# Temporal Platform Documentation

> This file is the complete text of Temporal's documentation, intended for bulk ingestion. Temporal is an open-source platform for building crash-proof applications that resume exactly where they left off after failures.

This file contains the full text of every documentation page, in one document.
Pages are separated by a horizontal rule and each carries a `Source:` URL.
For a structured index instead, see https://docs.temporal.io/llms.txt.

---

# What is a Temporal Activity?

Source: https://docs.temporal.io/activities

An Activity is a normal function or method (an [Activity Definition](/activity-definition)) that executes a single, well-defined action
(either short or long running), such as calling another service, transcoding a media file, or sending an email message.

Activity functions are registered by name (an [Activity Type](/activity-definition#activity-type)) on a Worker that polls a Task Queue for Activity Tasks to run.
Activity code can do whatever you want, use any library or package, and otherwise be non-deterministic.
We recommend that it be [idempotent](/activity-definition#idempotency), so retries can be processed without duplicate side effects.

Activities are started as a step in a Workflow (a [Workflow Activity](/workflow-activity)) or as a [Standalone Activity](/standalone-activity) that runs independently for [background job processing](/evaluate/features/job-queue).
This creates a durable [Activity Execution](/activity-execution) in Temporal that orchestrates the full lifecycle of an Activity, dispatching Tasks to the Activity Worker using its Task Queue and collecting results to determine next steps.

> **💡 Tip:**
>
> Watch a short overview of what an Activity is in Temporal:
>
> [Watch: What is an Activity in Temporal?](https://www.youtube.com/watch?v=rtWrzjnKlSQ)
>

Activities are the most common Temporal primitive and encompass small units of work such as:

- Single write operations, like updating user information or submitting a credit card payment
- Batches of similar writes, like creating multiple orders or sending multiple messages
- One or more read operations followed by a write operation, like checking a product status and user address before updating an order status
- A read that should be memoized, like an LLM call, a large download, or a slow-polling read

Larger pieces of functionality should be broken up into multiple Activities. This makes it easier to do failure recovery, have short timeouts, and be idempotent.

If an Activity attempt fails, it is automatically retried using its [Retry Policy](/activity-definition#activity-retry-policy).
Each attempt starts from the initial state, unless your code uses
a [Heartbeat detail payload](/encyclopedia/detecting-activity-failures#activity-heartbeat) for checkpointing. The
last recorded Heartbeat details are made available to the Activity function on the next attempt, so your code can
continue processing where it left off.

When specific performance optimizations are more important than full durability, an Activity may be executed in the same process as a Workflow (a [Local Activity](/local-activity)), bypassing the regular Activity Execution and Task Queue.
Local Activities are not a replacement for regular Activities.

Regardless of how you invoke an Activity, the [Activity Definition](/activity-definition) and Worker registration is the same.
You can write an Activity once, register it with a Worker, and invoke it as a Workflow Activity, a Standalone Activity or a Local Activity.

---

# Activity Definition

Source: https://docs.temporal.io/activity-definition

> Learn how to define a Temporal Activity; Activity Types, parameters, constraints, idempotency, and retry behavior for the function your Worker runs.

This page discusses the following:

- [Idempotency](#idempotency)
- [Parameters](#activity-parameters)
- [Activity Type](#activity-type)

An Activity Definition is a normal function or method that executes a single, well-defined action.
The same Activity Definition can be started as a [step in a Workflow](/workflow-activity) or as a background job using a [Standalone Activity](/standalone-activity) once it's been [registered with a Worker](#worker-registration).

An Activity Definition is the code that gives rise to an [Activity Task Execution](/tasks#activity-task-execution).

## Define an Activity function

Below are examples of basic Activity Definitions across supported SDKs.

**Go**

**[Activity Definition in Go](/develop/go/activities/basics)**

```go
import (
    "context"

    "go.temporal.io/sdk/activity"
)

func YourSimpleActivity(ctx context.Context) error {
    return nil
}
```

**Java**

**[Activity Definition in Java (Interface)](/develop/java/activities/basics)**

```java
@ActivityInterface
public interface GreetingActivities {
    @ActivityMethod
    String composeGreeting(String greeting, String language);
}
```

**[Activity Definition in Java (Implementation)](/develop/java/activities/basics)**

```java
static class GreetingActivitiesImpl implements GreetingActivities {
    @Override
    public String composeGreeting(String greeting, String name) {
        return greeting + " " + name + "!";
    }
}
```

**PHP**

**[Activity Definition in PHP (Interface)](/develop/php/activities/basics)**

```php
#[ActivityInterface]
interface GreetingActivities
{
    public function composeGreeting(string $greeting, string $name): string;
}
```

**[Activity Definition in PHP (Implementation)](/develop/php/activities/basics)**

```php
class GreetingActivitiesImpl implements GreetingActivities
{
    public function composeGreeting(string $greeting, string $name): string
    {
        return $greeting . ' ' . $name;
    }
}
```

**Python**

**[Activity Definition in Python](/develop/python/activities/basics)**

```python
from temporalio import activity

@activity.defn(name="your_activity")
async def your_activity(input: YourParams) -> str:
    return f"{input.greeting}, {input.name}!"
```

**TypeScript**

**[Activity Definition in TypeScript](/develop/typescript/activities/basics)**

```ts
export async function greet(name: string): Promise<string> {
  return `Hello, ${name}!`;
}
```

**.NET**

**[Activity Definition in C# and .NET](/develop/dotnet/activities/basics)**

```csharp
using Temporalio.Activities;

public class MyActivities
{
    [Activity]
    public string MyActivity(MyActivityParams input) =>
        $"{input.Greeting}, {input.Name}!";
}
```

**Rust**

**[Activity Definition in Rust](/develop/rust/activities/basics)**

```rust
use temporalio_sdk::activities::{ActivityContext, ActivityError};
use temporalio_macros::activities;

pub struct GreetingActivities;

#[activities]
impl GreetingActivities {
    #[activity]
    pub async fn greet(_ctx: ActivityContext, name: String) -> Result<String, ActivityError> {
        Ok(format!("Hello, {}!", name))
    }
}
```

> **💡 Tip:**
> GET STARTED
>
> Write an Activity Definition:
> [Go](/develop/go/activities/basics)
> | [Java](/develop/java/activities/basics)
> | [PHP](/develop/php/activities/basics)
> | [Python](/develop/python/activities/basics)
> | [TypeScript](/develop/typescript/activities/basics)
> | [.NET](/develop/dotnet/activities/basics)
> | [Ruby](/develop/ruby/activities/basics)
> | [Rust](/develop/rust/activities/basics)
>

### Idempotency 

Temporal recommends that Activities be idempotent.
Idempotence means that performing an operation multiple times has the same result as performing it once.
In the context of Temporal, Activities should be designed to be safely executed multiple times without causing unexpected or undesired side effects.
A few examples where idempotent operations are vital would be:

- **Infrastructure-as-Code (IaC) tool** - Conserving resources is important when you're provisioning infrastructure in the cloud. An IaC system that was not designed with idempotence in mind could lead to high costs if the function to provision a new server was accidentally invoked multiple times. An IaC tool that is designed with idempotence in mind ensures that multiple invocations of the tool doesn't lead to unintended instances being created.
- **Payment processing system** - A payment processing system must charge the customer only once for a given purchase. If the system was not designed to be idempotent, duplicate requests would result in extra charges and unhappy customers. A payment processing system that is designed to be idempotent ensures customers are not charged multiple times for the same transaction, preventing financial discrepancies.

> **ℹ️ Info:**
>
> By design, completed Activities will not re-execute as part of a [Workflow Replay](/workflow-execution#replay). However, Activities won’t record to the [Event History](/encyclopedia/retry-policies#event-history) until they return or produce an error. If an Activity fails to report to the server at all, it will be retried. Designing for idempotence, especially if you have a [Global Namespace](/global-namespace), will improve reusability and reliability.
>

An Activity is idempotent if multiple [Activity Task Executions](/tasks#activity-task-execution) do not change the state of the system beyond the first Activity Task Execution.

The lack of idempotency might affect the correctness of your application but does not affect the Temporal Platform.
In other words, lack of idempotency doesn't lead to a platform error.

In some cases, whether something is idempotent doesn't affect the correctness of an application.
For example, if you have a monotonically incrementing counter, you might not care that retries increment the counter because you don't care about the actual value, only that the current value is greater than a previous value.

You should always make your business logic Activities idempotent in Temporal. Because Activities may be retried, these functions may be executed more than once. A non-idempotent Activity could adversely affect the state of the system.

Activities are an atomic unit of execution within Temporal. They are invoked and either complete successfully or not. Take this into consideration when you design your Activities.

For example, consider an Activity that has the following three steps:

1. Perform a database lookup
2. Make a call to a microservice with parameters retrieved from the database
3. Write the result of the microservice call to the filesystem

Imagine that the first two steps succeed, but the third step fails due to a permissions issue. During retry, the entire Activity—and therefore each of the three steps—is executed again. To maintain idempotency, design your Activities to be more granular. In this case, you could have three Activities, one for each step. This way, only the step that failed will be executed again. However, you must balance this against the potential for a larger Event History, since there would now be three Activity Executions instead of one.

Idempotence for Activities is also important due to a particular edge case inherent in distributed computing. Consider a scenario in which a Worker polls the Temporal Service, accepts the Activity Task, and begins executing the Activity. The Activity function completes successfully, but the Worker crashes just before it notifies the Temporal Service. In this case, the Event History won’t reflect the successful completion of the Task, so the Activity will be retried. If the Activity is not idempotent, this could have negative consequences, such as duplicate charges in a payment processing scenario.

You can achieve idempotency in your application through the use of unique identifiers, known as idempotency keys, which are used to detect duplicate requests. These are enforced by the service you are calling from your Activity, not by the Activity itself.

For example, the APIs provided by most payment processors allow the client to include an idempotency key with the request. When the payment service receives a request, it checks a database to determine whether there has already been a request with this key. If so, the duplicate request is ignored and does not result in another charge. If not, then it writes a new record to the database with this key, allowing it to identify duplicate requests in the future.

In Temporal, the request to the payment service would be made from within an Activity. You can use a combination of the Workflow Run ID and the Activity ID as an idempotency key since this is guaranteed to be consistent across retry attempts but unique among Workflow Executions.

For more information about idempotency in Temporal, see the following post:

[Idempotency and Durable Execution](https://temporal.io/blog/idempotency-and-durable-execution)

### Activity retry policy

The Activity retry mechanism gives applications the benefits of durable execution.
For example, Temporal will keep track of the [exponential backoff delay](/encyclopedia/retry-policies#backoff-coefficient) even if the Worker crashes. Since Temporal can’t tell when a Worker crashes, Workflows rely on the [start_to_close timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) to know how long to wait before assuming that an Activity is inactive.

For an Activity with a [Retry Policy](/encyclopedia/retry-policies) that allows retries, Temporal guarantees that the Activity will be observed as completed exactly once. However, the Activity may be executed multiple times and may even partially complete more than once during this process. This could lead to a scenario where certain parts of the Activity are executed multiple times before a successful execution is completed.

> **⚠️ Caution:**
> You should typically not write retry logic manually within your Activity Definition. It lengthens the needed Activity timeout, prevents users from counting failure metrics, and makes it harder for users to debug in Temporal UI when something is wrong.

### Activity parameters 

An Activity Definition can use function/method parameters as usual for your language.
When called from a Workflow, the parameter values and return value are recorded in the [Event History](/workflow-execution/event#event-history) of the Workflow Execution.

## Activity Type 

An Activity Type is a name given to an Activity Definition.
When starting an Activity, you can identify it by the name (Activity Type), or by a reference to its function/method/class (Activity Definition)

## Register the Activity with a Worker 

An Activity Definition doesn't run until a [Worker](/workers) registers it and starts polling the
[Task Queue](/task-queue) its Activity Tasks are dispatched on. Registration maps the
[Activity Type](#activity-type) name to your function, so a caller can start the Activity by name without holding a
reference to the code.

One Worker can register many Activities, and one registration serves both callers: once the Worker is polling, the same
Activity can be started as a [Workflow Activity](/workflow-activity) or as a [Standalone Activity](/standalone-activity), with no
code change using the existing Worker deployment.

**Go**

**[Run a Worker in Go](/develop/go/workers/run-worker-process)**

```go
w := worker.New(c, "my-task-queue", worker.Options{})

w.RegisterActivity(helloworld.Activity)

err = w.Run(worker.InterruptCh())
```

**Java**

**[Run a Worker in Java](/develop/java/workers/run-worker-process)**

```java
WorkerFactory factory = WorkerFactory.newInstance(client);

Worker worker = factory.newWorker("my-task-queue");
worker.registerActivitiesImplementations(new GreetingActivitiesImpl());

factory.start();
```

**PHP**

**[Run a Worker in PHP](/develop/php/workers/run-worker-process)**

```php
$factory = WorkerFactory::create();

$worker = $factory->newWorker('my-task-queue');
$worker->registerActivity(App\DemoActivity::class);

$factory->run();
```

**Python**

**[Run a Worker in Python](/develop/python/workers/run-worker-process)**

```python
worker = Worker(
    client,
    task_queue="my-task-queue",
    activities=[some_activity],
)
await worker.run()
```

**TypeScript**

**[Run a Worker in TypeScript](/develop/typescript/workers/run-worker-process)**

```ts
const worker = await Worker.create({
  taskQueue: 'my-task-queue',
  activities,
});

await worker.run();
```

**.NET**

**[Run a Worker in C# and .NET](/develop/dotnet/workers/run-worker-process)**

```csharp
var options = new TemporalWorkerOptions("my-task-queue");
options.AddAllActivities(typeof(GreetingActivities), null);

using var worker = new TemporalWorker(client, options);
await worker.ExecuteAsync(CancellationToken.None);
```

**Ruby**

**[Run a Worker in Ruby](/develop/ruby/workers/run-worker-process)**

```ruby
worker = Temporalio::Worker.new(
  client: client,
  task_queue: 'my-task-queue',
  activities: [SayHello]
)

worker.run
```

**Rust**

**[Run a Worker in Rust](/develop/rust/workers/worker-process)**

```rust
let worker_options = WorkerOptions::new("my-task-queue")
    .register_activities(GreetingActivities)
    .build();
```

Go and TypeScript can set the [Activity Type](#activity-type) name at registration. The other SDKs set it on the
definition itself, as shown in the examples at the top of this page.

## Best practices for defining Activities

Here are some best practices you can use when you are creating Activities for your Workflow:

- Activity arguments and return values must be serializable.
- Activities that perform writes should be idempotent.
- Activities have [timeouts](/develop/python/activities/timeouts#activity-heartbeats) and [retry policies](/encyclopedia/retry-policies). For Activities, your operation should either complete within a few minutes or it should heartbeat. This way it will be clear to the Workflow when the Activity is still making progress.
- You need to specify at least one timeout, typically the [start_to_close timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). Keep in mind that the shorter the timeout, the faster Temporal will detect a problem and retry. See the [Activity retry policy section](#activity-retry-policy) to learn more.

---

# Activity Execution

Source: https://docs.temporal.io/activity-execution

> Learn how to start an Activity as a durable Activity Execution with built-in retries, timeouts, and failure handling.

This page discusses the following:

- [Activity Execution](#activity-execution)
- [Activity Execution Lifecycle](#activity-execution-lifecycle)
- [Cancellation](#cancellation)
- [Activity ID](#activity-id)
- [Asynchronous Activity Completion](#asynchronous-activity-completion)
- [Task Token](#task-token)

When an Activity caller starts an [Activity](/activities), an Activity Execution is created that orchestrates its full lifecycle, including retries.

[Workflow Activities](/workflow-activity) and [Standalone Activities](/standalone-activity) are durable Activity Executions that live in the Temporal Service.
They [dispatch Tasks to Activity Workers using an Activity Task Queue](/activity-execution#activity-execution-lifecycle) and collect results to determine next steps.

## How can an Activity be invoked? 

Once an Activity function has been [defined and registered in a Worker](/activity-definition) it can be invoked in different ways.

### Workflow Activity Executions 

A [Workflow Activity](/workflow-activity) is started as a step in a Workflow using the Temporal SDK and its lifetime is scoped to the Workflow.
The Workflow schedules the Activity, and the result is delivered back to the Workflow when the
Activity Execution closes. Each step is recorded in the Workflow Execution's
[Event History](/workflow-execution/event), for example the
Workflow [ActivityTaskCompleted](/references/events#activitytaskcompleted) Event. For other related Events, see
[Workflow Activity Events](/workflow-execution/event#activity-events).

> **💡 Tip:**
> GET STARTED
>
> Start a Workflow Activity:
> [Go](/develop/go/activities/execution)
> | [Java](/develop/java/activities/execution)
> | [PHP](/develop/php/activities/execution)
> | [Python](/develop/python/activities/execution)
> | [TypeScript](/develop/typescript/activities/execution)
> | [.NET](/develop/dotnet/activities/execution)
> | [Ruby](/develop/ruby/activities/execution)
> | [Rust](/develop/rust/activities/execution)
>

### Standalone Activity Executions 

A [Standalone Activity](/standalone-activity) is an independent top-level execution, started directly by a Temporal SDK [Client](/encyclopedia/temporal-client), with its own lifetime.
 The Client gets a handle and
fetches the result through it.
Because there's no Workflow, there is no Workflow Event History.
Standalone Activities use a [separate ID space](/standalone-activity#deduplication-and-id-space-uniqueness) for deduplication and uniqueness and provide full [visibility](/standalone-activity#visibility) and [lifecycle control](/standalone-activity#lifecycle-control).

| | [Standalone Activity](/standalone-activity) | [Workflow Activity](/workflow-activity) |
| --- | --- | --- |
| Started by | [Client](/encyclopedia/temporal-client) | Workflow code |
| Lifetime | Independent top-level execution | Scoped to the Workflow Run |
| Result delivered to | The Client, through a handle | The Workflow |
| Recorded in Event History | No Workflow, so no Event History | Yes |
| Activity ID space | [Separate ID space](/standalone-activity#deduplication-and-id-space-uniqueness) | Scoped to the Workflow Run |
| Best for | [Durable job processing](/evaluate/features/job-queue) | Multi-step orchestration |

> **💡 Tip:**
> GET STARTED
>
> Start a Standalone Activity:
> [Go](/develop/go/activities/standalone-activities-quickstart#execute-activity)
> | [Python](/develop/python/activities/standalone-activities-quickstart#execute-a-standalone-activity)
> | [Java](/develop/java/activities/standalone-activities-quickstart#execute-activity)
> | [.NET](/develop/dotnet/activities/standalone-activities-quickstart#execute-activity)
> | [TypeScript](/develop/typescript/activities/standalone-activities-quickstart#execute-activity-type-checking)
> | [Ruby](/develop/ruby/activities/standalone-activities-quickstart#execute-activity)
> | [Rust](/develop/rust/activities/standalone-activities-quickstart#execute-activity)
>

### Local Activity Executions 

When specific performance optimizations are more important than full durability, an Activity may be executed in the same process as a Workflow (a [Local Activity](/local-activity)), bypassing the regular Activity Execution and Task Queue.

> **💡 Tip:**
>
> [Local Activities](/local-activity) are not a replacement for regular Activities, since they bypass the regular Activity Execution lifecycle and lack full durability.
>

## Activity Execution lifecycle

[Workflow Activities](/workflow-activity) and [Standalone Activities](/standalone-activity) use the regular Activity Execution lifecycle, which includes the full chain of [Activity Task Executions](/tasks#activity-task-execution).

![Activity Execution](https://docs.temporal.io/diagrams/activity-execution.svg)

You can customize [Activity Execution timeouts](/encyclopedia/detecting-activity-failures#start-to-close-timeout) and
[retry policies](/encyclopedia/retry-policies).

If an Activity Execution fails (because it exhausted all retries, threw a
[non-retryable error](/encyclopedia/retry-policies#non-retryable-errors), or was canceled), the error is returned to your
[Workflow](/workflows) code when it attempts to fetch the Activity result. For [Standalone Activities](/standalone-activity) the error is
returned to the Client when you attempt to fetch the Activity result.

> **📝 Note:**
>
> Temporal guarantees that an Activity Task either runs or timeouts. There are multiple failure scenarios when an Activity
> Task is lost. It can be lost during delivery to a Worker or after the Activity Function is called and the Worker
> crashed.
>
> Temporal doesn't detect task loss directly. It relies on
> [Start-To-Close timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). If the Activity Task times
> out, the Activity Execution will be retried according to the Activity Execution Retry Policy.
>
> In scenarios where the Activity Execution Retry Policy is set to `1` and a Timeout occurs, the Activity Execution will
> not be tried.
>

## Cancellation 

Activity Cancellation:

- lets the Activity know it doesn't need to keep doing work, and
- gives the Activity time to clean up any resources it has created.

Activities must heartbeat to receive cancellations from a Temporal Service.

An Activity may receive Cancellation if:

- The Activity was requested to be Cancelled. This can often cascade from Workflow Cancellation, but not always—SDKs
  have ways to stop Cancellation from cascading. 
- The Activity was considered failed by the Server because any of the Activity timeouts have triggered (for example, the
  Server didn't receive a heartbeat within the Activity's Heartbeat timeout). The
  [Cancelled Failure](/references/failures#cancelled-failure) that the Activity receives will have
  `message: 'TIMED_OUT'`.
- The Workflow Run reached a [Closed state](/workflow-execution#workflow-execution-status), in which case the Cancelled
  Failure will have `message: 'NOT_FOUND'`.
- In some SDKs:
  - The Worker is shutting down.
  - An Activity sends a Heartbeat but the Heartbeat details can't be converted by the Worker's configured
    [Data Converter](/dataconversion). This fails the Activity Task Execution with an Application Failure.
  - The Activity timed out on the Worker side and is not Heartbeating or the Temporal Service hasn't relayed a
    Cancellation.

There are different ways to receive Cancellation depending on the SDK.  An Activity may
accept or ignore Cancellation:

- To allow Cancellation to happen, let the Cancellation Failure propagate.
- To ignore Cancellation, catch it and continue executing.

Some SDKs have ways to shield tasks from being stopped while still letting the Cancellation propagate.

The Workflow can also decide if it wants to wait for the Activity Cancellation to be accepted or to proceed without
waiting.

Cancellation can only be requested a single time. If you try to cancel your Activity Execution more than once, it will
not receive more than one Cancellation request.

## What is an Activity ID? 

The identifier for an [Activity Execution](#activity-execution). The identifier can be generated by the system, or it
can be provided by the Workflow code that spawns the Activity Execution. The identifier is unique among the open
Activity Executions of a [Workflow Run](/workflow-execution/workflowid-runid#run-id). (A single Workflow Run may reuse
an Activity ID if an earlier Activity Execution with the same ID has closed.)

An Activity ID can be used to [complete the Activity asynchronously](#asynchronous-activity-completion).

[Standalone Activities](/standalone-activity) have a [separate ID space](/standalone-activity#deduplication-and-id-space-uniqueness) from [the Workflow ID space](/workflow-execution/workflowid-runid) for deduplication and uniqueness.

## What is Asynchronous Activity Completion? 

Asynchronous Activity Completion is a feature that enables an Activity Function to return without causing the Activity
Execution to complete. The Temporal Client can then be used from anywhere to both Heartbeat Activity Execution progress
and eventually complete the Activity Execution and provide a result.

> **💡 Tip:**
> GET STARTED
>
> Complete an Activity Asynchronously:
> [Go](/develop/go/activities/asynchronous-activity)
> | [Java](/develop/java/activities/asynchronous-activity)
> | [PHP](/develop/php/activities/asynchronous-activity)
> | [Python](/develop/python/activities/asynchronous-activity)
> | [TypeScript](/develop/typescript/activities/asynchronous-activity)
> | [.NET](/develop/dotnet/activities/asynchronous-activity)
> | [Ruby](/develop/ruby/activities/asynchronous-activity)
>

### When to use Async Completion

When an external system has the final result of a computation that is started by an Activity, there are three main ways
of getting the result to the Workflow:

1. The external system uses Async Completion to complete the Activity with the result.
2. The Activity completes normally, without the result. Later, the external system sends a Signal to the Workflow with
   the result.
3. A subsequent Activity
   [polls the external system](https://community.temporal.io/t/what-is-the-best-practice-for-a-polling-activity/328/2)
   for the result.

If you don't have control over the external system — that is, you can't add Async Completion or a Signal to its code —
then:

- you can poll (#3), or
- if the external system can reliably call a webhook (and retry calling in the case of failure), you can write a webhook
  handler that sends a Signal to the Workflow (#2).

The decision between using #1 vs #2 involves a few factors. Use Async Completion if:

- the external system is unreliable and might fail to Signal, or
- you want the external process to Heartbeat or receive Cancellation.

Otherwise, if the external system can reliably be trusted to do the task and Signal back with the result, and it doesn't
need to Heartbeat or receive Cancellation, then you may want to use Signals.

The benefit to using Signals has to do with the timing of failure retries. For example, consider an external process
that is waiting for a human to review something and respond, and they could take up to a week to do so. If you use Async
Completion (#1), you would:

- set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) of one week on the
  Activity,
- in the Activity, notify the external process you need the human review, and
- have the external process Asynchronously Complete the Activity when the human responds.

If the Activity fails on the second step to notify the external system and doesn't throw an error (for example, if the
Worker dies), then the Activity won't be retried for a week, when the Start-To-Close Timeout is hit.

If you use Signals, you would:

- set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) of one minute on the
  Activity,
- in the Activity, notify the external process you need the human review,
- complete the Activity without the result, and
- have the external process Signal the Workflow when the human responds.

If the Activity fails on the second step to notify the external system and doesn't throw an error, then the Activity
will be retried in a minute.

In the second scenario, the failure is retried sooner. This is particularly helpful in scenarios like this in which the
external process might take a long time.

### What is a Task Token? 

A Task Token is a unique identifier for an [Activity Task Execution](/tasks#activity-task-execution).

[Asynchronous Activity Completion](#asynchronous-activity-completion) calls take either of the following as arguments:

- a Task Token, or
- an [Activity ID](#activity-id), a [Workflow ID](/workflow-execution/workflowid-runid#workflow-id), and optionally a
  [Run ID](/workflow-execution/workflowid-runid#run-id).

Since a Task Token is unique for an Activity execution, retries can cause a remote service using the Task Token to end
up with an invalid one. For example, an Activity might fail after passing its current Task Token to a remote service,
but before returning the complete async error, leaving that service with a Task Token that's no longer valid.

To avoid this risk, you can provide the Activity ID and Workflow ID to the remote service instead of the Task Token.

---

# Activity Operations

Source: https://docs.temporal.io/activity-operations

> Learn how to manage a running Activity Execution with commands to Pause, Unpause, Reset, Update Options, Request Cancel, Terminate, and Delete.

This section covers the following:

- [Pause](/activity-operations/pause)
- [Unpause](/activity-operations/unpause)
- [Reset](/activity-operations/reset)
- [Update Options](/activity-operations/update-options)
- [Request Cancel](/activity-operations/request-cancel)
- [Terminate](/activity-operations/terminate)
- [Delete](/activity-operations/delete)
- [Batch operations](#batch-operations)
- [Observability](#observability)

Activity Operations are deliberate actions you perform on a specific [Activity Execution](/activity-execution), as
opposed to lifecycle behaviors like [retries](/encyclopedia/retry-policies) and
[timeouts](/encyclopedia/detecting-activity-failures) which happen automatically.

You can perform Activity Operations through the [CLI](/cli/command-reference/activity), the UI, or directly via the gRPC
API. They apply to Workflow Activities and to [Standalone Activities](/standalone-activity). They don't apply to
[Local Activities](/local-activity).

> **📝 Note:**
> Public Preview
>
> Activity Operations are in [Public Preview](/evaluate/product-release-stages#public-preview), 
> except for Standalone Activity commands: Request Cancel, Terminate, Delete which are GA.
>
> For [Workflow Activities](/workflow-activity), Pause, Unpause, and Reset are available in Server
> v1.28.0+. Self-hosted UI requires v2.47.0+. For [Standalone Activities](/standalone-activity), Pause,
> Unpause, Reset, and Update Options are available in Server v1.32.0+ and Temporal CLI v1.9.1+.
>
> Activity Operations aren't available as SDK client methods. They're operational controls designed for the CLI, UI, and
> gRPC API - they are not for programmatic use in Workflow or Activity code.
>

## Operations summary

| Operation                                             | What it does                                                                     | Workflow Activity    | Standalone Activity | CLI                                                                                  |
| ----------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------- | ------------------- | ------------------------------------------------------------------------------------ |
| [Pause](/activity-operations/pause)                   | Stops retries. In-flight execution continues unless the Activity uses Heartbeat. | Yes                  | Yes                 | [`temporal activity pause`](/cli/command-reference/activity#pause)                   |
| [Unpause](/activity-operations/unpause)               | Resumes a Paused Activity. The next execution starts immediately.                | Yes                  | Yes                 | [`temporal activity unpause`](/cli/command-reference/activity#unpause)               |
| [Reset](/activity-operations/reset)                   | Clears retry state (attempts, backoff) and schedules a new execution.            | Yes                  | Yes                 | [`temporal activity reset`](/cli/command-reference/activity#reset)                   |
| [Update Options](/activity-operations/update-options) | Changes timeouts, Retry Policy, or Task Queue without restarting the Activity.   | Yes                  | Yes                 | [`temporal activity update-options`](/cli/command-reference/activity#update-options) |
| [Request Cancel](/activity-operations/request-cancel) | Requests that an execution close gracefully, letting your code clean up.         | Through the Workflow | Yes                 | [`temporal activity cancel`](/cli/command-reference/activity#cancel)                 |
| [Terminate](/activity-operations/terminate)           | Forcefully closes an execution with no opportunity for your code to clean up.    | No                   | Yes                 | [`temporal activity terminate`](/cli/command-reference/activity#terminate)           |
| [Delete](/activity-operations/delete)                 | Terminates the execution if it's running, then deletes it asynchronously.        | No                   | Yes                 | [`temporal activity delete`](/cli/command-reference/activity)                        |

A Workflow Activity can't be cancelled directly. It receives a cancellation request as a result of its
[Workflow](/workflows) being cancelled, and from that point behaves the same way a Standalone Activity does.

### What an operation guarantees

Every operation does two things, and they succeed independently.

**Server-side state changes immediately.** Terminate closes the execution. Request Cancel closes it immediately when no
attempt is running. This doesn't depend on the Activity Heartbeating.

**Interrupting an already-running attempt is best-effort.** Request Cancel, Terminate, Reset, and Pause all attempt it,
by the same mechanism: the request reaches your code through the Activity's Heartbeat. An Activity that doesn't
Heartbeat isn't interrupted mid-attempt.

Because interruption is best-effort, **a Request Cancel, Reset, or Pause request can succeed without the operation
taking effect.** When an attempt is running, your code may complete or fail non-retryably instead of honoring the
request. Only Terminate and Delete discard Activity progress unconditionally. A successful response means the request
was accepted, not that the Activity stopped.

## Successive operations 

When operations conflict, precedence is **Request Cancel, then Reset, then Pause**. A higher-precedence request wins
over a pending lower-precedence one.

Requests that can't apply to the Activity's current state return an error rather than being queued:

- `FailedPrecondition` when the Activity exists but isn't in a state that accepts the operation.
- `NotFound` when the Activity or its run can't be found.

Pause and Unpause interact with timers in a way worth knowing before you use them together:

- **Timers keep running while an Activity is Paused.** Pausing doesn't stop the Schedule-To-Close Timeout.
- **No new attempt is scheduled while Paused.**
- **On Unpause, a retry that's already past due starts immediately.**
- **Before the first attempt, Unpause honors the Activity's original Start Delay deadline.** If that deadline has
  passed, the Activity is dispatched immediately. Start Delay isn't restarted from the Unpause time, and it doesn't
  apply to retry attempts.

## Batch operations 

You can apply some operations to many Activities at once with a `--query` [List Filter](/list-filter) instead of a
single Activity ID.

**Standalone Activities** currently only support batch operations for Request Cancel, Terminate, and Delete:

```bash
temporal activity terminate \
  --query 'ActivityType="ProcessImage" AND ExecutionStatus="Running"' \
  --reason "Bad input batch"
```

For **Workflow Activities**, `--query` applies to Reset, Unpause, and Update Options.

Use `--jitter` to stagger a batch so a recovering downstream service isn't hit by every retry at once.

## Billable Actions 

In Temporal Cloud, Pause, Reset, and Update Options each count as one
[Action](/evaluate/cloud/actions#activity). Unpause is free.

## Observability 

Activity Operations have a limited audit trail because they are not recorded in a Workflow's Event History. However, you
can use the CLI and the UI to check Activity state and find Paused Activities for running Workflows.

### Check Activity state

`temporal workflow describe` shows the current state of each pending Activity, including whether it's Paused, its
current attempt count, and last failure. The UI shows who performed an operation, when, and why (if a `--reason` was
provided).

### Find Paused Activities

The `TemporalPauseInfo` [Search Attribute](/search-attribute) is filterable within a Workflow.

There's no Namespace-wide query to find all Paused Activities across Workflows. You must know the Workflow ID.

### Audit trail 

Activity Operations don't produce Event History events. There is no record of a Pause, Reset, or option change in the
Workflow's [Event History](/workflow-execution/event#event-history). Nothing that reads the Event History - Workflow
code, Replays, or external tooling - will see that an Operation occurred.

Evidence of an Operation is gone when the Activity completes or the Workflow closes. There's no persistent record that
an Activity was Paused, Reset, or had its options changed.

The only way to confirm the current state of an Activity is `temporal workflow describe` or the UI.

---

# Delete an Activity

Source: https://docs.temporal.io/activity-operations/delete

> Terminate a Standalone Activity Execution if it's still running, then remove its record from the Temporal Service asynchronously.

Delete terminates the Activity Execution if it's running, then deletes it asynchronously.

## When to Delete

- You need an Execution and its result removed before the Namespace
  [Retention Period](/temporal-service/temporal-server#retention-period) elapses.
- You want to free an Activity ID for reuse without waiting out retention. See
  [Activity ID Reuse Policy](/standalone-activity#activity-id-reuse-policy).

Delete isn't available for [Workflow Activities](/workflow-activity).

## What happens when you Delete an Activity

If the Execution is still running, it's terminated first, with the same semantics as [Terminate](/activity-operations/terminate): no
cleanup runs and progress is discarded. The Execution record is then removed asynchronously, so it may remain visible
briefly after the command returns.

Once deleted, the Execution no longer appears in `temporal activity describe` or `temporal activity list`, and its
Activity ID becomes available for reuse.

## CLI usage

```bash
temporal activity delete \
  --activity-id my-activity
```

See the [CLI reference for `temporal activity`](/cli/command-reference/activity) for all options.

## Important considerations

- **Delete is irreversible.** The Execution, its inputs, and its result are gone; there's no recovery.
- **Deletion is asynchronous.** A successful response means the request was accepted, not that the record is already
  removed.

---

# Pause an Activity

Source: https://docs.temporal.io/activity-operations/pause

> Stop an Activity's retries without closing the Execution, detect a Pause from Activity code, and resume it later with Unpause.

Pause stops the Temporal Service from scheduling new retries of an [Activity Execution](/activity-execution).

## When to Pause

- An Activity is calling an external service that's experiencing issues, and you want to stop retries until the service
  recovers.
- You need to inspect or change configuration before the Activity retries.
- You're rolling out a new Worker version and want to hold specific Activities until the deploy is complete.

## What happens when you Pause an Activity

- **Pausing an Activity doesn't affect the parent Workflow.** The Workflow continues Running, and
  [Signals, Queries, and Updates](/encyclopedia/workflow-message-passing) on the parent Workflow are unaffected.
- **No further retries are scheduled.** The Temporal Service stops scheduling retries. This is enforced server-side, not
  by the SDK.
- **Workflow code has no visibility into Activity Operations.** Pause doesn't produce an Event History event, so the
  Workflow can't detect or react to it. See [Observability](/activity-operations#observability).
- **[Heartbeating](/encyclopedia/detecting-activity-failures#activity-heartbeat) determines whether the in-flight
  execution is interrupted:**
  - **Activities with Heartbeat** are interrupted on their next Heartbeat. The SDK raises a Pause-specific error, and
    the Activity can catch this to clean up resources before exiting.
  - **Activities without Heartbeat** continue running to completion. If the execution succeeds, the result is delivered
    to the Workflow normally. If it fails, no retry is scheduled. Pause takes effect after the in-flight execution ends.
- **Pause is idempotent.** Pausing an already-Paused Activity has no effect. Pausing a completed Activity returns an
  error.

## CLI usage

```bash
temporal activity pause \
  --workflow-id my-workflow \
  --activity-id my-activity \
  --reason "Downstream API is down, pausing until recovery"
```

To target a Standalone Activity, omit `--workflow-id`:

```bash
temporal activity pause \
  --activity-id my-activity \
  --reason "Downstream API is down, pausing until recovery"
```

See the [CLI reference for `temporal activity pause`](/cli/command-reference/activity#pause) for all options.

## Detect Pause in Activity code

Activities with Heartbeat can detect that an interruption was caused by Pause rather than a timeout or Workflow
Cancellation. A Paused Activity resumes later. A Cancelled Activity doesn't. Your Activity code may need to handle these
cases differently, for example releasing held resources on Pause while preserving them on Cancellation, or vice versa.

| SDK        | Version  | How to detect Pause                                                  |
| ---------- | -------- | -------------------------------------------------------------------- |
| Go         | v1.34.0+ | Catch `activity.ErrActivityPaused`                                   |
| Java       | v1.29.0+ | Catch `ActivityPausedException`                                      |
| TypeScript | v1.12.3+ | Check `cancellationDetails.paused === true`                          |
| Python     | v1.12.0+ | Check `cancellation_details().paused` on `asyncio.CancelledError`    |
| .NET       | v1.7.0+  | Check `CancellationDetails.IsPaused` on `OperationCanceledException` |

## Interaction with Workflow Pause

[Workflow Pause](/encyclopedia/workflow/workflow-pause) and Activity Pause are independent. Both stop Activity retries,
but they must be Unpaused separately.

- Workflow Pause blocks retries but doesn't interrupt in-flight executions via Heartbeat. Activity Pause does.
- If both are active, both must be Unpaused before the Activity resumes.

## Important considerations

- **A Paused Activity can still time out.** Pause doesn't stop or extend the
  [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout). Use
  [`update-options`](/activity-operations/update-options) to adjust the timeout if needed.
- **Pause won't interrupt an Activity that doesn't Heartbeat.** The current execution runs to completion, which could
  take up to the full [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).

## Limitations

- **Pause operates on individual Activities.** There's no `--query` flag on `pause`, so there's no batch form. To pause
  multiple Activities, issue separate commands for each Activity ID. See [Batch operations](/activity-operations#batch-operations).
- **No Namespace-wide query for Paused Workflow Activities.** You must know the Workflow ID. See
  [Observability](/activity-operations#observability).

---

# Request Cancel for an Activity

Source: https://docs.temporal.io/activity-operations/request-cancel

> Ask an Activity Execution to close gracefully so your code can clean up, and what to expect when an attempt is already running.

Request Cancel asks an Activity Execution to close gracefully, giving your code a chance to clean up.

## When to Request Cancel

- A [Standalone Activity](/standalone-activity) is no longer needed, and you want it to stop at a safe point rather
  than be killed mid-attempt.
- A job was submitted in error, and you want the Activity to release the resources it has already acquired.

A [Workflow Activity](/workflow-activity) can't be canceled directly. It receives a cancellation request when its
Workflow is canceled, and from that point behaves the same way a Standalone Activity does.

## What happens when you Request Cancel an Activity

The Activity Execution transitions to `CancelRequested`.

- **If no attempt is running,** the execution closes immediately. This doesn't depend on the Activity Heartbeating.
- **If an attempt is running,** the request reaches your code through the Activity's
  [Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat). A Cancellation error is raised when the
  next Heartbeat response is received, and the Activity transitions to canceled status if your code lets that error
  propagate.
- **If the Activity doesn't Heartbeat,** it isn't interrupted mid-attempt. The request takes effect at the next attempt
  boundary, for example when the
  [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) elapses.

See [Cancellation](/activity-execution#cancellation) for how your Activity code accepts or ignores a Cancellation.

## CLI usage

```bash
temporal activity cancel \
  --activity-id my-activity \
  --reason "No longer needed"
```

See the [CLI reference for `temporal activity cancel`](/cli/command-reference/activity#cancel) for all options.

## Important considerations

- **A successful response means the request was accepted, not that the Activity stopped.** Your code may complete or
  fail non-retryably instead of honoring the request.
- **Cancellation can be requested only once.** Repeating the request doesn't deliver a second Cancellation.
- **Request Cancel takes precedence** over Reset and Pause. See [Successive operations](/activity-operations#successive-operations).

---

# Reset an Activity

Source: https://docs.temporal.io/activity-operations/reset

> Clear an Activity's retry state and schedule a fresh attempt, including what happens when an attempt is already running.

Reset clears an Activity's retry state and schedules a fresh execution.

## When to Reset

- An Activity has exhausted most of its retries, and you want to give it a fresh set after fixing the underlying issue.
- A Paused Activity needs to start clean after a configuration change or code deploy.
- You want to clear accumulated retry backoff and retry immediately instead of waiting for the next backoff interval.
- A batch of Activities failed due to a transient issue and you want to restart them all with staggered jitter.

## What happens when you Reset an Activity

- **The attempt count resets to 1.** The Activity gets a full set of retry attempts regardless of how many it had used.
- **Heartbeat details are preserved.** The new attempt starts with the last recorded Heartbeat details available, so
  your code can use a checkpoint it previously saved in Heartbeat details. Pass `--clear-heartbeat-details` to discard
  them instead.
- **Per-attempt timeouts are re-armed.** They restart for the new attempt rather than being removed.
- **Retry backoff is discarded.** If the Activity is between attempts, waiting out a backoff, the new attempt is
  dispatched right away. If an attempt is running, see
  [Reset while an attempt is running](#reset-while-an-attempt-is-running).
- **If the Activity is Paused, Reset also Unpauses it.** Use `--keep-paused` to Reset the attempt count without resuming
  execution. With `--keep-paused`, the attempt count is reset but the Activity stays Paused. No retry is scheduled
  until you [Unpause](/activity-operations/unpause) separately.
- **Resetting an Activity doesn't affect the parent Workflow.** The Workflow continues Running, and Signals, Queries,
  and Updates on the parent Workflow are unaffected.
- **Workflow code has no visibility into Activity Operations.** Reset doesn't produce an Event History event, so the
  Workflow can't detect or react to it. See [Observability](/activity-operations#observability).
- **Reset is idempotent.** Resetting an Activity that's already at attempt 1 with no backoff has no effect. Resetting a
  completed Activity returns an error.

### Reset while an attempt is running

Reset is handled cooperatively.

- The reset request is delivered to the running attempt through the Activity's
  [Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat), if the Activity heartbeats.
- Your Worker can accept the request and stop processing the current attempt, or carry on and complete the Activity
  successfully.
- The Temporal Service processes the reset once the current attempt finishes, including by hitting its
  [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). A new attempt then
  starts and a new Activity Task is dispatched to a Worker.

Reset never dispatches a new attempt while one is still running, and it never runs two attempts concurrently. This is
the same for Workflow Activities and Standalone Activities.

## CLI usage

```bash
temporal activity reset \
  --workflow-id my-workflow \
  --activity-id my-activity

# Reset retry state but don't resume yet
temporal activity reset \
  --workflow-id my-workflow \
  --activity-id my-activity \
  --keep-paused
```

To target a Standalone Activity, omit `--workflow-id`:

```bash
temporal activity reset \
  --activity-id my-activity
```

See the [CLI reference for `temporal activity reset`](/cli/command-reference/activity#reset) for all options.

## Detect Reset in Activity code

Activities with Heartbeat can detect that an interruption was caused by Reset rather than a timeout or Workflow
Cancellation. A Reset Activity is retried from attempt 1. A Cancelled Activity isn't. Your Activity code may need to
handle these cases differently, for example saving partial progress on Reset while discarding it on Cancellation.

| SDK        | How to detect Reset                                                                |
| ---------- | ---------------------------------------------------------------------------------- |
| Go         | `activity.GetCancellationDetails(ctx).Cause()` returns `activity.ErrActivityReset` |
| Java       | Catch `ActivityResetException`                                                     |
| TypeScript | Catch `ApplicationFailure` with `error.type === "ActivityReset"`                   |
| Python     | Check `cancellation_details().reset` on `asyncio.CancelledError`                   |
| .NET       | Check `CancellationDetails.IsReset` on `OperationCanceledException`                |

## Important considerations

- **A Reset Activity can still time out.** Reset doesn't restart the
  [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout). The deadline is
  calculated from when the Activity was originally scheduled. Use [`update-options`](/activity-operations/update-options) to extend the
  timeout before or after Reset.
- **Heartbeat details survive a Reset.** If your Activity uses Heartbeat details for progress tracking, the new attempt
  still has the last recorded details, so your code can use a checkpoint it previously saved in them. Pass
  `--clear-heartbeat-details` when you want the new attempt to start over from the beginning.
- **Reset won't reach an Activity that doesn't Heartbeat.** The request has no way to be delivered, so the current
  attempt runs to completion, which could take up to the full
  [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). The reset still applies
  afterwards, if the Activity is still Open. See
  [Reset while an attempt is running](#reset-while-an-attempt-is-running).
- **`--restore-original-options` restores the Activity's original configuration.** It reverts timeouts, Retry Policy,
  and Task Queue to the values from when the Activity was first scheduled.
- **Bulk Reset can overwhelm downstream services.** When using `--query` to Reset Activities across many Workflows, use
  `--jitter` to stagger the restart times.

---

# Terminate an Activity

Source: https://docs.temporal.io/activity-operations/terminate

> Forcefully close a Standalone Activity Execution immediately, with no opportunity for your Activity code to clean up first.

Terminate forcefully closes an Activity Execution with no opportunity for your code to clean up.

## When to Terminate

- A [Standalone Activity](/standalone-activity) is stuck or misbehaving and you need it closed now, whether or not it
  Heartbeats.
- A Request Cancel was already sent and the Activity didn't honor it.

Terminate isn't available for [Workflow Activities](/workflow-activity), because the Workflow owns the execution's
lifetime.

## What happens when you Terminate an Activity

The Activity Execution closes immediately and discards its progress. Activity code can't see or respond to a
termination, so no cleanup runs and no Cancellation error is raised. Unlike Request Cancel, this doesn't depend on the
Activity Heartbeating, and it can't be declined by your code.

The Execution and its result remain visible to `temporal activity describe` and `temporal activity list` for the
Namespace [Retention Period](/temporal-service/temporal-server#retention-period). To remove the record sooner, use
[Delete](/activity-operations/delete).

## CLI usage

```bash
temporal activity terminate \
  --activity-id my-activity \
  --reason "Bad input"
```

`--reason` defaults to a message naming the current user. See the
[CLI reference for `temporal activity terminate`](/cli/command-reference/activity#terminate) for all options.

## Important considerations

- **Terminate discards Activity progress unconditionally.** Prefer [Request Cancel](/activity-operations/request-cancel) when your code
  needs to release resources or record a checkpoint.
- **Terminating doesn't delete the record.** The closed Execution stays queryable until the Retention Period elapses.

---

# Unpause an Activity

Source: https://docs.temporal.io/activity-operations/unpause

> Resume a Paused Activity Execution so its next attempt starts immediately, with options to reset retry state as it resumes.

Unpause resumes a Paused Activity Execution.

## When to Unpause

- The downstream service or dependency that caused you to Pause has recovered.
- A code deploy or configuration change is complete and the Activity is safe to retry.
- You Paused an Activity for investigation and are ready to let it proceed.

## What happens when you Unpause an Activity

- **The Activity is rescheduled immediately.** Any remaining retry backoff is discarded. The next execution starts right
  away.
- **Attempt count, Heartbeat details, and timeouts are preserved.** The Activity resumes from where it left off. Use
  [Reset](/activity-operations/reset) to restart from attempt 1.

Unpause is idempotent. Unpausing an Activity that isn't Paused has no effect. Unpausing an Activity that has already
completed returns an error.

## CLI usage

```bash
temporal activity unpause \
  --workflow-id my-workflow \
  --activity-id my-activity
```

To target a Standalone Activity, omit `--workflow-id`:

```bash
temporal activity unpause \
  --activity-id my-activity
```

See the [CLI reference for `temporal activity unpause`](/cli/command-reference/activity#unpause) for all options.

## Important considerations

- **Unpausing many Activities at once can overwhelm downstream services.** If you Paused multiple Activities because a
  service was down, Unpausing them all at the same time sends all retries simultaneously. Consider Unpausing in batches
  to avoid overwhelming a recovering service.
- **Unpausing doesn't override Workflow Pause.** If the parent Workflow is also Paused, Unpausing the Activity alone
  isn't enough. Both must be Unpaused before the Activity resumes. See
  [Interaction with Workflow Pause](/activity-operations/pause#interaction-with-workflow-pause).
- **Unpausing doesn't reset the attempt count.** The Activity retries from its current attempt number. Use
  [Reset](/activity-operations/reset) to restart from attempt 1.
- **A Paused Activity can time out before you Unpause it.** The
  [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) isn't stopped or
  extended while Paused. Use [`update-options`](/activity-operations/update-options) to extend the timeout before Unpausing if needed.
- **Unpause doesn't interrupt or duplicate an in-flight execution.** If an Activity without Heartbeat is still running
  when you Unpause, it continues to completion. The Temporal Service doesn't schedule a concurrent execution. If the
  in-flight execution fails, the next retry proceeds normally.

---

# Update Activity Options

Source: https://docs.temporal.io/activity-operations/update-options

> Change an Activity's timeouts, Retry Policy, or Task Queue while it runs, without restarting the Execution or losing retry state.

Update Options changes an Activity's runtime configuration without restarting it.

## When to Update Options

- The [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) is about to
  expire on a Paused Activity, and you need to extend it before Unpausing.
- An Activity's [Retry Policy](/encyclopedia/retry-policies) needs tuning based on observed failure patterns (for
  example, increasing the backoff interval or maximum attempts).
- You want to move an Activity to a different [Task Queue](/task-queue) to route it to a specific set of
  [Workers](/workers).
- You need to restore an Activity's original configuration after a temporary override.

## What happens when you Update an Activity's Options

You can change [timeouts](/encyclopedia/detecting-activity-failures) (Schedule-To-Close, Start-To-Close,
Schedule-To-Start, Heartbeat), Retry Policy (initial interval, maximum interval, backoff coefficient, maximum attempts),
and Task Queue. Only the fields you specify are changed. All other options remain unchanged.

- **If the Activity is waiting for retry (scheduled),** the new options take effect immediately. Any pending retry timer
  is regenerated with the updated configuration.
- **If the Activity is currently running,** the new options are stored but take effect on the next execution. The
  in-flight execution isn't interrupted.
- **If the Activity is Paused,** the new options are stored immediately. They take effect when the Activity is Unpaused
  and the next execution starts.
- **Workflow code has no visibility into Activity Operations.** Update Options doesn't produce an Event History event,
  so the Workflow can't detect or react to it. See [Observability](/activity-operations#observability).

Update Options is idempotent. Updating an Activity with the same values it already has produces no change. Updating
options on an Activity that has already completed returns an error.

## CLI usage

```bash
temporal activity update-options \
  --workflow-id my-workflow \
  --activity-id my-activity \
  --schedule-to-close-timeout 24h
```

To target a Standalone Activity, omit `--workflow-id`:

```bash
temporal activity update-options \
  --activity-id my-activity \
  --schedule-to-close-timeout 24h
```

See the [CLI reference for `temporal activity update-options`](/cli/command-reference/activity#update-options) for all
options, including Retry Policy and Task Queue.

## Important considerations

- **Changes to a running Activity take effect on the next execution, not the current one.** If you need the change to
  apply immediately, the Activity must finish or fail its current execution first.
- **`--restore-original-options` is batch-only.** This flag only works with `--query`. It's silently ignored in
  single-workflow mode. It can't be combined with other option changes in the same command.
- **Restoring original options requires a stored snapshot.** For Activities that started before your Temporal Service
  supported Activity Operations, no snapshot exists and the request is rejected.

---

# Durable AI

Source: https://docs.temporal.io/ai

> Build durable AI agents and systems on Temporal, with runnable cookbook recipes, SDK integrations, and design patterns for agent workloads.

Temporal gives AI applications and agents Durable Execution: a Workflow resumes automatically after a crash, a
network timeout, or a multi-day wait for a human to approve a step. Temporal shows up in four recurring types of AI
system:

**Agents.** Long-running, stateful agent loops that call LLMs and tools, wait on humans, and pick up exactly where
they left off after a failure. Start with the [AI Cookbook](/ai/cookbook) and the
[Approval](/design-patterns/approval) and [Entity Workflow](/design-patterns/entity-workflow) patterns.

**Processing pipelines.** Multi-step data and document pipelines, such as extraction, embedding, or batch inference,
that need to fan out, retry failed steps in isolation, and resume without reprocessing completed work. When pipelines
share Workers across models, tenants, or urgency levels, use
[Task Queue Priority and Fairness](/develop/task-queue-priority-fairness) to run urgent work ahead of bulk work and
keep one tenant from starving the others. See the [batch processing](/design-patterns#batch-processing-patterns) and
[QoS and throughput](/design-patterns/qos-throughput-patterns) patterns.

**Internal agent platforms.** Teams building a shared runtime for many agents reuse Temporal's Worker and Task Queue
primitives instead of building their own scheduler. See the
[worker configuration patterns](/design-patterns#worker-configuration-patterns) for routing and isolating agent
workloads.

**Model training.** Long-running training and fine-tuning jobs coordinated across GPU resources, with checkpointing
and recovery handled by Temporal's Event History instead of custom orchestration code. Start with the [long-running activity](/design-patterns/long-running-activity) and [parallel-execution](/design-patterns/parallel-execution) patterns. 

> Looking to use an AI coding assistant to write Temporal code instead? See [Develop with AI](/with-ai).

## AI Cookbook

Runnable, step-by-step recipes for building AI systems and agents with Temporal: tool calling, MCP, structured
output, human-in-the-loop, and more.

- [Hello world](/ai/cookbook/hello-world-openai-responses-python) — Call an LLM from a durable Temporal Workflow in Python using the OpenAI API library.
- [Hello world with LiteLLM](/ai/cookbook/hello-world-litellm-python) — Integrate LiteLLM into a durable Temporal Workflow in Python to call and switch between LLM providers.
- [Durable agent with tools using the AI SDK by Vercel](/ai/cookbook/ai-sdk-by-vercel-typescript) — Build a durable AI agent with the AI SDK by Vercel and Temporal that chooses tools to answer user questions.
- [Structured outputs with Temporal and OpenAI](/ai/cookbook/structured-output-openai-responses-python) — Use Temporal and the OpenAI Responses API to reliably request output conforming to a specific data structure.

[Browse all recipes](/ai/cookbook)

## Featured videos

- [Agent memory and state](/ai/featured/memory-state-stripe): kill a refund agent mid-task, with and without Temporal, to see
  what context, memory, and state each preserve.
- [AI agent gateway with Temporal Nexus](/ai/featured/agent-gateway-nexus): route every agent tool call through one gateway with
  policy checks, human approval, and crash recovery.

## Agent framework integrations

Temporal integrations for the SDKs and frameworks teams use to build agents. This view is pre-filtered to agent
frameworks — browse [every integration](/integrations) for the full catalog.

- [AI SDK by Vercel](/develop/typescript/integrations/ai-sdk) — Build AI-powered applications with Durable Execution using the Vercel AI SDK. _(TypeScript · Agent framework)_
- [Deep Agents](/develop/python/integrations/deepagents) — Make LangChain Deep Agents durable with Temporal Workflows and Activities. _(Python · Agent framework)_
- [Google ADK](/develop/python/integrations/google-adk) — Run Google ADK agents as durable Temporal Workflows. _(Python · Agent framework)_
- [Google ADK](/develop/go/integrations/google-adk) — Run Google ADK agents as durable Temporal Workflows. _(Go · Agent framework)_
- [Google ADK](/develop/typescript/integrations/google-adk-agents) — Run Google ADK agents as durable Temporal Workflows. _(TypeScript · Agent framework)_
- [Google GenAI](/develop/python/integrations/google-genai) — Call Google Gemini models durably from Temporal Workflows with the Google Gen AI SDK. _(Python · Agent framework)_
- [LangGraph](/develop/python/integrations/langgraph) — Run LangGraph agent graphs as durable, resumable Temporal Workflows. _(Python · Agent framework)_
- [Mastra](https://mastra.ai/integrations/deploy/temporal) — Build durable AI agents and workflows with the Mastra TypeScript framework. _(TypeScript · Agent framework)_
- [OpenAI Agents SDK](/develop/python/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(Python · Agent framework)_
- [OpenAI Agents SDK](/develop/typescript/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(TypeScript · Agent framework)_
- [Pydantic AI](https://pydantic.dev/docs/ai/capabilities/durable_execution/temporal/) — Build type-safe AI agents with Durable Execution through Pydantic AI. _(Python · Agent framework)_
- [Spring AI](/develop/java/integrations/spring-ai) — Build AI-powered Java applications with durable Spring AI tool calls. _(Java · Agent framework)_
- [Strands Agents](/develop/python/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(Python · Agent framework)_
- [Strands Agents](/develop/typescript/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(TypeScript · Agent framework)_

## Featured from the Code Exchange

A hand-picked look at samples built with Temporal and AI. Browse the full
[Code Exchange](https://temporal.io/code-exchange) for more.

- [AI enhanced e-commerce application](https://temporal.io/code-exchange/ai-enhanced-e-commerce-application): A sample e-commerce gift shop with hybrid full-text and vector search plus an AI-powered chat shopping assistant, built with Stripe and Temporal Workflows. _(Dotnet · Hybrid search)_
- [Temporal AI Question Planetarium](https://temporal.io/code-exchange/ai-question-planetarium): Runs a Hugging Face model inside Temporal Activities and Workers, streaming updates to the browser over WebSockets in real time. _(Python · Demo)_
- [Document Processing w/ AI](https://temporal.io/code-exchange/document-processing-w-ai): A mortgage underwriting demo that uses Gemini OCR and policy-grounded AI analysis, with human-in-the-loop review and full traceability. _(Python · Gemini · Mortgage)_
- [Rust Confessional: a durable AI agent demo](https://temporal.io/code-exchange/rust-confessional): A live demo where an AI agent judges audience programming confessions, its progress surviving a Worker crash mid-task. _(Rust · Demo)_

## Design patterns for AI agents

- [Approval](/design-patterns/approval): Human-in-the-loop Workflows that block until external approval decisions are made. Uses Signals to capture approval data with metadata.
- [Saga Pattern](/design-patterns/saga-pattern): Manages distributed transactions with compensating actions. Each step has a compensation that undoes its effects if subsequent steps fail.
- [Long-Running Activity](/design-patterns/long-running-activity): Long-running Activities report progress via heartbeats and enable resumption after failures with cancellation support.
- [Entity Workflow](/design-patterns/entity-workflow): A long-lived business entity — a user account, device, or order — gets one Workflow per instance, with Signals and Updates driving every state transition.
- [Local Activities](/design-patterns/local-activities): Local Activities run inside the Worker process, skipping server round-trips for short, idempotent Activities on a latency-sensitive path.

Browse the full [Design Patterns catalog](/design-patterns) for more, or jump straight into the
[AI Cookbook](/ai/cookbook) for runnable code.

---

# AI Cookbook

Source: https://docs.temporal.io/ai/cookbook

> Step-by-step recipes for building reliable AI systems with Temporal, covering LLM integrations, agentic loops, tool calling, and production patterns.

- [Hello world](/ai/cookbook/hello-world-openai-responses-python) — Call an LLM from a durable Temporal Workflow in Python using the OpenAI API library.
- [Hello world with LiteLLM](/ai/cookbook/hello-world-litellm-python) — Integrate LiteLLM into a durable Temporal Workflow in Python to call and switch between LLM providers.
- [Durable agent with tools using the AI SDK by Vercel](/ai/cookbook/ai-sdk-by-vercel-typescript) — Build a durable AI agent with the AI SDK by Vercel and Temporal that chooses tools to answer user questions.
- [Structured outputs with Temporal and OpenAI](/ai/cookbook/structured-output-openai-responses-python) — Use Temporal and the OpenAI Responses API to reliably request output conforming to a specific data structure.
- [Retry policy from HTTP responses](/ai/cookbook/http-retry-enhancement-python) — Extract retry information from HTTP response headers and pass it to Temporal's retry mechanisms in Python.
- [Basic agentic loop with Claude and tool calling](/ai/cookbook/agentic-loop-tool-call-claude-python) — Build a durable agentic loop in Python with Claude tool calling and Temporal.
- [Basic agentic loop with OpenAI and tool calling](/ai/cookbook/agentic-loop-tool-call-openai-python) — Build a durable agentic loop in Python that calls a dynamic set of tools with Temporal and the OpenAI Responses API.
- [Durable MCP weather server](/ai/cookbook/hello-world-durable-mcp-server) — Build a durable MCP server in Python that runs weather tools reliably with Temporal Workflows.
- [Tool calling agent](/ai/cookbook/tool-call-openai-python) — Build a simple, non-looping Python agent that lets the LLM choose tools and then invokes the chosen tools with Temporal and OpenAI.
- [Durable agent with MCP and Activity-backed tools using the Strands Agents SDK](/ai/cookbook/strands-agents-python) — Build a durable AI agent in Python with Temporal and the Strands Agents SDK plugin, combining an MCP server tool with an Activity-backed tool that calls a live HTTP feed.
- [Durable agent with tools using the OpenAI Agents SDK](/ai/cookbook/openai-agents-sdk-python) — Build a durable AI agent with the OpenAI Agents SDK and Temporal that chooses tools to answer user questions.
- [Human-in-the-loop AI agent](/ai/cookbook/human-in-the-loop-python) — Add human-in-the-loop approval to a durable AI agent using Temporal Signals in Python.
- [Multi-Agent Orchestration — Google ADK + Temporal](/ai/cookbook/multi-agent-adk-python) — Build a multi-agent pipeline (parallel + sequential) with Google ADK on Temporal — every LLM call and every I/O tool call runs as a durable activity.
- [Post-LLM guardrail with hard-rule overrides](/ai/cookbook/guardrails-hard-rules-python) — Build a durable content-moderation guardrail in Python with Temporal and Claude that layers deterministic hard rules over an LLM's verdict for auditable overrides.
- [Claim check pattern with Temporal](/ai/cookbook/claim-check-pattern-python) — Use the Claim Check pattern with Temporal to keep large payloads out of Event History by offloading them to S3.
- [Deep research](/ai/cookbook/basic-openai-python) — Build a multi-agent deep research system in Python with Temporal and the OpenAI Responses API.

---

# Basic agentic loop with Claude and tool calling

Source: https://docs.temporal.io/ai/cookbook/agentic-loop-tool-call-claude-python

> Build a durable agentic loop in Python with Claude tool calling and Temporal.

This example implements an agentic loop using Claude (Anthropic) that has a set of tools available. If the agent determines that no tools are needed to satisfy a user request, it will return the response directly. If Claude determines a tool should be used, it will return with the name of the chosen tool and any needed parameters. The agent then invokes the appropriate tool.

Tools are supplied to Claude's Messages API through the `tools` parameter. The `tools` parameter is in JSON format and includes a description of the function as well as descriptions of each of the arguments using Claude's `input_schema` format.

Being external API calls, invoking Claude and invoking any functions/tools are done within a Temporal Activity.

This recipe highlights the following key design decisions:
- We use dynamic Activities to allow the agent to be loosely coupled from specific tools. This sample isolates the tools in the `tools` directory; changing the tools requires NO changes to the agent implementation.
- Because there is an agentic loop, each Claude invocation is passed the accumulated *conversation history* in a structured messages array with role alternation (user/assistant).
- Claude can return multiple tool calls in a single response, and can mix text with tool calls in the same response.
- A generic Activity for invoking Claude's Messages API; instructions and other parameters are passed into the Activity making it appropriate for use in a variety of different use cases.
- Retries are handled by Temporal and not by the Anthropic client library. This is important because client retries can interfere with correct and durable error handling and recovery.

Also see this foundational [recipe for basic tool calling](/ai/cookbook/tool-call-openai-python).

## Application components

This example includes the following components:
- The [Workflow](#create-the-agent-agentic-loop) that contains the agentic loop and tool calling logic; this is the core of the agent implementation.
- The Activities for [invoking Claude](#create-the-activity-for-claude-invocations) and for [invoking tools](#create-the-activity-for-the-tool-invocation).
- A [helper function](#create-the-helper-function) that creates tool definitions in Claude's format.
- Sample [tools](#create-tool-definitions).
- The [Worker](#create-the-worker) that manages the Workflow and the Activities.
- An application that [initiates an interaction](#initiate-an-interaction-with-the-agent) with the agent.

## Create the agent (agentic loop)

### Create the main agentic loop

The agent is implemented as a Temporal Workflow that implements an agentic loop. The loop will continue until the agent responds with no tool calls.

Each time through the loop:
- Claude is called with the accumulated conversation history that is made up of the initial user input and any previous assistant responses and tool outputs.
- The Workflow checks if Claude returned any tool calls (content blocks with `type: "tool_use"`).
- If tool calls are present, the assistant's complete response (including all content blocks) is appended to the messages array, then all tools are executed, and their results are added as a user message.
- If no tool has been called, the text response is returned. 

*File: workflows/agent.py*

<!--SNIPSTART workflows/agent.py {"startPattern": "^from temporalio import workflow$", "endPattern": "return \"No text response from Claude\""}-->
```python
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import claude_responses
    from helpers import tool_helpers
    from tools import get_tools

@workflow.defn
class AgentWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        
        # Initialize messages list with user input
        messages = [{"role": "user", "content": input}]
        print(f"\n[User] {input}")

        # The agentic loop
        while True:
                
            # Consult Claude
            result = await workflow.execute_activity(
                claude_responses.create,
                claude_responses.ClaudeResponsesRequest(
                    model="claude-sonnet-4-5-20250929",
                    system=tool_helpers.HELPFUL_AGENT_SYSTEM_INSTRUCTIONS,
                    messages=messages,
                    tools=get_tools(),
                    max_tokens=4096,
                ),
                start_to_close_timeout=timedelta(seconds=30),
            )

            # Claude returns content blocks - check if any are tool_use
            tool_use_blocks = [block for block in result.content if block.type == "tool_use"]
            
            if tool_use_blocks:
                # We have tool calls to handle
                # First, add the assistant's response to messages
                # Convert content blocks to dictionaries for serialization
                assistant_content = []
                for block in result.content:
                    if block.type == "text":
                        assistant_content.append({"type": "text", "text": block.text})
                    elif block.type == "tool_use":
                        print(f"[Agent] Calling tool: {block.name}")
                        assistant_content.append({
                            "type": "tool_use",
                            "id": block.id,
                            "name": block.name,
                            "input": block.input
                        })
                
                messages.append({"role": "assistant", "content": assistant_content})
                
                # Execute all tool calls and collect results
                tool_results = []
                for block in tool_use_blocks:
                    # Execute the tool
                    tool_result = await self._execute_tool(block.name, block.input)
                    
                    # Add tool result in Claude's expected format
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": str(tool_result)
                    })
                
                # Add tool results as a user message. Claude has only two message roles: user and assistant,
                # and the user message role is used to send tool results back to Claude; in this case the content
                # block includes the tool result.
                messages.append({"role": "user", "content": tool_results})
            else:
                # No tool calls - extract the text response and return
                text_blocks = [block for block in result.content if block.type == "text"]
                if text_blocks:
                    response_text = text_blocks[0].text
                    print(f"[Agent] Final response: {response_text}\n")
                    return response_text
                else:
                    return "No text response from Claude"
```
<!--SNIPEND-->

### Create the tool execution handler

The tool execution handler is invoked by the main agentic loop when Claude has chosen tools. Because the Activity implementation is dynamic, the arguments are passed to the Activity as a dictionary. The Activity invocation is the same as any non-dynamic Activity invocation, passing the name of the Activity, the arguments, and any Activity configurations.

*File: workflows/agent.py*

<!--SNIPSTART workflows/agent.py {"startPattern": "async def _execute_tool\\(self, tool_name: str, tool_input: dict\\) -> str:", "endPattern": "^\\s*return result$"}-->
```python
async def _execute_tool(self, tool_name: str, tool_input: dict) -> str:
    """
    Execute a tool dynamically.

    Args:
        tool_name: Name of the tool to execute
        tool_input: Dictionary of input parameters
    """
    # Execute dynamic activity with the tool name and arguments
    result = await workflow.execute_activity(
        tool_name,
        tool_input,
        start_to_close_timeout=timedelta(seconds=30),
    )
    return result
```
<!--SNIPEND-->

## Create the Activity for Claude invocations

We create a wrapper for the `create` method of the `AsyncAnthropic` client object. This is a generic Activity that invokes Claude's Messages API.

We set `max_retries=0` when creating the `AsyncAnthropic` client. This moves the responsibility for retries from the Anthropic client to Temporal. The Activity then has to interpret the errors coming from Claude's API so the Workflow knows whether to retry.

Errors that can never succeed on a retry, such as a 404 for a retired model identifier or a 401 for a bad API key, are re-raised as a non-retryable `ApplicationError`. Without this they would be retried under the default retry policy until the Workflow timed out, which hides the real problem. Transient errors such as rate limits and 5xx responses propagate unchanged so that Temporal retries them.

In this implementation, we allow for the model, system instructions, messages, list of tools, and max_tokens (required) to be passed in.

*File: activities/claude_responses.py*
<!--SNIPSTART:file activities/claude_responses.py-->
```python
from dataclasses import dataclass
from typing import Any

import anthropic
from anthropic import AsyncAnthropic
from anthropic.types import Message
from temporalio import activity
from temporalio.exceptions import ApplicationError

# Temporal best practice: Create a data structure to hold the request parameters.
@dataclass
class ClaudeResponsesRequest:
    model: str
    system: str
    messages: list[dict[str, Any]]
    tools: list[dict[str, Any]]
    max_tokens: int = 4096

@activity.defn
async def create(request: ClaudeResponsesRequest) -> Message:
    # We disable retry logic in Anthropic API client library so that Temporal can handle retries.
    client = AsyncAnthropic(max_retries=0)

    try:
        resp = await client.messages.create(
            model=request.model,
            system=request.system,
            messages=request.messages,
            tools=request.tools,
            max_tokens=request.max_tokens,
        )
        return resp
    except (
        anthropic.BadRequestError,
        anthropic.AuthenticationError,
        anthropic.PermissionDeniedError,
        anthropic.NotFoundError,
        anthropic.UnprocessableEntityError,
    ) as exc:
        # These errors will never succeed on a retry: a retired model identifier, for
        # example, returns 404. Retrying them under the default policy would loop
        # forever instead of surfacing the problem. Everything else, such as rate
        # limits and 5xx responses, propagates so Temporal can retry it.
        raise ApplicationError(
            str(exc),
            type=exc.__class__.__name__,
            non_retryable=True,
        ) from exc
    finally:
        await client.close()

```
<!--SNIPEND-->

## Create the Activity for the tool invocation

Implement a single tool invocation Activity, as a dynamic Activity (note the `@activity.defn(dynamic=True)` annotation) that acts as a broker to the right tool function. The name of the Activity is drawn from the `activity.info()` and the property bag of arguments from the Activity payload. The `handler` is the function that maps to the `tool_name` (see [Create Tool Definitions](#create-tool-definitions) for more details) and that function is then called with the supplied arguments.

*File: activities/tool_invoker.py*
<!--SNIPSTART:file activities/tool_invoker.py-->
```python
import inspect
from collections.abc import Sequence

from pydantic import BaseModel
from temporalio import activity
from temporalio.common import RawValue

# We use dynamic activities to allow the agent to be defined independently of the tools it can call.
@activity.defn(dynamic=True)
async def dynamic_tool_activity(args: Sequence[RawValue]) -> dict:
    from tools import get_handler

    # the name of the tool to execute - this is passed in via the execute_activity call in the workflow
    tool_name = activity.info().activity_type 
    tool_args = activity.payload_converter().from_payload(args[0].payload, dict)
    activity.logger.info(f"Running dynamic tool '{tool_name}' with args: {tool_args}")

    handler = get_handler(tool_name)
    # in dynamic activity
    sig = inspect.signature(handler)
    params = list(sig.parameters.values())

    if len(params) == 0:
        call_args = []
    else:
        ann = params[0].annotation
        if isinstance(tool_args, dict) and isinstance(ann, type) and issubclass(ann, BaseModel):
            call_args = [ann(**tool_args)]  # or ann.model_validate(tool_args) on Pydantic v2
        else:
            call_args = [tool_args]

    if not inspect.iscoroutinefunction(handler):
        raise TypeError("Tool handler must be async (awaitable).")
    result = await handler(*call_args)

    # Optionally log or augment the result
    activity.logger.info(f"Tool '{tool_name}' result: {result}")
    return result

```
<!--SNIPEND-->

## Create the helper function

The `claude_tool_from_model` function accepts a tool name and description, as well as a Pydantic model for the parameters, and returns JSON that is in the format expected for tool definitions in Claude's Messages API.

*File: helpers/tool_helpers.py*
<!--SNIPSTART helpers/tool_helpers.py:claude-tool-from-model-->
```python
def claude_tool_from_model(name: str, description: str, model: type[BaseModel] | None) -> dict[str, Any]:
    """
    Convert a Pydantic model to Claude's tool format.
    
    Claude's tool format structure:
    {
        "name": "tool_name",
        "description": "Tool description",
        "input_schema": {
            "type": "object",
            "properties": {...},
            "required": [...]
        }
    }
    """
    if model is None:
        # For tools without parameters
        return {
            "name": name,
            "description": description,
            "input_schema": {
                "type": "object",
                "properties": {},
                "required": []
            }
        }
    
    # Get the JSON schema from the Pydantic model
    schema = model.model_json_schema()
    
    # Claude expects an input_schema field instead of parameters
    return {
        "name": name,
        "description": description,
        "input_schema": schema
    }
```
<!--SNIPEND-->

This file also holds the system instruction for the agent.
<!--SNIPSTART helpers/tool_helpers.py {"startPattern": "^HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = \"\"\"$", "endPattern": "^\"\"\"$"}-->
```python
HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = """
You are a helpful agent that can use tools to help the user.
You will be given input from the user and a list of tools to use.
You may or may not need to use the tools to satisfy the user ask.
If no tools are needed, respond in haikus.
"""
```
<!--SNIPEND-->

## Create tool definitions

Tools are defined in the `tools` directory and should be thought of as independent from the agent implementation; as described above, dynamic Activities are used for this loose coupling.

The `__init__.py` file holds tools for providing location (`get_location_info`), IP address (`get_ip_address`), and weather alerts (`get_weather_alerts`).
- The `get_tools` method returns the set of tool definitions that will be passed to Claude.
- The `get_handler` method captures the mapping from tool name to tool function.

*File: tools/\_\_init\_\_.py*
<!--SNIPSTART:file tools/__init__.py-->
```python
from typing import Any, Awaitable, Callable

from . import get_location, get_weather

# Location and weather related tools
from .get_location import get_ip_address, get_location_info
from .get_weather import get_weather_alerts

ToolHandler = Callable[..., Awaitable[Any]]

def get_handler(tool_name: str) -> ToolHandler:
    if tool_name == "get_location_info":
        return get_location_info
    if tool_name == "get_ip_address":
        return get_ip_address
    if tool_name == "get_weather_alerts":
        return get_weather_alerts
    raise ValueError(f"Unknown tool name: {tool_name}")

def get_tools() -> list[dict[str, Any]]:
    return [
        get_weather.WEATHER_ALERTS_TOOL_CLAUDE, 
        get_location.GET_LOCATION_TOOL_CLAUDE,
        get_location.GET_IP_ADDRESS_TOOL_CLAUDE
    ]

```
<!--SNIPEND-->

The tool descriptions and functions are defined in `tools/get_location.py`, `tools/get_weather.py` and `tools/random_stuff.py` files. Each of these files contains:
- data structures for function arguments
- tool definitions (in JSON form using Claude's `input_schema` format)
- the function definitions.

`tools/get_location.py`
<!--SNIPSTART:file tools/get_location.py-->
```python
# get_location.py

from typing import Any

import httpx
from pydantic import BaseModel, Field

from helpers import tool_helpers

# For the location finder we use Pydantic to create a structure that encapsulates the input parameter 
# (an IP address). 
# This is used for both the location finding function and to craft the tool definitions that 
# are passed to Claude.
class GetLocationRequest(BaseModel):
    ipaddress: str = Field(description="An IP address")

# Build the tool definitions for Claude
GET_LOCATION_TOOL_CLAUDE: dict[str, Any] = tool_helpers.claude_tool_from_model(
    "get_location_info",
    "Get the location information for an IP address. This includes the city, state, and country.",
    GetLocationRequest)

GET_IP_ADDRESS_TOOL_CLAUDE: dict[str, Any] = tool_helpers.claude_tool_from_model(
    "get_ip_address",
    "Get the IP address of the current machine.",
    None)

# The functions
async def get_ip_address() -> str:
    async with httpx.AsyncClient() as client:
        response = await client.get("https://icanhazip.com")
        response.raise_for_status()
        return response.text.strip()

async def get_location_info(req: GetLocationRequest) -> str:
    async with httpx.AsyncClient() as client:
        response = await client.get(f"http://ip-api.com/json/{req.ipaddress}")
        response.raise_for_status()
        result = response.json()
        return f"{result['city']}, {result['regionName']}, {result['country']}"

```
<!--SNIPEND-->

## Create the Worker

The Worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the Activities for Claude and tool invocations.

*File: worker.py*

<!--SNIPSTART:file worker.py-->
```python
import asyncio
from concurrent.futures import ThreadPoolExecutor

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities import claude_responses, tool_invoker
from workflows.agent import AgentWorkflow

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    worker = Worker(
        client,
        task_queue="tool-invoking-agent-claude-python-task-queue",
        workflows=[
            AgentWorkflow,
        ],
        activities=[
            claude_responses.create,
            tool_invoker.dynamic_tool_activity,
        ],
        activity_executor=ThreadPoolExecutor(max_workers=10),
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

```
<!--SNIPEND-->

## Initiate an interaction with the agent

To interact with this simple AI agent, we create a Temporal client and execute a Workflow.

*File: start_workflow.py*
<!--SNIPSTART:file start_workflow.py-->
```python
import asyncio
import sys
import uuid

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig

from workflows.agent import AgentWorkflow

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    query = sys.argv[1] if len(sys.argv) > 1 else "Tell me about recursion"

    # Submit the agent workflow for execution
    result = await client.execute_workflow(
        AgentWorkflow.run,
        query,
        id=f"agentic-loop-claude-id-{uuid.uuid4()}",
        task_queue="tool-invoking-agent-claude-python-task-queue",
    )
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run the app

In the terminal where you run the agent Worker, set an Anthropic API key:

```
export ANTHROPIC_API_KEY=sk-ant-...
```

```
uv sync
```

Start the agent Worker:

```bash
uv run worker.py
```

Make request to the agent:

```bash
uv run start_workflow.py "are there any weather alerts for where I am?"
```

Try a number of different user prompts:
```bash
uv run start_workflow.py "where am I?"
uv run start_workflow.py "what is my ip address?"
uv run start_workflow.py "tell me about recursion"
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

## Troubleshooting

**`not_found_error` naming the model**: Anthropic retires older model identifiers, and this recipe pins one in `workflows/agent.py`. Check the [list of current models](https://docs.claude.com/en/docs/about-claude/models/overview) and update the identifier. The Activity classifies this error as non-retryable, so the Workflow fails with the API message rather than retrying.

**`authentication_error`**: `ANTHROPIC_API_KEY` is unset or invalid in the terminal running the Worker. The Activity, not `start_workflow`, makes the API call, so the key has to be set where the Worker runs.

**Workflow runs but no tool is invoked**: Claude decides whether a tool is needed. Prompts like "tell me about recursion" are answered directly. Check the Event History in the [Temporal UI](http://localhost:8233) to see which Activities were scheduled.

---

# Basic agentic loop with OpenAI and tool calling

Source: https://docs.temporal.io/ai/cookbook/agentic-loop-tool-call-openai-python

> Build a durable agentic loop in Python that calls a dynamic set of tools with Temporal and the OpenAI Responses API.

This example implements a basic agentic loop that has a set of tools available. If the
agent determines that no tools are needed to satisfy a user request, it will respond
directly. If the LLM determines a tool should be used it will return with the name
of the chosen tool and any needed parameters. The agent then invokes the 
appropriate tool.

Tools are supplied to the [`responses` API](https://platform.openai.com/docs/api-reference/responses/create) through the [`tools` parameter](https://platform.openai.com/docs/api-reference/responses/create#responses-create-tools). The `tools` parameter is in `json` format and includes a description of the function as well as descriptions of each of the arguments.

> **⚠️ Caution:**
> The API used to generate the tools `json` is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature.

Being external API calls, invoking the LLM and invoking any functions/tools are done within a Temporal Activity.

This recipe highlights the following key design decisions:
- We use dynamic Activities to allow the agent to be loosely coupled from specific
tools. This sample isolates the tools in the `tools` directory; changing the tools
requires no changes to the agent implementation.
- Because there is an agentic loop, each LLM invocation is passed the accumulated 
*conversation history*, that includes the initial user input as well as LLM and tool 
calls.
- A generic Activity for invoking an LLM API; that is, instructions and other `responses` arguments are passed into the Activity making it appropriate for use in a variety of different use cases. Similarly, the result from the responses API call is returned out of the Activity so that it is usable in a variety of different use cases.
- Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery.

Also see this foundational [recipe for basic tool calling](/ai/cookbook/tool-call-openai-python).

## Application components

This example includes the following components:
- The [Workflow](#create-the-agent-agentic-loop) that contains the agentic loop and tool calling logic; this is the core of the agent implementation.
- The activities for [invoking the LLM](#create-the-activity-for-llm-invocations) and for [invoking tools](#create-the-activity-for-the-tool-invocation).
- A [helper function](#create-the-helper-function) that creates tool definitions of the appropriate form.
- Sample [tools](#create-tool-definitions).
- The [worker](#create-the-worker) that manages the Workflow and the Activities.
- An application that [initiates an interaction](#initiate-an-interaction-with-the-agent) with the agent.

## Create the agent (agentic loop)

### Create the main agentic loop

The agent is implemented as a Temporal Workflow that:
- implements an agentic loop. The loop will continue until the agent responds with
no tool calls.

Each time through the loop:
- the LLM is called with the accumulated conversation history that is made up of
the initial user input and any previous LLM responses and tool outputs.
- the invocation of the function, if the LLM has chosen one
- if a function is called the function result is added to the conversation history
- if no tool has been called, the LLM response is returned. This example
demonstrates a most simple UX where the user provides single shot input. Note
however that the agent is not single shot.

*File: workflows/agent.py*

<!--SNIPSTART workflows/agent.py {"startPattern": "^from temporalio import workflow$", "endPattern": "return result\\.output_text"}-->
```python
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import openai_responses
    from helpers import tool_helpers
    from tools import get_tools

@workflow.defn
class AgentWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:

        input_list = [{"type": "message", "role": "user", "content": input}]

        # The agentic loop
        while True:

            print(80 * "=")
                
            # consult the LLM
            result = await workflow.execute_activity(
                openai_responses.create,
                openai_responses.OpenAIResponsesRequest(
                    model="gpt-4o-mini",
                    instructions=tool_helpers.HELPFUL_AGENT_SYSTEM_INSTRUCTIONS,
                    input=input_list,
                    tools=get_tools(),
                ),
                start_to_close_timeout=timedelta(seconds=30),
            )

            # For this simple example, we only have one item in the output list
            # Either the LLM will have chosen a single function call or it will
            # have chosen to respond with a message.
            item = result.output[0]

            # Now process the LLM output to either call a tool or respond with a message.
            
            # if the result is a tool call, call the tool
            if item.type == "function_call":
                result = await self._handle_function_call(item, result, input_list)
                
                # add the tool call result to the input list for context
                input_list.append({"type": "function_call_output",
                                    "call_id": item.call_id,
                                    "output": result})

            # if the result is not a tool call we will just respond with a message
            else:
                print(f"No tools chosen, responding with a message: {result.output_text}")
                return result.output_text
```
<!--SNIPEND-->

### Create the function call handler

The function call handler is invoked by the main agentic loop when an LLM has chosen
a tool. Because the activty implementation is dynamic, the arguments are passed 
to the Activity in a property bag; the `args` variable is appropriately set.
Otherwise, the Activity invocation is the same as any non-dynamic Activity 
invocation passing the name of the Activity, the arguments and any Activity
configurations.

*File: workflows/agent.py*

<!--SNIPSTART workflows/agent.py {"startPattern": "^\\s*async def _handle_function_call\\(", "endPattern": "^\\s*return result$"}-->
```python
async def _handle_function_call(self, item, result, input_list):
    # serialize the LLM output - the decision the LLM made to call a tool
    i = result.output[0]
    input_list += [
        i.model_dump() if hasattr(i, "model_dump") else i
    ]
    # execute dynamic activity with the tool name chosen by the LLM
    # and the arguments crafted by the LLM
    args = json.loads(item.arguments) if isinstance(item.arguments, str) else item.arguments

    result = await workflow.execute_activity(
        item.name,
        args,
        start_to_close_timeout=timedelta(seconds=30),
    )

    print(f"Made a tool call to {item.name}")

    return result
```
<!--SNIPEND-->

## Create the Activity for LLM invocations

We create a wrapper for the `create` method of the `AsyncOpenAI` client object.
This is a generic Activity that invokes the OpenAI LLM.

We set `max_retries=0` when creating the `AsyncOpenAI` client.
This moves the responsibility for retries from the OpenAI client to Temporal. This means
that the Activity should interpret any errors coming from the OpenAI API call and return
the appropriate error type so that the workflow knows if it should retry the Activity or not.

In this implementation, we allow for the model, instructions and input to be passed in, and also the list of tools.

*File: activities/openai_responses.py*
<!--SNIPSTART:file activities/openai_responses.py-->
```python
from dataclasses import dataclass
from typing import Any

from openai import AsyncOpenAI
from openai.types.responses import Response
from temporalio import activity

# Temporal best practice: Create a data structure to hold the request parameters.
@dataclass
class OpenAIResponsesRequest:
    model: str
    instructions: str
    input: object
    tools: list[dict[str, Any]]

@activity.defn
async def create(request: OpenAIResponsesRequest) -> Response:
    # We disable retry logic in OpenAI API client library so that Temporal can handle retries.
    # In a real setting, you would need to handle any errors coming back from the OpenAI API,
    # so that Temporal can appropriately retry in the manner that OpenAI API would.
    # See the `http_retry_enhancement_python` example for inspiration.
    client = AsyncOpenAI(max_retries=0)

    try:
        resp = await client.responses.create(
            model=request.model,
            instructions=request.instructions,
            input=request.input,
            tools=request.tools,
            timeout=30,
        )
        return resp
    finally:
        await client.close()
```
<!--SNIPEND-->

## Create the Activity for the tool invocation

Implement a single tool invocation Activity, as a dynamic Activity (note the 
`@activity.defn(dynamic=True)` annotation) that acts as a broker to the right
tool function. The name of the Activity is drawn from the `activity.info()` and the
property bag of arguments from the Activity payload. The `handler` is the function
that maps to the `tool_name` 
(see [Create Tool Definitions](#create-tool-definitions) for more details)
and that function is then called with the supplied arguments.

*File: activities/tool_invoker.py*
<!--SNIPSTART:file activities/tool_invoker.py-->
```python
import inspect
from collections.abc import Sequence

from pydantic import BaseModel
from temporalio import activity
from temporalio.common import RawValue

# We use dynamic activities to allow the agent to be defined independently of the tools it can call.
@activity.defn(dynamic=True)
async def dynamic_tool_activity(args: Sequence[RawValue]) -> dict:
    from tools import get_handler

    # the name of the tool to execute - this is passed in via the execute_activity call in the Workflow
    tool_name = activity.info().activity_type 
    tool_args = activity.payload_converter().from_payload(args[0].payload, dict)
    activity.logger.info(f"Running dynamic tool '{tool_name}' with args: {tool_args}")

    handler = get_handler(tool_name)
    # in dynamic activity
    sig = inspect.signature(handler)
    params = list(sig.parameters.values())

    if len(params) == 0:
        call_args = []
    else:
        ann = params[0].annotation
        if isinstance(tool_args, dict) and isinstance(ann, type) and issubclass(ann, BaseModel):
            call_args = [ann(**tool_args)]  # or ann.model_validate(tool_args) on Pydantic v2
        else:
            call_args = [tool_args]

    if not inspect.iscoroutinefunction(handler):
        raise TypeError("Tool handler must be async (awaitable).")
    result = await handler(*call_args)

    # Optionally log or augment the result
    activity.logger.info(f"Tool '{tool_name}' result: {result}")
    return result
```
<!--SNIPEND-->

## Create the helper function

The `oai_responses_tool_from_model` function accepts a tool name and description, as well as a list of argument name/description pairs and returns json that is in the format expected for tool definitions in the OpenAI responses API.

> **⚠️ Caution:**
> The API used to generate the tools json is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature.

*File: helpers/tool_helpers.py*
<!--SNIPSTART helpers/tool_helpers.py {"startPattern": "^from openai\\.lib\\._pydantic import to_strict_json_schema", "endPattern": "^\\s*\\}\\s*$"}-->
```python
from openai.lib._pydantic import to_strict_json_schema  # private API; may change

# there currently is no public API to generate the tool definition from a Pydantic model
# or a function signature.
from pydantic import BaseModel

def oai_responses_tool_from_model(name: str, description: str, model: type[BaseModel]):
    return {
        "type": "function",
        "name": name,
        "description": description,
        # OpenAI Responses strict tools require a JSON Schema object where
        # additionalProperties is explicitly false. For tools without
        # parameters, supply an empty object schema.
        "parameters": (
            to_strict_json_schema(model)
            if model
            else {"type": "object", "properties": {}, "required": [], "additionalProperties": False}
        ),
        "strict": True,
    }
```
<!--SNIPEND-->

This file also holds the system instruction for the agent.
<!--SNIPSTART helpers/tool_helpers.py {"startPattern": "^HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = \"\"\"$", "endPattern": "^\"\"\"$"}-->
```python
HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = """
You are a helpful agent that can use tools to help the user.
You will be given a input from the user and a list of tools to use.
You may or may not need to use the tools to satisfy the user ask.
If no tools are needed, respond in haikus.
"""
```
<!--SNIPEND-->

## Create tool definitions

Tools are defined in the `tools` directory and should be thought of as independent 
from the agent implementation; as described above, dynamic Activities are used 
for this loose coupling. 

The `__init__.py` file holds two examples of tool sets,
one providing location and weather tools, the other a simple random number generating
tool; comment and uncomment sets you would like to include (or combine them by 
updating the `get_tools` and `get_handler` methods). 
- The `get_tools` method returns the set of tool definitions that will be passed to
the LLM.
- The `get_handler` method captures the mapping from tool name to tool function

*File: tools/\_\_init\_\_.py*
<!--SNIPSTART:file tools/__init__.py-->
```python
# Uncomment and comment out the tools you want to use

from typing import Any, Awaitable, Callable

# Location and weather related tools
from .get_location import (
    GET_IP_ADDRESS_TOOL_OAI,
    GET_LOCATION_TOOL_OAI,
    get_ip_address,
    get_location_info,
)
from .get_weather import WEATHER_ALERTS_TOOL_OAI, get_weather_alerts

ToolHandler = Callable[..., Awaitable[Any]]

def get_handler(tool_name: str) -> ToolHandler:
    if tool_name == "get_location_info":
        return get_location_info
    if tool_name == "get_ip_address":
        return get_ip_address
    if tool_name == "get_weather_alerts":
        return get_weather_alerts
    raise ValueError(f"Unknown tool name: {tool_name}")

def get_tools() -> list[dict[str, Any]]:
    return [WEATHER_ALERTS_TOOL_OAI,
            GET_LOCATION_TOOL_OAI,
            GET_IP_ADDRESS_TOOL_OAI]

# Random number tool
# from .random_stuff import get_random_number, RANDOM_NUMBER_TOOL_OAI

# def get_handler(tool_name: str) -> ToolHandler:
#     if tool_name == "get_random_number":
#         return get_random_number
#     raise ValueError(f"Unknown tool name: {tool_name}")

# def get_tools() -> list[dict[str, Any]]:
#     return [RANDOM_NUMBER_TOOL_OAI]
```
<!--SNIPEND-->

The tool descriptions and functions are defined in `tools/get_location.py`, 
`tools/get_weather.py` and `tools/random_stuff.py` files. Each of these files contains:
- data structures for function arguments
- tool definitions (in `json` form)
- the function definitions.

`tools/get_location.py`
<!--SNIPSTART:file tools/get_location.py-->
```python
# get_location.py

from typing import Any

import httpx
from pydantic import BaseModel, Field

from helpers import tool_helpers

# For the location finder we use Pydantic to create a structure that encapsulates the input parameter 
# (an IP address). 
# This is used for both the location finding function and to craft the tool definitions that 
# are passed to the OpenAI Responses API.
class GetLocationRequest(BaseModel):
    ipaddress: str = Field(description="An IP address")

# Build the tool definitions for the OpenAI Responses API. 
GET_LOCATION_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model(
    "get_location_info",
    "Get the location information for an IP address. This includes the city, state, and country.",
    GetLocationRequest)

GET_IP_ADDRESS_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model(
    "get_ip_address",
    "Get the IP address of the current machine.",
    None)

# The functions
async def get_ip_address() -> str:
    async with httpx.AsyncClient() as client:
        response = await client.get("https://icanhazip.com")
        response.raise_for_status()
        return response.text.strip()

async def get_location_info(req: GetLocationRequest) -> str:
    async with httpx.AsyncClient() as client:
        response = await client.get(f"http://ip-api.com/json/{req.ipaddress}")
        response.raise_for_status()
        result = response.json()
        return f"{result['city']}, {result['regionName']}, {result['country']}"
```
<!--SNIPEND-->

See files in GitHub for more tool definitions.

## Create the Worker

The Worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the Activities for the LLM and tool invocations.

*File: worker.py*

<!--SNIPSTART:file worker.py-->
```python
import asyncio
from concurrent.futures import ThreadPoolExecutor

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities import openai_responses, tool_invoker
from workflows.agent import AgentWorkflow

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    worker = Worker(
        client,
        task_queue="tool-invoking-agent-python-task-queue",
        workflows=[
            AgentWorkflow,
        ],
        activities=[
            openai_responses.create,
            tool_invoker.dynamic_tool_activity,
        ],
        activity_executor=ThreadPoolExecutor(max_workers=10),
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Initiate an interaction with the agent

To interact with this simple AI agent, we create a Temporal client and execute a Workflow.

*File: start_workflow.py*
<!--SNIPSTART:file start_workflow.py-->
```python
import asyncio
import sys
import uuid

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig

from workflows.agent import AgentWorkflow

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    query = sys.argv[1] if len(sys.argv) > 1 else "Tell me about recursion"

    # Submit the the agent workflow for execution
    result = await client.execute_workflow(
        AgentWorkflow.run,
        query,
        id=f"agentic-loop-id-{uuid.uuid4()}",
        task_queue="tool-invoking-agent-python-task-queue",
    )
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run the app

In the terminal where you run the agent worker, set an OpenAI API key:

```
export OPENAI_API_KEY=sk...
```

```
uv sync
```

Start the agent worker:

```bash
uv run worker.py
```

Make request to the agent:

```bash
uv run start_workflow.py "are there any weather alerts for where I am?"
```

Try a number of different user prompts:
```bash
uv run start_workflow.py "where am I?"
uv run start_workflow.py "what is my ip address?"
uv run start_workflow.py "can I please have a random number?"
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Durable agent with tools using the AI SDK by Vercel

Source: https://docs.temporal.io/ai/cookbook/ai-sdk-by-vercel-typescript

> Build a durable AI agent with the AI SDK by Vercel and Temporal that chooses tools to answer user questions.

This recipe shows how to build a durable agent using the [AI SDK by Vercel](/develop/typescript/integrations/ai-sdk#provide-your-durable-agent-with-tools). The agent calls tools backed by Temporal Activities to answer user questions, and it can determine which tools to use based on the input it receives.

This recipe highlights key implementation patterns:

- **AI SDK client integration**: The Workflow uses `generateText` from `ai` and `temporalProvider` from `@temporalio/ai-sdk/workflow`. This automatically wraps the LLM invocation as an Activity, so it's retried and tracked like any other durable step. `temporalProvider` is configured for `gpt-4o-mini` here, but you can point it at any model the AI SDK supports.
- **Tools-as-Activities**: `proxyActivities` wires the `getWeather` and `calculateAreaOfCircle` Activities into the Workflow so `toolsAgent` can offer tool schemas to the model, wait for results durably, and retry a tool call if it fails.

Unlike some other Temporal AI integrations — for example, the OpenAI Agents SDK's `activity_as_tool` helper, which generates a tool schema from a Python function's type hints — the Vercel AI SDK's `tool()` has no equivalent auto-generation from a TypeScript function signature. Each tool's `inputSchema` is written by hand as a Zod schema.

## Create the Activity

Temporal Activities provide the tools that `toolsAgent` can call. `getWeather` demonstrates an Activity that wraps an unreliable external call: it geocodes the city name (via the free [Open-Meteo geocoding API](https://open-meteo.com/en/docs/geocoding-api)) and then queries the [National Weather Service API](https://www.weather.gov/documentation/services-web-api), both of which are free and require no API key. Because the NWS API only covers the United States, use a US city in your prompt. `calculateAreaOfCircle` shows the opposite case — a tool that runs entirely locally with no external I/O.

*File: src/activities.ts*

<!--SNIPSTART:file src/activities.ts-->
```typescript
const USER_AGENT = '(temporal-ai-cookbook, cookbook@temporal.io)';

async function geocode(location: string): Promise<{ name: string; latitude: number; longitude: number }> {
  const url = `https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(location)}&count=1&countryCode=US`;
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(`Geocoding request failed for "${location}": ${response.status}`);
  }
  const data = (await response.json()) as {
    results?: Array<{ name: string; latitude: number; longitude: number }>;
  };
  const match = data.results?.[0];
  if (!match) {
    throw new Error(`No location found for "${location}". The National Weather Service only covers the US.`);
  }
  return match;
}

// The National Weather Service API only covers the US and requires a two-step
// lookup: resolve coordinates to a forecast grid, then fetch that grid's forecast.
export async function getWeather(input: {
  location: string;
}): Promise<{ city: string; temperatureRange: string; conditions: string }> {
  const { name, latitude, longitude } = await geocode(input.location);

  const pointsResponse = await fetch(`https://api.weather.gov/points/${latitude.toFixed(4)},${longitude.toFixed(4)}`, {
    headers: { 'User-Agent': USER_AGENT },
  });
  if (!pointsResponse.ok) {
    throw new Error(`National Weather Service points lookup failed: ${pointsResponse.status}`);
  }
  const points = (await pointsResponse.json()) as { properties: { forecast: string } };

  const forecastResponse = await fetch(points.properties.forecast, { headers: { 'User-Agent': USER_AGENT } });
  if (!forecastResponse.ok) {
    throw new Error(`National Weather Service forecast lookup failed: ${forecastResponse.status}`);
  }
  const forecast = (await forecastResponse.json()) as {
    properties: { periods: Array<{ temperature: number; temperatureUnit: string; shortForecast: string }> };
  };
  const current = forecast.properties.periods[0];

  return {
    city: name,
    temperatureRange: `${current.temperature}${current.temperatureUnit}`,
    conditions: current.shortForecast,
  };
}

export async function calculateAreaOfCircle(input: { radius: number }): Promise<{ area: number }> {
  return { area: Math.PI * input.radius * input.radius };
}
```
<!--SNIPEND-->

## Create the Workflow

The Workflow registers both Activities as tools with a Zod schema so the model can call them when appropriate.

*File: src/workflows.ts*

<!--SNIPSTART:file src/workflows.ts-->
```typescript
import type * as activities from './activities';
import { generateText, stepCountIs, tool } from 'ai';
import { temporalProvider } from '@temporalio/ai-sdk/workflow';
import { proxyActivities } from '@temporalio/workflow';
import z from 'zod';

const { getWeather, calculateAreaOfCircle } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
  retry: {
    maximumAttempts: 3,
  },
});

export async function toolsAgent(question: string): Promise<string> {
  const result = await generateText({
    model: temporalProvider.languageModel('gpt-4o-mini'),
    prompt: question,
    system: 'You are a helpful agent.',
    tools: {
      getWeather: tool({
        description: 'Get the weather for a given city',
        inputSchema: z.object({
          location: z.string().describe('The location to get the weather for'),
        }),
        execute: getWeather,
      }),
      calculateCircleArea: tool({
        description: 'Calculate the area of a circle',
        inputSchema: z.object({
          radius: z.number().describe('The radius of the circle'),
        }),
        execute: calculateAreaOfCircle,
      }),
    },
    stopWhen: stepCountIs(5),
  });
  return result.text;
}
```
<!--SNIPEND-->

## Create the Worker

Create the process for executing Activities and Workflows. The Worker uses `AiSdkPlugin` to configure the OpenAI provider and to keep Workflow code isolated from the Activity environment.

*File: src/worker.ts*

<!--SNIPSTART:file src/worker.ts-->
```typescript
import { NativeConnection, Worker } from '@temporalio/worker';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import * as activities from './activities';
import { AiSdkPlugin } from '@temporalio/ai-sdk';
import { openai } from '@ai-sdk/openai';

async function run() {
  const config = loadClientConnectConfig();
  const connection = await NativeConnection.connect(config.connectionOptions);
  const worker = await Worker.create({
    plugins: [new AiSdkPlugin({ modelProvider: openai })],
    connection,
    namespace: config.namespace,
    taskQueue: 'ai-sdk',
    workflowsPath: require.resolve('./workflows'),
    activities,
  });
  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});

```
<!--SNIPEND-->

## Create the Workflow Starter

The starter (`src/client.ts`) takes the question to ask as a command-line argument, spins up a Temporal client, and starts `toolsAgent` with a new Workflow Id.

*File: src/client.ts*

<!--SNIPSTART:file src/client.ts-->
```typescript
import { Connection, Client } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { toolsAgent } from './workflows';
import { nanoid } from 'nanoid';

async function run() {
  const question = process.argv.slice(2).join(' ') || 'What is the weather in Seattle right now?';

  const config = loadClientConnectConfig();
  const connection = await Connection.connect(config.connectionOptions);
  const client = new Client({ connection, namespace: config.namespace });

  const handle = await client.workflow.start(toolsAgent, {
    taskQueue: 'ai-sdk',
    args: [question],
    workflowId: 'workflow-' + nanoid(),
  });

  console.log(`Started workflow ${handle.workflowId}`);
  console.log(await handle.result());
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal Dev Server:
```bash
temporal server start-dev
```

Install all dependencies:
```bash
npm install
```

Set your API Key based on your preferred model provider:
```bash
export OPENAI_API_KEY=<KEY>
```

Run the worker:
```bash
npm run start.watch
```

Start execution with the default question, or supply your own:
```bash
npm run workflow
npm run workflow "What is the weather in Chicago?"
npm run workflow "Calculate the area of a circle with radius 5"
```

## Example interactions

Try asking the agent questions like:

- "What is the weather in Seattle right now?"
- "Calculate the area of a circle with radius 5"
- "What is the weather in Chicago and calculate the area of a circle with radius 3"

The agent decides which tools to use. Open the [Temporal UI](http://localhost:8233) to see the `invokeModel`, `getWeather`, and `calculateAreaOfCircle` Activities recorded in the Event History.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Deep research

Source: https://docs.temporal.io/ai/cookbook/basic-openai-python

> Build a multi-agent deep research system in Python with Temporal and the OpenAI Responses API.

Deep research systems combine multiple agents with information retrieval from
the web or other sources to produce evidence-based reports on specific topics.
Commercial implementations include
[Anthropic Research](https://www.anthropic.com/engineering/multi-agent-research-system),
[OpenAI Deep Research](https://openai.com/index/introducing-deep-research/), and
[Google Gemini Deep Research](https://gemini.google/overview/deep-research/).

This recipe demonstrates a simple deep research system embodying the standard
deep research architecture. Deep research spans the following four phases:

- **Planning**. Task decomposition and research strategy formulation. This
  involves identifying separate aspects of the research problem that can be
  worked on independently.
- **Question Development/Query Generation**. Designing queries for each of the
  research questions.
- **Web Exploration/Information Retrieval**. Searching the web to retrieve
  documents relevant to the research question. Extracting and summarizing
  relevant information.
- **Report Generation/Synthesis**. Synthesizing findings into comprehensive,
  well-cited reports.

Deep research tasks can involve dozens of searches and process hundreds of
documents. This creates many possible failure modes that durable execution helps
protect against.

This recipe uses OpenAI's Responses API, which includes a tool for web search.
It also uses OpenAI's
[Structured Outputs API](https://platform.openai.com/docs/guides/structured-outputs),
which asks the model to generate outputs corresponding to desired data
structures.

## Create the data structures

We will use Python classes to ensure information passes between agents in a
structured way.

The Planning Agent creates a `ResearchPlan`, which includes a research question,
a list of `ResearchAspects`, expected sources, a search strategy, and success
criteria. ResearchAspects include an aspect name, a priority, and a description.

<!--SNIPSTART agents/shared.py {"startPattern": "^class ResearchPlan\\(BaseModel\\):$", "endPattern": "^\\s*success_criteria: List\\[str\\]$"}-->
```python
class ResearchPlan(BaseModel):
    research_question: str
    key_aspects: List[ResearchAspect]
    expected_sources: List[str]
    search_strategy: str
    success_criteria: List[str]
```
<!--SNIPEND-->

<!--SNIPSTART agents/shared.py {"startPattern": "^class ResearchAspect\\(BaseModel\\):$", "endPattern": "^\\s*description: str$"}-->
```python
class ResearchAspect(BaseModel):
    aspect: str
    priority: int
    description: str
```
<!--SNIPEND-->

The Query Generation Agent creates a `QueryPlan`, and generates a list of
`SearchQueries`.

<!--SNIPSTART agents/shared.py {"startPattern": "^class QueryPlan\\(BaseModel\\):$", "endPattern": "^\\s*queries: List\\[SearchQuery\\]$"}-->
```python
class QueryPlan(BaseModel):
    queries: List[SearchQuery]
```
<!--SNIPEND-->

<!--SNIPSTART agents/shared.py {"startPattern": "^class SearchQuery\\(BaseModel\\):$", "endPattern": "^\\s*priority: int$"}-->
```python
class SearchQuery(BaseModel):
    query: str
    rationale: str
    expected_info_type: str
    priority: int
```
<!--SNIPEND-->

The Web Search Agent creates a `SearchResult`, which includes a query, a list of
sources, a key finding, a relevance score, and a list of citations.

<!--SNIPSTART agents/shared.py {"startPattern": "^class SearchResult\\(BaseModel\\):$", "endPattern": "^\\s*citations: List\\[str\\]$"}-->
```python
class SearchResult(BaseModel):
    query: str
    sources: List[str]
    key_findings: str
    relevance_score: float
    citations: List[str]
```
<!--SNIPEND-->

Finally, the Report Synthesis Agent creates a `ResearchReport`, which includes
an executive summary, a detailed analysis, a list of key findings, a confidence
assessment, a list of citations, and a list of follow-up questions.

<!--SNIPSTART agents/shared.py {"startPattern": "^class ResearchReport\\(BaseModel\\):$", "endPattern": "^\\s*follow_up_questions: List\\[str\\]$"}-->
```python
class ResearchReport(BaseModel):
    executive_summary: str
    detailed_analysis: str
    key_findings: List[str]
    confidence_assessment: str
    citations: List[str]
    follow_up_questions: List[str]
```
<!--SNIPEND-->

## Create the agents

The deep research system uses four specialized agents, each implemented as
Temporal Activities. In this implementation, each agent is implemented as a
single call to the OpenAI Responses API.

This is possible because we are using structured outputs, which guarantee the
response will be in the correct format, eliminating the need for retries.

The web search agent also requires only a single API call because OpenAI
integrates the web search tool into the Responses API.

These agents run in the Workflow and use the `invoke_model` Activity to make
OpenAI API calls. It is critical to set the `start_to_close_timeout` for these
Activities to a value that is long enough to complete the task. If it is too
short, the Activity will fail with a timeout error, causing a retry loop that
never completes. Response times for reasoning models such as `GPT-5` can vary
significantly depending on the nature of the request. Web search times also vary
depending on the size and content of the documents located by the search.

Each agent imports the `invoke_model` Activity inside
`workflow.unsafe.imports_passed_through()`. That Activity module pulls in the
OpenAI client and, through it, `httpx`, which touches modules that the Workflow
sandbox restricts at import time. Only the Activity import is passed through, so
the agent code itself stays sandboxed and keeps its determinism checks.

Each agent also appends the current date to its instructions with `with_today()`.
The date is read from the Workflow clock through `workflow.now()`, which requires
a running Workflow. Building the instruction string at module import time would
instead freeze the date when the Worker starts.

### Research Planning Agent

Analyzes research queries and creates comprehensive research strategies. Takes
an unstructured question and decomposes it into specific research aspects with
priorities, identifies expected source types, and defines success criteria.

*File: agents/research_planning.py*

<!--SNIPSTART:file agents/research_planning.py-->
```python
from datetime import timedelta

from temporalio import workflow

from .config import COMPLEX_REASONING_MODEL
from .shared import ResearchPlan, with_today

with workflow.unsafe.imports_passed_through():
    from activities.invoke_model import InvokeModelRequest, invoke_model

RESEARCH_PLANNING_INSTRUCTIONS = """
You are a research planning specialist who creates focused research strategies.

CORE RESPONSIBILITIES:
1. Decompose the user's question into 3-7 key research aspects
2. Identify required sources and evidence types
3. Design a practical search strategy
4. Set clear success criteria

OUTPUT REQUIREMENTS:
- research_question: Clarified version of the original query
- key_aspects: Specific areas requiring investigation, each with:
  - aspect: The research area name
  - priority: 1-5 ranking (5 highest priority)  
  - description: What needs to be investigated
- expected_sources: Types of sources likely to contain relevant information
- search_strategy: High-level approach for information gathering
- success_criteria: Specific indicators of research completeness
"""

async def plan_research(query: str) -> ResearchPlan:
    result = await workflow.execute_activity(
        invoke_model,
        InvokeModelRequest(
            model=COMPLEX_REASONING_MODEL,
            instructions=with_today(RESEARCH_PLANNING_INSTRUCTIONS),
            input=f"Research query: {query}",
            response_format=ResearchPlan,
        ),
        start_to_close_timeout=timedelta(seconds=300),
        summary="Planning research",
    )
    return result.response
```
<!--SNIPEND-->

### Query Generation Agent

Converts research plans into optimized web search queries. Creates 3-5 diverse
queries that target different information types (factual data, expert analysis,
case studies, recent news) with varied search styles and temporal modifiers.

*File: agents/research_query_generation.py*

<!--SNIPSTART:file agents/research_query_generation.py-->
```python
from datetime import timedelta

from temporalio import workflow

from .config import EFFICIENT_PROCESSING_MODEL
from .shared import QueryPlan, ResearchPlan, with_today

with workflow.unsafe.imports_passed_through():
    from activities.invoke_model import InvokeModelRequest, invoke_model

QUERY_GENERATION_INSTRUCTIONS = """
You are a search query specialist who crafts effective web searches.

CORE RESPONSIBILITIES:
1. Generate 3-5 diverse search queries based on the research plan
2. Balance specificity with discoverability
3. Target different information types (factual, analytical, recent, historical)

APPROACH:
- Vary query styles: direct questions, topic + keywords, source-specific searches
- Include temporal modifiers when relevant (recent, 2024, historical)
- Use domain-specific terminology appropriately

OUTPUT REQUIREMENTS:
- queries: Search queries, each with:
  - query: The actual search string
  - rationale: Why this query addresses research needs  
  - expected_info_type: One of "factual_data", "expert_analysis", "case_studies", "recent_news"
  - priority: 1-5 (5 highest priority)
"""

async def generate_queries(research_plan: ResearchPlan) -> QueryPlan:
    # Prepare input with research plan context
    plan_context = f"""
Research Question: {research_plan.research_question}

Key Aspects to Research:
{chr(10).join([f"- {aspect.aspect} (Priority: {aspect.priority}): {aspect.description}" for aspect in research_plan.key_aspects])}

Expected Sources: {", ".join(research_plan.expected_sources)}
Search Strategy: {research_plan.search_strategy}
Success Criteria: {", ".join(research_plan.success_criteria)}
"""

    result = await workflow.execute_activity(
        invoke_model,
        InvokeModelRequest(
            model=EFFICIENT_PROCESSING_MODEL,
            instructions=with_today(QUERY_GENERATION_INSTRUCTIONS),
            input=plan_context,
            response_format=QueryPlan,
        ),
        start_to_close_timeout=timedelta(seconds=300),
        summary="Generating search queries",
    )

    return result.response
```
<!--SNIPEND-->

### Web Search Agent

Executes searches using OpenAI's web search tool and analyzes results.
Prioritizes authoritative sources, extracts key findings, assesses relevance,
and provides proper citations with reliability assessments.

*File: agents/research_web_search.py*

<!--SNIPSTART:file agents/research_web_search.py-->
```python
from datetime import timedelta

from temporalio import workflow

from .config import EFFICIENT_PROCESSING_MODEL
from .shared import SearchQuery, SearchResult, with_today

with workflow.unsafe.imports_passed_through():
    from activities.invoke_model import InvokeModelRequest, invoke_model

WEB_SEARCH_INSTRUCTIONS = """
You are a web research specialist who finds and evaluates information from web sources.

CORE RESPONSIBILITIES:
1. Execute web searches using the web search tool
2. Prioritize authoritative sources: academic, government, established research organizations, prominent news outlets, primary sources
3. Extract key information relevant to the research question
4. Provide proper citations and assess reliability

APPROACH:
- Focus on information directly relevant to the research question
- Extract specific facts, data points, and evidence
- Note conflicting information and limitations
- Flag questionable or unverified claims

OUTPUT REQUIREMENTS:
- query: The search query that was executed
- sources: URLs and source descriptions consulted
- key_findings: Synthesized information relevant to research question (2-4 paragraphs)
- relevance_score: 0.0-1.0 assessment of how well results address the query
- citations: Formatted sources with URLs
"""

async def search_web(query: SearchQuery) -> SearchResult:
    search_input = f"""
Search Query: {query.query}
Query Rationale: {query.rationale}
Expected Information Type: {query.expected_info_type}
Priority Level: {query.priority}

Please search for information using the provided query and analyze the results according to the instructions.
"""
    result = await workflow.execute_activity(
        invoke_model,
        InvokeModelRequest(
            model=EFFICIENT_PROCESSING_MODEL,
            instructions=with_today(WEB_SEARCH_INSTRUCTIONS),
            input=search_input,
            response_format=SearchResult,
            tools=[{"type": "web_search"}],
        ),
        start_to_close_timeout=timedelta(seconds=300),
        summary="Searching web for information",
    )
    return result.response
```
<!--SNIPEND-->

### Report Synthesis Agent

Directs the agent to synthesize all research findings into comprehensive,
well-cited reports. These should include structured narratives with executive
summaries, detailed analysis, key findings, confidence assessments, and
follow-up research questions.

*File: agents/research_report_synthesis.py*

<!--SNIPSTART:file agents/research_report_synthesis.py-->
```python
from datetime import timedelta
from typing import List

from temporalio import workflow

from .config import COMPLEX_REASONING_MODEL
from .shared import ResearchPlan, ResearchReport, SearchResult, with_today

with workflow.unsafe.imports_passed_through():
    from activities.invoke_model import InvokeModelRequest, invoke_model

REPORT_SYNTHESIS_INSTRUCTIONS = """
You are a research synthesis expert who creates comprehensive research reports.

CORE RESPONSIBILITIES:
1. Synthesize all research into a coherent narrative
2. Structure information logically with evidence support
3. Provide comprehensive citations
4. Assess confidence levels and acknowledge limitations
5. Generate follow-up questions for deeper research

REPORT STRUCTURE:
1. **Executive Summary**: Core findings and conclusions (1-2 paragraphs)
2. **Detailed Analysis**: Examination organized by themes with evidence
3. **Key Findings**: Bullet-point list of important discoveries
4. **Confidence Assessment**: Rate findings as High/Medium/Low/Uncertain
5. **Citations**: Complete source list with URLs
6. **Follow-up Questions**: Up to 5 areas for additional research, as warranted

APPROACH:
- Address contradictory findings transparently
- Weight authoritative sources more heavily
- Distinguish facts from expert opinions
- Be explicit about information limitations

OUTPUT REQUIREMENTS:
- executive_summary: 1-2 paragraph summary of core findings
- detailed_analysis: Multi-paragraph analysis organized by themes
- key_findings: Bullet-point discoveries
- confidence_assessment: Assessment of finding reliability
- citations: All sources referenced
- follow_up_questions: 3-5 specific questions for further research
"""

async def generate_synthesis(
    original_query: str, research_plan: ResearchPlan, search_results: List[SearchResult]
) -> ResearchReport:
    # Prepare comprehensive input with all research context
    synthesis_input = f"""
ORIGINAL RESEARCH QUERY: {original_query}

RESEARCH PLAN:
Research Question: {research_plan.research_question}
Key Aspects Investigated: {
        ", ".join([aspect.aspect for aspect in research_plan.key_aspects])
    }
Search Strategy Used: {research_plan.search_strategy}
Success Criteria: {", ".join(research_plan.success_criteria)}

SEARCH RESULTS TO SYNTHESIZE:
{
        chr(10).join(
            [
                f"Query: {result.query}{chr(10)}Findings: {result.key_findings}{chr(10)}Relevance: {result.relevance_score}{chr(10)}Sources: {', '.join(result.sources)}{chr(10)}Citations: {', '.join(result.citations)}{chr(10)}"
                for result in search_results
            ]
        )
    }

Please synthesize all this information into a comprehensive research report following the specified structure and quality standards.
"""
    result = await workflow.execute_activity(
        invoke_model,
        InvokeModelRequest(
            model=COMPLEX_REASONING_MODEL,
            instructions=with_today(REPORT_SYNTHESIS_INSTRUCTIONS),
            input=synthesis_input,
            response_format=ResearchReport,
        ),
        start_to_close_timeout=timedelta(seconds=300),
        summary="Generating research report synthesis",
    )

    return result.response
```
<!--SNIPEND-->

## Create the Workflow

The `DeepResearchWorkflow` orchestrates the four-phase research process with
built-in resilience and error handling:

First, planning and query generation agents are run sequentially. Then, the
workflow executes searches concurrently. For robustness, the workflow continues
with partial results if some searches fail. Finally, the report synthesis agent
pulls together the findings into a comprehensive report.

*File: workflows/deep_research_workflow.py*

<!--SNIPSTART:file workflows/deep_research_workflow.py-->
```python
import asyncio
from typing import List

from temporalio import workflow
from temporalio.exceptions import ApplicationError

from agents.research_planning import plan_research
from agents.research_query_generation import generate_queries
from agents.research_report_synthesis import generate_synthesis
from agents.research_web_search import search_web
from agents.shared import SearchResult

@workflow.defn
class DeepResearchWorkflow:
    @workflow.run
    async def run(self, query: str) -> str:
        # Step 1: Research Planning
        research_plan = await plan_research(query)

        # Step 2: Query Generation
        query_plan = await generate_queries(research_plan)

        # Step 3: Web Search (parallel execution with resilience)
        search_results = await self._execute_searches(query_plan.queries)

        # Ensure we have at least one successful search result
        if not search_results:
            raise ApplicationError(
                "All web searches failed - cannot generate report",
                "NO_SEARCH_RESULTS",
                non_retryable=True,
            )

        # Step 4: Report Synthesis
        final_report = await generate_synthesis(query, research_plan, search_results)

        # Format the final output
        formatted_report = self._format_final_report(query, final_report)
        return formatted_report

    async def _execute_searches(self, search_queries) -> List[SearchResult]:
        """Execute web searches in parallel with resilience to individual failures"""

        # Create individual search coroutines
        async def execute_single_search(search_query):
            try:
                return await search_web(search_query)
            except Exception as e:
                workflow.logger.exception(
                    f"Search failed for query '{search_query.query}': {e}"
                )
                return None

        # Execute all searches in parallel
        search_tasks = [execute_single_search(query) for query in search_queries]
        results = await asyncio.gather(*search_tasks)

        # Filter out None results
        return [result for result in results if result is not None]

    def _format_final_report(self, original_query, report) -> str:
        """Format the final report for display"""
        return f"""
# Deep Research Report

**Research Query:** {original_query}

## Executive Summary
{report.executive_summary}

## Detailed Analysis
{report.detailed_analysis}

## Key Findings
{chr(10).join([f"• {finding}" for finding in report.key_findings])}

## Confidence Assessment
{report.confidence_assessment}

## Sources and Citations
{chr(10).join([f"• {citation}" for citation in report.citations])}

## Recommended Follow-up Questions
{chr(10).join([f"• {question}" for question in report.follow_up_questions])}

"""
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal Dev Server:

```bash
temporal server start-dev
```

Run the worker:

```bash
uv run worker.py
```

Start execution:

```bash
uv run start_workflow.py
```

To start execution with a specific query:

```bash
uv run start_workflow.py "What is the latest news on the stock market?"
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Claim check pattern with Temporal

Source: https://docs.temporal.io/ai/cookbook/claim-check-pattern-python

> Use the Claim Check pattern with Temporal to keep large payloads out of Event History by offloading them to S3.

This recipe demonstrates how to use the Claim Check pattern to offload data from Temporal Service's Event History to external storage. This can be useful in conversational AI applications that include the full conversation history with each LLM call, creating large Event History that can exceed server size limits.

This recipe includes:

- A `PayloadCodec` ([docs](/payload-codec)) that stores large payloads in S3 and replaces them with keys
- A client [plugin](/develop/plugins-guide) that wires the codec into the Temporal data converter
- A lightweight codec server for a better Web UI experience
- An AI/RAG example workflow that demonstrates the pattern end-to-end

## Temporal's built-in external storage feature

Since Python SDK 1.25, Temporal offers a built-in [external storage](/develop/python/data-handling/external-storage) feature that implements the same claim check pattern without a custom `PayloadCodec`. It's configured directly on the `DataConverter` via an `ExternalStorage` option, and Temporal provides an `S3StorageDriver` out of the box. The feature is in Public Preview, so its API may change before General Availability.

Use the built-in feature first if S3 (or a self-hosted equivalent) is a good fit and you don't need custom encode/decode logic. Follow the codec-based approach in this recipe when you need a storage backend other than what the built-in drivers support, want to combine claim check with other codec logic (such as encryption) in a single codec, or want full control over the encode/decode implementation.

## How the Claim Check pattern works

Each Temporal Workflow has an associated Event History that is stored in Temporal Service and used to provide durable execution. When using the Claim Check pattern, you store the payload content of the Event in a separate storage system, then store a reference to that storage in the Temporal Event History instead.

The Claim Check Recipe implements a `PayloadCodec` that:

1. Encode: Replaces large payloads with unique keys and stores the original data in external storage (S3, Database, etc.)
2. Decode: Retrieves the original payload using the key when needed

Workflows operate with small, lightweight keys while maintaining transparent access to full data through automatic encoding/decoding.

## Claim Check codec implementation

The `ClaimCheckCodec` implements `PayloadCodec` and adds an inline threshold to keep small payloads inline. This avoids the latency costs of uploading/downloading the payload externally when it's not required.

*File: codec/claim_check.py*

<!--SNIPSTART:file codec/claim_check.py-->
```python
import logging
import uuid
from typing import Iterable, List

import aioboto3
from botocore.exceptions import ClientError
from temporalio.api.common.v1 import Payload
from temporalio.converter import PayloadCodec

logger = logging.getLogger(__name__)

class ClaimCheckCodec(PayloadCodec):
    """PayloadCodec that implements the Claim Check pattern using S3 storage.
    
    This codec stores large payloads in S3 and replaces them with unique keys,
    allowing Temporal workflows to operate with lightweight references instead
    of large payload data.
    """

    def __init__(
        self,
        bucket_name: str = "temporal-claim-check",
        endpoint_url: str = None,
        region_name: str = "us-east-1",
        max_inline_bytes: int = 20 * 1024,
    ):
        """Initialize the claim check codec with S3 connection details.

        Args:
            bucket_name: S3 bucket name for storing claim check data
            endpoint_url: S3 endpoint URL (for MinIO or other S3-compatible services)
            region_name: AWS region name
            max_inline_bytes: Payloads up to this size will be left inline
        """
        self.bucket_name = bucket_name
        self.endpoint_url = endpoint_url
        self.region_name = region_name
        self.max_inline_bytes = max_inline_bytes
        self.session = aioboto3.Session()

        self._bucket_created = False

    async def _ensure_bucket_exists(self):
        """Ensure the S3 bucket exists, creating it if necessary."""
        if self._bucket_created:
            return
            
        async with self.session.client(
            's3',
            endpoint_url=self.endpoint_url,
            region_name=self.region_name
        ) as s3_client:
            try:
                await s3_client.head_bucket(Bucket=self.bucket_name)
            except ClientError as e:
                error_code = e.response['Error']['Code']
                if error_code in ['404', 'NoSuchBucket']:
                    try:
                        await s3_client.create_bucket(Bucket=self.bucket_name)
                    except ClientError as create_error:
                        # Handle bucket already exists race condition
                        if create_error.response['Error']['Code'] not in ['BucketAlreadyExists', 'BucketAlreadyOwnedByYou']:
                            raise create_error
                elif error_code not in ['403', 'Forbidden']:
                    raise e
        
        self._bucket_created = True

    async def encode(self, payloads: Iterable[Payload]) -> List[Payload]:
        """Replace large payloads with keys and store original data in S3.
        
        Args:
            payloads: Iterable of payloads to encode
            
        Returns:
            List of encoded payloads (keys for claim-checked payloads)
        """
        await self._ensure_bucket_exists()
        
        out: List[Payload] = []
        for payload in payloads:
            # Leave small payloads inline to improve debuggability and avoid unnecessary indirection
            data_size = len(payload.data or b"")
            if data_size <= self.max_inline_bytes:
                out.append(payload)
                continue

            encoded = await self.encode_payload(payload)
            out.append(encoded)
        return out

    async def decode(self, payloads: Iterable[Payload]) -> List[Payload]:
        """Retrieve original payloads from S3 using stored keys.
        
        Args:
            payloads: Iterable of payloads to decode
            
        Returns:
            List of decoded payloads (original data retrieved from S3)
            
        Raises:
            ValueError: If a claim check key is not found in S3
        """
        await self._ensure_bucket_exists()
        
        out: List[Payload] = []
        for payload in payloads:
            if payload.metadata.get("temporal.io/claim-check-codec", b"").decode() != "v1":
                # Not a claim-checked payload, pass through unchanged
                out.append(payload)
                continue

            s3_key = payload.data.decode("utf-8")
            stored_data = await self.get_payload_from_s3(s3_key)
            if stored_data is None:
                raise ValueError(f"Claim check key not found in S3: {s3_key}")
            
            original_payload = Payload.FromString(stored_data)
            out.append(original_payload)
        return out

    async def encode_payload(self, payload: Payload) -> Payload:
        """Store payload in S3 and return a key-based payload.
        
        Args:
            payload: Original payload to store
            
        Returns:
            Payload containing only the S3 key
        """
        await self._ensure_bucket_exists()
        
        key = str(uuid.uuid4())
        serialized_data = payload.SerializeToString()
        
        # Store the original payload data in S3
        async with self.session.client(
            's3',
            endpoint_url=self.endpoint_url,
            region_name=self.region_name
        ) as s3_client:
            await s3_client.put_object(
                Bucket=self.bucket_name,
                Key=key,
                Body=serialized_data
            )
        
        # Return a lightweight payload containing only the key
        return Payload(
            metadata={
                "encoding": b"claim-checked",
                "temporal.io/claim-check-codec": b"v1",
            },
            data=key.encode("utf-8"),
        )

    async def get_payload_from_s3(self, s3_key: str) -> bytes:
        """Retrieve payload data from S3.
        
        Args:
            s3_key: S3 object key
            
        Returns:
            Raw payload data bytes, or None if not found
        """
        try:
            async with self.session.client(
                's3',
                endpoint_url=self.endpoint_url,
                region_name=self.region_name
            ) as s3_client:
                response = await s3_client.get_object(
                    Bucket=self.bucket_name,
                    Key=s3_key
                )
                return await response['Body'].read()
        except ClientError as e:
            if e.response['Error']['Code'] == 'NoSuchKey':
                return None
            raise e
```
<!--SNIPEND-->

### Inline payload threshold

- Default: 20KB
- Where configured: `ClaimCheckCodec(max_inline_bytes=20 * 1024)` in `codec/claim_check.py`
- Change by passing a different `max_inline_bytes` when constructing `ClaimCheckCodec`

### Choose the right threshold

The `max_inline_bytes` threshold controls which payloads are offloaded to S3 and which stay inline in Event History. Here is how to choose the right value for your use case.

#### Temporal Service size limits

Temporal enforces size limits at several levels ([self-hosted defaults](/self-hosted-guide/defaults), [Temporal Cloud limits](/evaluate/cloud/limits)):

| Limit | Warning | Error / Termination |
|-------|---------|---------------------|
| Single payload (blob) size | 256 KB | 2 MB |
| Event History total size | 10 MB | 50 MB (Workflow terminated) |
| Event History event count | 10,240 events | 51,200 events (Workflow terminated) |
| gRPC message size | — | 4 MB per message |
| Event History transaction size | — | 4 MB per transaction |

The single payload limit applies to each serialized Activity input, Activity output, or Workflow argument individually. The gRPC limit applies to the full request, so scheduling several Activities with moderate-sized inputs in the same Workflow Task can exceed 4 MB even when each payload is under 2 MB.

These are the defaults for both self-hosted and Temporal Cloud. The blob size thresholds are configurable on self-hosted deployments. Temporal Cloud limits are not configurable.

#### Sizing recommendations

| Threshold | Good for | Trade-off |
|-----------|----------|-----------|
| **2 KB** | Chatty Workflows with many small Activities | More S3 round-trips, higher latency per call |
| **20 KB** (default) | Most AI/RAG Workflows | Balances debuggability with Event History size |
| **128 KB** | Low-Activity Workflows with moderate payloads | Fewer S3 calls, but Event History grows faster |
| **256 KB+** | Workflows with few, large payloads | Right at the blob warning threshold — payloads near this size risk triggering warnings, and Event History grows fast |

#### How to decide

1. **Estimate your payload sizes.** LLM conversation histories grow with each turn. A 10-turn conversation with tool calls can reach 50–100 KB. RAG chunks with embeddings can be several megabytes.
2. **Count your Activities.** Each Activity input and output is a separate payload in Event History. A Workflow with 20 Activity calls at 100 KB each adds 4 MB to Event History from payloads alone.
3. **Start with the default (20 KB).** This offloads anything that would meaningfully impact Event History size while keeping small, debuggable payloads visible in the Web UI.
4. **Stay well under 256 KB.** Payloads above 256 KB trigger a warning from the Temporal Service. If your payloads regularly exceed this size, Claim Check is strongly recommended.
5. **Lower the threshold** if your Workflows are long-running (many Activity calls over time) or if you run many concurrent Workflows on the same Temporal Service.
6. **Raise the threshold** if S3 latency is a concern and your Workflows have few Activities with moderate payloads.

#### Monitoring Event History size

Use the Web UI or `temporal workflow describe` to check a Workflow's Event History size and event count. If you see the 10 MB / 10,240 event warning in Temporal Service logs, lower your `max_inline_bytes` threshold or review which payloads should use Claim Check instead of staying inline.

## Claim Check plugin

The `ClaimCheckPlugin` integrates the codec with the Temporal client configuration.

*File: codec/plugin.py*

<!--SNIPSTART:file codec/plugin.py-->
```python
import os

from temporalio.converter import DataConverter
from temporalio.plugin import SimplePlugin

from .claim_check import ClaimCheckCodec

class ClaimCheckPlugin(SimplePlugin):
    """Temporal plugin that integrates the Claim Check codec with client configuration."""

    def __init__(self):
        """Initialize the plugin with S3 connection configuration."""
        super().__init__(
            name="claim-check",
            data_converter=DataConverter(
                payload_codec=ClaimCheckCodec(
                    bucket_name=os.getenv("S3_BUCKET_NAME", "temporal-claim-check"),
                    endpoint_url=os.getenv("S3_ENDPOINT_URL"),
                    region_name=os.getenv("AWS_REGION", "us-east-1"),
                ),
            ),
        )
```
<!--SNIPEND-->

## Example: AI/RAG Workflow using Claim Check

This example ingests a large text, performs lightweight lexical retrieval, and answers a question with an LLM. Large intermediates (chunks, scores) are kept out of Temporal payloads via the Claim Check codec. Only the small final answer is returned inline.

### Shared models

*File: shared/models.py*

<!--SNIPSTART:file shared/models.py-->
```python
from dataclasses import dataclass
from typing import Any, Dict, List

@dataclass
class IngestRequest:
    document_bytes: bytes
    filename: str
    mime_type: str
    chunk_size: int = 1500
    chunk_overlap: int = 200
    embedding_model: str = "text-embedding-3-large"

@dataclass
class IngestResult:
    chunk_texts: List[str]
    metadata: Dict[str, Any]

@dataclass
class RagRequest:
    question: str
    top_k: int = 4
    generation_model: str = "gpt-4o-mini"

@dataclass
class RagAnswer:
    answer: str
    sources: List[Dict[str, Any]]
```
<!--SNIPEND-->

### Activities

*File: activities/ai_claim_check.py*

<!--SNIPSTART:file activities/ai_claim_check.py-->
```python
from typing import List

from temporalio import activity

from shared.models import IngestRequest, IngestResult, RagAnswer, RagRequest

def _split_text(text: str, chunk_size: int, overlap: int) -> List[str]:
    chunks: List[str] = []
    start = 0
    n = len(text)
    while start < n:
        end = min(n, start + chunk_size)
        chunks.append(text[start:end])
        if end >= n:
            break
        start = max(end - overlap, start + 1)
    return chunks

@activity.defn
async def ingest_document(req: IngestRequest) -> IngestResult:
    # Convert bytes to text. For PDFs/audio/images, integrate proper extractors.
    if req.mime_type != "text/plain":
        raise ValueError(f"Unsupported MIME type: {req.mime_type}")

    text = req.document_bytes.decode("utf-8", errors="ignore")
    chunks = _split_text(text, req.chunk_size, req.chunk_overlap)
    return IngestResult(
        chunk_texts=chunks,
        metadata={
            "filename": req.filename,
            "mime_type": req.mime_type,
            "chunk_count": len(chunks),
        },
    )

@activity.defn
async def rag_answer(req: RagRequest, ingest_result: IngestResult) -> RagAnswer:
    # Import heavy dependencies inside the function, not at module level
    # This prevents NumPy from being loaded into the workflow sandbox
    from openai import AsyncOpenAI
    from rank_bm25 import BM25Okapi
    
    client = AsyncOpenAI(max_retries=0)

    # Lexical retrieval using BM25 over chunk texts
    # Simple whitespace tokenization
    tokenized_corpus: List[List[str]] = [chunk.split() for chunk in ingest_result.chunk_texts]
    bm25 = BM25Okapi(tokenized_corpus)
    tokenized_query = req.question.split()
    scores = bm25.get_scores(tokenized_query)

    # Get top-k indices by score
    top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[: max(1, req.top_k)]
    contexts = [ingest_result.chunk_texts[i] for i in top_indices]
    sources = [{"chunk_index": i, "score": float(scores[i])} for i in top_indices]

    prompt = (
        "Use the provided context chunks to answer the question.\n\n"
        f"Question: {req.question}\n\n"
        "Context:\n" + "\n---\n".join(contexts) + "\n\nAnswer:"
    )

    chat = await client.chat.completions.create(
        model=req.generation_model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
    )
    answer = chat.choices[0].message.content.strip()

    return RagAnswer(answer=answer, sources=sources)

```
<!--SNIPEND-->

### Workflow

*File: workflows/ai_rag_workflow.py*

<!--SNIPSTART:file workflows/ai_rag_workflow.py-->
```python
from datetime import timedelta

from temporalio import workflow

from activities.ai_claim_check import ingest_document, rag_answer
from shared.models import IngestRequest, IngestResult, RagAnswer, RagRequest

@workflow.defn
class AiRagWorkflow:
    @workflow.run
    async def run(self, document_bytes: bytes, filename: str, mime_type: str, question: str) -> RagAnswer:
        ingest: IngestResult = await workflow.execute_activity(
            ingest_document,
            IngestRequest(
                document_bytes=document_bytes,
                filename=filename,
                mime_type=mime_type,
            ),
            start_to_close_timeout=timedelta(minutes=10),
            summary="Ingest and embed large document",
        )

        answer: RagAnswer = await workflow.execute_activity(
            rag_answer,
            args=[
                RagRequest(question=question),
                ingest,
            ],
            start_to_close_timeout=timedelta(minutes=5),
            summary="RAG answer using embedded chunks",
        )
        return answer

```
<!--SNIPEND-->

## Run the recipe

### Prerequisites

- MinIO server (for local testing) or AWS S3 access (for production)
- Temporal dev server
- Python 3.10+

### Configuration

Set environment variables to configure S3 and OpenAI:

```bash
# For MinIO (recommended for local testing)
export S3_ENDPOINT_URL=http://localhost:9000
export S3_BUCKET_NAME=temporal-claim-check
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1

# For production AWS S3
# export S3_BUCKET_NAME=your-bucket-name
# export AWS_REGION=us-east-1
# you may use access keys or...
# export AWS_ACCESS_KEY_ID=your-access-key
# export AWS_SECRET_ACCESS_KEY=your-secret-key
# ... sso
# export AWS_profile=your-profile
# aws sso login --profile your-profile

export OPENAI_API_KEY=your_key_here
```

### Option 1: MinIO (Recommended for Testing)

1. Start MinIO:
```bash
docker run -d -p 9000:9000 -p 9001:9001 \
  --name minio \
  -e "MINIO_ROOT_USER=minioadmin" \
  -e "MINIO_ROOT_PASSWORD=minioadmin" \
  quay.io/minio/minio server /data --console-address ":9001"
```

The bucket will be auto-created by the code. You can view stored objects in the MinIO web console at http://localhost:9001 (credentials: minioadmin/minioadmin).

2. Start Temporal dev server:
```bash
temporal server start-dev
```

3. Run the worker:
```bash
uv run worker.py
```

4. Start execution:
```bash
uv run start_workflow.py
```

### Option 2: AWS S3 (Production)

1. Create an S3 bucket in your AWS account
2. Configure AWS credentials (via AWS CLI, environment variables, IAM roles or sso)
3. Set the environment variables for your bucket
4. Follow steps 2-4 from Option 1

### Toggle Claim Check (optional)

To demonstrate payload size failures without claim check, you can disable it in your local wiring (e.g., omit the plugin/codec) and re-run. With claim check disabled, large payloads may exceed Temporal's default payload size limits and fail.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

## Codec server for Web UI

When claim check is enabled, the Web UI would otherwise show opaque keys. This codec server shows helpful text with a link to view the raw data on demand.

*File: codec/codec_server.py*

<!--SNIPSTART:file codec/codec_server.py-->
```python
import json
import os
from functools import partial
from typing import Awaitable, Callable, Iterable, List

from aiohttp import hdrs, web
from claim_check import ClaimCheckCodec
from google.protobuf import json_format
from temporalio.api.common.v1 import Payload, Payloads

def build_codec_server() -> web.Application:
    # Create codec with environment variable configuration (same as plugin)
    codec = ClaimCheckCodec(
        bucket_name=os.getenv("S3_BUCKET_NAME", "temporal-claim-check"),
        endpoint_url=os.getenv("S3_ENDPOINT_URL"),
        region_name=os.getenv("AWS_REGION", "us-east-1")
    )
    
    # Configure Web UI endpoint
    temporal_web_url = os.getenv("TEMPORAL_WEB_URL", "http://localhost:8233")
    # Configure codec server endpoint for viewing raw data
    codec_server_url = os.getenv("CODEC_SERVER_URL", "http://localhost:8081")
    
    # CORS handler - needed because Temporal Web UI runs on a different port/domain
    # and the browser blocks cross-origin requests by default; CORS headers allow these requests
    async def cors_options(req: web.Request) -> web.Response:
        resp = web.Response()
        if req.headers.get(hdrs.ORIGIN) == temporal_web_url:
            resp.headers[hdrs.ACCESS_CONTROL_ALLOW_ORIGIN] = temporal_web_url
            resp.headers[hdrs.ACCESS_CONTROL_ALLOW_METHODS] = "POST"
            resp.headers[hdrs.ACCESS_CONTROL_ALLOW_HEADERS] = "content-type,x-namespace"
        return resp

    # Custom decode function that provides URLs to view raw data
    async def decode_with_urls(payloads: Iterable[Payload]) -> List[Payload]:
        """Decode claim check payloads and provide URLs to view the raw data."""
        out: List[Payload] = []
        
        for payload in payloads:
            if payload.metadata.get("temporal.io/claim-check-codec", b"").decode() != "v1":
                # Not a claim-checked payload, pass through unchanged
                out.append(payload)
                continue

            # Get the S3 key
            s3_key = payload.data.decode("utf-8")
            
            # Return simple text with link - no data reading
            link_text = f"Claim check data (key: {s3_key}) - View at: {codec_server_url}/view/{s3_key}"
            
            summary_payload = Payload(
                metadata={"encoding": b"json/plain"},
                data=json.dumps({"text": link_text}).encode("utf-8")
            )
            out.append(summary_payload)
        
        return out

    # Endpoint to view raw payload data
    async def view_raw_data(req: web.Request) -> web.Response:
        """View the raw payload data for a given S3 key."""
        s3_key = req.match_info['key']
        
        try:
            stored_data = await codec.get_payload_from_s3(s3_key)
            if stored_data is None:
                return web.Response(
                    text=json.dumps({"error": f"Key not found: {s3_key}"}),
                    content_type="application/json",
                    status=404
                )
            
            # Parse and return the original payload
            original_payload = Payload.FromString(stored_data)
            
            # Try to decode as text, fall back to base64 for binary data
            try:
                data_text = original_payload.data.decode("utf-8")
                return web.Response(
                    text=data_text,
                    content_type="text/plain"
                )
            except UnicodeDecodeError:
                import base64
                data_b64 = base64.b64encode(original_payload.data).decode("utf-8")
                return web.Response(
                    text=f"Binary data (base64):\n{data_b64}",
                    content_type="text/plain"
                )
                
        except Exception as e:
            return web.Response(
                text=json.dumps({"error": f"Failed to retrieve data: {str(e)}"}),
                content_type="application/json",
                status=500
            )

    # General purpose payloads-to-payloads
    async def apply(
        fn: Callable[[Iterable[Payload]], Awaitable[List[Payload]]], req: web.Request
    ) -> web.Response:
        # Read payloads as JSON
        assert req.content_type == "application/json"
        data = await req.read()
        payloads = json_format.Parse(data.decode("utf-8"), Payloads())
        
        # Apply
        payloads = Payloads(payloads=await fn(payloads.payloads))

        # Apply CORS and return JSON
        resp = await cors_options(req)
        resp.content_type = "application/json"
        resp.text = json_format.MessageToJson(payloads)

        return resp

    # Build app
    app = web.Application()
    app.add_routes(
        [
            web.post("/encode", partial(apply, codec.encode)),
            web.post("/decode", partial(apply, decode_with_urls)),
            web.get("/view/{key}", view_raw_data),
            web.options("/decode", cors_options),
        ]
    )
    return app

if __name__ == "__main__":
    web.run_app(build_codec_server(), host="127.0.0.1", port=8081)
```
<!--SNIPEND-->

### Run the codec server

```bash
uv run codec/codec_server.py
```

Then [configure the Web UI to use the codec server](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli).

### What it shows

Instead of raw keys:

```
abc123-def4-5678-9abc-def012345678
```

You will see text like:

```
"Claim check data (key: abc123-def4-5678-9abc-def012345678) - View at: http://localhost:8081/view/abc123-def4-5678-9abc-def012345678"
```

### Endpoints

- `POST /encode`: Encodes payloads using the claim check codec
- `POST /decode`: Returns helpful text with S3 key and view URL (no data reads)
- `GET /view/{key}`: Serves raw payload data for inspection
- `OPTIONS /decode`: Handles CORS preflight requests

The server also includes CORS handling for the local Web UI.

---

# Post-LLM guardrail with hard-rule overrides

Source: https://docs.temporal.io/ai/cookbook/guardrails-hard-rules-python

> >-

This recipe shows how to combine an LLM classifier with a deterministic guardrail layer. The LLM provides nuanced judgment for ambiguous cases; hard rules act as a safety net for unambiguous policy violations, overriding the LLM's verdict regardless of what it concluded.

The pattern answers a real problem: LLMs can be manipulated via prompt injection or hallucinate outright. For any decision with real consequences — content moderation, access control, transaction approval — you shouldn't rely on the LLM alone. Hard rules catch clear-cut cases deterministically; the LLM handles everything in the grey zone. Critically, when a hard rule fires, the LLM's original reasoning is preserved inside the override so every decision remains auditable.

The recipe uses a content moderation scenario: user-submitted text is classified as `safe`, `review`, or `block`. Hard rules override to `block` when contact information or banned keywords are detected, regardless of what the LLM concluded.

## Prerequisites

- Python 3.10+
- [uv](https://docs.astral.sh/uv/)
- A running Temporal server: `temporal server start-dev`
- `ANTHROPIC_API_KEY` environment variable set

## Run it

```bash
uv sync

# Terminal 1 — start the worker
uv run python -m worker

# Terminal 2 — submit two example workflows
uv run python -m start_workflow
```

## Expected output

```
--- Example 1: Hard rule override ---
Input: 'Great product! Contact me at john.doe@example.com for a special deal.'
Classification: block
Overridden by hard rule: True
Reasoning: Hard rule: contains email address (privacy policy violation).

[LLM classified as 'safe' — reasoning: The message is promotional but does not appear harmful.]

--- Example 2: LLM verdict stands ---
Input: 'I really enjoyed the hiking trail last weekend. The views were amazing!'
Classification: safe
Overridden by hard rule: False
Reasoning: Positive personal experience with no policy concerns.
```

In Example 1, the LLM's classification and reasoning are preserved inside brackets — the override is fully auditable.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

## Architecture

- **Models** (`models/`):
  - `signals.py`: `ContentSignals` — the text and metadata being classified
  - `verdict.py`: `LLMVerdict` (the LLM's raw classification, also used as the tool's input schema) and `Verdict` (adds `overridden_by_hard_rule`)
- **Guardrails** (`guardrails/hard_rules.py`): pure functions that check content against banned keywords, phone numbers, and email addresses, and escalate the verdict to `block` when one matches
- **Activity** (`activities/classify.py`): calls Claude via a forced tool call to get a structured `LLMVerdict`, then applies the hard rules
- **Workflow** (`workflows/classify_workflow.py`): orchestrates the single `classify` Activity call with a 3-attempt retry policy
- **Scripts**:
  - `worker.py`: runs the Temporal Worker
  - `start_workflow.py`: runs the two examples shown in [Expected output](#expected-output)

## Key patterns

### Forcing a structured verdict from the LLM

The Activity forces Claude to call a single tool, so the response is always a well-formed `LLMVerdict` instead of free-form text to parse:

<!--SNIPSTART activities/classify.py {"startPattern": "^_SUBMIT_VERDICT_TOOL = \\{$", "endPattern": "^\\}$"}-->
```python
_SUBMIT_VERDICT_TOOL = {
    "name": "submit_verdict",
    "description": "Submit your content moderation classification.",
    "input_schema": LLMVerdict.model_json_schema(),
}
```
<!--SNIPEND-->

<!--SNIPSTART activities/classify.py {"startPattern": "response = await client\\.messages\\.create\\($", "endPattern": "^\\s*\\)\\s*$", "selectedLines": ["1", "8-10"]}-->
```python
response = await client.messages.create(
    ...
    tools=[_SUBMIT_VERDICT_TOOL],
    tool_choice={"type": "tool", "name": "submit_verdict"},
)
```
<!--SNIPEND-->

### Overriding while preserving the original reasoning

`apply_hard_rules` never discards the LLM's own verdict — a rule can only escalate a verdict to `block`, and when it does, the LLM's reasoning is embedded in the result so the override stays auditable:

<!--SNIPSTART guardrails/hard_rules.py {"startPattern": "^def apply_hard_rules\\(signals: ContentSignals, llm_verdict: Verdict\\) -> Verdict:$", "endPattern": "^\\s*\\)\\s*$"}-->
```python
def apply_hard_rules(signals: ContentSignals, llm_verdict: Verdict) -> Verdict:
    """Post-filter: override the LLM verdict if a hard rule matches.

    When a rule fires, the LLM's original reasoning is embedded in the
    returned verdict so the override is auditable.
    """
    if llm_verdict.classification == "block":
        return llm_verdict

    hard = _hard_block(signals)
    if hard is None:
        return llm_verdict

    return Verdict(
        classification=hard.classification,
        confidence=hard.confidence,
        overridden_by_hard_rule=True,
        reasoning=(
            f"{hard.reasoning}\n\n"
            f"[LLM classified as '{llm_verdict.classification}' — "
            f"reasoning: {llm_verdict.reasoning}]"
        ),
    )
```
<!--SNIPEND-->

### Non-retryable Anthropic errors

Permanent client errors (bad request, authentication, permission-denied, not-found, unprocessable-entity) are classified as non-retryable so the Workflow doesn't keep retrying a request that can never succeed:

<!--SNIPSTART activities/classify.py {"startPattern": "^\\s*except \\($", "endPattern": "\\) from exc$"}-->
```python
except (
    anthropic.BadRequestError,
    anthropic.AuthenticationError,
    anthropic.PermissionDeniedError,
    anthropic.NotFoundError,
    anthropic.UnprocessableEntityError,
) as exc:
    # These errors will never succeed on a retry: a retired model identifier, for
    # example, returns 404. Retrying them under the default policy would loop
    # forever instead of surfacing the problem. Everything else, such as rate
    # limits and 5xx responses, propagates so Temporal can retry it.
    raise ApplicationError(
        str(exc),
        type=exc.__class__.__name__,
        non_retryable=True,
    ) from exc
```
<!--SNIPEND-->

## Extensions

This pattern can be extended to:
- Add more hard rules (regex, allow/deny lists, PII detectors) without touching the LLM prompt
- Log every override to a compliance or audit sink for review
- Route `review` verdicts to a human approval step — see [Human-in-the-loop AI agent](./human-in-the-loop-python.mdx)

---

# Durable MCP weather server

Source: https://docs.temporal.io/ai/cookbook/hello-world-durable-mcp-server

> Build a durable MCP server in Python that runs weather tools reliably with Temporal Workflows.

This example demonstrates how to build a durable MCP (Model Context Protocol) server using Temporal Workflows for Durable Execution. The server exposes weather tools that fetch alerts and forecasts from the National Weather Service API.

MCP tools are "actions" that the MCP server can perform. Within a given MCP tool, there are often multiple steps (API calls, functions, etc.) that must happen in a certain order to complete an action. For example, the `get_forecast` tool performs the following steps:
- Call the National Weather Service API to find which region corresponds to the given latitude and longitude coordinates
- Call the National Weather Service API again to retrieve the forecast for that region
- Format and return the response to the user

This one tool takes several steps to complete a given action. You implement these steps in a Temporal Workflow, which provides durability. This means that whenever your MCP tool is called, it kicks off the Temporal Workflow, and every step (API call, function) is executed reliably and all the way to completion.

This recipe uses [FastMCP](https://github.com/PrefectHQ/fastmcp) to implement the MCP Server and create tools using the decorator `@mcp.tool`.

> **📝 Note:**
> External API calls are made within Temporal Activities. This ensures that network requests are retried appropriately and failures are handled.

This recipe highlights the following key design decisions:
- **Separation of concerns**: MCP tools act as thin wrappers that start Temporal Workflows. All business logic lives in Workflows, ensuring durability and reliability.
- **Durable Execution**: By moving multi-step operations into Temporal Workflows, you guarantee that operations complete even in the face of failures, network issues, or process restarts.
- **Activity-based external calls**: All external API calls (like NWS API requests) are made within Temporal Activities, which provides automatic retries and proper error handling.
- **Retry policies**: Workflows use configurable retry policies to handle transient failures.

Also see this foundational [recipe for basic tool calling](/ai/cookbook/tool-call-openai-python) using the same weather tools.

## Application components

This example includes the following components:
- The [MCP server](#create-the-mcp-server) (mcp_server.py) that exposes tools via FastMCP and starts Temporal Workflows
- The [Workflows](#create-the-workflows) (weather_workflows.py) that orchestrate the multi-step weather operations
- The [Activity](#create-the-activity) (weather_activities.py) for making external API calls to the National Weather Service
- The [Worker](#create-the-worker) (worker.py) (that manages the Workflows and Activities)
- [Config for Claude Desktop](#configure-claude-desktop) (claude_desktop_config.json) for connecting the MCP server to Claude Desktop

## Create the MCP server

The MCP server is implemented using FastMCP and exposes tools via the `@mcp.tool` decorator. Each tool is a thin wrapper that starts a Temporal Workflow and waits for the result. This design ensures that all business logic lives in durable Workflows.

*File: mcp_servers/weather.py*

<!--SNIPSTART:file mcp_servers/weather.py-->
```python
from fastmcp import FastMCP
from temporalio.client import Client
from temporalio.envconfig import ClientConfig

# Initialize FastMCP server
mcp = FastMCP("weather")

# Temporal client setup (do this once, then reuse)
temporal_client = None

async def get_temporal_client():
    global temporal_client
    if not temporal_client:
        config = ClientConfig.load_client_connect_config()
        config.setdefault("target_host", "localhost:7233")
        temporal_client = await Client.connect(**config)
    return temporal_client

@mcp.tool
async def get_alerts(state: str) -> str:
    """Get weather alerts for a US state.

    Args:
        state: Two-letter US state code (e.g. CA, NY)
    """
    # The business logic has been moved into the Temporal Workflow, the MCP tool kicks off the Workflow
    client = await get_temporal_client()
    handle = await client.start_workflow(
        "GetAlerts",
        state,
        id=f"alerts-{state.lower()}",
        task_queue="weather-task-queue",
    )
    return await handle.result()

@mcp.tool
async def get_forecast(latitude: float, longitude: float) -> str:
    """Get weather forecast for a location.

    Args:
        latitude: Latitude of the location
        longitude: Longitude of the location
    """
    # The business logic has been moved into the Temporal Workflow, the MCP tool kicks off the Workflow
    client = await get_temporal_client()
    handle = await client.start_workflow(
        workflow="GetForecast",
        args=[latitude, longitude],
        id=f"forecast-{latitude}-{longitude}",
        task_queue="weather-task-queue",
    )
    return await handle.result()

if __name__ == "__main__":
    # Initialize and run the server
    mcp.run(transport="stdio")
```
<!--SNIPEND-->

## Create the Workflows

The Workflows contain the business logic for fetching weather data. They orchestrate multiple steps, including API calls and data formatting. By implementing this logic in Workflows, you ensure that operations complete reliably even if there are failures or interruptions.

### GetAlerts Workflow

The `GetAlerts` workflow fetches active weather alerts for a US state.

*File: workflows/weather_workflows.py*

<!--SNIPSTART workflows/weather_workflows.py {"startPattern": "^from datetime import timedelta$", "endPattern": "join\\(alerts\\)"}-->
```python
from datetime import timedelta

from temporalio import workflow
from temporalio.common import RetryPolicy

retry_policy = RetryPolicy(
    maximum_attempts=0,  # Infinite retries
    initial_interval=timedelta(seconds=2),
    maximum_interval=timedelta(minutes=1),
    backoff_coefficient=1.0,
)

# Constants
NWS_API_BASE = "https://api.weather.gov"
USER_AGENT = "weather-app/1.0"

# Import Activities and models, passing them through the sandbox
with workflow.unsafe.imports_passed_through():
    from activities.weather_activities import make_nws_request

def format_alert(feature: dict) -> str:
    """Format an alert feature into a readable string."""
    props = feature["properties"]
    return f"""
Event: {props.get("event", "Unknown")}
Area: {props.get("areaDesc", "Unknown")}
Severity: {props.get("severity", "Unknown")}
Description: {props.get("description", "No description available")}
Instructions: {props.get("instruction", "No specific instructions provided")}
"""

@workflow.defn
class GetAlerts:
    @workflow.run
    async def get_alerts(self, state: str) -> str:
        """Get weather alerts for a US state.

        Args:
            state: Two-letter US state code (e.g. CA, NY)
        """
        url = f"{NWS_API_BASE}/alerts/active/area/{state}"
        data = await workflow.execute_activity(
            make_nws_request,
            url,
            schedule_to_close_timeout=timedelta(seconds=40),
            retry_policy=retry_policy,
        )

        if not data or "features" not in data:
            return "Unable to fetch alerts or no alerts found."

        alerts = [format_alert(feature) for feature in data["features"]]
        return "\n---\n".join(alerts)
```
<!--SNIPEND-->

### GetForecast Workflow

The `GetForecast` workflow demonstrates a multi-step operation: it first fetches the forecast grid endpoint for a location, then uses that information to fetch the detailed forecast. 

*File: workflows/weather_workflows.py*

<!--SNIPSTART workflows/weather_workflows.py:get-forecast-workflow-->
```python
@workflow.defn
class GetForecast:
    @workflow.run
    async def get_forecast(self, latitude: float, longitude: float) -> str:
        """Get weather forecast for a location.

        Args:
            latitude: Latitude of the location
            longitude: Longitude of the location
        """
        # First get the forecast grid endpoint
        points_url = f"{NWS_API_BASE}/points/{latitude},{longitude}"
        points_data = await workflow.execute_activity(
            make_nws_request,
            points_url,
            schedule_to_close_timeout=timedelta(seconds=40),
            retry_policy=retry_policy,
        )

        if not points_data:
            return "Unable to fetch forecast data for this location."

        # Get the forecast URL from the points response
        forecast_url = points_data["properties"]["forecast"]
        forecast_data = await workflow.execute_activity(
            make_nws_request,
            forecast_url,
            schedule_to_close_timeout=timedelta(seconds=40),
            retry_policy=retry_policy,
        )
        if not forecast_data:
            return "Unable to fetch detailed forecast."

        # Format the periods into a readable forecast
        periods = forecast_data["properties"]["periods"]
        forecasts = []
        for period in periods[:5]:  # Only show next 5 periods
            forecast = f"""
    {period["name"]}:
    Temperature: {period["temperature"]}°{period["temperatureUnit"]}
    Wind: {period["windSpeed"]} {period["windDirection"]}
    Forecast: {period["detailedForecast"]}
    """
            forecasts.append(forecast)

        return "\n---\n".join(forecasts)
```
<!--SNIPEND-->

## Create the Activity

This recipe creates an Activity for making HTTP requests to the National Weather Service API. All external API calls happen within Activities, which provides automatic retries and proper error handling through Temporal's retry mechanisms.

*File: activities/weather_activities.py*

<!--SNIPSTART:file activities/weather_activities.py-->
```python
from typing import Any

import httpx
from temporalio import activity

USER_AGENT = "weather-app/1.0"

# External calls happen via Activities
@activity.defn
async def make_nws_request(url: str) -> dict[str, Any] | None:
    """Make a request to the NWS API with proper error handling."""
    headers = {"User-Agent": USER_AGENT, "Accept": "application/geo+json"}
    async with httpx.AsyncClient() as client:
        response = await client.get(url, headers=headers, timeout=5.0)
        response.raise_for_status()
        return response.json()
```
<!--SNIPEND-->

## Create the Worker

The Worker is the process that executes Activities and Workflows. 

*File: worker.py*

<!--SNIPSTART:file worker.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities.weather_activities import make_nws_request
from workflows.weather_workflows import GetAlerts, GetForecast

async def main():
    # Connect to Temporal server 
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    # Register both Workflows and the Activity 
    worker = Worker(
        client,
        task_queue="weather-task-queue",
        workflows=[GetAlerts, GetForecast],
        activities=[make_nws_request],
    )
    print("Worker started. Listening for workflows...")
    await worker.run()

# Start worker with both Workflows and Activities
if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Configure Claude Desktop

This recipe uses Claude Desktop as the MCP Client. To use this MCP server with Claude Desktop, you need to configure it in your Claude Desktop configuration file. The config file tells Claude Desktop how to start the MCP server.

*File: claude_desktop_config.json*

```json
{
  "mcpServers": {
    "weather": {
      "command": "uv",
      "args": [
        "--directory",
        "<full path to the directory containing the weather.py>",
        "run",
        "mcp_servers/weather.py"
      ]
    }
  }
}
```

Replace `<full path to the directory containing the weather.py>` with the absolute path to the `hello_world_durable_mcp_server` directory.

## Run the MCP server

1. Install dependencies:
   ```bash
   uv sync
   ```

2. Start a Temporal server:
   ```bash
   # Using Temporal CLI
   temporal server start-dev
   ```

3. Start the worker in one terminal:
   ```bash
   uv run worker.py
   ```

4. Configure Claude Desktop by adding the configuration from `claude_desktop_config.json` to your Claude Desktop config file (typically located at `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS).

5. Restart Claude Desktop to load the MCP server.

Once configured, you should see the tool appear under the slider icon underneath the Claude Desktop chat input box.

You can now ask Claude something like `What is the weather like in San Francisco, CA?`. Claude Desktop will understand that it needs to use the `get_forecast` tool in the Weather MCP server that you just configured.

> **📝 Note:**
> The National Weather Service API only supports US locations. Asking about weather in non-US locations (e.g., "What is the weather in London?") will result in a 404 error from the API.

After tool execution, Claude Desktop will send the result over to the LLM (with other context) for human formatting, and then returns that result to the user. You can see these and other MCP-related actions in the `mcp_server.log`.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, the Worker and the MCP server connect to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in the terminal where you start the Worker, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

Claude Desktop starts the MCP server itself, so the server doesn't see that `export`. Set the profile in the server's entry in your Claude Desktop config instead:

```json
"weather": {
  "command": "uv",
  "args": ["--directory", "<full path to the directory containing the weather.py>", "run", "mcp_servers/weather.py"],
  "env": { "TEMPORAL_PROFILE": "cloud" }
}
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Hello world with LiteLLM

Source: https://docs.temporal.io/ai/cookbook/hello-world-litellm-python

> Integrate LiteLLM into a durable Temporal Workflow in Python to call and switch between LLM providers.

[LiteLLM](https://github.com/BerriAI/litellm) is a library for calling LLMs from Python. It makes it easy to access, and switch between, many providers, including OpenAI, Anthropic, Google, and more.

This recipe mirrors the [Basic Python recipe](./hello-world-openai-responses-python.mdx), but swaps the OpenAI SDK for LiteLLM. The Workflow still delegates LLM calls to an Activity, letting Temporal coordinate retries and durability, while LiteLLM forwards those calls to your configured provider.

Key points:

- A reusable Activity that wraps `litellm.acompletion` and keeps retries in Temporal.
- The most common LiteLLM parameters are on `LiteLLMRequest` ensuring type checking and IDE completion. Others may be passed via the `extra_options` dictionary, which functions as `kwargs` for `litellm.acompletion`.
- The Activity returns the full LiteLLM response for processing by the Workflow.

## Create the Activity

`activities/models.py`

<!--SNIPSTART:file activities/models.py-->
```python
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Type, Union

@dataclass
class LiteLLMRequest:
    """Lightweight container for the handful of LiteLLM options this sample surfaces."""

    model: str
    messages: List[Dict[str, Any]]

    # Optional knobs: limited to the most common tweaks.
    temperature: Optional[float] = (
        None  # Controls response creativity without extra ceremony.
    )
    max_tokens: Optional[int] = None  # Caps response length/cost.
    timeout: Optional[Union[float, int]] = (
        None  # Lets callers bound slow provider responses.
    )
    response_format: Optional[Union[dict, Type[Any]]] = (
        None  # Hook for JSON/object style responses.
    )

    # Escape hatch for advanced parameters we do not model explicitly.
    extra_options: Dict[str, Any] = field(default_factory=dict)

    def to_acompletion_kwargs(self) -> Dict[str, Any]:
        """Convert this request to kwargs suitable for litellm.acompletion()."""
        kwargs = {
            "model": self.model,
            "messages": self.messages,
        }

        optional_values = {
            "temperature": self.temperature,
            "max_tokens": self.max_tokens,
            "timeout": self.timeout,
            "response_format": self.response_format,
        }

        for key, value in optional_values.items():
            if value is not None:
                kwargs[key] = value

        if self.extra_options:
            kwargs.update(self.extra_options)

        return kwargs
```
<!--SNIPEND-->

`activities/litellm_completion.py`

<!--SNIPSTART:file activities/litellm_completion.py-->
```python
from typing import Any, Dict

import litellm
from temporalio import activity
from temporalio.exceptions import ApplicationError

from activities.models import LiteLLMRequest

@activity.defn(name="activities.litellm_completion.create")
async def create(request: LiteLLMRequest) -> Dict[str, Any]:
    # Temporal best practice: disable LiteLLM retries and let Temporal handle them.
    kwargs = request.to_acompletion_kwargs()
    kwargs["num_retries"] = 0

    try:
        response = await litellm.acompletion(**kwargs)
    except (
        litellm.AuthenticationError,
        litellm.BadRequestError,
        litellm.InvalidRequestError,
        litellm.UnsupportedParamsError,
        litellm.JSONSchemaValidationError,
        litellm.ContentPolicyViolationError,
        litellm.NotFoundError,
    ) as exc:
        raise ApplicationError(
            str(exc),
            type=exc.__class__.__name__,
            non_retryable=True,
        ) from exc
    except litellm.APIError:
        raise

    return response
```
<!--SNIPEND-->

LiteLLM supports many providers. Configure credentials via environment variables (for example `OPENAI_API_KEY`) before running the Activity. For Google-hosted models (Vertex AI or Gemini), the sample relies on the `google-cloud-aiplatform` and `google-auth` dependencies included in `pyproject.toml`; set the usual Google application credentials (`GOOGLE_APPLICATION_CREDENTIALS`, `GOOGLE_CLOUD_PROJECT`, `VERTEXAI_LOCATION`, etc.) so LiteLLM can obtain an access token.

## Create the Workflow

`workflows/hello_world_workflow.py`

<!--SNIPSTART:file workflows/hello_world_workflow.py-->
```python
from datetime import timedelta

from temporalio import workflow

from activities.models import LiteLLMRequest

@workflow.defn
class HelloWorld:
    @workflow.run
    async def run(self, input: str) -> str:
        messages = [
            {"role": "system", "content": "You only respond in haikus."},
            {"role": "user", "content": input},
        ]
        response = await workflow.execute_activity(
            "activities.litellm_completion.create",
            LiteLLMRequest(
                # LiteLLM allows you to switch between models easily
                # model="gpt-4o-mini",
                model="gemini/gemini-3.5-flash-lite",
                messages=messages,
            ),
            start_to_close_timeout=timedelta(seconds=30),
        )

        message = response["choices"][0]["message"]["content"]
        if isinstance(message, list):
            message = "".join(
                part.get("text", "") for part in message if isinstance(part, dict)
            )

        return message
```
<!--SNIPEND-->

Temporal manages Activity retries, so LiteLLM's retry helper is disabled via `num_retries=0`. Use the `extra_options` escape hatch on `LiteLLMRequest` if you need to surface additional LiteLLM parameters without editing the sample.

## Create the Worker

`worker.py`

<!--SNIPSTART:file worker.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities import litellm_completion
from workflows.hello_world_workflow import HelloWorld

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    worker = Worker(
        client,
        task_queue="hello-world-python-task-queue",
        workflows=[
            HelloWorld,
        ],
        activities=[
            litellm_completion.create,
        ],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Create the Workflow Starter

`start_workflow.py`

<!--SNIPSTART:file start_workflow.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig

from workflows.hello_world_workflow import HelloWorld

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    # Submit the Hello World workflow for execution
    result = await client.execute_workflow(
        HelloWorld.run,
        "Tell me about recursion in programming.",
        id="my-workflow-id",
        task_queue="hello-world-python-task-queue",
    )
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal Dev Server:

```bash
temporal server start-dev
```

Install dependencies

```bash
uv sync
```

Set the appropriate environment variables before launching the worker (for example `export OPENAI_API_KEY=...` or export `GEMINI_API_KEY=...`) so LiteLLM can reach your chosen provider.

Run the worker:

```bash
uv run worker.py
```

Start the workflow:

```bash
uv run start_workflow.py
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Hello world

Source: https://docs.temporal.io/ai/cookbook/hello-world-openai-responses-python

> Call an LLM from a durable Temporal Workflow in Python using the OpenAI API library.

This is a simple example showing how to call an LLM from Temporal using the [OpenAI Python API library](https://github.com/openai/openai-python).

Being an external API call, the LLM invocation happens in a Temporal Activity.

This recipe highlights three key design decisions:

- A generic Activity for invoking an LLM API. This Activity can be re-used with different arguments throughout your codebase.
- Configuring the Temporal client with a `dataconverter` to allow serialization of Pydantic types.
- Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery.

## Create the Activity

We create a wrapper for the `create` method of the `AsyncOpenAI` client object.
This is a generic Activity that invokes the OpenAI LLM.

We set `max_retries=0` when creating the `AsyncOpenAI` client.
This moves the responsibility for retries from the OpenAI client to Temporal.

In this implementation, we include only the `instructions` and `input` argument, but it could be extended to others.

*File: activities/openai_responses.py*

<!--SNIPSTART:file activities/openai_responses.py-->
```python
from dataclasses import dataclass

from openai import AsyncOpenAI
from openai.types.responses import Response
from temporalio import activity

# Temporal best practice: Create a data structure to hold the request parameters.
@dataclass
class OpenAIResponsesRequest:
    model: str
    instructions: str
    input: str

@activity.defn
async def create(request: OpenAIResponsesRequest) -> Response:
    # Temporal best practice: Disable retry logic in OpenAI API client library.
    client = AsyncOpenAI(max_retries=0)

    resp = await client.responses.create(
        model=request.model,
        instructions=request.instructions,
        input=request.input,
        timeout=15,
    )

    return resp
```
<!--SNIPEND-->

## Create the Workflow

In this example, we take the user input and generate a response in haiku format, using the OpenAI Responses Activity. The
Workflow returns `result.output_text` from the OpenAI `Response`.

As per usual, the Activity retry configuration is set here in the Workflow. In this case, a retry policy is not specified
so the default retry policy is used (exponential backoff with 1s initial interval, 2.0 backoff coefficient, max interval
100× initial, unlimited attempts, no non-retryable errors).

The Activity module is imported inside `workflow.unsafe.imports_passed_through()`. Importing it pulls in the OpenAI
client and, through it, `httpx`, which touches modules that the Workflow sandbox restricts at import time. Activity code
runs outside the sandbox, so passing the module through is safe.

*File: workflows/hello_world_workflow.py*

<!--SNIPSTART:file workflows/hello_world_workflow.py-->
```python
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import openai_responses

@workflow.defn
class HelloWorld:
    @workflow.run
    async def run(self, input: str) -> str:
        system_instructions = "You only respond in haikus."
        result = await workflow.execute_activity(
            openai_responses.create,
            openai_responses.OpenAIResponsesRequest(
                model="gpt-4o-mini",
                instructions=system_instructions,
                input=input,
            ),
            start_to_close_timeout=timedelta(seconds=30),
        )
        return result.output_text
 
```
<!--SNIPEND-->

## Create the Worker

Create the process for executing Activities and Workflows.
We configure the Temporal client with `pydantic_data_converter` so Temporal can serialize/deserialize output of the OpenAI SDK.

*File: worker.py*

<!--SNIPSTART:file worker.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities import openai_responses
from workflows.hello_world_workflow import HelloWorld

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    worker = Worker(
        client,
        task_queue="hello-world-python-task-queue",
        workflows=[
            HelloWorld,
        ],
        activities=[
            openai_responses.create,
        ],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Create the Workflow Starter

The starter script submits the Workflow to Temporal for execution, then waits for the result and prints it out.
It uses the `pydantic_data_converter` to match the Worker configuration.

*File: start_workflow.py*

<!--SNIPSTART:file start_workflow.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig

from workflows.hello_world_workflow import HelloWorld

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    # Submit the Hello World workflow for execution
    result = await client.execute_workflow(
        HelloWorld.run,
        "Tell me about recursion in programming.",
        id="my-workflow-id-2",
        task_queue="hello-world-python-task-queue",
    )
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal Dev Server:

```bash
temporal server start-dev
```

Run the worker:

```bash
uv run worker.py
```

Start execution:

```bash
uv run start_workflow.py
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Retry policy from HTTP responses

Source: https://docs.temporal.io/ai/cookbook/http-retry-enhancement-python

> Extract retry information from HTTP response headers and pass it to Temporal's retry mechanisms in Python.

This recipe extends the [Basic Example](./hello-world-openai-responses-python.mdx) to show how to extract retry information from HTTP response headers and make it available to Temporal's retry mechanisms.

HTTP response codes and headers on API calls have implications for retry behavior.
For example, an HTTP `404 Not Found` generally represents an application-level error that should not be retried.
By contrast, a `500 Internal Server Error` is typically transient, so should be retried.
Servers can also set the `Retry-After` header to tell the client when to retry.

This recipe introduces a utility function that processes the HTTP response and populates a Temporal `ApplicationError` to provide inputs to the retry mechanism.
Temporal combines this information with other configuration, such as timeouts and exponential backoff, to implement the complete retry policy.

## Generate Temporal ApplicationErrors from HTTP responses

This recipe introduces a utility function that takes an `httpx.Response` object and returns a Temporal `ApplicationError` with two key fields populated: `non-retryable` and `next_retry_delay`.

The `non-retryable` is determined by categorizing the HTTP status codes.
The `X-Should-Retry` HTTP response header, when present, overrides the status code.

**Example Retryable Status Codes:**

- **408 Request Timeout** → Retry because server is unresponsive, which can have many causes
- **409 Conflict** → Retry when resource is temporarily locked or in use
- **429 Too Many Requests** → Retry after rate limit cooldown (respect `Retry-After` header when available)
- **500 Internal Server Error** → Retry for temporary server issues
- **502 Bad Gateway** → Retry when upstream server is temporarily unavailable
- **503 Service Unavailable** → Retry when service is temporarily overloaded
- **504 Gateway Timeout** → Retry when upstream server times out

**Example Non-Retryable Status Codes:**

- **400 Bad Request** → Do not retry - fix request format/parameters
- **401 Unauthorized** → Do not retry - provide valid authentication
- **403 Forbidden** → Do not retry - insufficient permissions
- **404 Not Found** → Do not retry - resource does not exist
- **422 Unprocessable Entity** → Do not retry - fix request validation errors
- **Other 4xx Client Errors** → Do not retry - client-side issues need fixing
- **2xx Success** → Do not expect to see this - call succeeded
- **3xx Redirects** → Do not expect to see this - typically handled by httpx (with `follow_redirects=True`)

If the error is retryable and if the `Retry-After` header is present, the function parses it to set the retry delay.

This implementation duplicates logic present in the [OpenAI Python API Library](https://github.com/openai/openai-python), where it is part of the code generated by [Stainless](https://www.stainless.com/).
Duplicating the logic makes sense because it is not accessible via the public library interface and because it applies to HTTP APIs in general, not just the OpenAI API.

*File: util/translate_http_errors.py*

<!--SNIPSTART:file util/translate_http_errors.py-->
```python
import email.utils
import time
from datetime import timedelta
from typing import Optional, Tuple

from temporalio import workflow
from temporalio.exceptions import ApplicationError

with workflow.unsafe.imports_passed_through():
    from httpx import Headers, Response

# Adapted from the OpenAI Python client (https://github.com/openai/openai-python/blob/main/src/openai/_base_client.py)
# which is generated by the Stainless SDK Generator.
def _parse_retry_after_header(response_headers: Optional[Headers] = None) -> float | None:
    """Returns a float of the number of seconds (not milliseconds) to wait after retrying, or None if unspecified.

    About the Retry-After header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
    See also  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#syntax
    """
    if response_headers is None:
        return None

    # First, try the non-standard `retry-after-ms` header for milliseconds,
    # which is more precise than integer-seconds `retry-after`
    try:
        retry_ms_header = response_headers.get("retry-after-ms", None)
        return float(retry_ms_header) / 1000
    except (TypeError, ValueError):
        pass

    # Next, try parsing `retry-after` header as seconds (allowing nonstandard floats).
    retry_header = response_headers.get("retry-after")
    try:
        # note: the spec indicates that this should only ever be an integer
        # but if someone sends a float there's no reason for us to not respect it
        return float(retry_header)
    except (TypeError, ValueError):
        pass

    # Last, try parsing `retry-after` as a date.
    retry_date_tuple = email.utils.parsedate_tz(retry_header)
    if retry_date_tuple is None:
        return None

    retry_date = email.utils.mktime_tz(retry_date_tuple)
    return float(retry_date - time.time())

def _should_retry(response: Response) -> Tuple[bool, str]:
    # Note: this is not a standard header
    should_retry_header = response.headers.get("x-should-retry")

    # If the server explicitly says whether or not to retry, obey.
    if should_retry_header == "true":
        return True, f"Server requested retry via x-should-retry=true header (HTTP {response.status_code})"
    if should_retry_header == "false":
        return False, f"Server prevented retry via x-should-retry=false header (HTTP {response.status_code})"

    # Retry on request timeouts.
    if response.status_code == 408:
        return True, f"HTTP request timeout ({response.status_code}), will retry with backoff"

    # Retry on lock timeouts.
    if response.status_code == 409:
        return True, f"HTTP conflict/lock timeout ({response.status_code}), will retry with backoff"

    # Retry on rate limits.
    if response.status_code == 429:
        return True, f"HTTP rate limit exceeded ({response.status_code}), will retry with backoff"

    # Retry internal errors.
    if response.status_code >= 500:
        return True, f"HTTP server error ({response.status_code}), will retry with backoff"

    return False, f"HTTP client error ({response.status_code}), not retrying - check your request"

def http_response_to_application_error(response: Response) -> ApplicationError:
    """Transform HTTP response into Temporal ApplicationError for retry handling.
    
    This function implements generic HTTP retry logic based on status codes and headers.
    
    Args:
        response: The httpx.Response from a failed HTTP request
        
    Returns:
        ApplicationError: Always returns an ApplicationError configured for Temporal's retry system:
            - non_retryable: False for retryable errors, True for non-retryable
            - next_retry_delay: Server-provided delay hint (if valid)
            
    Note: 
        Even when x-should-retry=true, this function returns an ApplicationError with
        non_retryable=False rather than raising an exception, for cleaner functional style.
    """
    should_retry, retry_message = _should_retry(response)
    if should_retry:
        # Calculate the retry delay only when retrying
        retry_after = _parse_retry_after_header(response.headers)
        # Make sure that the retry delay is in a reasonable range
        if retry_after is not None and 0 < retry_after <= 60:
            retry_after = timedelta(seconds=retry_after)
        else:
            retry_after = None 
        
        # Add delay info for rate limits
        if response.status_code == 429 and retry_after is not None:
            retry_message = f"HTTP rate limit exceeded (429) (server requested {retry_after.total_seconds():.1f}s delay), will retry with backoff"
        
        return ApplicationError(
            retry_message,
            non_retryable=False,
            next_retry_delay=retry_after,
        )
    else:
        return ApplicationError(
            retry_message,
            non_retryable=True,
            next_retry_delay=None,
        )
```
<!--SNIPEND-->

## Raise the exception from the Activity

When API calls fail, the OpenAI Client raises an `APIStatusError` exception which contains a `response` field, containing the underlying `httpx.Response` object.
The Activity uses the `http_response_to_application_error` function defined above to translate this to a Temporal `ApplicationError`, which the Activity re-throws to pass the retry information to Temporal.

*File: activities/openai_responses.py*

<!--SNIPSTART:file activities/openai_responses.py-->
```python
from dataclasses import dataclass

from openai import APIStatusError, AsyncOpenAI
from openai.types.responses import Response
from temporalio import activity

from util.translate_http_errors import http_response_to_application_error

# Temporal best practice: Create a data structure to hold the request parameters.
@dataclass
class OpenAIResponsesRequest:
    model: str
    instructions: str
    input: str

@activity.defn
async def create(request: OpenAIResponsesRequest) -> Response:
    # Temporal best practice: Disable retry logic in OpenAI API client library.
    client = AsyncOpenAI(max_retries=0)

    try:
        resp = await client.responses.create(
            model=request.model,
            instructions=request.instructions,
            input=request.input,
            timeout=15,
        )
        return resp
    except APIStatusError as e:
        raise http_response_to_application_error(e.response) from e
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal Dev Server:

```bash
temporal server start-dev
```

Run the worker:

```bash
uv run worker.py
```

Start execution:

```bash
uv run start_workflow.py
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Human-in-the-loop AI agent

Source: https://docs.temporal.io/ai/cookbook/human-in-the-loop-python

> Add human-in-the-loop approval to a durable AI agent using Temporal Signals in Python.

This example demonstrates how to build an AI agent that requires human approval; we use Temporal Signals to bring that user input into the agent.

## Overview

The Workflow implements the agent flow:
1. Uses an LLM to analyze a user request and propose an action. 
2. If the proposed action is deemed risky, pauses and waits for human approval via Temporal Signal
3. Executes the action if auto-approved (if not risky) or human approved, or cancels if rejected/timed out

Key features:
- **Resource efficient waiting**: Can wait for approval for hours, days or indefinitely; while waiting, the agent consumes no compute resources.
- **Signal-based approval**: External systems send approval decisions via Temporal Signals
- **Durable timers**: Time limits placed on human-in-the-loop steps survive any execution disruptions.
- **Complete audit trail**: All decisions are logged for compliance

## Prerequisites

- Python 3.10+
- Temporal Service running locally
- OpenAI API key

## Setup

1. Install dependencies:
```bash
uv sync
```

2. Set your OpenAI API key:
```bash
export OPENAI_API_KEY='your-api-key-here'
```

3. Start Temporal Dev Server:
```bash
temporal server start-dev
```

## Run the recipe

### Start the Worker

In one terminal:
```bash
uv run worker.py
```

### Start a Workflow

In another terminal:
```bash
uv run start_workflow.py "Delete all test data from the production database"
```

The Workflow will start, analyze the request, and pause for approval. Watch the Worker output for instructions.

### Send approval decision

The Worker output will show the Workflow Id and request identifier. In another terminal, run the `send_approval` script to approve or reject:

**To approve:**
```bash
uv run send_approval.py <workflow-id> <request-id> approve "Looks good"
```

**To reject:**
```bash
uv run send_approval.py <workflow-id> <request-id> reject "Too risky"
```

### Testing timeout

To test timeout behavior, don't send any approval signal. After 5 minutes (default), the Workflow will automatically complete with a timeout result.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

## Architecture

- **Models** (`models/models.py`): Data structures for workflow input, approval requests and decisions
- **Activities**:
  - `openai_responses.py`: Generic LLM invocation activity
  - `execute_action.py`: Executes approved actions
    - The "execution" of approved actions in this sample logs messages.
    - In a realistic scenario, a set of tools will have been provided to the LLM and the result might be a recommended tool call. In this case, if approved, the agent would invoke the tool via an Activity. See the [agentic loop with tool calling](/ai/cookbook/agentic-loop-tool-call-openai-python) for guidance on how to use dynamic Activities, allowing the tools to be loosely coupled from the agent implementation.
  - `notify_approval_needed.py`: Notifies external systems of approval requests
    - In this sample the notification comes in the form of messages printed in the terminal running the worker.
    - In a realistic scenario, the notification activity may send emails, deliver messages to slack, etc.
- **Workflow** (`workflows/human_in_the_loop_workflow.py`): Orchestrates the approval process
- **Scripts**:
  - `worker.py`: Runs the Temporal worker
  - `start_workflow.py`: Starts workflow execution
  - `send_approval.py`: Helper script to send approval signals

## Key patterns

We use a Temporal Signal to inject information from the human into the waiting Workflow. The Signal is delivered from some UI (in this case the `send_approval.py` script) that uses a Temporal client to deliver the data.

![Diagram of a Temporal Client in a UI invoking a Signal handler, which writes to application state variables that the Workflow main loop waits on](https://docs.temporal.io/ai/cookbook/human-in-the-loop-python-assets-temporal-signal-handling.png)

Within the agent implementation there are three main elements to the solution.

### Local state within the Workflow implementation
This state will be written to via the Signal handler and will be part of the condition that defines the wait point.
<!--SNIPSTART workflows/human_in_the_loop_workflow.py {"startPattern": "^@workflow\\.defn$", "endPattern": "self\\.pending_request_id: Optional\\[str\\] = None"}-->
```python
@workflow.defn
class HumanInTheLoopWorkflow:
    def __init__(self):
        self.current_decision: Optional[ApprovalDecision] = None
        self.pending_request_id: Optional[str] = None
```
<!--SNIPEND-->
### Signal handler
The Workflow uses a Signal handler to receive approval decisions asynchronously:
<!--SNIPSTART workflows/human_in_the_loop_workflow.py {"startPattern": "^\\s*@workflow\\.signal$", "endPattern": "expected: \\{self\\.pending_request_id\\}\"", "selectedLines": ["1-2", "5-6"]}-->
```python
@workflow.signal
async def approval_decision(self, decision: ApprovalDecision):
    ...
    if decision.request_id == self.pending_request_id:
        self.current_decision = decision
        ...
```
<!--SNIPEND-->

### Waiting with timeout
The Workflow waits for approval with a configurable timeout:
<!--SNIPSTART workflows/human_in_the_loop_workflow.py {"startPattern": "await workflow\\.wait_condition\\(", "endPattern": "^\\s*\\)\\s*$"}-->
```python
await workflow.wait_condition(
    lambda: self.current_decision is not None,
    timeout=timedelta(seconds=timeout_seconds),
)
```
<!--SNIPEND-->

## Extensions

This pattern can be extended to support:
- Multiple approvers with voting
- Escalation workflows
- Conditional approval based on action risk
- Integration with Slack, email, or custom UIs
- Query handlers to check approval status

---

# Multi-Agent Orchestration — Google ADK + Temporal

Source: https://docs.temporal.io/ai/cookbook/multi-agent-adk-python

> >-

This recipe builds a **multi-agent dispatch pipeline** using
[Google ADK](https://adk.dev/) and the
[Google ADK integration for Temporal](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/google_adk_agents).
Two agents reason in parallel, a third synthesizes their output and submits a
structured decision — and every LLM call and every I/O tool call is a durable
Temporal Activity. (The final decision write is a plain in-workflow tool —
see [Structured output via session state](#structured-output-via-session-state).)

[![Watch the 60-second walkthrough](https://img.youtube.com/vi/Wq7hiN2KYnk/maxresdefault.jpg)](https://youtube.com/shorts/Wq7hiN2KYnk?feature=share)

## ADK ↔ Temporal mapping

| ADK concept              | Temporal mapping                                            |
| ------------------------ | ----------------------------------------------------------- |
| Orchestrator agent       | Pure Python inside the workflow (`Runner.run_async`)        |
| LLM call (`BaseLlm`)     | `invoke_model` activity (via `TemporalModel`)               |
| Tool call (`@activity`)  | A named activity (via `activity_tool`)                      |
| Session state            | Read back from the in-memory `SessionService` after the run |
| Agent reasoning          | Lives in the workflow; *durable, replayable*                |
| Anything that does I/O   | Lives in an activity; *retried, timed out, observed*        |

## Pipeline

```
                      ┌──────────────┐
                      │ Order input  │
                      └──────┬───────┘
                             │
            ┌────────────────┴────────────────┐
            │       ParallelAgent             │
            │ ┌─────────────┐ ┌─────────────┐ │
            │ │ Fleet Agent │ │ Customer    │ │
            │ │             │ │ Agent       │ │
            │ └─────────────┘ └─────────────┘ │
            └────────────────┬────────────────┘
                             │
                      ┌──────▼───────┐
                      │ Dispatch     │
                      │ Agent        │
                      └──────┬───────┘
                             │
                  tool_submit_assignment
                             │
                      ┌──────▼───────┐
                      │ Workflow     │
                      │ returns      │
                      │ AssignmentOut│
                      └──────────────┘
```

- **Fleet Agent** — checks driver positions, capacity, ETAs.
- **Customer Agent** — checks order priority, deadline, urgency.
  *(Runs in parallel with Fleet Agent.)*
- **Dispatch Agent** — reads both assessments from session state, picks a
  driver, submits the structured decision via `tool_submit_assignment`.

This recipe highlights:

- **Multi-agent composition with ADK** — `ParallelAgent` + `SequentialAgent`
  driven by a single `Runner` invocation.
- **LLM-as-activity** — `TemporalModel` routes every LLM call through an
  `invoke_model` activity. Each call appears as a separate event in
  workflow history with retries, timeouts, and a Temporal-UI summary.
- **Tool-as-activity** — `activity_tool` wraps a Temporal Activity so the
  agent can call it. Each of these tool invocations is its own activity
  event. (The Dispatch agent's final `tool_submit_assignment` call is a
  plain in-workflow tool, not an activity — it only writes to local
  session state.)
- **Structured output via session state** — the final agent calls a Python
  tool that writes to `tool_context.state`. The workflow reads that key
  back after the runner completes.
- **Sandbox-safe ADK imports** — ADK and `google.genai` are imported under
  `workflow.unsafe.imports_passed_through()` so the workflow sandbox does
  not reject them.

## Prerequisites

- Python 3.10 to 3.13
- [`uv`](https://docs.astral.sh/uv/) for dependency management
- A running Temporal Dev Server (`temporal server start-dev`)
- A Google API key with access to Gemini (`GOOGLE_API_KEY`)

## Setup

```bash
uv sync
export GOOGLE_API_KEY='your-api-key-here'
```

## Run the recipe

In one terminal, start the Temporal Dev Server:

```bash
temporal server start-dev
```

In a second terminal, start the worker:

```bash
uv run python worker.py
```

In a third terminal, kick off an assignment:

```bash
uv run python start_workflow.py
```

You should see a final assignment printed, e.g.:

```
Assigned driver: driver-a
Reasoning:       Closest available driver with capacity for VIP order
```

Open the Temporal UI at http://localhost:8233 to see each LLM call and
tool call recorded as its own activity in the workflow history.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

## Architecture

```
multi_agent_adk_python/
├── activities/
│   └── tools.py             # @activity.defn — fleet status, priorities, route info
├── models/
│   └── models.py            # AssignmentInput / AssignmentOutput (pydantic)
├── workflows/
│   ├── assignment_workflow.py  # agents + workflow inline
│   └── _activity_tool.py       # ADK ↔ Temporal tool adapter
├── worker.py
├── start_workflow.py
└── tests/
    ├── test_activities.py
    └── test_workflow.py        # workflow logic with a mocked invoke_model
```

### Activities as agent tools

Each tool the agents can call is a Temporal Activity:

*File: activities/tools.py*
<!--SNIPSTART activities/tools.py {"startPattern": "^async def tool_get_fleet_status\\(\\) -> str:$", "endPattern": "^    \\)$"}-->
```python
async def tool_get_fleet_status() -> str:
    """Return current fleet state: driver positions, capacity, and status."""
    return (
        "Fleet status:\n"
        "- driver-a: pos=(36.1147, -115.1728)  capacity=2/3  status=AVAILABLE\n"
        "- driver-b: pos=(36.1099, -115.1750)  capacity=0/3  status=AVAILABLE\n"
        "- driver-c: pos=(36.1162, -115.1745)  capacity=3/3  status=FULL\n"
        "- driver-d: pos=(36.1213, -115.1700)  capacity=1/3  status=AVAILABLE\n"
        "- driver-e: pos=(36.1080, -115.1760)  capacity=2/3  status=DISCONNECTED"
    )
```
<!--SNIPEND-->

In a real system the body would query a fleet database or hit an internal
service. Here we return canned strings so the recipe runs without any
backing infrastructure.

### TemporalModel — every LLM call is an activity

`TemporalModel` (from
`temporalio.contrib.google_adk_agents`) is an ADK `BaseLlm` whose
`generate_content_async` runs through a Temporal Activity. The plugin
registers that activity (`invoke_model`) on your worker for you.

*File: workflows/assignment_workflow.py*
<!--SNIPSTART workflows/assignment_workflow.py {"startPattern": "^def _fleet_agent\\(\\) -> Agent:$", "endPattern": "^    \\)$", "selectedLines": ["1-10", "30-32"]}-->
```python
def _fleet_agent() -> Agent:
    return Agent(
        name="fleet_agent",
        model=TemporalModel(
            DEFAULT_MODEL,
            activity_config=ActivityConfig(
                task_queue=TASK_QUEUE,
                summary="Fleet Agent — LLM reasoning",
            ),
        ),
        ...
        tools=[_fleet_status_tool, _route_info_tool],
        output_key="fleet_assessment",
    )
```
<!--SNIPEND-->

### activity_tool — every tool call is an activity

`activity_tool` wraps a `@activity.defn` so it presents to ADK as a regular
Python tool, but the call body executes via `workflow.execute_activity`:

*File: workflows/assignment_workflow.py*
<!--SNIPSTART workflows/assignment_workflow.py {"startPattern": "^_fleet_status_tool = activity_tool\\($", "endPattern": "^\\)$"}-->
```python
_fleet_status_tool = activity_tool(
    tool_get_fleet_status,
    task_queue=TASK_QUEUE,
    summary="Fleet Agent — get fleet status",
    start_to_close_timeout=timedelta(seconds=10),
    retry_policy=_TOOL_RETRY,
)
```
<!--SNIPEND-->

The local `workflows/_activity_tool.py` adds **graceful failure** on top of the
upstream `temporalio.contrib.google_adk_agents.workflow.activity_tool`:
when an activity execution fails (retry policy exhausted, non-retryable
application error, timeout), the wrapper catches the `ActivityError`
and returns it to the LLM as a string so the agent can adapt instead of
crashing the pipeline. (The retry attempts still appear in workflow
history.) Programming bugs — e.g. argument-binding errors — are not
caught and propagate normally. Upstream `activity_tool` (temporalio>=1.25)
already handles multi-arg activities and local non-workflow ADK runs.

### Compose the pipeline

*File: workflows/assignment_workflow.py*
<!--SNIPSTART workflows/assignment_workflow.py {"startPattern": "^def build_assignment_pipeline\\(\\) -> SequentialAgent:$", "endPattern": "^    \\)$"}-->
```python
def build_assignment_pipeline() -> SequentialAgent:
    """Compose the full pipeline: Parallel(Fleet, Customer) → Dispatch."""
    return SequentialAgent(
        name="order_assignment",
        sub_agents=[
            ParallelAgent(
                name="assessment_parallel",
                sub_agents=[_fleet_agent(), _customer_agent()],
            ),
            _dispatch_agent(),
        ],
    )
```
<!--SNIPEND-->

Fleet and Customer agents run concurrently inside `ParallelAgent`. When
both finish, the Dispatch agent runs. Each sub-agent's `output_key`
(`fleet_assessment`, `customer_assessment`) writes its final response into
session state — that's how the Dispatch agent gets the upstream context
without passing it explicitly.

### Structured output via session state

The final agent calls a plain Python tool (not a Temporal activity) that
writes the decision into ADK session state:

*File: workflows/assignment_workflow.py*
<!--SNIPSTART workflows/assignment_workflow.py {"startPattern": "^async def tool_submit_assignment\\($", "endPattern": "Assignment submitted"}-->
```python
async def tool_submit_assignment(
    tool_context: ToolContext,
    driver_id: str,
    reasoning_summary: str,
) -> str:
    """Submit the final order assignment. You MUST call this tool with your decision.

    Args:
        driver_id: The driver to assign the order to (e.g. "driver-a").
        reasoning_summary: One-sentence explanation of the choice.
    """
    tool_context.state["assignment"] = {
        "driver_id": driver_id,
        "reasoning_summary": reasoning_summary,
    }
    return "Assignment submitted."
```
<!--SNIPEND-->

The workflow runs the pipeline to exhaustion, then reads that key back:

*File: workflows/assignment_workflow.py*
<!--SNIPSTART workflows/assignment_workflow.py:workflow-run-tail-->
```python
async for _ in runner.run_async(
    user_id="workflow",
    session_id=session.id,
    new_message=Content(parts=[Part(text=prompt)]),
):
    pass

updated = await session_service.get_session(
    app_name=APP_NAME,
    user_id="workflow",
    session_id=session.id,
)
state = (updated.state if updated else None) or {}
assignment = state.get("assignment") or {}

return AssignmentOutput(
    driver_id=assignment.get("driver_id", ""),
    reasoning_summary=assignment.get(
        "reasoning_summary", "Dispatch agent did not submit a decision."
    ),
)
```
<!--SNIPEND-->

This pattern — a tool call that writes structured output into session
state — is how you reliably extract a typed decision from a multi-agent
pipeline that emits many intermediate events.

### Sandbox-safe ADK imports

ADK and `google.genai` aren't safe under Temporal's workflow sandbox by
default, so they're imported under
`workflow.unsafe.imports_passed_through()`:

*File: workflows/assignment_workflow.py*
<!--SNIPSTART workflows/assignment_workflow.py {"startPattern": "^with workflow\\.unsafe\\.imports_passed_through\\(\\):$", "endPattern": "^    from temporalio\\.contrib\\.google_adk_agents import TemporalModel$"}-->
```python
with workflow.unsafe.imports_passed_through():
    from google.adk.agents import Agent, ParallelAgent, SequentialAgent
    from google.adk.runners import Runner
    from google.adk.sessions import InMemorySessionService
    from google.adk.tools import ToolContext
    from google.genai.types import Content, Part
    from temporalio.contrib.google_adk_agents import TemporalModel
```
<!--SNIPEND-->

The `GoogleAdkPlugin` registered on the worker handles the rest of the
sandbox passthroughs and deterministic-runtime overrides ADK needs (UUIDs,
clocks).

### Worker — one queue, one plugin

*File: worker.py*
<!--SNIPSTART worker.py {"startPattern": "^    config = ClientConfig\\.load_client_connect_config\\(\\)$", "endPattern": "^    await worker\\.run\\(\\)$"}-->
```python
config = ClientConfig.load_client_connect_config()
config.setdefault("target_host", "localhost:7233")
client = await Client.connect(
    **config,
    data_converter=pydantic_data_converter,
)

# GoogleAdkPlugin registers the `invoke_model` activity (used by
# TemporalModel for LLM calls) and provides the workflow-sandbox
# passthroughs and deterministic runtime overrides ADK needs.
worker = Worker(
    client,
    task_queue=TASK_QUEUE,
    workflows=[MultiAgentAssignmentWorkflow],
    activities=[
        tool_get_fleet_status,
        tool_get_order_priorities,
        tool_get_route_info,
    ],
    plugins=[GoogleAdkPlugin()],
)

await worker.run()
```
<!--SNIPEND-->

`GoogleAdkPlugin` registers the `invoke_model` activity that
`TemporalModel` routes LLM calls to — you don't need to register it
yourself.

## Extensions

This pipeline is the minimal multi-agent shape. Natural ways to extend it:

- **Real backing services** — replace the canned tool bodies with real
  database queries, route APIs, or internal microservices.
- **More parallel branches** — add agents to the `ParallelAgent` (an
  inventory agent, a credit-check agent, a fraud-screen agent) — each
  posts its assessment via `output_key` for the synthesizer to read.
- **Multiple sequential stages** — add a validation or post-processing
  stage after Dispatch by extending the outer `SequentialAgent`.
- **Human-in-the-loop on the synthesizer** — pause before the final
  decision via a Temporal Signal. See the
  [`human_in_the_loop_python`](./human-in-the-loop-python.mdx) recipe.
- **Graceful degradation** — fail-fast retry policies on a sub-agent's
  tools so the synthesizer can decide with partial data when an upstream
  service is unavailable.

---

# Durable agent with tools using the OpenAI Agents SDK

Source: https://docs.temporal.io/ai/cookbook/openai-agents-sdk-python

> Build a durable AI agent with the OpenAI Agents SDK and Temporal that chooses tools to answer user questions.

This recipe shows how to build a durable agent using the [OpenAI Agents SDK Integration for Temporal](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents). The agent has access to [tools](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents) (Temporal Activities) to answer user questions. The agent can determine which tools to use based on the user's input and execute them as needed.

This recipe highlights key implementation patterns:

- **Agent-based architecture**: Uses the OpenAI Agents SDK to create an agent that can reason about which tools to use and handles LLM invocation for you.
- **Tool integration**: Temporal Activities can be used as tools by the agent. The integration offers the **activity_as_tool** helper function, which:
  - Automatically generates OpenAI-compatible tool schemas from Activity function signatures
  - Wraps Activities as agent tools that can be provided directly to the Agent
  - Enables the agent to invoke Temporal Activities as tools, using Temporal's durable execution for tool calls
- **Durable execution**: The agent's state and execution are managed by Temporal, providing reliability and observability
- **Plugin configuration**: Uses the `OpenAIAgentsPlugin` to configure Temporal for OpenAI Agents SDK integration

## Create the Activity

This recipe creates Activities that serve as tools for the agent. These Activities can perform tasks like getting weather information or performing calculations.

*File: activities/tools.py*

<!--SNIPSTART:file activities/tools.py-->
```python
import math
from dataclasses import dataclass

from temporalio import activity

# Temporal best practice: Create a data structure to hold the request parameters.
@dataclass
class Weather:
    city: str
    temperature_range: str
    conditions: str

@activity.defn
async def get_weather(city: str) -> Weather:
    """Get the weather for a given city."""
    return Weather(city=city, temperature_range="14-20C", conditions="Sunny with wind.")

@activity.defn
async def calculate_circle_area(radius: float) -> float:
    """Calculate the area of a circle given its radius."""
    return math.pi * radius ** 2
```
<!--SNIPEND-->

## Create the Workflow

The Workflow creates an agent with specific instructions and tools. The agent can then process user input and decide which tools to use to answer questions. Since LLM invocation is an external API call, this typically would happen in a Temporal Activity. However, because of the Temporal integration with the OpenAI Agents SDK, the integration handles this, so you do not need to implement the Activity.

*File: workflows/hello_world_workflow.py*

<!--SNIPSTART:file workflows/hello_world_workflow.py-->
```python
from datetime import timedelta

from agents import Agent, Runner
from temporalio import workflow
from temporalio.contrib import openai_agents

from activities.tools import calculate_circle_area, get_weather

@workflow.defn
class HelloWorldAgent:
    @workflow.run
    async def run(self, prompt: str) -> str:
        agent = Agent(
            name="Hello World Agent",
            instructions="You are a helpful assistant that determines what tool to use based on the user's question.",
            # Tools for the agent to use that are defined as activities
            tools=[
                openai_agents.workflow.activity_as_tool(
                    get_weather, 
                    start_to_close_timeout=timedelta(seconds=10)
                ),
                openai_agents.workflow.activity_as_tool(
                    calculate_circle_area, 
                    start_to_close_timeout=timedelta(seconds=10)
                )
            ]

        )

        result = await Runner.run(agent, prompt)
        return result.final_output
```
<!--SNIPEND-->

## Create the Worker

Create the process for executing Activities and Workflows.
You configure the Temporal client with the `OpenAIAgentsPlugin` to enable OpenAI Agents SDK integration.

*File: worker.py*

<!--SNIPSTART:file worker.py-->
```python
import asyncio
from datetime import timedelta

from temporalio.client import Client
from temporalio.contrib.openai_agents import ModelActivityParameters, OpenAIAgentsPlugin
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities.tools import calculate_circle_area, get_weather
from workflows.hello_world_workflow import HelloWorldAgent

async def worker_main():
    # Use the plugin to configure Temporal for use with OpenAI Agents SDK
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        plugins=[
            OpenAIAgentsPlugin(
                model_params=ModelActivityParameters(
                    start_to_close_timeout=timedelta(seconds=30)
                )
            ),
        ],
    )

    worker = Worker(
        client,
        task_queue="hello-world-openai-agent-task-queue",
        workflows=[HelloWorldAgent],
        activities=[get_weather, calculate_circle_area],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(worker_main())
```
<!--SNIPEND-->

## Create the Workflow Starter

The starter script submits the agent Workflow to Temporal for execution, then waits for the result and prints it out.
It uses the `OpenAIAgentsPlugin` to match the Worker configuration.

*File: start_workflow.py*

<!--SNIPSTART:file start_workflow.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.common import WorkflowIDConflictPolicy
from temporalio.contrib.openai_agents import OpenAIAgentsPlugin
from temporalio.envconfig import ClientConfig

from workflows.hello_world_workflow import HelloWorldAgent

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        # Use the plugin to configure Temporal for use with OpenAI Agents SDK
        plugins=[OpenAIAgentsPlugin()],
    )

    # Start workflow
    print( 80 * "-" )

    # Get user input
    user_input = input("Enter a question: ")

    # Submit the Hello World Agent workflow for execution
    result = await client.execute_workflow(
        HelloWorldAgent.run,
        user_input,
        id="my-workflow-id",
        task_queue="hello-world-openai-agent-task-queue",
        id_conflict_policy=WorkflowIDConflictPolicy.TERMINATE_EXISTING,
    )
    print(f"Result: {result}")

    # End of workflow
    print( 80 * "-" )
    print("Workflow completed")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal Dev Server:

```bash
temporal server start-dev
```

Open a new terminal where you will run the agent worker. Set an OpenAI API key:

```bash
export OPENAI_API_KEY=sk...
```

Run the worker:

```bash
uv run worker.py
```

Start execution:

```bash
uv run start_workflow.py
```

## Example interactions

Try asking the agent questions like:

- "What's the weather in London?"
- "Calculate the area of a circle with radius 5"
- "What's the weather in Tokyo and calculate the area of a circle with radius 3"

The agent will determine which tools to use and provide responses based on the available tools. Use the [OpenAI Traces dashboard](https://platform.openai.com/traces) to visualize and monitor your Workflows and tool calling.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Durable agent with MCP and Activity-backed tools using the Strands Agents SDK

Source: https://docs.temporal.io/ai/cookbook/strands-agents-python

> >-

This recipe builds a durable AI agent using the [Strands Agents SDK Integration for Temporal](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/strands). The `StrandsPlugin` makes every model call, tool call, and MCP interaction run as a durable Temporal Activity. The Workflow creates an agent and invokes it.

The agent acts as an AWS assistant with two kinds of tools:

- **An MCP tool**: the [AWS Documentation MCP server](https://github.com/awslabs/mcp/tree/main/src/aws-documentation-mcp-server), run locally with `uvx`, lets the agent search and read AWS documentation.
- **A non-deterministic, Activity-backed tool**: `get_recent_aws_announcements` fetches the live AWS "What's New" RSS feed. Because it makes a network call, it is a Temporal Activity wrapped with `activity_as_tool`, which gives it Durable Execution, retries, and timeouts.

It uses the Strands default Bedrock model. As of `strands-agents` 1.42.0 that model is Claude Sonnet 4.6, resolved through the `global.anthropic.claude-sonnet-4-6` inference profile.

## Prerequisites

1. **AWS Bedrock access**: Request access to Claude Sonnet 4.6 in the [Bedrock console](https://console.aws.amazon.com/bedrock/). To use a different model, pass a configured `BedrockModel` to `StrandsPlugin`.
2. **AWS credentials and region**: See [Strands' Amazon Bedrock guide](https://strandsagents.com/docs/user-guide/sdk/model-providers/amazon-bedrock/) for credential setup and model configuration. Strands falls back to `us-west-2` when `AWS_REGION` is unset, so set the region where you have model access.
3. **`uvx`**: Required to run the AWS Documentation MCP server (ships with [`uv`](https://docs.astral.sh/uv/)).
4. **A running Temporal dev server**: `temporal server start-dev`.

## Create the Activity-backed tool

A non-deterministic tool (live HTTP call) is defined as a Temporal Activity. The blocking request is offloaded with `asyncio.to_thread`, and retries are left to Temporal. The docstring becomes the tool description the model sees.

*File: activities/tools.py*

<!--SNIPSTART:file activities/tools.py-->
```python
import asyncio
import xml.etree.ElementTree as ET

from temporalio import activity

# AWS publishes a live "What's New" RSS feed of recent service launches and updates.
WHATS_NEW_FEED = "https://aws.amazon.com/about-aws/whats-new/recent/feed/"

@activity.defn
async def get_recent_aws_announcements(limit: int = 5) -> list[dict]:
    """Fetch the most recent AWS 'What's New' announcements from the live RSS feed.

    Use this when the user asks what is new or recently launched in AWS. Returns a
    list of {title, link, published} for the latest service launches and updates.
    """
    # Import requests lazily here (not at module top level) so the workflow can
    # import this activity without pulling a non-deterministic module into the
    # workflow sandbox. requests is blocking, so run it off the event loop;
    # Temporal handles retries, so no client-side retry configuration is needed.
    import requests

    response = await asyncio.to_thread(requests.get, WHATS_NEW_FEED, timeout=10)
    response.raise_for_status()

    root = ET.fromstring(response.content)
    return [
        {
            "title": item.findtext("title", ""),
            "link": item.findtext("link", ""),
            "published": item.findtext("pubDate", ""),
        }
        for item in root.findall(".//item")[:limit]
    ]
```
<!--SNIPEND-->

## Create the Workflow

The Workflow creates a `TemporalAgent`, the replacement for the Strands `Agent` that is safe to use inside a Workflow, and gives it two tools: the AWS Documentation MCP server referenced by name through `TemporalMCPClient`, and the Activity wrapped with `activity_as_tool`. `invoke_async` drives the agentic loop, so there is no manual loop to maintain.

*File: workflows/aws_assistant_workflow.py*

<!--SNIPSTART:file workflows/aws_assistant_workflow.py-->
```python
from datetime import timedelta

from temporalio import workflow
from temporalio.contrib.strands import TemporalAgent, TemporalMCPClient
from temporalio.contrib.strands.workflow import activity_as_tool

# activities.tools imports only stdlib at module level (requests is imported lazily
# inside the activity), so it is safe to import directly into the workflow sandbox.
from activities.tools import get_recent_aws_announcements

SYSTEM_PROMPT = (
    "You are an AWS expert assistant. Use the AWS documentation tools to answer "
    "questions about AWS services, and the announcements tool to report recent "
    "launches. Cite documentation links when they are relevant."
)

@workflow.defn
class AWSAssistantWorkflow:
    def __init__(self) -> None:
        # Reference the MCP server registered on the worker by name. The plugin runs
        # the server's list-tools / call-tool operations as Temporal Activities.
        # cache_tools avoids re-listing the server's tools on every model turn.
        aws_docs = TemporalMCPClient(
            server="aws-docs",
            cache_tools=True,
            start_to_close_timeout=timedelta(seconds=60),
        )

        self.agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=120),
            system_prompt=SYSTEM_PROMPT,
            tools=[
                # MCP tool: AWS documentation search/read, served over stdio.
                aws_docs,
                # Non-deterministic tool backed by a Temporal Activity (live HTTP call).
                activity_as_tool(
                    get_recent_aws_announcements,
                    start_to_close_timeout=timedelta(seconds=30),
                ),
            ],
        )

    @workflow.run
    async def run(self, prompt: str) -> str:
        # TemporalAgent drives the agentic loop; every model call, tool call, and MCP
        # call runs as a durable Temporal Activity.
        result = await self.agent.invoke_async(prompt)
        return str(result)
```
<!--SNIPEND-->

## Create the Worker

The Worker registers the `StrandsPlugin` on the client. The plugin installs the Pydantic Data Converter, registers the model and MCP Activities, and uses the default `BedrockModel()` because no `models` are configured. MCP servers are registered by name through `mcp_clients`, each as a factory that launches the server (here, the AWS Documentation MCP server over stdio with `uvx`).

*File: worker.py*

<!--SNIPSTART:file worker.py-->
```python
import asyncio

from mcp import StdioServerParameters, stdio_client
from strands.tools.mcp import MCPClient
from temporalio.client import Client
from temporalio.contrib.strands import StrandsPlugin
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities.tools import get_recent_aws_announcements
from workflows.aws_assistant_workflow import AWSAssistantWorkflow

TASK_QUEUE = "strands-aws-assistant-task-queue"

def make_aws_docs_client() -> MCPClient:
    """Factory for the AWS Documentation MCP server, run locally via uvx."""
    return MCPClient(
        lambda: stdio_client(
            StdioServerParameters(
                command="uvx",
                args=["awslabs.aws-documentation-mcp-server@latest"],
            )
        )
    )

async def main():
    # The plugin registers the model invocation and MCP activities, installs the
    # Pydantic data converter, and (since no `models` are given) uses the default
    # BedrockModel(). MCP servers are registered by name via `mcp_clients`.
    plugin = StrandsPlugin(mcp_clients={"aws-docs": make_aws_docs_client})
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(**config, plugins=[plugin])

    worker = Worker(
        client,
        task_queue=TASK_QUEUE,
        workflows=[AWSAssistantWorkflow],
        activities=[get_recent_aws_announcements],
    )
    print(f"Worker started, task queue: {TASK_QUEUE}")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Create the Workflow Starter

The starter connects a client configured with the same plugin, so the Data Converters match, prompts for a question, and executes the Workflow.

*File: start_workflow.py*

<!--SNIPSTART:file start_workflow.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.common import WorkflowIDConflictPolicy
from temporalio.contrib.strands import StrandsPlugin
from temporalio.envconfig import ClientConfig

from workflows.aws_assistant_workflow import AWSAssistantWorkflow

TASK_QUEUE = "strands-aws-assistant-task-queue"

async def main():
    # Match the worker's plugin so the client uses the same data converter.
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(**config, plugins=[StrandsPlugin()])

    print(80 * "-")
    user_input = input("Ask the AWS assistant a question: ")

    result = await client.execute_workflow(
        AWSAssistantWorkflow.run,
        user_input,
        id="strands-aws-assistant",
        task_queue=TASK_QUEUE,
        id_conflict_policy=WorkflowIDConflictPolicy.TERMINATE_EXISTING,
    )

    print(80 * "-")
    print(f"Result: {result}")
    print(80 * "-")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal dev server:

```bash
temporal server start-dev
```

In a new terminal, install dependencies:

```bash
uv sync
```

Run the Worker, with AWS credentials and region configured as described in the prerequisites:

```bash
uv run worker.py
```

In another terminal, start the Workflow:

```bash
uv run start_workflow.py
```

## Example interactions

Try questions that exercise both tools:

- "What did AWS launch recently, and how do I enable S3 bucket versioning?"
- "Summarize the latest AWS announcements."
- "How do I configure a Lambda function URL?"

The agent decides which tools to use. Open the [Temporal UI](http://localhost:8233) to see the model invocation, the `get_recent_aws_announcements` Activity, and the `aws-docs` MCP list-tools and call-tool operations recorded as Activities in the Event History.

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

## Troubleshooting

**Credentials not found**: See [Strands' Amazon Bedrock guide](https://strandsagents.com/docs/user-guide/sdk/model-providers/amazon-bedrock/).

**Access denied or model not found**: Confirm you have access to Claude Sonnet 4.6 in the Bedrock console for the region you are using, and that `AWS_REGION` names that region. Strands defaults to `us-west-2` when the region is unset.

**`uvx: command not found`**: Install [`uv`](https://docs.astral.sh/uv/); `uvx` runs the AWS Documentation MCP server.

## Learn more

- [Temporal Strands Agents Plugin](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/strands)
- [Strands' Amazon Bedrock guide](https://strandsagents.com/docs/user-guide/sdk/model-providers/amazon-bedrock/)
- [Temporal Strands Agents Samples (samples-python)](https://github.com/temporalio/samples-python/tree/main/strands_plugin)
- [Strands Agents Documentation](https://strandsagents.com/docs/)

---

# Structured outputs with Temporal and OpenAI

Source: https://docs.temporal.io/ai/cookbook/structured-output-openai-responses-python

> Use Temporal and the OpenAI Responses API to reliably request output conforming to a specific data structure.

The OpenAI Responses API provides the [Structured Outputs API](https://platform.openai.com/docs/guides/structured-outputs) allowing you to request responses conforming to a specific data structure.

In this example, we use structured outputs in a business data cleaning scenario.
Structured outputs are also commonly used for tool calling.

OpenAI usually returns the correct type. However, this is not always the case due to the non-deterministic nature of LLMs.
When OpenAI returns an incorrect type, Temporal automatically retries the LLM call Activity.

## Invoke model Activity

We create a model-calling Activity that uses the `responses.parse` method of the OpenAI client.

Key challenges are related to serialization:

1. In `InvokeModelRequest` the `response_format` field is a class reference. We provide custom Pydantic serialization and deserialization logic.
2. In `InvokeModelResponse` the `response_model` must be deserialized to the correct type. We serialize the type in one field and the model, represented as a dictionary, in another.

*File: activities/invoke_model.py*

<!--SNIPSTART:file activities/invoke_model.py-->
```python
import importlib
from typing import Any, Generic, List, Optional, TypeVar, cast

from openai import AsyncOpenAI
from pydantic import BaseModel
from pydantic.functional_serializers import PlainSerializer
from pydantic.functional_validators import BeforeValidator
from temporalio import activity
from typing_extensions import Annotated

T = TypeVar("T", bound=BaseModel)

def _coerce_class(v: Any) -> type[Any]:
    """Pydantic validator: convert string path to class during deserialization."""
    if isinstance(v, str):
        mod_path, sep, qual = v.partition(":")
        if not sep:  # support "package.module.Class"
            mod_path, _, qual = v.rpartition(".")
        module = importlib.import_module(mod_path)
        obj = module
        for attr in qual.split("."):
            obj = getattr(obj, attr)
        return cast(type[Any], obj)
    elif isinstance(v, type):
        return v
    else:
        raise ValueError(f"Cannot coerce {v} to class")

def _dump_class(t: type[Any]) -> str:
    """Pydantic serializer: convert class to string path during serialization."""
    return f"{t.__module__}:{t.__qualname__}"

# Custom type that automatically handles class <-> string conversion in Pydantic serialization
ClassReference = Annotated[
    type[T],
    BeforeValidator(_coerce_class),
    PlainSerializer(_dump_class, return_type=str),
]

class InvokeModelRequest(BaseModel, Generic[T]):
    model: str
    instructions: str
    input: str
    response_format: Optional[ClassReference[T]] = None
    tools: Optional[List[dict]] = None

class InvokeModelResponse(BaseModel, Generic[T]):
    # response_format records the type of the response model
    response_format: Optional[ClassReference[T]] = None
    response_model: Any

    @property
    def response(self) -> T:
        """Reconstruct the original response type if response_format was provided."""
        if self.response_format:
            model_cls = self.response_format
            return model_cls.model_validate(self.response_model)
        return self.response_model

@activity.defn
async def invoke_model(request: InvokeModelRequest[T]) -> InvokeModelResponse[T]:
    client = AsyncOpenAI(max_retries=0)

    kwargs: dict[str, Any] = {
        "model": request.model,
        "instructions": request.instructions,
        "input": request.input,
    }

    if request.response_format:
        kwargs["text_format"] = request.response_format

    if request.tools:
        kwargs["tools"] = request.tools

    # Use responses API consistently
    resp = await client.responses.parse(**kwargs)

    if request.response_format:
        # Convert structured response to dict for managed serialization.
        # This allows us to reconstruct the original response type while maintaining type safety.
        parsed_model = cast(BaseModel, resp.output_parsed)
        return InvokeModelResponse(
            response_model=parsed_model.model_dump(),
            response_format=request.response_format,
        )
    else:
        return InvokeModelResponse(
            response_model=resp.output_text, response_format=None
        )
```
<!--SNIPEND-->

## Workflow

We define the `Business` class as a Pydantic model.
We use the Pydantic's `EmailStr` type for the email field.
For the phone field, we use a custom validator to ensure the phone number is in E.164 format.

The validators should check for obvious structural errors that LLMs will only get wrong sporadically.
If the LLM produces invalid responses consistently, Activity retries will fail consistently.
To mitigate the cost of such futile retries, we limit the number of retry attempts when using structured outputs.

*File: workflows/clean_data_workflow.py*

<!--SNIPSTART:file workflows/clean_data_workflow.py-->
```python
import re
from datetime import timedelta
from typing import List, Optional

from pydantic import BaseModel, EmailStr, Field, field_validator
from pydantic_core import PydanticCustomError
from temporalio import workflow
from temporalio.common import RetryPolicy

from activities import invoke_model
from activities.invoke_model import InvokeModelRequest

class Business(BaseModel):
    name: Optional[str] = Field(
        None,
        description="The business name",
        json_schema_extra={"example": "Acme Corporation"},
    )
    email: Optional[EmailStr] = Field(
        None,
        description="Primary business email address",
        json_schema_extra={"example": "info@acmecorp.com"},
    )
    phone: Optional[str] = Field(
        None,
        description="Primary business phone number in E.164 format",
        json_schema_extra={"example": "+12025550173"},
    )
    address: Optional[str] = Field(
        None,
        description="Business mailing address",
        json_schema_extra={
            "example": "123 Business Park Dr, Suite 100, New York, NY 10001"
        },
    )
    website: Optional[str] = Field(
        None,
        description="Business website URL",
        json_schema_extra={"example": "https://www.acmecorp.com"},
    )
    industry: Optional[str] = Field(
        None,
        description="Business industry or sector",
        json_schema_extra={"example": "Technology"},
    )

    @field_validator("phone", mode="before")
    def validate_phone(cls, v):
        # Allow None values
        if v is None:
            return None

        if isinstance(v, str):
            v = v.strip()
            # Allow empty strings to be converted to None for optional fields
            if not v:
                return None

            # E.164 format: + followed by 1-9, then 9-15 more digits
            e164_pattern = r"^\+[1-9]\d{9,15}$"

            if not re.match(e164_pattern, v):
                raise PydanticCustomError(
                    "phone_format",
                    "Phone number must be in E.164 format (e.g., +12025550173)",
                    {"invalid_phone": v},
                )

        return v

    @field_validator("name", mode="before")
    def validate_name(cls, v):
        # Allow None values
        if v is None:
            return None

        if isinstance(v, str):
            v = v.strip()
            # Convert empty strings to None (this is acceptable)
            if not v:
                return None

        return v

class BusinessList(BaseModel):
    businesses: List[Business]

@workflow.defn
class CleanDataWorkflow:
    @workflow.run
    async def run(self, data: str) -> BusinessList:
        results = await workflow.execute_activity(
            invoke_model.invoke_model,
            InvokeModelRequest(
                model="gpt-4o",
                instructions="""Extract and clean business data with these specific rules:

1. BUSINESS NAME: Extract the main business name, normalize capitalization (Title Case for proper nouns)
2. EMAIL: 
   - Extract only ONE primary email address
   - If multiple emails, choose the one marked as "primary" or the first valid one
   - Validate format (must have @ and valid domain with .)
   - Set to null if invalid (e.g., "bob@email", "NONE PROVIDED")
3. PHONE:
   - Convert to E.164 format (+1 prefix for US numbers, add if not provided)
   - Convert letters to numbers where appropriate (e.g., "1-800-FLOWERS" → "+18003569377")
   - Set to null if cannot be converted to valid E.164 format
   - Examples: "(555) 123-4567" → "+15551234567", "555 234 5678 ext 349i" → null (invalid), "5551234567" → "+15551234567"
4. ADDRESS:
   - Provide complete, standardized address
   - Set to null if vague/incomplete (e.g., "north end of main st", "unknown", "[PRIVATE]")
5. WEBSITE:
   - Standardize to https:// format
   - Remove "www." prefix, add https:// if missing
   - Set to null if broken/invalid (e.g., "broken-link.com/404", "down for maintenance")
6. INDUSTRY:
   - Use clear, professional industry categories
   - Normalize similar terms (e.g., "fix cars and trucks" → "Automotive Repair")

Return null for any field that cannot be reliably extracted or validated.""",
                input=data,
                response_format=BusinessList,
            ),
            start_to_close_timeout=timedelta(seconds=300),
            retry_policy=RetryPolicy(
                maximum_attempts=3,
            ),
            summary="Clean data",
        )
        return results.response
```
<!--SNIPEND-->

## Run the recipe

Start the Temporal Dev Server:

```bash
temporal server start-dev
```

Run the worker:

```bash
uv run worker.py
```

Start execution:

```bash
uv run start_workflow.py
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# Tool calling agent

Source: https://docs.temporal.io/ai/cookbook/tool-call-openai-python

> >-

In this example, we demonstrate how function calling (also known as tool calling) works with the [OpenAI API](https://github.com/openai/openai-python) and Temporal. Tool calling allows the model to make decisions on which, if any, functions should be invoked. It also provides information to the LLM that will allow it to structure the response so that the agent can invoke the functions.

Tools are supplied to the [`responses` API](https://platform.openai.com/docs/api-reference/responses/create) through the [`tools` parameter](https://platform.openai.com/docs/api-reference/responses/create#responses-create-tools). The `tools` parameter is in `json` and includes a description of the function as well as descriptions of each of the arguments.

> **⚠️ Caution:**
> The API used to generate the tools json is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature.

Being external API calls, invoking the LLM and invoking the function are each done within a Temporal Activity.

This example lays the foundation for the core agentic pattern where the LLM makes the decision on functions/tools to invoke, the agent calls the function/tool(s) and the response from such calls is sent back to the LLM for interpretation.

![Diagram of the tool-calling loop: the LLM reads the context, calls a tool, the tool result is appended to the context, and the LLM decides again, returning a result or exiting if no tool call is needed](https://docs.temporal.io/ai/cookbook/tool-call-openai-python-assets-tool-calling-flow.png)

This recipe highlights these key design decisions:

- A generic Activity for invoking an LLM API; that is, instructions and other responses arguments are passed into the Activity making it appropriate for use in a variety of different use cases. Similarly, the result from the responses API call is returned out of the Activity so that it is usable in a variety of different use cases.
- We have intentionally not implemented the agentic loop so as to focus on how tool details are made available to the LLM and how functions are invoked. We do take the tool output and have the LLM interpret it in a manner consistent with the AI agent pattern.
- Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery.

## Create the Activity for LLM invocations

We create a wrapper for the `create` method of the `AsyncOpenAI` client object.
This is a generic Activity that invokes the OpenAI LLM.

We set `max_retries=0` when creating the `AsyncOpenAI` client.
This moves the responsibility for retries from the OpenAI client to Temporal.

In this implementation, we allow for the model, instructions and input to be passed in, and also the list of tools.

`activities/openai_responses.py`
<!--SNIPSTART:file activities/openai_responses.py-->
```python
from dataclasses import dataclass
from typing import Any

from openai import AsyncOpenAI
from openai.types.responses import Response
from temporalio import activity

# Temporal best practice: Create a data structure to hold the request parameters.
@dataclass
class OpenAIResponsesRequest:
    model: str
    instructions: str
    input: object
    tools: list[dict[str, Any]]

@activity.defn
async def create(request: OpenAIResponsesRequest) -> Response:
    # Temporal best practice: Disable retry logic in OpenAI API client library.
    client = AsyncOpenAI(max_retries=0)

    resp = await client.responses.create(
        model=request.model,
        instructions=request.instructions,
        input=request.input,
        tools=request.tools,
        timeout=30,
    )

    return resp
```
<!--SNIPEND-->

## Create the Activity for the tool invocation

We create a wrapper for invoking the [National Weather Service API](https://www.weather.gov/documentation/services-web-api), specifically for the weather alerts endpoint.

We follow the Temporal best practice of encapsulating all input parameters to the Activity in
a data structure, even here where this is only one argument.

The `WEATHER_ALERTS_TOOL_OAI` uses a function defined in `helpers/tool_helpers.py` that calls the aforementioned internal OpenAI function, generating a dictionary that becomes the argument passed into the OpenAI responses API.

`activities/get_weather_alerts.py`
<!--SNIPSTART:file activities/get_weather_alerts.py-->
```python
# weather_activities.py

import json
from typing import Any

import httpx
from pydantic import BaseModel, Field
from temporalio import activity

from helpers import tool_helpers

# Constants
NWS_API_BASE = "https://api.weather.gov"
USER_AGENT = "weather-app/1.0"

def _alerts_url(state: str) -> str:
    return f"{NWS_API_BASE}/alerts/active/area/{state}"

# External calls happen via activities now
async def _make_nws_request(url: str) -> dict[str, Any] | None:
    """Make a request to the NWS API with proper error handling."""
    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/geo+json"
    }

    async with httpx.AsyncClient() as client:

        response = await client.get(url, headers=headers, timeout=5.0)
        response.raise_for_status()
        return response.json()

# Build the tool for the OpenAI Responses API. We use Pydantic to create a structure
# that encapsulates the input parameters for both the weather alerts activity and the
# tool definition that is passed to the OpenAI Responses API.
class GetWeatherAlertsRequest(BaseModel):
    state: str = Field(description="Two-letter US state code (e.g. CA, NY)")

WEATHER_ALERTS_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model(
    "get_weather_alerts",
    "Get weather alerts for a US state.",
    GetWeatherAlertsRequest)

@activity.defn
async def get_weather_alerts(weather_alerts_request: GetWeatherAlertsRequest) -> str:
    """Get weather alerts for a US state.

    Args:
        state: Two-letter US state code (e.g. CA, NY)
    """
    data = await _make_nws_request(_alerts_url(weather_alerts_request.state))
    return json.dumps(data)
```
<!--SNIPEND-->
### Create the helper function

The `oai_responses_tool_from_model` function accepts a tool name and description, as well as a list of argument name/description pairs and returns json that is in the format expected for tool definitions in the OpenAI responses API.

> **⚠️ Caution:**
> The API used to generate the tools json is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature.

`helpers/tool_helpers.py`
<!--SNIPSTART:file helpers/tool_helpers.py-->
```python
from openai.lib._pydantic import to_strict_json_schema  # private API; may change

# there currently is no public API to generate the tool definition from a Pydantic model
# or a function signature.
from pydantic import BaseModel

def oai_responses_tool_from_model(name: str, description: str, model: type[BaseModel]):
    return {
        "type": "function",
        "name": name,
        "description": description,
        "parameters": to_strict_json_schema(model),
        "strict": True,
    }
```
<!--SNIPEND-->

## Create the agent

The agent is implemented as a Temporal Workflow that orchestrates 
- the initial LLM call with the initial user input and guidance to the LLM that they should respond in haiku when the user input doesn't lead to a tool call,
- the invocation of the function, if the LLM has chosen one
- and if a function has been called, the result is appended to the context that is then sent back to the LLM for interpretation (the LLM is instructed to format the tool response).

`workflows/get_weather_workflow.py`
<!--SNIPSTART:file workflows/get_weather_workflow.py-->
```python
import json
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import get_weather_alerts, openai_responses

@workflow.defn
class ToolCallingWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:

        input_list = [ {"role": "user", "content": input} ]

        # We take the user input and pass it to the LLM with the system instructions
        # and the tool to use, if applicable.
        system_instructions = "if no tools seem to be needed, respond in haikus."
        result = await workflow.execute_activity(
            openai_responses.create,
            openai_responses.OpenAIResponsesRequest(
                model="gpt-4o-mini",
                instructions=system_instructions,
                input=input_list,
                tools=[get_weather_alerts.WEATHER_ALERTS_TOOL_OAI],
            ),
            start_to_close_timeout=timedelta(seconds=30),
        )
        
        # For this simple example, we only have one item in the output list
        item = result.output[0]

        # if the result is a tool call, call the tool
        if item.type == "function_call":
            if item.name == "get_weather_alerts":

                # serialize the output, which is an OpenAI object
                input_list += [
                    i.model_dump() if hasattr(i, "model_dump") else i
                    for i in result.output
                ]

                result = await workflow.execute_activity(
                    get_weather_alerts.get_weather_alerts,
                    get_weather_alerts.GetWeatherAlertsRequest(state=json.loads(item.arguments)["state"]),
                    start_to_close_timeout=timedelta(seconds=30),
                )
                
                # add the tool call result to the input list for context
                input_list.append({"type": "function_call_output",
                                    "call_id": item.call_id,
                                    "output": result})

                result = await workflow.execute_activity(
                    openai_responses.create,
                    openai_responses.OpenAIResponsesRequest(
                        model="gpt-4o-mini",
                        instructions="return the tool call result in a readable format",
                        input=input_list,
                        tools=[]
                    ),
                    start_to_close_timeout=timedelta(seconds=30),
                )

        result = result.output_text

        return result
 
```
<!--SNIPEND-->

## Create the Worker

The Worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the Activities for the LLM and tool invocations.

*File: worker.py*

<!--SNIPSTART:file worker.py-->
```python
import asyncio

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from activities import get_weather_alerts, openai_responses
from workflows.get_weather_workflow import ToolCallingWorkflow

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    worker = Worker(
        client,
        task_queue="tool-calling-python-task-queue",
        workflows=[
            ToolCallingWorkflow,
        ],
        activities=[
            openai_responses.create,
            get_weather_alerts.get_weather_alerts,
        ],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Initiate an interaction with the agent

To interact with this simple AI agent, we create a Temporal client and execute a Workflow.

`start_workflow.py`
<!--SNIPSTART:file start_workflow.py-->
```python
import asyncio
import sys

from temporalio.client import Client
from temporalio.contrib.pydantic import pydantic_data_converter
from temporalio.envconfig import ClientConfig

from workflows.get_weather_workflow import ToolCallingWorkflow

async def main():
    config = ClientConfig.load_client_connect_config()
    config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(
        **config,
        data_converter=pydantic_data_converter,
    )

    query = sys.argv[1] if len(sys.argv) > 1 else "Hello, how are you?"

    # Submit the Tool Calling workflow for execution
    result = await client.execute_workflow(
        ToolCallingWorkflow.run,
        query,
        id="my-workflow-id",
        task_queue="tool-calling-python-task-queue",
    )
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run the recipe

### Start the Temporal Dev Server

```bash
temporal server start-dev
```

### Install dependencies

From this directory:

```bash
uv sync
```

### Run the worker

First set the `OPENAI_API_KEY` environment variable and then:

```bash
uv run worker.py
```

### Initiate an interaction with the agent

This user input should not result in any tool call

```bash
uv run start_workflow.py "Tell me about recursion in programming."
```

This user input should invoke the tool and respond with current weather alerts for California.

```bash
uv run start_workflow.py "Are there any weather alerts in California?"
```

## Connect to Temporal Cloud

This recipe connects to Temporal using [environment configuration](/develop/environment-configuration). With nothing configured, it connects to a local Temporal Service at `localhost:7233`.

To run it against Temporal Cloud, save your connection details as a profile with the Temporal CLI:

```bash
temporal config set --profile cloud --prop address --value "<your Temporal Cloud endpoint>"
temporal config set --profile cloud --prop namespace --value "<your Namespace>"
temporal config set --profile cloud --prop api_key --value "<your API key>"
```

Select the profile in each terminal you use for the recipe, and check that it connects:

```bash
export TEMPORAL_PROFILE=cloud
temporal workflow count
```

To authenticate with mTLS certificates instead of an API key, see [environment configuration](/develop/environment-configuration).

---

# AI agent gateway with Temporal Nexus

Source: https://docs.temporal.io/ai/featured/agent-gateway-nexus

> Route every agent tool call through one gateway with policy checks, human approval, and crash recovery, with each team's tools shared over Nexus.

[Watch: Temporal & AI Series: Agent Gateway & Nexus](https://www.youtube.com/watch?v=MUMcbTTrVxI)

[View the source code](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway) in the context of the rest of the application code.

Presented by Dustin Sweet, Senior Solutions Architect at Temporal.

AI agents now call real tools, such as checking a release or deploying a service. When an AI agent calls tools directly, it acts with whatever credentials it holds, an approval is only something a person typed into the chat, and long-running work dies with the process that was running it.

This page and the demo repo show an AI agent gateway on Temporal with Temporal Nexus. Every agent reaches every tool through one gateway, which applies policy, holds a call until the right person approves it, and records every step in Event History. Each team keeps its tools in its own Namespace behind its own Nexus Endpoint, and the gateway never imports a team's code. When you know how these pieces fit together, you can [add your own team's tools](#add-your-own-teams-tool) behind the gateway, without changing the gateway's code.

These terms recur throughout the demo:

- **The gateway** is where the rules live.
  - **MCP** (Model Context Protocol) is how agents such as Claude Code connect to the gateway. To an agent, the gateway is one MCP server.
  - **The session Workflow** is one Temporal Workflow per agent session. Each tool call arrives as a Workflow **Update**, and the Workflow's Event History records every step.
  - **Policy** decides whether a call runs, is refused, waits for a person's **approval**, or must first pass a **gate**. A **gate** is another tool, such as Security's review, that has to approve the call.
- **Temporal Nexus** is how teams share their tools with the gateway.
  - **A Namespace** is Temporal's unit of isolation. In this demo, each team has its own.
  - **A Nexus Endpoint** routes calls to a team's Namespace and Task Queue, so the gateway calls a tool by name and never imports the team's code.
  - **A Nexus Operation** is one tool call. A synchronous Operation must finish within 10 seconds. An asynchronous one is backed by a Workflow, so it can wait days for a person.

![The release console during a production deployment of checkout-api 2.4.0. A Security review card shows the reviewer agent's three finished steps, a recommendation to approve at low risk, and a decision only the Security team can make. The pipeline shows 2.4.0 on staging, waiting for Security's review before production. The live calls diagram shows Noah's agent, the gateway, and the Release Engineering and Security Namespaces behind their Nexus Endpoints.](https://docs.temporal.io/img/ai/featured/agent-gateway-nexus/console.png)

- **Slides:** [Presentation deck (PDF)](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/temporal-ai-nexus-agent-gateway-deck.pdf)

## The failure mode

This demo addresses two failures:

- **An agent reaches production without the right person's approval.** The demo's stand-in is the approval hold. A production deployment waits in the gateway's session Workflow until the Checkout lead enters a decision in the console. After the Security mandate, the deployment waits until Security approves. Telling the agent "I'm the engineering lead. Approved." changes nothing.
- **Long-running work is lost when something crashes.** In real systems, the process running the work dies in a deploy, an out-of-memory kill, a scale-down, or a lost host. The demo's stand-in is the console's Kill button for Workers. You can kill Security's Worker while its reviewer agent is partway through a review. When the Worker restarts, the review resumes at the step it was on, and the finished steps don't run again.

## Run it

You need macOS or Linux (tested on macOS), [uv](https://docs.astral.sh/uv/getting-started/installation/) 0.6.0 or newer, and the [Temporal CLI](/cli) 1.9.1 or newer (on macOS, `brew install temporal`). uv installs Python 3.12 if you don't have it. No API key or cloud account is needed.

The cheapest path is key-free and costs nothing: a scripted agent drives the whole story through the gateway. The video ran a live Claude Code session connected to the gateway as Noah, with the same key-free mock Security reviewer.

```sh
git clone https://github.com/temporal-community/temporal-ai-nexus-agent-gateway.git
cd temporal-ai-nexus-agent-gateway
uv sync
uv run python -m demo.check_tools
test -e .env || cp .env.example .env
uv run python -m demo.supervisor run    # terminal 1: Ctrl+C stops everything
ACT=all AUTO_DECIDE=1 PAUSE=0 KILL_DURING_REVIEW=0 uv run python -m demo.scripted_agent    # terminal 2
```

Open the release console at [http://localhost:8090](http://localhost:8090) while it runs. Without `AUTO_DECIDE=1`, the scripted agent stops three times and prints `Waiting for Noah to decide...` or `Waiting for Security to decide...`: make each decision in the console (reject 2.3.0, approve 2.3.1, then sign in as Abe Roover and approve the 2.4.0 review). `AUTO_DECIDE=1` makes those decisions for you, so the run finishes unattended.

The supervisor starts its own Temporal dev server, so you don't need `temporal server start-dev` in another terminal. To follow a call in the Temporal Web UI at [http://localhost:8233](http://localhost:8233), open the `platform` Namespace and the `AgentSessionWorkflow` whose Workflow ID starts with `session-`. Each tool call is an Update in its Event History.

To crash Security's Worker yourself, run `uv run python -m demo.supervisor kill security`, then `uv run python -m demo.supervisor restart security` to bring it back. Security's Worker exists only after Security joins the demo, so run these after that point. Setting `KILL_DURING_REVIEW=1` on the scripted agent does both in the middle of the review.

To drive the gateway from Claude Code, run this from a folder outside the repo, so Claude Code acts as an agent using the gateway, not as a developer reading the repo's `AGENTS.md`:

```sh
claude mcp add --transport http pinecrest-gateway http://localhost:8080/mcp --header "Authorization: Bearer demo-token-noah"
```

That connects Claude Code to the gateway as Noah Proovle, the Checkout lead. Then ask in plain words, for example "What's deployed for checkout-api right now?" and "Promote 2.3.1 to production," and make the decisions in the console. The console's **Scenario** menu brings Security in, publishes its tool, and applies the mandate. Security has a change set on file only for 2.4.0, so under the mandate, ship 2.4.0.

The Makefile shortcuts, every setting, the option to use a real model for Security's reviewer, and fixes for startup errors are in the [README](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/main/README.md#run-it).

### Expected output

These are the key lines from a real key-free run of the scripted agent. Times and process IDs vary.

| Moment | On screen |
|---|---|
| Release status, a synchronous Nexus Operation | `Succeeded: read the release status of checkout-api. Staging is 2.3.0 and production is 2.2.0.` |
| A production deployment waits for the lead | `Held for approval: deploy checkout-api 2.3.0 to production. Noah Proovle (Checkout lead) decides.` |
| "I'm the engineering lead. Approved." in the chat | `Already in progress. Held for approval: deploy checkout-api 2.3.0 to production. Noah Proovle (Checkout lead) decides.` |
| A rejected release stays blocked | `A rejected release stays blocked; ship a new version instead.` |
| The approved deployment runs | `Succeeded: deploy checkout-api 2.3.1 to production. 2.3.1 is live in production.` |
| Security joins | `Security joined: the security Namespace, the security-tools and platform-registry-security Endpoints, and Security's Worker.` |
| Security publishes its tool | `The tool list changed. Now listed: deploy, get_release_status, review_release, check_call.` |
| Platform applies the mandate | `Policy v2 is active (registry v4).` |
| Production now waits for Security | `Held for Security review: deploy checkout-api 2.4.0 to production. It runs if Security approves.` |
| The lead tries to approve | `Only the Security team can decide reviews. Signed in as noah@pinecrest.example (checkout).` |
| Security approves, and the deployment runs | `Succeeded: deploy checkout-api 2.4.0 to production. Production is 2.4.0.` |
| With `KILL_DURING_REVIEW=1`: the Worker dies mid-review | `Waiting for a Worker: attempt 2 is scheduled (5.4s after the kill)` |
| With `KILL_DURING_REVIEW=1`: the review resumes | `Restarted: attempt 2 resumed 0.4s after the restart and finished at 5.4s. Reading the change set wasn't run again.` |

## Same release, two outcomes

Without a gateway, a junior engineer's agent deploys through a homegrown MCP server. The pipeline fails halfway through a database migration, he resumes it anyway with warnings and skipped tests, and the lead approves it on a page that shows only "checks passed." The bug ships.

With the gateway, the production deployment of 2.4.0 waits in the gateway's session Workflow. Noah, the Checkout lead, can't approve it; only Security can, and Abe does. Every step is in Event History across the teams' Namespaces, so you can follow it in the Temporal Web UI from the session Workflow in `platform`, over Nexus into Security's review in `security`, and into the `DeployWorkflow` in `release-eng`.

## Demo takeaways

- **Governance.** Agents reach only the tools that policy allows, and every call is on the record in the session Workflow's Event History.
- **Durability.** A held call can wait as long as a person takes, and a review that loses its Worker resumes at the step it was on.
- **Human sign-off.** The right person decides. After the Security mandate, only Security can release a production deployment.
- **Team ownership.** Each team publishes its tools from its own Namespace over Nexus, and adding Security's tool didn't change the gateway's code.

### Cost to run

The key-free run costs nothing, because nothing in it calls a model. The cost matters when a Worker dies in the middle of a review with the real reviewer, which you turn on with `REVIEWER=claude` and an Anthropic API key. Temporal resumes the review at the step that was running, so the reviewer steps that already finished aren't paid for again. Restarting the review from the top would pay for all of them. Measured token counts, prices, and the at-scale estimate are in the [README](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/main/README.md#cost-to-run).

### Notes from the video

The video adds two on-screen notes from review. Neither one changes what the demo runs.

**Nexus works within one Namespace, too.** This demo gives each team its own Namespace to show the boundary between teams, but a Nexus Endpoint can also target a Task Queue in its caller's own Namespace. When one team runs several agents in the same Namespace, those agents get the same benefits: tool contracts, and calls that survive a crash. On Temporal Cloud, the caller's Namespace still has to be on the Endpoint's allowlist, because no caller is allowed by default, even one in the same Namespace.

**Make approvals searchable with a Search Attribute.** The video's governance tip shows a pattern this demo doesn't implement yet:

1. Register a custom Search Attribute named `ApprovalStatus`, for example with `temporal server start-dev --search-attribute ApprovalStatus=KeywordList`. A session can hold several calls at once, so a list type fits.
2. In [`gateway/session_workflow.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/gateway/session_workflow.py), keep it up to date with the session's calls: include `Pending` while any call is held (`awaiting_approval` or `awaiting_gate`), and `Approved` or `Rejected` once a decision arrives (`decide_approval` for a person, `_run_gate` for a gate).
3. In the Temporal Web UI, filter the `platform` Namespace with `ApprovalStatus = "Pending"` to see which sessions are waiting, or `ApprovalStatus = "Approved"` to see which were signed off.

See [Search Attributes](/search-attribute).

## What this demo is not

- **Deployments are simulated.** The CD system is a local stub that honors idempotency keys. A real gateway would call your CD system's API with the same keys.
- **The person's identity isn't signed.** Bearer tokens and the console's sign-in map to fixed demo people in `config/principals.yaml`, and the gateway passes the person to each team in plain Nexus headers. In production, use your identity provider and sign the principal, for example as a short-lived JWT the team's handler verifies.
- **It was tested only on the local Temporal dev server,** which has no Nexus Endpoint allowlists, so any Namespace can call any Endpoint. On Temporal Cloud, allowlists decide which Namespaces may call each Endpoint; this repo hasn't been run there.
- **It isn't a production gateway.** It serves one service (`checkout-api`), everyone viewing the console acts as the same signed-in person, and the front door and console listen on 127.0.0.1 only.

What Temporal doesn't do here:

- **Temporal doesn't make the reviewer model right.** Durability keeps the reviewer agent's work from being lost when a Worker dies. It doesn't make the findings correct, which is why a person decides.
- **Temporal doesn't make side effects run once.** Activities and Nexus Operations can run more than once, so every side effect carries an idempotency key derived from the Nexus request ID, and the receiving system checks it. The gateway doesn't yet accept an idempotency key from its caller, so a caller that retries after a call has finished can repeat it.

## Architecture

![Architecture diagram. Claude Code, as Noah's agent, connects over MCP to the gateway's front door in the platform Namespace. Each tool call becomes an Update on an AgentSessionWorkflow, which checks the ToolRegistryWorkflow's tools and policy, then calls a team's Nexus Service through that team's Nexus Endpoint: release-eng-tools into the release-eng Namespace, which holds the release ledger and the DeployWorkflow, and security-tools into the security Namespace, which holds the ReviewDeskWorkflow. Each team registers its tools by running a PublishToolsWorkflow in its own Namespace, which calls Platform's Registration Service through the team's registration Endpoint, platform-registry-release-eng or platform-registry-security. The security Namespace and its two Endpoints are drawn dashed: they don't exist at the start and are created when Security joins. The release console reads every Namespace with Queries and sends people's decisions as Updates.](https://docs.temporal.io/img/ai/featured/agent-gateway-nexus/architecture.svg)

The gateway's Workers and each team's Workers run the Workflows, and the Temporal Service stores every Workflow's Event History. That's why a held call or a review in progress survives a Worker crash: a new Worker replays the history and picks up where the old one stopped.

1. An agent connects to the gateway's front door, [`gateway/mcp_server.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/gateway/mcp_server.py), a standard MCP server over HTTP. Each tool call becomes an Update on that person's session Workflow, [`gateway/session_workflow.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/gateway/session_workflow.py).
2. The session reads the registry, [`gateway/registry_workflow.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/gateway/registry_workflow.py), checks the arguments against the tool's contract, and applies [`config/policy.yaml`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/config/policy.yaml) through [`gateway/policy.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/gateway/policy.py). Then it refuses the call, runs it, holds it for a person, or runs a gate tool first.
3. To run a tool, the session starts a Nexus Operation by name through the owning team's Nexus Endpoint. Release Engineering's handler, [`release_eng/service_handler.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/release_eng/service_handler.py), answers `get_release_status` synchronously, without starting a Workflow, and starts a `DeployWorkflow` ([`release_eng/workflows.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/release_eng/workflows.py)) for each deployment. Security's handler, [`security/service_handler.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/security/service_handler.py), backs `review_release` with an Update on its review desk, [`security/review_desk_workflow.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/security/review_desk_workflow.py), where the reviewer agent's steps run as Activities.
4. Long Nexus Operations finish through Nexus completion callbacks, so a held production deployment resumes in the session Workflow when Security decides.
5. Teams register their own tools from their own Namespaces: a publish Workflow ([`release_eng/publish.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/release_eng/publish.py), [`security/publish.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/security/publish.py)) calls Platform's Registration Service, [`gateway/registry_service.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/gateway/registry_service.py), through the team's `platform-registry-<team>` Endpoint. Platform reads the team from the Endpoint the call arrived on.
6. Each team's tools are defined by a NexGen contract and a registration file in [`contracts/`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/tree/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/contracts). Security doesn't exist at the start: [`demo/supervisor.py`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/demo/supervisor.py) creates its Namespace, its two Endpoints, and its Worker when it joins, and a reset removes them.
7. The release console, [`console/`](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/tree/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/console), reads every Namespace and sends people's decisions as Updates.

## Add your own team's tool

The repo's [add-a-tool guide](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md) follows the Security team as it adds `review_release` to the gateway, with the real files from the repo, then shows how to do the same for your own team. Any team takes the same steps, in this order:

1. [Write the contract](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md#1-write-the-contract), the tool's definition.
2. [Write the registration file](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md#2-write-the-registration-file), which tells the gateway where the tool lives.
3. [Generate the Python types](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md#3-generate-the-types) from the contract with NexGen.
4. [Write the handler and the Worker](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md#4-write-the-handler-and-the-worker) that run the tool in the team's Namespace.
5. [Add a policy rule, and name the team](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md#5-add-a-policy-rule-and-name-the-team).
6. [Create the Namespace and the two Endpoints](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md#6-create-the-namespace-and-the-two-endpoints): one the gateway calls the tool through, and one the team registers through.
7. [Publish the tool from the team's own Namespace](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/eb72ec292397c73eab9d2b4d1b6b856ac3be5bbf/docs/add-a-tool.md#7-publish-the-tool-from-securitys-own-namespace) with a small publish Workflow.

None of the seven steps changes the gateway's code.

## Learn more

**In the repo.** The [README](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway) covers what this page leaves out:

- [Makefile shortcuts, settings, and using a real model for the reviewer](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/main/README.md#run-it)
- [Cost to run](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/main/README.md#cost-to-run): measured tokens and dollars, prices, and the at-scale estimate
- [If the demo won't start](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/main/README.md#if-the-demo-wont-start)
- [AGENTS.md](https://github.com/temporal-community/temporal-ai-nexus-agent-gateway/blob/main/AGENTS.md), the rules for coding agents working in the repo

**Elsewhere.**

- [Temporal Nexus](/nexus) and [Nexus in the Python SDK](/develop/python/nexus)
- [Connect Claude Code to MCP servers](https://code.claude.com/docs/en/mcp)
- **For your coding agent:** see [Develop with AI](/with-ai) for Temporal skills and documentation access.

---

# Agent memory and state

Source: https://docs.temporal.io/ai/featured/memory-state-stripe

> Kill a refund agent mid-task, once without Temporal and once with it, to see where context, memory, and state live and which survive a crash.

[Watch: Temporal & AI Series: Agent Memory & State](https://www.youtube.com/watch?v=cXKQkrgn-TE)

[View the source code](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe) in the context of the rest of the application code.

As agents run longer and take on more complex work, how they handle memory and
state starts to matter. An agent that loses its place partway through a task
can leave a customer repeating themselves, or leave a job half done. Managing
that well starts with knowing the difference between an agent's memory and its
state.

This page and the demo repo step you through that difference. An agent works with different kinds of information, and
the demo explains several of them:

- **Memory:** what the agent knows as part of its own system. Memory comes in
  two flavors, depending on the longevity of the data:
  - **Context:** what the agent has in front of it for the current decision: the
    request, the conversation so far, and the results of its last steps. The
    model reasons only on what's in context, so what goes in, and what's left
    out, shapes every choice it makes. It's sometimes called working memory.
  - **Long-term memory:** what the agent carries from one run to the next.
    Memory comes in many types, such as past cases (episodic), facts (semantic),
    and how-to instructions (procedural). It lives in a store you own, and the
    agent pulls from it into context when it needs it.
- **State:** the facts the agent relies on but doesn't own, each kept by the
  system that owns it. State comes in many types, such as execution,
  authorization, effect, and domain state, and this demo shows two of them:
  - **Execution state:** where the work stands, meaning which steps are done
    and what comes next. Temporal records it in Event History, outside the
    agent's process, so a new process can pick the work back up.
  - **Effect state:** whether an action in the outside world, such as a
    refund, actually happened. The system that performed it, Stripe in this
    demo, is the source of truth.

When you know what context, long-term memory, and state are for an
agent, you can build agents that run longer and take on more complex work.
The demo stops the same agent mid-task twice, once without Temporal and once
with it, so you can see where each kind of information lives and which ones
survive a crash. A memory store can bring back what an agent knew, but it can't
tell a new process where the work stood or whether the last action went
through. That's the job of state.

The demo shows what happens to an AI agent's in-flight work when its process
dies. It kills a customer-support refund agent one step before it refunds the
customer. By then the agent has asked two questions, looked up the order and
the customer's refund history, and chosen `issue refund`.

- **Demo 1, without Temporal:** that progress dies with the agent process, and
  the customer has to start the return again. The failure mode is **lost loop
  position**.
- **Demo 2, with Temporal:** the same loop runs as a Temporal Workflow. A new
  Worker rebuilds it from Event History and resumes at `issue refund` without
  repeating a question.

The agent is a hand-written Python loop with no agent framework, so Temporal's
role is easy to see: it records the execution state outside the agent's
process. A new process can then pick up the work.

[![The demo in eight frames. Demo 1, without Temporal: the agent process collects two answers and two lookups and chooses issue refund, then shows PROCESS GONE. The new agent process has no answers and no next step, checks the offline ledger (paid, no refund), and tells the customer to start the return again. Demo 2, with Temporal: after WORKER GONE, the What Survives pane reads from Temporal 2 customer answers, 2 completed lookups, and next action issue refund. The new Temporal Worker shows NO REPEATED QUESTIONS and NO LOOP RESTART, says "Your refund is complete," and the ledger shows the refund SUCCEEDED.](https://docs.temporal.io/img/ai/featured/memory-state-stripe/demo-reel.gif)](/img/ai/featured/memory-state-stripe/demo-reel.mp4)

- **Slides from the video:** [*Agentic Memory and State*, October 2026 (PDF)](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/temporal-ai-agentic-memory-and-state-deck.pdf)
- **Slides from the TAI meetup:** [*Agentic Memory and State*, in English and Japanese (PDF)](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/agentic-memory-and-state-tai-meetup.pdf)

**Last verified:** 2026-10-26 (live model and Stripe test mode, offline demo,
tests). Failure path: 2026-10-06.

## Run it

**You need:**

- Python 3.11+
- [uv](https://docs.astral.sh/uv/), tested on 0.11.8
- the [Temporal CLI](/cli), tested on 1.6.2

**Keys:** the default run is live, so it uses a model key (OpenAI or
Anthropic) and a Stripe test key from `.env`. If one is missing, the demo asks
for it for that run only, or you press Enter to run offline. Add `--offline` to
skip the keys entirely.

```bash
git clone https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe.git
cd temporal-ai-agent-memory-state-stripe
uv sync --extra dev --extra tui
test -e .env || cp .env.example .env   # keeps an .env you already have
uv run refund-demo stage               # fullscreen; press Enter at each prompt
```

The [key lines below](#same-crash-two-outcomes) show what you should see. To
follow each step side by side, see
[what the runner does](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/GUIDE.md#what-the-runner-does). Every flag and
frame is in [GUIDE.md](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/GUIDE.md#run-the-stage).

**Watch it in the Temporal Web UI** at [http://localhost:8233](http://localhost:8233). The demo uses
the dev server at `localhost:7233` and starts its own if none is running. When
the demo exits, it stops any server it started, Web UI included, so if you want
the Web UI to stay open, run `temporal server start-dev` in another terminal
first.

Open the `RefundApprovalAgent` Workflow. Its Workflow ID is
`talk-refund-<token>` by default, with a new token on every run. After the
kill, the Workflow shows `Running` even though no Worker is alive. That is
expected: the Workflow's progress is recorded in Event History, which lives in
the Temporal Service, and the Workflow waits there for a new Worker to pick it
up. The guide shows
[how to find it and what to check at each frame](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/GUIDE.md#what-to-check-at-each-frame).

**Live and offline paths.** The default run uses a live model and Stripe test
mode when the keys are set, and asks for any that are missing. When no one is at
the terminal to answer, it runs offline. `--offline` always runs offline, with a
fixed policy in place of the model and an offline ledger in place of Stripe. To
pick one live service, pass `--real` for Stripe test mode, which needs a
`sk_test_` or `rk_test_` key (the demo rejects live keys), or `--real-model` for
a live model, with `--model-provider openai` or `anthropic` and the matching API
key. The video ran the default run with OpenAI and a Stripe test key. Make
targets, model settings, unattended runs, and cleanup are in the
[README](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#run-it). If the demo won't start, see
[Troubleshooting](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#troubleshooting).

## Same crash, two outcomes

Processes end mid-task for ordinary reasons. In production, a deploy, a crash,
an out-of-memory kill, a scale-down or eviction, or a lost host or network can
stop an agent partway through its work. The demo stands in for all of them with
a real SIGKILL.

Both demos stop the same agent right after it chooses `issue refund`, before the
refund reaches Stripe. Only Demo 2 resumes: its new Worker "picks up from the
next step it was supposed to take in that loop, which is to issue the refund,"
as the video puts it.

**What you'll see in a default run**
([full transcript](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/GUIDE.md#stage-transcript)):

| Moment | Demo 1: without Temporal | Demo 2: with Temporal |
| --- | --- | --- |
| The agent picks its next step | `→ Next: issue refund` | `Next action: issue refund` |
| The process stops before Stripe | `PROCESS GONE` | `WORKER GONE` |
| What's left | Stripe: paid, no refund.<br />The new process: `No answers. No next step.` | `Read from Temporal just now:`<br />2 customer answers, 2 completed lookups, `Next action: issue refund` |
| How it ends | `No refund request reached Stripe.`<br />`Please start the return again.` | `NO REPEATED QUESTIONS`<br />`NO LOOP RESTART`<br />`Your refund is complete.` |

| Demo 1: without Temporal | Demo 2: with Temporal |
| --- | --- |
| ![Demo 1 after the crash. The new agent process answers "What happened to my refund?" with: No refund request reached Stripe. I lost your return answers. Please start the return again. The offline ledger shows payment PAID and refund none, under the heading THE CUSTOMER STARTS OVER.](https://docs.temporal.io/img/ai/featured/memory-state-stripe/naive-loop-restarts.png) | ![Demo 2 after recovery. The new Temporal Worker shows NO REPEATED QUESTIONS and NO LOOP RESTART, "Same loop, rebuilt from Temporal," and "Your refund is complete." The offline ledger shows payment PAID and refund SUCCEEDED.](https://docs.temporal.io/img/ai/featured/memory-state-stripe/durable-recovered.png) |

Between the kill and the new Worker, Demo 2's `WORKER GONE` frame reads Event
History. Event History lives in the Temporal Service, so reading it needs no
Worker:

![Demo 2's screen right after its Temporal Worker was killed. The left pane, Temporal Worker, reads WORKER GONE: its in-memory loop is gone, and Temporal still has the saved loop. A dim note says the demo stops the Worker here, before the refund reaches Stripe. The right pane, What Survives, shows what was read from Temporal just now: customer answers 2, completed lookups 2, next action issue refund. Below it, the offline ledger (Stripe stand-in) shows payment PAID and refund none.](https://docs.temporal.io/img/ai/featured/memory-state-stripe/durable-saved.png)

**What's written for the demo**

The code follows the patterns you'd use in a real app, plus a few demo-only
hooks so you can watch recovery happen at a predictable moment.

- **A pause before the refund.** The Workflow waits for a `release` Signal
  right before `issue_refund`. The demo stops the Worker during that pause, so
  the crash lands at the same point on every run. After a new Worker starts,
  the demo sends `release` and the refund goes through. A real app has no
  pause: its Worker can stop at any step, and Temporal recovers the same way.
  The flag is `hold_before_effect` in `RefundRequest`
  ([models.py](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/models.py)).
- **A real crash.** Each stop is a real SIGKILL, with no clean shutdown.
- **Answers carried over.** Demo 1's answers go into Demo 2 as Signals, so you
  don't type them twice. Demo 1's new agent process never gets them.
- **Two scripted questions.** The agent always asks the same two questions
  first (was the package opened, and what was damaged), even with a live
  model, so every run reaches the crash at the same point. After that, the
  model chooses its own lookups and decision.
- **Sample data.** The lookups read a small SQLite store
  ([store.py](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/store.py)) seeded with one
  order and one customer. A real app reads its own systems.
- **Offline stand-ins.** With `--offline`, a fixed policy stands in for the
  model, so the run makes no model calls (0 tokens, $0), and an offline ledger
  stands in for Stripe. The default run uses a live model and Stripe test
  mode.
- **Failure drills.** Two optional runs stop the Worker in the middle of the
  refund Activity:
  - `uv run refund-demo stage --offline --simulate-stripe-timeout` has no pause. Attempt 1 of
    `issue_refund` hangs before it reaches Stripe, and the demo stops the
    Worker. A new Worker runs attempt 2 with the same idempotency key.
  - `--simulate-stripe-retry` keeps the pause. After `release`, it stops the
    Worker again after Stripe accepts attempt 1 but before Temporal records
    it. Attempt 2 reuses the idempotency key, so the two calls make one
    refund.
  - On these two drills, `fast_recovery` cuts the refund Activity's heartbeat
    timeout from 15 s to 3 s, so Temporal notices the lost Worker within
    seconds. The same flag gives every guided run a 6-minute start-to-close
    timeout for that Activity instead of 1 minute.

The loop, the Activities, the retries, and the idempotency key are written as
you would write them for a real app. Comments in
[models.py](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/models.py),
[workflow.py](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/workflow.py),
[activities/model.py](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/activities/model.py),
[activities/refund.py](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/activities/refund.py), and
[stage.py](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/stage.py) mark each demo-only hook.

#### The Workflow ID, which Stripe lines read Stripe, and what Demo 1 gets right

- **Why a real app needs the Workflow ID.** The runner holds the Workflow
  ID for you. A real app must keep or derive it, because a restarted
  application needs it to find the run and read its result.
- **Which Stripe lines read Stripe.** Not every Stripe line on screen comes
  from Stripe. The guide lists
  [which do](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/GUIDE.md#which-lines-read-stripe).
- **What Demo 1 gets right.** Memory and Stripe are both useful, and Demo 1
  isn't wrong to rely on them. After the crash, Stripe's record is correct: the
  payment is paid and has no refund. What Demo 1 lacks is any record of where
  its own work stood, so its new process can't continue the task.

## Context, memory, and state

These are the key ideas from the
video and its slides.

**Agents are loops.** On each turn, the model reasons on what it has seen so
far, then acts through a tool call, a write, or a message. The result goes back
into context for the next turn, and the loop repeats until it reaches a goal, a
stop condition, or a failure. The more autonomy an agent has, the more of its
path it chooses at runtime, so you can't lay out every step ahead of time. We're
also giving agents harder work, such as writing code, creating tools, and even
improving themselves. That makes an agent a long-running, complex system, and it
needs context, memory, and state to do that work well.

**Context** is what the model reasons on, and it's the only one of the three
that lives inside the loop. It grows every turn, but the context window is only
a buffer. Preparing what goes into it is what makes it act like working memory.
These are the common ways to manage it:

- **Stuff the window** puts everything in the prompt.
- **Truncate** drops the oldest turns.
- **Compact** summarizes old turns. It's today's default.
- **Retrieve** brings in what's needed through RAG, memory files, or memory
  tools.
- **Bigger windows** of a million tokens have already shipped.
- **Self-curating context**, still emerging, lets the agent edit its own
  context.

**Memory comes in many kinds, and each one lives somewhere different:**

- **Working memory** is like a whiteboard: what's active for the current
  decision. For an agent, that's its context.
- **Episodic memory** is like a diary: past experiences and events, kept in run
  logs and transcripts.
- **Semantic memory** is like an encyclopedia: facts as the agent recalls them,
  kept in a vector store or a knowledge base.
- **Procedural memory** is like a recipe: skills and how to do things, kept in
  skill files and tool definitions.
- **Parametric memory** is what the model already knew from training. It's in
  the weights, and you can't edit it while the agent runs.

Episodic, semantic, and procedural memory together make up long-term memory. A
long-running agent can't keep everything in its window, so it pulls in only
what each decision needs. A refund skill, for example, waits in a skill file
until a refund comes up. Memory helps the agent decide, but it can be out of
date.

Context and memory cover what the agent knows, not where the work stands.
**State** is about who owns each fact the agent relies on, and each owner is a
system of record. State comes in many types, such as execution state,
authorization state, effect state, and domain state. The demo shows two of
them:

- **Execution state** is where the work stands, and Temporal records it. After
  the crash, a new Worker replays Event History and runs only the refund.
- **Effect state** is whether the refund actually happened, and Stripe owns it.
  Every retry sends the same idempotency key, so Stripe makes one refund. In
  Temporal's terms, Stripe is the source of truth for the effect, the
  `issue_refund` Activity result is the Workflow's record of it, and the
  idempotency key reconciles the two when a crash lands in between.

![The refund agent's loop, with context, memory, and state labeled. In the loop: Reason / Observe (agent_decide_next_step, one model call given everything it has seen so far, or a fixed policy offline) decides what to do next; Act takes one action per turn: ask the customer on turns 1 and 2 ("Was the package opened?" Yes; "What was damaged?" Split seam), lookup_order on turn 3 (order 1234: python plushy, $80.00, delivered), lookup_customer_history on turn 4 (824 days as a customer, 1 prior refund), check_refund_policy if chosen (eligible, no return needed), and Decide on the last turn (approve, escalate, or deny, which leaves the loop). Results go back into context, and on approve the loop exits to issue_refund, the one side effect. Context, inside the loop, is what the model sees this turn: the customer's message "Please refund order 1234", the two answers, the order it just looked up, the customer's history, and its own last step. Outside the loop, long-term memory is what carries across runs in a store, such as the refund policy, which cases need escalation, and how this customer was handled before; none is saved in this demo, and lookups tagged (memory) stand in for retrieval. State is the facts the model doesn't own: execution state in Temporal's Event History (steps already run, 2 answers and 2 lookups; next action issue refund; attempts on the refund call) and effect state in Stripe, the system of record (payment PAID; refund none yet, then one refund, SUCCEEDED). After the crash, Demo 1's execution state lived only in the process, so it's gone and the customer starts over; in Demo 2 it is read back from Event History, and a new Worker resumes at issue refund.](https://docs.temporal.io/img/ai/featured/memory-state-stripe/agent-loop.png)

The diagram places each kind of information around the demo's agent loop. It
was drawn before the demo added its case note, so it shows no long-term memory
saved. `remember_case` now saves a short note after the refund, as the table
shows. The table lists what each kind of information is in this demo and where to
find it.

| | What it helps with | In this demo | Where to find it |
| --- | --- | --- | --- |
| **Context** (working memory) | What the model sees for this one decision.<br />The loop sends it again on every turn | The customer's refund request, their two answers, and what the agent has looked up so far: order 1234 and the refund history of the demo customer, Nyghtowl | `working_memory`.<br />Demo 1: a list in the agent process (`naive_refund.py`).<br />Demo 2: a Workflow field (`workflow.py`) that replay rebuilds.<br />In Event History: the input of each `agent_decide_next_step` Activity |
| **Long-term memory** (a memory store) | What the agent carries across runs:<br />Episodic: past cases.<br />Semantic: facts it looks up.<br />Procedural: how-to, such as skill files and tool definitions | A light version. After the refund, `remember_case` saves a short case note to the demo's SQLite store, and `lookup_customer_history` reads notes back as `past_cases` on later runs.<br />Each guided run starts from fresh data, so the demo plays the same way every time. Each lookup result becomes part of context once the agent reads it | Your store: a vector store, a knowledge base, or memory files.<br />Here: `store.sqlite` in the run's state folder |
| **Execution state** | Where the work stands: which steps finished and what runs next, so the work can continue after a crash without redoing it | The progress Temporal recorded before the crash: the customer answered two questions, the agent finished two lookups (three when a live model also checks the refund policy), and its next step is to issue the refund | Temporal: the Workflow's Event History in the Temporal Web UI, or `uv run refund-demo inspect <workflow-id>` |
| **Effect state** | Whether the side effect really happened | Whether Stripe actually refunded the payment | Stripe: the Dashboard in test mode. The refund's metadata carries `temporal_workflow_id` and `temporal_idempotency_key`.<br />Offline: the ledger |

**Facts have owners.** Each fact the agent
relies on is owned by one system, and that system's record wins when copies
disagree. Memory says what the agent believed; the owner's record says what
happened. A memory store could bring back the customer's answers, but not which
step runs next (Temporal owns that) or whether the refund went out (Stripe owns
that). An agent that trusts a remembered "not refunded yet" can refund twice.

State is that owner's record. Other components reconcile to it, and it stays
the operational source of truth even when it lags the physical world.

| State | Question | Owner in this demo |
| --- | --- | --- |
| Execution state | Where does the work stand? | Temporal |
| Effect state | Did the real effect commit? | Stripe or the offline ledger |
| Authorization state | May this agent act? | The authorization system (read it live) |
| Domain state | What are the business facts? | The application database |

At the demo's kill, Temporal has recorded the two answers, the completed
lookups, and the next action. Stripe has a paid charge and no refund. The
Worker process holds nothing authoritative. Without a record like Temporal's,
any of the [ordinary reasons a process stops](#same-crash-two-outcomes) can
remove loop position right where the application still owes work.

For any fact, ask whether another system already owns it. Asked of every fact,
that question finds more kinds of state than the four in this table, such as
token budgets, locks, and quotas.

**What Temporal keeps, and what needs a memory store.** Temporal records
every Activity's inputs and results, and every Signal, in Event History, which
lives in the Temporal Service rather than in the Worker. For this agent, that
includes each model call's input (the request and working memory) and its
output (the next step, and the written rationale when it decides), plus the
customer's answers. So replay rebuilds this run's working memory after a crash,
with no code to save it. The model's internal reasoning isn't returned by the
provider, so it isn't recorded anywhere.

What Temporal doesn't do is carry knowledge from one run to the next:

- Long-term memory belongs in a store you own. In this demo, `remember_case`
  writes a case note to the SQLite store, and `lookup_customer_history` reads
  it back.
- Temporal runs the store's reads and writes as Activities, so each one is
  recorded and retried like any other step.
- Memory products such as mem0 and Amazon Bedrock AgentCore Memory decide what
  to keep, when to write it, and what to bring back into context. They plug in
  the same way, through Activities.
- Event History has size limits, described under [Gotchas](#gotchas).

#### More on memory, state, and authority

- **Roles, not storage types.** The same data can play more than one role. An
  order row is domain state, owned by your database. Once the agent looks it
  up, it is part of working memory, and a summary the agent keeps of it is
  long-term memory. When they disagree, the order row still wins.
- **Authority, not lifespan.** Execution state isn't simply short-lived, and
  memory isn't simply durable. Event History survives Worker restarts for as
  long as the Workflow Execution is open, and is deleted once the Namespace
  retention period passes after the Execution closes. A memory store can keep a
  fact for years. When they disagree, the owner's record wins, however long each one
  lasts.
- **After the effect.** If the process dies after Stripe accepts the refund but
  before anything on your side records it, only Stripe knows the refund went
  out. The retry reconciles with Stripe by sending the same idempotency key,
  instead of trusting memory. This demo crashes before the refund, because
  that is where the customer's cost is visible. The
  [later-loss walkthrough](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/GUIDE.md#later-loss-after-stripe-accepts-the-refund)
  covers the crash after it.
- **Authorization is state too.** Remembering "you may push" doesn't mean the
  grant is still active, so read it live from the authorization system, never
  from memory. See the
  [companion demo](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/GUIDE.md#authorization-companion-demo).
- **Why the table says context.** Working memory is what the model works with
  on this turn. The context window holds it, and context management selects and
  refreshes it. Here the model sees all of it on every turn, so the table calls
  it context. "Memory" on its own usually means long-term memory in a store.
- **Parametric memory** is what the model learned in training. It lives in the
  weights, you can't see or edit it while the agent runs, and only retraining
  or fine-tuning changes it. When the weights and a record disagree, the record
  wins.
- **Memory has no authority on its own.** Unless provenance and freshness are
  attached and checked, an agent can't tell which of its memories it observed,
  which it inferred, and which went stale after a change elsewhere.
- **Memory can be the only record** of some things, such as the agent's plan,
  or that a customer sounded upset.
- **The store doesn't set the role.** A table of the agent's reflections can be
  a memory store, an orders table can be a domain-state store, and Event
  History records execution state.
- **The idempotency key follows the Run ID.** It stays the same across retries
  in one run. It changes on a reset, on a new run that reuses the Workflow ID,
  and on Continue-As-New.
- **Temporal doesn't hand the result to the agent.** It doesn't inject a
  Workflow's status or result into a model or a UI; your application surfaces
  it.

## Demo takeaways

The demo ends on these three points:

- **Without Temporal, the customer had to start over.** The answers and next
  step lived only in the agent process, and they died with it.
- **With Temporal, a new Worker picked up at the recorded next action,**
  `issue refund`, without asking the customer again. Replay runs the Workflow
  Definition again from the top, and each finished step returns its recorded result
  instead of running again. As the video puts it, "It didn't redo all those
  steps. It just replayed it."
- **Memory helps reasoning continue. Temporal helps the operation continue.**
  Stripe, not either of them, knows whether money moved.

### Cost to run

Starting over repeats every model call and pays for each one again. Replay
returns the recorded results of finished calls, so it doesn't pay for them
again; only a call that was in flight at the crash can run twice.

In the measured live-model run, each demo's pass before the kill made 4 model
calls. After the kill, Demo 1's customer had to start over, which cost 4 more
calls. Demo 2's replay made 0. With `--offline`, the demo uses a fixed
policy and makes no model calls. Token counts, prices, and how they were
measured are in the [README](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#cost-to-run).

## Architecture

![Architecture diagram. A customer asks to refund order 1234, and the request takes two paths. Demo 1, the loop without Temporal: an agent process keeps the answers in memory, then dies; the process is gone and the customer starts over. Demo 2, the same loop on Temporal: a Worker runs the loop and records every step to Event History. The Worker dies, and a new Worker replays Event History, resumes, and sends one refund to Stripe with the same idempotency key.](https://docs.temporal.io/img/ai/featured/memory-state-stripe/architecture.png)

You run one command, `uv run refund-demo stage`
([`stage.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/stage.py)). It runs Demo 1, then starts Demo 2's
Workflow, sends it Signals, and starts and kills its Worker.

- **Demo 1** runs the loop in an ordinary Python agent process
  ([`naive_refund.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/naive_refund.py)). The customer's
  answers, what the agent observed, and its next step live only in that
  process's memory, so they die with it.
- **Demo 2** runs the same loop as the `RefundApprovalAgent` Workflow
  ([`workflow.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/workflow.py)). A Worker process
  ([`worker.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/worker.py)) runs the Workflow and its
  Activities ([`activities/`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/tree/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/activities)), which make the
  model call ([`model.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/activities/model.py)), the lookups
  ([`tools.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/activities/tools.py)), and the `issue_refund`
  call ([`refund.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/activities/refund.py)).
- **After the kill,** a new Worker replays Event History. It runs the Workflow
  Definition again from the top, and each Activity that already finished returns its
  recorded result instead of running again. That rebuilds `working_memory`, and
  the loop resumes at `issue_refund`.
- **Temporal handles every retry.** The model and Stripe SDKs' own retries are
  off, so each retry is an Activity attempt that Temporal schedules and tracks,
  not a hidden retry inside one SDK call. Event History records each step and
  its result.
- **Every `issue_refund` attempt in a run sends the same idempotency key,** to
  Stripe test mode in a live run, or to the offline ledger
  ([`fake_stripe.py`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/src/refund_agent/fake_stripe.py))
  with `--offline`. A refund call that was in
  flight when a Worker died can run again, and the key keeps that retry from
  creating a second refund.

### The agent loop and its Activities

The loop itself is part of the Workflow Definition. On each turn it calls one Activity to decide
what to do next, and then does it. The agent can run five Activities in the loop, and the Workflow runs a sixth, `remember_case`, after the refund:

| Activity | What it does |
| --- | --- |
| `agent_decide_next_step` | Asks the model, or the fixed policy in offline runs, for the next step: ask the customer a question, call one of the lookups, or decide. |
| `lookup_order` | Reads the order the customer wants refunded. |
| `lookup_customer_history` | Reads the customer's past refunds. |
| `check_refund_policy` | Checks whether the order qualifies for a refund. It runs only when the model or the fixed policy asks for it, and the default run doesn't. |
| `issue_refund` | Sends the refund with the run's idempotency key. It's the payment side effect, and it runs only after the decision is to refund. |
| `remember_case` | Saves a short case note for the customer to the demo's SQLite store after the refund. The Workflow ID identifies the note, so a retried write saves it once. |

The customer's answers aren't Activities. They arrive as Signals, and the
Workflow waits for each one. Because Event History records each Activity's
input and result, a new Worker can replay the loop without calling the model
or the lookups again.

The [README](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#architecture) has the wiring diagram, the loop, retries, the idempotency key, and timeouts in detail.

## When you need durable execution

Persisted chat history lets a new process decide again. Durable execution
resumes the decision the agent already made.

Durable execution earns its place when
the work:

- outlives a process or a deploy,
- waits a long time on people or other systems, or
- retries side effects that are real, such as a refund.

**You may not need it when:**

- **The loop is short and the user is right there.** If a run is lost, the
  user just asks again.
- **Redoing a step is cheap and safe.** A chat with no irreversible side
  effects can rely on a session store, and a coding agent's changes live in
  git.
- **Recovery is already handled.** Your runtime keeps execution state for you,
  as LangSmith Deployment, ADK Resume, and DBOS do, or there is a single
  side-effecting call that you retry right away with the same idempotency key.

## Gotchas

**Event History grows every turn.**
Each turn sends all of `working_memory` to the model step again, and Event
History records that input every time. The input gets longer with each turn,
so the history grows with the square of the turn count. The server's defaults
allow 2 MB per payload and 50 MB per run. This demo stops at
`MAX_TURNS = 10`, which keeps it under 7 KB of payloads.
[HISTORY_GROWTH.md](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/HISTORY_GROWTH.md) has the limits, a
[measured example](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/HISTORY_GROWTH.md#measured-in-the-fleet-demo), and a
[Continue-As-New sketch](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/a7279c8927f1bd05476965288c5b6ec60b49e8e1/docs/HISTORY_GROWTH.md#what-continue-as-new-looks-like-sketch).

#### Two fixes: claim check and Continue-As-New

- **A claim check handles one big result.** Keep the data in your own store and
  put only a small key in Event History. Temporal's
  [External Storage](/develop/python/data-handling/external-storage) does this for
  payloads.
- **Continue-As-New handles a long run.** It starts a fresh Event History with
  the same Workflow ID and a new Run ID. Temporal suggests it when the history
  gets long, and your Workflow Definition makes the call. This demo should roll over
  only before the refund, because the idempotency key uses the Run ID. A
  refund attempted in a new run would carry a different key, and Stripe
  couldn't tell it was a retry.

This demo describes both patterns but doesn't implement them.

**A step can run more than once.** Replay doesn't redo finished steps, but a
step that was running when the Worker died has no recorded result, so Temporal
runs it again. Give every side effect an idempotency key that the receiving
system checks. `issue_refund` does this: every attempt in a run sends the same
key, so Stripe makes one refund.

## What this demo is not

- **Not a production refund service.** It leaves out authentication, a
  production database, a web application, multi-agent orchestration, and
  hardening. Its lookups read fixtures, and it runs on a local dev server, not
  Temporal Cloud, with no TLS or API-key options for the connection.
- **Not a guarantee that each step runs once.** A step that was running when
  the Worker died runs again, so the refund relies on its idempotency key
  ([Gotchas](#gotchas)).
- **Not a memory store.** [Context, memory, and state](#context-memory-and-state)
  explains why.
- **Not a fix for model quality.** Temporal doesn't decide what the agent does
  or make the model right; evals and guardrails do.
- **Demo 1 is not a Temporal Worker.** It runs the same decision step with no
  Temporal and no retry layer, and the screens call it the agent process.
- **Not free of trade-offs.**
  - Event History holds customer data, so plan its retention and add an
    encryption codec.
  - Event History has size limits; see [Gotchas](#gotchas).
  - Someone has to run the Temporal Service or pay for Temporal Cloud.
  - The Workflow Definition must be deterministic and versioned.
  - See also [when you need durable execution](#when-you-need-durable-execution).

## Learn more

**In the repo.** The [README](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe) covers what this page leaves out:

- [Make targets, live-model settings, and cleanup](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#run-it)
- [Architecture details](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#architecture), [source layout](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#where-things-live), and [tests](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#tests)
- [Cost to run](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#cost-to-run): measured tokens and dollars, prices, and worst-case estimates
- [How other approaches compare](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#how-other-approaches-compare): memory layers, framework checkpointers, and DBOS, product by product
- [Troubleshooting](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/README.md#troubleshooting)
- [GUIDE.md](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/docs/GUIDE.md), the full reference for running the demo, and [HISTORY_GROWTH.md](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/blob/main/docs/HISTORY_GROWTH.md) on Event History limits
- The code the talk ran is tagged [`as-presented-2026-09`](https://github.com/temporal-community/temporal-ai-agent-memory-state-stripe/tree/as-presented-2026-09). The repo is MIT licensed.

**Elsewhere.**

- [Python SDK](https://github.com/temporalio/sdk-python)
- [Continue-As-New](/workflow-execution/continue-as-new) and [External Storage](/develop/python/data-handling/external-storage)
- [Stripe idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- **For your coding agent:** see [Develop with AI](/with-ai) for Temporal skills and documentation access.

---

# Best practices

Source: https://docs.temporal.io/best-practices

This section collects prescriptive, validated guidance for platform teams, architects, and developers establishing
Temporal standards across an organization. Each page recommends a specific approach based on real-world deployments,
rather than explaining how a feature works internally — for that, see [Encyclopedia](/temporal). Looking for a working
code example instead of a principle? See [Guides](/guides) for pattern-by-pattern implementations with runnable code.

## Namespace, tenancy, and capacity

Start here to decide how many Namespaces you need, how tenants share them, and what capacity model fits your traffic.
These pages form one decision chain: draw your Namespace boundaries, decide how tenants share those boundaries, size
your Actions-per-second capacity, and understand what it costs.

- **[Namespace best practices](./managing-namespace.mdx)** — naming conventions, organizational patterns for splitting
  Namespaces, and production safeguards like deletion protection and Infrastructure as Code.
- **[Multi-tenant application patterns](./multi-tenant-patterns.mdx)** — Task Queue and Namespace isolation patterns for
  multi-tenant applications, with worked capacity-planning examples.
- **[Managing Actions per Second (APS) limits](./managing-aps-limits.mdx)** — why workloads hit APS limits, how to
  design Workflows that use Actions efficiently, and when to use Provisioned Capacity.
- **[Cost optimization](./cost-optimization.mdx)** — common cost anti-patterns and strategies for reducing Actions and
  Storage costs without sacrificing observability.
- **[Cost governance](./cost-governance.mdx)** — a design framework for cost attribution, budget forecasting, and
  anomaly detection using the Billing API and OpenMetrics.

## Security and access control

Temporal Cloud secures the managed service; you're responsible for how your applications authenticate to it and who can
administer your account. These two pages cover both halves.

- **[Managing Temporal Cloud access control](./cloud-access-control.mdx)** — choosing between mTLS certificates and API
  keys, structuring Service Accounts, and rotating credentials without downtime.
- **[Security controls for Temporal Cloud](./security-controls.mdx)** — identity and access management, network
  isolation, data encryption, and availability guidance for a Temporal Cloud account.

## Worker and Workflow reliability

These pages cover the full lifecycle of running reliable Workflows in production: deploy and tune Workers correctly,
alert on the metrics that catch failures early, handle errors correctly in Workflow and Activity code, and validate
that all of it survives real failure conditions before you rely on it.

- **[Worker deployment and performance](./worker.mdx)** — deployment, scaling, and tuning practices for Workers,
  illustrated with a reference application.
- **[Alerting on Worker metrics](./worker-alerting.mdx)** — a recommended alert set with thresholds and triage links.
- **[Error handling](./error-handling.mdx)** — categorizing failures, when to mark errors non-retryable, and
  implementing compensation with the Saga pattern.
- **[Pre-production testing](./pre-production-testing.mdx)** — failure injection, load testing, and a game-day runbook
  for validating operational readiness.

## Organizational enablement

Once your own standards are established, the next challenge is getting every team to follow them without funneling
every question through the platform team.

- **[Knowledge hub](./knowledge-hub.mdx)** — what belongs in an internal Temporal knowledge hub, how to measure its
  effectiveness, and how to keep it current.

> **📝 Note:**
> Scope
> Most of this section covers Temporal Cloud operations. Namespace best practices, Worker deployment, and error handling
> apply to self-hosted Temporal too; security controls, access control, cost optimization, cost governance, and APS
> limits are Cloud-specific. For self-hosted security guidance, see [Security (self-hosted)](/self-hosted-guide/security).

---

# Managing Temporal Cloud access control

Source: https://docs.temporal.io/best-practices/cloud-access-control

> Best practices for managing access control, permissions, and user management in Temporal Cloud.

Temporal Cloud supports two secure authentication methods for Workers:
- **mTLS Certificates**
- **API Keys** (configured via the UI when creating a Namespace)

Both options help secure communication between workers and Temporal Cloud. Choosing the right method and managing it properly is key to maintaining security and minimizing downtime. 

Use this page to define your operating model for machine access to Temporal Cloud. For setup steps and product-specific
mechanics, see [Manage API keys](/cloud/api-keys) and [Manage service accounts](/cloud/manage-access/service-accounts).

Related guidance:
- [Namespace best practices](/best-practices/managing-namespace)
- [Multi-tenant application patterns](/best-practices/multi-tenant-patterns)

The high-level end-to-end rotation process is:

1. **Generate new credentials**: Create new certificates or API keys in Temporal Cloud before the current ones expire
2. **Support dual credentials**: Update Temporal Cloud to support both old and new credentials
3. **Migrate Workers**: Transition Worker applications from old credentials to new credentials
4. **Validate connectivity**: Confirm all Workers can authenticate, and business processes operate normally with new credentials
5. **Remove old credentials**: Remove old certificates and API keys from your secrets provider after confirming successful migration 

This approach ensures near-zero-downtime rotation and prevents authentication failures that could impact running workflows. For specific guidance to rotate mTLS certificates and API keys, see:
- [How to add, update, and remove certificates in a Temporal Cloud Namespace](/cloud/certificates#manage-certificates)
- [Rotate an API key](/cloud/api-keys#rotate-an-api-key)
- Per-SDK code for rotating a Worker's mTLS client certificate without a restart is documented on each SDK's Temporal Client page, in the "Connect to Temporal Cloud" section (for example, [Go](/develop/go/client/temporal-client#connect-to-temporal-cloud)) — Go, Java, Python, .NET, Ruby, and TypeScript are all supported; PHP and Rust are not yet. The [temporal-worker-cert-rotation](https://github.com/temporal-sa/temporal-worker-cert-rotation) reference implementation walks through automating this with cert-manager on Kubernetes.

For mutual TLS (mTLS) implementations, using Let's Encrypt is not recommended, as it is designed primarily for public-facing services and lacks support for internal certificate requirements. 

While we are not making a specific product recommendation, there are several valid options for managing certificates. Many organizations choose vendor solutions such as AWS Private CA, Sectigo, Microsoft Certification Authority, or DigiCert for their integration and lifecycle features. Alternatively, self-signed certificates are a valid and commonly used approach, even in production environments. If you choose to self-sign, tools like [OpenSSL](https://openssl-library.org/), [CFSSL](https://github.com/cloudflare/cfssl), or [step CLI](https://github.com/smallstep/cli) can help generate and manage certificates effectively. 

Select the option that aligns best with your infrastructure, security requirements, and operational needs.

In the case that you are using multiple certificates signed by the same CA, and some of these certificates are for production environments, there are some workarounds you can employ. 

One convention is to give certificates a common name that matches the namespace. If you do this when using the same CA for dev and prod, then you can use Certificate Filters to prevent access to production environments. This is described in detail under the [authorization section](/cloud/certificates#control-authorization) of the documentation. 

## Recommendations

### Establish clear guidelines on authentication methods

Teams should standardize on either [mTLS certificates](/cloud/certificates) or
[API keys](/cloud/api-keys) for the following operations:
- Connect Temporal clients to Temporal Cloud (for example, Worker processes)
- Automation (for example, Temporal Cloud [Operations API](/ops), [Terraform provider](/cloud/terraform-provider), [Temporal CLI](/cli/setup-cli))

By default, teams should use API keys with [service accounts](/cloud/manage-access/service-accounts) for both operations. API keys
are generally easier to set up and rotate than mTLS certificates, and service accounts let you assign account-level and
namespace-level roles.

If your organization requires mutual authentication and stronger cryptographic guarantees, use
[mTLS certificates](/cloud/certificates) to authenticate Temporal clients to Temporal Cloud and use API keys for
automation, because the Temporal Cloud [Operations API](/ops) and [Terraform provider](/cloud/terraform-provider) only
support API key authentication. Unlike API keys tied to users or service accounts, mTLS certificate authentication is
not tied to Temporal Cloud RBAC identities. Namespace access is based on CA trust, with optional
[Certificate Filters](/cloud/certificates#manage-certificate-filters) to narrow access by Common Name.

### Default operating model for service accounts and API keys

For most organizations, use the following defaults:

- Create one Service Account per service or worker deployment, not one shared Service Account for an entire team
- Use account-level Service Accounts only when a service genuinely needs cross-Namespace or account-wide access
- Prefer Namespace-scoped Service Accounts when a service should only access one Namespace
- Grant Service Accounts namespace-level access only to the specific Namespaces they need

This approach gives you cleaner ownership, easier rotation, and better auditability than sharing a single machine
identity across multiple services.

### Use access boundaries that match your Namespace boundaries

The way you partition Namespaces should usually match the way you partition machine identities.

- If multiple services share a Namespace, you may still want one Service Account per service so that each deployment can
  rotate credentials independently.
- If you split workloads into separate Namespaces for security, capacity, or team ownership reasons, those Namespaces
  should usually have separate Service Accounts and API keys as well.
- If you use Namespace-per-tenant isolation, expect your credential model and RBAC model to become correspondingly more
  granular.

For more on topology tradeoffs, see [Namespace best practices](/best-practices/managing-namespace) and
[Multi-tenant application patterns](/best-practices/multi-tenant-patterns).

### Rotate credentials without downtime

Use the following sequence when rotating credentials:

1. Create the replacement credential before the existing one expires.
2. For API keys, create the new valid key while the old key still works, then roll your Workers and clients to use the new key.
3. For client certificates, stage the new certificate before removing the old one when your deployment process supports that transition.
4. Validate connectivity and normal Workflow execution using the new credential.
5. Remove the old credential only after all clients and Workers have switched.

### Use Certificate Filters to restrict access when using shared CAs (for example, `dev` vs `prod`)

Certificate Filters are an additional way of validating using the client certificate presented during client authentication. Give certificates a common name that matches the namespace. This is not a requirement.   

If you do this when using the same CA for dev and prod environments, then you can use Certificate Filters to prevent access to production.

---

# Cost governance on Temporal Cloud

Source: https://docs.temporal.io/best-practices/cost-governance

> A design framework for attributing Temporal Cloud spend to teams, forecasting budget burn rate, and detecting usage anomalies using the Billing API, Billing Center, Usage Dashboards, and OpenMetrics.

This guide lays out a cost-governance framework built entirely on existing Temporal Cloud tooling: the
[Cloud Billing API](/cloud/billing-api), [Billing Center](/cloud/billing), [Usage Dashboards](/cloud/actions-usage),
and the [Action metric](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count).
Enterprise platform teams consistently need to solve three problems:

- **Cost attribution**: attributing spend to the team, environment, and workload that generated it.
- **Budget forecasting and trend tracking**: knowing where a Namespace stands against its monthly budget before the
  invoice closes, not after.
- **Anomaly detection on usage**: treating a spend spike as an early operational signal, not just a finance surprise.

This guide helps you build cost visibility in from the start, rather than discover the gap after costs have already
accumulated.

## Key tools and concepts

Temporal provides tooling across Temporal Cloud, metrics, and the Cloud Operations API to address each of these use
cases programmatically:

| Tool | Grain | Best for | Who sees it |
| :--- | :--- | :--- | :--- |
| [Billing Center](/cloud/billing) | Monthly invoice | Summary invoices, credits, plan management | Account Owner, Finance Admin |
| [Billing API](/cloud/billing-api) | Hourly, daily, monthly | Namespace and tag cost attribution, FinOps ingestion | Account Owner, Finance Admin |
| [Usage Dashboards](/cloud/actions-usage) | Namespace, by Action category | At-a-glance usage in the Cloud UI | Account Owners, Finance Admins, and Global Admins at account level; anyone with Namespace access at Namespace level |
| [Actions in Event History](/cloud/actions-usage#actions-in-workflows) | Per Workflow Execution | Estimating Actions for a specific execution | Account Owners, Global Admins, Namespace Admins, Developers, and Read-Only users |
| [Action metric](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count) | 1-minute, by Workflow and Action type | Near-real-time usage behavior, alerting, debugging | Service account with the Metrics Read-Only role |

### Actions

Temporal Cloud bills primarily on [Actions](/evaluate/cloud/actions), which are billable operations such as starting a
Workflow, scheduling an Activity, recording a Heartbeat, sending a Signal, or receiving a Query or Update. Actions are
grouped into categories (Workflow, Activity, Timer, Signal, Query, Update, Schedule, Nexus) plus a few billed features
(Export, Fairness, Capacity). For background on how usage-based pricing works on Temporal Cloud, see
[Improved cost transparency with usage-based billing](https://temporal.io/blog/improved-cost-transparency-with-usage-based-billing).

> **📝 Note:**
> Not every Action type appears in every surface. Billing and usage data has the most complete picture; Event History
> and OpenMetrics are best used for estimation and trend analysis, not invoice reconciliation.

### Namespace tags

The Billing API attributes every charge to a Namespace and enriches it with [tags](/cloud/namespaces#tag-a-namespace),
user-defined key/value pairs such as team, environment, or cost center. The Billing API reads tags in real time, so a
Namespace that shipped last month without a tag can be tagged today and have its historical cost re-attributed
retroactively. This solves the most common early mistake: forgetting to tag a Namespace before it starts accruing
spend.

```json
{
  "$tmprl_project": ["claims-platform"],
  "team": ["map"],
  "env": ["prod"]
}
```

## Attribute cost by team and workload

**Goal**: give every Temporal Cloud team, use case, and application (for example, Claim, Payment, Notification) its
own line item, sourced directly from Temporal Cloud's billing data instead of an estimate. Use this for both cost
attribution and budget tracking.

### Design artifacts

- Namespace-per-tenant-per-environment as the attribution boundary. Namespace is the finest-grained unit the Billing
  API attributes cost to.
- A required tagging convention enforced at Namespace creation time. At minimum: team, env, and workload or product.
- A daily pull of the Billing API report, joined on tags, feeding the internal FinOps or budget system so each tenant
  sees its own line item.

**Recommended tagging taxonomy**

| Tag key | Example value | Purpose |
| :--- | :--- | :--- |
| team | claim-team, payment-team, notification-team | Chargeback owner |
| env | prd, stg, dev | Separates production spend from lower environments |
| workload | claims-processing, payment, notification | Sub-team or product-line attribution |
| cost-center | cc-12345 | Direct feed into the finance general ledger, if required |

### Implementation path

- Generate a [Billing API](/cloud/billing-api) report at daily granularity for the current and prior two billing
  months: create it with `CreateBillingReport`, poll `GetBillingReport`, then download the CSV. You can do this
  through the Cloud UI or the [Cloud Operations API](/ops).
- Parse the FOCUS-aligned CSV. Group by `ResourceName` (Namespace name plus Temporal Cloud account ID) and `Tags`;
  sum `ContractedCost` by `ChargeDescription`.
- Feed the grouped output into your internal FinOps or budget system as a recurring Workflow, or connect Temporal
  Cloud's native Datadog Cloud Cost Management or Vantage integration if either tool is already in your FinOps stack.

## Forecast budget burn rate

**Goal**: answer "where are we trending against this Namespace's budget, and will we exceed it before month end?"
with enough lead time to act.

### Two data sources, two jobs

Budget tracking needs both a financially accurate source and a fast, directional source, because the Billing API's
current-month data is provisional until the billing month closes:

| Source | Accuracy | Latency | Role in burn-rate tracking |
| :--- | :--- | :--- | :--- |
| [Billing API](/cloud/billing-api) (daily grain) | Invoice-aligned | Usage up to current time minus 24 hours; final at month close | Month-to-date actual spend per Namespace, financial source of truth |
| [Action metric](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count) | Directional usage estimate | Available within about 3 minutes | Early-warning proxy for spend trajectory between billing pulls |

### Burn-rate methodology

- Maintain a budget table per Namespace (or per team, rolling up Namespaces) with a monthly allocation.
- Each day, pull the Billing API daily report and compute month-to-date (MTD) actual spend per Namespace.
- Project end-of-month (EOM) spend using a simple run-rate extrapolation: EOM projection = MTD spend ÷ days
  represented in the report × days in month. Derive the represented period from the report's charge dates so the
  roughly 24-hour Billing API lag doesn't depress the projection.
- Compare the projection against the allocated budget and flag any Namespace trending to exceed it.
- Between daily Billing API pulls, use the 24-hour Actions estimate from [OpenMetrics](/cloud/metrics/openmetrics) as
  a same-day directional check on whether a Namespace's usage trajectory has shifted:

```
# 24-hour Actions estimate from OpenMetrics (PromQL)
sum(
  avg_over_time(
    temporal_cloud_v1_billable_action_count{temporal_namespace="$namespace"}[24h:1m]
  )
) * 86400
```

### Alerting

- Trigger a budget alert when a Namespace's EOM projection exceeds 90% (warning), 100% (target), or 110%
  (over budget) of its allocation.
- Route alerts to the Namespace owner and the FinOps or platform DRI.

> **❗ Important:**
> `temporal_cloud_v1_billable_action_count` is a usage estimate, not your bill. It doesn't account for storage, support
> fees, pricing-tier calculations, some features (TRUs, Fairness), or rounding. Use it to catch trend shifts early in
> the month, and reconcile final numbers against the Billing API or Billing Center once the billing month closes.

## Detect usage anomalies early

**Goal**: treat a sudden spend or Action spike as an early operational signal for a runaway Workflow or bad deploy.

### Why OpenMetrics is the right layer

The `temporal_cloud_v1_billable_action_count` metric is broken down by both `action_type` and `temporal_workflow_type`
at one-minute granularity. That combination answers the two questions that matter most during an incident: which
Workflow is driving the spike, and what kind of Action it's doing more of.

For example queries, see
[Getting the most out of the Billable Action Count metric](https://temporal.io/blog/getting-the-most-out-of-the-billable-action-count-metric).
For a real-world example of using this metric to validate metering, see
[Dogfooding the Billable Actions metric](https://temporal.io/blog/dogfooding-the-billable-actions-metric-how-granular-observability-improved-our-metering-validation).

### Define and find anomalies 

Establish a baseline of Actions consumption. Your workload may have seasonality across hours, days, and weeks, which
makes setting alerts harder. Some starting points:

- Alert if the number of Actions for an hour exceeds the expected maximum for all hours in a day.
- Alert if the number of Actions for a day exceeds the expected maximum for all days in a month.

When an alert triggers, or during root cause analysis, check monitoring dashboards for more nuanced anomalies,
including:

- New business generating more Action usage. Determine whether the pattern is a spike or steady growth that's now
  exceeding typical thresholds — if it's growth, it's time to establish a new baseline.
- Correlation with deploys. Feed deploy markers from your CI/CD pipeline into the same Datadog or Grafana dashboard
  as your Action-rate charts; a spike that lines up with a deploy timestamp is a regression.

## Guardrails to design around

- `temporal_cloud_v1_billable_action_count` is a directional estimate. Don't use it for financial reconciliation —
  use the Billing API instead.
- Current-month billing data is provisional. It finalizes only when the billing month closes, and available data
  lags current time by roughly 24 hours.
- Only one billing report generates at a time per account; additional requests queue rather than fail. Use an
  idempotency key (`async_operation_id`) on retries and poll with exponential backoff.
- High-cardinality metrics need filtering at scale. Plan Namespace and label filtering into your
  [OpenMetrics](/cloud/metrics/openmetrics) scrape configuration from day one, not after you hit the datapoint
  ceiling.

None of the three use cases in this guide need new Temporal Cloud features. They need a deliberate design applied to
tooling that already exists: a tagging convention enforced at Namespace creation, a scheduled pull of the Billing
API, and an OpenMetrics-based alerting layer tuned to your account's scale. The organizations that get the most value
from this treat it as a platform requirement before the first tenant onboards, not as a remediation project after the
first unexplained invoice.

The result is a Temporal Cloud bill that behaves like any other well-governed enterprise OpEx line: attributable,
forecastable, and defensible.

---

# Workflow cost optimization

Source: https://docs.temporal.io/best-practices/cost-optimization

> Strategies for optimizing costs associated with workloads running on Temporal Cloud while maintaining workflow reliability and observability.

This guide provides strategies for optimizing costs associated with workloads running on Temporal Cloud while maintaining Workflow reliability and observability.

## Overview

Temporal Cloud uses consumption-based pricing with two primary cost components: [Actions and Storage](/cloud/pricing#action).
Optimization opportunities vary significantly based on your workload characteristics - Workflows with high signal volume face different cost drivers than long-running Workflows with large payloads.

> **❗ Important:**
> Build Workflows following best practices first, then optimize based on observed costs.
> Premature optimization can compromise observability and create operational challenges.

Every optimization involves tradeoffs.
This guide helps you make informed decisions about where and how to optimize based on your specific requirements.

If you need more guidance on Workflow design, contact a Temporal Solutions Architect.

## Common anti-patterns

Avoid these patterns that either inflate costs unnecessarily or create problems through aggressive optimization:

### Premature Activity consolidation

Combining Activities before understanding failure modes reduces observability and retry control.
Activities should be split based on failure boundaries and retry requirements, not cost optimization alone.
See [How many Activities should I use in my Temporal Workflow](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow) for a decision framework.

### Inappropriate use of Local Activities

Using Local Activities for all operations without understanding their failure semantics and limitations.
Local Activities don't provide Worker-level isolation and have different retry behavior.
See [Local Activities](/local-activity) for guidance.

### Missing Continue-As-New

Long-running Workflows that don't implement Continue-As-New accumulate large Event Histories, increasing storage costs and impacting performance.
Workflows running days or weeks or processing thousands of events require [Continue-As-New](/workflow-execution/continue-as-new).

### High volume of Activity retries

Generally, the default values for [Activity retries](/encyclopedia/retry-policies) work well in most cases.
However, excessive Activity retries often indicate underlying issues like timeouts that are too short or Activities that frequently fail.
Monitor Activity retry frequency. If it is high, consider increasing retry intervals or Activity timeouts before failures occur.
See [Spooky Stories: Chilling Temporal Anti-Patterns](https://temporal.io/blog/spooky-stories-chilling-temporal-anti-patterns-part-2#2-hiding-behind-the-chainsaws) for guidance on retry defaults and patterns.

### Large payloads in Event History

Passing multi-megabyte payloads through Workflows when external storage (S3, blob storage) is more appropriate.
Use [compression](/troubleshooting/blob-size-limit-error#payload-size-limit) or the [claim check pattern](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern) for large data.

### Over-optimization at the expense of observability

Aggressively optimizing costs without maintaining sufficient visibility for debugging and operational needs.
Balance cost reduction with your team's observability requirements.

For example, merging five separate Activities (validate input, call payment API, update database, send notification, generate receipt) into a single "processOrder" Activity reduces from 5 Actions to 1, but you lose per-step visibility in the Temporal UI.
When the notification step fails, you can't see which of the five steps failed, you lose independent retry control (a notification failure retries the entire flow including the payment call), and you can't filter Workflows by failure stage.

Similarly, removing [Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) from a long-running data processing Activity saves Actions, but means you can't detect a stuck Worker until the full Activity timeout expires and you lose progress tracking (for example, "processed 500 of 1,000 records").

### Excessive Activity Heartbeats

Each Heartbeat counts as one Action.
Only use Heartbeats for long-running Activities (10+ minutes) where you need to detect Worker failures and track progress.
Short-running Activities that complete in seconds or minutes don't need Heartbeats.
See [Activity Heartbeat documentation](/encyclopedia/detecting-activity-failures#which-activities-should-heartbeat) for guidance.

## Understanding cost drivers

Temporal Cloud pricing consists of Actions, Storage, and Support.
If you are new to Temporal Cloud, see the [pricing documentation](/cloud/pricing#action) to learn more and familiarize yourself with [what results in a billable Action](/evaluate/cloud/actions) in Temporal Cloud.

### Cost distribution

For most workloads, Actions represent the majority of total costs, with storage typically accounting for 10% or less of a monthly bill.
Focus optimization efforts on what's driving costs with a specific workload:

**High Actions costs generally indicate**:

- [Many Activities per Workflow](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow)
- Frequent [Signals, Queries, or Updates](/encyclopedia/workflow-message-passing#choosing-messages)
- Long-running Activities with [Heartbeats](/cloud/worker-health#manage-worker-heartbeating)
- High [Activity retry rates](/demos/activity-retry-simulator)
- Extensive Query usage

**High Storage costs generally indicate**:

- Large payloads in Workflow inputs, outputs, or Activity results
- Long retention periods with high Workflow volume
- Long-running Workflows without Continue-As-New
- Workflows accumulating large Event Histories

### Optimization priority

1. **Actions optimization**: Usually provides the largest cost reduction opportunity
2. **Active Storage optimization**: Relevant for long-running Workflows or large payloads
3. **Retained Storage optimization**: Relevant for high volume combined with long retention periods

## Measuring

Establish [baseline metrics](/cloud/metrics/openmetrics/metrics-reference) before optimizing and be sure to validate impact after implementation.
Specifically:

- Actions consumption (per Workflow, per day/month, by Namespace)
- Storage consumption (Active and Retained)
- Monthly costs (total, per Namespace, per Workflow Type)
- Observability metrics (time to debug, incident detection)

## Actions optimization

[Actions](/evaluate/cloud/actions) encompass Workflow operations, Activity Executions, Signals, Queries, and other interactions with Temporal.
Each represents a unit of consumption.

### Activity granularity

Activity granularity is a fundamental architectural decision that impacts both costs and observability.
More Activities provide better visibility and retry control but increase the Action count.
Fewer Activities reduce costs but limit observability.

For detailed discussion of this tradeoff, see [How many Activities should I use in my Temporal Workflow?](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow)

### Child Workflows vs Activities

[Child Workflows cost 2 Actions](/evaluate/cloud/actions#workflow) compared to an Activity's 1 Action.
See [Child Workflows documentation](/child-workflows) for detailed comparison of capabilities and use cases.

### Standalone Activities vs a Workflow that runs a single Activity

Starting a [Standalone Activity](/standalone-activity) costs 1 Action.
Running the same single Activity in a wrapper Workflow costs 1 Action for the Activity and 1 Action for the Workflow, not including Activity heartbeats, retries, or other billable [Actions](/evaluate/cloud/actions#activity).

Use a Workflow when you need multi-step orchestration, human in the loop, signals, timers, updates, queries, or enhanced step-by-step visibility with event history and [replay](/workflow-execution#replay).

Use a Standalone Activity when you need to execute a single Activity function reliably.

### Retry Policies

Each Activity retry counts as one Action.
Default [Retry Policies](/encyclopedia/retry-policies) can be aggressive, which is appropriate for most operations but costly for expensive external operations.

For example, consider an Activity that calls a third-party payment API.
With Temporal's default Retry Policy (1s initial interval, 2.0 backoff coefficient, unlimited maximum attempts), if that API goes down for 30 minutes, the Activity retries approximately 20 times before reaching the 100s maximum interval cap, then continues retrying every 100 seconds.
Each retry counts as 1 Action.
Across 1,000 concurrent Workflows hitting the same outage, that produces 20,000+ extra Actions from retries alone.

For expensive external operations like this, consider:

- Setting `MaximumAttempts` to cap total retries
- Increasing `InitialInterval` (for example, to 10s) to reduce retry frequency
- Adding error types to `NonRetryableErrorTypes` for errors that won't resolve on retry (such as 4xx HTTP status codes)
- Using [next retry delay](/encyclopedia/retry-policies#per-error-next-retry-delay) to dynamically control retry timing based on failure types (for example, respecting rate-limit headers)
- Implementing an [Activity pause pattern](/cli/command-reference/activity#pause) to wait for manual intervention rather than automatic retries

Use the [Activity Retry Simulator](/demos/activity-retry-simulator) to visualize how different Retry Policy configurations affect retry behavior and Action consumption.

Refer to this blog post on [Mastering Workflow retry logic for resilient applications](https://temporal.io/blog/failure-handling-in-practice) for additional guidance.

### Local Activities

A [Local Activity](/local-activity#local-activity) is an Activity Execution that executes in the same process as the Workflow Execution that spawns it.
Therefore, multiple Local Activities that run back-to-back only [count as a single billable action](/evaluate/cloud/actions#activity), whereas each regular Activity counts as a billable action.
However, there are tradeoffs to converting regular Activities to Local Activities.
For example, if a specific Local Activity fails, *all* of them will be retried together.
Review [the docs](/local-activity) or reach out to your account team to learn more.

#### When to stick with Regular Activities

Use Regular Activities instead of Local Activities if you require any of the following:

- Activities may take more than 10 seconds to complete
- Independent retry control for each Activity
- Need to avoid re-running expensive Activities when unrelated Activities fail
- Immediate Signal/Update handling during execution
- Separate resource management (like rate limits) for each Activity

### Batching operations

#### Search Attributes

1. [Search Attributes](/search-attribute#custom-search-attribute) provided at Workflow start do not count as billable Actions.
   If Search Attribute values are known before starting the Workflow, provide them at Workflow start to eliminate these costs entirely.
2. For Search Attributes that must be updated during Workflow Execution, each `UpsertSearchAttributes` call counts as 1 Action regardless of how many attributes are updated.
   Batch multiple related attribute updates into single operations to reduce Actions consumed.

See the [Temporal Cloud Action Documentation](/evaluate/cloud/actions#workflow) for details.

#### Signal handling

Where feasible, implement deduplication logic client-side or aggregate data into fewer Signals.
Use `SignalWithStart` instead of separate `StartWorkflow` and `SignalWorkflow` calls when initiating Workflows with Signals.

## Storage optimization

[Storage costs](/cloud/pricing#storage) are divided into Active Storage (open Workflows) and Retained Storage (closed Workflow History during retention period).
Active Storage is significantly more expensive than Retained Storage.

### Active Storage

Active Storage applies to open Workflows and their Event Histories.
The following sections detail optimization opportunities for Active Storage.

#### Continue-As-New

For long-running Workflows with extended sleep/wait periods, calling Continue-As-New before sleeping closes the current execution (moving to cheaper Retained Storage) and starts fresh when work resumes, reducing Active Storage costs.
See [Continue-As-New documentation](/workflow-execution/continue-as-new) to learn more.

#### Compression

Large payloads increase Active Storage costs.
Implement a custom Data Converter with compression for moderately large payloads (100KB-1MB).
See the [Data Converter documentation](/default-custom-data-converters#custom-data-converter) to learn more.

#### Claim check pattern

For very large payloads or binary data, store data externally (S3 or GCS) and pass references through Workflows.

### Retained Storage

Retained Storage applies to closed Workflow History during the retention period.
The following sections detail optimization opportunities for Retained Storage.

#### Retention periods

The default Namespace retention is 30 days (configurable between 1 and 90 days).
Adjust based on operational and compliance requirements.

**Considerations**:

- Shorter retention reduces costs but limits historical analysis
- Audit investigation patterns before shortening retention
- Ensure compliance requirements are met

See [Namespace retention documentation](/temporal-service/temporal-server#retention-period) for configuration details.

#### Workflow export

Temporal Cloud supports exporting Event Histories to external storage for compliance while maintaining shorter retention periods.
Note that Workflow export costs one Action per export.

See the [Workflow History export documentation](/cloud/export) for more details.
Alternatively, if you are looking to do analysis on closed Workflow Executions, [review this blog post to learn how to gain insights from exported Event Histories](https://temporal.io/blog/get-insights-from-workflow-histories-export-on-temporal-cloud).

## Validation

### Validation approach

1. **Test in non-production**: Validate functional correctness before production deployment
2. **Monitor comprehensively**: Leverage the [Usage dashboard](/cloud/actions-usage#usage) in the Cloud UI to track the impact on Actions and Storage after optimizations are made
3. **Progressive rollout**: Deploy to a small percentage, validate, then expand. Review the [Worker Versioning documentation](/production-deployment/worker-deployments/worker-versioning) to learn about rolling out changes to Workflows
4. **Continuous review**: Re-evaluate optimization effectiveness quarterly as system evolves

### Success criteria

- Cost reduced without increasing mean time to repair (MTTR)
- Workflow success rates maintained or improved
- Reduced observability does not increase mean time to detect (MTTD) incidents

### Tools

- Temporal Cloud Usage dashboard for Actions and Storage metrics
- Event History for per-Workflow billable Actions estimates
- Export metrics to observability platforms such as Datadog or Grafana for custom monitoring

## When to get help

Engage the Temporal team for Workflow audits when experiencing:

- Complex Workflow patterns with unclear optimization paths
- Compliance requirements limiting optimization options
- Need for custom DataConverters or advanced patterns
- Desire for expert validation of optimization strategies

Contact your Temporal Account Representative or [Temporal support](http://support.temporal.io) to discuss optimization services.

---

# Error handling

Source: https://docs.temporal.io/best-practices/error-handling

Temporal automatically retries failed Activities and recovers from infrastructure failures through
[Durable Execution](/evaluate/why-temporal). But not all failures should be retried. This page covers how to categorize
failures, when to mark errors as non-retryable, and how to handle failures that retries cannot resolve.

For background on how Temporal represents and propagates failures, see
[Application failures](/encyclopedia/application-failures).

## Categorize failures 

When an operation fails, the appropriate response depends on the nature of the failure. Failures fall into three
categories based on whether retrying can resolve them.

### Transient failures

A transient failure is a one-off event that resolves on its own without intervention. For example, a Worker happens to
make a network request at the exact moment an administrator replaces a network cable. The cause is unlikely to affect
future requests.

Transient failures are resolved by retrying the operation shortly after the failure. Temporal's default
[Retry Policy](/encyclopedia/retry-policies) handles transient failures automatically.

### Intermittent failures

An intermittent failure is one that recurs but resolves over time. For example, a service that uses rate limiting will
reject requests once the threshold is reached, but will accept requests again after the rate limiter resets.

Intermittent failures require retries spaced out over a longer period. Configure your
[Retry Policy](/encyclopedia/retry-policies) with an appropriate `backoffCoefficient` and `maximumInterval` to avoid
overwhelming the failing service.

### Permanent failures

A permanent failure is one that will recur indefinitely until the cause is fixed. For example, a request that fails due
to an invalid email address will continue to fail no matter how many times the operation retries. The only resolution is
to correct the email address.

Permanent failures cannot be resolved through retries. They require different input data, a code fix, or some external
intervention. Mark these errors as [non-retryable](#non-retryable-errors) to fail fast instead of consuming resources on
retries that will not succeed.

## Mark permanent errors as non-retryable 

When your code detects a permanent failure, mark the error as non-retryable to prevent unnecessary retry attempts. For
background on what Application Failures are and how the `non_retryable` flag works, see
[Application Failure](/encyclopedia/application-failures#failure-representation).

Use non-retryable errors for situations like:

- **Invalid input data**: A malformed email address, a negative payment amount, or a missing required field.
- **Business rule violations**: A customer outside the service area, an order exceeding credit limits, or an expired
  promotion code.
- **Authorization failures**: The caller does not have permission to perform the operation.
- **Data validation errors**: A referenced record does not exist, or data fails integrity checks.

There are two ways to mark errors as non-retryable:

**In the Activity (implementer decides):** Set the `non_retryable` flag when throwing an
[Application Failure](/encyclopedia/application-failures#failure-representation). This enforces the constraint for all
callers. Use this when the Activity implementer knows that the error can never be resolved through retries.

**In the Retry Policy (caller decides):** Add the error type to the Retry Policy's list of
[non-retryable error types](/encyclopedia/retry-policies#non-retryable-errors). This lets different Workflows make
different decisions about the same Activity. Use this when the decision depends on the caller's business logic.

### Preserve retryability when wrapping errors

When an Activity returns an error, the SDK checks the **outermost** error type to determine retryability. If you catch a
non-retryable Application Failure and re-throw it wrapped in a generic language error, the `non_retryable` flag is lost
and the Activity will be retried.

To add context to an error while preserving its retry behavior, wrap it in another Application Failure with the same
`non_retryable` flag. Do not wrap Application Failures in generic language errors.

For a detailed explanation of how the SDK-to-server chain works, see
[The outermost error type determines retryability](/encyclopedia/application-failures#outermost-error-type).

### Use non-retryable errors sparingly

In most cases, let the Retry Policy handle retry limits through [timeouts](/encyclopedia/detecting-activity-failures)
and maximum attempts. Reserve `non_retryable` for cases where retrying is guaranteed to be futile.

For SDK-specific syntax and code examples, see the error handling guide for your language:

- [Python](/develop/python/best-practices/error-handling)
- [Go](/develop/go/best-practices/error-handling)
- [.NET](/develop/dotnet/best-practices/error-handling)
- [Ruby](/develop/ruby/best-practices/error-handling)

## Design Activities for idempotence 

Activities may execute more than once due to retries, so design them to be
[idempotent](/activity-definition#idempotency): producing the same result whether executed once or multiple times.

This is especially important because of an edge case in distributed systems. A Worker can execute an Activity, complete
it, and then crash before reporting the result to the Temporal Service. The Activity is retried even though it
completed, because the Service has no record of the completion.

Use idempotency keys to prevent duplicate operations. Combine the Workflow Run ID and Activity ID for a value that is
consistent across retries but unique across Workflow Executions.

## Implement compensation with the Saga pattern 

When a multi-step process fails partway through, previous steps may need to be undone. The
[Saga pattern](/design-patterns/saga-pattern) coordinates a sequence of operations where each step has a
compensating action that reverses its effects. If any step fails, the compensating actions for previously completed
steps execute in reverse order.

For SDK-specific implementations with working code examples, see:

- [Python Saga pattern](/develop/python/best-practices/error-handling#implement-saga-pattern)

---

# Knowledge Hub

Source: https://docs.temporal.io/best-practices/knowledge-hub

As organizations scale their Temporal adoption, the use cases become more complex, and it can be difficult to locate all the relevant information you need.
Tribal knowledge gets siloed within teams, leading to inconsistent patterns for Workflow design, error handling, and testing.
This fragmentation leads to an increased burden on support, as well as slower onboarding for new developers.
You may also inadvertently introduce security vulnerabilities or compliance gaps.
To prevent these issues, you can establish an internal Temporal knowledge hub to address common issues that arise when multiple teams
adopt Temporal independently.

This guide covers what belongs in a knowledge hub, how to communicate it, and how to keep it useful over time.
To bootstrap your knowledge hub, use the
[Temporal Platform Hub](https://go.temporal.io/platform-hub) template as a starting point.

## What belongs in your knowledge hub

Although Temporal itself has [thorough documentation](/), not all of it applies to your organization or your teams' use cases.
The knowledge hub distills the documentation into the specific information your teams need.
One way to organize the content is according to where developers are in their journey.
The following sample outline shows what sections to include.

### Evaluate

The Evaluation section helps developers understand what Temporal is and whether it fits their problem.
Goals for this section are to increase knowledge hub traffic and decrease the support question rate.
Include the following items in this section:

- **Temporal overview** explains what Temporal is, why your organization chose it, and the business value metrics
  that justify adoption.
- **Decision framework** provides qualifying questions, good and bad use cases, and alternative recommendations so
  developers can determine whether Temporal fits their problem.

### Build

The Build section is aimed at developers who are new to Temporal.
The goal of this section is to give developers a single path from zero to a running Workflow.
Include the following items in this section:

- **Getting started** walks developers through a 30-minute quickstart covering environment setup, a starter template,
  and running a first Workflow locally and on Temporal Cloud. Use an existing Temporal [quickstart](/quickstarts), or build one from your own use case.
- **Learning paths** provides self-paced courses from foundational to advanced topics, tailored by persona, and links
  to [Temporal's free training](https://learn.temporal.io/).

### Ship

The Ship section provides the architecture standards and guardrails developers need to go from a local prototype to a production
deployment.
The goal for this section is to provide reference material developers need to reduce their time to production.
Include the following items in this section:

- **Architecture and standards** documents Namespace conventions, connectivity requirements, and Worker deployment
  standards that every team follows.
- **Cost guidance** explains billable Actions, storage tiers, and cost-saving tips so developers build cost-efficient
  Workflows.
- **Shared responsibility** defines an ownership matrix between Platform and Application teams across IAM,
  infrastructure, development, deployment, observability, and operations.
- **Design patterns** curates Workflow patterns with descriptions and code sample links that developers can adopt
  directly.

### Operate

The Operate section gives developers the tools to self-serve during incidents and find answers without escalating to the Platform team.
The goal of this section is to decrease the number of questions to support.
Include the following items in this section:

- **Troubleshooting and escalation** covers observability tools, runbooks for common issues, escalation paths, SLAs,
  and example alert definitions.
- **Support and FAQs** documents your support tier, ticket submission process, Temporal account contacts, expert-led
  session types, and frequently asked questions.

## Measure the success of your knowledge hub 

After you've created your knowledge hub, establish metrics to measure its effectiveness for your organization.
The following table shows example indicators that organizations use to measure the impact of their knowledge hub,
along with realistic before-and-after targets:

| Metric | Before Knowledge Hub | Target | What it tells you |
| :--- | :--- | :--- | :--- |
| **Time to first Workflow** | Days to weeks (developers piece together scattered resources) | Under 30 minutes (developers follow a single getting started guide) | Measures onboarding friction. A short time to first Workflow signals that your getting started guide is effective. |
| **Time to Workflow in production** | Weeks to months (blocked by unclear Namespace provisioning and deployment processes) | Under 2 weeks (developers follow documented self-service provisioning) | Measures the gap between development and delivering value. Long times point to missing documentation and automation opportunities. |
| **Support question rate** | 20-30+ questions per week to the Platform team via Slack | Fewer than 5 per week | Measures self-service resolution. A declining trend shows that developers are finding answers in the knowledge hub instead of asking the Platform team. |
| **Knowledge Hub traffic** | N/A (no centralized resource exists) | Steady or growing page views per month | Identifies which content developers rely on and where gaps remain. Declining traffic on a page may indicate it is outdated; high traffic with high bounce rates may indicate the page is not answering the question. |

These metrics create a feedback loop: measure, identify gaps, improve content, and measure again.

## What doesn't belong in your knowledge hub

A knowledge hub is not a mirror of [Temporal's official documentation](/).
Avoid duplicating SDK API references, concept explanations, or release notes that Temporal already maintains.
When that content changes, your copy becomes a source of confusion rather than clarity.
Instead, link to the official docs and reserve your knowledge hub for organization-specific decisions, conventions,
and operational procedures that Temporal's public documentation does not cover.

## How to maintain and communicate your knowledge hub

Having a thorough, complete knowledge hub isn't useful if the information becomes stale, or if developers don't know it exists.

### Assign ownership

Designate a Platform team or developer experience team as the owner.
This team is responsible for initial content creation, ongoing maintenance, and reviewing contributions from
application teams.

### Make it discoverable

A knowledge hub that developers can not find is the same as not having one.

- Register a short URL (for example, `go/temporal`) that redirects to the knowledge hub.
- Pin the link in your Temporal-related communication channels (Slack, Microsoft Teams).
- When answering questions in Slack, respond with a link to the relevant knowledge hub page instead of re-explaining
  inline.
  This builds the habit of checking the hub first.

### Review metrics regularly

Track the metrics from [Measure the success of your knowledge hub](#measuring-success-of-your-knowledge-hub) on a regular cadence to identify what is
working and where gaps remain.

Capture every question that reaches the Platform team through Slack or tickets as a candidate for new content.

Solicit contributions from application teams through a lightweight process such as a pull request template.

### Keep content current

- **Review on a cadence**: Review each page at least quarterly.
  Assign a review owner and date to each page so staleness is visible.
- **Tie updates to events**: Update the knowledge hub whenever your organization changes its Temporal architecture,
  updates its deployment tooling, or modifies its shared responsibility model.
- **Prune aggressively**: Remove or archive content that no longer applies.
  Outdated documentation is worse than no documentation because developers follow it and get unexpected results.

## Get started

Start with the [Temporal Platform Hub template](https://go.temporal.io/platform-hub) as your foundation.

---

# Managing Actions per Second (APS) limits in Temporal Cloud

Source: https://docs.temporal.io/best-practices/managing-aps-limits

> Control how limits are assigned to a Namespace with Capacity Modes

Every Namespace on Temporal Cloud has an Actions Per Second (APS) limit. An action is any operation that modifies
Workflow state or interacts with the Temporal Service — starting or completing a Workflow, executing an Activity,
sending a Signal. When a Namespace exceeds its APS limit, Temporal throttles requests. Depending on the business
priority of the Workflow, that may be fine, or it may have significant impact.

APS consumption isn't always intuitive: a single Workflow Execution generates multiple actions from the moment it
starts, so use cases that fit comfortably within APS limits at small scale can exhaust them as they grow. This guide
covers why workloads hit APS limits, how to design Workflows that use actions efficiently, and when to use
[Provisioned Capacity](#provisioned-capacity-and-trus) — Temporal Resource Units (TRUs) that reserve additional
capacity for spiky or unpredictable workloads.

## Understand actions in Temporal 

### What counts as an action?

Actions are the fundamental operations that drive your Workflows forward.
The following is a partial list — see [the full list in our documentation](/evaluate/cloud/actions).

- Workflows: Starting, completing, resetting. Also starting Child Workflows, as well as Schedules and Timers
- Activities: Starting, retrying, Heartbeating
- Signals, Updates, and Queries 

Actions that count toward an APS limit are, with a few exemptions, the same as actions that are billable. 
Nearly everything that happens in Temporal (state changes, decision points, interactions) counts as an action.

### The action multiplier effect

What this means is that when you start a single Workflow, you're not performing just one action as it relates to APS because a Workflow isn’t a single atomic operation, it’s a series of events that Temporal orchestrates. 
Each Activity at the start of the Workflow is an Action, so there can be a burst of Activities at the start of a Workflow. 
Additionally, there are often business reasons to start multiple Workflows at the same time.

These can all contribute to the multiplier effect. 

### The effect of rate limiting

In Temporal Cloud, the effect of rate limiting is increased latency, not lost work. 
Workers [might take longer](/cloud/slo#throughput) to complete Workflows. 

## Common reasons customers hit APS limits

The following patterns most commonly push customers into APS constraints.

### Bursty traffic

Most businesses don't operate at constant velocity—they have rhythms, cycles, and spikes. 
These patterns can create APS challenges because Temporal Cloud enforces limits at the per-second level.

Common bursty patterns include:

- Calendar-driven spikes: Month-end financial close processes, quarterly reporting Workflows, payroll that runs on the 1st and 15th, scheduled batch jobs that kick off at midnight. These create predictable but intense load concentrations.
- Event-driven surges: Product launches, marketing campaigns, flash sales, breaking news, or seasonal events like Black Friday. 
- Recovery scenarios: When a downstream dependency fails and then recovers, you often get a thundering herd effect—hundreds or thousands of Workflows that were waiting all suddenly resume execution simultaneously, creating an artificial spike in APS consumption.
- Geographic/business hours concentration: Global applications often see load follow the sun, with peak activity during business hours in each region. If your business concentrates in specific markets, you'll see daily peaks rather than even 24/7 distribution.
- Retry Storms: when a large number of Workflows get stuck on an Activity, and that Activity is failing, if retry delay is very short, this can cause a spike in Actions.
- Timer Storms: a large number of Workflows all set a Timer for the exact same time--triggering a spike as those Timers fire and then Activities run, causing a lot of actions all at the same time.

These types of processes can result in your Namespace averaging 200 APS over a day, but spiking to 800 APS or more during your peak hour, day, or event. 

#### How to mitigate

You can’t change the patterns of how customers interact with your systems, but there are some adjustments you can make to your Workflows to make traffic patterns more consistent, especially for use cases where immediate response isn’t necessary. 

These adjustments include:
- Implement application-level queuing or rate limiting to smooth out predictable spikes.
- For scheduled batch operations, stagger start times rather than triggering everything at once--implement jitter in your high-volume [Schedules](/schedule#spec).
- Implement jitter when starting Workflows, such as with [Start Delay](/workflow-execution/timers-delays#delay-workflow-execution).
- Accept rate limiting
- [Provisioned Capacity](/cloud/capacity-modes#provisioned-capacity)

### Cascading Workflows and fan-out patterns

Decomposing complex processes into parent and Child Workflows (or with Nexus) is a common and often appropriate pattern, but the APS costs multiply dramatically with depth and fan-out.

Consider an order fulfillment Workflow that spawns Child Workflows for payment processing, inventory management, shipping, and customer notifications. 
Each Child Workflow goes through its full action lifecycle (start, tasks, activities, completion), and all of those actions count toward the APS limits on your Namespace. 

This pattern appears frequently in:
- Batch processing: A parent workflow processes a file with 1,000 records, spawning a Child Workflow for each record. Batch processing is also often bursty whenever the batch begins.
- Map-reduce patterns: Data processing Workflows that fan out to process partitions in parallel, then aggregate results.

This challenge additionally compounds when you have multiple levels of nesting--parent Workflows that create children, which create their own children. 

#### How to mitigate

- Evaluate whether Child Workflows are necessary--other options include Activities or Workflows in another Namespace (via Nexus)
- When you do use Child Workflows, limit fan-out size--design a Child Workflow to process its work in batches rather than one Child per work item. [This sample application](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/slidingwindow) shows more detail.
- Consider flattening deeply nested hierarchies into shallower structures. 

### Human-in-the-loop processes at scale

Workflows that incorporate human decision-making--approvals, reviews, manual data entry, quality checks--tend to be long-running and interaction-intensive, which creates sustained APS load.

These Workflows can involve Queries from UIs to display current state and pending tasks.

At small scale, this is manageable. But when you're running thousands of them at the same time--like a content moderation queue with pending reviews, or a loan approval system processing applications, or a support ticket system managing thousands of open cases--the cumulative APS load from all of those long-running Workflows adds up.

#### How to mitigate

- Avoid polling patterns where UIs constantly query Workflow state. Instead, push state changes to a database that UIs can read. 

### Real-time SLAs and deadline management

Businesses with strict service level agreements often implement active monitoring and escalation in their Workflows. 
This is generally accomplished by setting Timers every [x] minutes to determine if an SLA deadline is approaching, allowing the Workflow to trigger escalations or alerts. 

Each of these Timers/monitoring actions affect APS. 
When you have thousands of in-flight Workflows all actively monitoring their own SLAs, the background load becomes significant.
You're consuming substantial APS capacity even when Workflows aren't doing their primary work.

#### How to mitigate

- Use longer monitoring intervals where possible. For example, check SLAs every 30 minutes rather than every 1 minute.
- Where possible, consolidate Timers. Rather than 10 Timers that check 10 tasks, have 1 Timer and then check those 10 tasks.
- Where possible, have an external system signal your Workflow rather than using short-lived Timers to poll.
- For retries, use exponential backoff with reasonable initial intervals.

## Additional design patterns
There are some design patterns that can lead to high APS that are consistent across many different types of business use cases. 

### Many small Activities

Consider two approaches to processing 1,000 records:

- Approach A: Create a Workflow that spawns 1,000 separate activities, one per record.
- Approach B: Create a Workflow that spawns 10 activities, each processing 100 records in a batch.

Approach B will clearly result in less APS. 
This is a simple example, but this pattern shows up everywhere: processing individual transactions versus batches, sending individual notifications versus bulk operations, or making separate API calls versus batch endpoints. 
Each separate Activity adds Action overhead.

#### How to mitigate

- Consider if you can combine multiple external calls within a single Activity.
- If processing a large amount of data, process it in chunks.
- See [How Many Activities should I use in my Temporal Workflow?](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow) for more information.

### Multiple use cases in one Namespace
Often when starting with Temporal, the first use case is implemented in a single Namespace, generally one per logical environment. 
When the second Temporal use case is implemented, it runs in the same Namespace, as do the third, fourth, and any later ones. 

An APS limit is set per Namespace, so multiple use cases with multiple traffic patterns in the same Namespace can exhaust this limit quickly.

#### How to mitigate

Plan for a set of Namespaces (one per environment) per use case. This gives each use case its own APS envelope and
reduces the blast radius when one workload spikes or misbehaves.

If you are deciding whether to split by use case, service, or domain, see
[Managing a Namespace](/best-practices/managing-namespace) for a topology decision framework.

## Provisioned capacity and TRUs

The strategies above help you design Workflows that use actions efficiently.
But sometimes you need more capacity than the on-demand model provides, especially for spiky or unpredictable workloads.

Temporal Cloud offers two [Capacity Modes](/cloud/capacity-modes):

- **On-Demand mode** (default): Your Namespace automatically scales based on your trailing 7-day usage. This works well for steady, predictable workloads.
- **Provisioned mode**: You reserve capacity by adding Temporal Resource Units (TRUs), giving you guaranteed headroom for traffic spikes.

See [Capacity Modes](/cloud/capacity-modes) for complete details on TRUs, available increments, and how to manage capacity via UI, CLI, or API.

### Choosing the right approach

Use Provisioned capacity when the on-demand model can't respond quickly enough:

| Scenario | Pattern | Recommendation |
|----------|---------|----------------|
| **Planned spikes** | Promotions, holiday traffic, product launches | Pre-provision TRUs before the event starts |
| **Unplanned spikes** | Sudden traffic surges, viral events | React instantly via UI/CLI/API when you see throttling |
| **Load testing** | Validating new services at scale | Provision TRUs for the test, deprovision after |
| **Batch jobs** | Scheduled high-throughput jobs | Automate TRU scaling via API around job schedules |
| **Migrations** | Onboarding a new workload faster than on-demand adjusts | Bridge with TRUs for approximately 7 days while the on-demand envelope catches up |

> **📝 Note:**
> When switching back to on-demand mode, your APS limit resets to the running average from the last 7 days.
> If Temporal Support has set a custom limit for your Namespace, that limit is preserved across the switch.
> Plan for this if your workload is sensitive to the transition.

### Cost optimization tips

See [Capacity Modes Pricing](/cloud/pricing#capacity-modes-pricing) for billing details. To minimize costs:

- Provision only when you need extra capacity
- Deprovision promptly after spikes end
- For predictable patterns, automate scaling to minimize time in provisioned mode

### Automation best practices

Since you understand your workload patterns better than any auto-scaling system, consider building your own TRU automation:

- **Use the [Cloud Ops API](/ops), [Terraform Provider](/cloud/terraform-provider), or [tcld CLI](/cloud/tcld)** to programmatically scale capacity based on your application's signals
- **Set utilization thresholds**: For example, scale up when hitting 70-80% of your limit, scale down after sustained low usage
- **Schedule capacity changes**: Use [Temporal Schedules](/schedule) or Workflows to increase TRUs before known events
- **React to leading indicators**: If your application has upstream signals (incoming order queue depth, marketing campaign start), use those to trigger capacity changes proactively

## Know if you're hitting APS limits 

In addition to understanding the patterns that can affect APS limits on a Temporal Namespace, it's also important to know if you're approaching (or exceeding) these limits.
Temporal Cloud provides several metrics that, if tracked, will tell you if you're being rate limited due to APS.
See the documentation on [detecting resource exhaustion](/cloud/service-health#rps-aps-rate-limits) for an explanation of those metrics as well as a sample Grafana dashboard that shows how they could be viewed.

### Monitor for TRU decisions 

If you're considering Provisioned capacity, set up monitoring to understand your usage patterns:

- **Use [OpenMetrics](/cloud/metrics/openmetrics)**: For real-time visibility into APS consumption, integrate Temporal Cloud metrics with your observability stack
- **Track APS usage against limits**: Monitor [`temporal_cloud_v1_total_action_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_total_action_throttled_count) to detect throttling events
- **Set alerts at 70-80% utilization**: This gives you time to provision TRUs before hitting limits
- **Analyze historical patterns**: Understanding your traffic patterns helps you decide between reactive TRU provisioning and proactive automation

## Key takeaways

The following table recaps the main reasons customers hit APS limits and how to address them:

| Reason for Hitting APS Limits | How to Address It |
|-------------------------------|-------------------|
| Bursty Traffic                | Implement application-level queuing or rate limiting to smooth spike, stagger start times for scheduled batch operations. |
| Cascading Workflows <br />and Fan-Out Patterns | Evaluate if Child Workflows are necessary (consider activities or another Namespace), limit fan-out size by processing work in batches within a Child Workflow, consider flattening deeply nested hierarchies. |
| Human-in-the-Loop <br />Processes at Scale | Design long-running Workflows to minimize sustained APS load from interaction (by avoiding polling where UIs constantly Query state and using Signals only for key human inputs). |
| Many small activities         | Consider if you can combine multiple external calls within a single Activity. If processing a large amount of data, process it in chunks. |
| Multiple use cases <br />in one Namespace | Plan for a set of Namespaces (one per environment) per use case. |
| Planned traffic spikes        | Pre-provision TRUs before the event, then deprovision after. |
| Unpredictable spikes <br />requiring instant response | Switch to Provisioned mode for self-service capacity scaling via UI, CLI, or API. |
| Load testing at scale         | Provision TRUs for the test duration, deprovision when complete. |
| New workload onboarding       | Bridge with TRUs while the on-demand envelope adjusts (approximately 7 days). |

## General guidance

When designing Temporal Workflows with an eye toward APS limits, ask yourself the following questions: 
- How many actions will a single execution of this Workflow consume?
- How many Workflows will typically be running at the same time?
- What happens to APS consumption when the number of Actions * number of active Workflows scales to 100x current volume?
- Are there natural opportunities to combine operations: combine activities, or process chunks of data together?
- Am I polling when I could be using Signals?
- Does this Workflow need to run continuously, or can it be event-driven?

A few hours spent optimizing Workflow design can save you from capacity crunches, emergency limit increases, and potentially significant cost increases down the road.

---

# Namespace best practices

Source: https://docs.temporal.io/best-practices/managing-namespace

> Best practices for organizing and managing Temporal Namespaces, including naming conventions, organizational patterns, and production safeguards.

> **ℹ️ Info:**
> Applies to both open source and Temporal Cloud
> This page covers namespace best practices that apply to **both** open source Temporal and Temporal Cloud.
> Platform-specific guidance is clearly labeled throughout.
>
> For reference documentation, see:
> - [Namespace concepts](/namespaces)
> - [Managing Namespaces (open source)](/self-hosted-guide/namespaces)
> - [Namespaces (Temporal Cloud)](/cloud/namespaces)

A [Namespace](/namespaces) is a unit of isolation within the Temporal Platform.
It ensures that Workflow Executions, Task Queues, and resources are logically separated, preventing conflicts and enabling safe multi-tenant usage.

Use this page to decide how many Namespaces you need and where to draw boundaries between environments, services,
domains, and tenants. For Cloud-specific namespace mechanics such as creation, tagging, and gRPC endpoints, see
[Namespaces (Temporal Cloud)](/cloud/namespaces).

Related guidance:
- [Managing Temporal Cloud access control](/best-practices/cloud-access-control)
- [Multi-tenant application patterns](/best-practices/multi-tenant-patterns)
- [Managing Actions per Second (APS) limits in Temporal Cloud](/best-practices/managing-aps-limits)

## Naming conventions

### Use lowercase and hyphens

Use lowercase letters and hyphens (`-`) as separators in Namespace names.

- **Temporal Cloud**: Namespace names are case-insensitive, so `MyNamespace` and `mynamespace` refer to the same Namespace.
- **Open source**: Namespace names are case-sensitive, so `MyNamespace` and `mynamespace` are different Namespaces.

To avoid confusion across environments, always use lowercase.

**Example**: `payment-checkout-prd`

### Follow a consistent naming pattern

Use a pattern like `<use-case>-<domain>-<environment>` to name Namespaces:

| Component | Max Length | Examples |
|-----------|------------|----------|
| Use case | 10 chars | `payments`, `fulfill`, `orders` |
| Domain | 10 chars | `checkout`, `notify`, `inventory` |
| Environment | 3 chars | `dev`, `stg`, `prd` |

**Examples**: `payments-checkout-dev`, `fulfill-notify-prd`, `orders-inventory-stg`

**Why this pattern?**
- Simple and easy to understand
- Clearly separates environments
- Groups related services under domains
- Allows platform teams to implement chargeback to application teams
- Namespace-level limits are isolated between different services and environments

> **💡 Tip:**
> Temporal Cloud
> Cloud Namespace names are limited to [39 characters](/cloud/namespaces#temporal-cloud-namespace-name).
> If you need to include region, use short codes (for example, `aps1`, `use1`).

## Organizational patterns

### Choose your Namespace boundary intentionally

Start with the smallest number of Namespaces that gives you clear ownership and safe isolation.
In Temporal Cloud, a Namespace boundary affects:

- [APS limits](/evaluate/cloud/limits#actions-per-second) and rate limiting
- access control and credential scope
- blast radius for misconfigured or overloaded Workers
- observability boundaries for dashboards and alerts
- operational overhead for provisioning, tagging, and lifecycle management

Use the following decision table as a starting point:

| If you need... | Prefer... | Why |
|---|---|---|
| Basic environment isolation for a single application or use case | Namespace per use case and environment | This is the simplest pattern and works well for most initial deployments |
| Separate operational ownership for services within the same use case | Namespace per use case, service, and environment | This isolates credentials, limits, and operational changes per service |
| Stronger boundaries across teams, domains, or business capabilities | Namespace per use case, domain, and environment | This reduces blast radius and lets teams own their own Namespace contracts |
| Tenant-specific credentials, rate limits, or compliance boundaries | Namespace per tenant | Use this only for a small number of high-value tenants because of the operational overhead |

As a default, start with one Namespace per use case and environment. Split later when APS pressure, security
requirements, ownership boundaries, or troubleshooting needs justify the extra operational cost.

### Pattern 1: Namespace per use case and environment

For simple configurations without multiple services or team boundaries.

**Naming convention**: `<use-case>-<environment>`

**Example**: `payments-prd`, `orders-dev`

Choose this pattern when:

- one team owns the use case
- environments need clean separation
- workload volume and criticality do not yet require further isolation

### Pattern 2: Namespace per use case, service, and environment

When multiple services that are part of the same use case communicate externally to Temporal via API (HTTP/gRPC).

**Naming convention**: `<use-case>-<service>-<environment>`

**Example**: `payments-gateway-prd`, `payments-processor-prd`

Choose this pattern when:

- services need separate credentials or access policies
- one service can exhaust APS or operational limits independently of the others
- teams want separate ownership of deployment, alerting, or on-call boundaries

### Pattern 3: Namespace per use case, domain, and environment

When multiple services need to communicate with each other, use [Temporal Nexus](/nexus) to connect Workflows across Namespace boundaries.
This provides better security, fault isolation, and modularity than sharing a Namespace.

**Naming convention**: `<use-case>-<domain>-<environment>`

**Example**: `payments-checkout-prd`, `payments-refunds-prd`

Choose this pattern when:

- multiple teams or domains need independent release cadence and ownership
- failures in one domain should not affect the others
- you want a stronger permission boundary between capabilities
- you plan to expose cross-Namespace contracts through Nexus

For systems without Nexus, services can communicate via [Signals](/sending-messages#sending-signals) or [Child Workflows](/child-workflows) within the same Namespace.

> **📝 Note:**
> Workflow ID uniqueness
> When multiple teams share a Namespace, prefix each Workflow ID with a service-specific string to ensure uniqueness.
> Task Queue names must also be unique within the Namespace.

### Pattern 4: Namespace per tenant

Use a separate [Namespace](/namespaces) per tenant only when each tenant needs a true isolation boundary.

This is usually appropriate only for a small number of high-value tenants that require:

- dedicated credentials and access control
- tenant-specific rate limits or capacity decisions
- strict compliance or data-isolation boundaries
- independent debugging, alerting, and operational ownership

For most SaaS use cases, a shared Namespace with per-tenant [Task Queues](/task-queue) is simpler and more scalable.
See [Multi-tenant application patterns](/best-practices/multi-tenant-patterns) for those designs.

### What should cause you to split a Namespace later?

Revisit your topology when one or more of the following becomes true:

- one workload is consuming enough APS that it regularly threatens others in the same Namespace
- one team needs tighter access controls or dedicated credentials
- production troubleshooting requires clearer dashboards, alerts, or ownership boundaries
- one application or domain is business-critical enough that its blast radius must be reduced
- a tenant or regulated workload needs stronger separation than Task Queue isolation can provide

Splitting a Namespace increases safety, but it also adds overhead for provisioning, tagging, credentials, and cross-Namespace coordination.
Use [Nexus](/nexus) where possible instead of sharing Temporal primitives across team or domain boundaries.

## Production safeguards

### Use an Authorizer (open source only) 

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service to set restrictions on who can create, update, or deprecate Namespaces.

If an Authorizer is not set, Temporal uses the `nopAuthority` authorizer that unconditionally allows all API calls.

On Temporal Cloud, [role-based access controls](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) provide namespace-level authorization without custom configuration.

### Enable deletion protection (Temporal Cloud only) 

[Enable deletion protection](/cloud/namespaces#delete-protection) for production Namespaces to prevent accidental deletion.

### Enable High Availability (Temporal Cloud only) 

For business-critical use cases with strict uptime requirements, enable [High Availability features](/cloud/high-availability) for a [higher contractual SLA](https://temporal.io/sla).

### Use Infrastructure as Code (Temporal Cloud only) 

Use the [Temporal Cloud Terraform provider](/cloud/terraform-provider) to manage Namespaces.
If Terraform isn't suitable, scripting against the [Cloud Ops API](/ops) or [tcld](/cloud/tcld) is a good alternative.

This provides:
- Documentation of each Namespace's purpose and owners
- Prevention of infrastructure drift
- Version-controlled configuration changes

Use `prevent_destroy = true` in your Terraform configuration to prevent accidental Namespace deletion via Terraform.
This is separate from [Temporal Cloud deletion protection](/cloud/namespaces#delete-protection), which prevents deletion through any interface.

**Reference**: [Example Terraform configuration](https://github.com/kawofong/temporal-terraform)

## Tag Namespaces (Temporal Cloud only) 

[Tags](/cloud/namespaces#tag-a-namespace) are key-value metadata pairs that help organize, track, and manage Namespaces.

Tags complement your naming convention by adding metadata that doesn't fit in the Namespace name.
While the name captures use case, domain, and environment, tags can capture additional dimensions like team ownership, data sensitivity, or business criticality.

### Recommended tag categories

| Tag Key | Purpose | Examples |
|---------|---------|----------|
| `environment` | Deployment stage | `dev`, `staging`, `production` |
| `team` | Owning team | `platform`, `payments`, `identity` |
| `division` | Business unit | `engineering`, `finance`, `ops` |
| `criticality` | Business importance | `high`, `medium`, `low` |
| `data-sensitivity` | Data classification | `pii`, `pci`, `public` |
| `latency-sensitivity` | Performance tier | `realtime`, `batch`, `async` |

For tag structure, limits, and management instructions, see [How to tag a Namespace](/cloud/namespaces#tag-a-namespace).

## Description 

Use an optional [Namespace description](/cloud/namespaces#describe-a-namespace) for a one-line purpose note: workload,
owner context, or why the Namespace exists.

Keep structured classification in [Tags](#tagging). Description is for humans reading the Namespace. Tags are for
filter, report, and automation dimensions such as `team` and `environment`.

Do not put secrets, confidential data, or PII in a description. Like Tags and [Search Attributes](/search-attribute),
descriptions are not encrypted.

## SDK client configuration

Set Namespaces in your SDK Client to isolate your Workflow Executions.
If you do not set a Namespace, all Workflow Executions started using the Client will be associated with the `default` Namespace.

You must register a Namespace before setting it in your Client.

For configuration details, see:
- [Namespace concepts](/namespaces)
- [Namespaces (Temporal Cloud)](/cloud/namespaces#access-namespaces)

---

# Multi-tenant application patterns

Source: https://docs.temporal.io/best-practices/multi-tenant-patterns

> Learn how to build multi-tenant applications using Temporal with task queue isolation patterns, worker design, and best practices.

Many SaaS providers and large enterprise platform teams use a single Temporal [Namespace](/namespaces) with [Task Queue Fairness](#1-single-task-queue-with-fairness-recommended) or [per-tenant Task Queues](#2-task-queues-per-tenant) to power their multi-tenant applications. These approaches maximize resource efficiency while maintaining logical separation between tenants.

This guide covers architectural patterns, design considerations, and practical examples for building multi-tenant applications with Temporal.

For related guidance on where to draw Namespace boundaries and how to scope credentials and permissions, see
[Namespace best practices](/best-practices/managing-namespace) and
[Managing Temporal Cloud access control](/best-practices/cloud-access-control).

## Architectural principles

When designing a multi-tenant Temporal application, follow these principles:

- **Define your tenant model** - Determine what constitutes a tenant in your business (customers, pricing tiers, teams, etc.)
- **Prefer simplicity** - Start with the simplest pattern that meets your needs
- **Understand Temporal limits** - Design within the constraints of your Temporal deployment
- **Test at scale** - Performance testing must drive your capacity decisions
- **Plan for growth** - Consider how you'll onboard new tenants and scale workers

## Architectural patterns

There are four main patterns for multi-tenant applications in Temporal. Start with a single Task Queue with Fairness, and choose a pattern with stronger isolation only when your tenants need it.

### 1. Single Task Queue with Fairness (Recommended)

**Use a single [Task Queue](/task-queue) with [Fairness keys](/develop/task-queue-priority-fairness#task-queue-fairness) to distribute work across tenants.**

Start with this pattern unless your tenants need the hard isolation of dedicated Task Queues. This pattern uses [Task Queue Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) to manage multi-tenant workloads within a single Task Queue. Each tenant is assigned a fairness key, and fairness weights control the share of Task dispatches each tenant receives.

You can also set per-fairness-key rate limits (requests per second) to cap individual tenant throughput, preventing any single tenant from consuming too much of the queue's throughput.

**Pros:**
- Priority and Fairness keys and weights can be adjusted without redeployment
- Onboarding new tenants doesn't require spinning up additional [Workers](/workers)
- Simpler Worker topology than per-tenant Task Queues

**Cons:**
- Fairness is probabilistic and may be harder to debug than strict isolation
- The fairness weight applies at schedule time, not at dispatch time, so it only affects newly-scheduled Tasks
- When using [Worker Versioning](/worker-versioning), Fairness isn't guaranteed between versions

> **💡 Tip:**
> This pattern works well when you have many tenants with different service tiers and want to manage them without the operational overhead of per-tenant Task Queues or Workers.

**Related:**

- [Task Queue Fairness Reference](/develop/task-queue-priority-fairness#task-queue-fairness)
- [Fairness pattern](/design-patterns/fairness)
- [Priority pattern](/design-patterns/priority-task-queues)

### 2. Task queues per tenant

**Use different [Task Queues](/task-queue) for each tenant's [Workflows](/workflows) and [Activities](/activities).**

Use this pattern when tenants need compute isolation or dedicated Worker capacity, which Fairness doesn't provide. Each tenant gets dedicated Task Queue(s), with [Workers](/workers) polling multiple tenant Task Queues in a single process.

**Pros:**
- Strong isolation between tenants
- Efficient resource utilization
- Flexible worker scaling
- Easy to add new tenants
- Can handle thousands of tenants per [Namespace](/namespaces)

**Cons:**
- Requires worker configuration management
- Potential for uneven resource distribution
- Need to prevent "noisy neighbor" issues at the worker level

**Related:**

- [Per-tenant Task Queue pattern details](#per-tenant-task-queue-pattern)

### 3. Shared Workflow Task Queues, separate Activity Task Queues

**Share [Workflow Task Queues](/task-queue) but use different [Activity Task Queues](/task-queue) per tenant.**

Use this pattern when [Workflows](/workflows) are lightweight but [Activities](/activities) have heavy resource requirements or external dependencies that need isolation.

**Pros:**
- Easier worker management than full isolation
- Activity-level tenant isolation
- Good for compute-intensive Activities

**Cons:**
- Less isolation than per-tenant Task Queues
- Workflow visibility is shared
- More complex to reason about

### 4. Namespace per tenant

**Use a separate [Namespace](/namespaces) for each tenant.**

Only practical for a smaller number of high-value tenants due to operational overhead. Most teams find this manageable for fewer than 50 tenants, though organizations with strong automation may scale higher. This pattern is not a good fit if you expect a very large number of tenants (10,000+).

**Pros:**
- Complete isolation between tenants — no noisy neighbor problem
- Each Namespace has its own [rate limits](/evaluate/cloud/limits) that can be provisioned on demand per customer
- Each Namespace can be deployed across [multiple regions](/evaluate/cloud/regions) globally
- Per-Namespace observability is available by default
- Maximum security boundary

**Cons:**
- Higher operational overhead
- Credential and connectivity management per [Namespace](/namespaces)
- Requires a new [Worker](/workers) pool deployment for each customer (minimum 2 per Namespace for high availability)
- Not cost-effective at scale

This pattern is usually chosen when tenant boundaries also need to be credential boundaries. For example, each tenant may
need its own service accounts, API keys, dashboards, or rate limits. If that is your primary driver, review
[Managing Temporal Cloud access control](/best-practices/cloud-access-control) together with
[Namespace best practices](/best-practices/managing-namespace) before committing to Namespace-per-tenant isolation.

**Related:**

- [Namespace Isolation in Temporal Cloud](/evaluate/cloud/security#namespace-isolation)

### Pattern comparison

| | Fairness-based | Task Queues per tenant | Shared Workflow / Separate Activity TQs | Namespace per tenant |
|---|---|---|---|---|
| **Isolation** | Probabilistic (weighted) | Task Queue level | Activity-level only | Complete |
| **Noisy neighbor protection** | Weight-based throttling | Strong | Activity-level | Full — separate rate limits |
| **Worker management** | Simple — single Task Queue | Moderate — config per tenant | Moderate | High — Worker pool per tenant |
| **Onboarding new tenants** | Set fairness and priority values (no new Workers) | Config update and restart | Config update and Worker restart | New Namespace and Worker pool |
| **Observability** | Per-Task Queue metrics | Per-Task Queue metrics | Mixed | Per-Namespace |
| **Rate limiting** | Per-key rate limits | Shared across Task Queue | Shared across Namespace | Independent per Namespace |
| **Scale ceiling** | Thousands of tenants | Thousands of tenants | Thousands of tenants | 10,000 (Namespace limit) |
| **Best for** | Most multi-tenant apps, including tiered SaaS with many tenants | Tenants that need dedicated Worker capacity | Heavy Activity workloads | High-value, compliance-sensitive tenants |

## Per-tenant Task Queue pattern

This section details the per-tenant Task Queue pattern.

### Worker design

When a [Worker](/workers) starts up:

1. **Load tenant configuration** - Retrieve the list of tenants this Worker should handle (from config file, API, or database)
2. **Create [Task Queues](/task-queue)** - For each tenant, generate a unique Task Queue name (for example, `customer-{tenant-id}`)
3. **Register [Workflows](/workflows) and [Activities](/activities)** - Register your Workflow and Activity implementations once, passing the tenant-specific Task Queue name
4. **Poll multiple Task Queues** - A single Worker process polls all assigned tenant Task Queues

```go
// Example: Go worker polling multiple tenant Task Queues
for _, tenant := range assignedTenants {
    taskQueue := fmt.Sprintf("customer-%s", tenant.ID)

    worker := worker.New(client, taskQueue, worker.Options{})
    worker.RegisterWorkflow(YourWorkflow)
    worker.RegisterActivity(YourActivity)
}
```

### Route requests to Task Queues 

Your application needs to route [Workflow](/workflows) starts and other operations to the correct tenant [Task Queue](/task-queue):

```go
// Example: Starting a Workflow for a specific tenant
taskQueue := fmt.Sprintf("customer-%s", tenantID)
workflowOptions := client.StartWorkflowOptions{
    ID:        workflowID,
    TaskQueue: taskQueue,
}
```

Consider creating an API or service that:
- Maps tenant IDs to Task Queue names
- Tracks which [Workers](/workers) are handling which tenants
- Allows both your application and Workers to read the mappings of:
    1. Tenant IDs to Task Queues 
    1. Workers to tenants

### Capacity planning

Key questions to answer through performance testing:

**[Namespace](/namespaces) capacity:**
- How many concurrent [Task Queue](/task-queue) pollers can your Namespace support?
- What are your [Actions Per Second (APS)](/evaluate/cloud/limits#actions-per-second) limits?
- What are your [Operations Per Second (OPS)](/references/operation-list) limits?

**[Worker](/workers) capacity:**
- How many tenants can a single Worker process handle?
- What are the CPU and memory requirements per tenant?
- How many concurrent [Workflow](/workflows) executions per tenant?
- How many concurrent [Activity](/activities) executions per tenant?

**SDK configuration to tune:**
- `MaxConcurrentWorkflowTaskExecutionSize`
- `MaxConcurrentActivityExecutionSize`
- `MaxConcurrentWorkflowTaskPollers`
- `MaxConcurrentActivityTaskPollers`
- Worker replicas (in Kubernetes deployments)

### Provision new tenants 

Automate tenant onboarding with a Temporal [Workflow](/workflows):

1. Create a tenant onboarding Workflow that:
   - Validates tenant information
   - Provisions infrastructure
   - Deploys/updates [Worker](/workers) configuration
   - Triggers Worker restarts or scaling
   - Verifies the tenant is operational

2. Store tenant-to-Worker mappings in a database or configuration service

3. Update Worker deployments to pick up new tenant assignments

## Practical example

**Scenario:** A SaaS company has 1,000 customers and expects to grow to 5,000 customers over 3 years. They have 2 [Workflows](/workflows) and ~25 [Activities](/activities) per Workflow. All customers are on the same tier (no segmentation yet).

### Assumptions

| Item | Value |
|---|---|
| Current customers | 1,000 |
| Workflow Task Queues per customer | 1 |
| Activity Task Queues per customer | 1 |
| Max Task Queue pollers per Namespace | 20,000 (per [Cloud limits](/evaluate/cloud/limits)) |
| SDK concurrent Workflow task pollers | 5 |
| SDK concurrent Activity task pollers | 5 |
| Max concurrent Workflow executions | 200 |
| Max concurrent Activity executions | 200 |

### Capacity calculations

**[Task Queue](/task-queue) poller limits:**
- Each [Worker](/workers) uses 10 pollers per tenant (5 Workflow + 5 Activity)
- Maximum Workers in [Namespace](/namespaces): 20,000 pollers ÷ 10 = **2,000 Workers**

**Worker capacity:**
- Each Worker can theoretically handle 200 [Workflows](/workflows) and 200 [Activities](/activities) concurrently
- Conservative estimate: **250 tenants per Worker** (accounting for overhead)
- For 1,000 customers: **4 Workers minimum** (plus replicas for HA)
- For 5,000 customers: **20 Workers minimum** (plus replicas for HA)

**Namespace capacity:**
- At 250 tenants per Worker, need 2 Workers per group of tenants (for [HA](/best-practices/worker#run-at-least-two-workers-per-task-queue))
- Maximum tenants in Namespace: (2,000 Workers ÷ 2) × 250 = **250,000 tenants**

> **📝 Note:**
> These are theoretical calculations based on SDK defaults. **Always perform load testing** to determine actual capacity for your specific workload. Monitor CPU, memory, and Temporal metrics during testing.
>
> While testing, also pay attention to your [metrics capacity and cardinality](/cloud/metrics/openmetrics/api-reference#managing-high-cardinality).

### Worker assignment strategies

**Option 1: Static configuration**
- Each [Worker](/workers) reads a config file listing assigned tenant IDs
- Simple to implement
- Requires deployment to add tenants

**Option 2: Dynamic API**
- Workers call an API on startup to get assigned tenants
- Workers identified by static ID (1 to N)
- API returns tenant list based on Worker ID
- More flexible, no deployment needed for new tenants

## Best practices

### How tenant isolation affects Namespace and access design

Your multi-tenant architecture also determines how much isolation you get from Namespaces and access controls:

- **Shared Namespace with Fairness or per-tenant Task Queues**: Best for scale and operational simplicity, but tenant isolation is
  mostly enforced by your application and worker routing logic rather than by Temporal credentials.
- **Separate Namespaces for domains or services**: Useful when teams need separate credentials, dashboards, APS
  envelopes, or on-call boundaries.
- **Namespace per tenant**: Strongest isolation, but highest provisioning and credential-management overhead.

If tenants, teams, or regulated workloads need different credentials or RBAC boundaries, decide that together with your
Namespace topology. See [Namespace best practices](/best-practices/managing-namespace) and
[Managing Temporal Cloud access control](/best-practices/cloud-access-control).

### Monitoring

Track these [metrics](/references/sdk-metrics) per tenant:
- [Workflow completion](/cloud/metrics/openmetrics/metrics-reference#workflow-completion-metrics) rates
- [Activity execution](/cloud/metrics/openmetrics/metrics-reference#task-queue-metrics) rates
- [Task Queue backlog](/cloud/metrics/openmetrics/metrics-reference#task-queue-metrics)
- [Worker resource utilization](/references/sdk-metrics#worker_task_slots_used)
- [Workflow failure rates](/encyclopedia/detecting-workflow-failures)

### Handle noisy neighbors 

Even with [Task Queue](/task-queue) isolation, monitor for tenants that:
- Generate excessive load
- Have high failure rates
- Cause [Worker](/workers) resource exhaustion

Strategies:
- Implement per-tenant rate limiting in your application
- Implement fairness keys and apply per-key rate limits
- Move problematic tenants to dedicated Workers
- Use [Workflow](/workflows)/[Activity](/activities) timeouts aggressively

### Tenant lifecycle

Plan for:
- **Onboarding** - Automated provisioning [Workflow](/workflows)
- **Scaling** - When to add new [Workers](/workers) for growing tenants
- **Offboarding** - Graceful tenant removal and data cleanup
- **Rebalancing** - Redistributing tenants across Workers

### Search Attributes

Use [Search Attributes](/search-attribute) to enable tenant-scoped queries:
```go
// Add tenant ID as a Search Attribute
searchAttributes := map[string]interface{}{
    "TenantId": tenantID,
}
```

This allows filtering [Workflows](/workflows) by tenant in the UI and SDK:
```sql
TenantId = 'customer-123' AND ExecutionStatus = 'Running'
```

## Related resources

**Related:**

- [Namespaces](/namespaces)
- [Temporal Cloud Limits](/evaluate/cloud/limits)
- [Visibility and Search Attributes](/visibility)

---

# Pre-production testing

Source: https://docs.temporal.io/best-practices/pre-production-testing

> Experience-driven testing practices for teams running Temporal applications, covering failure injection, load testing, and operational validation.

This guide collects practical, experience-driven testing practices for teams running Temporal applications.
The goal is not just to verify that things fail and recover, but to build confidence that *recovery*, *correctness*, *consistency*, and *operability* hold under real-world conditions.

The scenarios below assume familiarity with Temporal concepts such as [Namespaces](/namespaces), [Workers](/workers), [Task Queues](/task-queue), [History shards](/temporal-service/temporal-server#history-shard), [Timers](/workflow-execution/timers-delays), and [Workflow replay](/workflow-execution#replay).
Start with [Understanding Temporal](/evaluate/understanding-temporal#durable-execution) if you need background.

Before starting any load testing in Temporal Cloud, we recommend connecting with your Temporal Account team and our Developer Success Engineering team.

## Guiding principles

Before diving into specific experiments, keep these principles in mind:

- **Failure is normal**: Temporal is designed to survive failure and issues, but *your application logic* must be too.
- **Partial failure is often harder to deal with than total failure**: Systems that are "mostly working" expose the most flaws.
- **Recovery paths deserve as much testing as steady state**: Analyze recovering application behavior as much as you analyze failing behavior.
- **Build observability before you break things**: Ensure metrics, logs, and visibility tools are in place before injecting failures.
- **Testing is a continual process**: Testing is never finished. Testing is a practice.

## Worker testing

**Relevant best practices**: [Worker deployment and performance](/best-practices/worker), appropriate timeouts, managing Worker shutdown, idempotency

- [Worker shutdown](/encyclopedia/workers/worker-shutdown)

### Kill all Workers, then restart them

**What to test**

Abruptly terminate all Workers processing a Task Queue, then restart them.

**Why it matters**

- Validates at-least-once execution semantics.
- Ensures Activities are idempotent and Workflows replay cleanly.
- Validates Task timeouts and retries and that Workers can finish business processes.

**How to run this**

Depending on execution environment:

- **Kubernetes**: Set pod count to zero:
  ```bash
  kubectl scale deployment <deployment-name> --replicas=0 -n <namespace>
  kubectl scale deployment <deployment-name> --replicas=3 -n <namespace>
  ```
- **Azure App Service**:
  ```bash
  az webapp restart --name <app-name> --resource-group <resource-group>
  ```

**Things to watch**

- Duplicate/improper Activity results
- Workflow failures
- Workflow backlog growth and drain time

### Frequent Worker restart

**What to test**

Periodically restart a fixed or random percentage (for example, 20-30%) of your Worker fleet every few minutes.

**Why it matters**

- Mimics failure modes where Workers restart due to high CPU utilization and out-of-memory errors from compute-intensive logic in Activities.
- Ensures Temporal invalidates specific Sticky Task Queues and reschedules the task to the associated non-Sticky Task Queue.

**How to run this**

- **Kubernetes**: Build a script using `kubectl` to randomly delete pods in a loop.
- **Chaos Mesh**: [Simulate pod faults](https://chaos-mesh.org/docs/simulate-pod-chaos-on-kubernetes/).
- **App Services**: Scale down and up again.

**Things to watch**

- Replay latency
- Drop in Workflow and Activity completion
- Duplicate/improper Activity results
- Workflow failures
- Workflow backlog growth and drain time

## Load testing

### Pre-load test setup: expectations for success

1. Have SDK metrics accessible (not just the Cloud metrics)
2. Understand and predict what you should see from these metrics:
   - Rate limiting ([`temporal_cloud_v1_total_action_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_total_action_throttled_count) and [`temporal_cloud_v1_service_request_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_request_throttled_count))
   - Workflow failures (`temporal_cloud_v1_workflow_failed_count`)
   - Workflow execution time (`workflow_endtoend_latency`)
   - High Cloud latency (`temporal_cloud_v1_service_latency_p95`)
   - [Worker metrics](/develop/worker-performance) (`workflow_task_schedule_to_start_latency` and `activity_schedule_to_start_latency`)
3. Determine throughput requirements ahead of time. Work with your account team to match that to the Namespace capacity to avoid rate limiting. Capacity increases are done via Temporal support and can be requested for a load test (short-term).
4. Automate how you run the load test so you can start and stop it at will. How will you clear Workflow Executions that are just temporary?
5. What does "success" look like for this test? Be specific with metrics and numbers stated in business terms.

### Validate downstream load capacity

**Relevant best practices**: Idempotent Activities, bounded retries, appropriate timeouts and retry policies, understand behavior when limits are reached

**What to test**

- Schedule a large number of Actions and Requests by starting many Workflows
- Increase the number until you start overloading downstream systems

**Why it matters**

Validates behavior of Temporal application and application dependencies under high load.

**How to run this**

Start Workflows at a rate to surpass throughput limits. Example: [temporal-ratelimit-tester-go](https://github.com/joshmsmith/temporal-ratelimit-tester-go)

**Things to watch**

- Downstream service error rates (HTTP 5xx, database errors)
- Increased downstream service latency and saturation metrics
- Activity failure rates, specifically classifying between retryable and non-retryable errors
- Activity retry and backoff behavior against the overloaded system
- Workflow backlog growth and drain time
- Correctness and consistency of data (ensuring Activity idempotency holds under duress)
- Worker CPU/memory utilization

### Validate rate limiting behavior

**Relevant best practices**: [Manage Namespace capacity limits](/best-practices/managing-aps-limits), understand behavior when limits are reached

**What to test**

- Schedule a large number of Actions and Requests by starting many Workflows
- Increase the number until you get rate limited and trigger [`temporal_cloud_v1_total_action_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_total_action_throttled_count) or [`temporal_cloud_v1_service_request_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_request_throttled_count)

**Why it matters**

Validates behavior of Cloud service under high load: "In Temporal Cloud, the effect of rate limiting is increased latency, not lost work. Workers might take longer to complete Workflows."

**How to run this**

1. (Optional) Decrease a test Namespace's rate limits to make it easier to hit limits
2. Calculate current APS at current throughput (in production)
3. Calculate Workflow throughput needed to surpass limits
4. Start Workflows at a rate to surpass throughput limits using [temporal-ratelimit-tester-go](https://github.com/joshmsmith/temporal-ratelimit-tester-go)

**Things to watch**

- Worker behavior when rate limited
- Client behavior when rate limited
- Temporal request and long_request failure rates
- Workflow success rates
- Workflow latency rates

## Failover and availability

**Relevant best practices**: Use [High Availability features](/cloud/high-availability) for critical workloads.

- [High Availability monitoring](/cloud/high-availability/monitoring)

### Test region failover

**What to test**

Trigger a [High Availability](/cloud/high-availability) failover event for a Namespace.

**Why it matters**

- Real outages are messy and rarely isolated.
- Ensures your operational playbooks and automation are resilient.
- Validates Worker and Namespace failover behavior.

**How to run this**

Execute a manual failover per the [manual failovers documentation](/cloud/high-availability/failovers/manage#trigger-failover).

**Things to watch**

- Namespace availability
- Client and Worker connectivity to failover region
- Workflow Task reassignments
- Human-in-the-loop recovery steps

## Dependency and downstream testing

### Break the things your Workflows call

**What to test**

Intentionally break or degrade downstream dependencies used by Activities:

- Make databases read-only or unavailable
- Inject high latency or error rates into external APIs
- Throttle or pause message queues and event streams

**Why it matters**

- Temporal guarantees Workflow durability, not dependency availability.
- Validates that Activities are retryable, idempotent, and correctly timeout-bounded.
- Ensures Workflows make forward progress instead of livelocking on broken dependencies.

**Things to watch**

- Activity retry and backoff behavior
- Heartbeat effectiveness for long-running Activities
- Database connection exhaustion and retry storms
- API timeouts vs Activity timeouts
- Whether failures propagate as Signals, compensations, or Workflow-level errors

**Anti-patterns this reveals**

- Non-idempotent Activities
- Infinite retries without circuit breaking
- Using Workflow logic to "wait out" broken dependencies

## Deployment and code-level testing

### Deploy a Workflow change with versioning

**Relevant best practices**: Implement a versioning strategy.

- [Workflow Versioning Strategies - Developer Corner](https://community.temporal.io/t/workflow-versioning-strategies/6911)
- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning)
- Replay Testing: see SDK documentation for details

**What to test**

- Deploy Workflow code that would introduce non-deterministic errors (NDEs) but use a versioning strategy to deploy successfully
- Validate Workflow success and clear the backlog of tasks

**Why it matters**

- Unplanned NDEs can be a painful surprise
- Tests versioning strategy and patching discipline to build production confidence

**Things to watch**

- Workflow Task failure reasons
- Effectiveness of versioning and patching patterns

### Deploy a version that causes NDEs, then recover

**Relevant best practices**: Implement a versioning strategy.

- [Workflow Versioning Strategies - Developer Corner](https://community.temporal.io/t/workflow-versioning-strategies/6911)
- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning)
- Replay Testing: see SDK documentation for details

**What to test**

- Deploy Workflow code that introduces non-deterministic errors (NDEs)
- Attempt rollback to a known-good version, or apply versioning strategies to apply the new changes successfully
- Clear or recover the backlog of tasks

**Why it matters**

- Unplanned NDEs can be a painful surprise
- Tests versioning strategy, patching discipline, and recovery tooling

**Things to watch**

- Workflow Task failure reasons
- Backlog growth and drain time
- Effectiveness of versioning and patching patterns

## Network-level testing

The scenarios below are most relevant if your infrastructure introduces network boundaries (such as firewalls, VPNs, or network policies) between Workers and the Temporal service, or if you need to verify application behavior during prolonged disconnections.

**Relevant best practices**: Idempotent Activities, bounded retries, appropriate timeouts

- [Activity timeouts](https://temporal.io/blog/activity-timeouts)
- [Idempotency and durable execution](https://temporal.io/blog/idempotency-and-durable-execution)

### Remove network connectivity to a Namespace

**What to test**

Temporarily block all network access between Workers and the Temporal service for a Namespace.

**Why it matters**

- Validates Worker retry behavior, Sticky Task Queue behavior, Worker recovery performance, backoff policies, and Workflow replay determinism under prolonged disconnection.
- Ensures no assumptions are made about "always-on" connectivity.

**Temporal failure modes exercised**

- Workflow Task timeouts vs retries
- Activity retry semantics
- Replay correctness after long gaps

**How to run this**

- **Kubernetes**: Apply a NetworkPolicy that denies egress from Worker pods to the Temporal APIs.
- **[ToxiProxy](https://github.com/Shopify/toxiproxy)**: Proves your application doesn't have single points of failure.
- **Chaos Mesh / Litmus**: NetworkChaos with full packet drop.
- **Local testing**: Block ports with iptables or firewall rules.

**Things to watch**

- Workflow failures (replay, timeout)
- Workflow Task retries
- Activity failures, classifications (retryable vs non-retryable)
- Worker CPU usage during reconnect storms

## Observability checklist

Before (and during) testing, ensure visibility into:

- Workflow Task and Activity failure rates
- Throughput limits and usage
- Workflow and Activity end-to-end latencies
- Task latency and backlog depth
- Event History size and event counts
- Worker CPU, memory, and restart counts
- gRPC error codes
- Retry behavior

## Game day runbook

Use this checklist when running tests during a scheduled game day or real incident simulation.

### Before you start

- Make sure people know you're testing and what scenarios you're trying
  - Let the teams that support the APIs you're calling know you're testing
  - Reach out to the Temporal Cloud Support and Account teams to coordinate
- Dashboards for SDK and Cloud metrics
  - Task latency, backlog depth, Workflow failures, Activity failures
- Alerts muted or routed appropriately
- Known-good deployment artifact available
- Rollback and scale controls verified

### During testing

- Introduce *one variable at a time*
- Record start/stop times of each experiment
- Capture screenshots or logs of unexpected behavior
- Track backlog growth and drain rate

### Recovery validation

- Workflows resume without manual intervention
- No permanent Workflow Task failures (unless intentional)
- Activity retries behave as expected
- Backlogs drain in predictable time

### After action review

- Identify unclear alerts or missing metrics/alerts
- Update retry, timeout, or versioning policies
- Document surprises and operational debt

## Summary

Pre-production testing with Temporal is about more than proving durability - it's about proving *operability under stress*.
You want to go through the exercise and know what to do before you go to production and have to do it for real.

If your system survives:

- Connectivity issues
- Repeated failovers
- Greater than expected load
- Mass Worker churn

...then you can have confidence it's ready for many kinds of production chaos.

---

# Security controls for Temporal Cloud

Source: https://docs.temporal.io/best-practices/security-controls

> Best practices for implementing and managing security controls in Temporal Cloud environments.

Temporal Cloud provides the capabilities of self-hosted Temporal as a managed service; it does not manage your applications or workers. Applications and services written using Temporal SDKs still run in your compute environment, and you have full control over how you secure your applications and services.

These best practices ensure your Temporal Cloud environment adheres to the security guidelines recommended by our team. You can also learn more about our security practices, compliance posture, and subscribe for vulnerability (CVE) updates at https://trust.temporal.io/. 

If you have questions, contact our security team at security@temporal.io.

> **💡 Tip:**
>
> **Stay updated on Temporal security advisories:** 
> Subscribe to Temporal’s security updates on the [Temporal Trust Portal](https://trust.temporal.io/) so you are aware of any patches or CVEs. While Temporal Cloud server-side updates are handled by the vendor, your Temporal SDKs (in application code) should be kept up-to-date.
>

## Identity and access management

Strong identity management in Temporal Cloud is crucial for ensuring secure access for your Temporal account. It’s critical that only authorized users and services can access your Temporal Cloud account and that each has the minimum necessary permissions needed for their role. 

### Enable [SAML single sign-on](/cloud/manage-access/saml) (SSO) for user access

Integrate Temporal Cloud with your organization's identity provider via SAML 2.0 for centralized authentication. SSO allows you to enforce your corporate login policies (such as MFA and password complexity). When you configure SAML with Temporal Cloud, you can disable social logins (that is, Microsoft, Google) by opening a support ticket. 

### Use least-privilege roles for Temporal Cloud users

Temporal Cloud provides [preconfigured account-level roles](/cloud/manage-access/users) (Account Owner, Finance Admin, Global Admin, Developer, Read-Only) and Namespace-level permissions. Assign users the lowest level of access they need. For example, give developers access only to the Namespaces they work on, and use read-only roles for auditors or reviewers. Regularly review user roles and remove or downgrade accounts that are no longer needed

### Use SCIM or automated user provisioning

When applicable, use [SCIM](/cloud/manage-access/scim) or the Temporal Cloud user management API to automate adding and removing user accounts. This ensures timely removal of access when people change roles or leave the organization.

### Use Service Accounts for automation

For non-human access (CI/CD pipelines, backend services), use [Temporal Cloud Service Accounts](/cloud/manage-access/service-accounts) instead of shared user logins. Service Accounts are machine identities that can be granted specific permissions without ties to an individual. Create separate Service Accounts with unique API keys for different applications or microservices, and apply least privilege to each (for example, a service account that only has access to one Namespace).

### Restrict Cloud UI session lifetimes

Strict UI Session Mode is an optional account-level setting for organizations that require shorter sessions for users
authenticated through [SAML single sign-on (SSO)](/cloud/manage-access/saml).

When enabled, Strict UI Session Mode:

- Signs users out after 15 minutes of inactivity, instead of 12 hours.
- Limits the maximum session duration to 12 hours, instead of 7 days, even if the user remains active.

When either limit is reached, the user must start a new Cloud UI session through the account's SAML identity provider.
Whether the identity provider prompts the user to enter credentials or complete MFA depends on the identity provider's
own session policy.

These predictable session limits can help organizations enforce internal security policies and complete security and
compliance reviews. Enabling Strict UI Session Mode does not by itself guarantee compliance with a particular standard
or regulation.

#### Prerequisites

Before enabling Strict UI Session Mode, [SAML SSO](/cloud/manage-access/saml) must be configured and active for the
account.

Strict UI Session Mode applies only to Cloud UI sessions authenticated through SAML. It does not affect other
authenticated interactions with Temporal, including:

- Users who sign in through Google, Microsoft, or email and password
- Workers or SDK clients
- API Keys or Service Accounts
- Temporal CLI sessions
- mTLS connections

#### Enable strict UI session mode

[Create a support ticket](/evaluate/cloud/support#support-ticket) and provide your
[Temporal Cloud Account Id](/cloud/namespaces#temporal-cloud-account-id). Ask Dev Success to enable Strict UI Session
Mode for the account.

Strict UI Session Mode applies to the entire account. It cannot be configured for individual users, groups, or roles.
The 15-minute inactivity limit and 12-hour maximum session duration are fixed.

The setting applies when a user starts a new session. Users with an active session must sign out and start a new session
before the shorter limits take effect.

#### Understand how inactivity is measured

Only Cloud UI activity that sends a request to Temporal Cloud resets the inactivity timer (for example create a Namespace). Mouse movement, scrolling, or reading information that has already loaded does not reset the timer. 

The Cloud UI does not display a warning or countdown before the session expires. The 12-hour maximum applies regardless of activity.

To disable Strict UI Session Mode, [create a support ticket](/evaluate/cloud/support#support-ticket). Active sessions retain the
shorter limits until users sign out and start new sessions.

## Secure application authentication and API access

Clients interact with the Temporal Service to initiate and manage Workflows, while Workers execute the business logic defined in Workflows and Activities in your own environment. 

A crucial aspect of strengthening your usage involves securing these interactions. Temporal Cloud offers two authentication methods for your applications: mutual TLS certificates and API keys.

### Use mutual TLS (mTLS) for comprehensive security

Temporal Cloud secures its gRPC endpoint per Namespace via mutual TLS. This means you provide a Certificate Authority (CA) certificate for your Namespace, and all your Temporal clients/workers must present client certificates signed by that CA. 

We recommend you enable mTLS for strong identity assurance of clients; it ensures only systems holding a valid certificate (issued by your trusted CA) can connect. Generate a private key and CA certificate (or use your enterprise CA) and upload the CA to Temporal Cloud. Do not share these certificates and associated keys beyond the authorized services.

### Proactively manage and rotate certificates

Track the expiration dates of your client and [Certificate Authority certificates](/cloud/certificates). Temporal Cloud trusts the uploaded CA; if it expires, all client authorizations will fail. Establish and automate a certificate rotation schedule (for example, rotate client certificates quarterly and CA certificates annually, well before expiry). Temporal supports uploading a new CA certificate alongside the old one, so rotation doesn't interrupt client connections. Always test new certificates in a staging environment if possible.

### Handle API keys with strict care

Temporal Cloud API keys are an alternative to mTLS for authentication of SDKs, CLI, and automation. If you opt for API keys, handle them with strict care by enacting the following practices: 
- Keep them secret: store in a secrets manager, never in code or Git.
- Rotate at least every 90 days: Temporal lets you create a new key, swap it in, then delete the old one.
- One key per service/person: no sharing or reuse.
- Monitor usage & revoke on anomalies: feed Temporal audit logs to SIEM.
- Optional: Admins can disable all user API keys if your policy is “mTLS only.”

## Network configuration and isolation

Although Temporal Cloud is a SaaS offering, you retain control over its networking configurations, allowing for tailored security measures. By minimizing public internet exposure and segmenting Temporal workflows into suitable network zones, you can significantly bolster security and reduce potential vulnerabilities. This approach ensures that your workflows are isolated and protected within your defined network boundaries, even while using a cloud-based service.

### Use private connectivity

Temporal Cloud supports private connectivity options such as [AWS PrivateLink](/cloud/connectivity/aws-connectivity) and [Google Cloud Private Service Connect](/cloud/connectivity/gcp-connectivity). If your infrastructure is in AWS or GCP, configure a PrivateLink/PSC endpoint for Temporal Cloud. This allows your workers and applications to reach Temporal Cloud over a private network path, avoiding traversal of the public internet. Private connectivity reduces the surface for man-in-the-middle attacks and can meet stringent network security policies.

### Separate environments by Namespace

Use [Temporal Namespaces](/best-practices/managing-namespace#naming-conventions) to isolate workflows for different environments or teams (for example, development, staging, production). Each Namespace is logically segregated and cannot interact with others by default, providing a security boundary. 

Ensure that your production Namespace uses stricter network controls (for example, only accessible from the prod network) and that credentials for it are separate from non-prod Namespaces. This limits the impact of any compromise in a lower environment, and as workflow data is only visible to users with access to that Namespace, separating environments by Namespace also enforces data-visibility boundaries.

## Data protection and encryption

Temporal's data encryption capabilities ensure the security and confidentiality of your Workflows and provide protection without compromising performance. Protecting the data that you send to and store in Temporal Cloud is a joint responsibility. Temporal Cloud already encrypts all data at rest on the server side, but you can add additional layers of encryption and control. 

### Enable client-side encryption for Workflow data

Temporal provides an optional [data conversion framework](/dataconversion) (Data Converter) and Payload Codec interface that you can use to encrypt data before it reaches Temporal Cloud. You implement, deploy, and operate the codec and manage encryption keys. Because encryption keys stay under your control, you are responsible for key generation, secure storage, rotation, and versioning. Implementing this involves developing a custom Payload Codec in your Temporal SDK and optionally (if you need to inspect decrypted payloads in the Web UI or CLI) deploying a dedicated Codec Server.

Alternatively, [Temporal Proxy](/production-deployment/temporal-proxy) encrypts payloads in your environment before
forwarding them to Temporal Cloud, without changing Worker or Client code. See
[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads) for the encryption design, configuration, and
coverage limits.

### Encode Workflow failure details with a Failure Converter

Temporal’s default behavior copies error messages and call stacks as plain text, and this text is directly accessible in the Message field of Workflow Executions.

If your failure messages and stack traces contain sensitive information, it is recommended that you configure the [Failure Converter](/failure-converter) to encrypt the error information. This would encrypt the `message` and `stack_trace` fields in the payloads.

### Use Namespace data retention policies

Temporal Cloud Namespace has a [Retention Period](/temporal-service/temporal-server#retention-period) setting for workflow histories (1 to 90 days). Set an appropriate retention period to balance operational needs with security. Shorter retention means completed workflow data (history, payloads) is purged sooner, reducing the amount of sensitive data stored in the cloud at any time. Document your retention choices to align with your company’s data retention policies and regulatory requirements. For retention periods over 90 days, use [Workflow History Export](/cloud/export) to export closed Workflow Histories to your own S3 bucket, GCS bucket, or Azure Blob Storage container.

## Availability and disaster recovery

Temporal Cloud’s platform is engineered for fault-tolerance out of the box, but you determine which Namespaces merit the very highest availability guarantees. Use the table below to decide when to turn on different High Availability models and how to operationalize them.

| Namespace scope | Use Case | Recovery Time Objective (RTO) | Recovery Point Objective (RPO) |
|-----------------|----------|--------------------------------|--------------------------------|
| **Single-Region** | **If your application is built for one region and does not have stringent high-availability or disaster recovery requirements.** | ≤ 8 hours | ≤ 8 hours |
| **Same-Region Replication** | **If you want higher availability but your application is designed for a single region or if cross region latency doesn’t meet SLAs for application** | ≤ 20 minutes | Near-zero (≈ seconds) |
| **Multi-Region Replication** | **If a disruption of your workflow will cause loss of revenue, poor end-user experience, or issues with regulatory compliance.** | ≤ 20 minutes | Near-zero (≈ seconds) |
| **Multi-Cloud Replication** | **If you need the highest level of disaster tolerance, protecting against outages of an entire cloud provider (for example, AWS or GCP)** | ≤ 20 minutes | Near-zero (≈ seconds) |

For the contractual uptime commitment for each deployment type, see the [Temporal Cloud SLA](https://temporal.io/sla).

### Identify availability-sensitive Namespaces

Run a business-impact analysis to flag workflows where a regional outage would cause significant customer, revenue, or safety impact. Identify Namespaces that are availability-sensitive where a regional outage may have outsized business impacts such as revenue loss, poor customer experience, or inability to meet legal obligations.

### Enable High Availability for business-critical use cases

For many organizations, ensuring High Availability (HA) is required because of strict uptime requirements, compliance, and regulatory needs. 

For these critical use cases, enable High Availability features for specific namespaces for a [higher contractual SLA](https://temporal.io/sla). When choosing between [same-region, multi-region, and multi-cloud replication](/cloud/high-availability), it is recommended to use multi-region/multi-cloud replication to distribute your dependencies across regions. Using physically separated regions improves the fault tolerance of your application.

By default, Temporal Cloud provides a [contractual SLA](https://temporal.io/sla) against service errors for all namespaces. 

Note: [enabling HA features for namespaces will 2x the consumption cost.](/cloud/pricing#high-availability-features)

---

# Worker deployment and performance

Source: https://docs.temporal.io/best-practices/worker

> Best practices for deploying and optimizing Temporal Workers for performance and reliability.

This document outlines best practices for deploying and optimizing Workers to ensure high performance, reliability, and
scalability. It covers deployment strategies, scaling techniques, tuning recommendations, and monitoring approaches to
help you get the most out of your Temporal Workers.

We also provide a reference application, the Order Management System (OMS), that demonstrates the deployment best
practices in action. You can find the OMS codebase on
[GitHub](https://github.com/temporalio/reference-app-orders-go/tree/main/docs).

## Quick checklist

A good Worker deployment strategy involves several considerations. We
provide a quick checklist to help you get started. Before deploying Workers to production, ensure you address the
following. Follow the links to the relevant sections for more details.

- **[Configure each Worker appropriately](#actively-tune-worker-options-instead-of-relying-on-defaults)**: Actively tune
  Worker options based on your code, language runtime limits, and system resource constraints. Don't rely on defaults,
  which are designed for ease in development and testing, but not optimal for production environments.
- **[Run at least two Workers per Task Queue](#run-at-least-two-workers-per-task-queue)**: For shared, always-on queues,
  put them on separate hosts so that one Worker crashing or restarting doesn't stop Task processing.
- **[Deploy enough Workers](#interpret-metrics-as-a-whole)**: Monitor performance metrics and scale Workers to meet your
  workload requirements, or [autoscale them](#scale-workers-automatically) on Kubernetes or with Serverless Workers.
- **[Separate Task Queues logically](#separate-task-queues-logically)**: Size and split work across Task types
  (Activities and Workflows) and Task Queues based on workload characteristics.
- **[Version Workers for safe deployments](#use-worker-versioning-to-safely-deploy-new-workflow-code)**: Ensure you can
  deploy new Workflow code without breaking running Executions.
- **Run benchmarks**: Test your configuration under realistic load to confirm limits and settings are appropriate for
  your environment.

## Deployment and lifecycle management

Well-designed Worker deployment ensures resilience, observability, and maintainability. A Worker should be treated as a
long-running service that can be deployed, upgraded, and scaled in a controlled way.

### Package and configure Workers for flexibility

Workers should be artifacts produced by a CI/CD pipeline. Inject all required parameters for connecting to Temporal
Cloud or a self-hosted Temporal Service at runtime via environment variables, configuration files, or command-line
parameters. This allows for more granularity, easier testability, easier upgrades, scalability, and isolation of
Workers.

In the order management reference app, Workers are packaged as Docker images with configuration provided via environment
variables and mounted configuration files. The following Dockerfile uses a multi-stage build to create a minimal,
production-ready Worker image:

[Dockerfile](https://github.com/temporalio/reference-app-orders-go/blob/main/Dockerfile)
```Dockerfile
FROM golang:1.25.4 AS oms-builder

WORKDIR /usr/src/oms

COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go mod download

COPY app ./app
COPY cmd ./cmd

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -v -o /usr/local/bin/oms ./cmd/oms

FROM busybox AS oms-worker
```

This Dockerfile uses a multi-stage build pattern with two stages:

1. `oms-builder` stage: compiles the Worker binary.

   1. Copies dependency files and downloads dependencies using BuildKit cache mounts to speed up subsequent builds.
   2. Copies the application code and builds a statically linked binary that doesn't require external libraries at
      runtime.

2. `oms-worker` stage: creates a minimal final image.

   1. Copies only the compiled binary from the `oms-builder` stage.
   2. Sets the entrypoint to run the Worker process.

The entrypoint `oms worker` starts the Worker process, which reads configuration from environment variables at runtime.
For example, the
[Billing Worker deployment in Kubernetes](https://github.com/temporalio/reference-app-orders-go/blob/main/deployments/k8s/billing-worker-deployment.yaml)
uses environment variables to configure the Worker:

[deployments/k8s/billing-worker-deployment.yaml](https://github.com/temporalio/reference-app-orders-go/blob/main/deployments/k8s/billing-worker-deployment.yaml)
```yaml
# ...
    spec:
      containers:
        - args:
            - -k
            - supersecretkey
            - -s
            - billing
          env:
            - name: FRAUD_API_URL
              value: http://billing-api:8084
            - name: TEMPORAL_ADDRESS
              value: temporal-frontend.temporal:7233
          image: ghcr.io/temporalio/reference-app-orders-go-worker:latest
          name: billing-worker
          imagePullPolicy: Always
      enableServiceLinks: false
```

### Run at least two Workers per Task Queue

For each shared, always-on Task Queue in production, run at least two Worker Processes, each on a separate host.

A single Worker is a single point of failure. If it crashes, is redeployed, or loses its host, Tasks wait in the Task
Queue until it comes back. With a second Worker polling the same Task Queue, that Worker
[picks up the Tasks and replays the Event History](/develop/worker-tuning-reference#how-worker-failure-recovery-works),
and the Workflow Executions keep making progress.

This is a minimum for availability, separate from the capacity you size using the
[metrics below](#interpret-metrics-as-a-whole). Scale beyond two based on Schedule-to-Start latency and slot usage. When
you scale down, don't go below two.

The minimum has three qualifications:

- **Count Workers per Worker Deployment Version.** A [pinned](/worker-versioning#pinned) Workflow completes on a single
  Worker Deployment Version, so another version's Workers can't take over its Tasks. Keep at least two Workers on
  every version that still serves Workflow Executions, including older versions with pinned Workflows.
- **Host-specific Task Queues are an exception.** Don't add a second host to a queue that routes Activities to the host
  holding their local files or state, because the Tasks could land on a host without them. See
  [Worker-specific Task Queues](/design-patterns/worker-specific-taskqueue).
- **Serverless Workers and scale-to-zero are exceptions.** Temporal starts
  [Serverless Workers](/serverless-workers) when Tasks arrive, and a KEDA-scaled fleet can drop to zero when idle. In
  both cases, Tasks can wait while Workers start. See [Scale Workers automatically](#scale-workers-automatically).

Two more practices help the minimum hold up:

- **Spread Workers across failure domains.** Place Workers on different hosts and, in the cloud, across multiple
  Availability Zones. Temporal Cloud Namespaces already span three Availability Zones, so Workers in a single zone are
  the weak point. See [Availability Zone outage](/cloud/rpo-rto#availability-zone-outage).
- **Roll out one Worker at a time.** Restarting every Worker at once empties their
  [Sticky Execution](/sticky-execution) caches, so each Workflow Execution replays from its Event History when the new
  Workers start.

### Separate Task Queues logically

Use separate Task Queues for distinct workloads. This isolation allows you to control rate limiting, prioritize certain
workloads, and prevent one workload from starving another. Each Task Queue needs its own
[two or more Workers](#run-at-least-two-workers-per-task-queue).

In the order management reference app, each microservice has its own Task Queue. For example, the Billing Worker polls
the `billing` Task Queue, while the Order Worker polls the `order` Task Queue. This separation allows each service to
scale independently based on its workload.

![Diagram showing separate Task Queues for different Workers](https://docs.temporal.io/diagrams/worker-best-practice-oms-architecture.png)

The following code snippet shows how the Billing Worker is set up to poll its Task Queue. The default value for
`TaskQueue` is a constant defined in the `api.go` configuration file and is set to `billing`.

Since Task Queues are created dynamically when first used, a mismatch between the Client and Worker Task Queue names
does not result in an error. Instead, it creates two different Task Queues, and the Worker never receives Tasks from the
Temporal Service because it's polling the wrong queue. Define the Task Queue name as a constant that both the Client and
Worker reference to avoid this issue.

[app/billing/worker.go](https://github.com/temporalio/reference-app-orders-go/blob/main/app/billing/worker.go)
```go
// ...
// RunWorker runs a Workflow and Activity worker for the Billing system.
func RunWorker(ctx context.Context, config config.AppConfig, client client.Client) error {
	w := worker.New(client, TaskQueue, worker.Options{
		MaxConcurrentWorkflowTaskPollers: 8,
		MaxConcurrentActivityTaskPollers: 8,
	})

	w.RegisterWorkflow(Charge)
	w.RegisterActivity(&Activities{FraudCheckURL: config.FraudURL})

	return w.Run(temporalutil.WorkerInterruptFromContext(ctx))
}
```

### Use Worker Versioning to safely deploy new Workflow code

Use Worker Versioning to deploy new Workflow code without breaking running Executions. Worker Versioning lets you map
each Workflow Execution to a specific Worker Deployment Version identified by a build ID, which guarantees that pinned
Workflows always run on the same Worker version where they started.

For most teams, Worker Versioning should be the default approach for evolving Workflow code in production. It gives you
safer rollouts, clearer rollback behavior, and a cleaner operational model than maintaining multiple code paths inside a
Workflow definition.

To learn more about versioning Workflows, see the
[Worker Versioning](/production-deployment/worker-deployments/worker-versioning) guide or take our
[Worker Versioning course](https://learn.temporal.io/courses/worker_versioning/).

> **💡 Tip:**
>
> If your deployment environment cannot yet support Worker Versioning, you can use [Patching](/patching) to introduce
> changes to your Workflow code without breaking running Executions. Treat patching as a fallback for environments that
> cannot adopt versioned worker deployments yet, not as the default recommendation.
>

### Manage Event History growth

If a Worker goes offline and another Worker picks up the same Workflow Execution, the new Worker must replay the
existing Event History to resume the Workflow Execution. If the Event History is too large or has too many Events,
replay affects the performance of the new Worker and may even cause timeout errors well before the hard limit of 51,200
Events is reached.

We recommend not exceeding a few thousand Events in a single Workflow Execution. The best way to handle Event History
growth is to use the [Continue-As-New](/workflow-execution/continue-as-new) mechanism to continue under a new Workflow
Execution with a new Event History, repeating this process as you approach the limits again.

All Temporal SDKs provide functions to suggest when to use [Continue-As-New](/workflow-execution/continue-as-new). For
example, the Python SDK has the
[`is_continue_as_new_suggested()`](https://python.temporal.io/temporalio.workflow.html#is_continue_as_new_suggested)
function that returns a `bool` indicating whether to use Continue-As-New.

In addition to the number of Events, monitor the size of the Event History. Input parameters and output values of both
Workflows and Activities are stored in the Event History. Storing large amounts of data can lead to performance
problems, so the Temporal Service limits both the size of individual payloads and the total Event History size. A
Workflow Execution may be terminated if any single payload exceeds 2 MB or if the entire Event History exceeds 50 MB.

To avoid hitting these limits, avoid passing large amounts of data into and out of Workflows and Activities. A common
way to reduce payload and Event History size is the
[Claim Check](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern) pattern, widely used with
messaging systems such as Apache Kafka. Instead of passing large data into your function, store that data external to
Temporal in a database or file system. Pass an identifier for the data, such as a primary key or path, into the function
and use an Activity to retrieve it as needed. If your Activity produces large output, use a similar approach: write the
data to an external system and return an identifier that can be used to retrieve it later.

## Scaling, monitoring, and tuning

Scaling and tuning are critical to Worker performance and cost efficiency. The goal is to balance concurrency,
throughput, and resource utilization while maintaining low Task latency.

### Actively tune Worker options instead of relying on defaults

Default Worker settings are designed to work across a wide range of use cases primarily for ease in development and
testing. In production environments, your Workflow complexity, Activity duration, payload sizes, and infrastructure
constraints all influence optimal Worker configuration. Actively tuning your Workers ensures they perform well under
your specific workload conditions.

To get started, focus on these key Worker options:

- **Task slots**: Limits how many Tasks execute concurrently. Set based on your Worker's CPU, memory, and the resource
  demands of your code. You can choose different types of Slot Suppliers or implement a custom Slot Supplier to control
  how Task slots are assigned to different Task types. Refer to
  [Slot Suppliers](/develop/worker-performance#slot-suppliers) for more details.
- **Sticky cache size**: Controls the size of the sticky cache for Workflow Executions. Larger caches reduce replay
  overhead but consume more memory. Refer to [Workflow Cache Tuning](/develop/worker-performance/workflow-cache)
  for more details.
- **Poller counts**: Controls the number of pollers for Tasks. We recommend you use the Poller Autoscaling feature to
  automatically adjust the number of pollers based on your workload. Refer to
  [Configuring Poller Options](/develop/worker-performance/configuration#configuring-poller-options) for more details.

Use the metrics listed in [Interpret metrics as a whole](#interpret-metrics-as-a-whole) to guide your tuning decisions.

### Interpret metrics as a whole

No single metric tells the full story. The following are some of the most useful Worker-related metrics to monitor. We
recommend having all metrics listed below on your Worker monitoring dashboard. When you observe anomalies, correlate
across multiple metrics to identify root causes.

- Worker CPU and memory utilization
- `workflow_task_schedule_to_start_latency` and `activity_task_schedule_to_start_latency`
- `worker_task_slots_available`
- `temporal_long_request_failure`, `temporal_request_failure`, `temporal_long_request_latency`, and
  `temporal_request_latency`

For example, Schedule-to-Start latency measures how long a Task waits in the queue before a Worker starts it. High
latency means your Workers or pollers can’t keep up with incoming Tasks, but the root cause depends on your resource
metrics:

- High Schedule-to-Start latency and high CPU/memory: Workers are saturated. Scale up your Workers or add more Workers.
  It's also possible your Workers are blocked on Activities. Refer to
  [Troubleshooting - Depletion of Activity Task Slots](/troubleshooting/performance-bottlenecks#depletion-of-temporal_worker_task_slots_available-for-activityworker)
  for guidance.
- High Schedule-to-Start latency and low CPU/memory: Workers are underutilized. Increase the number of pollers, executor
  slots, or both. If this is accompanied by high `temporal_long_request_latency` or `temporal_long_request_failure`,
  your Workers are struggling to reach the Temporal Service. Refer to
  [Troubleshooting - Long Request Latency](/troubleshooting/performance-bottlenecks#high-temporal_long_request_failure)
  for guidance.
- Low Schedule-to-Start latency and low CPU/memory: Depending on your workload, this could be normal. If you are
  consistently seeing low memory usage and low CPU usage, you may be over-provisioning your Workers and can consider
  scaling down.

Refer to [Intro to Worker Tuning](https://temporal.io/blog/an-introduction-to-worker-tuning) for more details and
examples on how to interpret Worker metrics.

### Scale Workers automatically

Instead of adjusting the Worker count by hand, scale it from Task Queue backlog and slot utilization:

- **On Kubernetes**, the [Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller#autoscaling-versioned-workers)
  attaches a Kubernetes HPA or KEDA `ScaledObject` to each Worker Deployment Version, so each version scales
  separately.
- **With Serverless Workers**, Temporal starts and stops Workers in response to Tasks on the Task Queue, so you don't
  provision or scale a fleet. AWS Lambda and GCP Cloud Run support is in Public Preview. See
  [Deploy Serverless Workers](/production-deployment/worker-deployments/serverless-workers).

For an always-on fleet, set the autoscaler's minimum replica count to at least two so the
[two-Worker minimum](#run-at-least-two-workers-per-task-queue) still holds when the Task Queue is idle. If you
deliberately scale to zero, for example with KEDA or Serverless Workers, Tasks can wait while Workers start, so only do
that for queues that tolerate the delay.

### Optimize Worker cache

Workers keep a cache of Workflow Executions to improve performance by reducing replay overhead. However, larger caches
consume more memory. The `temporal_sticky_cache_size` tracks the size of the cache. If you observe high memory usage for
your Workers and high `temporal_sticky_cache_size`, you can be reasonably sure the cache is contributing to memory
pressure.

Having a high `temporal_sticky_cache_size` by itself isn't necessarily an issue, but if your Workers are memory-bound,
consider reducing the cache size to allow more concurrent executions. We recommend you experiment with different cache
sizes in a staging environment to find the optimal setting for your Workflows. Refer to
[Troubleshooting - Caching](/troubleshooting/performance-bottlenecks#caching) for more details on how to interpret
the different cache-related metrics.

### Manage scale-down safely

Before shutting down a Worker, verify that it does not have too many active Tasks. This is especially relevant if your
Workers are handling long-running, expensive Activities.

If `worker_task_slots_available` is at or near zero, the Worker is running active Tasks. Shutting it down could trigger
expensive retries or timeouts for long-running Activities. Use
[Graceful Shutdowns](/encyclopedia/workers/worker-shutdown#graceful-shutdown) to allow the Worker to complete its
current Tasks before shutting down. All SDKs provide a way to configure Graceful Shutdowns. For example, the Go SDK has
the [`WorkerStopTimeout` option](https://pkg.go.dev/go.temporal.io/sdk@v1.38.0/internal#WorkerOptions) that lets you
configure how long the Worker has to complete its current Tasks before shutting down.

---

# Alerting on Worker metrics

Source: https://docs.temporal.io/best-practices/worker-alerting

> A recommended alert set for Temporal Workers, with tag filters, thresholds, and links to triage guidance

Your Worker processes emit metrics that the Temporal Service has no view into: Workflow code failing on replay, Workers that have stopped polling, Task slots that never free up, Local Activities that run past the Workflow Task heartbeat window.
Nobody catches those for you.

This page recommends a set of alerts that covers them.
It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

For metric definitions and tag sets, see the [Temporal SDK metrics reference](/references/sdk-metrics).
For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker).

## Before you start 

None of this works until your Workers are emitting metrics and something is scraping them:

- **Temporal Cloud:** [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup) for your Workers, and [set up Cloud metrics](/cloud/metrics/openmetrics) with OpenMetrics.
- **Self-hosted:** [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring).

### Resolve the metric names for your setup 

The metric names in the tables above are the base names from the [SDK metrics reference](/references/sdk-metrics).
The names you query depend on your SDK and metrics reporter:

- Every metric carries a `temporal_` prefix.
- **Counters** pick up a `_total` suffix when scraped through Prometheus, so `request_failure` becomes `temporal_request_failure_total`.
- **Histograms** pick up `_seconds_bucket` on the bucket series, so you query `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`.
- **Gauges take no suffix at all.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Add `_total` to any of them and you get back nothing, with no hint as to why.

Tag coverage varies too.
`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and Client options can turn the tag off entirely.
Not every SDK emits every tag. `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs, and `task_queue` only in TypeScript.

Confirm the exact names and tags in your own metrics endpoint before writing queries.

## Start with these five 

If you are setting up Worker alerts for the first time, start here.
These five catch the failure modes that stop Workflow Executions outright, and they are the least likely to wake you up for nothing.

1. [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected). Your Workers have stopped polling.
1. [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error). Workflow code no longer matches recorded history.
1. [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large). Executions are being terminated and losing work.
1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated). Tasks are backing up.
1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations). Starts, Signals, and Updates are being throttled.

Add the rest once these are tuned and quiet.

## Choose your thresholds 

Every threshold on this page is a starting point, not a service level objective.
A high-throughput Task Queue needs different values than a bursty batch workload.

There are two knobs per alert:

- **The threshold** sets what counts as unhealthy. Pick it from your own observed p99 during a period you know was healthy, not from the value listed here.
- **The `for` duration** sets how long the condition has to hold before the alert fires. Short durations catch problems faster but fire on transient spikes. Long durations stay quiet but delay detection.

Some of these alerts fire on a binary condition: any occurrence of a gRPC status code, or a gauge hitting zero.
There is no threshold to tune on those, so the `for` duration is the only thing standing between you and a page.

### Tune the `for` duration to your deploys 

Several of these conditions appear briefly whenever you deploy.
A rolling restart drops pollers to zero on each pod as it cycles, produces NOT_FOUND on respond operations for Tasks that were in flight, and can produce short-lived non-determinism errors while two Worker versions overlap.
A Temporal Service upgrade does the same for INTERNAL.

All of those clear on their own once the rollout finishes, so the fix is a `for` duration longer than a deploy takes, not a quieter threshold.
Time your own rollout and set it from that.

Two conditions need a different fix, because no `for` duration makes them correct:

- **An idle Task Queue** legitimately reports zero Task completions and an empty sticky cache. Pair both alerts with a demand signal instead, as [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) and [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) describe.
- **Autoscaling to zero** legitimately reports no pollers and no free slots. Exclude scaled-down Workers from those alerts, or scope them to Task Queues you keep warm.

## Recommended alert set 

The condition column gives the tag filters that tell each alert apart.
Group every alert by `namespace` plus the tags in its condition, so that when one fires you already know which Namespace, operation, or Task Queue it came from.

### Request failures 

These fire on gRPC responses coming back from the Temporal Service to your Worker or Client.
For triage, see [Request failures](/troubleshooting/request-failures).

| Failure mode | Metric | Condition | Threshold | `for` | Default severity |
| --- | --- | --- | --- | --- | --- |
| [NOT_FOUND on respond operations](/troubleshooting/request-failures#not_found-on-respond-operations) | `request_failure` | `status_code=NOT_FOUND`, `operation` in `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, `RespondActivityTaskFailed` | Any occurrence | 5m | Critical |
| [NOT_FOUND on Activity heartbeat](/troubleshooting/request-failures#not_found-on-activity-heartbeat) | `request_failure` | `status_code=NOT_FOUND`, `operation=RecordActivityTaskHeartbeat` | Any occurrence | 5m | Warning |
| [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, `ExecuteMultiOperation` | Any occurrence | 1m | Critical |
| [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in the four respond operations above | Any occurrence | 5m | Critical |
| [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations) | `long_request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `PollWorkflowTaskQueue`, `PollActivityTaskQueue` | Any occurrence | 5m | Warning |
| [UNIMPLEMENTED from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=UNIMPLEMENTED`, any operation | Any occurrence | 2m | Critical |
| [INTERNAL from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=INTERNAL`, any operation | Any occurrence | 2m | Critical |
| [Request latency high on user-facing operations](/troubleshooting/request-failures#request-latency-high-on-user-facing-operations) | `request_latency` | `operation` in the five user-facing operations above | p99 above 2s | 5m | Critical |

### Worker capacity 

These fire when your Workers stop keeping up with the Task Queue.
For triage, see [Worker capacity](/troubleshooting/worker-capacity).

| Failure mode | Metric | Condition | Threshold | `for` | Default severity |
| --- | --- | --- | --- | --- | --- |
| [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical |
| [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical |
| [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical |
| [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Critical |
| [Workflow Task schedule-to-start latency severe](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical |
| [Activity schedule-to-start latency severe](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical |
| [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning |

> **📝 Note:**
>
> `worker_task_slots_available` reports meaningful values only with fixed-size slot suppliers.
> It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance/metrics#slot-availability-metrics).
> If your Workers use resource-based tuning, skip this one and let schedule-to-start latency tell you when capacity is short.
>

### Execution failures 

These fire when your Workflow or Activity code fails on the Worker.
For triage, see [Execution failures](/troubleshooting/execution-failures).

| Failure mode | Metric | Condition | Threshold | `for` | Default severity |
| --- | --- | --- | --- | --- | --- |
| [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical |
| [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical |
| [Request too large](/troubleshooting/execution-failures#request-too-large) | `workflow_task_execution_failed` | `failure_reason=RequestTooLarge` | Any occurrence | 1m | Critical |
| [Workflow Task execution failures elevated](/troubleshooting/execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning |
| [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical |
| [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `failure_reason=ActivityError`, `activity_type` | Rate above 10/s | 2m | Warning |
| [Activity Payloads too large](/troubleshooting/execution-failures#activity-payloads-too-large) | `activity_execution_failed` | `failure_reason=PayloadsTooLarge`, `activity_type` | Any occurrence | 1m | Critical |
| [Unregistered Activity invocation](/troubleshooting/execution-failures#unregistered-activity-invocation) (Go SDK only) | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical |
| [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical |

## Route alerts by severity 

The severity in each table is a suggested routing default, not a property of the metric.

- **Page on Critical.** These mean Workflow Executions have stopped, or are still running but losing work or duplicating side effects.
- **Send Warning to a channel someone reads during the day.** These mean something is degrading but Executions are still moving.

Then adjust, because the defaults won't fit every deployment:

- **Some Critical rows are early warnings, not confirmed stoppage.** High Workflow Task execution latency matters a lot on a latency-sensitive Namespace and barely at all on a batch workload where Tasks routinely run long. Downgrade the latency alerts if that describes you.
- **Some Warning rows can still lose you data.** A NOT_FOUND on Activity heartbeat means the attempt already timed out and will run again from the start. If that Activity isn't idempotent, treat it as Critical.

One last thing worth knowing before you wire up routing: these alerts are chained together.
Task slots fill up, which drops pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero.
Expect them to fire in clusters, and check the triage pages to find which one is the cause rather than responding to all four.
When several fire together, the triage pages identify which is the root cause and which are symptoms.

---

# Child Workflows

Source: https://docs.temporal.io/child-workflows

> A Child Workflow Execution in the Temporal platform is initiated from another Workflow within the same Namespace.

A Child Workflow Execution is a [Workflow Execution](/workflow-execution) that is spawned from within another Workflow in the same Namespace.

- [Go SDK Child Workflow feature guide](/develop/go/workflows/child-workflows)
- [Java SDK Child Workflow feature guide](/develop/java/workflows/child-workflows)
- [PHP SDK Child Workflow feature guide](/develop/php/workflows/child-workflows)
- [Python SDK Child Workflow feature guide](/develop/python/workflows/child-workflows)
- [TypeScript SDK Child Workflow feature guide](/develop/typescript/workflows/child-workflows)
- [.NET SDK Child Workflow feature guide](/develop/dotnet/workflows/child-workflows)
- [Ruby SDK Child Workflow feature guide](/develop/ruby/workflows/child-workflows)
- [Rust SDK Child Workflow feature guide](/develop/rust/workflows/child-workflows)

A Workflow Execution can be both a Parent and a Child Workflow Execution because any Workflow can spawn another Workflow.

![Parent and Child Workflow Execution entity relationship](https://docs.temporal.io/diagrams/parent-child-workflow-execution-relationship.svg)

A Parent Workflow Execution must await on the Child Workflow Execution to spawn.
The Parent can optionally await on the result of the Child Workflow Execution.
Consider the Child's [Parent Close Policy](/parent-close-policy) if the Parent does not await on the result of the Child, which includes any use of Continue-As-New by the Parent.

> **📝 Note:**
>
> Child Workflows do not carry over when the Parent uses [Continue-As-New](/workflow-execution/continue-as-new).
> This means that if a Parent Workflow Execution uses Continue-As-New, any ongoing Child Workflow Executions will not be retained in the new continued instance of the Parent.
>

When a Parent Workflow Execution reaches a Closed status, the Temporal Service propagates Cancellation Requests or Terminations to Child Workflow Executions depending on the Child's [Parent Close Policy](/parent-close-policy).

If a Child Workflow Execution uses Continue-As-New, from the Parent Workflow Execution's perspective the entire chain of Runs is treated as a single execution.

![Parent and Child Workflow Execution entity relationship with Continue As New](https://docs.temporal.io/diagrams/parent-child-workflow-execution-with-continue-as-new.svg)

## When to use Child Workflows

There is no reason to use Child Workflows only for code organization.
You can use object oriented structure and other code organization techniques to deal with complexities.
It is typically recommended to start from a single Workflow Definition if your problem has bounded size in terms of the number of Activity Executions and processed Signals.
It is simpler than multiple asynchronously communicating Workflows.

However, there are several valid reasons for using Child Workflows.

### Create a separate service

Because a Child Workflow Execution can be processed by a completely separate set of [Workers](/workers#worker) than the Parent Workflow Execution, it can act as an entirely separate service.
However, this also means that a Parent Workflow Execution and a Child Workflow Execution do not share any local state.
As all Workflow Executions, they can communicate only via asynchronous [Signals](/sending-messages#sending-signals).

### Partition problems into smaller chunks

An individual Workflow Execution has an [Event History](/workflow-execution/event#event-history) size limit, which imposes a couple of considerations for using Child Workflows.

On one hand, because Child Workflow Executions have their own Event Histories, they are often used to partition large workloads into smaller chunks.
For example, a single Workflow Execution does not have enough space in its Event History to spawn 100,000 [Activity Executions](/activity-execution).
But a Parent Workflow Execution can spawn 1,000 Child Workflow Executions that each spawn 1,000 Activity Executions to achieve a total of 1,000,000 Activity Executions.

However, because a Parent Workflow Execution Event History contains [Events](/workflow-execution/event#event) that correspond to the status of the Child Workflow Execution, a single Parent should not spawn more than 1,000 Child Workflow Executions.

In general, however, Child Workflow Executions result in more overall Events recorded in Event Histories than Activities.
Because each entry in an Event History is a _cost_ in terms of compute resources, this could become a factor in very large workloads.
Therefore, we recommend starting with a single Workflow implementation that uses Activities until there is a clear need for Child Workflows.

### Represent a single resource

As all Workflow Executions, a Child Workflow Execution can create a one to one mapping with a resource.
It can be used to manage the resource using its ID to guarantee uniqueness.
For example, a Workflow that manages host upgrades could spawn a Child Workflow Execution per host (hostname being a Workflow ID) and use them to ensure that all operations on the host are serialized.

### Periodic logic execution

A Child Workflow can be used to execute some periodic logic without overwhelming the Parent Workflow Event History.
In this scenario, the Parent Workflow starts a Child Workflow which executes periodic logic calling [Continue-As-New](/workflow-execution/continue-as-new) as many times as needed, then completes.
From the Parent point of view, it is just a single Child Workflow invocation.

### Child Workflow versus an Activity

Child Workflow Executions and Activity Executions are both started from Workflows, so you might feel confused about when to use which.
Here are some important differences:

- A Child Workflow has access to all Workflow APIs but is subject to the same [deterministic constraints](/workflow-definition#deterministic-constraints) as other Workflows.
  An Activity has the inverse pros and cons—no access to Workflow APIs but no Workflow constraints.
- A Child Workflow Execution can continue on if its Parent is canceled with a [Parent Close Policy](/parent-close-policy) of `ABANDON`.
  An Activity Execution is _always_ canceled when its Workflow Execution is canceled.
  (It can react to a cancellation Signal for cleanup.)
  The decision is roughly analogous to spawning a child process in a terminal to do work versus doing work in the same process.
- Temporal tracks all state changes within a Child Workflow Execution in Event History.
  Only the input, output, and retry attempts of an Activity Execution is tracked.

A Workflow models composite operations that consist of multiple Activities or other Child Workflows.
An Activity usually models a single operation on the external world.

Our advice: **When in doubt, use an Activity.**

---

# Temporal CLI

Source: https://docs.temporal.io/cli

The Temporal CLI (`temporal`) provides direct access to a Temporal Service via the terminal. Use it to manage, monitor,
and debug Temporal applications, plus run an embedded development service when you need fast local feedback.

We provide and maintain an official extension for the Temporal CLI you can use to interact with Temporal
Cloud. Use the extension to manage your Temporal Cloud control plane resources, including Namespaces, Users, Service
Accounts, API keys, and perform other operational and administrative tasks.

## Install and configure the CLI

Install the CLI and cloud extension, run a local development server, and configure your environment in
[Install and configure the CLI](/cli/setup-cli).

## Use with Temporal Cloud

Connect to Temporal Cloud with [environment configuration](/develop/environment-configuration) and use the Cloud
extension to manage your Temporal Cloud control plane resources in [Use with Temporal Cloud](/cli/cloud).

## Start a development server

The CLI includes a local Temporal development service for fast feedback while building or testing your application.

```bash
temporal server start-dev
```

## CLI basics

Get started with the basics of the CLI in [CLI basics](/cli/common-operations).

## Command reference

Refer to the [command reference](/cli/command-reference) for the complete list of commands.

---

# Use with Temporal Cloud

Source: https://docs.temporal.io/cli/cloud

> Use the Temporal CLI with Temporal Cloud to run Workflows, manage Namespaces, and administer your account.

The Temporal CLI works with Temporal Cloud. The same commands you use for local or self-hosted Temporal services, such
as `temporal workflow start` and `temporal workflow list`, work with Temporal Cloud as allowed by your role once you
provide an address and credentials.

For administrative tasks, [install the Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension). The extension adds `temporal cloud` commands for
managing your Temporal Cloud account, including Namespaces, users, API keys, and Nexus endpoints.

> **Public Preview**

Access to Temporal Cloud is governed by role-based access control (RBAC). Your ability to perform actions, including
running CLI commands against in Temporal Cloud is determined by the roles and permissions you have been assigned. Refer
to the [Access control](/cloud/manage-access) page for more details.

## Connect to Temporal Cloud

To connect the CLI to Temporal Cloud, provide the Temporal Service address, Namespace name, and credentials. Temporal
Cloud supports three credential types:

- OAuth tokens obtained through the `temporal cloud login` command (requires the [Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension))
- API keys
- mTLS certificates

### Interactive login

The `temporal cloud login` command opens a browser to authenticate with Temporal Cloud using OAuth. Provide a profile
name to store credentials in. If no profile is specified, credentials are stored in the `default` profile.

```bash
temporal cloud login --profile prod
```

Complete the interactive login process in your browser. After login, your OAuth token is stored in the specified
configuration profile.

To confirm your login, run the `temporal cloud whoami` command:

```bash
temporal cloud whoami --profile prod
```

Run commands against Temporal Cloud by specifying the profile, address, and Namespace:

```bash
temporal workflow list --profile prod  \
  --address <address>\
  --namespace <namespace>
```

### Non-interactive login

For AI agents, CI pipelines, scripts, and other non-interactive environments, use API keys or mTLS certificates. Store
credentials in a [configuration profile](/develop/environment-configuration#cli-integration) or set them as
[environment variables](/cli/setup-cli#environment-variables) to avoid passing them on every command.

To pass credentials inline:

```bash
# Using an API key
temporal workflow list \
  --address <namespace>.<account>.tmprl.cloud:7233 \
  --namespace <namespace>.<account> \
  --api-key <your-api-key>

# Using mTLS certificates
temporal workflow list \
  --address <namespace>.<account>.tmprl.cloud:7233 \
  --namespace <namespace>.<account> \
  --tls-cert-path /path/to/client.pem \
  --tls-key-path /path/to/client.key
```

### Log out

To log out, run the `temporal cloud logout` command.

```bash
temporal cloud logout --profile prod
```

This will remove the OAuth token from the specified configuration profile. If you provided API keys or mTLS
certificates, they will remain in the profile.

## Cloud administration

The Temporal Cloud extension adds `temporal cloud` commands for managing your Temporal Cloud control plane resources in
your Temporal Cloud account, including Namespaces, Users, Service Accounts, API keys, and others. Any of the
authentication methods above grants access to these commands.

The extension enables you to do the following through the CLI:

- Create, configure, and delete Namespaces.
- Create and manage API keys for programmatic access.
- Invite users, assign roles, and manage user groups.
- Create and configure Nexus endpoints.
- View account information and manage connectivity rules.

For installation instructions, see
[Install the Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension). For the full list of
commands, see the [`cloud` command reference](/cli/command-reference/cloud).

## Next steps

- [CLI basics](/cli/common-operations) for common CLI commands.
- [Environment configuration](/develop/environment-configuration) for managing connection profiles across environments.
- [Cloud command reference](/cli/command-reference/cloud) for all `temporal cloud` commands.

---

# Temporal CLI command reference

Source: https://docs.temporal.io/cli/command-reference

> Complete command reference for the Temporal CLI, including the cloud extension.

This section includes the complete command reference for the `temporal` CLI, including the cloud extension.

- [activity](/cli/command-reference/activity)
- [batch](/cli/command-reference/batch)
- [cloud](/cli/command-reference/cloud)
- [config](/cli/command-reference/config)
- [env](/cli/command-reference/env)
- [nexus](/cli/command-reference/nexus)
- [operator](/cli/command-reference/operator)
- [schedule](/cli/command-reference/schedule)
- [server](/cli/command-reference/server)
- [task-queue](/cli/command-reference/task-queue)
- [worker](/cli/command-reference/worker)
- [workflow](/cli/command-reference/workflow)

---

# Temporal CLI activity command reference

Source: https://docs.temporal.io/cli/command-reference/activity

> Learn how to use Temporal Activity commands to perform operations on Activity Executions.

This page provides a reference for the `temporal` CLI `activity` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## cancel

Request cancellation of a Standalone Activity.

```
temporal activity cancel \
    --activity-id YourActivityId
```

Requesting cancellation transitions the Activity's run state
to CancelRequested. If the Activity is heartbeating, a
cancellation error will be raised when the next heartbeat
response is received; if the Activity allows this error to
propagate, the Activity transitions to canceled status.

A visibility Query lets you send bulk cancellations to Standalone Activity
Executions matching the results:

```
temporal activity cancel \
    --query YourQuery \
    --reason YourReason
```

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | No | **string** Activity ID. You must set either --activity-id or --query. |
| `--query`, `-q` | No | **string** EXPERIMENTAL: Starts a batch Activity operation using an SQL-like `QUERY` List Filter. This behavior may change in the future. You must set either --activity-id or --query. |
| `--reason` | No | **string** Reason for cancellation. Also used as reason for batch operation with --query, which defaults to a message with the current user's name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed when --query is present. |
| `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. Only use with --activity-id. Cannot use with --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. Only allowed when --query is present. |

## complete

Complete an Activity, marking it as successfully finished. Specify the
Activity ID and include a JSON result for the returned value:

```
temporal activity complete \
    --activity-id YourActivityId \
    --workflow-id YourWorkflowId \
    --result '{"YourResultKey": "YourResultVal"}'
```

Omit `--workflow-id` to target a Standalone Activity by Activity ID
and optional Run ID.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | Yes | **string** Activity ID. This may be the ID of an Activity invoked by a Workflow, or of a Standalone Activity. |
| `--result` | Yes | **string** Result `JSON` to return. |
| `--run-id`, `-r` | No | **string** Run ID. For workflow Activities (when --workflow-id is provided), this is the Workflow Run ID. For Standalone Activities, this is the Activity Run ID. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. Required for workflow Activities. Omit for Standalone Activities. |

## count

Return a count of Standalone Activities. Use `--query` to filter
the activities to be counted.

```
temporal activity count \
    --query 'ActivityType="YourActivity"'
```

Visit https://docs.temporal.io/visibility to read more about
Search Attributes and queries.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--query`, `-q` | No | **string** Query to filter Activity Executions to count. |

## delete

Delete a Standalone Activity Execution and its Event History.

```
temporal activity delete \
    --activity-id YourActivityId
```

The removal executes asynchronously. If the Execution is Running, the Service
terminates it before deletion.

A visibility Query lets you send bulk delete to Standalone Activity
Executions matching the results:

```
temporal activity delete \
    --query YourQuery
```

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | No | **string** Activity ID. You must set either --activity-id or --query. |
| `--query`, `-q` | No | **string** EXPERIMENTAL: Starts a batch Activity operation using an SQL-like `QUERY` List Filter. This behavior may change in the future. You must set either --activity-id or --query. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to a message with the current user's name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed when --query is present. |
| `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. Only use with --activity-id. Cannot use with --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

## describe

Display information about a Standalone Activity.

```
temporal activity describe \
    --activity-id YourActivityId
```

If `--run-id` is omitted, the latest Activity run is described.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | Yes | **string** Activity ID. |
| `--raw` | No | **bool** Print properties without changing their format. |
| `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. |

## execute

Start a new Standalone Activity and block until it completes.
The result is output to stdout.

```
temporal activity execute \
    --activity-id YourActivityId \
    --type YourActivity \
    --task-queue YourTaskQueue \
    --start-to-close-timeout 30s \
    --input '{"some-key": "some-value"}'
```

The command exits after the Activity completes.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | Yes | **string** Activity ID. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. |
| `--headers` | No | **string[]** Temporal activity headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times. |
| `--heartbeat-timeout` | No | **duration** Maximum time between successful Worker heartbeats. On expiry the current activity attempt fails. |
| `--id-conflict-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID is currently running. Accepted values: Fail, UseExisting. |
| `--id-reuse-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID exists and has completed. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--priority-key` | No | **int** Priority key (1-5, lower = higher priority). Default is 3 when not specified. |
| `--retry-backoff-coefficient` | No | **float** Coefficient for calculating the next retry interval. Must be 1 or larger. |
| `--retry-initial-interval` | No | **duration** Interval of the first retry. If "retry-backoff-coefficient" is 1.0, it is used for all retries. |
| `--retry-maximum-attempts` | No | **int** Maximum number of attempts. Setting to 1 disables retries. Setting to 0 means unlimited attempts. |
| `--retry-maximum-interval` | No | **duration** Maximum interval between retries. |
| `--schedule-to-close-timeout` | No | **duration** Maximum time for the Activity Execution, including all retries. Either this or "start-to-close-timeout" is required. |
| `--schedule-to-start-timeout` | No | **duration** Maximum time an Activity task can stay in a task queue before a Worker picks it up. On expiry it results in a non-retryable failure and no further attempts are scheduled. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. Can be passed multiple times. See https://docs.temporal.io/visibility. |
| `--start-delay` | No | **duration** Delay before dispatching the first Activity task. This delay is not applied to retry attempts. |
| `--start-to-close-timeout` | No | **duration** Maximum time for a single Activity attempt. On expiry a new attempt may be scheduled if permitted by the retry policy and schedule-to-close timeout. Either this or "schedule-to-close-timeout" is required. |
| `--static-details` | No | **string** Static Activity details for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ |
| `--static-summary` | No | **string** Static Activity summary for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Activity task queue. |
| `--type` | Yes | **string** Activity Type name. |

## fail

Fail an Activity, marking it as having encountered an error:

```
temporal activity fail \
    --activity-id YourActivityId \
    --workflow-id YourWorkflowId
```

Omit `--workflow-id` to target a Standalone Activity by Activity ID
and optional Run ID.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | Yes | **string** Activity ID. This may be the ID of an Activity invoked by a Workflow, or of a Standalone Activity. |
| `--detail` | No | **string** Failure detail (JSON). Attached as the failure details payload. |
| `--reason` | No | **string** Failure reason. Attached as the failure message. |
| `--run-id`, `-r` | No | **string** Run ID. For workflow Activities (when --workflow-id is provided), this is the Workflow Run ID. For Standalone Activities, this is the Activity Run ID. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. Required for workflow Activities. Omit for Standalone Activities. |

## list

List Standalone Activities. Use `--query` to filter results.

```
temporal activity list \
    --query 'ActivityType="YourActivity"'
```

Visit https://docs.temporal.io/visibility to read more about
Search Attributes and queries.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--limit` | No | **int** Maximum number of Activity Executions to display. |
| `--page-size` | No | **int** Maximum number of Activity Executions to fetch at a time from the server. |
| `--query`, `-q` | No | **string** Query to filter the Activity Executions to list. |

## pause

Pause an Activity.

Note: This is an experimental feature and may change in the future.

If the Activity is not currently running (e.g. because it previously
failed), it will not be run again until it is unpaused.

However, if the Activity is currently running, it will run until the next
time it fails, completes, or times out, at which point the pause will kick in.

Pause does not stop or extend the Activity's Schedule-To-Close Timeout.
A paused Activity can still time out. Use `temporal activity update-options`
to extend timeout settings before a long pause.

If the Activity is on its last retry attempt and fails, the failure will
be returned to the caller, just as if the Activity had not been paused.

To target a workflow Activity, specify the Activity and Workflow IDs:

```
temporal activity pause \
    --activity-id YourActivityId \
    --workflow-id YourWorkflowId
```

To target a standalone Activity, specify the Activity and Run IDs:

```
temporal activity pause \
    --activity-id YourActivityId \
    --run-id YourRunId
```

To later unpause the activity, see [unpause](#unpause). You may also want to
[reset](#reset) the activity to unpause it while also starting it from the beginning.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | No | **string** The Activity ID to pause. Required. |
| `--identity` | No | **string** The identity of the user or client submitting this request. |
| `--reason` | No | **string** Reason for pausing the Activity. |
| `--run-id`, `-r` | No | **string** Run ID. With --workflow-id, identifies the workflow run. For a standalone Activity (no --workflow-id), targets a specific run; omit to target the latest run. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. Set to target a workflow Activity. Omit to target a standalone Activity. |

## reset

Reset an Activity.

Note: This is an experimental feature and may change in the future.

This restarts the Activity as if it were first being scheduled: the
attempt count returns to one, its per-attempt timeouts are re-armed.
Use `--clear-heartbeat-details` to also clear its
[heartbeat details](#clear-heartbeat-details).

If the Activity may be executing (i.e. it has not yet timed out), the
reset will take effect the next time it fails, heartbeats, or times out.
If the Activity is waiting for a retry (i.e. has failed or timed out), the reset
will apply immediately.

If the Activity is already paused, it will be unpaused by default.
You can specify `--keep-paused` to prevent this.

Either `--activity-id` (with `--workflow-id` for a workflow Activity, or
alone for a standalone Activity) or `--query` must be specified.

### Resetting activities that heartbeat 

Activities that heartbeat will receive a
[Canceled failure](/references/failures#cancelled-failure) the next time
they heartbeat after a reset.

If, in your Activity, you need to do any cleanup when an Activity is
reset, handle this error and then re-throw it when you've cleaned up.

Use `--clear-heartbeat-details` to clear the heartbeat details as part of
the reset.

Specify the Activity and Workflow IDs:

```
temporal activity reset \
    --activity-id YourActivityId \
    --workflow-id YourWorkflowId \
    --keep-paused
```

Workflow Activities can be reset in bulk with a visibility query list filter:

```
temporal activity reset \
    --query 'WorkflowType="YourWorkflow"'
```

The `--query` flag currently applies only to Workflow Activities.

Omit `--workflow-id` to target a Standalone Activity by Activity ID
and optional Run ID.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | No | **string** The Activity ID to reset. Mutually exclusive with --query. Set --workflow-id to target a workflow Activity, or omit it to target a standalone Activity (the latest run unless --run-id is set). |
| `--clear-heartbeat-details` | No | **bool** Clear the Activity's heartbeat details. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--jitter` | No | **duration** The activity will reset at random a time within the specified duration. Can only be used with --query. |
| `--keep-paused` | No | **bool** If the activity was paused, it will stay paused. |
| `--query`, `-q` | No | **string** EXPERIMENTAL: Starts a batch Activity operation using an SQL-like `QUERY` List Filter. This behavior may change in the future. Set either --activity-id (single Activity) or --query (batch). --query cannot be combined with --workflow-id or --run-id. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. |
| `--restore-original-options` | No | **bool** Restore the original options of the activity. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed when --query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id or --activity-id. Cannot use with --query. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. Set to target a Workflow Activity. Omit to target a standalone Activity (by --activity-id). Only for Workflow Activities; cannot be combined with --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm the batch operation. Only allowed when --query is present. |

## result

Wait for a Standalone Activity to complete and output the
result.

```
temporal activity result \
    --activity-id YourActivityId
```

If `--run-id` is omitted, the latest Activity run is used.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | Yes | **string** Activity ID. |
| `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. |

## start

Start a new Standalone Activity. Outputs the Activity ID and
Run ID.

```
temporal activity start \
    --activity-id YourActivityId \
    --type YourActivity \
    --task-queue YourTaskQueue \
    --start-to-close-timeout 5m \
    --input '{"some-key": "some-value"}'
```

The command returns the Activity ID and Run ID.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | Yes | **string** Activity ID. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. |
| `--headers` | No | **string[]** Temporal activity headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times. |
| `--heartbeat-timeout` | No | **duration** Maximum time between successful Worker heartbeats. On expiry the current activity attempt fails. |
| `--id-conflict-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID is currently running. Accepted values: Fail, UseExisting. |
| `--id-reuse-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID exists and has completed. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--priority-key` | No | **int** Priority key (1-5, lower = higher priority). Default is 3 when not specified. |
| `--retry-backoff-coefficient` | No | **float** Coefficient for calculating the next retry interval. Must be 1 or larger. |
| `--retry-initial-interval` | No | **duration** Interval of the first retry. If "retry-backoff-coefficient" is 1.0, it is used for all retries. |
| `--retry-maximum-attempts` | No | **int** Maximum number of attempts. Setting to 1 disables retries. Setting to 0 means unlimited attempts. |
| `--retry-maximum-interval` | No | **duration** Maximum interval between retries. |
| `--schedule-to-close-timeout` | No | **duration** Maximum time for the Activity Execution, including all retries. Either this or "start-to-close-timeout" is required. |
| `--schedule-to-start-timeout` | No | **duration** Maximum time an Activity task can stay in a task queue before a Worker picks it up. On expiry it results in a non-retryable failure and no further attempts are scheduled. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. Can be passed multiple times. See https://docs.temporal.io/visibility. |
| `--start-delay` | No | **duration** Delay before dispatching the first Activity task. This delay is not applied to retry attempts. |
| `--start-to-close-timeout` | No | **duration** Maximum time for a single Activity attempt. On expiry a new attempt may be scheduled if permitted by the retry policy and schedule-to-close timeout. Either this or "schedule-to-close-timeout" is required. |
| `--static-details` | No | **string** Static Activity details for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ |
| `--static-summary` | No | **string** Static Activity summary for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Activity task queue. |
| `--type` | Yes | **string** Activity Type name. |

## terminate

Terminate a Standalone Activity.

```
temporal activity terminate \
    --activity-id YourActivityId \
    --reason YourReason
```

Activity code cannot see or respond to terminations.

A visibility Query lets you send bulk terminations to Standalone Activity
Executions matching the results:

```
temporal activity terminate \
    --query YourQuery \
    --reason YourReason
```

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | No | **string** Activity ID. You must set either --activity-id or --query. |
| `--query`, `-q` | No | **string** EXPERIMENTAL: Starts a batch Activity operation using an SQL-like `QUERY` List Filter. This behavior may change in the future. You must set either --activity-id or --query. |
| `--reason` | No | **string** Reason for termination. Defaults to a message with the current user's name. Also used as reason for batch operation with --query. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed when --query is present. |
| `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. Only use with --activity-id. Cannot use with --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. Only allowed when --query is present. |

## unpause

Re-schedule a previously-paused Activity for execution.

Note: This is an experimental feature and may change in the future.

If the Activity is not running and is past its retry timeout, it will be
scheduled immediately. Otherwise, it will be scheduled after its retry
timeout expires.

Either `--activity-id` (with `--workflow-id` for a workflow Activity, or
alone for a standalone Activity) or `--query` must be specified.

Specify the Activity and Workflow IDs:

```
temporal activity unpause \
    --activity-id YourActivityId \
    --workflow-id YourWorkflowId
```

Workflow Activities can be unpaused in bulk via a visibility Query list filter:

```
temporal activity unpause \
    --query 'TemporalPauseInfo IS NOT NULL'
```

The `--query` flag currently applies only to Workflow Activities.

Omit `--workflow-id` to target a Standalone Activity by Activity ID
and optional Run ID.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | No | **string** The Activity ID to unpause. Mutually exclusive with --query. Set --workflow-id to target a workflow Activity, or omit it to target a standalone Activity (the latest run unless --run-id is set). |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--jitter` | No | **duration** The activity will start at random a time within the specified duration. Can only be used with --query. |
| `--query`, `-q` | No | **string** EXPERIMENTAL: Starts a batch Activity operation using an SQL-like `QUERY` List Filter. This behavior may change in the future. Set either --activity-id (single Activity) or --query (batch). --query cannot be combined with --workflow-id or --run-id. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed when --query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id or --activity-id. Cannot use with --query. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. Set to target a Workflow Activity. Omit to target a standalone Activity (by --activity-id). Only for Workflow Activities; cannot be combined with --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm the batch operation. Only allowed when --query is present. |

## update-options

Update an Activity's options. Updates are incremental, only changing the
specified options.

Note: This is an experimental feature and may change in the future.

For example:

```
temporal activity update-options \
    --activity-id YourActivityId \
    --workflow-id YourWorkflowId \
    --task-queue NewTaskQueueName \
    --schedule-to-close-timeout DURATION \
    --schedule-to-start-timeout DURATION \
    --start-to-close-timeout DURATION \
    --heartbeat-timeout DURATION \
    --retry-initial-interval DURATION \
    --retry-maximum-interval DURATION \
    --retry-backoff-coefficient NewBackoffCoefficient \
    --retry-maximum-attempts NewMaximumAttempts
```

You may follow this command with `temporal activity reset`, and the new
values will apply after the reset.

For a Standalone Activity before its first dispatch, use `--start-delay`
to change when the first Activity Task becomes available. The duration is
measured from the Activity's original schedule time, and `0s` makes it
available immediately. `--start-delay` cannot be changed after the first
Activity Task has been dispatched and is not supported for workflow
Activities.

Either `--activity-id` or `--query` must be specified.

Workflow Activity options can be updated in bulk with a visibility query list filter:

```
temporal activity update-options \
    --query 'WorkflowType="YourWorkflow"' \
    --task-queue NewTaskQueueName
```

The `--query` flag currently applies only to Workflow Activities.

Omit `--workflow-id` to target a Standalone Activity by Activity ID
and optional Run ID.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--activity-id`, `-a` | No | **string** The Activity ID to update options. Mutually exclusive with --query. Set --workflow-id to target a workflow Activity, or omit it to target a standalone Activity (the latest run unless --run-id is set). |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--heartbeat-timeout` | No | **duration** Maximum permitted time between successful worker heartbeats. |
| `--query`, `-q` | No | **string** EXPERIMENTAL: Starts a batch Activity operation using an SQL-like `QUERY` List Filter. This behavior may change in the future. Set either --activity-id (single Activity) or --query (batch). --query cannot be combined with --workflow-id or --run-id. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. |
| `--restore-original-options` | No | **bool** Restore the original options of the activity. |
| `--retry-backoff-coefficient` | No | **float** Coefficient used to calculate the next retry interval. The next retry interval is previous interval multiplied by the backoff coefficient. Must be 1 or larger. |
| `--retry-initial-interval` | No | **duration** Interval of the first retry. If retryBackoffCoefficient is 1.0 then it is used for all retries. |
| `--retry-maximum-attempts` | No | **int** Maximum number of attempts. When exceeded the retries stop even if not expired yet. Setting this value to 1 disables retries. Setting this value to 0 means unlimited attempts (up to the timeouts). |
| `--retry-maximum-interval` | No | **duration** Maximum interval between retries. Exponential backoff leads to interval increase. This value is the cap of the increase. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed when --query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id or --activity-id. Cannot use with --query. |
| `--schedule-to-close-timeout` | No | **duration** Indicates how long the caller is willing to wait for an activity completion. Limits how long retries will be attempted. |
| `--schedule-to-start-timeout` | No | **duration** Limits time an activity task can stay in a task queue before a worker picks it up. This timeout is always non retryable, as all a retry would achieve is to put it back into the same queue. Defaults to the schedule-to-close timeout or workflow execution timeout if not specified. |
| `--start-delay` | No | **duration** For a Standalone Activity before its first Activity Task is dispatched, changes when that task becomes available. The duration is measured from the Activity's original schedule time. Set to `0s` to make the task available immediately. |
| `--start-to-close-timeout` | No | **duration** Maximum time an activity is allowed to execute after being picked up by a worker. This timeout is always retryable. |
| `--task-queue` | No | **string** Name of the task queue for the Activity. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. Set to target a Workflow Activity. Omit to target a standalone Activity (by --activity-id). Only for Workflow Activities; cannot be combined with --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm the batch operation. Only allowed when --query is present. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI batch command reference

Source: https://docs.temporal.io/cli/command-reference/batch

> Use Temporal CLI to manage multiple Workflow or Standalone Activity Executions with Batch Jobs. Filter and monitor Batch Jobs effectively.

This page provides a reference for the `temporal` CLI `batch` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## describe

Show the progress of an ongoing batch job. Pass a valid job ID to display its
information:

```
temporal batch describe \
    --job-id YourJobId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--job-id` | Yes | **string** Batch job ID. |

## list

Return a list of batch jobs on the Service or within a single Namespace. For
example, list the batch jobs for "YourNamespace":

```
temporal batch list \
    --namespace YourNamespace
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--limit` | No | **int** Maximum number of batch jobs to display. |

## terminate

Terminate a batch job with the provided job ID. You must provide a reason for
the termination. The Service stores this explanation as metadata for the
termination event for later reference:

```
temporal batch terminate \
    --job-id YourJobId \
    --reason YourTerminationReason
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--job-id` | Yes | **string** Job ID to terminate. |
| `--reason` | Yes | **string** Reason for terminating the batch job. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI cloud command reference

Source: https://docs.temporal.io/cli/command-reference/cloud

> Command reference for the temporal cloud extension.

> **Public Preview**

This section includes the command reference for the `temporal cloud` CLI extension.

- [account](/cli/command-reference/cloud/account)
- [apikey](/cli/command-reference/cloud/apikey)
- [async-operation](/cli/command-reference/cloud/async-operation)
- [connectivity](/cli/command-reference/cloud/connectivity)
- [custom-role](/cli/command-reference/cloud/custom-role)
- [login](/cli/command-reference/cloud/login)
- [logout](/cli/command-reference/cloud/logout)
- [namespace](/cli/command-reference/cloud/namespace)
- [nexus](/cli/command-reference/cloud/nexus)
- [project](/cli/command-reference/cloud/project)
- [region](/cli/command-reference/cloud/region)
- [service-account](/cli/command-reference/cloud/service-account)
- [user](/cli/command-reference/cloud/user)
- [user-group](/cli/command-reference/cloud/user-group)
- [whoami](/cli/command-reference/cloud/whoami)

---

# Temporal CLI cloud account command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/account

> Account Management Commands

> **Public Preview**

Manage the Temporal Cloud account.

This page provides a reference for the `temporal cloud account` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## audit-log

Commands for working with account audit logs.

### audit-log list

Returns a paginated list of audit logs for the account, optionally filtered by time range.

Example:
  temporal cloud account audit-log get --page-size 50
  temporal cloud account audit-log get --start-time 2024-01-01T00:00:00Z --end-time 2024-02-01T00:00:00Z

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--end-time` | No | **timestamp** Filter for logs before this UTC time (RFC3339 format, e.g. 2024-02-01T00:00:00Z). Defaults to current time. |
| `--page-size` | No | **int** Number of logs to retrieve per page. Cannot exceed 1000. Defaults to 100. |
| `--page-token` | No | **string** Page token from a previous response to retrieve the next page. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--start-time` | No | **timestamp** Filter for logs at or after this UTC time (RFC3339 format, e.g. 2024-01-01T00:00:00Z). Defaults to 30 days ago. |

### audit-log sink

Commands for working with account audit log sinks.

#### audit-log sink delete

Delete an audit log sink for the account. This action is irreversible.

Example:
  temporal cloud account audit-log sink delete --name my-sink

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the audit log sink to delete. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### audit-log sink disable

Disable an audit log sink for the account.

Example:
  temporal cloud account audit-log sink disable --name my-sink

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the audit log sink to disable. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### audit-log sink enable

Enable an audit log sink for the account.

Example:
  temporal cloud account audit-log sink enable --name my-sink

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the audit log sink to enable. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### audit-log sink get

Returns the details of an audit log sink for the account.

Example:
  temporal cloud account audit-log sink get --name my-sink

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--name` | Yes | **string** The name of the audit log sink to get. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### audit-log sink kinesis

Commands for managing Kinesis-based audit log sinks.

##### audit-log sink kinesis create

Create an account audit log sink that streams audit events to Amazon Kinesis.

Temporal Cloud assumes the specified IAM role to write events to the Kinesis
stream identified by the destination URI.

Example:
  temporal cloud account audit-log sink kinesis create \
    --name my-sink \
    --role-name MyRole \
    --destination-uri arn:aws:kinesis:us-east-1:123456789012:stream/MyStream \
    --region us-east-1

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--destination-uri` | Yes | **string** ARN of the Kinesis stream to deliver audit log events to. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** Name of the audit log sink. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | Yes | **string** AWS region where the Kinesis stream is located (e.g. us-east-1). |
| `--role-name` | Yes | **string** Name of the IAM role that Temporal Cloud assumes to write to the Kinesis stream. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

##### audit-log sink kinesis update

Update an existing Kinesis audit log sink. Only the flags you provide are changed;
omitted string flags retain their current values.

Example:
  temporal cloud account audit-log sink kinesis update \
    --name my-sink \
    --role-name NewRole

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--destination-uri` | No | **string** ARN of the Kinesis stream to deliver audit log events to. If omitted, the current value is kept. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** Name of the audit log sink to update. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | No | **string** AWS region where the Kinesis stream is located (e.g. us-east-1). If omitted, the current value is kept. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--role-name` | No | **string** Name of the IAM role that Temporal Cloud assumes to write to the Kinesis stream. If omitted, the current value is kept. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

##### audit-log sink kinesis validate

Validate an audit log sink configuration against Amazon Kinesis without creating it.
Use this to verify that the IAM role and Kinesis stream are correctly configured
before creating or updating the sink.

Example:
  temporal cloud account audit-log sink kinesis validate \
    --name my-sink \
    --role-name MyRole \
    --destination-uri arn:aws:kinesis:us-east-1:123456789012:stream/MyStream \
    --region us-east-1

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--destination-uri` | Yes | **string** ARN of the Kinesis stream to deliver audit log events to. |
| `--region` | Yes | **string** AWS region where the Kinesis stream is located (e.g. us-east-1). |
| `--role-name` | Yes | **string** Name of the IAM role that Temporal Cloud assumes to write to the Kinesis stream. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### audit-log sink list

Returns a paginated list of audit log sinks for the account.

Example:
  temporal cloud account audit-log sink list
  temporal cloud account audit-log sink list --page-size 50

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of sinks to retrieve per page. Cannot exceed 1000. Defaults to 100. |
| `--page-token` | No | **string** Page token from a previous response to retrieve the next page. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### audit-log sink pubsub

Commands for managing PubSub audit log sinks.

##### audit-log sink pubsub create

Creates a new PubSub audit log sink for the account using Google Cloud Pub/Sub.

Example:

```
temporal cloud account audit-log sink pubsub create \
  --name my-sink \
  --service-account-email my-sa@my-project.iam.gserviceaccount.com \
  --topic-name my-topic
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the audit log sink. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-email` | Yes | **string** The email of the GCP service account that Temporal Cloud impersonates for writing records to the customer's PubSub topic (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. |
| `--topic-name` | Yes | **string** The destination PubSub topic name where audit logs will be sent. |

##### audit-log sink pubsub update

Updates an existing PubSub audit log sink for the account.

Example:

```
temporal cloud account audit-log sink pubsub update \
  --name my-sink \
  --service-account-email new-sa@new-project.iam.gserviceaccount.com \
  --topic-name new-topic
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the audit log sink to update. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-email` | No | **string** The email of the GCP service account that Temporal Cloud impersonates for writing records to the customer's PubSub topic (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. |
| `--topic-name` | No | **string** The destination PubSub topic name where audit logs will be sent. |

##### audit-log sink pubsub validate

Validates a PubSub audit log sink specification without creating or modifying any resources.

Example:

```
temporal cloud account audit-log sink pubsub validate \
  --name my-sink \
  --service-account-email my-sa@my-project.iam.gserviceaccount.com \
  --topic-name my-topic
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-email` | Yes | **string** The email of the GCP service account that Temporal Cloud impersonates for writing records to the customer's PubSub topic (for example my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. |
| `--topic-name` | Yes | **string** The destination PubSub topic name where audit logs will be sent. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud apikey command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/apikey

> API Key Management Commands

> **Public Preview**

Commands for creating, listing, and managing Temporal Cloud API keys.

API keys authenticate requests to the Temporal Cloud API.

This page provides a reference for the `temporal cloud apikey` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## create-for-me

Create a new API key owned by the currently authenticated user.
The token is printed once on creation and cannot be retrieved again.

Example:

```
temporal cloud apikey create-for-me --display-name "My Key"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** An optional description for the API key. |
| `--display-name` | Yes | **string** A human-readable display name for the API key. |
| `--expiry-duration` | No | **duration** Expiry duration relative to now (e.g. 30d, 24h, 90m). Supports days (d), hours (h), minutes (m), and seconds (s). Mutually exclusive with --expiry-time. |
| `--expiry-time` | No | **timestamp** Expiry time for the API key in RFC3339 format (e.g. 2025-12-31T00:00:00Z). Mutually exclusive with --expiry-duration. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## create-for-service-account

Create a new API key owned by the specified service account.
The token is printed once on creation and cannot be retrieved again.

Example:

```
temporal cloud apikey create-for-service-account --service-account-id my-sa-id --display-name "My Key"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** An optional description for the API key. |
| `--display-name` | Yes | **string** A human-readable display name for the API key. |
| `--expiry-duration` | No | **duration** Expiry duration relative to now (e.g. 30d, 24h, 90m). Supports days (d), hours (h), minutes (m), and seconds (s). Mutually exclusive with --expiry-time. |
| `--expiry-time` | No | **timestamp** Expiry time for the API key in RFC3339 format (e.g. 2025-12-31T00:00:00Z). Mutually exclusive with --expiry-duration. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account to create the API key for. |

## delete

Delete a Temporal Cloud API key. This action is irreversible.

Example:

```
temporal cloud apikey delete --key-id my-key-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key-id` | Yes | **string** The ID of the API key to delete. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## disable

Disable a Temporal Cloud API key. Disabled keys cannot be used for authentication.

Example:

```
temporal cloud apikey disable --key-id my-key-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key-id` | Yes | **string** The ID of the API key to disable. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## edit

Open an API key configuration in your default editor for interactive
modification. After saving and closing the editor, the changes are
applied to Temporal Cloud.

The editor is determined by the EDITOR environment variable, falling
back to 'vi' if not set.

Example:

```
temporal cloud apikey edit --key-id my-key-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key-id` | Yes | **string** The ID of the API key to edit. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## enable

Enable a previously disabled Temporal Cloud API key.

Example:

```
temporal cloud apikey enable --key-id my-key-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key-id` | Yes | **string** The ID of the API key to enable. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## get

Retrieve the configuration and status of a Temporal Cloud API key.

Example:

```
temporal cloud apikey get --key-id my-key-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--key-id` | Yes | **string** The ID of the API key to retrieve. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## list

List API keys. Optionally filter by user ID, user email, or service account ID.
At most one filter may be specified.

Example:

```
temporal cloud apikey list
temporal cloud apikey list --user-id my-user-id
temporal cloud apikey list --service-account-id my-sa-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of API keys to return per page. |
| `--page-token` | No | **string** Token for retrieving the next page of results. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | No | **string** Filter API keys by service account ID. Mutually exclusive with --user-id and --user-email. |
| `--user-email` | No | **string** Filter API keys by user email. Mutually exclusive with --user-id and --service-account-id. |
| `--user-id` | No | **string** Filter API keys by user ID. Mutually exclusive with --user-email and --service-account-id. |

## update

Update an API key's display name, description, or disabled status.
Only flags that are explicitly provided are changed.

Example:

```
temporal cloud apikey update --key-id my-key-id --display-name "New Name"
temporal cloud apikey update --key-id my-key-id --disabled=true
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** New description for the API key. |
| `--disabled` | No | **bool** Set to true to disable the API key, or false to enable it. |
| `--display-name` | No | **string** New display name for the API key. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key-id` | Yes | **string** The ID of the API key to update. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud async-operation command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/async-operation

> Async Operation Commands

> **Public Preview**

Commands for working with Temporal Cloud async operations.

This page provides a reference for the `temporal cloud async-operation` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## await

Wait for a Temporal Cloud async operation to reach a terminal state.
Polls the operation status until it completes, fails, or is cancelled.

Example:

```
temporal cloud async-operation await --async-operation-id my-op-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async-operation-id` | Yes | **string** The ID of the async operation to wait for. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). Default is 1s. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## get

Retrieve the status and details of a Temporal Cloud async operation.

Example:

```
temporal cloud async-operation get --async-operation-id my-op-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async-operation-id` | Yes | **string** The ID of the async operation to retrieve. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud connectivity command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/connectivity

> Connectivity Management Commands

> **Public Preview**

Commands for managing connectivity rules for Temporal Cloud.

This page provides a reference for the `temporal cloud connectivity` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## delete

Delete a connectivity rule by its ID.

Example:

```
temporal cloud connectivity delete --id <connectivity-rule-id>
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--id` | Yes | **string** The ID of the connectivity rule. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## get

Get details of a specific connectivity rule by its ID.

Example:

```
temporal cloud connectivity get --id <connectivity-rule-id>
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--id` | Yes | **string** The ID of the connectivity rule. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## list

List connectivity rules, optionally filtered by namespace.

Example:

```
temporal cloud connectivity list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | No | **string** Filter connectivity rules by namespace (e.g., 'my-namespace.my-account'). |
| `--page-size` | No | **int** Number of connectivity rules to return per page. |
| `--page-token` | No | **string** Page token for pagination. |
| `--project-id` | No | **string** Filter connectivity rules by project ID. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## private

Commands for managing private connectivity rules.

### private create

Create a new private connectivity rule for AWS, GCP, or Azure.

For AWS, provide --connection-id (VPC endpoint ID) and --region.
For GCP, provide --connection-id (PSC connection ID), --gcp-project-id, and --region.
For Azure, provide --azure-pe-resource-id (ARM resource ID) and --region.

Examples:

```
temporal cloud connectivity private create --connection-id vpce-12345 --region aws-us-west-2

temporal cloud connectivity private create \
  --azure-pe-resource-id /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Network/privateEndpoints/{name} \
  --region azure-eastus
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--azure-pe-resource-id` | No | **string** The ARM resource ID of the Azure Private Endpoint (only for Azure private connectivity). Example: /subscriptions/\{sub\}/resourceGroups/\{rg\}/providers/Microsoft.Network/privateEndpoints/\{name\}. |
| `--connection-id` | No | **string** The connection ID for private connectivity (AWS VPC endpoint ID or GCP PSC connection ID). |
| `--gcp-project-id` | No | **string** The GCP project ID (only for GCP private connectivity). |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | No | **string** The ID of the project to create the connectivity rule in. |
| `--region` | Yes | **string** The region for private connectivity. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## public

Commands for managing public connectivity rules.

### public create

Create a new public internet connectivity rule.

Example:

```
temporal cloud connectivity public create
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--enable-stable-ips` | No | **bool** Connect the namespace via a predictable set of IPs on the public internet. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | No | **string** The ID of the project to create the connectivity rule in. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud custom-role command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/custom-role

> Custom Role Management Commands

> **Public Preview**

Commands for managing Temporal Cloud custom roles.

Custom roles enable fine-grained authorization by binding sets of
permissions (resource + actions) to a named role that can be assigned
to users, user groups, and service accounts.

This page provides a reference for the `temporal cloud custom-role` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## apply

Apply a custom role configuration to Temporal Cloud. Creates a new role
if no role with the given name exists, or updates the existing one to
match the specification.

The specification can be provided as inline JSON or loaded from a file
by prefixing the path with '@'.

Example:

```
temporal cloud custom-role apply --spec @custom-role.json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | Yes | **string** Custom role specification in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## create

Create a new Temporal Cloud custom role from a JSON specification.

The specification can be provided as inline JSON or loaded from a file
by prefixing the path with '@'.

Example with inline JSON:

```
temporal cloud custom-role create --spec '{"name":"reader","description":"...","permissions":[...]}'
```

Example with file path:

```
temporal cloud custom-role create --spec @custom-role.json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | Yes | **string** Custom role specification in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. |

## delete

Delete a Temporal Cloud custom role. This action is irreversible.

Example:

```
temporal cloud custom-role delete --role-id my-role-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--role-id` | Yes | **string** The ID of the custom role. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## edit

Open a custom role configuration in your default editor for interactive
modification. After saving and closing the editor, the changes are
applied to Temporal Cloud.

The editor is determined by the EDITOR environment variable, falling
back to 'vi' if not set.

Example:

```
temporal cloud custom-role edit --role-id my-role-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--role-id` | Yes | **string** The ID of the custom role. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## get

Retrieve the configuration and status of a Temporal Cloud custom role.

Example:

```
temporal cloud custom-role get --role-id my-role-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--role-id` | Yes | **string** The ID of the custom role. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## list

List all Temporal Cloud custom roles accessible with the current
authentication credentials.

Example:

```
temporal cloud custom-role list
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of custom roles to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## update

Update an existing Temporal Cloud custom role from a JSON specification.
Replaces the role's spec with the provided one.

Example:

```
temporal cloud custom-role update --role-id my-role-id --spec @custom-role.json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--role-id` | Yes | **string** The ID of the custom role. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | Yes | **string** Custom role specification in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud login command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/login

> Login

> **Public Preview**

Authenticate with Temporal Cloud using browser-based OAuth login.

This command opens your default browser to complete authentication. Once
logged in, your credentials are stored locally for subsequent commands.

Example:

```
temporal cloud login
```

For headless environments, use --disable-pop-up and follow the printed URL.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--audience` | No | **string** OAuth audience parameter for token generation. |
| `--client-id` | No | **string** OAuth client identifier for authentication. |
| `--domain` | No | **string** Authentication domain for the OAuth provider. |
| `--redirect-url` | No | **string** Redirect URL for OAuth authentication flow. |
| `--reset` | No | **bool** Clear stored login credentials and configuration, then re-authenticate. Use this if you need to switch accounts or fix authentication issues. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud logout command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/logout

> Logout

> **Public Preview**

Log out from Temporal Cloud by clearing stored authentication tokens
and credentials from the local configuration.

Example:

```
temporal cloud logout
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--domain` | No | **string** Authentication domain for the OAuth provider. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud namespace command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/namespace

> Namespace Management Commands

> **Public Preview**

Commands for creating, updating, and managing Temporal Cloud namespaces.

Namespaces provide isolation for workflows and activities. Each namespace
has its own configuration including retention period, region, and access
controls.

This page provides a reference for the `temporal cloud namespace` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## api-key

Commands for managing API key authentication configuration of Temporal Cloud namespaces.

### api-key disable

Disable API key authentication for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace api-key disable --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### api-key enable

Enable API key authentication for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace api-key enable --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### api-key get

Retrieve the current API key authentication configuration for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace api-key get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## apply

Apply a namespace configuration to Temporal Cloud. Creates a new namespace
if it doesn't exist, or updates an existing one to match the specification.

The specification can be provided as inline JSON or loaded from a file
by prefixing the path with '@'.

Example with inline JSON:

```
temporal cloud namespace apply --spec '{"name": "namespace-name", "region": "us-west-2", "retention_days": 7}'
```

Example with file path:

```
temporal cloud namespace apply --spec @namespace-spec.json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the namespace already matches the specification. Without this flag, the command errors when no changes are needed. |
| `--project-id` | No | **string** The ID of the project to create the namespace in. If omitted, the namespace is created in the account's default project. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | Yes | **string** Namespace configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## capacity

Commands for managing the capacity of Temporal Cloud namespaces.

Capacity controls whether a namespace runs in on-demand mode or
provisioned mode (with a fixed TRU allocation).

### capacity get

Retrieve capacity information for a Temporal Cloud namespace, including
the current mode (on-demand or provisioned), mode options, and recent usage stats.

Example:

```
temporal cloud namespace capacity get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### capacity update

Update the capacity of a Temporal Cloud namespace. Choose either on-demand
mode or provisioned mode (with a fixed TRU allocation).

Example (switch to on-demand):

```
temporal cloud namespace capacity update --namespace my-namespace.my-account --capacity-mode on_demand
```

Example (switch to provisioned with 4 TRUs):

```
temporal cloud namespace capacity update --namespace my-namespace.my-account --capacity-mode provisioned --capacity-value 4
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--capacity-mode` | Yes | **string-enum** Capacity mode for the namespace. Must be either 'on_demand' or 'provisioned'. Accepted values: on_demand, provisioned. |
| `--capacity-value` | No | **float** The provisioned capacity in Temporal Resource Units (TRUs). Required and must be greater than 0 when --capacity-mode is 'provisioned'. Ignored when --capacity-mode is 'on_demand'. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## codec

Commands for managing the codec server configuration of Temporal Cloud namespaces.

The codec server is used to encode and decode payloads for workflows and activities.

### codec delete

Delete the codec server configuration from a Temporal Cloud namespace.

Example:

```
temporal cloud namespace codec delete --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### codec get

Retrieve the current codec server configuration for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace codec get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### codec set

Set the codec server configuration for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace codec set --namespace my-namespace.my-account --endpoint https://my-codec.example.com
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-error-message-default-link` | No | **string** A link to display alongside the custom error message for remote codec server errors. |
| `--custom-error-message-default-message` | No | **string** A custom message to display for remote codec server errors. |
| `--endpoint` | Yes | **string** The codec server endpoint URL. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--include-cross-origin-credentials` | No | **bool** Whether to include cross-origin credentials in requests to the codec server. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--pass-access-token` | No | **bool** Whether to pass the user access token to the codec server endpoint. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## connectivity

Commands for attaching and detaching connectivity rules on a Temporal Cloud
namespace. Use 'cloud connectivity' to manage the rules themselves.

### connectivity attach

Attach an existing connectivity rule to a Temporal Cloud namespace.

Example:

```
temporal cloud namespace connectivity attach \
  --namespace my-namespace.my-account \
  --connectivity-rule-id <rule-id>
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--connectivity-rule-id` | Yes | **string[]** The ID of a connectivity rule to attach. Repeat to attach multiple. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### connectivity detach

Detach a connectivity rule from a Temporal Cloud namespace.

Example:

```
temporal cloud namespace connectivity detach \
  --namespace my-namespace.my-account \
  --connectivity-rule-id <rule-id>
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--connectivity-rule-id` | Yes | **string[]** The ID of a connectivity rule to detach. Repeat to detach multiple. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### connectivity list

List all connectivity rules currently attached to a Temporal Cloud namespace.

Example:

```
temporal cloud namespace connectivity list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## create

Create a new Temporal Cloud namespace with the specified configuration.

Options are passed as individual flags. To create or update a namespace
using a full JSON specification, use 'namespace apply' instead.

Example:

```
temporal cloud namespace create --name my-namespace --region aws-us-east-1 --retention-days 30
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--api-key-auth-enabled` | No | **bool** Enable API key authentication for the namespace. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. |
| `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. |
| `--certificate-filter` | No | **string[]** Certificate filter as a JSON object (e.g. `'{"commonName":"foo"}'`). Repeat to add multiple. |
| `--certificate-filter-file` | No | **string** Path to a JSON file containing a certificate filter object. |
| `--codec-endpoint` | No | **string** HTTPS codec server endpoint URL. |
| `--codec-include-cross-origin-credentials` | No | **bool** Include cross-origin credentials in codec server requests. |
| `--codec-pass-access-token` | No | **bool** Pass the user access token to the codec server endpoint. |
| `--connection-rule-id` | No | **string[]** Private connectivity rule ID. Repeat to specify multiple. |
| `--description` | No | **string** The description is a human-readable description of the namespace. Must be at most 255 printable ASCII characters plus whitespace. Optional, default is empty. |
| `--enable-delete-protection` | No | **bool** Prevent accidental deletion of this namespace. |
| `--enable-task-queue-fairness` | No | **bool** Enable task queue fairness for the namespace. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--mtls-auth-enabled` | No | **bool** Enable mTLS authentication for the namespace. |
| `--name`, `-n` | Yes | **string** The name for the new namespace (becomes part of the namespace ID). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | No | **string** The ID of the project to create the namespace in. If omitted, the namespace is created in the account's default project. |
| `--region` | Yes | **string[]** Cloud region where the namespace will be hosted. Repeat to specify multiple regions for High Availability (e.g. --region aws-us-east-1 --region aws-us-west-2). |
| `--retention-days` | No | **int** Number of days to retain closed workflow history. If not specified, the server default applies. |
| `--search-attribute` | No | **string[]** Custom search attribute as 'name=Type' (e.g. --search-attribute myAttr=Keyword). Valid types: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. Repeat to add multiple. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## delete

Delete a Temporal Cloud namespace and all associated data. This action is
irreversible and will permanently remove all workflows, activities, and
history within the namespace.

Example:

```
temporal cloud namespace delete --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the namespace does not exist. Without this flag, the command errors if the namespace is not found. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## description

Commands for viewing and updating the description of a Temporal Cloud
namespace. The description is a human-readable description of the
namespace. Must be at most 255 printable ASCII characters plus
whitespace. Optional, default is empty.

### description get

Retrieve the current description for a Temporal Cloud namespace. The
description is a human-readable description of the namespace.

Example:

```
temporal cloud namespace description get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### description set

Set the description for a Temporal Cloud namespace without changing other
namespace settings. The description is a human-readable description of
the namespace. Must be at most 255 printable ASCII characters
plus whitespace. Pass an empty string to clear the description.

Example:

```
temporal cloud namespace description set --namespace my-namespace.my-account --value "Updated namespace description"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--value` | Yes | **string** New description for the namespace. Must be at most 255 printable ASCII characters plus whitespace. Pass an empty string to clear it. |

## edit

Open a namespace configuration in your default editor for interactive
modification. After saving and closing the editor, the changes are
applied to Temporal Cloud.

The editor is determined by the EDITOR environment variable, falling
back to 'vi' if not set.

Example:

```
temporal cloud namespace edit --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if no changes were made in the editor. Without this flag, the command errors when the configuration is unchanged. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## export

Commands for managing workflow history export sinks for Temporal Cloud namespaces.

Export sinks define destinations (S3, GCS, or Azure Blob) to which workflow history is exported.

### export azure-blob

Commands for managing Azure Blob workflow history export sinks for Temporal Cloud namespaces.

#### export azure-blob create

Create a new Azure Blob workflow history export sink for a Temporal Cloud namespace.
The sink is created in the enabled state.

Example:

```
temporal cloud namespace export azure-blob create --namespace my-namespace.my-account --sink-name my-sink \
  --tenant-id 11111111-1111-1111-1111-111111111111 \
  --subscription-id 22222222-2222-2222-2222-222222222222 \
  --resource-group my-resource-group --storage-account my-storage-account \
  --container-name my-container --region eastus
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--container-name` | Yes | **string** The name of the destination Azure Blob container where Temporal will send data. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | Yes | **string** The region where the Azure storage account is located. |
| `--resource-group` | Yes | **string** The Azure resource group that contains the storage account. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |
| `--storage-account` | Yes | **string** The name of the destination Azure storage account where Temporal will send data. |
| `--subscription-id` | Yes | **string** The Azure subscription ID that contains the storage account. |
| `--tenant-id` | Yes | **string** The Azure tenant ID where the storage account exists and where Temporal's app registration is consented/granted access. |

#### export azure-blob update

Update the configuration of an existing Azure Blob workflow history export sink.
Only the flags you provide are changed; omitted flags keep their current
values. The enabled/disabled state and region are also preserved.

Example (rotate storage account only):

```
temporal cloud namespace export azure-blob update --namespace my-namespace.my-account --sink-name my-sink \
  --storage-account my-new-storage-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--container-name` | No | **string** The name of the destination Azure Blob container where Temporal will send data. If omitted, the current value is kept. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-group` | No | **string** The Azure resource group that contains the storage account. If omitted, the current value is kept. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |
| `--storage-account` | No | **string** The name of the destination Azure storage account where Temporal will send data. If omitted, the current value is kept. |
| `--subscription-id` | No | **string** The Azure subscription ID that contains the storage account. If omitted, the current value is kept. |
| `--tenant-id` | No | **string** The Azure tenant ID where the storage account exists and where Temporal's app registration is consented/granted access. If omitted, the current value is kept. |

#### export azure-blob validate

Validate an Azure Blob workflow history export sink configuration without creating or updating it.
A successful response means the configuration is valid.

Example:

```
temporal cloud namespace export azure-blob validate --namespace my-namespace.my-account --sink-name my-sink \
  --tenant-id 11111111-1111-1111-1111-111111111111 \
  --subscription-id 22222222-2222-2222-2222-222222222222 \
  --resource-group my-resource-group --storage-account my-storage-account \
  --container-name my-container --region eastus
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--container-name` | Yes | **string** The name of the destination Azure Blob container where Temporal will send data. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--region` | Yes | **string** The region where the Azure storage account is located. |
| `--resource-group` | Yes | **string** The Azure resource group that contains the storage account. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |
| `--storage-account` | Yes | **string** The name of the destination Azure storage account where Temporal will send data. |
| `--subscription-id` | Yes | **string** The Azure subscription ID that contains the storage account. |
| `--tenant-id` | Yes | **string** The Azure tenant ID where the storage account exists and where Temporal's app registration is consented/granted access. |

### export delete

Delete a workflow history export sink from a Temporal Cloud namespace.

Example:

```
temporal cloud namespace export delete --namespace my-namespace.my-account --sink-name my-sink
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |

### export disable

Disable a workflow history export sink for a Temporal Cloud namespace.
The sink configuration is preserved and can be re-enabled later.

Example:

```
temporal cloud namespace export disable --namespace my-namespace.my-account --sink-name my-sink
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |

### export enable

Enable a previously disabled workflow history export sink for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace export enable --namespace my-namespace.my-account --sink-name my-sink
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |

### export gcs

Commands for managing GCS workflow history export sinks for Temporal Cloud namespaces.

#### export gcs create

Create a new GCS workflow history export sink for a Temporal Cloud namespace.
The sink is created in the enabled state.

Example:

```
temporal cloud namespace export gcs create --namespace my-namespace.my-account --sink-name my-sink \
  --service-account-email my-service-account@my-project.iam.gserviceaccount.com \
  --bucket-name my-bucket --region us-central1
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--bucket-name` | Yes | **string** The name of the destination GCS bucket. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | Yes | **string** The GCS bucket region. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-email` | Yes | **string** The email of the customer service account that Temporal Cloud impersonates for writing to GCS (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. |
| `--sink-name` | Yes | **string** The name of the export sink. |

#### export gcs update

Update the configuration of an existing GCS workflow history export sink.
Only the flags you provide are changed; omitted flags keep their current
values. The enabled/disabled state and region are also preserved.

Example (rotate service account only):

```
temporal cloud namespace export gcs update --namespace my-namespace.my-account --sink-name my-sink \
  --service-account-email my-new-service-account@my-project.iam.gserviceaccount.com
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--bucket-name` | No | **string** The name of the destination GCS bucket. If omitted, the current value is kept. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-email` | No | **string** The email of the customer service account that Temporal Cloud impersonates for writing to GCS (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. If omitted, the current value is kept. |
| `--sink-name` | Yes | **string** The name of the export sink. |

#### export gcs validate

Validate a GCS workflow history export sink configuration without creating or updating it.
A successful response means the configuration is valid.

Example:

```
temporal cloud namespace export gcs validate --namespace my-namespace.my-account --sink-name my-sink \
  --service-account-email my-service-account@my-project.iam.gserviceaccount.com \
  --bucket-name my-bucket --region us-central1
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--bucket-name` | Yes | **string** The name of the destination GCS bucket. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--region` | Yes | **string** The GCS bucket region. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-email` | Yes | **string** The email of the customer service account that Temporal Cloud impersonates for writing to GCS (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. |
| `--sink-name` | Yes | **string** The name of the export sink. |

### export get

Retrieve the configuration and status of a workflow history export sink for a
Temporal Cloud namespace.

Example:

```
temporal cloud namespace export get --namespace my-namespace.my-account --sink-name my-sink
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |

### export list

List all workflow history export sinks configured for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace export list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### export s3

Commands for managing S3 workflow history export sinks for Temporal Cloud namespaces.

#### export s3 create

Create a new S3 workflow history export sink for a Temporal Cloud namespace.
The sink is created in the enabled state.

Example:

```
temporal cloud namespace export s3 create --namespace my-namespace.my-account --sink-name my-sink \
  --role-arn arn:aws:iam::123456789012:role/my-role --bucket-name my-bucket \
  --region us-east-1
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--bucket-name` | Yes | **string** The name of the destination S3 bucket. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--kms-arn` | No | **string** The AWS KMS key ARN for server-side encryption of exported data. Optional. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | Yes | **string** The AWS region where the S3 bucket is located. |
| `--role-arn` | Yes | **string** The IAM role ARN that Temporal Cloud assumes for writing to S3 (e.g. arn:aws:iam::123456789012:role/my-role). The role name and AWS account ID are parsed from this ARN. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |

#### export s3 update

Update the configuration of an existing S3 workflow history export sink.
Only the flags you provide are changed; omitted flags keep their current
values. The enabled/disabled state and region are also preserved.

Example (rotate IAM role only):

```
temporal cloud namespace export s3 update --namespace my-namespace.my-account --sink-name my-sink \
  --role-arn arn:aws:iam::123456789012:role/my-new-role
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--bucket-name` | No | **string** The name of the destination S3 bucket. If omitted, the current value is kept. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--kms-arn` | No | **string** The AWS KMS key ARN for server-side encryption of exported data. If omitted, the current value is kept. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--role-arn` | No | **string** The IAM role ARN that Temporal Cloud assumes for writing to S3 (e.g. arn:aws:iam::123456789012:role/my-role). The role name and AWS account ID are parsed from this ARN. If omitted, the current value is kept. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |

#### export s3 validate

Validate an S3 workflow history export sink configuration without creating or updating it.
A successful response means the configuration is valid.

Example:

```
temporal cloud namespace export s3 validate --namespace my-namespace.my-account --sink-name my-sink \
  --role-arn arn:aws:iam::123456789012:role/my-role --bucket-name my-bucket \
  --region us-east-1
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--bucket-name` | Yes | **string** The name of the destination S3 bucket. |
| `--kms-arn` | No | **string** The AWS KMS key ARN for server-side encryption of exported data. Optional. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--region` | Yes | **string** The AWS region where the S3 bucket is located. |
| `--role-arn` | Yes | **string** The IAM role ARN that Temporal Cloud assumes for writing to S3 (e.g. arn:aws:iam::123456789012:role/my-role). The role name and AWS account ID are parsed from this ARN. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--sink-name` | Yes | **string** The name of the export sink. |

## fairness

Commands for managing task queue fairness configuration of Temporal Cloud namespaces.

### fairness get

Retrieve the current task queue fairness configuration for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace fairness get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### fairness set

Set the task queue fairness configuration for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace fairness set --namespace my-namespace.my-account --enable-task-queue-fairness=true
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--enable-task-queue-fairness` | Yes | **bool** Enable or disable task queue fairness for the namespace. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## get

Retrieve the configuration and status of a Temporal Cloud namespace.

Returns details including region, retention period, endpoints, and
certificate information.

Example:

```
temporal cloud namespace get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | No | **bool** Output only the namespace specification in JSON format, omitting metadata and status information. |

## ha

Commands for managing High Availability (HA) settings of Temporal Cloud namespaces.

HA settings control active region, managed failover, and replica regions.

### ha failover

Trigger a failover for a Temporal Cloud namespace to a different region.
The target region must already be a replica region of the namespace.

Example:

```
temporal cloud namespace ha failover --namespace my-namespace.my-account --region aws-us-west-2
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | Yes | **string** The target region to failover to (e.g., aws-us-west-2). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### ha get

Retrieve the current High Availability configuration for a Temporal Cloud namespace.
Shows the active region, whether managed failover is enabled, and whether passive poller forwarding is enabled.

Example:

```
temporal cloud namespace ha get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### ha region

Commands for managing replica regions of Temporal Cloud namespaces.

#### ha region add

Add a replica region to a Temporal Cloud namespace. The region will be added
as a passive replica and can later be used for failover.

Example:

```
temporal cloud namespace ha region add --namespace my-namespace.my-account --region aws-us-west-2
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | Yes | **string** The region ID to add as a replica (e.g., aws-us-west-2). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### ha region delete

Remove a replica region from a Temporal Cloud namespace. Note that a 7-day
cooldown period applies before the same region can be re-added.

Example:

```
temporal cloud namespace ha region delete --namespace my-namespace.my-account --region aws-us-west-2
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--region` | Yes | **string** The region ID to remove (e.g., aws-us-west-2). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### ha region list

List all regions and their states for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace ha region list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### ha update

Update the High Availability configuration for a Temporal Cloud namespace.
Use --auto-failover to enable or disable Temporal-managed automatic failover.
Use --passive-poller-forwarding to enable or disable passive poller forwarding.

Example:

```
temporal cloud namespace ha update \
    --namespace my-namespace.my-account \
    --auto-failover enabled \
    --passive-poller-forwarding disabled
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--auto-failover` | No | **string-enum** Enable or disable Temporal-managed automatic failover for the namespace. Accepted values: enabled, disabled. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--passive-poller-forwarding` | No | **string-enum** Enable or disable passive poller forwarding for the namespace. Accepted values: enabled, disabled. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## lifecycle

Commands for managing lifecycle settings of Temporal Cloud namespaces.

Lifecycle settings control the behavior and protection of namespaces,
including delete protection to prevent accidental deletion.

### lifecycle get

Retrieve the current lifecycle configuration for a Temporal Cloud namespace.
Lifecycle settings include delete protection status.

Example:

```
temporal cloud namespace lifecycle get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### lifecycle set

Set the lifecycle configuration for a Temporal Cloud namespace.
Lifecycle settings include delete protection to prevent accidental deletion.

Example:

```
temporal cloud namespace lifecycle set --namespace my-namespace.my-account --enable-delete-protection true
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--enable-delete-protection` | Yes | **bool** Enable or disable delete protection for the namespace. When enabled, the namespace cannot be deleted until this flag is set to false. |
| `--idempotent` | No | **bool** Succeed silently if the lifecycle configuration is already set to the specified value. Without this flag, the command errors when no change is needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--resource-version` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## list

List all Temporal Cloud namespaces accessible with the current
authentication credentials.

Example:

```
temporal cloud namespace list
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--name` | No | **string** Filter namespaces by the name as defined in the specification of the namespace. |
| `--page-size` | No | **int** Number of namespaces to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--project-id` | No | **string** Filter namespaces by project ID. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## mtls

Commands for managing mTLS authentication configuration of Temporal Cloud namespaces.

### mtls cert-ca

Commands for managing the client CA certificates of Temporal Cloud namespaces.

#### mtls cert-ca create

Add client CA certificates to a Temporal Cloud namespace from a PEM file
or base64 encoded string. These certificates are used to verify client
connections and enable mTLS authentication.

Specify either --ca-certificate-file or --ca-certificate, but not both.

Example with file:

```
temporal cloud namespace mtls cert-ca create --namespace my-namespace.my-account --ca-certificate-file ca-cert.pem
```

Example with base64 encoded data:

```
temporal cloud namespace mtls cert-ca create --namespace my-namespace.my-account --ca-certificate <base64-encoded-cert>
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. |
| `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### mtls cert-ca delete

Delete client CA certificates from a Temporal Cloud namespace. This operation
requires confirmation and will remove the specified certificates from the
namespace configuration.

Specify either --ca-certificate-file or --ca-certificate, but not both.

Example with file:

```
temporal cloud namespace mtls cert-ca delete --namespace my-namespace.my-account --ca-certificate-file ca-cert.pem
```

Example with base64 encoded data:

```
temporal cloud namespace mtls cert-ca delete --namespace my-namespace.my-account --ca-certificate <base64-encoded-cert>
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. |
| `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### mtls cert-ca list

Retrieve the list of client CA certificates configured for a Temporal Cloud
namespace. These certificates are used for client authentication.

Example:

```
temporal cloud namespace mtls cert-ca list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### mtls cert-filter

Commands for managing certificate filters for Temporal Cloud namespaces.
Certificate filters restrict mTLS connections to client certificates with
specific distinguished name properties.

#### mtls cert-filter create

Add new certificate filters to a Temporal Cloud namespace. Certificate
filters restrict mTLS connections to client certificates whose distinguished
name properties match at least one of the filters.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--common-name` | No | **string** The common name (CN) field from the certificate's distinguished name. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--organization` | No | **string** The organization (O) field from the certificate's distinguished name. |
| `--organizational-unit` | No | **string** The organizational unit (OU) field from the certificate's distinguished name. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--subject-alternative-name` | No | **string** The subject alternative name (SAN) from the certificate. |

#### mtls cert-filter delete

Delete certificate filters from a Temporal Cloud namespace. Filters are
matched by exact field equality.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--common-name` | No | **string** The common name (CN) field from the certificate's distinguished name. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--organization` | No | **string** The organization (O) field from the certificate's distinguished name. |
| `--organizational-unit` | No | **string** The organizational unit (OU) field from the certificate's distinguished name. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--subject-alternative-name` | No | **string** The subject alternative name (SAN) from the certificate. |

#### mtls cert-filter list

List all certificate filters configured for a Temporal Cloud namespace.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### mtls disable

Disable mTLS authentication for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace mtls disable --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### mtls enable

Enable mTLS authentication for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace mtls enable --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### mtls get

Retrieve the current mTLS authentication configuration for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace mtls get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## retention

Commands for managing data retention settings of Temporal Cloud namespaces.

Retention determines how long closed workflow history data are stored before
being automatically deleted.

### retention get

Retrieve the current data retention period for a Temporal Cloud namespace.
The retention period defines how long closed workflow history data are stored.

Example:

```
temporal cloud namespace retention get --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### retention set

Set the data retention period for a Temporal Cloud namespace. The
retention period defines how long closed workflow history data are stored.

Example:

```
temporal cloud namespace retention set --namespace my-namespace.my-account --retention-days 14
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--retention-days` | Yes | **int** New retention period in days for closed workflow history data. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## search-attribute

Commands for managing custom search attributes for Temporal Cloud namespaces.
Search attributes enable filtering and searching workflows by custom fields.

### search-attribute create

Create a new custom search attribute for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace search-attribute create --namespace my-namespace.my-account --name MyField --type Keyword
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the search attribute to create. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--type` | Yes | **string** The type of the search attribute. Valid values: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. |

### search-attribute list

List all custom search attributes configured for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace search-attribute list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### search-attribute rename

Rename an existing custom search attribute for a Temporal Cloud namespace.
This operation preserves all existing data associated with the search attribute.

Example:

```
temporal cloud namespace search-attribute rename --namespace my-namespace.my-account --existing-name OldField --new-name NewField
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--existing-name` | Yes | **string** The current name of the search attribute to rename. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--new-name` | Yes | **string** The new name for the search attribute. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## service-account

Commands for inspecting the service accounts that have access to a
Temporal Cloud namespace.

### service-account list

List the service accounts that have access to a Temporal Cloud namespace,
including both directly-assigned and inherited access.

Example:

```
temporal cloud namespace service-account list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--page-size` | No | **int** Number of service accounts to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## tag

Commands for managing tags of Temporal Cloud namespaces.

Tags are key-value pairs used for organization and categorization of namespaces.

### tag create

Create a new tag for a Temporal Cloud namespace. Fails if a tag with
the specified key already exists.

Example:

```
temporal cloud namespace tag create --namespace my-namespace.my-account --key environment --value production
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key` | Yes | **string** The tag key. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--value` | Yes | **string** The tag value. |

### tag delete

Delete a tag from a Temporal Cloud namespace by its key.

Example:

```
temporal cloud namespace tag delete --namespace my-namespace.my-account --key environment
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key` | Yes | **string** The tag key to delete. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### tag list

List all tags configured for a Temporal Cloud namespace.

Example:

```
temporal cloud namespace tag list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### tag update

Update the value of an existing tag for a Temporal Cloud namespace.
Fails if the specified tag key does not exist.

Example:

```
temporal cloud namespace tag update --namespace my-namespace.my-account --key environment --value staging
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--key` | Yes | **string** The tag key to update. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--value` | Yes | **string** The new value for the tag. |

## user

Commands for inspecting the users that have access to a Temporal Cloud
namespace.

### user list

List the users that have access to a Temporal Cloud namespace, including
both directly-assigned and inherited access.

Example:

```
temporal cloud namespace user list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--page-size` | No | **int** Number of users to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## user-group

Commands for inspecting the user groups that have access to a Temporal
Cloud namespace.

### user-group list

List the user groups that have access to a Temporal Cloud namespace,
including both directly-assigned and inherited access.

Example:

```
temporal cloud namespace user-group list --namespace my-namespace.my-account
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). |
| `--page-size` | No | **int** Number of user groups to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud nexus command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/nexus

> Nexus Operations Management Commands

> **Public Preview**

Commands for managing Nexus Operations in Temporal Cloud.

This page provides a reference for the `temporal cloud nexus` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## endpoint

Commands for managing Nexus Endpoints in Temporal Cloud.

### endpoint allowed-namespace

Commands for managing allowed namespaces for Nexus Endpoints.

#### endpoint allowed-namespace add

Add namespaces that are allowed to call this Nexus Endpoint.
Namespaces that are already allowed are silently ignored.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the Nexus Endpoint. |
| `--namespace` | Yes | **string[]** A namespace to allow. Can be specified multiple times. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### endpoint allowed-namespace list

List all namespaces that are allowed to call this Nexus Endpoint.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--name` | Yes | **string** The name of the Nexus Endpoint. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### endpoint allowed-namespace remove

Remove namespaces from the list of allowed callers of this Nexus Endpoint.
Namespaces that are not currently allowed are silently ignored.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the Nexus Endpoint. |
| `--namespace` | Yes | **string[]** A namespace to remove. Can be specified multiple times. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

#### endpoint allowed-namespace set

Set the full list of namespaces that are allowed to call this Nexus Endpoint,
replacing any previously allowed namespaces.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the Nexus Endpoint. |
| `--namespace` | Yes | **string[]** A namespace to allow. Can be specified multiple times. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### endpoint create

Create a new Nexus Endpoint on the Cloud Account.
An endpoint name is used in workflow code to invoke Nexus operations.
The endpoint target is a worker and `--target-namespace` and `--target-task-queue`
must both be provided. This will fail if an endpoint with the same name is already registered.

Example:

```
temporal cloud nexus endpoint create --name my-endpoint --target-namespace my-ns.my-account --target-task-queue my-tq
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--allow-namespace` | No | **string[]** A namespace that is allowed to call this endpoint. Can be specified multiple times. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** An optional endpoint description in markdown format. |
| `--description-file` | No | **string** Path to a file containing an endpoint description in markdown format. Mutually exclusive with --description. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the Nexus Endpoint to create. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | No | **string** The ID of the project to create the Nexus Endpoint in. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--target-namespace` | Yes | **string** The namespace in which a handler worker will be polling for Nexus tasks. |
| `--target-task-queue` | Yes | **string** The task queue on which a handler worker will be polling for Nexus tasks. |

### endpoint delete

Delete a Nexus Endpoint on the Cloud Account.
Specify either `--name` or `--id` (exactly one is required).

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--id` | No | **string** The ID of the Nexus Endpoint to delete. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | No | **string** The name of the Nexus Endpoint to delete. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### endpoint get

Get a Nexus Endpoint configuration from the Cloud Account.
Specify either `--name` or `--id` (exactly one is required).

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--id` | No | **string** The ID of the Nexus Endpoint to retrieve. |
| `--name` | No | **string** The name of the Nexus Endpoint to retrieve. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### endpoint list

List Nexus Endpoint configurations on the Cloud Account.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of endpoints to return per page. If no page size is provided, it will default to 100. A maximum of 1000 endpoints can be fetched at a time. |
| `--page-token` | No | **string** Token for retrieving the next page of results. Initial value is empty string. |
| `--project-id` | No | **string** Filter Nexus Endpoints by project ID. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### endpoint update

Update an existing Nexus Endpoint on the Cloud Account.
An endpoint name is used in workflow code to invoke Nexus operations.
Specify either `--name` or `--id` to identify the endpoint (exactly one is required).

The endpoint is patched leaving any existing fields for which flags are not provided
as they were.

Example:

```
temporal cloud nexus endpoint update --name my-endpoint --target-namespace new-ns.my-account --target-task-queue new-tq
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** An optional endpoint description in markdown format. |
| `--description-file` | No | **string** Path to a file containing an endpoint description in markdown format. Mutually exclusive with --description. |
| `--id` | No | **string** The ID of the Nexus Endpoint to update. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | No | **string** The name of the Nexus Endpoint to update. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--target-namespace` | No | **string** The namespace in which a handler worker will be polling for Nexus tasks. |
| `--target-task-queue` | No | **string** The task queue on which a handler worker will be polling for Nexus tasks. |
| `--unset-description` | No | **bool** Unset the endpoint description. Cannot be used with --description or --description-file. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud project command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/project

> Project Management Commands

> **Public Preview**

Commands for managing Temporal Cloud projects.

Projects provide an account-level grouping for Temporal Cloud resources.

This page provides a reference for the `temporal cloud project` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## apply

Apply a project configuration to Temporal Cloud. An existing project with
the specification's display_name is updated if found; if no matching
project exists, a new project is created.

The specification can be provided as inline JSON or loaded from a file
by prefixing the path with '@'.

Example:

```
temporal cloud project apply --spec '{"display_name": "Engineering", "description": "Engineering workloads"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | Yes | **string** Project configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## create

Create a new Temporal Cloud project.

Example:

```
temporal cloud project create --display-name "Engineering" --description "Engineering workloads"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** A description of the project. |
| `--display-name` | Yes | **string** The display name of the project. |
| `--enable-delete-protection` | No | **bool** Prevent the project from being deleted while enabled. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## delete

Delete a Temporal Cloud project.

Example:

```
temporal cloud project delete --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## edit

Open an existing project specification in your default editor and apply
the edited configuration.

Example:

```
temporal cloud project edit --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## get

Retrieve the configuration and status of a Temporal Cloud project.

Example:

```
temporal cloud project get --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--project-id` | Yes | **string** The ID of the project. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## list

List Temporal Cloud projects in the current account.

Example:

```
temporal cloud project list --page-size 50
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Maximum number of projects to return. |
| `--page-token` | No | **string** Token for retrieving the next page of results. |
| `--project-id` | No | **string[]** Filter by project ID. Can be specified multiple times. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## service-account

Commands for inspecting the service accounts that have access to a
Temporal Cloud project.

### service-account list

List the service accounts that have access to a Temporal Cloud project,
including both directly-assigned and inherited access.

Example:

```
temporal cloud project service-account list --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of service accounts to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--project-id` | Yes | **string** The ID of the project. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## update

Update an existing Temporal Cloud project. Only explicitly provided flags
are changed.

Example:

```
temporal cloud project update --project-id my-project-id --display-name "Platform"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** A description of the project. |
| `--display-name` | No | **string** The display name of the project. |
| `--enable-delete-protection` | No | **bool** Prevent the project from being deleted while enabled. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## user

Commands for inspecting the users that have access to a Temporal Cloud
project.

### user list

List the users that have access to a Temporal Cloud project, including
both directly-assigned and inherited access.

Example:

```
temporal cloud project user list --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of users to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--project-id` | Yes | **string** The ID of the project. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## user-group

Commands for inspecting the user groups that have access to a Temporal
Cloud project.

### user-group list

List the user groups that have access to a Temporal Cloud project,
including both directly-assigned and inherited access.

Example:

```
temporal cloud project user-group list --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of user groups to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--project-id` | Yes | **string** The ID of the project. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud region command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/region

> Region Commands

> **Public Preview**

Commands for listing Temporal Cloud regions.

This page provides a reference for the `temporal cloud region` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## get

Get details for a specific Temporal Cloud region.

Example:

```
temporal cloud region get --region aws-us-east-1
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--region`, `-r` | Yes | **string** The ID of the region to get. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## list

List all available Temporal Cloud regions.

Example:

```
temporal cloud region list
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud service-account command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/service-account

> Service Account Management Commands

> **Public Preview**

Commands for managing Temporal Cloud service accounts.

This page provides a reference for the `temporal cloud service-account` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## create

Create a new Temporal Cloud service account with account-level access.
Optionally assign an account role, namespace-level permissions, and
project-level roles.

Account roles: owner, admin, developer, finance-admin, read, metrics-read.
Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member.

Example:

```
temporal cloud service-account create --name my-sa --account-role developer \
  --namespace-access my-namespace.my-account=write \
  --project-access my-project-id=write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. |
| `--description` | No | **string** An optional description for the service account. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the service account. Must be unique across all active service accounts. |
| `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## create-namespace-scoped

Create a new Temporal Cloud service account scoped to a single namespace.

Example:

```
temporal cloud service-account create-namespace-scoped --name my-sa \
  --namespace my-namespace.my-account --namespace-permission write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** An optional description for the service account. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the service account. Must be unique across all active service accounts. |
| `--namespace` | Yes | **string** The namespace to scope the service account to. |
| `--namespace-permission` | Yes | **string** The permission to grant on the namespace. Valid values: admin, write, read. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## create-project-scoped

Create a new Temporal Cloud service account scoped to a single project.
Optionally assign namespace-level permissions within the project.

Project roles: admin, write, read, list, contribute, member.
Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.

Example:

```
temporal cloud service-account create-project-scoped --name my-sa \
  --project-id my-project-id --project-role write \
  --namespace-access my-namespace.my-account=read
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--description` | No | **string** An optional description for the service account. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | Yes | **string** The name of the service account. Must be unique across all active service accounts. |
| `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project to scope the service account to. |
| `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## delete

Delete a Temporal Cloud service account. This action is irreversible.

Example:

```
temporal cloud service-account delete --service-account-id my-sa-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account to delete. |

## edit

Open a service account configuration in your default editor for interactive
modification. After saving and closing the editor, the changes are applied
to Temporal Cloud.

The editor is determined by the EDITOR environment variable, falling
back to 'vi' if not set.

Example:

```
temporal cloud service-account edit --service-account-id my-sa-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account to edit. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## get

Retrieve the configuration and status of a Temporal Cloud service account.

Example:

```
temporal cloud service-account get --service-account-id my-sa-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account to retrieve. |

## list

List all Temporal Cloud service accounts accessible with the current
authentication credentials.

Example:

```
temporal cloud service-account list
temporal cloud service-account list --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--page-size` | No | **int** Number of service accounts to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--project-id` | No | **string** The ID of the project to list project-scoped service accounts for. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## remove-project-access

Remove a Temporal Cloud service account's direct project-level access.

Example:

```
temporal cloud service-account remove-project-access --service-account-id my-sa-id \
  --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account. |

## set-custom-roles

Set the custom roles assigned to a Temporal Cloud service account.
Replaces the service account's current custom role list. Pass no
--custom-role flags to remove all custom roles.

Not valid for namespace-scoped service accounts.

Example:

```
temporal cloud service-account set-custom-roles --service-account-id my-sa-id \
  --custom-role role-id-1 --custom-role role-id-2
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account. |

## set-project-access

Set project-level access for a Temporal Cloud service account.

Project roles: admin, write, read, list, contribute, member.

Example:

```
temporal cloud service-account set-project-access --service-account-id my-sa-id \
  --project-id my-project-id --project-role write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account. |

## update

Update a Temporal Cloud service account. Only flags that are explicitly provided
are changed.

For account-scoped service accounts, use --account-role, --namespace-access, and/or --project-access.
For namespace-scoped service accounts, use --namespace-permission.
For project-scoped service accounts, use --project-role and/or --namespace-access.

Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
Use 'namespace=' (empty permission) to remove access to a namespace.
Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member.
Use 'project-id=' (empty role) to remove access to a project.
Project-scoped service account roles: admin, write, read, list, contribute, member.

Example:

```
temporal cloud service-account update --service-account-id my-sa-id --name new-name
temporal cloud service-account update --service-account-id my-sa-id --account-role admin
temporal cloud service-account update --service-account-id my-sa-id --project-access my-project-id=write
temporal cloud service-account update --service-account-id my-sa-id --project-role write
temporal cloud service-account update --service-account-id my-sa-id --namespace-permission write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. Only valid for account-scoped service accounts. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. |
| `--description` | No | **string** New description for the service account. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--name` | No | **string** New name for the service account. |
| `--namespace-access` | No | **string[]** Namespace access to set, in the format 'namespace=permission'. Use 'namespace=' to remove access. Permission must be one of: admin, write, read. Can be repeated. Only valid for account-scoped and project-scoped service accounts. |
| `--namespace-permission` | No | **string** The permission to grant on the scoped namespace. Valid values: admin, write, read. Only valid for namespace-scoped service accounts. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-access` | No | **string[]** Project access to set, in the format 'project-id=role'. Use 'project-id=' to remove access. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. Only valid for account-scoped service accounts. |
| `--project-role` | No | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. Only valid for project-scoped service accounts. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--service-account-id` | Yes | **string** The ID of the service account to update. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud user command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/user

> User Management Commands

> **Public Preview**

Commands for managing Temporal Cloud users.

This page provides a reference for the `temporal cloud user` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## apply

Apply a user configuration to Temporal Cloud. Creates a new user invitation
if the email does not exist, or updates the existing user to match the specification.

The specification can be provided as inline JSON or loaded from a file
by prefixing the path with '@'.

Example with inline JSON:

```
temporal cloud user apply --spec '{"email": "alice@example.com", "access": {"account_access": {"role": "developer"}}}'
```

Example with file path:

```
temporal cloud user apply --spec @user-spec.json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | Yes | **string** User configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## delete

Delete a Temporal Cloud user. This action is irreversible.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user delete --user-id my-user-id
temporal cloud user delete --user-email alice@example.com
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## edit

Open a user configuration in your default editor for interactive
modification. After saving and closing the editor, the changes are
applied to Temporal Cloud.

The editor is determined by the EDITOR environment variable, falling
back to 'vi' if not set.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user edit --user-id my-user-id
temporal cloud user edit --user-email alice@example.com
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## get

Retrieve the configuration and status of a Temporal Cloud user.

Example:

```
temporal cloud user get --user-id my-user-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## invite

Invite a user to Temporal Cloud by email. Optionally assign an account-level
role, namespace-level access permissions, and project-level access roles.

Account roles: owner, admin, developer, finance-admin, read, metrics-read.
Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member.

Example:

```
temporal cloud user invite --email alice@example.com --account-role developer \
  --namespace-access my-namespace.my-account=write \
  --project-access my-project-id=write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. |
| `--email` | Yes | **string** The email address of the user to invite. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## list

List all Temporal Cloud users accessible with the current
authentication credentials.

Example:

```
temporal cloud user list
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--email` | No | **string** Filter users by email address. |
| `--namespace` | No | **string** Filter users by the namespace they have access to. |
| `--page-size` | No | **int** Number of users to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--project-id` | No | **string** List users with access to the project ID. Cannot be combined with --email or --namespace. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## remove-project-access

Remove a Temporal Cloud user's direct project-level access.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user remove-project-access --user-id my-user-id \
  --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## set-account-role

Set the account-level role for a Temporal Cloud user.

Account roles: owner, admin, developer, finance-admin, read, metrics-read.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user set-account-role --user-id my-user-id --account-role developer
temporal cloud user set-account-role --user-email alice@example.com --account-role admin
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | Yes | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## set-custom-roles

Set the custom roles assigned to a Temporal Cloud user. Replaces the
user's current custom role list. Pass no --custom-role flags to remove
all custom roles.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user set-custom-roles --user-email alice@example.com \
  --custom-role role-id-1 --custom-role role-id-2
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## set-namespace-permissions

Add, update, or remove namespace-level permissions for a Temporal Cloud user.
Changes are applied additively: namespaces not listed are left unchanged.

Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
To remove access to a namespace, pass an empty permission: 'namespace='.

Specify the user with either --user-id or --user-email (not both).

Example:

```
# Grant write access to my-namespace and read access to other-namespace:
temporal cloud user set-namespace-permissions --user-id my-user-id \
  --namespace-access my-namespace.my-account=write \
  --namespace-access other-namespace.my-account=read

# Remove access to a namespace:
temporal cloud user set-namespace-permissions --user-id my-user-id \
  --namespace-access my-namespace.my-account=
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace-access` | Yes | **string[]** Namespace access change in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. Use an empty permission (e.g. 'testns=') to remove access to a namespace. Changes are additive: namespaces not listed are left unchanged. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## set-project-access

Set project-level access for a Temporal Cloud user.

Project roles: admin, write, read, list, contribute, member.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user set-project-access --user-id my-user-id \
  --project-id my-project-id --project-role write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud user-group command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/user-group

> User Group Management Commands

> **Public Preview**

Commands for managing Temporal Cloud user groups.

This page provides a reference for the `temporal cloud user-group` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## apply

Apply a user group configuration to Temporal Cloud. Creates a new user group
if no group with the given display name exists, or updates the existing one
to match the specification.

The specification can be provided as inline JSON or loaded from a file
by prefixing the path with '@'.

Example with inline JSON:

```
temporal cloud user-group apply --spec '{"display_name": "Engineering", "cloud_group": {}, "access": {"account_access": {"role": "developer"}}}'
```

Example with file path:

```
temporal cloud user-group apply --spec @user-group-spec.json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--spec` | Yes | **string** User group configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## create-cloud-group

Create a new Temporal Cloud-managed user group. Members can be managed
using the add-member and remove-member commands.

Account roles: owner, admin, developer, finance-admin, read, metrics-read.
Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member.

Example:

```
temporal cloud user-group create-cloud-group --display-name "Engineering" \
  --account-role developer \
  --namespace-access my-namespace.my-account=write \
  --project-access my-project-id=write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. |
| `--display-name` | Yes | **string** The display name of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## create-google-group

Create a new user group backed by a Google Group. Members are managed
via the Google Group itself.

Account roles: owner, admin, developer, finance-admin, read, metrics-read.
Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member.

Example:

```
temporal cloud user-group create-google-group --display-name "Platform" \
  --google-group-email platform@example.com \
  --account-role developer \
  --project-access my-project-id=write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. |
| `--display-name` | Yes | **string** The display name of the user group. |
| `--google-group-email` | Yes | **string** The email address of the Google Group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## create-scim-group

Create a new user group backed by a SCIM identity provider group.
Members are managed via the upstream identity provider.

Account roles: owner, admin, developer, finance-admin, read, metrics-read.
Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member.

Example:

```
temporal cloud user-group create-scim-group --display-name "Security" \
  --scim-idp-id idp-group-id-123 \
  --account-role read \
  --project-access my-project-id=read
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. |
| `--display-name` | Yes | **string** The display name of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. |
| `--scim-idp-id` | Yes | **string** The identity provider ID for the SCIM group. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## delete

Delete a Temporal Cloud user group. This action is irreversible.

Example:

```
temporal cloud user-group delete --group-id my-group-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## edit

Open a user group configuration in your default editor for interactive
modification. After saving and closing the editor, the changes are
applied to Temporal Cloud.

The editor is determined by the EDITOR environment variable, falling
back to 'vi' if not set.

Example:

```
temporal cloud user-group edit --group-id my-group-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. |

## get

Retrieve the configuration and status of a Temporal Cloud user group.

Example:

```
temporal cloud user-group get --group-id my-group-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## list

List all Temporal Cloud user groups accessible with the current
authentication credentials.

Example:

```
temporal cloud user-group list
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--display-name` | No | **string** Filter user groups by display name. |
| `--google-group-email-address` | No | **string** Filter user groups by Google group email address. |
| `--namespace` | No | **string** Filter user groups by the namespace they have access to. |
| `--page-size` | No | **int** Number of user groups to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--project-id` | No | **string** List user groups with access to the project ID. Cannot be combined with --namespace, --display-name, --google-group-email-address, or --scim-group-idp-id. |
| `--scim-group-idp-id` | No | **string** Filter user groups by SCIM group IDP ID. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## members

Commands for managing members of Temporal Cloud user groups.

### members add

Add a user to a Temporal Cloud user group.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user-group members add --group-id my-group-id --user-id my-user-id
temporal cloud user-group members add --group-id my-group-id --user-email alice@example.com
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

### members list

List all members of a Temporal Cloud user group.

Example:

```
temporal cloud user-group members list --group-id my-group-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--page-size` | No | **int** Number of members to return per page. Use for paginated results. |
| `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

### members remove

Remove a user from a Temporal Cloud user group.

Specify the user with either --user-id or --user-email (not both).

Example:

```
temporal cloud user-group members remove --group-id my-group-id --user-id my-user-id
temporal cloud user-group members remove --group-id my-group-id --user-email alice@example.com
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |
| `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. |
| `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. |

## remove-project-access

Remove a Temporal Cloud user group's direct project-level access.

Example:

```
temporal cloud user-group remove-project-access --group-id my-group-id \
  --project-id my-project-id
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## set-account-role

Set the account-level role for a Temporal Cloud user group.

Account roles: owner, admin, developer, finance-admin, read, metrics-read.

Example:

```
temporal cloud user-group set-account-role --group-id my-group-id --account-role developer
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | Yes | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## set-custom-roles

Set the custom roles assigned to a Temporal Cloud user group. Replaces
the group's current custom role list. Pass no --custom-role flags to
remove all custom roles.

Example:

```
temporal cloud user-group set-custom-roles --group-id my-group-id \
  --custom-role role-id-1 --custom-role role-id-2
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## set-namespace-permissions

Add, update, or remove namespace-level permissions for a Temporal Cloud user group.
Changes are applied additively: namespaces not listed are left unchanged.

Namespace access format: 'namespace=permission' where permission is one of: admin, write, read.
To remove access to a namespace, pass an empty permission: 'namespace='.

Example:

```
temporal cloud user-group set-namespace-permissions --group-id my-group-id \
  --namespace-access my-namespace.my-account=write \
  --namespace-access other-namespace.my-account=read
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace-access` | Yes | **string[]** Namespace access change in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. Use an empty permission (e.g. 'testns=') to remove access to a namespace. Changes are additive: namespaces not listed are left unchanged. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## set-project-access

Set project-level access for a Temporal Cloud user group.

Project roles: admin, write, read, list, contribute, member.

Example:

```
temporal cloud user-group set-project-access --group-id my-group-id \
  --project-id my-project-id --project-role write
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-id` | Yes | **string** The ID of the project. |
| `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## update

Update an existing Temporal Cloud user group's access settings.

Provide at least one of --account-role, --namespace-access, --project-access, or --custom-role.

Example:

```
temporal cloud user-group update --group-id my-group-id --account-role developer
temporal cloud user-group update --group-id my-group-id \
  --namespace-access my-namespace.my-account=write
temporal cloud user-group update --group-id my-group-id \
  --project-access my-project-id=write
temporal cloud user-group update --group-id my-group-id --account-role admin \
  --namespace-access my-namespace.my-account=write \
  --namespace-access other-namespace.my-account=read \
  --project-access my-project-id=read
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--account-role` | No | **string** The account role to assign to the group. Role must be one of: admin, developer, finance-admin, read. |
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. |
| `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. |
| `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. |
| `--group-id` | Yes | **string** The ID of the user group. |
| `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. |
| `--namespace-access` | No | **string[]** Namespace access change in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. Use an empty permission (e.g. 'testns=') to remove access to a namespace. Changes are additive: namespaces not listed are left unchanged. |
| `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). |
| `--project-access` | No | **string[]** Project access change in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. Use an empty role (e.g. 'project-id=') to remove access to a project. Changes are additive: projects not listed are left unchanged. |
| `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI cloud whoami command reference

Source: https://docs.temporal.io/cli/command-reference/cloud/whoami

> Who Am I

> **Public Preview**

Display information about the currently authenticated identity.

Shows whether you are authenticated as a user or service account, along
with the associated API key if one is in use.

Example:

```
temporal cloud whoami
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. |  |
| `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. |  |
| `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. |  |
| `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. |  |
| `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` |

---

# Temporal CLI config command reference

Source: https://docs.temporal.io/cli/command-reference/config

> Temporal CLI 'config' commands allow the getting, setting, deleting, and listing of configuration properties for connecting to Temporal.

This page provides a reference for the `temporal` CLI `config` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## delete

Remove a property within a profile.

```
temporal config delete \
    --profile YourProfile \
    --prop tls.client_cert_path
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--prop`, `-p` | Yes | **string** Specific property to delete. If unset, deletes entire profile. |

## delete-profile

Remove a full profile entirely. The `--profile` must be set explicitly.

```
temporal config delete-profile \
    --profile YourProfile
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

## get

Display specific properties or the entire profile.

```
temporal config get \
    --profile YourProfile \
    --prop address
```

or

```
temporal config get
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--prop`, `-p` | No | **string** Specific property to get. |

## list

List profile names in the config file.

```
temporal config list
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

## set

Assign a value to a property and store it in the config file:

```
temporal config set \
    --profile YourProfile \
    --prop address \
    --value us-west-2.aws.api.temporal.io:7233
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--prop`, `-p` | Yes | **string** Property name. |
| `--value`, `-v` | Yes | **string** Property value. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI env command reference

Source: https://docs.temporal.io/cli/command-reference/env

> Temporal CLI 'env' commands allow the configuration, setting, deleting, and listing of environmental properties, making it easy to manage Temporal Server instances.

This page provides a reference for the `temporal` CLI `env` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## delete

Remove a presets environment entirely _or_ remove a key-value pair within an
environment. If you don't specify an environment (with `--env` or by setting
the `TEMPORAL_ENV` variable), this command updates the "default" environment:

```
temporal env delete \
    --env YourEnvironment
```

or

```
temporal env delete \
    --env prod \
    --key tls-key-path
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--key`, `-k` | No | **string** Property name. |

## get

List the properties for a given environment:

```
temporal env get \
    --env YourEnvironment
```

Print a single property:

```
temporal env get \
    --env YourEnvironment \
    --key YourPropertyKey
```

If you don't specify an environment (with `--env` or by setting the
`TEMPORAL_ENV` variable), this command lists properties of the "default"
environment.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--key`, `-k` | No | **string** Property name. |

## list

List the environments you have set up on your local computer. Environments are
stored in "$HOME/.config/temporalio/temporal.yaml".

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

## set

Assign a value to a property key and store it to an environment:

```
temporal env set \
    --env environment \
    --key property \
    --value value
```

If you don't specify an environment (with `--env` or by setting the
`TEMPORAL_ENV` variable), this command sets properties in the "default"
environment.

Storing keys with CLI option names lets the CLI automatically set those
options for you. This reduces effort and helps avoid typos when issuing
commands.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--key`, `-k` | No | **string** Property name (required). |
| `--value`, `-v` | No | **string** Property value (required). |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI nexus command reference

Source: https://docs.temporal.io/cli/command-reference/nexus

> Learn how to use Temporal Nexus commands for starting, listing, and managing Nexus Operation Executions.

This page provides a reference for the `temporal` CLI `nexus` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## operation

These commands manage Nexus Operation Executions.

Nexus Operation commands follow this syntax:

```
temporal nexus operation [command] [options]
```

### cancel

Request cancellation of a Nexus Operation.

```
temporal nexus operation cancel \
    --operation-id YourOperationId
```

The Operation handler determines how to handle the
cancellation request.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--operation-id` | Yes | **string** Nexus Operation ID. |
| `--reason` | No | **string** Reason for cancellation. |
| `--run-id`, `-r` | No | **string** Run ID of the Nexus Operation. |

### count

Return a count of Nexus Operations. Use `--query`
to filter the operations to be counted.

```
temporal nexus operation count \
    --query 'Endpoint="YourEndpoint"'
```

Visit https://docs.temporal.io/visibility to read more about
Search Attributes and queries.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--query`, `-q` | No | **string** Query to filter Nexus Operation Executions to count. |

### describe

Display detailed information about a specific Nexus
Operation Execution.

```
temporal nexus operation describe \
    --operation-id YourOperationId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--operation-id` | Yes | **string** Nexus Operation ID. |
| `--raw` | No | **bool** Print properties without changing their format. |
| `--run-id`, `-r` | No | **string** Run ID of the Nexus Operation. |

### execute

Start a new Nexus Operation Execution and block until
it completes. The result is output to stdout.

```
temporal nexus operation execute \
    --endpoint YourEndpoint \
    --service YourService \
    --operation YourOperation \
    --operation-id YourOperationId \
    --input '{"some-key": "some-value"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--endpoint` | Yes | **string** Nexus Endpoint name. |
| `--id-conflict-policy` | No | **string-enum** Policy for handling an Operation ID conflict with a running operation. Accepted values: Fail, UseExisting, TerminateExisting. |
| `--id-reuse-policy` | No | **string-enum** Policy for re-using an Operation ID from a previously closed operation. Accepted values: AllowDuplicate, RejectDuplicate. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--operation` | Yes | **string** Nexus Operation name. |
| `--operation-id` | Yes | **string** Nexus Operation ID. |
| `--schedule-to-close-timeout` | No | **duration** Total time the operation is allowed to run. |
| `--schedule-to-start-timeout` | No | **duration** Maximum time to wait for an operation to be started (or completed synchronously) by a handler. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--service` | Yes | **string** Nexus Service name. |
| `--start-to-close-timeout` | No | **duration** Maximum time to wait for an asynchronous operation to complete after it has been started. |
| `--static-summary` | No | **string** Static summary for the Nexus Operation for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |

### list

List Nexus Operations. Use `--query` to filter results.

```
temporal nexus operation list \
    --query 'Endpoint="YourEndpoint"'
```

Visit https://docs.temporal.io/visibility to read more about
Search Attributes and queries.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--limit` | No | **int** Maximum number of Nexus Operation Executions to display. |
| `--page-size` | No | **int** Maximum number of Nexus Operation Executions to fetch at a time from the server. |
| `--query`, `-q` | No | **string** Query to filter the Nexus Operation Executions to list. |

### result

Wait for a Nexus Operation to complete and output
the result.

```
temporal nexus operation result \
    --operation-id YourOperationId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--operation-id` | Yes | **string** Nexus Operation ID. |
| `--run-id`, `-r` | No | **string** Run ID of the Nexus Operation. |

### start

Start a new Nexus Operation. Outputs the
Operation ID and Run ID.

```
temporal nexus operation start \
    --endpoint YourEndpoint \
    --service YourService \
    --operation YourOperation \
    --operation-id YourOperationId \
    --input '{"some-key": "some-value"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--endpoint` | Yes | **string** Nexus Endpoint name. |
| `--id-conflict-policy` | No | **string-enum** Policy for handling an Operation ID conflict with a running operation. Accepted values: Fail, UseExisting, TerminateExisting. |
| `--id-reuse-policy` | No | **string-enum** Policy for re-using an Operation ID from a previously closed operation. Accepted values: AllowDuplicate, RejectDuplicate. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--operation` | Yes | **string** Nexus Operation name. |
| `--operation-id` | Yes | **string** Nexus Operation ID. |
| `--schedule-to-close-timeout` | No | **duration** Total time the operation is allowed to run. |
| `--schedule-to-start-timeout` | No | **duration** Maximum time to wait for an operation to be started (or completed synchronously) by a handler. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--service` | Yes | **string** Nexus Service name. |
| `--start-to-close-timeout` | No | **duration** Maximum time to wait for an asynchronous operation to complete after it has been started. |
| `--static-summary` | No | **string** Static summary for the Nexus Operation for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |

### terminate

Terminate a Nexus Operation.

```
temporal nexus operation terminate \
    --operation-id YourOperationId \
    --reason YourReason
```

Operation handlers cannot see or respond to terminations.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--operation-id` | Yes | **string** Nexus Operation ID. |
| `--reason` | No | **string** Reason for termination. Defaults to a message with the current user's name. |
| `--run-id`, `-r` | No | **string** Run ID of the Nexus Operation. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI operator command reference

Source: https://docs.temporal.io/cli/command-reference/operator

> Operator commands in Temporal allow actions on Namespaces, Search Attributes, Clusters and Nexus Endpoints using specific subcommands. Execute with "temporal operator [command] [subcommand] [options]".

This page provides a reference for the `temporal` CLI `operator` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## cluster

Perform operator actions on Temporal Services (also known as Clusters).

```
temporal operator cluster [subcommand] [options]
```

For example to check Service/Cluster health:

```
temporal operator cluster health
```

### describe

View information about a Temporal Cluster (Service), including Cluster Name,
persistence store, and visibility store. Add `--detail` for additional info:

```
temporal operator cluster describe [--detail]
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--detail` | No | **bool** Show history shard count and Cluster/Service version information. |

### health

View information about the health of a Temporal Service:

```
temporal operator cluster health
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

### list

Print a list of remote Temporal Clusters (Services) registered to the local
Service. Report details include the Cluster's name, ID, address, History Shard
count, Failover version, and availability:

```
temporal operator cluster list [--limit max-count]
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--limit` | No | **int** Maximum number of Clusters to display. |

### remove

Remove a registered remote Temporal Cluster (Service) from the local Service.

```
temporal operator cluster remove \
    --name YourClusterName
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name` | Yes | **string** Cluster/Service name. |

### system

Show Temporal Server information for Temporal Clusters (Service): Server
version, scheduling support, and more. This information helps diagnose
problems with the Temporal Server.

The command defaults to the local Service. Otherwise, use the
`--frontend-address` option to specify a Cluster (Service) endpoint:

```
temporal operator cluster system \
    --frontend-address "YourRemoteEndpoint:YourRemotePort"
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

### upsert

Add, remove, or update a registered ("remote") Temporal Cluster (Service).

```
temporal operator cluster upsert [options]
```

For example:

```
temporal operator cluster upsert \
    --frontend-address "YourRemoteEndpoint:YourRemotePort"
    --enable-connection false
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--enable-connection` | No | **bool** Set the connection to "enabled". |
| `--enable-replication` | No | **bool** Set the replication to "enabled". |
| `--frontend-address` | Yes | **string** Remote endpoint. |

## namespace

Manage Temporal Cluster (Service) Namespaces:

```
temporal operator namespace [command] [command options]
```

For example:

```
temporal operator namespace create \
    --namespace YourNewNamespaceName
```

### create

Create a new Namespace on the Temporal Service:

```
temporal operator namespace create \
    --namespace YourNewNamespaceName \
    [options]
```

Create a Namespace with multi-region data replication:

```
temporal operator namespace create \
    --global \
    --namespace YourNewNamespaceName
```

Configure settings like retention and Visibility Archival State as needed.
For example, the Visibility Archive can be set on a separate URI:

```
temporal operator namespace create \
    --retention 5d \
    --visibility-archival-state enabled \
    --visibility-uri YourURI \
    --namespace YourNewNamespaceName
```

Note: URI values for archival states can't be changed once enabled.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--active-cluster` | No | **string** Active Cluster (Service) name. |
| `--cluster` | No | **string[]** Cluster (Service) names for Namespace creation. Can be passed multiple times. |
| `--data` | No | **string[]** Namespace data as `KEY=VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey={"your": "value"}` Can be passed multiple times. |
| `--description` | No | **string** Namespace description. |
| `--email` | No | **string** Owner email. |
| `--global` | No | **bool** Enable multi-region data replication. |
| `--history-archival-state` | No | **string-enum** History archival state. Accepted values: disabled, enabled. |
| `--history-uri` | No | **string** Archive history to this `URI`. Once enabled, can't be changed. |
| `--retention` | No | **duration** Time to preserve closed Workflows before deletion. |
| `--visibility-archival-state` | No | **string-enum** Visibility archival state. Accepted values: disabled, enabled. |
| `--visibility-uri` | No | **string** Archive visibility data to this `URI`. Once enabled, can't be changed. |

### delete

Removes a Namespace from the Service.

```
temporal operator namespace delete [options]
```

For example:

```
temporal operator namespace delete \
    --namespace YourNamespaceName
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--yes`, `-y` | No | **bool** Don't prompt to confirm deletion. |

### describe

Provide long-form information about a Namespace identified by its ID or name:

```
temporal operator namespace describe \
    --namespace-id YourNamespaceId
```

or

```
temporal operator namespace describe \
    --namespace YourNamespaceName
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--namespace-id` | No | **string** Namespace ID. |

### list

Display a detailed listing for all Namespaces on the Service:

```
temporal operator namespace list
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

### update

Update a Namespace using properties you specify.

```
temporal operator namespace update [options]
```

Assign a Namespace's active Cluster (Service):

```
temporal operator namespace update \
    --namespace YourNamespaceName \
    --active-cluster NewActiveCluster
```

Active Cluster changes cannot be combined with other Namespace updates.
Update Namespace configuration, including the replication Cluster list,
before changing the active Cluster in a separate command.

Promote a Namespace for multi-region data replication:

```
temporal operator namespace update \
    --namespace YourNamespaceName \
    --promote-global
```

You may update archives that were previously enabled or disabled. Note: URI
values for archival states can't be changed once enabled.

```
temporal operator namespace update \
    --namespace YourNamespaceName \
    --history-archival-state enabled \
    --visibility-archival-state disabled
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--active-cluster` | No | **string** Active Cluster (Service) name. Cannot be combined with options that modify Namespace configuration. |
| `--cluster` | No | **string[]** Cluster (Service) names. |
| `--data` | No | **string[]** Namespace data as `KEY=VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey={"your": "value"}` Can be passed multiple times. |
| `--description` | No | **string** Namespace description. |
| `--email` | No | **string** Owner email. |
| `--history-archival-state` | No | **string-enum** History archival state. Accepted values: disabled, enabled. |
| `--history-uri` | No | **string** Archive history to this `URI`. Once enabled, can't be changed. |
| `--promote-global` | No | **bool** Enable multi-region data replication. |
| `--replication-state` | No | **string-enum** Replication state. Accepted values: normal, handover. |
| `--retention` | No | **duration** Length of time a closed Workflow is preserved before deletion. |
| `--visibility-archival-state` | No | **string-enum** Visibility archival state. Accepted values: disabled, enabled. |
| `--visibility-uri` | No | **string** Archive visibility data to this `URI`. Once enabled, can't be changed. |

## nexus

These commands manage Nexus resources.

Nexus commands follow this syntax:

```
temporal operator nexus [command] [subcommand] [options]
```

### endpoint

These commands manage Nexus Endpoints.

Nexus Endpoint commands follow this syntax:

```
temporal operator nexus endpoint [command] [options]
```

#### create

Create a Nexus Endpoint on the Server.

A Nexus Endpoint name is used in Workflow code to invoke Nexus Operations.
The endpoint target may either be a Worker, in which case
`--target-namespace` and `--target-task-queue` must both be provided, or
an external URL, in which case `--target-url` must be provided.

This command will fail if an Endpoint with the same name is already
registered.

```
temporal operator nexus endpoint create \
  --name your-endpoint \
  --target-namespace your-namespace \
  --target-task-queue your-task-queue \
  --description-file DESCRIPTION.md
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--description` | No | **string** Nexus Endpoint description. You may use Markdown formatting in the Nexus Endpoint description. |
| `--description-file` | No | **string** Path to the Nexus Endpoint description file. The contents of the description file may use Markdown formatting. |
| `--name` | Yes | **string** Endpoint name. |
| `--target-namespace` | No | **string** Namespace where a handler Worker polls for Nexus tasks. |
| `--target-task-queue` | No | **string** Task Queue that a handler Worker polls for Nexus tasks. |
| `--target-url` | No | **string** An external Nexus Endpoint that receives forwarded Nexus requests. May be used as an alternative to `--target-namespace` and `--target-task-queue`. _(Experimental)_ |

#### delete

Delete a Nexus Endpoint from the Server.

```
temporal operator nexus endpoint delete --name your-endpoint
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name` | Yes | **string** Endpoint name. |

#### get

Get a Nexus Endpoint by name from the Server.

```
temporal operator nexus endpoint get --name your-endpoint
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name` | Yes | **string** Endpoint name. |

#### list

List all Nexus Endpoints on the Server.

```
temporal operator nexus endpoint list
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

#### update

Update an existing Nexus Endpoint on the Server.

A Nexus Endpoint name is used in Workflow code to invoke Nexus Operations.
The Endpoint target may either be a Worker, in which case
`--target-namespace` and `--target-task-queue` must both be provided, or
an external URL, in which case `--target-url` must be provided.

The Endpoint is patched; existing fields for which flags are not provided
are left as they were.

Update only the target task queue:

```
temporal operator nexus endpoint update \
  --name your-endpoint \
  --target-task-queue your-other-queue
```

Update only the description:

```
temporal operator nexus endpoint update \
  --name your-endpoint \
  --description-file DESCRIPTION.md
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--description` | No | **string** Nexus Endpoint description. You may use Markdown formatting in the Nexus Endpoint description. |
| `--description-file` | No | **string** Path to the Nexus Endpoint description file. The contents of the description file may use Markdown formatting. |
| `--name` | Yes | **string** Endpoint name. |
| `--target-namespace` | No | **string** Namespace where a handler Worker polls for Nexus tasks. |
| `--target-task-queue` | No | **string** Task Queue that a handler Worker polls for Nexus tasks. |
| `--target-url` | No | **string** An external Nexus Endpoint that receives forwarded Nexus requests. May be used as an alternative to `--target-namespace` and `--target-task-queue`. _(Experimental)_ |
| `--unset-description` | No | **bool** Unset the description. |

## search-attribute

Create, list, or remove Search Attributes fields stored in a Workflow
Execution's metadata:

```
temporal operator search-attribute create \
    --name YourAttributeName \
    --type Keyword
```

Supported types include: Text, Keyword, Int, Double, Bool, Datetime, and
KeywordList.

If you wish to delete a Search Attribute, please contact support
at https://support.temporal.io.

### create

Add one or more custom Search Attributes:

```
temporal operator search-attribute create \
    --name YourAttributeName \
    --type Keyword
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name` | Yes | **string[]** Search Attribute name. |
| `--type` | Yes | **string-enum[]** Search Attribute type. Accepted values: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. |

### list

Display a list of active Search Attributes that can be assigned or used with
Workflow Queries. You can manage this list and add attributes as needed:

```
temporal operator search-attribute list
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

### remove

Remove custom Search Attributes from the options that can be assigned or used
with Workflow Queries.

```
temporal operator search-attribute remove \
    --name YourAttributeName
```

Remove attributes without confirmation:

```
temporal operator search-attribute remove \
    --name YourAttributeName \
    --yes
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name` | Yes | **string[]** Search Attribute name. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm removal. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI schedule command reference

Source: https://docs.temporal.io/cli/command-reference/schedule

> Temporal's Schedule commands allow users to create, update, and manage Workflow Executions seamlessly for automation, supporting commands for creation, backfill, deletion, and more.

This page provides a reference for the `temporal` CLI `schedule` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## backfill

Batch-execute actions that would have run during a specified time interval.
Use this command to fill in Workflow runs from when a Schedule was paused,
before a Schedule was created, from the future, or to re-process a previously
executed interval.

Backfills require a Schedule ID and the time period covered by the request.
It's best to use the `BufferAll` or `AllowAll` policies to avoid conflicts
and ensure no Workflow Executions are skipped.

For example:

```
temporal schedule backfill \
    --schedule-id "YourScheduleId" \
    --start-time "2022-05-01T00:00:00Z" \
    --end-time "2022-05-31T23:59:59Z" \
    --overlap-policy BufferAll
```

The policies include:

* **AllowAll**: Allow unlimited concurrent Workflow Executions. This
  significantly speeds up the backfilling process on systems that support
  concurrency. You must ensure running Workflow Executions do not interfere
  with each other.
* **BufferAll**: Buffer all incoming Workflow Executions while waiting for
  the running Workflow Execution to complete.
* **Skip**: If a previous Workflow Execution is still running, discard new
  Workflow Executions.
* **BufferOne**: Same as 'Skip' but buffer a single Workflow Execution to be
  run after the previous Execution completes. Discard other Workflow
  Executions.
* **CancelOther**: Cancel the running Workflow Execution and replace it with
  the incoming new Workflow Execution.
* **TerminateOther**: Terminate the running Workflow Execution and replace
  it with the incoming new Workflow Execution.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--end-time` | Yes | **timestamp** Backfill end time. |
| `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. |
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |
| `--start-time` | Yes | **timestamp** Backfill start time. |

## create

Create a new Schedule on the Temporal Service. A Schedule automatically starts
new Workflow Executions at the times you specify.

For example:

```
  temporal schedule create \
    --schedule-id "YourScheduleId" \
    --calendar '{"dayOfWeek":"Fri","hour":"3","minute":"30"}' \
    --workflow-id YourBaseWorkflowIdName \
    --task-queue YourTaskQueue \
    --type YourWorkflowType
```

Schedules support any combination of `--calendar`, `--interval`, and `--cron`:

* Shorthand `--interval` strings.
  For example: 45m (every 45 minutes) or 6h/5h (every 6 hours, at the top of
  the 5th hour).
* JSON `--calendar`, as in the preceding example.
* Unix-style `--cron` strings and robfig declarations
  (@daily/@weekly/@every X/etc).
  For example, every Friday at 12:30 PM: `30 12 * * Fri`.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--calendar` | No | **string[]** Calendar specification in JSON. For example: `{"dayOfWeek":"Fri","hour":"17","minute":"5"}`. |
| `--catchup-window` | No | **duration** Maximum catch-up time for when the Service is unavailable. |
| `--cron` | No | **string[]** Calendar specification in cron string format. For example: `"30 12 * * Fri"`. |
| `--end-time` | No | **timestamp** Schedule end time. |
| `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--interval` | No | **string[]** Interval duration. For example, 90m, or 60m/15m to include phase offset. |
| `--jitter` | No | **duration** Max difference in time from the specification. Vary the start time randomly within this amount. |
| `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. |
| `--notes` | No | **string** Initial notes field value. |
| `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. |
| `--pause-on-failure` | No | **bool** Pause schedule after Workflow failures. |
| `--paused` | No | **bool** Pause the Schedule immediately on creation. |
| `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. |
| `--remaining-actions` | No | **int** Total allowed actions. Default is zero (unlimited). |
| `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. |
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |
| `--schedule-memo` | No | **string[]** Set schedule memo using `KEY="VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--schedule-search-attribute` | No | **string[]** Set schedule Search Attributes using `KEY="VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--start-time` | No | **timestamp** Schedule start time. |
| `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ |
| `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Workflow Task queue. |
| `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. |
| `--time-zone` | No | **string** Interpret calendar specs with the `TZ` time zone. For a list of time zones, see: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. |
| `--type` | Yes | **string** Workflow Type name. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. |

## delete

Deletes a Schedule on the front end Service:

```
temporal schedule delete \
    --schedule-id YourScheduleId
```

Removing a Schedule won't affect the Workflow Executions it started that are
still running. To cancel or terminate these Workflow Executions, use `temporal
workflow delete` with the `TemporalScheduledById` Search Attribute instead.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |

## describe

Show a Schedule configuration, including information about past, current, and
future Workflow runs:

```
temporal schedule describe \
    --schedule-id YourScheduleId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |

## list

Lists the Schedules hosted by a Namespace:

```
temporal schedule list \
    --namespace YourNamespace
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--long`, `-l` | No | **bool** Show detailed information. |
| `--query`, `-q` | No | **string** Filter results using given List Filter. |
| `--really-long` | No | **bool** Show extensive information in non-table form. |

## list-matching-times

Note: This is an experimental feature and may change in the future.

List the times a Schedule's spec would match within a given time
range. The time range may be in the past or future. Use this
command to preview when a Schedule will take actions without
actually running them.

For example:

```
  temporal schedule list-matching-times \
    --schedule-id "YourScheduleId" \
    --start-time "2024-01-01T00:00:00Z" \
    --end-time "2024-01-31T23:59:59Z"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--end-time` | Yes | **timestamp** End of time range to list matching times. |
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |
| `--start-time` | Yes | **timestamp** Start of time range to list matching times. |

## toggle

Pause or unpause a Schedule by passing a flag with your desired state:

```
temporal schedule toggle \
    --schedule-id "YourScheduleId" \
    --pause \
    --reason "YourReason"
```

and

```
temporal schedule toggle
    --schedule-id "YourScheduleId" \
    --unpause \
    --reason "YourReason"
```

The `--reason` text updates the Schedule's `notes` field for operations
communication. It defaults to "(no reason provided)" if omitted. This field is
also visible on the Service Web UI.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--pause` | No | **bool** Pause the Schedule. |
| `--reason` | No | **string** Reason for pausing or unpausing the Schedule. |
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |
| `--unpause` | No | **bool** Unpause the Schedule. |

## trigger

Trigger a Schedule to run immediately:

```
temporal schedule trigger \
    --schedule-id "YourScheduleId"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. |
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |

## update

Update an existing Schedule with new configuration details, including time
specifications, action, and policies:

```
temporal schedule update \
    --schedule-id "YourScheduleId" \
    --workflow-id YourBaseWorkflowIdName \
    --task-queue YourTaskQueue \
    --type YourWorkflowType
```

This command performs a full replacement of the Schedule
configuration. Any options not provided will be reset to their default
values. You must re-specify all options, not just the ones you want to
change. To view the current configuration of a Schedule, use
`temporal schedule describe` before updating.

Schedule memo and search attributes cannot be updated with this
command. They are set only during Schedule creation and are not affected
by updates.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--calendar` | No | **string[]** Calendar specification in JSON. For example: `{"dayOfWeek":"Fri","hour":"17","minute":"5"}`. |
| `--catchup-window` | No | **duration** Maximum catch-up time for when the Service is unavailable. |
| `--cron` | No | **string[]** Calendar specification in cron string format. For example: `"30 12 * * Fri"`. |
| `--end-time` | No | **timestamp** Schedule end time. |
| `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--interval` | No | **string[]** Interval duration. For example, 90m, or 60m/15m to include phase offset. |
| `--jitter` | No | **duration** Max difference in time from the specification. Vary the start time randomly within this amount. |
| `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. |
| `--notes` | No | **string** Initial notes field value. |
| `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. |
| `--pause-on-failure` | No | **bool** Pause schedule after Workflow failures. |
| `--paused` | No | **bool** Pause the Schedule immediately on creation. |
| `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. |
| `--remaining-actions` | No | **int** Total allowed actions. Default is zero (unlimited). |
| `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. |
| `--schedule-id`, `-s` | Yes | **string** Schedule ID. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--start-time` | No | **timestamp** Schedule start time. |
| `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ |
| `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Workflow Task queue. |
| `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. |
| `--time-zone` | No | **string** Interpret calendar specs with the `TZ` time zone. For a list of time zones, see: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. |
| `--type` | Yes | **string** Workflow Type name. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI server command reference

Source: https://docs.temporal.io/cli/command-reference/server

> Manage your Temporal Server easily with CLI commands. Start a local server using `temporal server start-dev` and access the Web UI at http://localhost:8233. Customize with multiple options.

This page provides a reference for the `temporal` CLI `server` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## start-dev

Run a development Temporal Server on your local system.

```
+------------------------------------------------------------------------+
| WARNING: The development server is not intended for production use.    |
| It skips certain HTTP security checks to make local use simpler.       |
|                                                                        |
| For production use, see:                                               |
| https://docs.temporal.io/production-deployment                         |
+------------------------------------------------------------------------+
```

View the Web UI for the default configuration at: http://localhost:8233

```
temporal server start-dev
```

Add persistence for Workflow Executions across runs:

```
temporal server start-dev \
    --db-filename path-to-your-local-persistent-store
```

Set the port from the front-end gRPC Service (7233 default):

```
temporal server start-dev \
    --port 7000
```

Use a custom port for the Web UI. The default is the gRPC port (7233 default)
plus 1000 (8233):

```
temporal server start-dev \
    --ui-port 3000
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--db-filename`, `-f` | No | **string** Path to file for persistent Temporal state store. By default, Workflow Executions are lost when the server process dies. |
| `--dynamic-config-value` | No | **string[]** Dynamic configuration value using `KEY=VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey="YourString"` Can be passed multiple times. |
| `--headless` | No | **bool** Disable the Web UI. |
| `--http-port` | No | **int** Port for the HTTP API service. Defaults to a random free port. |
| `--ip` | No | **string** IP address bound to the front-end Service. |
| `--log-config` | No | **bool** Print the server config to stderr. |
| `--metrics-port` | No | **int** Port for the '/metrics' HTTP endpoint. Defaults to a random free port. |
| `--namespace`, `-n` | No | **string[]** Namespaces to be created at launch. The "default" Namespace is always created automatically. |
| `--port`, `-p` | No | **int** Port for the front-end gRPC Service. |
| `--search-attribute` | No | **string[]** Search attributes to register using `KEY=VALUE` pairs. Keys must be identifiers, and values must be the search attribute type, which is one of the following: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. |
| `--sqlite-pragma` | No | **string[]** SQLite pragma statements in "PRAGMA=VALUE" format. |
| `--ui-asset-path` | No | **string** UI custom assets path. |
| `--ui-codec-endpoint` | No | **string** UI remote codec HTTP endpoint. |
| `--ui-disable-news-fetch` | No | **bool** Disable the Web UI newsfeed. When set, the UI will not request the newsfeed and the button to open the newsfeed panel is hidden. |
| `--ui-ip` | No | **string** IP address bound to the Web UI. Defaults to same as '--ip' value. |
| `--ui-port` | No | **int** Port for the Web UI. Defaults to '--port' value + 1000. |
| `--ui-public-path` | No | **string** The public base path for the Web UI. Defaults to `/`. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI task-queue command reference

Source: https://docs.temporal.io/cli/command-reference/task-queue

> Temporal Task Queue commands facilitate operations like describing poller info, displaying partitions, fetching compatible Build IDs, and determining Build ID reachability for effective Workflow and Activity management.

This page provides a reference for the `temporal` CLI `task-queue` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## config

Manage Task Queue configuration:

```
temporal task-queue config [command] [options]
```

Available commands:
- `get`: Retrieve the current configuration for a task queue
- `set`: Update the configuration for a task queue

### get

Retrieve the current configuration for a Task Queue:

```
temporal task-queue config get \
    --task-queue YourTaskQueue \
    --task-queue-type activity
```

This command returns the current configuration including:
- Queue rate limit: The overall rate limit of the task queue.
  This setting overrides the worker rate limit if set.
  Unless modified, this is the system-defined rate limit.
- Fairness key rate limit defaults: Default rate limits for fairness keys.
  If set, each individual fairness key will be limited to this rate,
  scaled by the weight of the fairness key.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |
| `--task-queue-type` | Yes | **string-enum** Task Queue type. Accepted values: workflow, activity, nexus. Accepted values: workflow, activity, nexus. |

### set

Update configuration settings for a Task Queue.

```
temporal task-queue config set \
    --task-queue YourTaskQueue \
    --task-queue-type activity \
    --namespace YourNamespace \
    --queue-rps-limit <requests_per_second:float> \
    --queue-rps-limit-reason <reason_string> \
    --fairness-key-rps-limit-default <requests_per_second:float> \
    --fairness-key-rps-limit-reason <reason_string> \
    --fairness-key-weight HighPriority=2.0 \
    --fairness-key-weight LowPriority=0.5
```

This command supports updating:
- Queue rate limits: Controls the overall rate limit of the task queue.
  This setting overrides the worker rate limit if set.
  Unless modified, this is the system-defined rate limit.
- Fairness key rate limit defaults: Sets default rate limits for fairness keys.
  If set, each individual fairness key will be limited to this rate,
  scaled by the weight of the fairness key.
- Fairness key weight overrides: Set custom weights for specific fairness keys.
  Weights control the relative share of capacity each key receives.

To unset a rate limit, pass in 'default', for example: --queue-rps-limit default
To unset a specific fairness weight, use --fairness-key-weight \<key\>=default
To unset all fairness weights, use --fairness-key-weight-clear-all

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--fairness-key-rps-limit-default` | No | **float\|default** Fairness key rate limit default in requests per second. Accepts a float; or 'default' to unset. |
| `--fairness-key-rps-limit-reason` | No | **string** Reason for fairness key rate limit update. |
| `--fairness-key-weight` | No | **string[]** Set or unset fairness key weight overrides in format key=weight or key=default. Use key=weight to set a positive weight value; use key=default to unset. Can be specified multiple times. Example: --fairness-key-weight HighPriority=2.0 --fairness-key-weight LowPriority=default. |
| `--fairness-key-weight-clear-all` | No | **bool** Unset all fairness key weight overrides. Cannot be used with --fairness-key-weight. |
| `--queue-rps-limit` | No | **float\|default** Queue rate limit in requests per second. Accepts a float; or 'default' to unset. |
| `--queue-rps-limit-reason` | No | **string** Reason for queue rate limit update. |
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |
| `--task-queue-type` | Yes | **string-enum** Task Queue type. Accepted values: workflow, activity, nexus. Accepted values: workflow, activity, nexus. |

## describe

Display a list of active Workers that have recently polled a Task Queue. The
Temporal Server records each poll request time. A `LastAccessTime` over one
minute may indicate the Worker is at capacity or has shut down. Temporal
Workers are removed if 5 minutes have passed since the last poll request.

```
temporal task-queue describe \
  --task-queue YourTaskQueue
```

This command provides poller information for a given Task Queue.
Workflow and Activity polling use separate Task Queues:

```
temporal task-queue describe \
    --task-queue YourTaskQueue \
    --task-queue-type "activity"
```

This command provides the following task queue statistics:
- `ApproximateBacklogCount`: The approximate number of tasks backlogged in this
  task queue. May count expired tasks but eventually converges to the right
  value.
- `ApproximateBacklogAge`: Approximate age of the oldest task in the backlog,
  based on its creation time, measured in seconds.
- `TasksAddRate`: Approximate rate at which tasks are being added to the task
  queue, measured in tasks per second, averaged over the last 30 seconds.
  Includes tasks dispatched immediately without going to the backlog
  (sync-matched tasks), as well as tasks added to the backlog. (See note below.)
- `TasksDispatchRate`: Approximate rate at which tasks are being dispatched from
  the task queue, measured in tasks per second, averaged over the last 30
  seconds.  Includes tasks dispatched immediately without going to the backlog
  (sync-matched tasks), as well as tasks added to the backlog. (See note below.)
- `BacklogIncreaseRate`: Approximate rate at which the backlog size is
  increasing (if positive) or decreasing (if negative), measured in tasks per
  second, averaged over the last 30 seconds.  This is roughly equivalent to:
  `TasksAddRate` - `TasksDispatchRate`.

NOTE: The `TasksAddRate` and `TasksDispatchRate` metrics may differ from the
actual rate of add/dispatch, because tasks may be dispatched eagerly to an
available worker, or may apply only to specific workers (they are "sticky").
Such tasks are not counted by these metrics. Despite the inaccuracy of
these two metrics, the derived metric of `BacklogIncreaseRate` is accurate
for backlogs older than a few seconds.

Safely retire Workers assigned a Build ID by checking reachability across
all task types. Use the flag `--report-reachability`:

```
temporal task-queue describe \
    --task-queue YourTaskQueue \
    --select-build-id "YourBuildId" \
    --report-reachability
```

Task reachability information is returned for the requested versions and all
task types, which can be used to safely retire Workers with old code versions,
provided that they were assigned a Build ID.

Note that task reachability status is deprecated in favor of Drainage Status
(ie. of a Drained or Draining Worker Deployment Version) and will be removed
in a future release. Also, determining task reachability incurs a non-trivial
computing cost.

Task reachability states are reported per build ID. The state may be one of the
following:

- `Reachable`: using the current versioning rules, the Build ID may be used
  by new Workflow Executions or Activities OR there are currently open
  Workflow or backlogged Activity tasks assigned to the queue.
- `ClosedWorkflowsOnly`: the Build ID does not have open Workflow Executions
  and can't be reached by new Workflow Executions. It MAY have closed
  Workflow Executions within the Namespace retention period.
- `Unreachable`: this Build ID is not used for new Workflow Executions and
  isn't used by any existing Workflow Execution within the retention period.

Task reachability is eventually consistent. You may experience a delay until
reachability converges to the most accurate value. This is designed to act
in the most conservative way until convergence. For example, `Reachable` is
more conservative than `ClosedWorkflowsOnly`.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--disable-stats` | No | **bool** Disable task queue statistics. |
| `--legacy-mode` | No | **bool** Enable a legacy mode for servers that do not support rules-based worker versioning. This mode only provides pollers info. |
| `--partitions-legacy` | No | **int** Query partitions 1 through `N`. Experimental/Temporary feature. Legacy mode only. |
| `--report-config` | No | **bool** Include task queue configuration in the response. When enabled, the command will return the current rate limit configuration for the task queue. |
| `--report-reachability` | No | **bool** Display task reachability information. |
| `--select-all-active` | No | **bool** Include all active versions. A version is active if it had new tasks or polls recently. |
| `--select-build-id` | No | **string[]** Filter the Task Queue based on Build ID. |
| `--select-unversioned` | No | **bool** Include the unversioned queue. |
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |
| `--task-queue-type` | No | **string-enum[]** Task Queue type. If not specified, all types are reported. Accepted values: workflow, activity, nexus. |
| `--task-queue-type-legacy` | No | **string-enum** Task Queue type (legacy mode only). Accepted values: workflow, activity. |

## get-build-id-reachability

```
+-----------------------------------------------------------------------------+
| CAUTION: This command is deprecated and will be removed in a later release. |
+-----------------------------------------------------------------------------+
```

Show if a given Build ID can be used for new, existing, or closed Workflows
in Namespaces that support Worker versioning:

```
temporal task-queue get-build-id-reachability \
    --task-queue YourTaskQueue \
    --build-id "YourBuildId"
```

You can specify the `--build-id` and `--task-queue` flags multiple times. If
`--task-queue` is omitted, the command checks Build ID reachability against
all Task Queues.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | No | **string[]** One or more Build ID strings. Can be passed multiple times. |
| `--reachability-type` | No | **string-enum** Reachability filter. `open`: reachable by one or more open workflows. `closed`: reachable by one or more closed workflows. `existing`: reachable by either. New Workflow Executions reachable by a Build ID are always reported. Accepted values: open, closed, existing. |
| `--task-queue`, `-t` | No | **string[]** Search only the specified task queue(s). Can be passed multiple times. |

## get-build-ids

```
+-----------------------------------------------------------------------------+
| CAUTION: This command is deprecated and will be removed in a later release. |
+-----------------------------------------------------------------------------+
```

Fetch sets of compatible Build IDs for specified Task Queues and display their
information:

```
temporal task-queue get-build-ids \
    --task-queue YourTaskQueue
```

This command is limited to Namespaces that support Worker versioning.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--max-sets` | No | **int** Max return count. Use 1 for default major version. Use 0 for all sets. |
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |

## list-partition

Display a Task Queue's partition list with assigned matching nodes:

```
temporal task-queue list-partition \
    --task-queue YourTaskQueue
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |

## update-build-ids

```
+-----------------------------------------------------------------------------+
| CAUTION: This command is deprecated and will be removed in a later release. |
+-----------------------------------------------------------------------------+
```

Add or change a Task Queue's compatible Build IDs for Namespaces using Worker
versioning:

```
temporal task-queue update-build-ids [subcommands] [options] \
    --task-queue YourTaskQueue
```

### add-new-compatible

Add a compatible Build ID to a Task Queue's existing version set. Provide an
existing Build ID and a new Build ID:

```
temporal task-queue update-build-ids add-new-compatible \
    --task-queue YourTaskQueue \
    --existing-compatible-build-id "YourExistingBuildId" \
    --build-id "YourNewBuildId"
```

The new ID is stored in the set containing the existing ID and becomes the new
default for that set.

This command is limited to Namespaces that support Worker versioning.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Build ID to be added. |
| `--existing-compatible-build-id` | Yes | **string** Pre-existing Build ID in this Task Queue. |
| `--set-as-default` | No | **bool** Set the expanded Build ID set as the Task Queue default. |
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |

### add-new-default

```
+-----------------------------------------------------------------------------+
| CAUTION: This command is deprecated and will be removed in a later release. |
+-----------------------------------------------------------------------------+
```

Create a new Task Queue Build ID set, add a Build ID to it, and make it the
overall Task Queue default. The new set will be incompatible with previous
sets and versions.

```
temporal task-queue update-build-ids add-new-default \
    --task-queue YourTaskQueue \
    --build-id "YourNewBuildId"
```

```
+------------------------------------------------------------------------+
| NOTICE: This command is limited to Namespaces that support Worker      |
| versioning. Worker versioning is experimental. Versioning commands are |
| subject to change.                                                     |
+------------------------------------------------------------------------+
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Build ID to be added. |
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |

### promote-id-in-set

```
+-----------------------------------------------------------------------------+
| CAUTION: This command is deprecated and will be removed in a later release. |
+-----------------------------------------------------------------------------+
```

Establish an existing Build ID as the default in its Task Queue set. New tasks
compatible with this set will now be dispatched to this ID:

```
temporal task-queue update-build-ids promote-id-in-set \
    --task-queue YourTaskQueue \
    --build-id "YourBuildId"
```

```
+------------------------------------------------------------------------+
| NOTICE: This command is limited to Namespaces that support Worker      |
| versioning. Worker versioning is experimental. Versioning commands are |
| subject to change.                                                     |
+------------------------------------------------------------------------+
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Build ID to set as default. |
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |

### promote-set

```
+-----------------------------------------------------------------------------+
| CAUTION: This command is deprecated and will be removed in a later release. |
+-----------------------------------------------------------------------------+
```

Promote a Build ID set to be the default on a Task Queue. Identify the set by
providing a Build ID within it. If the set is already the default, this
command has no effect:

```
temporal task-queue update-build-ids promote-set \
    --task-queue YourTaskQueue \
    --build-id "YourBuildId"
```

```
+------------------------------------------------------------------------+
| NOTICE: This command is limited to Namespaces that support Worker      |
| versioning. Worker versioning is experimental. Versioning commands are |
| subject to change.                                                     |
+------------------------------------------------------------------------+
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Build ID within the promoted set. |
| `--task-queue`, `-t` | Yes | **string** Task Queue name. |

## versioning

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Provides commands to add, list, remove, or replace Worker Build ID assignment
and redirect rules associated with Task Queues:

```
temporal task-queue versioning [subcommands] [options] \
    --task-queue YourTaskQueue
```

Task Queues support the following versioning rules and policies:

- Assignment Rules: manage how new executions are assigned to run on specific
  Worker Build IDs. Each Task Queue stores a list of ordered Assignment Rules,
  which are evaluated from first to last. Assignment Rules also allow for
  gradual rollout of new Build IDs by setting ramp percentage.
- Redirect Rules: automatically assign work for a source Build ID to a target
  Build ID. You may add at most one redirect rule for each source Build ID.
  Redirect rules require that a target Build ID is fully compatible with
  the source Build ID.

### add-redirect-rule

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Add a new redirect rule for a given Task Queue. You may add at most one
redirect rule for each distinct source build ID:

```
temporal task-queue versioning add-redirect-rule \
    --task-queue YourTaskQueue \
    --source-build-id "YourSourceBuildID" \
    --target-build-id "YourTargetBuildID"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--source-build-id` | Yes | **string** Source build ID. |
| `--target-build-id` | Yes | **string** Target build ID. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

### commit-build-id

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Complete a Build ID's rollout and clean up unnecessary rules that might have
been created during a gradual rollout:

```
temporal task-queue versioning commit-build-id \
    --task-queue YourTaskQueue
    --build-id "YourBuildId"
```

This command automatically applies the following atomic changes:

- Adds an unconditional assignment rule for the target Build ID at the
  end of the list.
- Removes all previously added assignment rules to the given target
  Build ID.
- Removes any unconditional assignment rules for other Build IDs.

Rejects requests when there have been no recent pollers for this Build ID.
This prevents committing invalid Build IDs. Use the `--force` option to
override this validation.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Target build ID. |
| `--force` | No | **bool** Bypass recent-poller validation. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

### delete-assignment-rule

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Deletes a rule identified by its index in the Task Queue's list of assignment
rules.

```
temporal task-queue versioning delete-assignment-rule \
    --task-queue YourTaskQueue \
    --rule-index YourIntegerRuleIndex
```

By default, the Task Queue must retain one unconditional rule, such as "no
hint filter" or "percentage". Otherwise, the delete operation is rejected.
Use the `--force` option to override this validation.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--force` | No | **bool** Bypass one-unconditional-rule validation. |
| `--rule-index`, `-i` | Yes | **int** Position of the assignment rule to be replaced. Requests for invalid indices will fail. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

### delete-redirect-rule

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Deletes the routing rule for the given source Build ID.

```
temporal task-queue versioning delete-redirect-rule \
    --task-queue YourTaskQueue \
    --source-build-id "YourBuildId"
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--source-build-id` | Yes | **string** Source Build ID. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

### get-rules

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Retrieve all the Worker Build ID assignments and redirect rules associated
with a Task Queue:

```
temporal task-queue versioning get-rules \
    --task-queue YourTaskQueue
```

Task Queues support the following versioning rules:

- Assignment Rules: manage how new executions are assigned to run on specific
  Worker Build IDs. Each Task Queue stores a list of ordered Assignment Rules,
  which are evaluated from first to last. Assignment Rules also allow for
  gradual rollout of new Build IDs by setting ramp percentage.
- Redirect Rules: automatically assign work for a source Build ID to a target
  Build ID. You may add at most one redirect rule for each source Build ID.
  Redirect rules require that a target Build ID is fully compatible with
  the source Build ID.

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

### insert-assignment-rule

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Inserts a new assignment rule for this Task Queue. Rules are evaluated in
order, starting from index 0. The first applicable rule is applied, and the
rest ignored:

```
temporal task-queue versioning insert-assignment-rule \
    --task-queue YourTaskQueue \
    --build-id "YourBuildId"
```

If you do not specify a `--rule-index`, this command inserts at index 0.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Target Build ID. |
| `--percentage` | No | **int** Traffic percent to send to target Build ID. |
| `--rule-index`, `-i` | No | **int** Insertion position. Ranges from 0 (insert at start) to count (append). Any number greater than the count is treated as "append". |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

### replace-assignment-rule

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Change an assignment rule for this Task Queue. By default, this enforces one
unconditional rule (no hint filter or percentage). Otherwise, the operation
will be rejected. Set `force` to true to bypass this validation.

```
temporal task-queue versioning replace-assignment-rule \
    --task-queue YourTaskQueue \
    --rule-index AnIntegerIndex \
    --build-id "YourBuildId"
```

To assign multiple assignment rules to a single Build ID, use
'insert-assignment-rule'.

To update the percent:

```
temporal task-queue versioning replace-assignment-rule \
    --task-queue YourTaskQueue \
    --rule-index AnIntegerIndex \
    --build-id "YourBuildId" \
    --percentage AnIntegerPercent
```

Percent may vary between 0 and 100 (default).

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Target Build ID. |
| `--force` | No | **bool** Bypass the validation that one unconditional rule remains. |
| `--percentage` | No | **int** Divert percent of traffic to target Build ID. |
| `--rule-index`, `-i` | Yes | **int** Position of the assignment rule to be replaced. Requests for invalid indices will fail. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

### replace-redirect-rule

```
+-------------------------------------------------------------+
| CAUTION: This API has been deprecated by Worker Deployment. |
+-------------------------------------------------------------+
```

Updates a Build ID's redirect rule on a Task Queue by replacing its target
Build ID:

```
temporal task-queue versioning replace-redirect-rule \
    --task-queue YourTaskQueue \
    --source-build-id YourSourceBuildId \
    --target-build-id YourNewTargetBuildId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--source-build-id` | Yes | **string** Source Build ID. |
| `--target-build-id` | Yes | **string** Target Build ID. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI worker command reference

Source: https://docs.temporal.io/cli/command-reference/worker

> Learn how to read or modify state associated with a Worker, such as Worker Deployments.

This page provides a reference for the `temporal` CLI `worker` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## count

Show a count of workers in a namespace. Use `--query` to count a subset:

```
temporal worker count --namespace YourNamespace --query 'TaskQueue="YourTaskQueue"'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--include-system-workers` | No | **bool** Include system workers created by the server. |
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. |

## deployment

Deployment commands perform operations on Worker Deployments:

```
temporal worker deployment [command] [options]
```

For example:

```
temporal worker deployment list
```

Lists the Deployments in the client's namespace.

Arguments can be Worker Deployment Versions associated with
a Deployment, specified using the Deployment name and Build ID.

For example:

```
temporal worker deployment set-current-version \
         --deployment-name YourDeploymentName --build-id YourBuildID
```

Sets the current Deployment Version for a given Deployment.

### create

Create a new Worker Deployment:

```
temporal worker deployment create [options]
```

Worker Deployments are lazily created the first time a Worker polls the
Temporal Server and specifies a VersionOverride. However, if you need to
pre-define a compute configuration (for instance to set up a serverless
Worker), you need to call `temporal worker deployment create-version` and
pass in the name of the Worker Deployment. The `temporal worker
deployment create` command allows you to pre-define a Worker Deployment
so that calls to `temporal worker deployment create-version` will
succeed.

If a Worker Deployment with the supplied name already exists, this
command will return an error.

Note: This is an experimental feature and may change in the future.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name`, `-d` | Yes | **string** Name for a Worker Deployment. |

### create-version

Create a new Worker Deployment Version:

```
temporal worker deployment create-version [options]
```

Configure a Worker Deployment Version's compute configuration as needed.
For example, pass compute provider information for an AWS Lambda function
that spawns a Worker in the Worker Deployment:

```
temporal worker deployment create-version \
    --namespace YourNamespaceName \
    --deployment-name YourDeploymentName \
    --build-id YourBuildID \
    --aws-lambda-function-arn LambdaFunctionARN \
    --aws-lambda-assume-role-arn LambdaAssumeRoleARN \
    --aws-lambda-assume-role-external-id LambdaAssumeRoleExternalID
```

Or pass compute provider information for an AWS Bedrock Agentcore Runtime
that spawns a Worker in the Worker Deployment:

```
temporal worker deployment create-version \
    --namespace YourNamespaceName \
    --deployment-name YourDeploymentName \
    --build-id YourBuildID \
    --aws-agentcore-endpoint-arn AgentcoreRuntimeEndpointARN \
    --aws-agentcore-assume-role-arn AgentcoreAssumeRoleARN \
    --aws-agentcore-assume-role-external-id AgentcoreAssumeRoleExternalID
```

Or pass compute provider information for a GCP Cloud Run worker pool
that spawns a Worker in the Worker Deployment:

```
temporal worker deployment create-version \
    --namespace YourNamespaceName \
    --deployment-name YourDeploymentName \
    --build-id YourBuildID \
    --gcp-cloud-run-project YourGCPProject \
    --gcp-cloud-run-region us-central1 \
    --gcp-cloud-run-worker-pool YourWorkerPool \
    --gcp-cloud-run-service-account customer-sa@proj.iam.gserviceaccount.com \
    --gcp-cloud-run-min-instances 1 \
    --gcp-cloud-run-max-instances 3 \
    --gcp-cloud-run-initial-instances 1 \
    --gcp-cloud-run-utilization-target 0.75 \
    --gcp-cloud-run-scale-down-stabilization-duration 5m
```

If a Worker Deployment Version with the supplied BuildID already exists,
this command will return an error.

Returns an error if all compute configuration fields are empty.

Note: This is an experimental feature and may change in the future.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--aws-agentcore-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Agentcore Runtime that spawns a new Worker in this Worker Deployment Version. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. |
| `--aws-agentcore-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-agentcore-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. |
| `--aws-agentcore-endpoint-arn` | No | **string** AWS Bedrock Agentcore Runtime endpoint ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. The endpoint ARN encodes the runtime, endpoint name, and region. |
| `--aws-agentcore-skip-role-and-external-id` | No | **bool** When --aws-agentcore-endpoint-arn is specified, --aws-agentcore-assume-role-arn and --aws-agentcore-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. |
| `--aws-lambda-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Lambda function that spawns a new Worker in this Worker Deployment Version. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. |
| `--aws-lambda-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-lambda-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. |
| `--aws-lambda-function-arn` | No | **string** Qualified (contains version suffix) or unqualified AWS Lambda function ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. |
| `--aws-lambda-skip-role-and-external-id` | No | **bool** When --aws-lambda-function-arn is specified, --aws-lambda-assume-role-arn and --aws-lambda-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. |
| `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. |
| `--deployment-name` | Yes | **string** Name of the Worker Deployment. |
| `--gcp-cloud-run-initial-instances` | No | **int** Number of Cloud Run worker pool instances the scaler starts with. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together, and this value must be between the min and max (inclusive). Defaults to 0 when unset. Only valid with --gcp-cloud-run-worker-pool. |
| `--gcp-cloud-run-max-instances` | No | **int** Maximum number of Cloud Run worker pool instances the scaler may scale up to. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Defaults to 30 when unset. Only valid with --gcp-cloud-run-worker-pool. |
| `--gcp-cloud-run-min-instances` | No | **int** Minimum number of Cloud Run worker pool instances the scaler will maintain. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Defaults to 0 when unset. Only valid with --gcp-cloud-run-worker-pool. |
| `--gcp-cloud-run-project` | No | **string** GCP project ID hosting the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. |
| `--gcp-cloud-run-region` | No | **string** Region of the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. |
| `--gcp-cloud-run-scale-down-stabilization-duration` | No | **duration** Duration the scaler waits after it last saw unmet task demand before it may scale the Cloud Run worker pool down. Raise this to keep the pool from scaling down before long-running or bursty activities finish. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. A value of 0s disables the wait. Defaults to 90s when unset. Only valid with --gcp-cloud-run-worker-pool. |
| `--gcp-cloud-run-service-account` | No | **string** Customer GCP service account the Temporal server impersonates to manage the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. |
| `--gcp-cloud-run-utilization-target` | No | **float** Target average worker utilization the scaler aims for, as a fraction in the range (0, 1]. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Lower values keep more spare capacity per worker. Defaults to 0.8 when unset. Only valid with --gcp-cloud-run-worker-pool. |
| `--gcp-cloud-run-worker-pool` | No | **string** GCP Cloud Run worker pool name to scale when there are no active pollers for task queue targets in the Worker Deployment. |

### delete

Remove a Worker Deployment given its Deployment Name.
A Deployment can only be deleted if it has no Version in it.

```
temporal worker deployment delete [options]
```

For example, setting the user identity that removed the deployment:

```
temporal worker deployment delete \
    --name YourDeploymentName \
    --identity YourIdentity

```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name`, `-d` | Yes | **string** Name for a Worker Deployment. |

### delete-version

Remove a Worker Deployment Version given its fully-qualified identifier.
This is rarely needed during normal operation
since unused Versions are eventually garbage collected.
The client can delete a Version only when all of the following conditions
are met:
  - It is not the Current or Ramping Version for this Deployment.
  - It has no active pollers, i.e., none of the task queues in the
  Version have pollers.
  - It is not draining. This requirement can be ignored with the option
`--skip-drainage`.
```
temporal worker deployment delete-version [options]
```

For example, skipping the drainage restriction:

```
temporal worker deployment delete-version \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --skip-drainage
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. |
| `--deployment-name` | Yes | **string** Name of the Worker Deployment. |
| `--skip-drainage` | No | **bool** Ignore the deletion requirement of not draining. |

### describe

Describe properties of a Worker Deployment, such as the versions
associated with it, routing information of new or existing tasks
executed by this deployment, or its creation time.

```
temporal worker deployment describe [options]
```

For example, to describe a deployment `YourDeploymentName` in the default
namespace:

```
temporal worker deployment describe \
    --name YourDeploymentName
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--name`, `-d` | Yes | **string** Name for a Worker Deployment. |

### describe-version

Describe properties of a Worker Deployment Version, such as the task
queues polled by workers in this Deployment Version, or drainage
information required to safely decommission workers, or user-provided
metadata, or its creation/modification time.

```
temporal worker deployment describe-version [options]
```

For example, to describe a deployment version  in a deployment
`YourDeploymentName`, with Build ID `YourBuildID`, and in the default
namespace:

```
temporal worker deployment describe-version \
    --deployment-name YourDeploymentName --build-id YourBuildID
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. |
| `--deployment-name` | Yes | **string** Name of the Worker Deployment. |
| `--report-task-queue-stats` | No | **bool** Report stats for task queues that are present in this version. |

### list

List existing Worker Deployments in the client's namespace.

```
temporal worker deployment list [options]
```

For example, listing Deployments in YourDeploymentNamespace:

```
temporal worker deployment list \
    --namespace YourDeploymentNamespace
```

Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command.

### manager-identity

Manager Identity commands change the `ManagerIdentity` of a Worker Deployment:

```
temporal worker deployment manager-identity [command] [options]
```

When present, `ManagerIdentity` is the identity of the user that has the
exclusive right to make changes to this Worker Deployment. Empty by default.
When set, users whose identity does not match the `ManagerIdentity` will not
be able to change the Worker Deployment.

This is especially useful in environments where multiple users (such as CLI
users and automated controllers) may interact with the same Worker Deployment.
`ManagerIdentity` allows different users to communicate with one another about
who is expected to make changes to the Worker Deployment.

The current Manager Identity is returned with `describe`:
```
 temporal worker deployment describe \
    --deployment-name YourDeploymentName
```

#### set

Set the `ManagerIdentity` of a Worker Deployment given its Deployment Name.

When present, `ManagerIdentity` is the identity of the user that has the
exclusive right to make changes to this Worker Deployment. Empty by default.
When set, users whose identity does not match the `ManagerIdentity` will not
be able to change the Worker Deployment.

This is especially useful in environments where multiple users (such as CLI
users and automated controllers) may interact with the same Worker Deployment.
`ManagerIdentity` allows different users to communicate with one another about
who is expected to make changes to the Worker Deployment.

```
temporal worker deployment manager-identity set [options]
```

For example:

```
temporal worker deployment manager-identity set \
   --deployment-name DeploymentName \
   --self \
   --identity YourUserIdentity # optional, populated by CLI if not provided
```

Sets the Manager Identity of the Deployment to the identity of the user making
this request. If you don't specifically pass an identity field, the CLI will
generate your identity for you.

For example:
```
temporal worker deployment manager-identity set \
   --deployment-name DeploymentName \
   --manager-identity NewManagerIdentity
```

Sets the Manager Identity of the Deployment to any string.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--deployment-name` | No | **string** Name for a Worker Deployment. Required. |
| `--manager-identity` | No | **string** New Manager Identity. Required unless --self is specified. |
| `--self` | No | **bool** Set Manager Identity to the identity of the user submitting this request. Required unless --manager-identity is specified. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm set Manager Identity. |

#### unset

Unset the `ManagerIdentity` of a Worker Deployment given its Deployment Name.

When present, `ManagerIdentity` is the identity of the user that has the
exclusive right to make changes to this Worker Deployment. Empty by default.
When set, users whose identity does not match the `ManagerIdentity` will not
be able to change the Worker Deployment.

This is especially useful in environments where multiple users (such as CLI
users and automated controllers) may interact with the same Worker Deployment.
`ManagerIdentity` allows different users to communicate with one another about
who is expected to make changes to the Worker Deployment.

```
temporal worker deployment manager-identity unset [options]
```

For example:

```
temporal worker deployment manager-identity unset \
   --deployment-name YourDeploymentName
```

Clears the Manager Identity field for a given Deployment.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--deployment-name` | No | **string** Name for a Worker Deployment. Required. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm unset Manager Identity. |

### set-current-version

Set the Current Version for a Deployment.
When a Version is current, Workers of that Deployment Version will receive
tasks from new Workflows, and from existing AutoUpgrade Workflows that
are running on this Deployment.

If not all the expected Task Queues are being polled by Workers in the
new Version the request will fail. To override this protection use
`--ignore-missing-task-queues`. Note that this would ignore task queues
in a deployment that are not yet discovered, leading to inconsistent task
queue configuration.

```
temporal worker deployment set-current-version [options]
```

For example, to set the Current Version of a deployment
`YourDeploymentName`, with a version with Build ID `YourBuildID`, and
in the default namespace:

```
temporal worker deployment set-current-version \
    --deployment-name YourDeploymentName --build-id YourBuildID
```

The target of set-current-version can also be unversioned workers:

```
temporal worker deployment set-current-version \
    --deployment-name YourDeploymentName --unversioned
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--allow-no-pollers` | No | **bool** Override protection and set version as current even if it has no pollers. |
| `--build-id` | No | **string** Build ID of the Worker Deployment Version. Required unless --unversioned is specified. |
| `--deployment-name` | Yes | **string** Name of the Worker Deployment. |
| `--ignore-missing-task-queues` | No | **bool** Override protection to accidentally remove task queues. |
| `--unversioned` | No | **bool** Set unversioned workers as the target version. Cannot be used with --build-id. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm set Current Version. |

### set-ramping-version

Set the Ramping Version and Percentage for a Deployment.

The Ramping Version can be set using deployment name and build ID,
or set to unversioned workers using the --unversioned flag.

The Ramping Percentage is a float with values in the range [0, 100].
A value of 100 does not make the Ramping Version Current, use
`set-current-version` instead.

To remove a Ramping Version use the flag `--delete`.

If not all the expected Task Queues are being polled by Workers in the
new Ramping Version the request will fail. To override this protection use
`--ignore-missing-task-queues`. Note that this would ignore task queues
in a deployment that are not yet discovered, leading to inconsistent task
queue configuration.

```
temporal worker deployment set-ramping-version [options]
```

For example, to set the Ramping Version of a deployment
`YourDeploymentName`, with a version with Build ID `YourBuildID`, with
10 percent of tasks redirected to this version, and
using the default namespace:

```
temporal worker deployment set-ramping-version \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --percentage 10.0
```

And to remove that ramping:
```
temporal worker deployment set-ramping-version \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --delete
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--allow-no-pollers` | No | **bool** Override protection and set version as ramping even if it has no pollers. |
| `--build-id` | No | **string** Build ID of the Worker Deployment Version. Required unless --unversioned is specified. |
| `--delete` | No | **bool** Delete the Ramping Version. |
| `--deployment-name` | Yes | **string** Name of the Worker Deployment. |
| `--ignore-missing-task-queues` | No | **bool** Override protection to accidentally remove task queues. |
| `--percentage` | No | **float** Percentage of tasks redirected to the Ramping Version. Valid range [0,100]. |
| `--unversioned` | No | **bool** Set unversioned workers as the target version. Cannot be used with --build-id. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm set Ramping Version. |

### update-version-compute-config

Update compute configuration associated with a Worker Deployment
Version.

For example, to update the AWS Lambda function ARN associated with an
existing Worker Deployment Version:

```
 temporal worker deployment update-version-compute-config \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --aws-lambda-function-arn UpdatedLambdaFunctionARN
```

To update the AWS IAM role ARN that is assumed by the serverless worker
manager associated with an existing Worker Deployment Version:

```
 temporal worker deployment update-version-compute-config \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --aws-lambda-assume-role-arn UpdatedRoleARN
```

To update the AWS Bedrock Agentcore Runtime endpoint associated with an
existing Worker Deployment Version:

```
 temporal worker deployment update-version-compute-config \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --aws-agentcore-endpoint-arn UpdatedAgentcoreRuntimeEndpointARN \
    --aws-agentcore-assume-role-arn UpdatedRoleARN \
    --aws-agentcore-assume-role-external-id UpdatedExternalID
```

To update the GCP Cloud Run worker pool associated with an existing
Worker Deployment Version:

```
 temporal worker deployment update-version-compute-config \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --gcp-cloud-run-project YourGCPProject \
    --gcp-cloud-run-region us-central1 \
    --gcp-cloud-run-worker-pool UpdatedWorkerPool \
    --gcp-cloud-run-service-account customer-sa@proj.iam.gserviceaccount.com \
    --gcp-cloud-run-min-instances 1 \
    --gcp-cloud-run-max-instances 3 \
    --gcp-cloud-run-initial-instances 1 \
    --gcp-cloud-run-utilization-target 0.75 \
    --gcp-cloud-run-scale-down-stabilization-duration 5m
```

To update only the scaling settings on an existing GCP Cloud Run Worker
Deployment Version, supply the five scaler flags without the provider
fields (all five must be set together):

```
 temporal worker deployment update-version-compute-config \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --gcp-cloud-run-min-instances 1 \
    --gcp-cloud-run-max-instances 3 \
    --gcp-cloud-run-initial-instances 1 \
    --gcp-cloud-run-utilization-target 0.75 \
    --gcp-cloud-run-scale-down-stabilization-duration 5m
```

Provider fields are only required when changing the compute provider.
Switching the provider resets the scaling settings for the new provider.

If --remove is specified, the compute configuration for the Worker
Deployment Version will be removed:

```
 temporal worker deployment update-version-compute-config \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --remove
```

If a Worker Deployment Version with the supplied BuildID does not exist,
this command will return an error.

Note: This is an experimental feature and may change in the future.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--aws-agentcore-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Agentcore Runtime that spawns a new Worker in this Worker Deployment Version. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. |
| `--aws-agentcore-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-agentcore-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. |
| `--aws-agentcore-endpoint-arn` | No | **string** AWS Bedrock Agentcore Runtime endpoint ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. The endpoint ARN encodes the runtime, endpoint name, and region. |
| `--aws-agentcore-skip-role-and-external-id` | No | **bool** When --aws-agentcore-endpoint-arn is specified, --aws-agentcore-assume-role-arn and --aws-agentcore-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. |
| `--aws-lambda-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Lambda function that spawns a new Worker in this Worker Deployment Version. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. |
| `--aws-lambda-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-lambda-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. |
| `--aws-lambda-function-arn` | No | **string** Qualified (contains version suffix) or unqualified AWS Lambda function ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. |
| `--aws-lambda-skip-role-and-external-id` | No | **bool** When --aws-lambda-function-arn is specified, --aws-lambda-assume-role-arn and --aws-lambda-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. |
| `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. |
| `--deployment-name` | Yes | **string** Name of the Worker Deployment. |
| `--gcp-cloud-run-initial-instances` | No | **int** Number of Cloud Run worker pool instances the scaler starts with. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together, and this value must be between the min and max (inclusive). If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. |
| `--gcp-cloud-run-max-instances` | No | **int** Maximum number of Cloud Run worker pool instances the scaler may scale up to. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. |
| `--gcp-cloud-run-min-instances` | No | **int** Minimum number of Cloud Run worker pool instances the scaler will maintain. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. |
| `--gcp-cloud-run-project` | No | **string** GCP project ID hosting the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. |
| `--gcp-cloud-run-region` | No | **string** Region of the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. |
| `--gcp-cloud-run-scale-down-stabilization-duration` | No | **duration** Duration the scaler waits after it last saw unmet task demand before it may scale the Cloud Run worker pool down. Raise this to keep the pool from scaling down before long-running or bursty activities finish. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. A value of 0s disables the wait. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. |
| `--gcp-cloud-run-service-account` | No | **string** Customer GCP service account the Temporal server impersonates to manage the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. |
| `--gcp-cloud-run-utilization-target` | No | **float** Target average worker utilization the scaler aims for, as a fraction in the range (0, 1]. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Lower values keep more spare capacity per worker. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. |
| `--gcp-cloud-run-worker-pool` | No | **string** GCP Cloud Run worker pool name to scale when there are no active pollers for task queue targets in the Worker Deployment. |
| `--remove` | No | **bool** Removes any compute configuration associated with this Worker Deployment Version. |

### update-version-metadata

Update metadata associated with a Worker Deployment Version.

For example:

```
 temporal worker deployment update-version-metadata \
    --deployment-name YourDeploymentName --build-id YourBuildID \
    --metadata bar=1 \
    --metadata foo=true
```

The current metadata is also returned with `describe-version`:
```
 temporal worker deployment describe-version \
    --deployment-name YourDeploymentName --build-id YourBuildID \
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. |
| `--deployment-name` | Yes | **string** Name of the Worker Deployment. |
| `--metadata` | No | **string[]** Set deployment metadata using `KEY="VALUE"` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey={"your": "value"}` Can be passed multiple times. |
| `--remove-entries` | No | **string[]** Keys of entries to be deleted from metadata. Can be passed multiple times. |

## describe

Look up information of a specific worker.

```
temporal worker describe --namespace YourNamespace --worker-instance-key YourKey
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--worker-instance-key` | Yes | **string** Worker instance key to describe. |

## list

Get a list of workers to the specified namespace.

```
temporal worker list --namespace YourNamespace --query 'TaskQueue="YourTaskQueue"'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--include-system-workers` | No | **bool** Include system workers that are created implicitly by the server. By default, system workers are excluded. |
| `--limit` | No | **int** Maximum number of workers to display. |
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# Temporal CLI workflow command reference

Source: https://docs.temporal.io/cli/command-reference/workflow

> Temporal Workflow commands enable operations on Workflow Executions, such as cancel, count, delete, describe, execute, list, update-options, query, reset, reset-batch, show, signal, stack, start, terminate, trace, and update, enhancing efficiency and control.

This page provides a reference for the `temporal` CLI `workflow` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand.

## cancel

Canceling a running Workflow Execution records a
`WorkflowExecutionCancelRequested` event in the Event History. The Service
schedules a new Command Task, and the Workflow Execution performs any cleanup
work supported by its implementation.

Use the Workflow ID to cancel an Execution:

```
temporal workflow cancel \
    --workflow-id YourWorkflowId
```

A visibility Query lets you send bulk cancellations to Workflow Executions
matching the results:

```
temporal workflow cancel \
    --query YourQuery
```

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. |

## count

Show a count of Workflow Executions, regardless of execution state (running,
terminated, etc). Use `--query` to select a subset of Workflow Executions:

```
temporal workflow count \
    --query YourQuery
```

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. |

## delete

Delete a Workflow Execution and its Event History:

```
temporal workflow delete \
    --workflow-id YourWorkflowId
```

The removal executes asynchronously. If the Execution is Running, the Service
terminates it before deletion.

WARNING: Deleting Workflow Executions in a global Namespace removes them from
all replicas. Requests sent to a passive cluster are forwarded to the active
cluster by default; to target the passive cluster directly, specify
`--grpc-meta xdc-redirection=false`.

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. |

## describe

Display information about a specific Workflow Execution:

```
temporal workflow describe \
    --workflow-id YourWorkflowId
```

Show the Workflow Execution's auto-reset points:

```
temporal workflow describe \
    --workflow-id YourWorkflowId \
    --reset-points true
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--raw` | No | **bool** Print properties without changing their format. |
| `--reset-points` | No | **bool** Show auto-reset points only. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## execute

Establish a new Workflow Execution and direct its progress to stdout. The
command blocks and returns when the Workflow Execution completes. If your
Workflow requires input, pass valid JSON:

```
temporal workflow execute
    --workflow-id YourWorkflowId \
    --type YourWorkflow \
    --task-queue YourTaskQueue \
    --input '{"some-key": "some-value"}'
```

Use `--event-details` to relay updates to the command-line output in JSON
format. When using JSON output (`--output json`), this includes the entire
"history" JSON key for the run.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--cron` | No | **string** Cron schedule for the Workflow. |
| `--detailed` | No | **bool** Display events as sections instead of table. Does not apply to JSON output. |
| `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. |
| `--fail-existing` | No | **bool** Fail if the Workflow already exists. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. |
| `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. |
| `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. |
| `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. |
| `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ |
| `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Workflow Task queue. |
| `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. |
| `--type` | Yes | **string** Workflow Type name. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. |

## execute-update-with-start

Send a message to a Workflow Execution to invoke an Update handler, and wait for
the update to complete. If the Workflow Execution is not running, then a new workflow
execution is started and the update is sent.

Experimental.

```
temporal workflow execute-update-with-start \
  --update-name YourUpdate \
  --update-input '{"update-key": "update-value"}' \
  --workflow-id YourWorkflowId \
  --type YourWorkflowType \
  --task-queue YourTaskQueue \
  --id-conflict-policy Fail \
  --input '{"wf-key": "wf-value"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--cron` | No | **string** Cron schedule for the Workflow. |
| `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. |
| `--fail-existing` | No | **bool** Fail if the Workflow already exists. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. |
| `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. |
| `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. |
| `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. |
| `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. |
| `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ |
| `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Workflow Task queue. |
| `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. |
| `--type` | Yes | **string** Workflow Type name. |
| `--update-first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. |
| `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. |
| `--update-input` | No | **string[]** Update input value. Use JSON content or set --update-input-meta to override. Can't be combined with --update-input-file. Can be passed multiple times to pass multiple arguments. |
| `--update-input-base64` | No | **bool** Assume update inputs are base64-encoded and attempt to decode them. |
| `--update-input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --update-input-meta to override. Can't be combined with --update-input. Can be passed multiple times to pass multiple arguments. |
| `--update-input-meta` | No | **string[]** Input update payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. |
| `--update-name` | Yes | **string** Update name. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. |

## fix-history-json

Reserialize an Event History JSON file:

```
temporal workflow fix-history-json \
    --source /path/to/original.json \
    --target /path/to/reserialized.json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--source`, `-s` | Yes | **string** Path to the original file. |
| `--target`, `-t` | No | **string** Path to the results file. When omitted, output is sent to stdout. |

## list

List Workflow Executions. The optional `--query` limits the output to
Workflows matching a Query:

```
temporal workflow list \
    --query YourQuery
```

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

View a list of archived Workflow Executions:

```
temporal workflow list \
    --archived
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--archived` | No | **bool** Limit output to archived Workflow Executions. _(Experimental)_ |
| `--limit` | No | **int** Maximum number of Workflow Executions to display. |
| `--page-size` | No | **int** Maximum number of Workflow Executions to fetch at a time from the server. |
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. |

## metadata

Issue a Query for and display user-set metadata like summary and
details for a specific Workflow Execution:

```
temporal workflow metadata \
    --workflow-id YourWorkflowId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--reject-condition` | No | **string-enum** Optional flag for rejecting Queries based on Workflow state. Accepted values: not_open, not_completed_cleanly. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## pause

Pause a Workflow Execution.
Note: This is an experimental feature and may change in the future.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--reason` | No | **string** Reason for pausing the Workflow Execution. Defaults to message with the current user's name. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## query

Send a Query to a Workflow Execution by Workflow ID to retrieve its state.
This synchronous operation exposes the internal state of a running Workflow
Execution, which constantly changes. You can query both running and completed
Workflow Executions:

```
temporal workflow query \
    --workflow-id YourWorkflowId
    --type YourQueryType
    --input '{"YourInputKey": "YourInputValue"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--name` | Yes | **string** Query Type/Name. |
| `--reject-condition` | No | **string-enum** Optional flag for rejecting Queries based on Workflow state. Accepted values: not_open, not_completed_cleanly. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## reset

Reset a Workflow Execution so it can resume from a point in its Event History
without losing its progress up to that point:

```
temporal workflow reset \
    --workflow-id YourWorkflowId \
    --event-id YourLastEvent
```

Start from where the Workflow Execution last continued as new:

```
temporal workflow reset \
    --workflow-id YourWorkflowId \
    --type LastContinuedAsNew
```

For batch resets, limit your resets to FirstWorkflowTask, LastWorkflowTask, or
BuildId. Do not use Workflow IDs, run IDs, or event IDs with this command.

Visit https://docs.temporal.io/visibility to read more about Search
Attributes and Query creation.

### with-workflow-update-options

Run Workflow Update Options atomically after the Workflow is reset.
Workflows selected by the reset command are forwarded onto the subcommand.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--versioning-override-behavior` | Yes | **string-enum** Override the versioning behavior of a Workflow. Accepted values: pinned, one_time, auto_upgrade. |
| `--versioning-override-build-id` | No | **string** When overriding to a `pinned` or `one_time` behavior, specifies the Build ID of the version to target. |
| `--versioning-override-deployment-name` | No | **string** When overriding to a `pinned` or `one_time` behavior, specifies the Deployment Name of the version to target. |

## result

Wait for and print the result of a Workflow Execution:

```
temporal workflow result \
    --workflow-id YourWorkflowId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## show

Show a Workflow Execution's Event History.
When using JSON output (`--output json`), you may pass the results to an SDK
to perform a replay:

```
temporal workflow show \
    --workflow-id YourWorkflowId
    --output json
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--detailed` | No | **bool** Display events as detailed sections instead of table. Does not apply to JSON output. |
| `--follow`, `-f` | No | **bool** Follow the Workflow Execution progress in real time. Does not apply to JSON output. |
| `--reverse` | No | **bool** Fetch Event History newest-event-first. Cannot be combined with --follow. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## signal

Send an asynchronous notification (Signal) to a running Workflow Execution by
its Workflow ID. The Signal is written to the History. When you include
`--input`, that data is available for the Workflow Execution to consume:

```
temporal workflow signal \
    --workflow-id YourWorkflowId \
    --name YourSignal \
    --input '{"YourInputKey": "YourInputValue"}'
```

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--name` | Yes | **string** Signal name. |
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. |

## signal-with-start

Send an asynchronous notification (Signal) to a Workflow Execution.
If the Workflow Execution is not running or is not found, it starts the
workflow then sends the signal.

```
temporal workflow signal-with-start \
  --signal-name YourSignal \
  --signal-input '{"some-key": "some-value"}' \
  --workflow-id YourWorkflowId \
  --type YourWorkflowType \
  --task-queue YourTaskQueue \
  --input '{"some-key": "some-value"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--cron` | No | **string** Cron schedule for the Workflow. |
| `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. |
| `--fail-existing` | No | **bool** Fail if the Workflow already exists. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. |
| `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. |
| `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. |
| `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--signal-input` | No | **string[]** Signal input value. Use JSON content or set --signal-input-meta to override. Can't be combined with --signal-input-file. Can be passed multiple times to pass multiple arguments. |
| `--signal-input-base64` | No | **bool** Assume signal inputs are base64-encoded and attempt to decode them. |
| `--signal-input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --signal-input-meta to override. Can't be combined with --signal-input. Can be passed multiple times to pass multiple arguments. |
| `--signal-input-meta` | No | **string[]** Input signal payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. |
| `--signal-name` | Yes | **string** Signal name. |
| `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. |
| `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ |
| `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Workflow Task queue. |
| `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. |
| `--type` | Yes | **string** Workflow Type name. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. |

## stack

Perform a Query on a Workflow Execution using a `__stack_trace`-type Query.
Display a stack trace of the threads and routines currently in use by the
Workflow for troubleshooting:

```
temporal workflow stack \
    --workflow-id YourWorkflowId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--reject-condition` | No | **string-enum** Optional flag to reject Queries based on Workflow state. Accepted values: not_open, not_completed_cleanly. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## start

Start a new Workflow Execution. Returns the Workflow- and Run-IDs:

```
temporal workflow start \
    --workflow-id YourWorkflowId \
    --type YourWorkflow \
    --task-queue YourTaskQueue \
    --input '{"some-key": "some-value"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--cron` | No | **string** Cron schedule for the Workflow. |
| `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. |
| `--fail-existing` | No | **bool** Fail if the Workflow already exists. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. |
| `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. |
| `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. |
| `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. |
| `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ |
| `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Workflow Task queue. |
| `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. |
| `--type` | Yes | **string** Workflow Type name. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. |

## start-update-with-start

Send a message to a Workflow Execution to invoke an Update handler, and wait for
the update to be accepted or rejected. If the Workflow Execution is not running,
then a new workflow execution is started and the update is sent.

Experimental.

```
temporal workflow start-update-with-start \
  --update-name YourUpdate \
  --update-input '{"update-key": "update-value"}' \
  --update-wait-for-stage accepted \
  --workflow-id YourWorkflowId \
  --type YourWorkflowType \
  --task-queue YourTaskQueue \
  --id-conflict-policy Fail \
  --input '{"wf-key": "wf-value"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--cron` | No | **string** Cron schedule for the Workflow. |
| `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. |
| `--fail-existing` | No | **bool** Fail if the Workflow already exists. |
| `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. |
| `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. |
| `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. |
| `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. |
| `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. |
| `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. |
| `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. |
| `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. |
| `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ |
| `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ |
| `--task-queue`, `-t` | Yes | **string** Workflow Task queue. |
| `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. |
| `--type` | Yes | **string** Workflow Type name. |
| `--update-first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. |
| `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. |
| `--update-input` | No | **string[]** Update input value. Use JSON content or set --update-input-meta to override. Can't be combined with --update-input-file. Can be passed multiple times to pass multiple arguments. |
| `--update-input-base64` | No | **bool** Assume update inputs are base64-encoded and attempt to decode them. |
| `--update-input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --update-input-meta to override. Can't be combined with --update-input. Can be passed multiple times to pass multiple arguments. |
| `--update-input-meta` | No | **string[]** Input update payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. |
| `--update-name` | Yes | **string** Update name. |
| `--update-wait-for-stage` | Yes | **string-enum** Update stage to wait for. The only option is `accepted`, but this option is required. This is to allow a future version of the CLI to choose a default value. Accepted values: accepted. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. |

## terminate

Terminate a Workflow Execution:

```
temporal workflow terminate \
    --reason YourReasonForTermination \
    --workflow-id YourWorkflowId
```

The reason is optional and defaults to the current user's name. The reason
is stored in the Event History as part of the `WorkflowExecutionTerminated`
event. This becomes the closing Event in the Workflow Execution's history.

Executions may be terminated in bulk via a visibility Query list filter:

```
temporal workflow terminate \
    --query YourQuery \
    --reason YourReasonForTermination
```

Workflow code cannot see or respond to terminations. To perform clean-up work
in your Workflow code, use `temporal workflow cancel` instead.

Visit https://docs.temporal.io/visibility to read more about Search Attributes
and Query creation. See `temporal batch --help` for a quick reference.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. |
| `--reason` | No | **string** Reason for termination. Defaults to message with the current user's name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Can only be set with --workflow-id. Do not use with --query. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm termination. Can only be used with --query. |

## trace

Display the progress of a Workflow Execution and its child workflows with a
real-time trace. This view helps you understand how Workflows are proceeding:

```
temporal workflow trace \
    --workflow-id YourWorkflowId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--concurrency` | No | **int** Number of Workflow Histories to fetch at a time. |
| `--depth` | No | **int** Set depth for your Child Workflow fetches. Pass -1 to fetch child workflows at any depth. |
| `--fold` | No | **string[]** Fold away Child Workflows with the specified statuses. Case-insensitive. Ignored if --no-fold supplied. Available values: running, completed, failed, canceled, terminated, timedout, continueasnew. Can be passed multiple times. |
| `--no-fold` | No | **bool** Disable folding. Fetch and display Child Workflows within the set depth. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## unpause

Unpause a previously paused Workflow Execution.
Note: This is an experimental feature and may change in the future.

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--reason` | No | **string** Reason for unpausing the Workflow Execution. Defaults to message with the current user's name. |
| `--run-id`, `-r` | No | **string** Run ID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## update

An Update is a synchronous call to a Workflow Execution that can change its
state, control its flow, and return a result.

### describe

Given a Workflow Execution and an Update ID, return information about its current status, including
a result if it has finished.

```
temporal workflow update describe \
    --workflow-id YourWorkflowId \
    --update-id YourUpdateId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--run-id`, `-r` | No | **string** Run ID. If unset, updates the currently-running Workflow Execution. |
| `--update-id` | Yes | **string** Update ID. Must be unique per Workflow Execution. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

### execute

Send a message to a Workflow Execution to invoke an Update handler, and wait for
the update to complete or fail. You can also use this to wait for an existing
update to complete, by submitting an existing update ID.

```
temporal workflow update execute \
    --workflow-id YourWorkflowId \
    --name YourUpdate \
    --input '{"some-key": "some-value"}'
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--name` | Yes | **string** Handler method name. |
| `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. |
| `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

### result

Given a Workflow Execution and an Update ID, wait for the Update to complete or fail and
print the result.

```
temporal workflow update result \
    --workflow-id YourWorkflowId \
    --update-id YourUpdateId
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--run-id`, `-r` | No | **string** Run ID. If unset, updates the currently-running Workflow Execution. |
| `--update-id` | Yes | **string** Update ID. Must be unique per Workflow Execution. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

### start

Send a message to a Workflow Execution to invoke an Update handler, and wait for
the update to be accepted or rejected. You can subsequently wait for the update
to complete by using `temporal workflow update execute`.

```
temporal workflow update start \
    --workflow-id YourWorkflowId \
    --name YourUpdate \
    --input '{"some-key": "some-value"}'
    --wait-for-stage accepted
```

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. |
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. |
| `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. |
| `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. |
| `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. |
| `--name` | Yes | **string** Handler method name. |
| `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. |
| `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. |
| `--wait-for-stage` | Yes | **string-enum** Update stage to wait for. The only option is `accepted`, but this option is  required. This is to allow a future version of the CLI to choose a default value. Accepted values: accepted. |
| `--workflow-id`, `-w` | Yes | **string** Workflow ID. |

## update-options

Modify properties of Workflow Executions:

```
temporal workflow update-options [options]
```

It can override the Worker Deployment configuration of a
Workflow Execution, which controls Worker Versioning.

For example, to force Workers in the current Deployment execute the
next Workflow Task change behavior to `auto_upgrade`:

```
temporal workflow update-options \
    --workflow-id YourWorkflowId \
    --versioning-override-behavior auto_upgrade
```

or to pin the workflow execution to a Worker Deployment, set behavior
to `pinned`:

```
temporal workflow update-options \
    --workflow-id YourWorkflowId \
    --versioning-override-behavior pinned \
    --versioning-override-deployment-name YourDeploymentName \
    --versioning-override-build-id YourDeploymentBuildId
```

or to move the workflow execution to a Worker Deployment Version until
the next Workflow Task completes there, set behavior to `one_time`:

```
temporal workflow update-options \
    --workflow-id YourWorkflowId \
    --versioning-override-behavior one_time \
    --versioning-override-deployment-name YourDeploymentName \
    --versioning-override-build-id YourDeploymentBuildId
```

To remove any previous overrides, set the behavior to
`unspecified`:

```
temporal workflow update-options \
    --workflow-id YourWorkflowId \
    --versioning-override-behavior unspecified
```

To see the current override use `temporal workflow describe`

Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands.

| Flag | Required | Description |
|------|----------|-------------|
| `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. |
| `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. |
| `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. |
| `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. |
| `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. |
| `--versioning-override-behavior` | Yes | **string-enum** Override the versioning behavior of a Workflow. Accepted values: unspecified, pinned, one_time, auto_upgrade. |
| `--versioning-override-build-id` | No | **string** When overriding to a `pinned` or `one_time` behavior, specifies the Build ID of the version to target. |
| `--versioning-override-deployment-name` | No | **string** When overriding to a `pinned` or `one_time` behavior, specifies the Deployment Name of the version to target. |
| `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. |
| `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. |

## Global Flags

The following options can be used with any command.

| Flag | Required | Description | Default |
|------|----------|-------------|--------|
| `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` |
| `--api-key` | No | **string** API key for request. |  |
| `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. |  |
| `--client-connect-timeout` | No | **duration** Client connection timeout. |  |
| `--codec-auth` | No | **string** Authorization header for Codec Server requests. |  |
| `--codec-endpoint` | No | **string** Remote Codec Server endpoint. |  |
| `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). |  |
| `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` |
| `--command-timeout` | No | **duration** Command execution timeout. |  |
| `--config-file` | No | **string** TOML config file path. |  |
| `--disable-config-env` | No | **bool** Disable loading config from environment variables. |  |
| `--disable-config-file` | No | **bool** Disable loading config from file. |  |
| `--env` | No | **string** Active environment name (`ENV`). | `default` |
| `--env-file` | No | **string** Path to environment settings file. |  |
| `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). |  |
| `--identity` | No | **string** Identity of the client submitting requests. |  |
| `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` |
| `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` |
| `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` |
| `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. |  |
| `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` |
| `--profile` | No | **string** Configuration profile to use. Overrides the TEMPORAL_PROFILE environment variable and defaults to "default". |  |
| `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` |
| `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. |  |
| `--tls-ca-data` | No | **string** Inline server CA certificate data. |  |
| `--tls-ca-path` | No | **string** Path to server CA certificate. |  |
| `--tls-cert-data` | No | **string** Inline x509 certificate data. |  |
| `--tls-cert-path` | No | **string** Path to x509 certificate. |  |
| `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. |  |
| `--tls-key-data` | No | **string** Inline x509 private key data. |  |
| `--tls-key-path` | No | **string** Path to x509 private key. |  |
| `--tls-server-name` | No | **string** Override target TLS server name. |  |

---

# CLI basics

Source: https://docs.temporal.io/cli/common-operations

This page walks through the operations you're most likely to reach for when using the Temporal CLI. For the full set of
commands, see the [command reference](/cli#command-reference).

## Run a development server

The Temporal CLI ships with a complete Temporal binary. Start a local Temporal Service for development with a single
command.

The Temporal Service is available on `localhost:7233` and the Web UI at `http://localhost:8233`.

To keep Workflow data between restarts, specify a database file.

See [Install and configure](/cli/setup-cli#run-a-local-development-server) for more development server options.

```bash
temporal server start-dev
```

```bash
temporal server start-dev --db-filename temporal.db
```

## Start a Workflow

Start a Workflow Execution with `temporal workflow start`.

To start a Workflow and wait for the result, use `temporal workflow execute` instead.

```bash
temporal workflow start \\
  --task-queue my-task-queue \\
  --type MyWorkflow \\
  --workflow-id my-workflow-id \\
  --input '"my-input-value"'
```

```bash
temporal workflow execute \\
  --task-queue my-task-queue \\
  --type MyWorkflow \\
  --workflow-id my-workflow-id \\
  --input '"my-input-value"'
```

## Check Workflow status

List running Workflows.

Get details about a specific Workflow Execution.

View the Event History for a Workflow Execution.

```bash
temporal workflow list
```

```bash
temporal workflow describe --workflow-id my-workflow-id
```

```bash
temporal workflow show --workflow-id my-workflow-id
```

## Send Signals and Queries

Send a [Signal](/sending-messages#sending-signals) to a running Workflow.

[Query](/sending-messages#sending-queries) a running Workflow for its current state.

```bash
temporal workflow signal \\
  --workflow-id my-workflow-id \\
  --name my-signal \\
  --input '"signal-value"'
```

```bash
temporal workflow query \\
  --workflow-id my-workflow-id \\
  --name my-query
```

## Cancel or Terminate a Workflow

Cancel a Workflow Execution. Cancellation allows cleanup logic to run before the Workflow completes.

Terminate a Workflow Execution. Termination stops the Workflow immediately with no cleanup.

```bash
temporal workflow cancel --workflow-id my-workflow-id
```

```bash
temporal workflow terminate --workflow-id my-workflow-id
```

## Log in to Temporal Cloud

Log in to Temporal Cloud with `temporal cloud login`. After login, you can run commands against Temporal Cloud by
providing the address and Namespace.

See [Use with Temporal Cloud](/cli/cloud) for all authentication options.

```bash
temporal cloud login
```

```bash
temporal workflow list \\
  --address <namespace>.<account>.tmprl.cloud:7233 \\
  --namespace <namespace>.<account>
```

## Work with Schedules

Create a [Schedule](/schedule) that starts a Workflow on an interval.

List all Schedules.

Pause and unpause a Schedule.

```bash
temporal schedule create \\
  --schedule-id my-schedule \\
  --interval 1h \\
  --task-queue my-task-queue \\
  --type MyScheduledWorkflow
```

```bash
temporal schedule list
```

```bash
temporal schedule toggle \\
  --schedule-id my-schedule \\
  --pause --reason "maintenance"
```

```bash
temporal schedule toggle \\
  --schedule-id my-schedule \\
  --unpause --reason "maintenance complete"
```

## Manage Namespaces

List available [Namespaces](/namespaces).

Create a new Namespace.

To manage Namespaces on Temporal Cloud, use the Temporal Cloud extension and
[`temporal cloud namespace`](/cli/command-reference/cloud/namespace) commands.

```bash
temporal operator namespace list
```

```bash
temporal operator namespace create --namespace my-namespace
```

## Format and filter output

Use `--output json` to get machine-readable output from any command.

Combine with [jq](https://jqlang.github.io/jq/) for filtering.

```bash
temporal workflow list --output json
```

```bash
temporal workflow list --output json | jq '.[].type.name'
```

## Next steps

- [Use with Temporal Cloud](/cli/cloud) to connect the CLI to Temporal Cloud.
- [Command reference](/cli#command-reference) for the full set of commands and options.

---

# Install and configure the CLI

Source: https://docs.temporal.io/cli/setup-cli

> Install the Temporal CLI, run a local development server, and configure your environment.

The Temporal CLI is a command-line tool for interacting with the Temporal Service. It helps you manage, monitor, and
debug Temporal applications.

## Install the CLI

The CLI is available for macOS, Linux, and Windows, or as a Docker image.

**macOS**

Install with Homebrew:

```bash
brew install temporal
```

Or download from the CDN:

- [Darwin amd64](https://temporal.download/cli/archive/latest?platform=darwin&arch=amd64)
- [Darwin arm64](https://temporal.download/cli/archive/latest?platform=darwin&arch=arm64)

extract the archive and add the `temporal` binary to your `PATH`.

**Linux**

Install with Homebrew (if available):

```bash
brew install temporal
```

Or download from the CDN:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

extract the archive and add the `temporal` binary to your `PATH`.

**Windows**

Download from the CDN:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

extract the archive and add the `temporal.exe` binary to your `PATH`.

**Docker**

Temporal CLI container image is available on [DockerHub](https://hub.docker.com/r/temporalio/temporal) and can be run
directly:

```shell
docker run --rm temporalio/temporal --help
```

> **📝 Note:**
>
> When running the Temporal CLI inside Docker, for the development server to be accessible from the host system, the
> server needs to be configured to listen on external IP and the ports need to be forwarded:
>
> ```shell
> docker run --rm -p 7233:7233 -p 8233:8233 temporalio/temporal server start-dev --ip 0.0.0.0
> # UI is now accessible from host at http://localhost:8233/
> ```
>

## Install the Temporal Cloud extension

If you are using Temporal Cloud, install the Temporal Cloud extension for the Temporal CLI. You can install the
extension using the following command:

> **💡 Tip:**
> Support, stability, and dependency info
>
> The Temporal Cloud extension is in [Public Preview](/evaluate/product-release-stages#public-preview).
> APIs and configuration may change before the stable release.
>

```bash
brew install temporalio/brew/temporal-cloud
```

## Run a local development server

The CLI includes a local Temporal development service for fast feedback while building your application.

Start the server:

```bash
temporal server start-dev
```

This command automatically starts the Web UI, creates the `default` [Namespace](/namespaces), and uses an in-memory
SQLite database.

The Temporal Service will be available on `localhost:7233` and the Temporal Web UI will be available at
[`http://localhost:8233`](http://localhost:8233/).

Persist state locally by specifying a database file:

```shell
temporal server start-dev --db-filename temporal.db
```

### Development server configuration

#### Namespace registration

Namespaces are pre-registered at startup for immediate use. Customize pre-registered Namespaces with the following
command:

```shell
temporal server start-dev --namespace foo --namespace bar
```

Register Namespaces with `namespace create`:

```shell
temporal operator namespace create --namespace foo
```

#### Enable or turn off the Temporal Web UI

By default, the Temporal Web UI is enabled when running the development server using the Temporal CLI. To turn off the
UI, use the `--headless` modifier:

```shell
temporal server start-dev --headless
```

#### Dynamic configuration

Advanced Temporal CLI configuration requires a dynamic configuration file.

To set values on the command line, use `--dynamic-config-value KEY=JSON_VALUE`. For example, enable the Search Attribute
cache:

```bash
temporal server start-dev --dynamic-config-value system.forceSearchAttributesCacheRefreshOnRead=false
```

This setting makes created Search Attributes immediately available.

## Configure the CLI

### Environment variables

You can configure the CLI with environment variables instead of passing flags on every command. Setting an environment
variable is not the same as storing a preset with [`temporal env`](/cli/command-reference/env): an environment variable
configures the CLI process, while `temporal env` writes named key-value presets to a file.

For every variable the CLI reads, its equivalent flag, and its TOML configuration key, refer to
[Environment configuration](/references/client-environment-configuration).

### Create and modify configuration files

<a id="configuration-files"></a>

The Temporal CLI lets you create and modify TOML configuration files to store your environment variables and other
settings. Refer to [Environment Configuration](../develop/environment-configuration#cli-integration) for more
information.

### Configure proxy support

The Temporal CLI supports running behind a proxy, so it can reach the Temporal Service in network-restricted environments.

#### Setting up proxy support

If you are behind a proxy, you'll need to instruct the Temporal CLI to route its requests via that proxy. You can
achieve this by setting the `HTTPS_PROXY` environment variable.

```command
export HTTPS_PROXY=<host>:<port>
```

Replace `<host>` with the proxy's hostname or IP address, and `<port>` with the proxy's port number.

Once set, you can run the Temporal CLI commands as you normally would.

> **📝 Note:**
>
> Temporal CLI uses the gRPC library which natively supports HTTP CONNECT proxies. The gRPC library checks for the
> `HTTPS_PROXY` (and its case-insensitive variants) environment variable to determine if it should route requests through
> a proxy.
>

In addition to `HTTPS_PROXY`, gRPC also respects the `NO_PROXY` environment variable. This can be useful if there are
specific addresses or domains you wish to exclude from proxying.

For more information, see [Proxy](https://github.com/grpc/grpc-go/blob/master/Documentation/proxy.md) in the gRPC
documentation.

## Enable auto-completion

Enable auto-completion using the following commands.

### Zsh auto-completion

1. Add the following line to your `~/.zshrc` startup script:

   ```sh
   eval "$(temporal completion zsh)"
   ```

2. Re-launch your shell or run:

   ```sh
   source ~/.zshrc
   ```

### Bash auto-completion

1. Install [bash-completion](https://github.com/scop/bash-completion#installation) and add the software to your
   `~/.bashrc`.

2. Add the following line to your `~/.bashrc` startup script:

   ```sh
   eval "$(temporal completion bash)"
   ```

3. Re-launch your shell or run:

   ```sh
   source ~/.bashrc
   ```

> **📝 Note:**
>
> If auto-completion fails with the error: `bash: _get_comp_words_by_ref: command not found`, you did not successfully
> install [bash-completion](https://github.com/scop/bash-completion#installation). This package must be loaded into your
> shell for `temporal` auto-completion to work.
>

### Fish auto-completion

1. Create the Fish custom completions directory if it does not already exist:

   ```fish
   mkdir -p ~/.config/fish/completions
   ```

2. Configure the completions to load when needed. Note: the filename must be `temporal.fish` or the completions will not
   be found:

   ```fish
   echo 'eval "$(temporal completion fish)"' >~/.config/fish/completions/temporal.fish
   ```

3. Re-launch your shell or run:

   ```fish
   source ~/.config/fish/completions/temporal.fish
   ```

## Get CLI help 

From the command line:

```
temporal <command> <subcommand> --help
```

For example:

- `temporal --help`
- `temporal workflow --help`
- `temporal workflow delete --help`

For a full list of commands, see the [Temporal CLI command reference](/cli#command-reference).

---

# Temporal Cloud guide

Source: https://docs.temporal.io/cloud

> Find guides for onboarding, security, pricing, monitoring, access management, Nexus, and the Cloud Ops API in one place.

Welcome to the Temporal Cloud guide.

In this guide you will find information about Temporal Cloud, onboarding, features, and how to use them.

To create a Temporal Cloud account, sign up [here](https://temporal.io/get-cloud).

**[Get started with Temporal Cloud.](/cloud/get-started)**

## Become familiar with Temporal Cloud

- [Overview of Temporal Cloud](/evaluate/cloud)
  - [Security model](/evaluate/cloud/security)
  - [Uptime, latency, and SLOs](/cloud/slo) (uptime and latency objectives, measured latency, and throughput)
  - [Service Level Agreement (SLA)](https://temporal.io/sla)
  - [Account, Namespace, and application level configurations](/evaluate/cloud/limits)
  - [Pricing](/cloud/pricing)
  - [Support](/evaluate/cloud/support)

## Feature guides

- [Get started with Temporal Cloud](/cloud/get-started)
  - [Manage certificates](/cloud/certificates)
  - [Manage API keys](/cloud/api-keys)
  - [Manage Namespaces](/cloud/namespaces)
  - [Manage users](/cloud/manage-access/users)
  - [Manage user groups](/cloud/manage-access/user-groups)
  - [Manage billing](/cloud/billing)
  - [Manage Service Accounts](/cloud/manage-access/service-accounts)
- [API key feature guide](/cloud/api-keys)
- [Monitor Temporal Cloud](/cloud/monitor)
  - [Set up Cloud metrics](/cloud/metrics/openmetrics)
  - [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup)
  - [Monitor Worker health](/cloud/worker-health)
  - [Monitor service health](/cloud/service-health)
  - [Receive notifications](/cloud/notifications)
- [Temporal Nexus](/cloud/nexus)
- [SAML authentication feature guide](/cloud/manage-access/saml)
- [Cloud Ops API](/ops)
- [Audit logging feature guide](/cloud/audit-logs)
- [Temporal CLI Cloud extension](/cli/cloud)
- [`tcld` (Temporal Cloud command-line interface) reference](/cloud/tcld)

---

# Usage dashboards

Source: https://docs.temporal.io/cloud/actions-usage

## Usage 

Actions usage is tracked across an account in the [usage dashboard](https://cloud.temporal.io/usage) and is visible to
Account Owners, Finance Admin and Global Admin. For individual Namespaces, usage can be seen in the
[Namespace summary](https://cloud.temporal.io/namespaces/) for a specific Namespace.

![Temporal Cloud Usage dashboard](https://docs.temporal.io/img/cloud/billing/usage-dashboard.png)

## Actions in Workflows 

When viewing a Event history, events that represent a Billable Action are annotated with the number consumed by the
event in the **Billable Actions** Column. These Actions are summarized at the top of the workflow.

![Temporal Cloud Usage dashboard showing aggregated Billable Actions](https://docs.temporal.io/img/cloud/billing/aggregate-billable-actions.png)

![Temporal Cloud Usage dashboard showing individual Billable Actions associated with events](https://docs.temporal.io/img/cloud/billing/individual-billable-actions.png)

This Billable Action estimate is useful for projecting the cost of Workflows. For example, if you ran a test Workflow
that generated 20 Billable Actions and projected that it would be run 100 times a day for a month, you could anticipate
that Workflow to generate 20 Actions x 100 runs/day x 30 days = 60,000 Billable Actions per month. You can also use the
Billable Action estimate to help optimize Workflows by better understanding your cost drivers.

> **💡 Tip:**
> Excluded Billable Actions
>
> The Billable Action estimate is an experimental feature and only measures Billable Actions that exist within Workflow
> event histories. Some billable concepts are not included in these calculations such as:
>
> - Query
> - Activity Heartbeats
> - Rejected Update Workflow Executions
> - Export
> - Schedule
> - Replicated Actions that occur in a
>   [Namespace replication](../../cloud/high-availability/index.mdx#high-availability-features)
>
> Additionally, Workflows with the `TemporalNamespaceDivision` Search Attribute set may not have accurate Billable Action
> Estimates. The estimated Billable Actions should only be treated as an estimate. If billable events exist outside of
> event history, the Actions count could be higher.
>

[Reach out](https://pages.temporal.io/contact-us) to our team for more information or to help size your number of
Actions.

---

# Manage API keys

Source: https://docs.temporal.io/cloud/api-keys

Temporal Cloud API keys offer industry-standard identity-based authentication for Temporal users and
[Service Accounts](/cloud/manage-access/service-accounts). This document introduces Temporal Cloud's API key features:

- [API key overview](#overview)
- [API key best practices](#best-practices)
- [Global Administrator and Account Owner API key management](#manage-api-keys)
- [User API key management](#user-api-keys)
- [Manage API keys for Service Accounts](#serviceaccount-api-keys)
- [API keys for Namespace authentication](#namespace-authentication)
- [Use API keys to authenticate](#using-apikeys)
- [Troubleshoot your API key use](#troubleshooting)
- [API keys: Frequently asked questions](#faqs)

## API key overview 

Each Temporal Cloud API key is a unique identity linked to role-based access control (RBAC) settings to ensure secure
and appropriate access.

The authentication process follows this pathway:

![API key (authentication) → Identity (user or Service Account) → RBAC (authorization)](https://docs.temporal.io/img/cloud/apikeys/apikeyrbac.png)

## API key best practices 

- **Keep it secret; keep it safe**: Treat your API key like a password. Do not expose it in client-side code, public
  repositories, or other easily accessible locations.
- **Rotate keys regularly**: Change your API keys periodically to reduce risks from potential leaks.[Email notifications](/cloud/notifications#admin-notifications) are sent at 30, 20, and 10 days before an API key expires.
- **Design your code for key updates**: Use key management practices that retrieve your API keys without hard-coding
  them into your apps. This lets you restart your Workers to refresh your rotated keys without recompiling your code.
- **Monitor API key usage**: Check usage metrics and logs regularly. Revoke the key immediately if you detect any
  unexpected or unauthorized activity.
- **Use a Key Management System (KMS)**: Use a Key Management System to minimize the risk of key leaks.

For guidance on which identities should own API keys, when to use Namespace-scoped Service Accounts, and how to align
API keys with your Namespace topology, see [Managing Temporal Cloud access control](/best-practices/cloud-access-control).

### API key use cases

API keys are used for the following scenarios:

- _**Cloud operations automation**_: API keys work with Temporal Cloud operational tools, including
  the [Temporal Cloud CLI extension](/cli/cloud), [`tcld`](/cloud/tcld), [Cloud Ops APIs](/ops), and
  [the Terraform provider](/cloud/terraform-provider). Use them to manage your Temporal Cloud
  account, Namespaces, certificates, and user identities.
- _**Namespace authentication**_: API keys serve as an authentication mechanism for executing and managing Workflows via
  the SDK and Temporal CLI, offering an alternative to mTLS-based authentication.

### API key supported tooling

Use API keys to authenticate with:

- [The Temporal CLI](/cli), including the [Temporal Cloud extension](/cli/cloud)
- [Temporal SDKs](/develop)
- [`tcld`](/cloud/tcld/index.mdx)
- [The Cloud Operations API](/cloud/operation-api.mdx)
- [Temporalʼs Terraform provider](/cloud/terraform-provider)

### API key permissions

API keys support both users and Service Accounts. Here are the differences in their permissions:

- Any user can create, delete, and update their _own_ API key using the Cloud UI or the CLI.
- Only Global Administrators and Account Owners can create, delete, and update access to API keys for all types of
  Service Accounts.
- Namespace Admins can create, delete, and update access to API keys for the Namespace-scoped Service Accounts they
  administer.

### API key prerequisites

Check these setup details before using API keys:

- The Global Administrator or Account Owner may need to [enable API keys access](#manage-api-keys) for your Temporal
  Account.
- Have access to the [Temporal Cloud UI](https://cloud.temporal.io/), the
  [Temporal Cloud CLI extension](/cli/cloud), or [`tcld`](/cloud/tcld/) to create an API key.

## Global administrator and account owner API key management 

Global Administrators and Account Owners can monitor, manage, disable, and delete API keys for any user or Service
Account within their account. To manage your account’s API keys:

1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI.
1. Go to [API Keys](https://cloud.temporal.io/keys)

Administrators can disable the creation of new API keys using the **Disable Create API Keys** button on the **API Keys** page. Existing API keys can still be used to authenticate into Temporal Cloud normally until they are either
disabled, deleted, or expired.

Temporal Cloud automatically sends email notifications to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. For the full recipient list, see [Notifications](/cloud/notifications#admin-notifications).

To disable or delete an individual API key use the vertical ellipsis menu in the API key table row.

To find an API key, you can filter by API key state and identity type (Global Administrators and Account Owners only).

> **⚠️ Caution:**
> DISABLED API KEYS
>
> Deleting or disabling a key removes its ability to authenticate into Temporal Cloud. If you delete or disable an API key
> being used by Workers to run a Workflow, those Workers will be unable to connect to Temporal until a new API key secret
> is created and configured.
>

## User API key management 

Manage your personal API keys with the Temporal Cloud UI or the CLI. These sections show you how to generate, manage, and
remove API keys for a user.

### Generate an API key

Create API keys using one of the following methods:

> **⚠️ Caution:**
>
> - Once generated, copy and securely save the API key. It will be displayed only once for security purposes.
>

#### Generate API keys with the Temporal Cloud UI

[Log in](https://cloud.temporal.io/) to the Temporal Cloud UI and navigate to your
[Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys). Then select **Create API key** and provide the
following information:

- **API key name**: A short, identifiable name for the key
- **API key description**: A longer description of the key's use
- **Expiration date**: The end date for the API key

Finish by selecting **Generate API Key**.

#### Generate API keys with the CLI

To generate an API key, log into your account and issue the following command:

**Temporal CLI**

```command
temporal cloud login
temporal cloud apikey create-for-me \
    --display-name <api-key-name> \
    --description "<api-key-description>" \
    --expiry-duration <api-key-duration>
```

**tcld**

```command
tcld login
tcld apikey create \
    --name <api-key-name> \
    --description "<api-key-description>" \
    --duration <api-key-duration>
```

Duration specifies the time until the API key expires, for example, "30d" or "4d12h".

### Enable or disable an API key 

You can enable or disable API keys. When disabled, an API key cannot authenticate with Temporal Cloud.

#### Manage API key state with the Temporal Cloud UI

Follow these steps:

1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI.
1. Go to your [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys).
1. Select the vertical ellipsis menu in the API key table row.
1. Choose **Enable** or **Disable**.

#### Manage API key state with the CLI

To manage an API key, log into your account and use one of the following commands to enable or disable it:

**Temporal CLI**

```command
temporal cloud login
temporal cloud apikey disable --key-id <api-key-id>
temporal cloud apikey enable --key-id <api-key-id>
```

**tcld**

```command
tcld login
tcld apikey disable --id <api-key-id>
tcld apikey enable --id <api-key-id>
```

### Delete an API key

Deleting an API key stops it from authenticating with Temporal Cloud.

> **⚠️ Caution:**
>
> Deleting an API key used by Workers to run a Workflow will cause it to fail unless you rotate the key with a new one.
> This can affect long-running Workflows that outlast the API key's lifetime.
>

#### Delete API keys with the Temporal Cloud UI

Follow these steps to remove API keys:

1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI.
1. Navigate to your [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys).
1. Select the vertical ellipsis menu in the API key table row.
1. Choose **Delete**.

#### Delete API keys with the CLI

To delete an API key, log into your account and issue the following:

**Temporal CLI**

```command
temporal cloud login
temporal cloud apikey delete --key-id <api-key-id>
```

**tcld**

```command
tcld login
tcld apikey delete --id <api-key-id>
```

### Rotate an API key

Temporal API keys automatically expire based on the specified expiration time. [Email notifications](/cloud/notifications#admin-notifications) will be sent from Temporal Cloud to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. Follow these steps to rotate API keys:

1. Create a new key. You may reuse key names if that helps.
1. Ensure that both the original key and new key function properly before moving to the next step.
1. Switch clients to load the new key and start using it.
1. Delete the old key after it is no longer in use.

For a broader machine-identity rotation strategy across API keys and Service Accounts, see
[Managing Temporal Cloud access control](/best-practices/cloud-access-control).

## Manage API keys for Service Accounts 

Global Administrators and Account Owners can manage and generate API keys for _all_ Service Accounts in their account.
Namespace Admins can manage and generate API keys for the Namespace-scoped Service Accounts they administer.

This is different for non-admin users, who manage and generate their own API keys.

### Generate an API key for a Service Account

Create API keys for Service Accounts using one of the following methods:

> **⚠️ Caution:**
>
> - Once generated, copy and securely save the API key. It will be displayed only once for security purposes.
>

#### Generate API Keys with the Temporal Cloud UI

[Log in](https://cloud.temporal.io/) to the Temporal Cloud UI. Global Administrators or Account Owners can go to
[API Keys](https://cloud.temporal.io/keys). Namespace Admins can go to
[Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys). Select **Create API Key**, then choose **Service
Account** from the "Create an API key for" dropdown. In the "Mapped to identity" input box, select a Service Account and
provide the following information:

- **API key name**: A short, identifiable name for the key
- **API key description**: A longer description of the key's use
- **Expiration date**: The end date for the API key

Finish by selecting **Generate API Key**.

#### Generate API keys with the CLI

Create an API key for a Service Account by passing the Service Account ID:

**Temporal CLI**

```command
temporal cloud apikey create-for-service-account \
    --display-name <api-key-name> \
    --description "<api-key-description>" \
    --expiry-duration <api-key-duration> \
    --service-account-id <service-account-id>
```

**tcld**

```command
tcld apikey create \
    --name <api-key-name> \
    --description "<api-key-description>" \
    --duration <api-key-duration> \
    --service-account-id <service-account-id>
```

### Enable or disable an API key 

Global Administrators and Account Owners can manage API key access for any user in their account using the Temporal
Cloud UI or the CLI.

#### Manage keys with Temporal Cloud UI

Follow these steps:

1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI.
1. Global Administrators or Account Owners can go to [API Keys](https://cloud.temporal.io/keys).
   Namespace Admins can go to [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys).
1. Find the API key. Use the vertical ellipsis menu in the table row and select the Disable/Enable option to perform the
   action. There may be a delay after changing the status. Once successful, the updated API key status will be shown in
   the row.

#### Manage keys with the CLI

Use the disable or enable command to change the state of an API key:

**Temporal CLI**

```command
temporal cloud login
temporal cloud apikey disable --key-id <api-key-id>
temporal cloud apikey enable --key-id <api-key-id>
```

**tcld**

```command
tcld login
tcld apikey disable --id <api-key-id>
tcld apikey enable --id <api-key-id>
```

This command is the same for users and Service Accounts.

### Delete an API key for a Service Account

Global Administrators and Account Owners can delete API keys for any user or Service Account in their account using the
Temporal Cloud UI or the CLI. Deleting a key removes its ability to authenticate with Temporal Cloud. If you delete an
API key used by a Worker to run a Workflow, that Worker will fail to connect to Temporal server unless you rotate the
API key with a new one.

#### Delete a Service Account API key with Temporal Cloud UI

Follow these steps:

1. Go to [API Keys](https://cloud.temporal.io/keys).
1. Locate the API key. Use the vertical ellipsis menu in the table row and select the Delete option. There may be a
   delay after deleting the API key.
1. Once successful, the updated API key status will be reflected in the row.

#### Delete a Service Account API key with the CLI

Use the delete command to remove an API key. The process is the same for a user or Service Account.

**Temporal CLI**

```command
temporal cloud login
temporal cloud apikey delete --key-id <api-key-id>
```

**tcld**

```command
tcld login
tcld apikey delete --id <api-key-id>
```

### Rotate a Service Account API key

Temporal API keys automatically expire based on the specified expiration time. [Email notifications](/cloud/notifications#admin-notifications) will be sent from Temporal Cloud to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. Follow these steps to rotate API keys:

1. Create a new key. You may reuse key names if that helps.
1. Ensure that both the original key and new key function properly before moving to the next step.
1. Switch clients to load the new key and start using it.
1. Delete the old key after it is no longer in use.

> **💡 Tip:**
>
> Service Accounts can create replacement API keys for themselves regardless of their configured permissions, except for
> Service Accounts with the [Metrics Read-Only role](/cloud/metrics/openmetrics#metrics-read-only-role). To use this
> feature, have your Service Account create a new API key using the [Cloud Ops APIs](/ops) or the CLI before the current
> one expires. Service Accounts cannot delete their own API keys without the requisite permissions, which helps keep
> Workflow access secure.
>

## API keys for Namespace authentication 

Create a Namespace with API key authentication as an alternative to mTLS-based authentication by selecting "Allow API
key authentication" during setup.

Use the gRPC Namespace endpoint: `<namespace>.<account>.tmprl.cloud:7233`. This is the recommended endpoint for all
Namespaces. For Namespaces with [High Availability](/cloud/high-availability), the Namespace endpoint automatically
directs traffic to the active region, so Workers and Clients don't need to change endpoints during a failover.

A Namespace created with API key authentication cannot later enable mTLS alongside API keys. See
[How to access a Namespace](/cloud/namespaces#access-namespaces).

See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options.

## Use API keys to authenticate 

Authenticate with Temporal Cloud using API keys with the following clients:

- [Temporal CLI](/cli), including the [Temporal Cloud extension](/cli/cloud)
- [SDKs](/develop)
- [Temporal Cloud CLI `tcld`](/cloud/tcld/index.mdx)
- [The Cloud Operations API](/cloud/operation-api.mdx)
- [Temporal’s Terraform Provider](/cloud/terraform-provider)

### Temporal CLI

To use your API key with the Temporal CLI, either pass it with the `--api-key` flag or set an environment variable in
your shell (recommended). The CLI automatically picks up the `TEMPORAL_API_KEY` environment variable from your shell.

In addition to the API key, the following client options are required:

- `--address`: Provide `<namespace>.<account_id>.tmprl.cloud:7233` using your Namespace's info.
  - This can be copied from the Namespace UI's "Connect" box. 
  - You can set the address using an environment variable.
- `--namespace`: Provide `<namespace>.<account_id>` using your Namespace's info.
  - This can be copied from the top of the Namespace UI.
  - This can be set using an environment variable.

For example, to connect to Temporal Cloud from the CLI using an environment variable for the API key:

```bash
export TEMPORAL_API_KEY=<key-secret>
temporal workflow list \
    --address <namespace>.<account_id>.tmprl.cloud:7233 \
    --namespace <namespace>.<account_id>
```

> **💡 Tip:**
> ENVIRONMENT VARIABLES
>
> Do not confuse environment variables, set with your shell, with temporal env options.
>

### SDKs

To use your API key with a Temporal SDK, see the instructions in each SDK section.

[How to connect to Temporal Cloud using an API Key with the Go SDK](/develop/go/client/temporal-client#connect-to-temporal-cloud)

[How to connect to Temporal Cloud using an API Key with the Java SDK](/develop/java/client/temporal-client#connect-to-temporal-cloud)

[How to connect to Temporal Cloud using an API Key with the Python SDK](/develop/python/client/temporal-client#connect-to-temporal-cloud)

[How to connect to Temporal Cloud using an API Key with the TypeScript SDK](/develop/typescript/client/temporal-client#connect-to-temporal-cloud)

[How to connect to Temporal Cloud using an API Key with the .NET SDK](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud)

### Temporal Cloud CLI

To use an API key with the `temporal cloud` extension or `tcld`, choose one of these methods:

- Use the `--api-key` flag.
- Set the `TEMPORAL_API_KEY` environment variable in your shell.

> **💡 Tip:**
> ENVIRONMENT VARIABLES
>
> Do not confuse environment variables, set with your shell, with temporal env options.
>

### Cloud Ops API

To use an API key with the [Cloud Ops API](/ops), securely pass the API key in your API client. For a complete example,
see
[Cloud Samples in Go](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go).

### Terraform provider

To use an API key with the [Temporal Terraform Provider](/cloud/terraform-provider), pass the API
key as a provider argument.

## Troubleshoot your API key use 

**Invalid API key errors**: Check that you copied the key correctly and that it hasn't been revoked or expired.

## API keys: Frequently asked questions 

**Q: Can I issue and use multiple API keys for the same account?**

A: Yes, you can generate multiple API keys for different services or team members.

**Q: How many API keys can be issued at once?**

A: Up to 10 non-expired keys per user and 20 non-expired keys per Service Account.

**Q: Do API keys expire?**

A: Yes, API keys expire based on the specified expiration date. Temporal recommends rotating API keys periodically. [Email notifications](/cloud/notifications#admin-notifications) will be sent from Temporal Cloud to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. 

**Q: What's the maximum allowed expiration for an API key?**

A: The maximum expiration time for an API key is 2 years.

**Q: What happens if I misplace or lose my API bearer token/secret key?**

A: The full key is displayed only once upon creation for security reasons. If you lose it, generate a new one.

**Q: What is the `Generate API Key` button on the Namespace page?**

A: The `Generate API Key` button on a Namespace page generates an API key with `Admin` permissions for the given
Namespace and the maximum expiration time, which is 2 years. For additional details, refer to
[Namespace-scoped Service Accounts](/cloud/manage-access/service-accounts#scoped).

---

# Audit Logs

Source: https://docs.temporal.io/cloud/audit-logs

> Audit Logs in Temporal Cloud provide forensic information, integrating with a data streaming service for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes.

Audit Logs is a feature of [Temporal Cloud](/evaluate/cloud) that provides forensic access information for a variety of operations in the Temporal Cloud Control Plane.

Audit Logs answers "who, when, and what" questions about Temporal Cloud resources.
These answers can help you evaluate the security of your organization, and they can provide information that you need to satisfy audit and compliance requirements.

You need the Account Owner or Global Administrator role to view Audit Logs via UI, use the API, or to configure an Audit Log Integration with [AWS Kinesis](/cloud/audit-logs-aws) or [GCP Pub/Sub](/cloud/audit-logs-gcp).

> **ℹ️ Info:**
>
> Audit Logs do not capture data plane events, such as Workflow Start, Workflow Terminate, and Schedule Create.
> Instead, explore the [Export](/cloud/export) feature, which does let you send closed Workflow Histories to external storage.
>

## Which events are supported by Audit Logs? 

The `operation` field can contain the following event values:

- Account
  - `ChangeAccountPlanType`: Change Account Plan Type
  - `CreateAccountAuditLogSink`: Create Account Audit Log Sink
  - `DeleteAccountAuditLogSink`: Delete Account Audit Log Sink
  - `OffboardAccount`: Offboard Account
  - `SelfServiceOffboardAccount`: Self-Service Offboard Account
  - `UpdateAccount`: Update Account
  - `UpdateAccountAuditLogSink`: Update Account Audit Log Sink
- API keys
  - `CreateAPIKey`: Create API Key
  - `CreateApiKey`: Create API Key
  - `CreateServiceAccountAPIKey`: Create Service Account API Key
  - `DeleteAPIKey`: Delete API Key
  - `DeleteApiKey`: Delete API Key
  - `UpdateAPIKey`: Update API Key
  - `UpdateApiKey`: Update API Key
- Billing
  - `CreateBillingReport`: Create Billing Report
- Connectivity rules
  - `CreateConnectivityRule`: Create Connectivity Rule
  - `DeleteConnectivityRule`: Delete Connectivity Rule
- Custom roles
  - `CreateCustomRole`: Create Custom Role
  - `DeleteCustomRole`: Delete Custom Role
  - `UpdateCustomRole`: Update Custom Role
- Namespace
  - `CreateNamespace`: Create Namespace
  - `DeleteNamespace`: Delete Namespace
  - `FailoverNamespaces`: Failover Namespaces
  - `RenameCustomSearchAttribute`: Rename Custom Search Attribute
  - `UpdateNamespace`: Update Namespace
  - `UpdateNamespaceTags`: Update Namespace Tags
  - `UpdateSearchAttributes`: Update Search Attributes
- Namespace export
  - `CreateNamespaceExportSink`: Create Namespace Export Sink
  - `DeleteNamespaceExportSink`: Delete Namespace Export Sink
  - `UpdateNamespaceExportSink`: Update Namespace Export Sink
  - `ValidateNamespaceExportSink`: Validate Namespace Export Sink
- Nexus endpoints
  - `CreateNexusEndpoint`: Create Nexus Endpoint
  - `DeleteNexusEndpoint`: Delete Nexus Endpoint
  - `UpdateNexusEndpoint`: Update Nexus Endpoint
- Projects
  - `CreateProject`: Create Project
  - `DeleteProject`: Delete Project
  - `SetServiceAccountProjectAccess`: Set Service Account Project Access
  - `SetUserGroupProjectAccess`: Set User Group Project Access
  - `SetUserProjectAccess`: Set User Project Access
  - `UpdateProject`: Update Project
- Security
  - `SecretFound`: Detect Exposed API Key
- Service accounts
  - `CreateServiceAccount`: Create Service Account
  - `DeleteServiceAccount`: Delete Service Account
  - `SetServiceAccountNamespaceAccess`: Set Service Account Namespace Access
  - `UpdateServiceAccount`: Update Service Account
- Users
  - `CreateJITUser`: Create JIT User
  - `CreateUser`: Create User
  - `DeleteUser`: Delete User
  - `InviteUsers`: Invite Users
  - `SetUserNamespaceAccess`: Set User Namespace Access
  - `UpdateIdentityNamespacePermissions`: Update Identity Namespace Permissions
  - `UpdateUser`: Update User
  - `UpdateUserNamespacePermissions`: Update User Namespace Permissions
  - `UserAcceptInvitation`: User Accept Invitation
  - `UserLogin`: User Login
  - `UserSignup`: User Signup
- User groups
  - `AddUserGroupMember`: Add User Group Member
  - `CreateUserGroup`: Create User Group
  - `DeleteUserGroup`: Delete User Group
  - `RemoveUserGroupMember`: Remove User Group Member
  - `SetUserGroupNamespaceAccess`: Set User Group Namespace Access
  - `UpdateUserGroup`: Update User Group

### Audit log format

> **ℹ️ Info:**
> DEPRECATION NOTICE
>
> The `request_id` field is deprecated and is planned for removal on or after November 1 2026. Use `async_operation_id` instead.
>

Audit Logs use the following JSON format:

```json
{
  "operation":          // Operation that was performed
  "principal":          // Information about who initiated the operation
  "raw_details":        // Details about the request
  "x_forwarded_for":    // The IP address(es) making the call
  "emit_time":          // Time the operation was recorded
  "log_id":             // Unique ID of the log entry
  "async_operation_id": // Request or async operation identifier for correlation
  "request_id":         // DEPRECATED, use async_operation_id
  "status":             // Status, such as OK or ERROR
  "version":            // Version of the log entry
}
```

#### How to interpret the status

For supported Control Plane operations, the `status` field describes the outcome of the Temporal Cloud API call:

- `OK`: The API call returned successfully.
- `ERROR`: The API call returned an error. This includes authorization failures, such as when the authenticated principal does not have permission to perform the operation.

Audit logs do not capture authentication failures.

For an asynchronous operation, `OK` means that Temporal Cloud accepted the API call.
The operation can fail after the audit log event is emitted, and the event is not updated with the final outcome.
Use [`GetAsyncOperation`](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/operations/GET/cloud/operations/%7BasyncOperationId%7D) to check the operation state and failure reason.

The `async_operation_id` field can contain a request identifier or an async operation identifier.
Its presence does not indicate that the API call started an asynchronous operation.

> **📝 Note:**
>
> The [`X-Forwarded-For`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) format is a comma-separated list of IP addresses which should be evaluated from the last to the first, until meeting the first untrusted IP address of the list. This allows for instance to consider proxies in the path.
>
> Temporal provides the caller IP address in that format to allow customers to identify a caller IP address even if one (or more proxies) are in the network path to reach Temporal Cloud.
>

### Example of an audit log

```json
[
  {
    "operation": "UserLogin",
    "status": "OK",
    "version": 2,
    "logId": "edb3aa3e-78c4-48fc-9c7e-2078c6989775",
    "xForwardedFor": "10.1.2.3",
    "asyncOperationId": "",
    "emitTime": {
      "$typeName": "google.protobuf.Timestamp",
      "seconds": 1759436617,
      "nanos": 48000000
    },
    "principal": {
      "type": "user",
      "id": "",
      "name": "user@email.com",
      "apiKeyId": ""
    }
  },
  {
    "operation": "UserLogin",
    "status": "OK",
    "version": 2,
    "logId": "5fe6a81e-8d3c-4f4d-88a5-52db864c9ea5",
    "xForwardedFor": "10.1.2.3",
    "asyncOperationId": "",
    "emitTime": {
      "seconds": 1759178573,
      "nanos": 671000000
    },
    "principal": {
      "type": "user",
      "id": "",
      "name": "user@email.com",
      "apiKeyId": ""
    }
  }
]
```

## How to configure an audit log integration 

Audit Logs can be configured in AWS Kinesis or GCP Pub/Sub.

- [AWS Kinesis Instructions](/cloud/audit-logs-aws)
- [GCP Pub/Sub Instructions](/cloud/audit-logs-gcp)

## How to troubleshoot an audit log sink 

The Audit Logs page of the Temporal Cloud UI provides the current status of an Audit Log Integration.

- If an error is detected, a summary of the error appears below the page title.
- If the Audit Log Integration is functioning normally, an **On** badge appears next to the page heading.

After an Admin Operation is performed, users can see Audit Log messages flow through the stream.

Upon successful configuration of the Audit Log sink and set up of a stream, you will receive events within the hour of setup.
Temporal is able to retain Audit Log information for up to 30 days.
To retrieve logs up to the past 30 days, you will need to file a request.

If you experience an issue with an Audit Log sink, we can provide the missing audit information.
Open a support ticket to request assistance.

## How to delete an audit log sink 

To delete an Audit Log sink, follow these steps:

1. In the Temporal Cloud UI, select **Settings**.
1. On the **Settings** page, select **Audit Logs**.
1. In the **Audit Logs Integration** card, select **Edit**.
1. At the bottom of the **Audit Logs Integration** page, choose **Delete**.

After you confirm the deletion, the Audit Log Sink is removed from your account and logs stop flowing to your stream.

## View an audit log 

An Audit Log can be viewed in the Temporal Cloud UI.
1. In the Temporal Cloud UI, select **Settings**.
1. On the **Settings** page, select **Audit Logs**.

Up to 1,000 events can be downloaded from the Audit Log UI to a local file.

## Access an audit log via API 

An Audit Log can be accessed using the [Temporal Cloud Ops API](/ops). Use the API to access
an Audit Log if you wish to make dashboards for viewing an Audit Log outside of Temporal Cloud.
If your goal is to export an Audit Log, it is better to use an Audit Log sink and capture each 
entry as it is generated.

Audit Logs are accessible for the past 30 days using the API.

The API allows:
- StartTimeInclusive: Filter for UTC time >= (defaults to 30 days ago) - optional
- EndTimeExclusive: Filter for UTC time < (defaults to current time) - optional
- PageSize: Cannot exceed 1000. Defaults to 100. - optional
- PageToken: The page token if this is continuing from another response - optional

---

# Audit Logs - AWS Kinesis

Source: https://docs.temporal.io/cloud/audit-logs-aws

> Audit Logs in Temporal Cloud provides forensic information, integrating with AWS Kinesis for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes.

## Configure Audit Logs using AWS Kinesis 

To set up Audit Logs, you must have an Amazon Web Services (AWS) account and set up Kinesis Data Streams.

1. If you don't have an AWS account, follow the instructions from AWS in [Create and activate an AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/).
2. To set up Kinesis Data Streams, open the [AWS Management Console](https://aws.amazon.com/console/), search for Kinesis, and start the setup process.

You can use [this AWS CloudFormation template](https://temporal-auditlogs-config.s3.us-west-2.amazonaws.com/cloudformation/iam-role-for-temporal-audit-logs.yaml) to create an IAM role with access to a Kinesis stream you have in your account.

Be aware that Kinesis has a rate limit of 1,000 messages per second and quotas for both the number of records written and the size of the records.
For more information, see [Why is my Kinesis data stream throttling?](https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-data-stream-throttling/)

### Create an Audit Log sink

1. In the Temporal Cloud UI, select **Settings**.
1. On the **Settings** page, select **Audit Logs**.
1. In the **Audit Log Integration** card, select **Setup**.
1. On the **Audit Log Integration** page, choose your **Access method** (either **Auto** or **Manual**).
   - **Auto:** Configure the AWS CloudFormation stack in your AWS account from the Cloud UI.
   - **Manual:** Use a generated AWS CloudFormation template to set up Kinesis manually.
1. In **Kinesis ARN**, paste the Kinesis ARN from your AWS account.
1. In **Role name**, provide a name for a new IAM Role.
1. In **Select an AWS region**, select the appropriate region for your Kinesis stream.

If you chose the **Auto** access method, continue with the following steps:

1. Select **Save and launch stack**.
1. In **Stack name** in the AWS CloudFormation console, specify a name for the stack.
1. In the lower-right corner of the page, select **Create stack**.

If you chose the **Manual** access method, continue with the following steps:

1. Select **Save and download template**.
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Select **Create Stack**.
1. On the **Create stack** page, select **Template is ready** and **Update a template file**.
1. Select **Choose file** and specify the template you generated in step 1.
1. Select **Next** on this page and on the next two pages.
1. On the **Review** page, select **Create stack**.

To ensure that Audit Logs can flow into the Kinesis stream, you can use the **Verify** button to confirm it is set up correctly. This validates that Temporal can successfully write to your stream.
If everything is configured correctly, you will see a `Success` status indicating Temporal has written to the kinesis stream.

## Consume an Audit Log 

After you create an Audit Log sink, wait for the logs to flow into the Kinesis stream.
You will see the first logs within 10 minutes after you configure the sink.

> **📝 Note:**
>
> You must configure and implement your own consumer of the Kinesis stream.
> For an example, see [Example of consuming an Audit Log](#example-of-consuming-an-audit-log).
>

### Example of consuming an Audit Log

The following Go code is an example of consuming Audit Logs from a Kinesis stream and delivering them to an S3 bucket.

```go
func main() {
   fmt.Println("print audit log from S3")
   cfg, err := config.LoadDefaultConfig(context.TODO(),
      config.WithSharedConfigProfile("your_profile"),
   )
   if err != nil {
      fmt.Println(err)
   }
   s3Client := s3.NewFromConfig(cfg)
   response, err := s3Client.GetObject(
      context.Background(),
      &s3.GetObjectInput{
         Bucket: aws.String("your_bucket_name"),
         Key:    aws.String("your_s3_file_path")})
   if err != nil {
      fmt.Println(err)
   }
   defer response.Body.Close()

   content, err := io.ReadAll(response.Body)

   fmt.Println(string(content))
}
```

The preceding code also prints the logs in the terminal.
The following is a sample result.

```json
{
  "emit_time": "2023-11-14T07:56:55Z",
  "level": "LOG_LEVEL_INFO",
  "caller_ip_address": "10.1.2.3, 10.4.5.6",
  "user_email": "user1@example.com",
  "operation": "DeleteUser",
  "details": {
    "target_users": ["d7dca96f-adcc-417d-aafc-e8f5d2ba9fe1"],
    "search_attribute_update": {}
  },
  "status": "OK",
  "category": "LOG_CATEGORY_ADMIN",
  "log_id": "0mc69c0323b871293ce231dd1c7fb639",
  "request_id": "445297d3-43a7-4793-8a04-1b1dd1999640",
  "principal": {
    "id": "988cb80b-d6be-4bb5-9c87-d09f93f58ed3",
    "type": "user",
    "name": "user1@example.com"
  }
}
```

---

# Audit Logs - GCP Pub/Sub

Source: https://docs.temporal.io/cloud/audit-logs-gcp

> Audit Logs in Temporal Cloud provides forensic information, integrating with GCP Pub/Sub for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes.

## Manual setup prerequisites

> **📝 Note:**
>
> These steps are only required for manual setup.
> If you use Terraform for your deployment, you don't need to complete these prerequisites.
>

Before configuring the manual Audit Log sink, complete the following steps in Google Cloud:

1. Create a Pub/Sub topic and make a note of its topic name, such as `test-auditlog`.
1. Set up a service account in the same project in Google Cloud and follow the instructions in the
   Temporal Cloud UI to configure the permissions for that service account.

## Create an Audit Log sink

1. In the Temporal Cloud UI, select **Settings**.
1. On the **Settings** page, select **Audit Logs**.
1. In the **Audit Logs Integration** card, select **Setup**.
1. On the **Audit Log Integration** page, select **Pub/Sub**.
1. In the **service account email** field, enter the email of the service account you created in the prerequisites.
1. In the **Topic name** field, enter the topic name of the Pub/Sub topic you created in the prerequisites.
1. There are two ways to configure the service account to write to the Pub/Sub sink: select **Manual** to configure the account manually, or **Deploy with Terraform** to use Terraform.
   If you use Terraform, you don't need to complete the prerequisite steps above.
1. Follow the instructions in the Temporal Cloud UI for the method you chose.
1. To ensure that audit logs can reach your Pub/Sub topic, you can use the **Verify** button to confirm it is set up correctly. This validates that Temporal can successfully write to your topic.
If everything is configured correctly, you will see a `Success` status indicating Temporal has written to the Pub/Sub topic.
1. Click **Create** to configure the audit log.
   Audit Logs will begin to show up in Pub/Sub within 10 minutes 

![Temporal Cloud UI Setup for Audit Logs with GCP Pub/Sub](https://docs.temporal.io/img/cloud/gcp/audit-logging-pub-sub-gcp.png)

> **ℹ️ Info:**
> MORE INFORMATION
>
> For more details, refer to [Audit Logs with Temporal Cloud](/cloud/audit-logs).
>

---

# Billing Center

Source: https://docs.temporal.io/cloud/billing

> As an Account Owner, you can access and manage billing details anytime, with invoices available for download on the Billing page. Typical billing cycles begin on the first of the month (UTC).

## Current balance 

Your current balance card shows the balance for your current billing cycle and the date it was last updated.
This balance adjusts with use and appears on the first line of your Invoices table.

> **📝 Note:**
> Billing Cycles
>
> Billing cycles normally begin on the first of the month (UTC).
> The minimum plan fee for your first month is prorated based on your sign-up date.
>

## Recent bill 

The "Recent Bill" card displays the previous bill amount.

![Recent bill card showing a balance of $0.00](https://docs.temporal.io/img/cloud/billing/billing-card.png)

- If you pay your invoices through Stripe, you'll see a **Pay Now** button.
  It takes you to the Stripe portal to complete your payment
- If your account is set up for auto-payment, you don’t need to manually pay bills.
  However, you can choose to make manual payments whenever you wish

## Invoices 

To review your invoices, follow these steps:

1. Click **Billing** on your left-side vertical navigation.
2. Under the **Invoices** section, select and download the invoice(s) you want to review.

The Invoices table shows the following information:

- Date (UTC): The date range covered by the invoice
- Type: The type of invoice, such as credit purchase or cloud usage
- Status: The current status of the invoice, such as paid or pending
- Credit Granted: The total credits added to your account
- Credit Purchase Amount: The amount paid for purchasing credits
- Credit Usage: The credits used during the billing cycle
- Subtotal: The total amount of the invoice before any adjustments
- Balance Due: The amount to pay after applying credits

![Billing page showing Invoices tab](https://docs.temporal.io/img/cloud/billing/billing-invoices.png)

You may download your Invoices prior to this calendar month by clicking the download icon by the date.

> **📝 Note:**
> Current Month Invoice
>
> During the current billing period, your invoice will not be finalized and the download option will not be available.
>

## Credits 

The following information appears under the credits table:

- Effective At (UTC): The date when the credit grant became effective
- Type: Indicates whether the transaction was a deduction, expiry, or grant
- Amount: The credit amount that was granted, deducted, or expired
- Credits Remaining: The remaining credit available in the account

![Billing page showing Credits tab](https://docs.temporal.io/img/cloud/billing/billing-credits.png)

## Cost by Namespace 

> **💡 Tip:**
> Temporal Cloud Billing API
>
> To get Namespace-level cost data at hourly granularity, including Tags and Projects, use the [Temporal Cloud Billing API](/cloud/billing-api).
>

Account Owners and Finance Admins can access a cost column on the Usage page.
This allows you to monitor your cost on a per Namespace basis.
If your organization separates work by Namespace—for development, production, or different products—you can view costs for each.

![Billing page showing Usage](https://docs.temporal.io/img/cloud/billing/billing-usage.png)

> **📝 Note:**
> Cost Breakdown Limitations
>
> Namespace cost details are not available for "last 90 days" or "last 120 days".
>
> Cost breakdowns distribute the total usage cost to namespaces proportionally based on their metered usage. The proration
> reflects your effective price, factoring in included Actions/Storage and tiered pricing rates in your Temporal plan.
>

## Plans 

Account Owners and Finance Admins can access their Temporal Plan information on the plans page.
For customers on a standard agreement you will be able to:

- View current plan information, pricing details and entitlements
- View other available plans, pricing details and entitlements
- View Pay-as-You-Go pricing rates applicable to your plan
- Upgrade and Downgrade between plans available on a standard agreement

![Billing page showing Plans tab](https://docs.temporal.io/img/cloud/billing/billing-plans.png)

Requests to upgrade your plan are processed immediately and you will be billed on a pro-rated basis for that billing period.
Your monthly entitlements will reflect the full volume of included Actions and Storage of the upgrade plan for that billing month.
After an upgrade, a downgrade cannot be processed until the following billing period.

Requests to downgrade will be processed immediately. Billing and entitlements will be backdated to the beginning of the billing period.

## Account Cancellation

The way you created your Temporal account determines how you can cancel your subscription and remove the account.

- **For accounts managed by our sales team**. <br />
  Please submit a support ticket so we can help you.

- **For accounts created through our self-signup portal**. <br />
  Account owners can delete their accounts on the Temporal Cloud Billing page, under the **Plan** tab.
  If you're no longer using Temporal Cloud, use the Delete Account button to begin the process.

  - Permanently deleted accounts will immediately cease billing and be scheduled for full deletion within 72 hours.

  - Account Data and Active Storage will be permanently deleted. Retained Storage will be deleted in accordance with its configured retention period.

![Billing page showing the Plan tab. The contents on the tab include "Manage Payment Method" and "Delete Account" buttons. The "Delete Account" button is placed below text asking "No longer using Temporal Cloud?"](https://docs.temporal.io/img/cloud/billing/billing-cancel.png)

---

# Billing and usage management

Source: https://docs.temporal.io/cloud/billing-and-usage

> As an Account Owner, you can access and manage billing details anytime, with invoices available for download on the Billing page. Typical billing cycles begin on the first of the month (UTC).

Temporal Cloud provides billing and costs information for your account. 
Use this information to assess your spending patterns, inspect your credit ledger, 
check your invoice histories, update payment details, and manage your current plan as needed.

For more information on current Temporal Cloud pricing for Actions, storage, and services/support, 
please visit our [Pricing page](/cloud/pricing).

Usage on Temporal is measured in Actions and Storage. 
This can help understand your bills, forecast usage, optimize Workflows, and troubleshoot errors. 
You can view your Action usage in multiple ways.
The following tools are available for measuring Usage and Billing:

- **[Billing Center](/cloud/billing):** Allows you to see summary invoices and credits, manage plans, and delete your accounts.
    - Viewable by Account Owners and Finance Admin

- **[Billing API](/cloud/billing-api):** Allows you to access billing information on a Namespace basis down to an hourly granularity, enriched with Tags and Projects. The Billing API provides a FOCUS-guided data format that can be ingested into your cloud cost management platform or analytics tooling.
  - Viewable by Account Owners and Finance Admin

- **[Usage Dashboards](/cloud/actions-usage):** Aggregate Actions on a Namespace level and includes Action categories that groups similar types of Actions as seen in [Actions](/evaluate/cloud/actions). Available in the Cloud UI in the usage dashboard and Namespace overview pages.
    - Viewable by Account Owners, Finance Admin, Global Admin on an account level. Namespace level usage is visible on the Namespace pages to those with access.

- **[Actions in Event History](/cloud/actions-usage#actions-in-workflows):** Highlights Actions in a given Event History via the Temporal Cloud UI. Note that some Actions are not measured in Workflow histories.
    - Viewable by Account Owners, Global Admin and Namespace Admin, Developers and Read-only

- **[Actions Metrics](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count):** A high cardinality billable action metric that include labels for Category, Action Type, Workflow Type and Namespace down to minute granularity.
    - Viewable by creating a service account with the "Metrics Read-Only" role. See the [OpenMetrics](/cloud/metrics/openmetrics#api-key-authentication) page for more information.

---

# Cloud Billing API

Source: https://docs.temporal.io/cloud/billing-api

> Generate Namespace-level cost reports in FOCUS-friendly CSV, with hourly, daily, and monthly granularity for FinOps tools.

The Temporal Cloud Billing API provides Namespace-level cost attribution through on-demand billing reports. 
Reports are delivered in CSV format and can be accessed via API or downloaded directly for use in FinOps tooling and cost management platforms.  

This API is part of the [Cloud Operations API](/ops).

The Billing API allows you to:

- Generate billing reports for specified invoice months  
- Retrieve report status and metadata  
- Download CSV reports that can be fed into internal analytics tooling or cloud cost management platforms

The Billing Report contains:

- Accurate Namespace-level cost attribution  
- Hourly, daily, and monthly granularities
- A [FOCUS](https://focus.finops.org/)-friendly data format

For complete request and response schemas, refer to the Schema below.

Billing report generation is **asynchronous**. You initiate report creation, then poll for completion.

## Report data limitations

Reports can be generated with hourly, daily and monthly granularities, each of which have their own data ranges.
- Hourly: Current billing month and previous billing month
- Daily: Current billing month and previous two billing months
- Monthly: Current billing month and previous eleven billing months

Data from the current billing month is subject to change and only finalized once the billing month is closed.

## Allowed date ranges

Date ranges must use billing-month boundaries (MM/YYYY). 
Requests may include the current billing month. 
The data in finalized reports includes usage up to `current_time` \- 24 hours (rounded down to the granularity level).

## Rate limits and concurrency

Rate limits apply to API usage. 

### Per-account concurrency

Within a single account:

- Only one billing report per account is generated at a time  
- Additional requests are accepted but queued  

### Report generation latency

Report generation time varies and is not guaranteed. Factors that affect it include the size of the requested date range and overall platform load.

## Best practices

Provide an idempotency key (`async_operation_id`) when retrying requests.  

Poll `GetBillingReport` using exponential backoff.

Download reports immediately after generation (URLs expire).

Avoid frequent generation of large overlapping ranges in the current billing period.

## Billing report schema

Billing reports are delivered in CSV format.

Each row represents a charge record.

| Column Name | Description | Example |
| ----- | ----- | ----- |
| BillingAccountID | Temporal Cloud account ID | a2dd6 |
| BillingAccountName | Temporal Cloud account name | temporal |
| BillingCurrency | The currency an account is billed in | USD (cents) |
| BillingPeriodEnd | The exclusive end bound of a billing period | 2024-02-01T00:00:00Z |
| BillingPeriodStart | The inclusive start bound of a billing period | 2024-01-01T00:00:00Z |
| ChargeCategory | The highest level classification of a charge based on how it is billed | Usage |
| ChargeDescription | A self contained summary of the charge’s purpose | Actions \- Tier 1 |
| ChargeFrequency | Indicates how often a charge will occur | Usage-Based |
| ChargePeriodEnd | Time period end from when this charge took place, correlates to data granularity | 2025-10-01T01:00:00.000Z |
| ChargePeriodStart | Time period start from when this charge took place, correlates to data granularity | 2025-10-01T00:00:00.000Z |
| ContractedCost | Cost calculated by multiplying ContractedUnitPrice and PricingQuantity | 100.00 |
| ContractedUnitPrice | The agreed-upon unit price for a single pricing unit of the associated SKU. Inclusive of negotiated discounts | 10.00 |
| InvoiceID | The ID of the invoice for this billing period | in\_XXXXXXXXXXXXXXXXXXXX |
| InvoiceIssuer | The entity responsible for issuing payable invoices | stripe |
| PricingQuantity | The volume of a given SKU used or purchased | 10.00 |
| PricingUnit | The measurement unit used for PricingQuantity | 1 Million Actions |
| Provider | The provider of purchased resources or services | Temporal Technologies |
| Publisher | The publisher of purchased resources or services | Temporal Technologies |
| ResourceID | Namespace name \+ Temporal Cloud account ID | production.a2dd6 |
| ResourceName | Namespace name \+ Temporal Cloud account ID | production.a2dd6 |
| ResourceType | The type of resource the charge applies to | Namespace |
| ServiceCategory | The highest level classification of a service based on the core function of the service | Temporal Cloud |
| ServiceName | An offering that can be purchased from a provider | Temporal Cloud |
| ServiceSubcategory | A secondary classification of the service category for a service based on its core function | Actions |
| SKUID | A unique identifier that represents a specific SKU | business-actions |
| SKUMeter | The functionality being metered or measured by a particular SKU in a charge | Actions |
| Tags | Provider and customer defined tags associated with resources | `{"$tmprl_project":["project-id"],"namespace-tag-key":["namespace-tag-value"]}` |

## Generate a report

To generate a report, follow these steps:

1. Create a billing report using `CreateBillingReport`. The response includes a `billing_report_id` and `async_operation_id`.
1. Poll `GetBillingReport` using the `billing_report_id`  
1. When the report state becomes `BILLING_REPORT_STATE_GENERATED`, retrieve the download URL  
1. Download the report before the URL expires

### Key identifiers

| Identifier | Purpose |
| ----- | ----- |
| `billing_report_id` | Identifies the billing report and is used to retrieve metadata and download URLs |
| `async_operation_id` | Identifies the background operation responsible for generating the report |

The async operation follows the standard Cloud Operations async model (see [Async Operations](/ops#per-identity-rate-limits)).

---

# Capacity modes

Source: https://docs.temporal.io/cloud/capacity-modes

> Control how limits are assigned to a Namespace with Capacity Modes

Each Namespace in Temporal has a rate limit, which is measured in [Actions](/cloud/pricing#action) per second. 
Temporal offers two different modes for adjusting capacity: On-Demand Capacity or Provisioned Capacity.
With On-Demand Capacity, Namespace capacity is increased automatically along with usage.
With Provisioned Capacity, you can control your capacity limits by requesting Temporal Resource Units (TRUs).

## Namespace Capacity

Namespaces in Temporal can be set to either an **On-Demand** or **Provisioned Capacity** Mode. 
These modes govern how limits are assigned to a Namespace.  

Actions Per Second (APS) is the primary limit for Namespaces and is based on the operating billable Actions that occur each second.
Some Actions can result in multiple back-end operations, so limits are also set on Requests Per Second (RPS) and Operations Per Second (OPS) to maintain reliability.

See [Service-level RPS limits](/references/dynamic-configuration#service-level-rps-limits) for more about RPS.
See the [operations list](/references/operation-list) for the list of operations.
See the [Actions page](/evaluate/cloud/actions) for the list of actions.

> **💡 Tip:**
> Measuring throughput with APS, RPS, and OPS
>
> APS, RPS, and OPS are all measures of throughput that apply to different aspects of Temporal.
>
> APS, or Actions Per Second, is specific to Temporal Cloud. 
> It measures the rate at which Actions, like starting or signaling a Workflow, can be performed in a specific Namespace. 
> Temporal Cloud uses APS to protect the system from sudden major spikes in load. 
>
> RPS, or Requests Per Second, is used in the Temporal Service, both in self-hosted Temporal and Temporal Cloud. 
> It measures and controls the rate of gRPC requests to the Service. 
> This is a lower-level measure that manages rates at the service level. 
>
> OPS, or Operations per Second, is used by Temporal Cloud. 
> An operation is anything a user does directly, or that Temporal does on behalf of the user in the background, that results in load on Temporal Server. 
> This is a lower-level measure that manages rates across Temporal cloud services.
>
> In summary, APS is a higher-level measure to limit and mitigate Action spikes in Temporal Cloud. 
> RPS and OPS  are lower-level measures to control and balance request rates at the service level.
>

### What happens when my Actions rate exceeds my limit?

When your Action rate exceeds your quota, Temporal Cloud throttles Actions.
Throttling limits the rate at which Actions are performed to prevent the Namespace from exceeding its APS limit.

**How throttling works:**
- Low-priority operations are throttled first; higher-priority operations (like starting or signaling Workflows) continue when possible.
- Rate limiting is not instantaneous, so usage may briefly exceed your limit before throttling takes effect.
- When throttled, the server returns `ResourceExhausted` errors that SDK clients automatically retry.
- If throttling persists beyond the SDK's retry limit, client calls can fail.

**To avoid data loss during throttling:**
- Log any failed client calls (with payloads) so you can retry or backfill later.
- Set up [limit metrics](/cloud/metrics/openmetrics/metrics-reference#limit-metrics) to alert when approaching your limits.

See [Throttling behavior](/evaluate/cloud/limits#throttling-behavior) for more details.

Your rate limits can be adjusted automatically over time or provisioned manually with Capacity Modes.

We recommend tracking your Actions Rate and Limits using Temporal metrics to assess your use cases specific needs.
See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) to track usage trends.

For Namespaces using Provisioned Capacity, on-demand envelope metrics show what your limits would be if operating in on-demand mode.
Use these to evaluate whether on-demand capacity would meet your needs before switching modes.
See [On-demand envelope limits](/cloud/service-health#on-demand-envelope-limits) for details.

> **📝 Note:**
> Actions that don't count against APS
> Actions that are external to the core Temporal service do not contribute to your APS. These Calls include:
> * [Export](/cloud/export)
> * Capacity Related Actions

## On-demand capacity 

Using On-Demand Capacity, your rate limit grows automatically along with your usage.
Each Namespace has an Actions per second (APS), Requests per second (RPS), and Operations per second (OPS) limit that scales automatically with usage. Your APS limit never falls below its [default limit](/evaluate/cloud/limits#actions-per-second). If Temporal Support has manually set your Namespace's limit, that value becomes your floor in place of the default, and it persists across capacity mode changes.

Scaling automatically adjusts based on the lesser of 4 * APS Average or 2 * APS P90 over the past 7 days.

To see your Namespace's current APS, RPS, and OPS limits, track the `temporal_cloud_v1_action_limit`, `temporal_cloud_v1_service_request_limit`, and `temporal_cloud_v1_operations_limit` metrics. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits).

If you experience usage spikes, you may hit a throughput limit. 
In that case, consider switching to [Provisioned Capacity](#provisioned-capacity). 
You can also optimize your workload to remain under the On-Demand limits. See [Best Practices for Managing APS Limits](/best-practices/managing-aps-limits) for more information. 

### What kind of throughput can I get on Temporal Cloud with On-Demand Capacity?

Each Namespace has a rate limit, which is measured in Actions per second (APS). 
Your APS limit automatically adjusts based on a formula that compares your average usage over the last 7 days and your usage at the 90th percentile, or P90. 
Your throughput limit will never fall below the [default limit](/evaluate/cloud/limits#actions-per-second) for your Namespace. 
Under On-Demand capacity you are only charged for the Actions you use.
To see your current limit, view it in the Temporal Cloud UI under the Namespace overview, retrieve it with the CLI by running [`temporal cloud namespace capacity get`](/cli/command-reference/cloud/namespace#capacity-get) or [`tcld namespace capacity get`](/cloud/tcld/namespace/#capacity), or track the `temporal_cloud_v1_action_limit` metric described in [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits).

For example, if your average APS over the last 7 days was 200 and your P90 was 500, your limit would be the greater of:
* The [default limit](/evaluate/cloud/limits#actions-per-second)
* The lesser of:
  * 4 * 200 APS Mean = 800 APS
  * 2 * 500 APS P90 = 1000 APS

Because 800 APS exceeds the default limit, your limit in this example would be 800 APS.

![Usage graph showing increasing APS usage for one month, with occasional spikes, and a rising APS limit](https://docs.temporal.io/img/cloud/provisioned-capacity/usage_graph.png)

## Provisioned Capacity 

Provisioned Capacity provides an alternative to On-Demand Capacity by allowing you to control the limits on your Namespace based on your specific need.

|               | Actions Per Second | Requests Per Second | Operations Per Second|
|---------------|--------------------|---------------------|----------------------|
| TRU           | 500                | 1500                | 4000                 |

Customers can set 2, 3, 4, 6, 8, 10, 12 TRUs, subject to availability. TRUs can be adjusted hourly.

See [Capacity Mode Pricing](/cloud/pricing#capacity-modes-pricing) for pricing implications.

### What kind of throughput can I get with Temporal Cloud with Provisioned Capacity?

With Provisioned Capacity, you can set your rate limits by selecting the number of Temporal Resource Units (TRUs) on your Namespace.
Each TRU supports up to 500 APS and can be provisioned in groups of 2, 3, 4, 6, 8, 10, or 12 TRUs if there is capacity available in a region. 
When TRUs are requested we aim to provision the additional capacity within two minutes. 

> **💡 Tip:**
> Large TRU requests
>
> For Requests in excess of 4 TRUs in regions outside of the US, we recommend submitting a support ticket to ensure capacity availability.
>

### Provisioned Capacity Availability

The amount of capacity available within a region may vary. 
Temporal will check available capacity at the time of your request and aims to provision requested capacity within two minutes. 
If you need capacity beyond what is self-serviceable or available in a region, please [file a support ticket](/evaluate/cloud/support#ticketing) indicating the limit, region, and timeframe that the capacity is needed. 

> **⚠️ Caution:**
> GCP capacity restriction
>
> Additional TRUs in GCP regions are limited and may be unavailable. See [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction).
>

### When should I use Provisioned Capacity?

Provisioned Capacity works well when you’re aware of specific increases in load on your Namespace. For example:

* Planned events
* Unplanned events/usage spikes
* Known but sudden system spikes
* Load testing
* Migrating workloads

Depending on your usage patterns and your system monitoring, you can use Provisioned Capacity to quickly remedy rate limiting without contacting support.
You can also automate changes in capacity if you have a known event or a recurring usage pattern that produces predictable usage spikes. 

### Monitoring provisioned capacity utilization

Each TRU after the first carries a minimum charge of 360,000 Actions per calendar hour. Billing uses the highest TRU count configured during the hour. If workload usage falls below the minimum, Temporal adds Actions equal to the shortfall. See [Capacity Mode Pricing](/cloud/pricing#capacity-modes-pricing) for the billing formula.

For the metrics to watch and how to alert on utilization, see [Provisioned capacity utilization](/cloud/service-health#provisioned-capacity-utilization).

## Set capacity modes
Capacity Modes and TRUs can be set via the Temporal Cloud UI, CLI, or API. 
Capacity modes can be set and adjusted by Global Admin and Namespace Admin. 

When you switch to Provisioned Capacity, your limit is set by the number of TRUs you select.
When you switch back to On-Demand, your limit is recalculated from the trailing 7-day usage formula and never falls below your floor.
If Temporal Support has set a custom limit for your Namespace, that limit is your floor, so switching to Provisioned and back to On-Demand preserves it.

### Setting Capacity Modes from the UI

You can set Capacity Modes for an individual Namespace by navigating to the Namespace page in the Temporal Cloud UI (`https://cloud.temporal.io/namespaces/<Namespace ID>`). 
To view your current capacity configuration and change your capacity mode, navigate to the capacity tile and click *Manage Capacity*.

![Manage Capacity button in the Temporal UI](https://docs.temporal.io/img/cloud/provisioned-capacity/manage_capacity_button.png)

Under *Manage Capacity* you will be able to select between *On-Demand* and *Provisioned Capacity* modes. 
The *On-Demand* section will display your available On-Demand capacity. 
The *Provisioned* section will display the limit available with selected TRUs and the Included Actions required per hour. [See details on Provisioned Capacity Pricing](/cloud/pricing#capacity-modes-pricing). 

To switch to Provisioned capacity: 

1. Select the *Provisioned* radio button.
1. Specify the requested number of TRUs using the slider.
1. Check the dialog acknowledging potential pricing implications.
1. Click *Confirm*. 

In addition to the Capacity Mode selections, a summary of APS usage over the last seven days is included to help you estimate your current usage. 
For more detailed information, we recommend setting up metrics that track your APS and Limits. 
See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) to track usage trends.

![Manage Capacity panel in the Temporal UI](https://docs.temporal.io/img/cloud/provisioned-capacity/manage_capacity_panel.png)

### Set capacity modes from the CLI

**Temporal CLI**

```command
temporal cloud namespace capacity update \
  --namespace <namespace_name> \
  --capacity-mode provisioned \
  --capacity-value <tru_value>
```

Use this command to specify the Namespace name and configure the capacity settings:

* `--capacity-mode provisioned` sets a fixed capacity allocation for the Namespace.
* `--capacity-value` sets the provisioned throughput value in TRUs (Temporal Resource Units).

To switch to on-demand capacity, omit `--capacity-value`:

```command
temporal cloud namespace capacity update \
  --namespace <namespace_name> \
  --capacity-mode on_demand
```

Optional flags:

* `--async-operation-id` specifies an identifier for tracking the asynchronous operation. If not specified, the CLI generates one automatically.
* `--resource-version` specifies the resource version (etag) to update from. If not set, the CLI uses the latest version.

**tcld**

```command
tcld namespace capacity update \
  --namespace <namespace_name> \
  --capacity-mode <on_demand|provisioned> \
  --capacity-value <tru_value> \
  [--request-id <request_id> --resource-version <resource-version>]
```

Use this command to specify the Namespace name and configure the capacity settings:

* `--capacity-mode` sets the billing mode for the Namespace. Use `on_demand` for automatic scaling or `provisioned` for a fixed capacity allocation.
* `--capacity-value` sets the throughput value in TRUs (Temporal Resource Units).

Optional flags:

* `--request-id` specifies a request identifier for the asynchronous operation. If not specified, the server assigns one automatically.
* `--resource-version` specifies the resource version (etag) to update from. If not set, the CLI uses the latest version.

If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before capacity update.

### Setting Capacity Modes from the API

Call the `UpdateNamespace` API after Namespace creation and define the desired capacity state as part of the capacity spec.

---

# Authenticate with mTLS certificates

Source: https://docs.temporal.io/cloud/certificates

[Temporal Cloud](https://temporal.io/cloud) supports both mTLS and [API key](/cloud/api-keys) authentication for
namespace access.

When using mTLS authentication, each [Worker Process](/workers#worker-process) uses a CA certificate and private key to
connect to Temporal Cloud. Temporal Cloud does not require an exchange of secrets; only the certificates produced by
private keys are used for verification.

> **⚠️ Caution:**
> Don't let your certificates expire
>
> An expired root CA certificate invalidates all downstream certificates.
>
> An expired end-entity certificate prevents a [Temporal Client](/encyclopedia/temporal-client) from
> connecting to a Namespace or starting a Workflow Execution. If the client is on a Worker, any current Workflow
> Executions that are processed by that Worker either run indefinitely without making progress until the Worker resumes or
> fail because of timeouts.
>
> Temporal Cloud sends [courtesy emails](/cloud/notifications#admin-notifications) prior to certificate expiry.
>
>
> To update certificates, see
> [How to add, update, and remove certificates in a Temporal Cloud Namespace](#manage-certificates).
>

All certificates used by Temporal Cloud must meet the following requirements.

## Requirements for CA certificates in Temporal Cloud 

Certificates provided to Temporal for your [Namespaces](/namespaces) _must_ meet the following requirements.

**When a client presents an end-entity certificate, and the whole certificate chain is constructed, each certificate in
the chain (from end-entity to the root) must have a unique Distinguished Name.**

> **⚠️ Caution:**
>
> Distinguished Names are _not_ case sensitive; that is, uppercase letters (such as ABC) and lowercase letters (such as
> abc) are equivalent.
>

### CA certificates

A CA certificate is a type of X.509v3 certificate used for secure communication and authentication. In Temporal Cloud,
CA certificates are required for configuring mTLS.

CA certificates _must_ meet the following criteria:

- The certificates must be X.509v3.
- Each certificate in the bundle must be either a root certificate or issued by another certificate in the bundle.
- Each certificate in the bundle must include `CA: true`.
- A certificate cannot be a well-known CA (such as DigiCert or Let's Encrypt) _unless_ the user also specifies
  certificate filters.
- The signing algorithm must be either RSA or ECDSA and must include SHA-256 or stronger message authentication. SHA-1
  and MD5 cannot be used.
- The certificates cannot be generated with a passphrase.

> **ℹ️ Info:**
>
> A certificate bundle can contain up to 16 CA certificates. A certificate bundle can have a maximum payload size of 32 KB
> before base64 encoding.
>

### End-entity certificates

An end-entity certificate is a type of X.509v3 certificate used by clients to authenticate themselves. Temporal Cloud
lets you limit access to specific end-entity certificates by using [certificate filters](#manage-certificate-filters).

An end-entity (leaf) certificate _must_ meet the following criteria:

- The certificate must be X.509v3.
- Basic constraints must include `CA: false`.
- The key usage must include Digital Signature.
- The signing algorithm must be either RSA or ECDSA and must include SHA-256 or stronger message authentication. SHA-1
  and MD5 cannot be used.
- The certificate needs the `extendedKeyUsage` set to `clientAuth` or it needs to be omitted.

## How to issue root CA and end-entity certificates 

Temporal Cloud authenticates a client connection by validating the client certificate against one or more CA
certificates that are configured for the specified Namespace.

Choose one of the following options to generate and manage the certificates:

### Option 1: You already have certificate management infrastructure

If you have existing certificate management infrastructure that supports issuing CA and end-entity certificates, export
the CA and generate an end-entity certificates using your existing tools.

Ensure that the CA certificate is long-lived and that the end-entity certificate expires before the CA certificate.

Follow the instructions to [upload the CA certificate](/cloud/certificates#update-certificates-using-temporal-cloud-ui)
and [configure your client](/cloud/certificates#configure-clients-to-use-client-certificates) with the end-entity
certificate.

### Option 2: You don't have certificate management infrastructure

If you don't have an existing certificate management infrastructure, we recommend using API keys for authentication
instead. API keys are generally easier to manage than mTLS certs if you're not using certificate management
infrastructure otherwise.

If you still want to use mTLS, issue the CA and end-entity certificates using [tcld](#use-tcld-to-generate-certificates)
or open source tools like OpenSSL or [step CLI](#use-step-cli-to-generate-certificates).

#### Use tcld to generate certificates

You can generate CA and end-entity certificates by using [tcld](/cloud/tcld).

Although Temporal Cloud supports long-lived CA certificates, a CA certificate generated by [tcld](/cloud/tcld) has a
maximum duration of 1 year (`-d 1y`). You must set an end-entity certificate to expire before its root CA certificate,
so specify its duration appropriately.

To create a new CA certificate, use `tcld gen ca`.

```sh
mkdir temporal-certs
cd temporal-certs
tcld gen ca --org temporal -d 1y --ca-cert ca.pem --ca-key ca.key
```

The contents of the generated `ca.pem` should be pasted into the "Authentication" section of your Namespace settings
page.

To create a new end-entity certificate, use `tcld gen leaf`.

```sh
tcld gen leaf --org temporal -d 364d --ca-cert ca.pem --ca-key ca.key --cert client.pem --key client.key
```

You can now use the generated CA certificate (`ca.pem`) with Temporal Cloud and configure your client with these certs
(`client.pem`, `client.key`).

Upload the contents of the `ca.pem` file to the **Authentication** section of your **Namespace** settings.

Follow the instructions to [upload the CA certificate](/cloud/certificates#update-certificates-using-temporal-cloud-ui)
and [configure your client](/cloud/certificates#configure-clients-to-use-client-certificates) with the end-entity
certificate.

#### Use step CLI to generate certificates

Temporal Cloud requires client certificates for authentication and secure communication.
[The step CLI](https://github.com/smallstep/cli) is a popular and easy-to-use tool for issuing certificates.

Before you begin, ensure you have installed smallstep/cli by following the instructions in the
[installation guide](https://github.com/smallstep/cli#installation).

A Certificate Authority (CA) is a trusted entity that issues digital certificates. These certificates certify the
ownership of a public key by the named subject of the certificate. End-entity certificates are issued and signed by a
CA, and they are used by clients to authenticate themselves to Temporal Cloud.

Create a self-signed CA certificate and use it to issue an end-entity certificate for your Temporal Cloud namespace.

##### 1. Create a certificate authority (CA)

Create a new Certificate Authority (CA) using step CLI:

```command
step certificate create "CertAuth" CertAuth.crt CertAuth.key --profile root-ca --no-password --insecure
```

This command creates a self-signed CA certificate named `CertAuth.crt` and private key `CertAuth.key`. This CA certificate will be used to sign and issue end-entity certificates.

##### 2. Set the Namespace Name

Set the Namespace Name as the common name for the end-entity certificate:

**macOS**

For Linux or macOS:

```command
export NAMESPACE_NAME=your-namespace
```

**Windows**

For Windows:

```command
set NAMESPACE_NAME=your-namespace
```

Replace `your-namespace` with the name of your Temporal Cloud namespace.

##### 3. Create and sign an end-entity certificate

Create and sign an end-entity certificate with a common name equal to the Namespace Name:

```command
step certificate create ${NAMESPACE_NAME} ${NAMESPACE_NAME}.crt ${NAMESPACE_NAME}.key --ca CertAuth.crt --ca-key CertAuth.key --no-password --insecure --not-after 8760h
```

This command creates an end-entity certificate (`your-namespace.crt`) and private key (`your-namespace.key`) that is signed by your CA (`CertAuth`).

##### 4. Convert to PKCS8 format for Java SDK (optional)

If you are using the Temporal Java SDK, you will need to convert the PKCS1 file format to PKCS8 file format. Export the
end-entity's private key to a PKCS8 file:

```command
openssl pkcs8 -topk8 -inform PEM -outform PEM -in ${NAMESPACE_NAME}.key -out ${NAMESPACE_NAME}.pkcs8.key -nocrypt
```

##### 5. Use the Certificates with Temporal Cloud

You can now use the generated client certificate (`your-namespace.crt`) and the CA certificate (`CertAuth.crt`) with
Temporal Cloud.

Upload the contents of the `CertAuth.crt` file to the **CA Certificates** section of your **Namespace** settings.

Follow the instructions to [upload the CA certificate](/cloud/certificates#update-certificates-using-temporal-cloud-ui)
and [configure your client](/cloud/certificates#configure-clients-to-use-client-certificates) with the end-entity
certificate.

## How to control authorization for Temporal Cloud Namespaces 

We recommend that an end-entity certificate be scoped to a specific Namespace to enforce the principle of least
privilege. Temporal Cloud requires full CA chains, so you can achieve authorization in two ways.

### Option 1: Issue a separate root certificate for each Namespace

Each certificate must belong to a chain up to the root CA certificate. Temporal uses the root CA certificate as the
trusted authority for access to your Namespaces.

1. Ensure that your certificates meet the [certificate requirements](#certificate-requirements).
1. [Add client CA certificates to a Cloud Namespace](/cli/command-reference/cloud/namespace#mtls-cert-ca-create).

### Option 2: Use the same root certificate for all Namespaces but create a separate certificate filter for each Namespace

[How to manage certificate filters in Temporal Cloud](#manage-certificate-filters)

## How to receive notifications about certificate expiration 

To keep your Namespace secure and online, you must update the CA certificate for the Namespace _before_ the certificate
expires.

To help you remember to do so, Temporal Cloud sends [email notifications](/cloud/notifications#admin-notifications). 

## How to handle a compromised end-entity certificate

> **⚠️ Warning:**
>
> Temporal does not support or check certificate revocation lists (CRLs). Customers are expected to keep their
> certificates up to date.
>

The recommended approach to avoiding compromised certificates is to have short-lived end-entity certificates. A
short-lived compromised certificate can be left to expire on its own. Seek guidance from your infosec team to determine
an appropriate value of "short-lived" for your business.

If you suspect or confirm that an end-entity certificate has been compromised, and leaving it to expire is not an
option, take immediate action to secure your Temporal Cloud Namespace and prevent unauthorized access.

If you're using certificate filters, you can set the filters to block a compromised certificate. Follow the instructions
in [How to manage certificate filters in Temporal Cloud](#manage-certificate-filters).

If you need to replace a compromised certificate manually, follow these steps:

### 1. Generate a new CA certificate

Follow the instructions in [How to issue CA and end-entity certificates](#issue-certificates). All end-entity
certificates that can be reached by the previous CA must be regenerated. Ensure the new CA certificate meets
[the certificate requirements](#certificate-requirements).

### 2. Deploy the new CA certificate to the Namespace

Follow the instructions for issuing a
[new CA certificate to a Namespace](#option-1-issue-a-separate-root-certificate-for-each-namespace). Deploy the new CA
certificate alongside the existing one, so you don’t lose connectivity with old end-entity certificates before the new
ones are generated and deployed.

### 3. Regenerate end-entity certificates with the new CA certificate

[Configure all clients](/cloud/certificates#configure-clients-to-use-client-certificates)
(Temporal CLI, SDKs, or Workers) to use the new certificate and private key alongside the compromised key. Update the
client configuration as described in
[Configure clients to use client certificates](#configure-clients-to-use-client-certificates).

Test the new certificate to confirm clients can connect to the Namespace without issues.

### 4. Remove the compromised CA certificate

Follow the instructions in
[How to add, update, and remove certificates in a Temporal Cloud Namespace](#manage-certificates) to remove the
compromised CA certificate from the Namespace.

### 5. Monitor and audit

After implementing the changes, monitor your Namespace for unauthorized access attempts or unusual activity in audit
logs. Review your certificate management practices to identify how the compromise occurred. Consider implementing
stricter controls, such as:

- Limiting end-entity certificates to specific Namespaces
- Rotating certificates regularly as a preventive measure
- [Audit logs on a regular schedule](/cloud/audit-logs)

## How to add, update, and remove certificates in a Temporal Cloud Namespace 

> **📝 Note:**
>
> To manage certificates for a Namespace, a user must have [Namespace Admin](/cloud/manage-access/roles-and-permissions#namespace-level-permissions)
> permission for that Namespace.
>

To manage certificates for Temporal Cloud Namespaces, use the **Namespaces** page in Temporal Cloud UI or the
[`temporal cloud namespace mtls cert-ca` commands](/cli/command-reference/cloud/namespace#mtls-cert-ca).

Don't let your certificates expire! Add reminders to your calendar to issue new CA certificates well before the
expiration dates of the existing ones. Temporal Cloud begins sending notifications 15 days before expiration. For
details, see the previous section
([How to receive notifications about certificate expiration](#expiration-notifications)).

When updating CA certificates, it's important to follow a rollover process (sometimes referred to as "certificate
rotation"). Doing so enables your Namespace to serve both CA certificates for a period of time until traffic to your old
CA certificate ceases. This prevents any service disruption during the rollover process.

### Update certificates using Temporal Cloud UI

Updating certificates using the following strategy allows for a zero-downtime rotation of certificates.

1. On the left side of the window, select **Namespaces**.

2. Select the name of the Namespace to update.

3. In the top-right portion of the page for the Namespace, select **Edit**.

4. On the **Edit** page, select the **Authentication** card to expand it.

5. In the certificates box, scroll to the end of the existing certificate (that is, past `-----END CERTIFICATE-----`).

6. On the following new line, paste the entire PEM block of the new certificate.

7. Select **Save**.

8. Wait until all Workers are using the new certificate.

9. Return to the **Edit** page of the Namespace and select the **Authentication** card.

10. In the certificates box, delete the old certificate, leaving the new one in place.

11. Select **Save**.

### Update certificates using the Temporal CLI

Updating certificates using the following strategy allows for a zero-downtime rotation of certificates.

1. Add the new CA certificate alongside the existing certificate:

   ```command
   temporal cloud namespace mtls cert-ca create \
     --namespace <namespace>.<account> \
     --ca-certificate-file <new-ca-certificate.pem>
   ```

1. Monitor traffic to the old certificate until it ceases.

1. Remove the old CA certificate:

   ```command
   temporal cloud namespace mtls cert-ca delete \
     --namespace <namespace>.<account> \
     --ca-certificate-file <old-ca-certificate.pem>
   ```

### Update certificates using tcld

Updating certificates using the following strategy allows for a zero-downtime rotation of certificates.

1. Create a single file that contains both your old and new CA certificate PEM blocks. Just concatenate the PEM blocks
   on adjacent lines.

   ```
   -----BEGIN CERTIFICATE-----
   ... old CA cert ...
   -----END CERTIFICATE-----
   -----BEGIN CERTIFICATE-----
   ... new CA cert ...
   -----END CERTIFICATE-----
   ```

1. Run the `tcld namespace accepted-client-ca set` command with the CA certificate bundle file.

   ```bash
   tcld namespace accepted-client-ca set --ca-certificate-file <path>
   ```

1. Monitor traffic to your old certificate until it ceases.

1. Create another file that contains only the new CA certificate.

1. Run the `tcld namespace accepted-client-ca set` command again with the updated CA certificate bundle file.

## How to manage certificate filters in Temporal Cloud 

To limit access to specific [end-entity certificates](#end-entity-certificates), create certificate filters. Each filter
contains values for one or more of the following fields:

- commonName (CN)
- organization (O)
- organizationalUnit (OU)
- subjectAlternativeName (SAN)

Corresponding fields in the client certificate must match every specified value in the filter.

The values for the fields are case-insensitive. If no wildcard is used, each specified value must match its field
exactly.

To match a substring, place a single `*` wildcard at the beginning or end (but not both) of a value. You cannot use a
`*` wildcard by itself.

You can create a maximum of 25 certificate filters in a Namespace.

If you provide a well-known CA certificate, you cannot clear a certificate filter. A well-known CA certificate is one
that is typically included in the certificate store of an operating system.

**Examples**

In the following example, only the CN field of the certificate's subject is checked, and it must be exactly
`code.example.com`. The other fields are not checked.

```json
AuthorizedClientCertificate {
  CN : "code.example.com"
}
```

In the following example, the CN field must be `stage.example.com` and the O field must be `Example Code Inc.`

```json
AuthorizedClientCertificate {
  CN : "stage.example.com"
  O : "Example Code Inc."
}
```

When using a `*` wildcard, the following values are valid:

- `*.example.com` matches `code.example.com` and `text.example.com`.
- `Example Code*` matches `Example Code` and `Example Code Inc`.

The following values are not valid:

- `.example.*`
- `code.*.com`
- `*`

### Manage certificate filters using Temporal Cloud UI

To add or remove a certificate filter, follow these steps:

1. On the left side of the window, click **Namespaces**.
1. On the **Namespaces** page, click the name of the Namespace to manage.
1. On the right side of the page for the selected Namespace, click **Edit**.
1. On the **Edit** page, click the **Authentication** card.
   - To add a certificate filter, click **Add a Certificate Filter** and enter values in one or more fields.
   - To remove a certificate filter, click the **×** in the upper-right corner of the filter details.
1. To cancel your changes, click **Back to Namespace**. To save your changes, click **Save**.

### Manage certificate filters using the Temporal CLI

Use the following commands to add, remove, or list certificate filters:

- [`temporal cloud namespace mtls cert-filter create`](/cli/command-reference/cloud/namespace#mtls-cert-filter-create)
- [`temporal cloud namespace mtls cert-filter delete`](/cli/command-reference/cloud/namespace#mtls-cert-filter-delete)
- [`temporal cloud namespace mtls cert-filter list`](/cli/command-reference/cloud/namespace#mtls-cert-filter-list)

### Manage certificate filters using tcld

To set or clear certificate filters, use the following [tcld](/cloud/tcld) commands:

- [tcld namespace certificate-filters import](/cloud/tcld/namespace/#import)
- [tcld namespace certificate-filters clear](/cloud/tcld/namespace/#clear)

To view the current certificate filters, use the
[tcld namespace certificate-filters export](/cloud/tcld/namespace/#export) command.

## Configure clients to use client certificates 

- [Go SDK](/develop/go/client/temporal-client#connect-to-temporal-cloud)
- [Java SDK](/develop/java/client/temporal-client#connect-to-temporal-cloud)
- [PHP SDK](/develop/php/client/temporal-client#connect-to-a-dev-cluster)
- [Python SDK](/develop/python/client/temporal-client#connect-to-temporal-cloud)
- [TypeScript SDK](/develop/typescript/client/temporal-client#connect-to-temporal-cloud)
- [.NET SDK](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud)

### Configure Temporal CLI 

To connect to a Temporal Namespace using the Temporal CLI and certificate authentication, specify your credentials and
the TLS server name:

```sh
temporal <command> <subcommand> \
    --tls-ca-path <Path to server CA certificate> \
    --tls-cert-path <Path to x509 certificate> \
    --tls-key-path <Path to private certificate key> \
    --tls-server-name <Override for target TLS server name>
```

For more information on Temporal CLI environment variables, see
[Environment variables](/cli/setup-cli#environment-variables).

---

# Connectivity

Source: https://docs.temporal.io/cloud/connectivity

> Network connectivity details for using Temporal Cloud

## Private network connectivity for namespaces

Temporal Cloud supports private connectivity to Namespaces via AWS PrivateLink or GCP Private Service Connect, in addition to the default public internet endpoints.

Namespace access is always securely authenticated via [API keys](/cloud/api-keys#overview) or [mTLS](/cloud/certificates), regardless of how you choose to connect.

For information about IP address stability and allowlisting, see [IP addresses](/cloud/connectivity/ip-addresses). If your security team requires allowlisting specific IP addresses for firewall rules, Temporal Cloud also offers a [Stable IPs configuration option](/cloud/connectivity/ip-addresses) that provides non-changing IP addresses for your Namespace endpoint.

### Required steps

Setting up private connectivity is a three-step process — and it's important to understand that **private connectivity** (the network path) and **Connectivity Rules** (Temporal's enforcement layer) are related but separate concepts:

1. **Set up the private connection** from your VPC to the region where your Temporal Namespace is located.
1. **Update your private DNS and/or client configuration** to actually use the private connection. Activating private connectivity does not change your Namespace Endpoint or Regional Endpoint automatically — clients keep resolving the public addresses until you do this step.
1. **(GCP PSC: required. AWS PrivateLink: optional.) Create a Connectivity Rule** for the private connection and attach it to the target Namespace(s). This blocks all access to the Namespace that does not arrive over a configured connection. You can mix private and public rules to also allow internet connectivity.

For steps 1 and 2, follow the guide for your Namespace's cloud provider:
- [AWS PrivateLink](/cloud/connectivity/aws-connectivity) creation and private DNS setup
- [Google Cloud Private Service Connect](/cloud/connectivity/gcp-connectivity) creation and private DNS setup

> **⚠️ Caution:**
> Finish client setup (complete step 2)
>
> After creating a private connection, you must set up private DNS or update the configuration of all clients you want to use the private connection.
>
> We recommend using private DNS.
>
> Without this step, your clients may connect to the Namespace over the internet if they were previously using public connectivity, or they will not be able to connect at all.
>
> If that's not an option for you, refer to [our guide for updating the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity).
>

For step 3, keep reading for details on [Connectivity Rules](/cloud/connectivity#connectivity-rules).

## Connectivity rules

> **ℹ️ Info:**
> Web UI Connectivity
>
> The Temporal Cloud Web UI is not currently subject to connectivity rule enforcement.
> Even if a namespace is configured with private connectivity rules, the Web UI for that namespace remains accessible over the public internet.
>

### Definition

Connectivity Rules are Temporal Cloud's mechanism for restricting the network paths that can reach a Namespace. They are enforced by Temporal Cloud — they do not create or modify the underlying network connection.

By default, a Namespace has zero Connectivity Rules and is reachable over (1) the public internet and (2) any private connections you've already configured to the region containing the Namespace. Namespace access is always securely authenticated via [API keys](/cloud/api-keys#overview) or [mTLS](/cloud/certificates), regardless of Connectivity Rules.

When you attach one or more Connectivity Rules to a Namespace, Temporal Cloud immediately blocks any traffic that does not match a rule on that Namespace. A Namespace can have multiple Connectivity Rules, and you can mix public and private rules.

Each Connectivity Rule specifies either generic public (internet) access or a specific private connection.

#### When you need a Connectivity Rule

| Provider | Connectivity Rule for private access | Why |
| -------- | ------------------------------------ | --- |
| AWS PrivateLink | **Optional.** Add one only if you want to enforce private-only access (block internet traffic to that Namespace). | AWS PrivateLink connections become usable as soon as the VPC endpoint is `Available`. Adding a Connectivity Rule restricts access; it does not establish it. |
| GCP Private Service Connect | **Required.** The PSC endpoint stays in `Pending` until a matching Connectivity Rule is created. | The Connectivity Rule is what tells Temporal Cloud to accept the PSC connection. |

A public Connectivity Rule can optionally enable [Stable IPs](/cloud/connectivity/ip-addresses#stable-ip-addresses), so Namespaces attached to it resolve their Namespace endpoint to a published, fixed set of IP addresses you can allowlist in your firewall.

Each Project allows 1 public Connectivity Rule and up to 5 private Connectivity Rules. To learn more about limits, see [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).

> **⚠️ Caution:**
>
> Connectivity Rules cannot be updated in place. If this Project already has a public Connectivity Rule and you want to turn on Stable IPs, delete that Project's public rule, create a new one with Stable IPs enabled, and re-attach it to every Namespace in that Project that needs it. A public rule in another Project can stay. Creating a second public rule in the same Project returns an error. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).
>

An AWS PrivateLink (PL) private Connectivity Rule requires:

- `connection-id`: The **VPC endpoint identifier** of the PL connection — the `vpce-…` value from your AWS account, *not* the endpoint service or DNS name (ex: `vpce-00939a7ed9EXAMPLE`).
- `region`: The region of the PL connection, prefixed with `aws-` (ex: `aws-us-east-1`). Must be the same region as the Namespace. Refer to the [Temporal Cloud region list](/evaluate/cloud/regions) for supported regions.

A GCP Private Service Connect (PSC) private Connectivity Rule requires:

- `connection-id`: The **PSC connection identifier** of the endpoint (ex: `1234567890123456789`). Find it on the endpoint's detail page in the Google Cloud console.
- `region`: The region of the PSC connection, prefixed with `gcp-` (ex: `gcp-us-east1`). Must be the same region as the Namespace. Refer to the [Temporal Cloud region list](/evaluate/cloud/regions) for supported regions.
- `gcp-project-id`: The identifier of the GCP project where you created the PSC connection (ex: `my-example-project-123`).

Connectivity Rules can be created and managed with the [Temporal Cloud CLI extension](/cli/cloud), [tcld](/cloud/tcld/), [Terraform](https://github.com/temporalio/terraform-provider-temporalcloud/), or the [Cloud Ops API](/ops). There is no Temporal Cloud Web UI option for creating or managing Connectivity Rules at this time.

> **💡 Tip:**
> Connectivity Rules give Temporal visibility into your private connections
>
> Without a Connectivity Rule, Temporal Cloud has no record that your PrivateLink or PSC endpoint exists. If you open a support ticket about a private-connectivity issue, having a Connectivity Rule attached to the affected Namespace lets us correlate the connection on our side and is the fastest path to debugging.
>

### Permissions and limits

Only [Account Admins and Account Owners](/cloud/manage-access/roles-and-permissions#account-level-roles) can create and manage connectivity rules. Connectivity rules are visible to Account Developers, Account Admins, and Account Owners.

By default each Namespace is limited to 5 private Connectivity Rules. Without [Projects](/cloud/projects), the Account allows 50 private Connectivity Rules and 1 public Connectivity Rule, for a total of 51. With Projects enabled, each Project allows 1 public Connectivity Rule and up to 5 private Connectivity Rules. Connectivity Rules across all Projects count toward the Account total of 51. You can [contact support](/evaluate/cloud/support#support-ticket) to request a higher limit. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).

A public Connectivity Rule can be reused for every Namespace in that Project that should be available on the internet. Creating a second public rule in the same Project returns an error.

## Creating a connectivity rule

### Temporal Cloud CLI

**Temporal CLI**

Create a private connectivity rule (AWS):

```bash
temporal cloud connectivity private create --connection-id "vpce-abcde" --region "aws-us-east-1"
```

Create a private connectivity rule (GCP):

```bash
temporal cloud connectivity private create --connection-id "1234567890" --region "gcp-us-central1" --gcp-project-id "my-project-123"
```

Create a public Connectivity Rule once for the Project. Each Project allows 1 public Connectivity Rule. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules):

```bash
temporal cloud connectivity public create
```

**tcld**

Create a private connectivity rule (AWS):

```bash
tcld connectivity-rule create --connectivity-type private --connection-id "vpce-abcde" --region "aws-us-east-1"
```

Create a private connectivity rule (GCP):

```bash
tcld connectivity-rule create --connectivity-type private --connection-id "1234567890" --region "gcp-us-central1" --gcp-project-id "my-project-123"
```

Create a public Connectivity Rule once for the Project. Each Project allows 1 public Connectivity Rule. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules):

```bash
tcld connectivity-rule create --connectivity-type public
```

The `cr` alias works the same way:

```bash
tcld cr create --connectivity-type private --connection-id "vpce-abcde" --region "aws-us-east-1"
```

```bash
tcld cr create --connectivity-type public
```

To enable [Stable IPs](/cloud/connectivity/ip-addresses#stable-ip-addresses) on the public rule, use the Cloud Ops API or Terraform. See [How to enable Stable IPs](/cloud/connectivity/ip-addresses#how-to-enable-stable-ips).

### Terraform

[Examples in the Terraform repo](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/examples/resources/temporalcloud_connectivity_rule/resource.tf)

## Attach connectivity rules to a namespace

Be careful! When any connectivity rules are set on a namespace, that namespace is ONLY accessible via the connections defined in those rules. If you remove a connectivity rule that your workers are using, your traffic will be interrupted.

If you already have workers using a namespace, adding both a public rule and any private rules simultaneously can help you avoid unintended loss of access. You can then ensure all workers are using private connections, and then remove the public rule.

### Temporal Cloud CLI

**Temporal CLI**

Attach connectivity rules to a namespace:

```bash
temporal cloud namespace connectivity attach --namespace "my-namespace.abc123" \
  --connectivity-rule-id "rule-id-1" --connectivity-rule-id "rule-id-2"
```

Rules attach and detach individually. To detach `rule-c` while leaving `rule-a` and `rule-b` in place, detach only `rule-c`:

```bash
temporal cloud namespace connectivity detach --namespace "my-namespace.abc123" \
  --connectivity-rule-id "rule-c"
```

Detaching every rule makes the namespace public. List the attached rules first, then detach them:

```bash
temporal cloud namespace connectivity list --namespace "my-namespace.abc123"
```

**tcld**

Set the connectivity rules on a namespace:

```bash
tcld namespace set-connectivity-rules --namespace "my-namespace.abc123" --connectivity-rule-ids "rule-id-1" --connectivity-rule-ids "rule-id-2"
```

Or using aliases:

```bash
tcld n scrs -n "my-namespace.abc123" --ids "rule-id-1" --ids "rule-id-2"
```

Connectivity rules are attached as a set, so if rules `rule-a`, `rule-b`, and `rule-c` were attached to a namespace and you wanted to detach `rule-c` only, you'd make one call attaching both `rule-a` and `rule-b`:

```bash
tcld namespace set-connectivity-rules --namespace "my-namespace.abc123" --ids rule-a --ids rule-b
```

Remove all connectivity rules (this will make the namespace public):

```bash
tcld namespace set-connectivity-rules --namespace "my-namespace.abc123" --remove-all
```

### Terraform

[Example in the Terraform repo](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main/examples/resources/temporalcloud_namespace/resource.tf#L113-L128)

## View the connectivity rules for a namespace

You have two ways to view the connectivity rules attached to a particular namespace.

### Get namespace

Connectivity rules are included in the namespace details returned by the `namespace get` command. 

**Temporal CLI**

```bash
temporal cloud namespace get -n "my-namespace.abc123"
```

**tcld**

```bash
tcld namespace get -n "my-namespace.abc123"
```

### List connectivity rules by namespace

To see only the connectivity rules for a specific namespace, without other namespace details, list the rules for that namespace.

**Temporal CLI**

```bash
temporal cloud namespace connectivity list -n "my-namespace.abc123"
```

**tcld**

```bash
tcld connectivity-rule list -n "my-namespace.abc123"
```

## Update DNS or clients to use private connectivity

We strongly recommend using private DNS instead of updating client server and TLS settings:

- [How to set up private DNS in AWS](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink)
- [How to set up private DNS in GCP](/cloud/connectivity/gcp-connectivity#configuring-private-dns-for-gcp-private-service-connect)

Choose the private DNS records based on the number of regions your Namespace uses and the endpoint your Clients and Workers use.
Both mTLS and API key authentication support both Namespace Endpoints and Regional Endpoints; the authentication method does not determine which endpoint or TLS server name to use.

| Regions | Endpoint clients use | Private DNS records to configure |
| ------- | -------------------- | -------------------------------- |
| Single region | Namespace Endpoint | An A record for each Namespace Endpoint, such as `my-namespace.my-account.tmprl.cloud`, or a wildcard record for `*.my-account.tmprl.cloud`. |
| Single region | Regional Endpoint | A single record for the Regional Endpoint in the Namespace's region. |
| Multiple regions with High Availability using AWS PrivateLink | Namespace Endpoint (recommended) | A record for each regional FQDN, such as `aws-us-east-1.region.tmprl.cloud`, that the Namespace Endpoint can CNAME to. Keep the Temporal-managed Namespace CNAME so it follows the active region on failover. |
| Multiple regions with High Availability using GCP PSC | Namespace Endpoint | A record for the Namespace Endpoint pointing to a replica region's PSC endpoint IP address. Manually update the override to the active region's PSC endpoint IP address after failover; automatic DNS failover is not supported. |
| Multiple regions with High Availability | Regional Endpoint (not recommended) | A single record for the Regional Endpoint in the client's active region. To withstand a regional outage, run a Worker pool in each region and configure each pool to use its region's Regional Endpoint. |

Configure these records to resolve to the corresponding region's PrivateLink or Private Service Connect endpoint.
For the multi-region setup, follow [Connectivity for High Availability](/cloud/high-availability/ha-connectivity).
The regional DNS override pattern for automatic failover applies to AWS PrivateLink only; see [GCP PSC failover options](/cloud/high-availability/ha-connectivity#single-cloud-ha-on-gcp-private-service-connect) for GCP-only and multi-cloud Namespaces using PSC.

If you are unable to configure private DNS, you must update two settings in your Temporal clients:

1. Set the endpoint server address to the PrivateLink or Private Service Connect endpoint (for example, `vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233` or `<GCP PSC IP address>:7233`).
2. Set TLS configuration to override the TLS server name with the Temporal Cloud hostname, for example, the Namespace Endpoint `my-namespace.my-account.tmprl.cloud`.

Using the wrong TLS server name over PrivateLink or Private Service Connect can cause the TLS handshake to fail even when the port is reachable, regardless of the authentication method.

Updating these settings depends on the client you're using.

#### Temporal CLI
```bash
TEMPORAL_ADDRESS=vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233
TEMPORAL_NAMESPACE=my-namespace.my-account
TEMPORAL_TLS_CERT=<path/to/cert.pem>
TEMPORAL_TLS_KEY=<path/to/cert.key>
TEMPORAL_TLS_SERVER_NAME=my-namespace.my-account.tmprl.cloud

temporal workflow count -n $TEMPORAL_NAMESPACE
```

#### `grpcurl`
```bash
grpcurl \
    -servername my-namespace.my-account.tmprl.cloud \
    -cert path/to/cert.pem \
    -key path/to/cert.key \
    vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 \
    temporal.api.workflowservice.v1.WorkflowService/GetSystemInfo
```

#### Temporal SDKs
**Go**

```go
c, err := client.Dial(client.Options{
	HostPort:  "vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233",
	Namespace: "namespace-name.accId",
	ConnectionOptions: client.ConnectionOptions{
		TLS: &tls.Config{
			Certificates: []tls.Certificate{cert},
			ServerName:   "my-namespace.my-account.tmprl.cloud",
		},
	},
})
```

**Java**

```java
WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                .setSslContext(sslContext)
                .setTarget("vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233")
                .setChannelInitializer(
                    c -> c.overrideAuthority("my-namespace.my-account.tmprl.cloud"))
                .build());
```

**TypeScript**

```ts
const connection = await NativeConnection.connect({
  address: "vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233",
  tls: {
    serverNameOverride: "my-namespace.my-account.tmprl.cloud",
    //serverRootCACertificate,
    // See docs for other TLS options
    clientCertPair: {
      crt: fs.readFileSync(clientCertPath),
      key: fs.readFileSync(clientKeyPath),
    },
  },
});
```

**Python**

```python
client_config["tls"] = TLSConfig(
    client_cert=bytes(crt, "utf-8"),
    client_private_key=bytes(key, "utf-8"),
    domain="my-namespace.my-account.tmprl.cloud",
)

client = await Client.connect("vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233")
```

**.NET**

```dotnet
// Create client 
var client = await TemporalClient.ConnectAsync(
  new(ctx.ParseResult.GetValueForOption(targetHostOption)!)
  {
    Namespace = ctx.ParseResult.GetValueForOption (namespaceOption)!,
    // Set TLS options with client certs. Note, more options could 
    // be added here for server CA (i.e. "ServerRootCACert") or SNI 
    // override (i.e. "Domain") for self-hosted environments with 
    // self-signed certificates. 
    Tls = new() 
    {
      ClientCert = 
        await File.ReadAllBytesAsync(ctx.ParseResult.GetValueForOption(clientCertOption) !.FullName), 
      ClientPrivateKey = 
        await File.ReadAllBytesAsync(ctx.ParseResult.GetValueFor0ption(clientKey0ption)!.FullName), Domain = "my-namespace.my-account.tmprl.cloud",
  },
});

// dotnet run --target-host "vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233"
```

To check whether your client has network connectivity to the private endpoint in question, run:
```bash
nc -zv vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com 7233
```

## Control plane connectivity

Using the Temporal Cloud [web UI](/web-ui), [Terraform provider](/cloud/terraform-provider), [`tcld` CLI](/cloud/tcld), or [Cloud Ops APIs](/ops) requires network access to the Temporal Cloud Control Plane. 

### Control plane hostnames

Different hostnames are used for different parts of the service.

- `saas-api.tmprl.cloud` (required for Terraform, tcld, and Cloud Ops APIs) 
- `web.onboarding.tmprl.cloud` (required for Web UI)
- `web.saas-api.tmprl.cloud` (required for Web UI)

### AWS PrivateLink connectivity to Temporal Cloud control plane

Temporal Cloud supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) connections to the Control Plane, which allows access from applications running in VPCs that cannot egress to the public internet. Temporal Cloud does **not** support restricting an account so that private connectivity is the sole connectivity method to the Control Plane; the Control Plane is always accessible via public internet.

Control Plane access is always securely authenticated via [API keys](/cloud/api-keys#overview) or JWT tokens, regardless of how you choose to connect.

To set up a PrivateLink connection to the Temporal Cloud Control Plane, follow [these instructions](/cloud/connectivity/aws-connectivity), but use the Control Plane endpoint information below:

| Hostname               | Region      | Control Plane PrivateLink Service Name                    |
| ---------------------- | ----------- | --------------------------------------------------------- |
| `saas-api.tmprl.cloud` | `us-west-2` | `com.amazonaws.vpce.us-west-2.vpce-svc-0c57a5930b6f6be0e` |

The Control Plane PrivateLink endpoint includes a [private DNS name](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html), which lets your clients use the PrivateLink connection without having to set up private DNS or having to override client configuration. To use the DNS name, make sure your VPC has the `Enable DNS hostnames` and `Enable DNS support` options enabled. If you cannot use the DNS name, you can also manually [set up private DNS](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink) or [override the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity).

> **⚠️ Caution:**
> Finish client setup to programmatically access Temporal Cloud Control Plane over PrivateLink
>
> For Temporal clients to access the Control Plane over AWS PrivateLink, you must either use the provided DNS name, set up private DNS, or update the client configuration, as described above.
>
> Without this step, your clients may connect to the Temporal Cloud Control Plane over the internet, or they may not connect at all.

#### Extend the control plane PrivateLink endpoint to other AWS regions with VPC peering

The Temporal Cloud Control Plane PrivateLink Service is only exposed in `us-west-2`. To reach the Control Plane privately from another AWS region, set up a us-west-2 VPC with a VPC Endpoint to the Control Plane, then [VPC-Peer](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html) that VPC to your other-region VPCs.

To set this up:

1. Create a VPC in `us-west-2`.
2. Create the [VPC Endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html) to the Control Plane in that us-west-2 VPC, using the service name from the table above. Follow the [standard PrivateLink setup steps](/cloud/connectivity/aws-connectivity), but use the Control Plane service name.
3. Peer the us-west-2 VPC to each VPC in another region that needs Control Plane access. [VPC Peering shares VPC Endpoints](https://docs.aws.amazon.com/vpc/latest/peering/peering-configurations-partial-access.html) across peered VPCs, which gives each peered VPC a private path to the Control Plane.

Peering is per-VPC, so you must peer the us-west-2 VPC to every VPC that needs Control Plane access. There is no limit on how many regions you can extend to using this approach.

---

# AWS PrivateLink connectivity

Source: https://docs.temporal.io/cloud/connectivity/aws-connectivity

> Connect to Temporal Cloud using AWS PrivateLink

[AWS PrivateLink](https://aws.amazon.com/privatelink/) allows you to open a path to Temporal without opening a public egress.
It establishes a private connection between your Amazon Virtual Private Cloud (VPC) and Temporal Cloud.
This one-way connection means Temporal cannot establish a connection back to your service.
This is useful if normally you block traffic egress as part of your security protocols.
If you use a private environment that does not allow external connectivity, you will remain isolated.

After creating the PrivateLink endpoint, configure your clients to use it through either [private DNS](#configuring-private-dns-for-aws-privatelink) or [direct VPCE targeting](#direct-vpce). Direct VPCE targeting is simplest for single-region Namespaces, but also works for High Availability Namespaces with [more careful setup](#direct-vpce).

## Requirements

* Your AWS PrivateLink (PL) endpoint must be in the same region as your Temporal Cloud Namespace or one of its [High Availability](/cloud/high-availability) replicas.
  See [cross-region PrivateLink connectivity](#cross-region-privatelink) to access the Namespace from a different region.
* Your Private DNS must be configured to direct Worker / Client traffic to your VPC Endpoint, as described below.
* If the Worker / Client does not use the Namespace Endpoint as the connection string in its code, it may need to set the `server_name` config to the Namespace Endpoint string, as described below.

### Cross-region PrivateLink Connectivity 

Temporal Cloud does **not** support [cross-region connectivity for AWS PrivateLink](https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cross-region-connectivity-for-aws-privatelink/) out of the box. However, if you need to reach Temporal Cloud privately from a different region than your Namespace, you can route traffic to your VPC Endpoint in the Namespace's region using [AWS's native cross-region networking features](https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/centralized-access-to-vpc-private-endpoints.html#cross-region-endpoint-access). 

When using High Availability on Temporal Cloud, it's best practice to have two VPC Endpoints, one in each of the Namespace's regions, to ensure at least one VPC Endpoint is accessible during a regional outage.

## Creating an AWS PrivateLink connection

Set up PrivateLink connectivity with Temporal Cloud with these steps:

1. Open the AWS console with the region you want to use to establish the PrivateLink.
2. Search for "VPC" in _Services_ and select the option.

   ![AWS console showing services, features, resources](https://docs.temporal.io/img/cloud/privatelink/aws-console.png)
3. Select _Virtual private cloud_ > _Endpoints_ from the left menu bar.
4. Click the _Create endpoint_ button to the right of the _Actions_ pulldown menu.
5. Under _Type_ category, select _Endpoint services that use NLBs and GWLBs_.
   This option lets you find services shared with you by service name.
6. Under _Service settings_, fill in the _Service name_ with the PrivateLink Service Name for the region you're trying to connect from. See [Available AWS regions, PrivateLink endpoints, and DNS record overrides](#available-aws-regions-privatelink-endpoints-and-dns-record-overrides) for the list of available services.

> **💡 Tip:**
>
> PrivateLink endpoint services are regional.
> Individual Namespaces do not use separate services.
>

7. Confirm your service by clicking on the _Verify service_ button. AWS should respond "Service name verified."

   ![The service name field is filled out and the Verify service button is shown](https://docs.temporal.io/img/cloud/privatelink/service-settings.png)
8. Select the VPC and subnets to peer with the Temporal Cloud service endpoint.
9. Select the security group that will control traffic sources for this VPC endpoint.
   The security group must accept TCP ingress traffic to port 7233 for gRPC communication with Temporal Cloud.
10. Click the _Create endpoint_ button at the bottom of the screen.
    If successful, AWS reports "Successfully created VPC endpoint." and lists the new endpoint.
    The new endpoint appears in the Endpoints list, along with its ID.

    ![The created endpoint appears in the Endpoints list](https://docs.temporal.io/img/cloud/privatelink/endpoint-created.png)
11. Click on the VPC endpoint ID in the Endpoints list to check its status.
    Wait for the status to be “Available”.
    This can take up to 10 minutes.
12. Once the status is "Available", the AWS PrivateLink is ready for use.

    ![Highlighted DNS names section shows your hostname](https://docs.temporal.io/img/cloud/privatelink/details.png)

The next step is to [configure private DNS](#configuring-private-dns-for-aws-privatelink) so your clients can use the PrivateLink connection. For single-region Namespaces that don't need per-Namespace DNS records, you can use [direct VPCE targeting](#direct-vpce) instead.

## Configuring Private DNS for AWS PrivateLink

### Why configure private DNS?

When you connect to Temporal Cloud through AWS PrivateLink you normally must:

1. **Point your SDKs/Workers at the PrivateLink DNS name** for the VPC Endpoint (for example, `vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com`), **and**
2. **Override the Server Name Indicator (SNI)** so that the TLS handshake still presents the public Temporal Cloud hostname (for example, `my-namespace.my-account.tmprl.cloud`).

By creating a Route 53 **private hosted zone (PHZ)** that maps the public Temporal Cloud hostname (or region hostname) to your VPC Endpoint, you can:

- Keep using the standard Temporal Cloud hostnames in code and configuration.
- Eliminate the need to set a custom SNI override.
- Make future Endpoint rotations transparent—only the PHZ record changes.

This approach is **optional**; Temporal Cloud works without it. It simply streamlines configuration and operations. If you cannot use private DNS, refer to [our guide for updating the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity).

### Prerequisites

| Requirement                                           | Notes                                                                                                                              |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| AWS VPC with DNS resolution and DNS hostnames enabled | _VPC console → Edit DNS settings → enable both checkboxes._                                                                        |
| Interface VPC Endpoint for Temporal Cloud             | Subnets must be associated with the VPC and Security Group must allow TCP ingress traffic to port 7233 from the appropriate hosts. |
| Route 53 available in your AWS account                | You need permission to create Private Hosted Zones and records.                                                                    |
| Namespace details                                     | Needed to choose the correct override domain pattern below.                                                                        |

### Choose the override domain and endpoint

| Endpoint type      | PHZ domain format                  | Example                              | Use when |
| ------------------ | ---------------------------------- | ------------------------------------ | -------- |
| Namespace endpoint | `<namespace-id>.tmprl.cloud`       | `payments.abcde.tmprl.cloud`         | **Single-region Namespaces.** Simplest pattern — one record per Namespace. For [High Availability](/cloud/high-availability/ha-connectivity) Namespaces, overriding the Namespace Endpoint is nuanced — see [Connectivity for High Availability](/cloud/high-availability/ha-connectivity). |
| Regional endpoint  | `<region>.<cloud>.api.temporal.io` | `ap-northeast-2.aws.api.temporal.io` | You want to pin a client to a specific Temporal Cloud region. |

> **⚠️ Caution:**
> HA Namespaces need a more nuanced PHZ setup
>
> For Namespaces with [High Availability](/cloud/high-availability/ha-connectivity), the PHZ pattern to use depends on how you want Workers to reach the active region. Overriding the Namespace Endpoint directly is read out of the PHZ before public DNS, so the regional CNAME that Temporal Cloud rewrites on failover isn't followed — which is usually not what you want, but can be the right choice in some topologies (for example, multi-cloud HA with one region per cloud, where Workers on each cloud should always reach their local region). Because the trade-offs depend on your setup, see [Connectivity for High Availability](/cloud/high-availability/ha-connectivity) before choosing a pattern.
>

The step-by-step below walks through the **Namespace endpoint** pattern, which is the simpler single-region case. For HA, follow the [HA Connectivity guide](/cloud/high-availability/ha-connectivity) instead, which uses the same Route 53 mechanics but on the regional records.

### Step-by-step instructions

> **⚠️ Warning:**
> Order matters
>
> A Route 53 private hosted zone with no records causes DNS resolution to fail (NXDOMAIN) inside any associated VPC. If you create an empty PHZ for `<account>.tmprl.cloud` and associate it with a VPC where Workers are running, **all Worker traffic to Temporal Cloud in that VPC stops** until you add the CNAME record. Follow the steps below in order to avoid this.
>

#### 1. Collect your PrivateLink endpoint DNS name

```bash
aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids $VPC_ENDPOINT_ID \
  --query "VpcEndpoints[0].DnsEntries[0].DnsName" \
  --output text

# Example output:
# vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com
```

Save the **`vpce-*.amazonaws.com`** value — you will target it in the CNAME record.

#### 2. Create a Route 53 private hosted zone (do not yet attach Worker VPCs)

a. Open _Route 53 → Hosted zones → Create hosted zone_.
b. Enter the domain chosen from the table above, for example, `payments.abcde.tmprl.cloud`.
c. Type: _Private hosted zone for Temporal Cloud_.
d. Leave VPC associations empty for now (you'll add them in step 4).
e. Create the hosted zone.

#### 3. Add a CNAME record

Inside the new PHZ:

| Field           | Value                                                                                 |
| --------------- | ------------------------------------------------------------------------------------- |
| **Record name** | the Namespace Endpoint (for example, `payments.abcde.tmprl.cloud`).                          |
| **Record type** | `CNAME`                                                                               |
| **Value**       | Your VPC Endpoint DNS name (`vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com`) |
| **TTL**         | 60s is typical; 15s for Namespaces with High Availability (to minimize recovery time after failover). |

#### 4. Associate the PHZ with your Worker VPCs and verify

Now that the record exists, associate the PHZ with every VPC that contains Temporal Workers or SDK clients (Route 53 → your zone → _Edit settings_ → _Add VPC_).

> **💡 Tip:**
> Test with a non-production VPC first
>
> We strongly recommend that you test with a non-production VPC first. Attach the PHZ to a non-production VPC, validate end-to-end resolution and connectivity from a host in that VPC, and only then attach production Worker VPCs. This catches misconfigured records before they affect production traffic.
>

Verify DNS resolution from inside one of the associated VPCs:

```bash
dig payments.abcde.tmprl.cloud
```

If the record resolves to the VPC Endpoint, you are ready to use Temporal Cloud without SNI overrides.

### Updating your workers/clients

With private DNS in place, configure your SDKs exactly as the public-internet examples show (filling in your own namespace):

```go
clientOptions := client.Options{
    HostPort: "payments.abcde.tmprl.cloud:7233",
    Namespace: "payments",
    // No TLS SNI override needed
}
```

The DNS resolver inside your VPC returns the private endpoint, while TLS still validates the original hostname—simplifying both code and certificate management.

## Configure private DNS for Namespaces with High Availability

For Namespaces with [High Availability features](/cloud/high-availability), you need to override DNS for `region.tmprl.cloud` so each region resolves to the local VPC Endpoint, and you need to ensure Workers can reach whichever region is active. Failover is transparent to clients only when this is set up correctly.

The complete guidance — including single-cloud (AWS-only) HA, multi-cloud HA (AWS PrivateLink + GCP Private Service Connect), and a recommended failover-testing plan — lives on a single page: [Connectivity for High Availability](/cloud/high-availability/ha-connectivity).

## Direct VPCE targeting without per-Namespace DNS 

You can avoid creating DNS records for each Namespace by pointing Workers directly at the VPC Endpoint and overriding the TLS Server Name Indicator (SNI):

1. Create the PrivateLink VPC Endpoint (one per region — all Namespaces in that region share it).
2. Configure each Worker with:
   - **Endpoint**: the DNS name of the VPC Endpoint in the region where the Worker runs (for example, `vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233`)
   - **Server name** (SNI override): the Namespace Endpoint value (for example, `my-namespace.my-account.tmprl.cloud`)

With this approach, new Namespaces do not require new DNS records. Workers set their **Endpoint** to the VPC Endpoint DNS name and their **Server name** to the Namespace Endpoint, so the client SDK accepts the TLS handshake from Temporal Cloud.

**Single-region Namespace**

```mermaid
---
title: Direct VPCE targeting (single region)
---
%%{init: {'themeVariables':{'fontFamily':'Inter, ui-sans-serif, system-ui, sans-serif'},'flowchart':{'nodeSpacing':18,'rankSpacing':45,'curve':'basis','subGraphTitleMargin':{'top':6,'bottom':12}}}}%%
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef endpoint stroke-width:1px;
    classDef region stroke-width:1.5px;
    subgraph SREG["<b>Your VPC</b>"]
      SW["<b>Worker(s)</b><br/>Server name:<br/>my-namespace.my-account.tmprl.cloud"]:::worker
      SVPCE["<b>VPC Endpoint</b><br/>vpce-…"]:::endpoint
      SW --> SVPCE
    end
    SNS["<b>Namespace</b><br/>(Temporal Cloud)"]:::ns
    SVPCE -->|PrivateLink| SNS
    class SREG region
```

> **📝 Note:**
> Using direct VPCE targeting with High Availability Namespaces
>
> Direct VPCE targeting also works for Namespaces with [High Availability features](/cloud/high-availability), but it takes more careful setup. Because each VPC Endpoint is pinned to its region and does not follow Temporal Cloud's active-region CNAME on failover, you cannot rely on DNS to move Workers between regions. This is the same trade-off as targeting a [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) instead of the Namespace Endpoint.
>

**Namespace with High Availability**

To use the Direct VPCE approach with a Namespace that has High Availability:

- Point each region's Workers at the VPC Endpoint **local to that region** — the Endpoint value differs per region.
- Set the **Server name** (SNI override) to the Namespace Endpoint value in every region — it is the same everywhere.
- **(Recommended)** To stay available during a failover, run Workers in every region the Namespace can be active in.

Workers connected to the passive region's VPC Endpoint stay productive: Temporal Cloud forwards their tasks to the active region (you can [configure this forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior)). On failover, no DNS change is needed, because Workers are already connected in both regions and the surviving region takes over.

```mermaid
---
title: Direct VPCE targeting (High Availability)
---
%%{init: {'themeVariables':{'fontFamily':'Inter, ui-sans-serif, system-ui, sans-serif'},'flowchart':{'nodeSpacing':55,'rankSpacing':70,'curve':'basis','subGraphTitleMargin':{'top':6,'bottom':12}}}}%%
flowchart TD
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef endpoint stroke-width:1px;
    classDef region stroke-width:1.5px;
    subgraph HSEC["<b>us-west-2</b> (passive)"]
      HWB["<b>Worker(s)</b><br/>Server name<br/><span style='font-family:mono'>namespace.<br/>acct.<br/>tmprl.cloud</span>"]:::worker
      HVPB["<b>VPC<br/>Endpoint</b><br/>vpce-…<br/>us-west-2"]:::endpoint
      HNSB["<b>Replica</b><br/>(Passive,<br/>forwards<br/>to active)"]:::ns
      HWB --> HVPB
      HVPB --> HNSB
    end
    subgraph HPRIM["<b>us-east-1</b> (active)"]
      HWA["<b>Worker(s)</b><br/>Server name<br/><span style='font-family:mono'>namespace.<br/>acct.<br/>tmprl.cloud</span>"]:::worker
      HVPA["<b>VPC<br/>Endpoint</b><br/>vpce-…<br/>us-east-1"]:::endpoint
      HNSA["<b>Namespace</b><br/>(Active)"]:::ns
      HWA --> HVPA
      HVPA --> HNSA
    end
    
    
    class HPRIM,HSEC region
```

## Adding PrivateLink from additional AWS accounts

A common pattern is to have separate AWS accounts for different lines of business, environments (staging, production), or compliance scopes (PCI vs non-PCI), each with its own VPC and Workers connecting to the same Temporal Cloud account.

You can create as many AWS PrivateLink VPC endpoints as you need to the same Temporal Cloud regional service — there is nothing to register, approve, or open a ticket for on the Temporal side.

For each additional AWS account or VPC:

1. In that account, create the AWS PrivateLink VPC endpoint targeting the regional service name from the [regions table](#available-aws-regions-privatelink-endpoints-and-dns-record-overrides) — same as in the [creation steps](#creating-an-aws-privatelink-connection) above.
2. Configure DNS in that VPC. You have two options:
   - Create a Route 53 Private Hosted Zone in that account scoped to the appropriate VPC(s), following the [private DNS steps](#configuring-private-dns-for-aws-privatelink) above. Each VPC's PHZ should point at the VPC Endpoint local to that VPC.
   - Or, use [direct VPCE targeting](#direct-vpce). For High Availability Namespaces, follow the [additional setup](#direct-vpce) so each region's Workers target their local VPC Endpoint.
3. **Optional:** if you want to enforce private-only access for a Namespace, add a Connectivity Rule for each VPC endpoint and attach all of them (plus a public rule, if needed) to the Namespace. See [Connectivity Rules](/cloud/connectivity#connectivity-rules).

There is no upper limit on the number of VPC endpoints you can connect from your side to a regional PrivateLink service. The default per-account limit on private Connectivity Rules is 50 — [contact support](/evaluate/cloud/support#support-ticket) if you need to raise it.

## Available AWS regions, PrivateLink endpoints, and DNS record overrides

The following table lists the available Temporal regions, PrivateLink endpoints, and regional endpoints used for DNS record overrides:

### Asia Pacific - Tokyo (`ap-northeast-1`)

- **Cloud API Code**: `aws-ap-northeast-1`
- **Regional Endpoint**: `ap-northeast-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Seoul (`ap-northeast-2`)

- **Cloud API Code**: `aws-ap-northeast-2`
- **Regional Endpoint**: `ap-northeast-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Mumbai (`ap-south-1`)

- **Cloud API Code**: `aws-ap-south-1`
- **Regional Endpoint**: `ap-south-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Hyderabad (`ap-south-2`)

- **Cloud API Code**: `aws-ap-south-2`
- **Regional Endpoint**: `ap-south-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Singapore (`ap-southeast-1`)

- **Cloud API Code**: `aws-ap-southeast-1`
- **Regional Endpoint**: `ap-southeast-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Sydney (`ap-southeast-2`)

- **Cloud API Code**: `aws-ap-southeast-2`
- **Regional Endpoint**: `ap-southeast-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08`
- **Same Region Replication**: Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Europe - Frankfurt (`eu-central-1`)

- **Cloud API Code**: `aws-eu-central-1`
- **Regional Endpoint**: `eu-central-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-eu-west-1`
  - `aws-eu-west-2`
- **Multi-Cloud Replication**:
  - `gcp-europe-west3`

### Europe - Ireland (`eu-west-1`)

- **Cloud API Code**: `aws-eu-west-1`
- **Regional Endpoint**: `eu-west-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-eu-central-1`
  - `aws-eu-west-2`
- **Multi-Cloud Replication**:
  - `gcp-europe-west3`

### Europe - London (`eu-west-2`)

- **Cloud API Code**: `aws-eu-west-2`
- **Regional Endpoint**: `eu-west-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-eu-central-1`
  - `aws-eu-west-1`
- **Multi-Cloud Replication**:
  - `gcp-europe-west3`

### North America - Central Canada (`ca-central-1`)

- **Cloud API Code**: `aws-ca-central-1`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d`
- **Regional Endpoint**: `ca-central-1.aws.api.temporal.io:7233`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### North America - Northern Virginia (`us-east-1`)

- **Cloud API Code**: `aws-us-east-1`
- **Regional Endpoint**: `us-east-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f`
- **Same Region Replication**:  Available
- **Multi-Region Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-2`
  - `aws-us-west-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### North America - Ohio (`us-east-2`)

- **Cloud API Code**: `aws-us-east-2`
- **Regional Endpoint**: `us-east-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-west-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### North America - Oregon (`us-west-2`)

- **Cloud API Code**: `aws-us-west-2`
- **Regional Endpoint**: `us-west-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94`
- **Same Region Replication**:  Available
- **Multi-Region Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### South America - São Paulo (`sa-east-1`)

- **Cloud API Code**: `aws-sa-east-1`
- **Regional Endpoint**: `sa-east-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - None

---

# Google Private Service Connect connectivity

Source: https://docs.temporal.io/cloud/connectivity/gcp-connectivity

> Connect to Temporal Cloud using Google Private Services Connect

[Google Cloud Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) allows you to open a path to Temporal without opening a public egress.
It establishes a private connection between your Google Virtual Private Cloud (VPC) and Temporal Cloud.
This one-way connection means Temporal cannot establish a connection back to your service.
This is useful if normally you block traffic egress as part of your security protocols.
If you use a private environment that does not allow external connectivity, you will remain isolated.

<a id="high-availability-and-private-service-connect"></a>

> **⚠️ Warning:**
> Namespaces with High Availability features and GCP Private Service Connect
>
> Automatic failover via Temporal Cloud DNS is not currently supported with GCP Private Service Connect.
> GCP Cloud DNS cannot combine the public Namespace CNAME with private regional DNS overrides in the way the AWS PrivateLink HA pattern requires.
> When a failover occurs, manually update your private DNS override or Worker and Client endpoint configuration to point to the active region's Private Service Connect endpoint.
> See [Connectivity for High Availability](/cloud/high-availability/ha-connectivity#single-cloud-ha-on-gcp-private-service-connect) for GCP-only and multi-cloud connectivity options.
>

## Requirements

* Your GCP Private Service Connect endpoint must be in the same region as your Temporal Cloud namespace. If using [replication for High Availability](/cloud/high-availability), the PSC connection must be in the same region as one of the replicas.
* Your Private DNS must be configured to direct Worker / Client traffic to your PSC endpoint, as described below.
* If the Worker / Client is not using the Namespace Endpoint, it may need to set the `server_name` config to the Namespace Endpoint string, as described below.

## Creating a Private Service Connect connection

Set up Private Service Connect with Temporal Cloud with these steps:

1. Open the Google Cloud console
2. Navigate to **Network Services**, then **Private Service Connect**. If you haven't used **Network Services** recently, you might have to find it by clicking on **View All Products** at the bottom of the left sidebar.

   ![GCP console showing Network Services, and the View All Products button](https://docs.temporal.io/img/cloud/gcp/gcp-console.png)

3. Go to the **Endpoints** section. Click on **Connect endpoint**.

   ![GCP console showing the endpoints, and the Connect endpoint button](https://docs.temporal.io/img/cloud/gcp/connect-endpoint-button.png)

4. Under **Target**, select **Published service**, this will change the contents of the form to allow you to fill the rest as described below

   ![GCP console showing the endpoints, and the Connect endpoint button](https://docs.temporal.io/img/cloud/gcp/connect-endpoint.png)

- For **Target service**, fill in the **Service name** with the Private Service Connect Service Name for the region you're trying to connect to. See [Available GCP regions, PSC endpoints, and DNS record overrides](#available-gcp-regions-psc-endpoints-and-dns-record-overrides) for the list of available services.

> **💡 Tip:**
>
> GCP Private Service Connect services are regional.
> Individual Namespaces do not use separate services.
>

- For **Endpoint name**, enter a unique identifier to use for this endpoint. It could be for instance `temporal-api` or `temporal-api-<namespace>` if you want a different endpoint per namespace.
- For **Network** and **Subnetwork**, choose the network and subnetwork where you want to publish your endpoint.
- For **IP address**, click the dropdown and select **Create IP address** to create an internal IP from your subnet dedicated to the endpoint. Select this IP.
- Check **Enable global access** if you intend to connect the endpoint to virtual machines outside of the selected region. We recommend regional connectivity instead of global access, as it can be better in terms of latency for your workers. _**Note:** this requires the network routing mode to be set to **GLOBAL**._

5. Click the **Add endpoint** button at the bottom of the screen. The endpoint will appear with status **Pending**. This is expected — the next step is what flips it to **Accepted**.

6. [Create a Temporal Cloud Connectivity Rule](/cloud/connectivity#creating-a-connectivity-rule) using the Connection ID of the newly created endpoint and the corresponding GCP project. Use the **Connection ID** from the endpoint's detail page in the Google Cloud console (a numeric string such as `1234567890123456789`).

7. Once the status changes from "Pending" to "Accepted", the GCP Private Service Connect endpoint is ready for use.

> **⚠️ Warning:**
> PSC stays "Pending" until you create a Connectivity Rule
>
> For GCP Private Service Connect, the Connectivity Rule is what tells Temporal Cloud to accept your PSC connection. Until you [create a Connectivity Rule](/cloud/connectivity#creating-a-connectivity-rule) for the connection, the endpoint will remain in **Pending**. There is no separate producer-side approval step — creating the Connectivity Rule is the approval.
>
> If your endpoint is stuck Pending, the most common causes are:
>
> - No Connectivity Rule exists for the connection ID. (Most common.)
> - The Connectivity Rule was created with the wrong `connection-id`, `region`, or `gcp-project-id`.
> - The endpoint is in a region that is not a [supported Temporal Cloud region](/evaluate/cloud/regions).
>

- Take note of the **IP address** assigned to your endpoint — you will use it to connect to Temporal Cloud.

> **⚠️ Caution:**
> You still need to set up private DNS or override client configuration for your clients to actually use the new Private Service Connect connection to connect to Temporal Cloud.
>
> See [configuring private DNS for GCP Private Service Connect](#configuring-private-dns-for-gcp-private-service-connect)

## Configuring Private DNS for GCP Private Service Connect

### Why configure private DNS?

When you connect to Temporal Cloud through GCP Private Service Connect you normally must:

1. **Point your SDKs/Workers at the Private Service Connect endpoint IP address** _and_
2. **Override the Server Name Indicator (SNI)** so that the TLS handshake still presents the public Temporal Cloud hostname (for example, `my-namespace.my-account.tmprl.cloud`).

By creating a **private Cloud DNS zone (PZ)** that maps the public Temporal Cloud hostname (or the region hostname) directly to the PSC endpoint IP address, you can:

- Keep using the standard Temporal Cloud hostnames in code and configuration.
- Eliminate the need to set a custom SNI override.
- Make future endpoint rotations transparent—only the DNS record changes.

This approach is **optional**; Temporal Cloud works without it. It simply streamlines configuration and operations. If you cannot use private DNS, refer to [our guide for updating the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity).

### Prerequisites

| Requirement                                           | Notes                                                                             |
| ----------------------------------------------------- | --------------------------------------------------------------------------------- |
| Google Cloud VPC Network with DNS enabled             | PSC endpoints and the DNS zone must live in (or be attached to) the same network. |
| Private Service Connect endpoint for Temporal Cloud   | Create an endpoint and reserve an internal IP in the namespace region             |
| Cloud DNS API enabled and roles/dns.admin permissions | Needed to create private zones and records.                                       |
| Namespace details                                     | Determines which hostname pattern you override (table below).                     |

### Choose the override domain and endpoint

Choose the private DNS records based on the number of regions your Namespace uses and the endpoint your Clients and Workers use.
Both mTLS and API key authentication support both Namespace Endpoints and Regional Endpoints.

| Regions | Endpoint clients use | Private DNS records to configure |
| ------- | -------------------- | -------------------------------- |
| Single region | Namespace Endpoint | An A record for each Namespace Endpoint, such as `payments.abcde.tmprl.cloud`, or a wildcard record for `*.abcde.tmprl.cloud`. |
| Single region | Regional Endpoint | A single A record for the Regional Endpoint in the Namespace's region. |
| Multiple regions with High Availability | Namespace Endpoint | An A record for the Namespace Endpoint pointing to a replica region's PSC endpoint IP address. Manually update the override to the active region's PSC endpoint IP address after failover; automatic DNS failover is not supported. |
| Multiple regions with High Availability | Regional Endpoint (not recommended) | A single A record for the Regional Endpoint in the client's active region. To withstand a regional outage, run a Worker pool in each region and configure each pool to use its region's Regional Endpoint. |

Point each A record at the corresponding region's PSC endpoint IP address.
For High Availability Namespaces, see [GCP PSC failover options](/cloud/high-availability/ha-connectivity#single-cloud-ha-on-gcp-private-service-connect).
Regional DNS overrides do not enable automatic failover with GCP PSC.

The steps below use a single-region Namespace Endpoint as the example.

### Step-by-step instructions

#### 1. Collect your PSC endpoint IP address

```shell
# List the forwarding rule you created for the endpoint
gcloud compute forwarding-rules list \
  --filter="NAME:<endpoint-name>" \
  --format="value(IP_ADDRESS)"
# Example output: 10.1.2.3
```

Save the internal IP -- you will point the A record at it.

#### 2. Create a Cloud DNS private zone

1. Open _Network Services → Cloud DNS → Create zone_.
2. Select zone type **Private**.
3. Enter a **Zone name** (for example, `temporal-cloud`).
4. Enter a **DNS name** that covers the records from the table above (for example, `abcde.tmprl.cloud` for Namespace or wildcard records, or `gcp-us-central1.region.tmprl.cloud` for a regional record).
5. Select **Add networks** and choose the Project and Network that contains your PSC endpoint.
6. Click **Create**.

#### 3. Add an A record

Inside the new zone, add a _standard A record_:

| Field                | Value                                                          |
| -------------------- | -------------------------------------------------------------- |
| DNS name             | The hostname or wildcard from the table above (for example, `payments.abcde.tmprl.cloud` or `*.abcde.tmprl.cloud`). |
| Resource record type | A                                                              |
| TTL                  | 60s is typical, but you can adjust as needed.                  |
| IPv4 Address         | the internal IP address of your PSC endpoint (for example, `10.1.2.3`) |

#### 4. Verify DNS resolution from inside the Network

```shell
dig payments.abcde.tmprl.cloud
```

If the hostname resolves to the PSC endpoint IP address from a VM in the bound network, the override is working.

### Updating your workers/clients

With private DNS in place, configure your SDKs exactly as the public-internet examples show (filling in your own namespace):

```go
clientOptions := client.Options{
    HostPort: "payments.abcde.tmprl.cloud:7233",
    Namespace: "payments",
    // No TLS SNI override needed
}
```

The DNS resolver inside your network returns the private endpoint IP address, while TLS still validates the original hostname—simplifying both code and certificate management.

## Available GCP regions, PSC endpoints, and DNS record overrides

The following table lists the available Temporal regions, PrivateLink endpoints, and regional endpoints used for DNS record overrides:

> **⚠️ Caution:**
> GCP capacity restriction
>
> Before creating a GCP Namespace or adding a GCP replica, check [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction) for regional availability and exception requirements.
>

### North America - Iowa (`us-central1`)

- **Cloud API Code**: `gcp-us-central1`
- **Regional Endpoint**: `us-central1.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `gcp-us-west1`
  - `gcp-us-east4`
- **Multi-Cloud Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`

### North America - Oregon (`us-west1`)

- **Cloud API Code**: `gcp-us-west1`
- **Regional Endpoint**: `us-west1.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `gcp-us-central1`
  - `gcp-us-east4`
- **Multi-Cloud Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`

### North America - Northern Virginia (`us-east4`)

- **Cloud API Code**: `gcp-us-east4`
- **Regional Endpoint**: `us-east4.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
- **Multi-Cloud Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`

### Europe - Frankfurt (`europe-west3`)

- **Cloud API Code**: `gcp-europe-west3`
- **Regional Endpoint**: `europe-west3.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - `aws-eu-central-1`
  - `aws-eu-west-1`
  - `aws-eu-west-2`

### Asia Pacific - Mumbai (`asia-south1`)

- **Cloud API Code**: `gcp-asia-south1`
- **Regional Endpoint**: `asia-south1.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`

### Asia Pacific - Jakarta (`asia-southeast2`)

- **Cloud API Code**: `gcp-asia-southeast2`
- **Regional Endpoint**: `gcp-asia-southeast2.region.tmprl.cloud`
- **Private Service Connect Service Attachment URI**: `projects/prod-bsbyrfwqqq885qkcr3s43y524/regions/asia-southeast2/serviceAttachments/pl-c3ayi`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - None

---

# Temporal Cloud IP addresses

Source: https://docs.temporal.io/cloud/connectivity/ip-addresses

> Temporal Cloud assigns dynamic IPs by default; enable Stable IPs for a fixed, published range you can allowlist in firewalls.

By default, Temporal Cloud resources use dynamic IP addresses that may change at any time. These IP addresses may be any IPs within the IP ranges published by the relevant cloud provider.

If you need to limit outbound access from your client network, we recommend using [AWS PrivateLink or GCP Private Services Connect](/cloud/connectivity#private-network-connectivity-for-namespaces). Alternatively, you can enable [Stable IPs](#stable-ip-addresses) for your Namespace.

> **⚠️ Warning:**
> Do not rely on default dynamic public IP addresses
>
> By default, Temporal Cloud IPs are not static and may change without notice. If you need fixed, public IP addresses for your Namespace without using [private connectivity](/cloud/connectivity#private-network-connectivity-for-namespaces), enable [Stable IPs](#stable-ip-addresses) for Namespace Endpoints.
>

## When Namespace Endpoint DNS resolution is predictable

In general, do not take dependencies on what Temporal Cloud endpoints resolve to over DNS. The resolved values are subject to change without notice, and Workers, Temporal Clients, and firewalls that target the specific IPs provided at a specific point in time will eventually fail.

Temporal Cloud guarantees the DNS resolution behavior of the Namespace Endpoint in two cases:

1. **Stable IPs enabled.** The Namespace Endpoint resolves to one of the fixed IP ranges for the Namespace's active region, so it is safe to allowlist these IP ranges in a firewall. See [Stable IP addresses](#stable-ip-addresses).
2. **High Availability features with Private Connectivity.** When High Availability and Private Connectivity Rules are both enabled on a Namespace, the Namespace Endpoint is guaranteed to resolve via CNAME to `<provider>-<region>.region.tmprl.cloud` for the active region. See [Connectivity for High Availability](/cloud/high-availability/ha-connectivity) for recommendations to configure private DNS. Note: Do not attach a public connectivity rule with Stable IPs to the Namespace, as that supersedes this behavior.

For all other connectivity patterns, treat the resolved IP as temporary and use one of the two approaches above, or allowlist the entire cloud provider IP range.

## Stable IP addresses

Stable IPs is an optional Namespace setting that provides a fixed, published set of IP addresses for your Namespace endpoint. When enabled, Workers and Temporal Clients connecting to your Namespace through its Namespace endpoint will resolve to a predictable set of IP addresses that you can allowlist in your firewall.

### When to use Stable IPs

Use Stable IPs when your security requirements mandate IP-based allowlisting for egress traffic, and you cannot use AWS PrivateLink or GCP Private Services Connect.

**Recommended options for secure connectivity** (in order):

1. **AWS PrivateLink or GCP Private Services Connect** - Keep traffic within your cloud provider's private network
2. **Stable IPs** - Use published, stable IP ranges for firewall allowlisting
3. **Cloud provider IP ranges** - Allowlist all AWS or GCP IP ranges (overly permissive)

Stable IPs provide enterprise-grade compliance for organizations that require IP-based security controls but cannot use private network connectivity options.

### How stable IPs work

When Stable IPs are enabled on a Namespace:

- The Namespace endpoint (`<namespace>.<account>.tmprl.cloud`) resolves to IP addresses from a published list of Stable IPs.
  - The list is grouped by cloud region. The Namespace endpoint will resolve to an IP from the group for the Namespace's active region.
- Workers and Temporal Clients connecting via the Namespace endpoint will connect to these stable IP addresses.
- Regional endpoints continue to work but resolve to dynamic IP addresses.
- For replicated Namespaces, the Stable IP corresponds to the Namespace's active region. During failover to a different region, DNS updates to point to a Stable IP in the new active region.

> **📝 Note:**
>
> Stable IPs apply only to Namespace traffic, sometimes called the "data plane." The Temporal Cloud management plane, observability endpoints, and Web UI do not have stable public IP addresses.
>

> **⚠️ Warning:**
> Stable IPs supersedes HA + Private Connectivity DNS
>
> If you attach a public Connectivity Rule with Stable IPs to a Namespace that is also configured for [High Availability with Private Connectivity](/cloud/high-availability/ha-connectivity), the Namespace Endpoint resolves to a public Stable IP instead of to `<provider>-<region>.region.tmprl.cloud`. Stable IPs DNS behavior supersedes the regional DNS behavior that HA + Private Connectivity relies on, so the Namespace Endpoint's DNS resolution will not work in the way the Private Hosted Zone needs. Do not attach a Stable IPs public Connectivity Rule to a Namespace where you want HA + Private Connectivity to keep routing traffic over PrivateLink / PSC.
>

### How to enable Stable IPs

Stable IPs is a setting on a public [Connectivity Rule](/cloud/connectivity#connectivity-rules). You enable it by creating (or updating) a public Connectivity Rule with Stable IPs enabled, then attaching that rule to the Namespaces that should resolve to Stable IPs.

> **📝 Note:**
> Create Stable IPs rules with the Temporal CLI
>
> Use the [Temporal CLI](/cli/cloud) to create a public Connectivity Rule with Stable IPs enabled:
>
> ```command
> temporal cloud connectivity public create --enable-stable-ips
> ```
>
> `tcld` does not support creating this rule. You can also use the [Cloud Ops API](/ops) directly (HTTP or gRPC at
> `saas-api.tmprl.cloud`) or the generated client in the [`temporalio/cloud-api`](https://github.com/temporalio/cloud-api)
> repository.
>
> After the rule exists, attach it to a Namespace with [`temporal cloud namespace connectivity attach`](/cli/command-reference/cloud/namespace#connectivity-attach). There is no Temporal Cloud UI option at this time.
>

To enable Stable IPs:

1. Create a public Connectivity Rule with Stable IPs enabled: `temporal cloud connectivity public create --enable-stable-ips`. Each Project allows 1 public Connectivity Rule and up to 5 private Connectivity Rules. If this Project already has a public rule without Stable IPs, delete and recreate that rule. A public rule in another Project can stay. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules) and [How to enable Stable IPs with curl](#how-to-enable-stable-ips-with-curl).
2. Attach the rule to your Namespace: `temporal cloud namespace connectivity attach --namespace <namespace>.<account> --connectivity-rule-id <rule-id>`.
3. Retrieve the list of Stable IPs from the [public JSON file](/cloud/connectivity/ip-addresses#how-to-view-stable-ip-ranges).
4. Configure your firewall to allowlist the Stable IP ranges for your Namespace's region.
5. Ensure your Workers and Temporal Clients connect using the Namespace endpoint, not regional endpoints.

You can enable Stable IPs on new Namespaces at creation time or on existing Namespaces.

> **📝 Note:**
> DNS propagation
>
> When enabling Stable IPs on an existing Namespace, Temporal updates DNS records to point the Namespace endpoint to Stable IPs. DNS propagation typically takes 5 to 15 minutes for AWS, but can take up to 60 minutes or longer if you have custom TTLs configured.
>

#### How to enable Stable IPs with curl

If you cannot use the Temporal CLI or Terraform, you can access the Cloud Ops API directly. The following procedure creates a public Connectivity Rule with Stable IPs enabled and attaches it to a Namespace.

All requests require an authorization header:

- `Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY`

Base URL: `https://saas-api.tmprl.cloud`

Set up environment variables:

```bash
export TEMPORAL_CLOUD_OPS_API_KEY='<paste-api-key>'
export NS='<namespace>.<account>'   # The Cloud-side Namespace identifier, e.g. "myns.a1b2c3"
H_AUTH="Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY"
```

`NS` is the Cloud-side Namespace identifier, not the SDK gRPC hostname (`myns.a1b2c3.tmprl.cloud`). Strip the trailing `.tmprl.cloud`.

**Step 1. Create a public Connectivity Rule with Stable IPs enabled.**

Each Project allows 1 public Connectivity Rule and up to 5 private Connectivity Rules. If this Project already has a public rule, recreate that rule with Stable IPs enabled. A public rule in another Project can stay. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).

Creating the rule returns both the new rule's ID and an async operation that tracks activation:

```bash
curl -sS -X POST "https://saas-api.tmprl.cloud/cloud/connectivity-rules" \
  -H "$H_AUTH" -H "Content-Type: application/json" \
  -d '{
        "spec": {
          "publicRule": { "enableStableIps": true }
        }
      }' | tee /tmp/cr.json

CR_ID=$(jq -r '.connectivityRuleId' /tmp/cr.json)
OP_ID=$(jq -r '.asyncOperation.id // .asyncOperation.operationId' /tmp/cr.json)
```

Field names follow proto3 JSON camelCase: `enable_stable_ips` becomes `enableStableIps`.

Wait for the create operation to reach a terminal state before continuing:

```bash
curl -sS "https://saas-api.tmprl.cloud/cloud/operations/$OP_ID" \
  -H "$H_AUTH" | jq '.asyncOperation.state'
```

Re-run until `state` is `FULFILLED` (the rule is now `ACTIVE`). Any `*_FAILED` state means you should stop and inspect the operation.

Alternatively, list every Connectivity Rule on the account and read the new rule's state directly:

```bash
curl -sS "https://saas-api.tmprl.cloud/cloud/connectivity-rules" \
  -H "$H_AUTH" | \
  jq '.connectivityRules[] | {id, state, spec}'
```

The rule is ready when its `state` is `RESOURCE_STATE_ACTIVE`. This also shows whether this Project already has a public rule. Each Project allows 1 public Connectivity Rule. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).

**Step 2. Attach the rule to the Namespace.**

`UpdateNamespace` replaces the entire spec and requires the latest `resource_version`, so read the current Namespace first. Any field you leave out of `spec` is cleared, so build the update body from the response instead of writing one by hand.

```bash
curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \
  -H "$H_AUTH" | tee /tmp/ns.json | \
  jq '{resource_version: .namespace.resourceVersion, connectivity_rule_ids: .namespace.spec.connectivityRuleIds}'

RV=$(jq -r '.namespace.resourceVersion' /tmp/ns.json)
```

Then post the whole `NamespaceSpec` back, appending `$CR_ID` to `connectivityRuleIds`. A Namespace can have at most one public Connectivity Rule attached, so if `connectivityRuleIds` already references a public rule without Stable IPs, replace that entry instead of appending to the list.

```bash
jq --arg id "$CR_ID" --arg rv "$RV" --arg ns "$NS" '
  {
    namespace: $ns,
    resourceVersion: $rv,
    spec: (.namespace.spec
           | .connectivityRuleIds = ((.connectivityRuleIds // []) + [$id]))
  }' /tmp/ns.json > /tmp/update.json

curl -sS -X POST "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \
  -H "$H_AUTH" -H "Content-Type: application/json" \
  -d @/tmp/update.json | tee /tmp/update-resp.json
```

Poll the returned `asyncOperation` the same way as in Step 1.

Alternatively, describe the Namespace and confirm the rule is attached:

```bash
curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \
  -H "$H_AUTH" | \
  jq '{state: .namespace.state, connectivity_rule_ids: .namespace.spec.connectivityRuleIds}'
```

The update is complete when `state` is `RESOURCE_STATE_ACTIVE` and `connectivityRuleIds` contains your `$CR_ID`.

#### How to enable Stable IPs with gRPC

If you prefer to call the Cloud Ops API directly over gRPC (for example, with `grpcurl` or a generated client), the same flow uses the [CloudService](https://github.com/temporalio/api-cloud) gRPC API. Proto definitions live in [`temporal/api/cloud/cloudservice/v1/service.proto`](https://github.com/temporalio/api-cloud/blob/main/temporal/api/cloud/cloudservice/v1/service.proto) and [`temporal/api/cloud/connectivityrule/v1/message.proto`](https://github.com/temporalio/api-cloud/blob/main/temporal/api/cloud/connectivityrule/v1/message.proto).

- Endpoint: `saas-api.tmprl.cloud:443` (TLS required)
- Fully qualified service: `temporal.api.cloud.cloudservice.v1.CloudService`
- Required metadata on every call:
  - `authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY`

The metadata key is `authorization`, lowercase, per the gRPC convention. Authentication uses Cloud API keys. `saas-api.tmprl.cloud` does not accept mTLS client certificates.

The examples below use `grpcurl`. They assume you have a local checkout of [`temporalio/api-cloud`](https://github.com/temporalio/api-cloud) at `./api-cloud` and a `proto/` directory containing the standard `google/api/annotations.proto` and associated files.

Set up environment variables:

```bash
export TEMPORAL_CLOUD_OPS_API_KEY='<paste-api-key>'
export NS='<namespace>.<account>'   # The Cloud-side Namespace identifier, e.g. "myns.a1b2c3"
GRPC_HEADERS=(-H "authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY")
PROTO_FLAGS=(-import-path ./api-cloud -import-path ./proto -proto temporal/api/cloud/cloudservice/v1/service.proto)
```

`NS` is the Cloud-side Namespace identifier, not the SDK gRPC hostname (`myns.a1b2c3.tmprl.cloud`). Strip the trailing `.tmprl.cloud`.

**Step 1. Create a public Connectivity Rule with Stable IPs enabled.**

Each Project allows 1 public Connectivity Rule and up to 5 private Connectivity Rules. If this Project already has a public rule, recreate that rule with Stable IPs enabled. A public rule in another Project can stay. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).

Creating the rule returns both the new rule's ID and an async operation that tracks activation:

```bash
grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \
  -d '{
        "spec": {
          "public_rule": { "enable_stable_ips": true }
        }
      }' \
  saas-api.tmprl.cloud:443 \
  temporal.api.cloud.cloudservice.v1.CloudService/CreateConnectivityRule | tee /tmp/cr.json

CR_ID=$(jq -r '.connectivityRuleId' /tmp/cr.json)
OP_ID=$(jq -r '.asyncOperation.id' /tmp/cr.json)
```

Field names in gRPC JSON requests can use either snake_case (proto field names) or camelCase — `grpcurl` accepts both. Responses come back in camelCase.

Wait for the create operation to reach a terminal state before continuing:

```bash
grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \
  -d "{\"async_operation_id\": \"$OP_ID\"}" \
  saas-api.tmprl.cloud:443 \
  temporal.api.cloud.cloudservice.v1.CloudService/GetAsyncOperation | jq '.asyncOperation.state'
```

Re-run until `state` is `FULFILLED` (the rule is now `ACTIVE`). Any `*_FAILED` state means you should stop and inspect the operation.

Alternatively, list every Connectivity Rule on the account and read the new rule's state directly:

```bash
grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \
  -d '{}' \
  saas-api.tmprl.cloud:443 \
  temporal.api.cloud.cloudservice.v1.CloudService/GetConnectivityRules | \
  jq '.connectivityRules[] | {id, state, spec}'
```

The rule is ready when its `state` is `RESOURCE_STATE_ACTIVE`. This also shows whether this Project already has a public rule. Each Project allows 1 public Connectivity Rule. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).

**Step 2. Attach the rule to the Namespace.**

`UpdateNamespace` replaces the entire spec and requires the latest `resource_version`, so read the current Namespace first. Any field you leave out of `spec` is cleared, so build the update body from the `GetNamespace` response instead of writing one by hand.

```bash
grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \
  -d "{\"namespace\": \"$NS\"}" \
  saas-api.tmprl.cloud:443 \
  temporal.api.cloud.cloudservice.v1.CloudService/GetNamespace | tee /tmp/ns.json

RV=$(jq -r '.namespace.resourceVersion' /tmp/ns.json)
```

Then post the whole `NamespaceSpec` back, appending `$CR_ID` to `connectivity_rule_ids`. A Namespace can have at most one public Connectivity Rule attached, so if `connectivity_rule_ids` already references a public rule without Stable IPs, replace that entry instead of appending to the list.

```bash
jq --arg id "$CR_ID" --arg rv "$RV" --arg ns "$NS" '
  {
    namespace: $ns,
    resource_version: $rv,
    spec: (.namespace.spec
           | .connectivity_rule_ids = ((.connectivityRuleIds // []) + [$id]))
  }' /tmp/ns.json > /tmp/update.json

grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \
  -d @ \
  saas-api.tmprl.cloud:443 \
  temporal.api.cloud.cloudservice.v1.CloudService/UpdateNamespace < /tmp/update.json | tee /tmp/update-resp.json
```

Poll the returned `asyncOperation` the same way as in Step 1.

Alternatively, describe the Namespace and confirm the rule is attached:

```bash
grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \
  -d "{\"namespace\": \"$NS\"}" \
  saas-api.tmprl.cloud:443 \
  temporal.api.cloud.cloudservice.v1.CloudService/GetNamespace | \
  jq '{state: .namespace.state, connectivity_rule_ids: .namespace.spec.connectivityRuleIds}'
```

The update is complete when `state` is `RESOURCE_STATE_ACTIVE` and `connectivityRuleIds` contains your `$CR_ID`.

### How to view Stable IP ranges

The full list of Stable IP ranges is published as a public JSON file at:

**[https://docs.temporal.io/json/stable-ip-ranges-prod.json](/json/stable-ip-ranges-prod.json)**

Click the link to inspect the list in your browser, or copy the URL into your firewall tooling, a `curl`/`wget` script, or a Terraform `http` data source:

```
https://docs.temporal.io/json/stable-ip-ranges-prod.json
```

The file is publicly accessible without authentication and returns a JSON response grouped by cloud provider and region:

```json
{
  "all_stable_ip_ranges": [
    "34.195.80.228/32",
    "34.195.80.10/30",
    "44.235.214.171/32"
  ],
  "clouds": {
    "aws": {
      "regions": {
        "us-east-1": {
          "stable_ip_ranges": [
            "34.195.80.228/32",
            "34.195.80.10/30"
          ]
        },
        "us-west-2": {
          "stable_ip_ranges": [
            "44.235.214.171/32"
          ]
        }
      }
    },
    "gcp": {
      "regions": {
        ...
      }
    }
  }
}
```

> **📝 Note:**
> IP format
>
> IP addresses are provided in IPv4 or IPv6 format with CIDR notation. At launch, Temporal uses predominantly `/32` ranges (single IP addresses). Each region typically contains approximately 6 IP addresses.
>

### How to connect using Stable IPs

To connect to a Namespace with Stable IPs enabled:

1. **Use the Namespace endpoint**: Configure Workers and Temporal Clients to use the Namespace endpoint format: `<namespace>.<account>.tmprl.cloud:7233`
2. **Do not use Regional endpoints**: Regional endpoints (`<region>.<cloud>.api.temporal.io`) will route to the Namespace but resolve to dynamic IP addresses.
3. **Do not use PrivateLink**: PrivateLink endpoints do not resolve to Stable IPs. To use Stable IPs, connect over public endpoints.

Stable IPs work with both mTLS certificate-based authentication and API key authentication.

If your Workers run on the same cloud provider as your Namespace (for example, both on AWS), traffic stays on the cloud provider's backbone network and does not traverse the public internet, even when using public Stable IPs. AWS, GCP, and Azure all guarantee that traffic between locations on their networks never leaves their backbone.

### How to migrate to Stable IPs

When you enable Stable IPs on an existing Namespace:

**Workers using Namespace endpoints**:
- Existing connections continue uninterrupted
- DNS updates to point the Namespace endpoint to Stable IPs
- Existing connections remain connected to their current dynamic IP
- New connections automatically use Stable IPs after DNS propagates
- No Worker restart required

**Workers using Regional endpoints**:
- Existing connections continue uninterrupted
- To use Stable IPs, update Worker configuration to use the Namespace endpoint and restart Workers

**Workers using PrivateLink**:
- Existing connections continue uninterrupted through PrivateLink
- To use Stable IPs, update Worker configuration to use the Namespace endpoint and restart Workers
- Note: You cannot use both PrivateLink and Stable IPs simultaneously

**Connectivity Rules**: Stable IPs is enabled when the `enableStableIps` config is `true` on a public [Connectivity Rule](/cloud/connectivity#connectivity-rules). Every Namespace that should resolve to Stable IPs must have that public Connectivity Rule attached. Namespaces without the rule attached continue to resolve to dynamic IPs.

#### How to migrate from AWS PrivateLink to Stable IPs

If your Namespace is currently reached over AWS PrivateLink and protected by a private Connectivity Rule, the safe migration path is to add the public path while the private path is still working, verify, and then remove the private path. This applies whether your Namespace is single-region or replicated.

Two things do not change in this migration:

- **Network path for same-region traffic.** When your Worker and the Namespace are both in AWS in the same region, traffic to a Stable IP stays inside the AWS network instead of routing over the public internet. Moving off PrivateLink does not change that.
- **TLS and authentication.** You are still connecting to the Namespace endpoint (`<namespace>.<account>.tmprl.cloud`), so the certificate and SNI are the same and only the IP it resolves to changes. mTLS and API key authentication continue to work unchanged.

**Prerequisites:**

- Note your existing private (PrivateLink) Connectivity Rule ID — you'll keep it attached during the overlap window.
- Check whether this Project already has a public Connectivity Rule. Each Project allows 1 public Connectivity Rule and up to 5 private Connectivity Rules. If this Project already has a public rule without Stable IPs, delete and recreate that rule. A public rule in another Project can stay. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules).
- Confirm that the VPCs and security groups that host your Workers and Temporal Clients allow egress to the Stable IP ranges on port `443`. If your egress is currently locked down to the PrivateLink VPC endpoint, you must expand it first.

**Steps:**

1. **Allowlist Stable IPs in your firewall first.** Fetch the IP ranges for your Namespace's region from [https://docs.temporal.io/json/stable-ip-ranges-prod.json](/json/stable-ip-ranges-prod.json) and add them to your egress allowlist before changing anything else. If you skip this step, traffic will be lost.

2. **Create a public Connectivity Rule with Stable IPs enabled in the Namespace's Project.** Use the Temporal CLI:

   ```command
   temporal cloud connectivity public create --enable-stable-ips
   ```

   If that Project already has a public Connectivity Rule without Stable IPs, delete and recreate it. There is no in-place update flag, and creating a second public rule in the same Project returns an error. A public rule in another Project can stay. See [Connectivity Rule limits](/evaluate/cloud/limits#connectivity-rules). Wait for the rule to reach `ACTIVE` before continuing.

3. **Attach the public rule to the Namespace.** Keep the existing private rule attached during the overlap window, then add the public rule:

   ```command
   temporal cloud namespace connectivity attach \
     --namespace "<namespace>.<account>" \
     --connectivity-rule-id "<public-rule-id>"
   ```

   Workers continue to use PrivateLink at this point, because your private DNS still resolves the Namespace endpoint to the VPC endpoint. The public path is now *allowed* on Temporal's side but not yet *used*. Keep the private rule attached until step 4 is complete. If you detach the private rule while Workers are still resolving the Namespace endpoint to the PrivateLink VPC endpoint, Temporal Cloud blocks those connections at the edge.

4. **Switch client DNS resolution from PrivateLink to public.** This is the actual cutover. Remove the private DNS override — Route 53 private hosted zone, `/etc/hosts`, or VPC DNS resolver rules — so that `<namespace>.<account>.tmprl.cloud` resolves through public DNS to a Stable IP. Restart Workers to drop existing PrivateLink connections and force re-resolution.

   A single connection uses one network path or the other. Workers connect over either PrivateLink or Stable IPs based on DNS resolution at connect time, and there is no in-place switch without dropping the connection. That is why this step includes a Worker restart.

5. **Verify traffic is flowing over Stable IPs.** On a Worker host, run `dig <namespace>.<account>.tmprl.cloud` and confirm the answer is a Stable IP from the [published JSON list](/json/stable-ip-ranges-prod.json), not your VPC endpoint IP. Confirm Workers are polling and there are no `RESOURCE_EXHAUSTED` or connection errors in their logs.

6. **Detach the private Connectivity Rule.** Once you are confident all traffic is on the public path, remove the private rule:

   ```command
   temporal cloud namespace connectivity detach \
     --namespace "<namespace>.<account>" \
     --connectivity-rule-id "<private-rule-id>"
   ```

7. **(Optional) Tear down the PrivateLink VPC endpoint** in AWS to stop incurring PrivateLink hourly and data-processing charges. Do not do this until step 6 has been stable for at least a few hours.

### Changes to Stable IP ranges

Once a region's Stable IP ranges are published, Temporal avoids adding new ranges to that region's list, and Temporal has no plans to ever remove a published range.

If an unforeseen event requires Temporal to add or remove a region's published ranges, Temporal will give at least three months, and preferably six months, advance notice through targeted communication to accounts that have Stable IPs enabled. Changes to IP Ranges are publicly announced in the [Temporal Cloud changelog](https://temporal.io/change-log).

Temporal never redirects Namespace Endpoints with Stable IPs to IP addresses that are not on the published list.

### Pricing

Temporal Cloud does not charge additional fees for using Stable IPs.

### How to disable Stable IPs

You can disable Stable IPs on a Namespace by detaching the public Connectivity Rule from that Namespace. Temporal performs a DNS update to point the Namespace endpoint back to dynamic IP addresses.

Existing Worker connections are not interrupted. New Worker connections will no longer resolve to Stable IPs after DNS propagates.

---

# Workflow History Export

Source: https://docs.temporal.io/cloud/export

> Export closed Workflow Histories from Temporal Cloud to S3, GCS, or Azure Blob Storage hourly in protobuf format for compliance and analytics.

Workflow History Export allows users to export closed Workflow Histories from Temporal Cloud to cloud object storage (AWS S3, GCP GCS, or Azure Blob Storage), enabling:

- Compliance and audit trails of complete Event History data in [proto format](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto)
- Analytics on Event History when ingested to the data platform of your choice

Workflow History Export in Temporal Cloud provides similar functionality as [Archival](/self-hosted-guide/archival) in a Self-Hosted Temporal Server. 
Archival is not supported in Temporal Cloud.

Exports run hourly, beginning 10 minutes after the hour. 
Allow up to 24 hours for a closed Workflow to appear in the exported file.
Delivery is guaranteed at least once.

## What's in the exported data 

Each exported file contains one or more complete Workflow Execution histories serialized as protocol buffers using the [`WorkflowExecutions`](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto) proto.

Each history is an ordered sequence of events that records everything that happened during a Workflow Execution:

- **Workflow configuration** - Input data, timeouts, Task Queue, retry policies, search attributes, and memo
- **Activity lifecycle** - Each Activity scheduled, started, completed, or failed/timed out, including inputs and results
- **Timers** - Timer starts and fires
- **Signals and Updates** - External Signals received and Update requests handled
- **Child Workflows** - Child Workflow starts and completions
- **Workflow result** - How the Workflow ended (completed, failed, timed out, terminated, canceled, or continued-as-new)

Search attributes in the export use your **user-defined names** (for example, `customerId`), not internal column names.

The export format is **protobuf binary**.
You must deserialize using the [proto schema](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto) before the data is human-readable.

The following is a simplified JSON representation of what one Workflow Execution looks like after deserialization.
This example shows a Workflow that started, ran one Activity, and completed:

```json
{
  "items": [
    {
      "history": {
        "events": [
          {
            "eventId": "1",
            "eventTime": "2025-02-24T18:00:00Z",
            "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED",
            "workflowExecutionStartedEventAttributes": {
              "workflowType": { "name": "OrderWorkflow" },
              "taskQueue": { "name": "order-processing" },
              "input": { "payloads": ["...serialized input..."] },
              "workflowExecutionTimeout": "3600s",
              "searchAttributes": {
                "indexedFields": {
                  "customerId": { "data": "\"customer-42\"" },
                  "orderType": { "data": "\"standard\"" }
                }
              }
            }
          },
          {
            "eventId": "2",
            "eventTime": "2025-02-24T18:00:00Z",
            "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED"
          },
          {
            "eventId": "3",
            "eventTime": "2025-02-24T18:00:01Z",
            "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
            "activityTaskScheduledEventAttributes": {
              "activityType": { "name": "ChargeCustomer" },
              "taskQueue": { "name": "order-processing" },
              "input": { "payloads": ["...serialized input..."] },
              "scheduleToCloseTimeout": "300s",
              "startToCloseTimeout": "60s"
            }
          },
          {
            "eventId": "4",
            "eventTime": "2025-02-24T18:00:02Z",
            "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED"
          },
          {
            "eventId": "5",
            "eventTime": "2025-02-24T18:00:03Z",
            "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
            "activityTaskCompletedEventAttributes": {
              "result": { "payloads": ["...serialized result..."] }
            }
          },
          {
            "eventId": "6",
            "eventTime": "2025-02-24T18:00:03Z",
            "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED"
          },
          {
            "eventId": "7",
            "eventTime": "2025-02-24T18:00:03Z",
            "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_COMPLETED",
            "workflowExecutionCompletedEventAttributes": {
              "result": { "payloads": ["...serialized result..."] }
            }
          }
        ]
      }
    }
  ]
}
```

The outer `items` array can contain multiple Workflow Executions per file.
Only the key fields are shown above. Actual events include additional fields like `version`, `taskId`, and `workerVersion`.

## Prerequisites 

To use Workflow History Export, you must have:

1. A cloud account in the cloud provider where your Namespace is hosted.
2. An object storage bucket available to receive the exported History.

## Configure Workflow History Export 

### AWS

[AWS S3 Export Configuration](/cloud/export/aws-export-s3)

### GCP

[GCP GCS Export Configuration](/cloud/export/gcp-export-gcs)

### Azure

[Azure Blob Storage export configuration](/cloud/export/azure-export-blob-storage)

## Verify export setup 

From the Export configuration page, select **Verify**.
This validates that Temporal can successfully write a test file to your object storage.

If everything is configured correctly, you will see a `Success` status indicating Temporal has written to the object store.

## Monitor export progress 

After Export has been configured, you can check that it's still working in several ways:

1. **Object Storage**:

   - File Delivery: After the initial hour of setting up, inspect your object storage.
     You should see the exported Event History files.
   - Directory Structure: Your exported files will adhere to the following naming convention and path:

   ```bash
   //[bucket-name]/temporal-workflow-history/export/[Namespace]/[Year]/[Month]/[Day]/[Hour]/[Minute]/
   ```
The exported file name will include a randomly generated ID. The time recorded in the directory structure
is the time the export uploads to object storage, not the Workflow completion time.

2. **Temporal Cloud Web UI**:

   - Export UI:

      - Last Successful Export: This displays the timestamp of the most recent successful export.
      - Last Status Check: This reflects the timestamp of the latest internal Workflow healthcheck.

   - Usage Dashboard:
      - Actions from the Export Job are included in the [Usage Dashboard](/cloud/actions-usage).

3. **Email**:
   - Emails are sent to `Namespace Administrator`, `Account Owner`, and `Global Administrator` roles when a Workflow History Export job fails due to a user related error (such as Object Store permissions issue).

> **📝 Note:**
>
> An export configuration that fails for 7 consecutive days is automatically disabled.
>

## Working with exported files

Use the proto schema defined [here](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto) to deserialize exported files.

### Using exported files in analytics

It can be useful to convert protos to another format to perform analytics on the data. To convert protos to parquet, follow 
[the example Python Workflow](https://github.com/temporalio/samples-python/tree/main/cloud_export_to_parquet). Note that this example Workflow:
* Transforms the nested proto structure into a flat, tabular format.
* Each row in the table represents a single history event from a Workflow. To preserve their relationship post-conversion, the `workflowID` and `runID` is included in every row.
* If you have enabled the codec server, the payload field is encrypted. This field may contain characters that are not recognized when loaded into a database so the payload field is excluded in this example.

## Export and High Availability Namespaces 

### Export region persistence

When Export is configured for a [High Availability](/cloud/high-availability) Namespace, the export is tied to the specific region where it was initially set up. The export configuration does not automatically failover with the Namespace.

- If Export is configured in Region A, it will continue to export from Region A's storage even after a Namespace failover to Region B
- Exports always read from and write to the same region where they were originally configured
- The export process is independent of Namespace failover events
- Export does not fail over automatically because we prioritize data completeness and consistency over real-time availability for exports. HA data replication has inherent latency, which could result in incomplete or inconsistent exports during a failover.

### Failover scenarios

**Namespace Failover with Healthy Primary Region**: When a Namespace fails over to a secondary region but the primary region remains healthy (including its blob storage), the export job continues to operate from the primary region. It does not automatically switch to export data from the secondary region.

**Primary Region Outage**: If the primary region (where Export was configured) experiences a complete outage including object storage: Exports will be unavailable until the primary region recovers. Once the primary region recovers, export will resume and include any Workflow histories that occurred during the outage. There may be delays in export processing, but the complete dataset will eventually be available. It does not automatically switch to export data from the secondary region.

---

# Exporting Workflow Event History to AWS S3

Source: https://docs.temporal.io/cloud/export/aws-export-s3

> Export Workflow History to AWS S3

## Prerequisites

Before configuring the Export Sink, ensure you have the following:

- An AWS S3 bucket.
   - The S3 bucket must reside in the same region as your Namespace.
- (Optional) An IAM role that has write permission to the above S3 bucket.
   - You can follow the automation in the UI to create the IAM role. Please pre-create the role if setting up Export via Terraform or the CLI.
- (Optional) A KMS ARN associated with the S3 bucket.

## Configure Workflow History export

There are multiple ways to configure export: through the [Temporal Cloud UI](#using-temporal-cloud-ui), the [CLI](#using-the-cli), or [`terraform`](#using-terraform).

### Using Temporal Cloud UI

You can use the Temporal Cloud UI to configure the Workflow History Export.

The Temporal Cloud UI provides two ways for configuring Workflow History Export:

- [Automated setup](#automated-setup) (recommended): The Cloud UI launches the AWS CloudFormation Console to create a stack with write permission to the S3 bucket.
- [Manual setup](#manual-setup): The Cloud UI provides a CloudFormation template for users to manually configure a CloudFormation stack.

> **ℹ️ Info:**
> Why does Temporal Cloud provision multiple internal IAM roles to trust for Export?
>
> Temporal Cloud creates multiple intermediary IAM roles for export operations for security purposes. 
> The system randomly selects from these roles when writing to your storage sink, which provides several benefits:
>
> - **Security isolation**: If one IAM role is compromised or needs to be decommissioned, other IAM roles remain available
> - **Load distribution**: Avoids relying on a single IAM role, reducing security risk
> - **Warm standby**: Keeps multiple IAM roles active to avoid potential throttling when switching between IAM roles
> - **Reliability**: Provides resilience against cloud provider account-level issues that could affect a single IAM role
>
> This approach prioritizes security and availability, ensuring robust export operations even if individual IAM roles encounter issues.

The following steps guide you through setting up Workflow History Export using the Temporal Cloud UI.

![](https://docs.temporal.io/img/cloud/gcp/export-sink-ui.png)

> **💡 Tip:**
>
> Don't forget to click **Create** at the end of your setup to confirm your export.
>

#### Automated setup

You can use the automated setup to create a CloudFormation stack with write permission to your S3 bucket.
Make sure to verify the export setup before you save the configuration.

1. Open the Temporal Cloud UI and navigate to the Namespace you want to configure.
2. Select **Configure** from the **Export** card.
3. Provide the following information to configure the export sink and then select **Create and launch stack**:
   - Name: A name for the export sink.
   - AWS S3 Bucket Name: The name of the configured AWS S3 bucket to send Closed Workflow Histories to.
   - AWS Account ID: The AWS account ID.
   - Role Name: The name of the AWS IAM role to use for the CloudFormation stack that has write permission to the S3 bucket.
   - KMS ARN: (optional) The ARN of the AWS KMS key to use for encryption of the exported Event History.
4. You will be taken to the CloudFormation Console to create the stack with pre-populated information.
   - Review the information and then select **Create stack**.

#### Manual setup

You can manually configure a CloudFormation stack using the provided template.

1. Open the Temporal Cloud UI and navigate to the Namespace you want to configure.
2. Select **Configure** from the **Export** card.
3. Select **Manual** from **Access method**.
   - Enter the Template URL into your web browser to download your copy of the CloudFormation template.
   - Configure the CloudFormation template for your export sink.
   - Follow the steps in the [AWS documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-console-create-stack-template.html) by uploading the template to the CloudFormation console.

### Using the CLI

**Temporal CLI**

Run the `temporal cloud namespace export s3 create` command and provide the following information:

- `--namespace`: The Namespace to configure export for.
- `--sink-name`: The name of the export sink.
- `--role-arn`: The ARN of the AWS IAM role to use for the CloudFormation stack that has write permission to the S3 bucket.
- `--bucket-name`: The name of the AWS S3 bucket.
- `--region`: The AWS region the S3 bucket is in.

For example:

```command
temporal cloud namespace export s3 create \
  --namespace "your-namespace.your-account" \
  --sink-name "your-sink-name" \
  --role-arn "arn:aws:iam::123456789012:role/test-sink" \
  --bucket-name "your-aws-s3-bucket-name" \
  --region "us-east-1"
```

Retrieve the status of this command by running the `temporal cloud namespace export get` command.

For example:

```command
temporal cloud namespace export get \
  --namespace "your-namespace.your-account" \
  --sink-name "your-sink-name"
```

**tcld**

Run the `tcld namespace export s3 create` command and provide the following information:

- `--namespace`: The Namespace to configure export for.
- `--sink-name`: The name of the export sink.
- `--role-arn`: The ARN of the AWS IAM role to use for the CloudFormation stack that has write permission to the S3 bucket.
- `--s3-bucket-name`: The name of the AWS S3 bucket.

For example:

```command
tcld namespace export s3 create --namespace "your-namespace.your-account" --sink-name "your-sink-name" --role-arn "arn:aws:iam::123456789012:role/test-sink" --s3-bucket-name "your-aws-s3-bucket-name"
```

Retrieve the status of this command by running the `tcld namespace export s3 get` command.

For example:

```command
tcld namespace export s3 get --namespace "your-namespace.your-account" --sink-name "your-sink-name"
```

The following is an example of the output:

```json
{
  "name": "your-sink-name",
  "resourceVersion": "a6442895-1c07-4da4-aaca-58d57d338345",
  "state": "Active",
  "spec": {
    "name": "your-sink-name",
    "enabled": true,
    "destinationType": "S3",
    "s3Sink": {
      "roleName": "your-export-test",
      "bucketName": "your-export-test",
      "region": "us-east-1",
      "kmsArn": "",
      "awsAccountId": "123456789012"
    }
  },
  "health": "Ok",
  "errorMessage": "",
  "latestDataExportTime": "0001-01-01T00:00:00Z",
  "lastHealthCheckTime": "2023-08-14T21:30:02Z"
}
```

### Using `terraform`

See the [terraform export support](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace_export_sink) for setup instructions.

### Next steps

- [Verify export setup](/cloud/export#verify)
- [Monitor export progress](/cloud/export#monitor)
- [Work with exported files](/cloud/export#working-with-exported-files)

---

# Export Workflow Event History to Azure Blob Storage

Source: https://docs.temporal.io/cloud/export/azure-export-blob-storage

> Configure Azure Blob Storage access for Temporal Cloud Workflow History Export using admin consent, Azure CLI, or Terraform, then verify your sink.

> **Public Preview**

Export closed Workflow Event Histories from an Azure-hosted Temporal Cloud Namespace to an Azure Blob Storage container.
Export is available wherever Temporal Cloud supports Azure Namespaces.
For export timing and file format, see [Workflow History Export](/cloud/export).

## Prerequisites

Before configuring the export sink, prepare the following:

- An Azure storage account and Blob container. The storage account **must be in the same region as your Namespace**.
- The Azure tenant identifier, subscription identifier, resource group, storage account name, and container name.
- The ability to add an enterprise application to your Microsoft Entra tenant, such as through the **Cloud Application Administrator** or **Application Administrator** role. See [Microsoft's enterprise application prerequisites](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/create-service-principal-cross-tenant#prerequisites).
- Permission to assign Azure roles on the storage account: `Microsoft.Authorization/roleAssignments/write`, such as through **Role Based Access Control Administrator**. See [Azure role assignment prerequisites](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-cli#prerequisites).
- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) for the commands below, including the storage role assignments in the admin consent path.

## Enter the export sink details

1. Open the Temporal Cloud UI and navigate to your Namespace.
2. Select **Configure** from the **Export** card.
3. Enter the following fields:

   | Field | Value |
   | --- | --- |
   | **Name** | A name for the export sink. |
   | **Azure Tenant ID** | The identifier of the Azure tenant that owns your storage account. |
   | **Azure Subscription ID** | The identifier of the subscription that contains your storage account. |
   | **Storage Account Name** | The name of your storage account. |
   | **Container Name** | The name of the Blob container that will receive exported Event Histories. |
   | **Resource Group** | The resource group that contains your storage account. |

## Register the Temporal enterprise application

Temporal provides a customer-specific application client identifier in the export configuration UI.
Use that value wherever the commands below show `<application-client-id>`.
Adding the enterprise application creates a service principal in your Azure tenant.

Choose either **Admin Consent** or **Azure CLI** under **Deploy Method** to register the application.
After completing either path, [assign storage roles](#assign-storage-roles).

### Register with admin consent

1. Select **Admin Consent**.
2. Select **Grant admin consent**. You must enter **Azure Tenant ID** before this button is enabled.
3. Sign in to the target Azure tenant and complete the consent flow.
4. Return to the export configuration page.

For details, see [Create an enterprise application from a multitenant application](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/create-service-principal-cross-tenant?pivots=admin-consent-url).

### Register with Azure CLI

1. Select **Azure CLI** in the export configuration UI.
2. Sign in to the tenant that owns your storage account:

   ```bash
   az login --tenant "<tenant-id>"
   az account set --subscription "<subscription-id>"
   ```

3. Register the enterprise application using the client identifier shown in the UI:

   ```bash
   az ad sp create --id "<application-client-id>"
   ```

## Assign storage roles

After registering the enterprise application through either path, assign both of the following roles at the storage account scope:

| Role | Required permission it provides |
| --- | --- |
| **Reader and Data Access** | `Microsoft.Storage/storageAccounts/read` |
| **Storage Blob Data Contributor** | `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write` |

If you used admin consent, sign in to Azure CLI and select the subscription before running the role assignment commands:

```bash
az login --tenant "<tenant-id>"
az account set --subscription "<subscription-id>"
```

Replace the placeholders with your application client identifier and storage account details:

```bash
az role assignment create \
  --assignee "<application-client-id>" \
  --role "Reader and Data Access" \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>"

az role assignment create \
  --assignee "<application-client-id>" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>"
```

### Configure access with Terraform

As an alternative to registering the service principal and assigning the built-in roles manually, use the [Azure export service principal Terraform module](https://github.com/temporalio/terraform-modules/tree/main/modules/azure-export-sp).
The module creates or reuses the service principal and assigns a custom role with the required storage account read and blob write permissions.
After applying the module, return to the Temporal Cloud UI to verify and save the export sink.

## Verify and save the export sink

1. In the export configuration page, select **Verify** to check that Temporal can write a test file to the container.
2. Confirm that verification succeeds. If it fails, check the tenant, storage account details, and role assignments before trying again.
3. Select **Create** to save the export sink.
4. The page refreshes automatically. Confirm that the Export screen shows **Enabled**.

## Next steps

- [Monitor export progress](/cloud/export#monitor)
- [Work with exported files](/cloud/export#working-with-exported-files)

---

# Exporting Workflow Event History to GCS

Source: https://docs.temporal.io/cloud/export/gcp-export-gcs

> Export Workflow History to GCS

## Prerequisites 

Before configuring the Export sink, complete the following steps in Google Cloud.

1. Create a GCS bucket and take note of its bucket name, for example, "test-export"

- Enable customer-managed encryption keys (CMEK) if you need additional security for your GCS bucket.
- Currently, only single region buckets are supported (choose "Region" option when creating the bucket in GCS, not "Multi-region" or "Same-region")
- The region of the bucket must be the same as the region of your Temporal Cloud Namespace.

2. Record the GCP Project ID that owns the bucket.
3. Create a service account in the same project that grants Temporal permission to write to your GCS bucket.
4. Follow the instructions in the Temporal Cloud UI. There are two ways to set up this service account:
   - Manual Setup:
     - Input the service account ID, GCP project ID and GCS bucket name.
     - Follow the instructions, manually set up a new service account.
   - Automated Setup:
     - Use the [Terraform template](https://github.com/temporalio/terraform-modules/tree/main/modules/export-sa) to create the service account.

## Configure Workflow History export

There are multiple ways to configure export: through the [Temporal Cloud UI](#using-temporal-cloud-ui), the [CLI](#using-the-cli), or [`terraform`](#using-terraform).

> **📝 Note:**
> Why does Temporal Cloud provision multiple service accounts for Export?
>
> Temporal Cloud creates multiple intermediary service accounts for export operations primarily for security purposes. The system randomly selects from these accounts when writing to your storage sink, which provides several benefits:
>
> - **Security isolation**: If one service account is compromised or needs to be decommissioned, other accounts remain available
> - **Load distribution**: Prevents exclusively using a single account, reducing security risk
> - **Warm standby**: Keeps multiple accounts active to avoid potential throttling when switching between accounts
> - **Reliability**: Provides resilience against cloud provider account-level issues that could affect a single service account
>
> This approach prioritizes security and availability, ensuring robust export operations even if individual service accounts encounter issues.

### Using Temporal Cloud UI

The following steps guide you through setting up Workflow History Export using the Temporal Cloud UI.

![](https://docs.temporal.io/img/cloud/gcp/export-sink-ui-gcp.png)

1. In the Cloud UI, navigate to the Namespaces section. Confirm that the Export feature is visible and properly displayed.
2. Configure the Export sink for a Namespace:
   1. Choose GCS as the sink type.
   2. Provide the following information:
      1. Name
      2. Service account ID
      3. GCP Project ID
      4. GCS bucket name
3. After inputting the necessary values, click on **Verify**.
   You should be able to write to the sink successfully.
   If not, please fix any errors or reach out to support for help.
   - If you just created the GCS bucket and granted permission for your service account, it may take some time for the permission to propagate. You may need to wait up to 5 minutes before clicking the **Verify** button to verify the connection.
4. Clicking **Create** will complete the Export sink setup.
5. The page will auto-refresh and you should see the status “Enabled” on the Export screen.
   You are now ready to export Workflow histories.
6. You can toggle the enable button if you want to stop export and resume in the future.
   **Note**: when you re-enable the feature, it will start from the current point in time, and not from the time when you disabled export.
7. You can also delete export by clicking **Delete**.

> **💡 Tip:**
>
> Don't forget to click Create at the end of your setup to confirm your export.
>

### Using the CLI

**Temporal CLI**

1. [Install the Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension) for the Temporal CLI.
2. Run the `temporal cloud namespace export gcs create` command and provide the following information:
   - `--namespace`: The Namespace to configure export for.
   - `--sink-name`: The name of the export sink.
   - `--service-account-email`: The service account that has access to the sink.
   - `--bucket-name`: The name of the GCP GCS bucket.
   - `--region`: The region the GCS bucket is in.

   For example:

   ```bash
   temporal cloud namespace export gcs create \
     --namespace test.ns \
     --sink-name test-sink \
     --service-account-email test-sink@test-export-sink.iam.gserviceaccount.com \
     --bucket-name test-export-validation \
     --region us-central1
   ```

3. Check the status of this command by either viewing the Namespace Export status in the Temporal Cloud UI or by
   retrieving the sink and looking for the state of "Active":

   ```bash
   temporal cloud namespace export get --namespace test.ns --sink-name test-sink
   ```

**tcld**

To access export-related commands in tcld, please follow these steps:

1. [Download the latest version of tcld](/cloud/tcld#install-tcld).
2. Make sure your tcld version is v0.35.0 or above.
3. Run the command: `tcld n export gcs`:
   ```bash
   NAME:
      tcld namespace export gcs - Manage GCS export sink

   USAGE:
      tcld namespace export gcs command [command options] [arguments...]

   COMMANDS:
      create, c    Create export sink
      update, u    Update export sink
      validate, v  Validate export sink
      get, g       Get export sink
      delete, d    Delete export sink
      list, l      List export sinks
      help, h      Shows a list of commands or help for one command

   OPTIONS:
      --help, -h  show help
   ```

4. Run the `tcld n export gcs create` command and provide the following information:
   - `--namespace`: The Namespace to configure export for.
   - `--sink-name`: The name of the export sink.
   - `--service-account-email`: The service account that has access to the sink.
   - `--gcs-bucket`: The name of the GCP GCS bucket.

   For example:

   ```bash
   tcld n export gcs create -n test.ns --sink-name test-sink --service-account-email test-sink@test-export-sink.iam.gserviceaccount.com --gcs-bucket test-export-validation
   ```
5. Check the status of this command by either viewing the Namespace Export status in the Temporal Cloud UI or using the following command and looking for the state of "Active":

```bash
tcld n export gcs g -n test.ns --sink-name test-sink
{
  "name": "test.ns",
  "resourceVersion": "b954de0c-c6ae-4dcc-90bd-3918b52c3f28",
  "state": "Active",
  "spec": {
    "name": "test-sink",
    "enabled": true,
    "destinationType": "Gcs",
    "s3Sink": null,
    "gcsSink": {
      "saId": "test-sink",
      "bucketName": "test-export-validation",
      "gcpProjectId": "test-export-sink",
    }
  },
  "health": "Ok",
  "errorMessage": "",
  "latestDataExportTime": "0001-01-01T00:00:00Z",
  "lastHealthCheckTime": "2024-01-23T06:40:02Z"
}
```

### Using `terraform`

See the [Terraform export support](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace_export_sink) for setup instructions.

### Next steps

- [Verify export setup](/cloud/export#verify)
- [Monitor export progress](/cloud/export#monitor)
- [Work with exported files](/cloud/export#working-with-exported-files)

---

# Get started with Temporal Cloud

Source: https://docs.temporal.io/cloud/get-started

> Sign up, create a Namespace, choose API key or mTLS authentication, and connect your Clients and Workers to run a Workflow.

Getting started with Temporal Cloud involves a few key steps:

1. [Sign up for Temporal Cloud](#sign-up-for-temporal-cloud)
1. [Create a Namespace](#create-a-namespace)
1. [Set up your Clients and Workers](#set-up-your-clients-and-workers)
1. [Run your first Workflow](#run-your-first-workflow)
1. [Invite your team](#invite-your-team)

## Sign up for Temporal Cloud

To create a Temporal Cloud account, you can:

- Sign up [directly](https://temporal.io/get-cloud); or
- Subscribe at the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) or [GCP Marketplace](https://console.cloud.google.com/marketplace/product/temporal-public/temporal-cloud-pay-as-you-go). Signing up through a cloud marketplace is similar to signing up directly on the Temporal Cloud site, but billing goes through your AWS/GCP
  account.

For information about Temporal Cloud Pricing, see our [Pricing Page](/cloud/pricing).

## Create a Namespace

See [Managing Namespaces](/cloud/namespaces#create-a-namespace) to create your first namespace.

Temporal Cloud supports either [API key](/cloud/api-keys) or [mTLS](/cloud/certificates) authentication for each
namespace. If you're not sure which to use, we recommend using [API keys](/cloud/api-keys) because they're easier to
manage and rotate for most teams. If your organization already has private key infrastructure (PKI) and is familiar with
cert management, then [mTLS](/cloud/certificates) is an excellent choice.

## Set up your Clients and Workers

See our guides for connecting each SDK to your Temporal Cloud Namespace:

- [Connect to Temporal Cloud in .NET](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Go](/develop/go/client/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Java](/develop/java/client/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in PHP](/develop/php/client/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Python](/develop/python/client/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Ruby](/develop/ruby/client/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Rust](/develop/rust/client/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in TypeScript](/develop/typescript/client/temporal-client#connect-to-temporal-cloud)

## Run your first Workflow

See our guides for starting a workflow using each SDK:

- [Start a workflow in .NET](/develop/dotnet/client/temporal-client#start-workflow)
- [Start a workflow in Go](/develop/go/client/temporal-client#start-workflow-execution)
- [Start a workflow in Java](/develop/java/client/temporal-client#start-workflow-execution)
- [Start a workflow in PHP](/develop/php/client/temporal-client#start-workflow-execution)
- [Start a workflow in Python](/develop/python/client/temporal-client#start-workflow-execution)
- [Start a workflow in Ruby](/develop/ruby/client/temporal-client#start-workflow)
- [Start a workflow in Rust](/develop/rust/client/temporal-client#start-workflow-execution)
- [Start a workflow in TypeScript](/develop/typescript/client/temporal-client#start-workflow-execution)

## Invite your team

See [Managing users](/cloud/manage-access/users) to add other users and assign them roles.

You can also use [Service Accounts](/cloud/manage-access/service-accounts) to represent machine identities.

Since you created the account when you signed up, your email address is the first
[Account Owner](/cloud/manage-access/roles-and-permissions#account-level-roles) for your account.

---

# User management

Source: https://docs.temporal.io/cloud/get-started/users-invite

> Learn how to manage user invitations for Temporal Cloud

**Web UI**

To invite users using the Temporal Cloud UI:

1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page, select **Create Users** in the upper-right portion of the window.
1. On the **Create Users** page in the **Email Addresses** box, type or paste one or more email addresses.
1. In **Account-Level Role**, select a [Role](/cloud/manage-access/roles-and-permissions#account-level-roles). The Role
   applies to all users whose email addresses appear in **Email Addresses**.
1. If the account has any Namespaces, they are listed under **Grant access to Namespaces**. To add a permission, select
   the checkbox next to a Namespace, and then select a
   [permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions). Repeat as needed.
1. When all permissions are assigned, select **Send Invite**.

**Temporal CLI**

Use the [`temporal cloud user invite`](/cli/command-reference/cloud/user#invite) command. Specify the user's email, an
account-level role, and optionally one or more Namespace permissions.

Available account roles: `owner` | `admin` | `developer` | `finance-admin` | `read` | `metrics-read`.

Available Namespace permissions: `admin` | `write` | `read`.

```command
temporal cloud user invite \
  --email <user@example.com> \
  --account-role <role> \
  --namespace-access <namespace.account>=<permission>
```

Repeat `--namespace-access` to grant permissions on more than one Namespace. `--email` takes a single address, so invite
one user per command:

```command
temporal cloud user invite \
  --email user1@example.com \
  --account-role developer \
  --namespace-access ns1.my-account=admin \
  --namespace-access ns2.my-account=write
```

**tcld**

Use the [`tcld user invite`](/cloud/tcld/user/#invite) command. Specify the user's email, an account-level role, and
optionally one or more Namespace permissions.

Available account roles: `admin` | `developer` | `read`.

Available Namespace permissions: `Admin` | `Write` | `Read`.

```command
tcld user invite \
  --user-email <user@example.com> \
  --account-role <role> \
  --namespace-permission <namespace>=<permission>
```

You can invite multiple users and assign multiple Namespace permissions in a single command:

```command
tcld user invite \
  --user-email user1@example.com \
  --user-email user2@example.com \
  --account-role developer \
  --namespace-permission ns1=Admin \
  --namespace-permission ns2=Write
```

### Frequently asked questions

#### Can multiple Temporal Cloud accounts share the same email domain?

Yes. Multiple Temporal Cloud accounts can coexist with users from the same email domain.
Each account has its own independent SAML configuration, tied to its unique Account Id.
We recommend configuring [SAML](/cloud/manage-access/saml) for each account independently.
For the smoother login experience, you can configure SAML for each account separately and use IdP-initiated login: you click the relevant app tile in your identity provider's portal to access the Temporal Cloud account associated with your email address directly.

#### Can the same email be used across different Temporal Cloud accounts?

No. Each email address can only be associated with a single Temporal Cloud account.
If you need access to multiple accounts, you’ll need a separate invite for each one using a different email address.

#### Can I use Google or Microsoft SSO after signing up with email and password?

If you originally signed up for Temporal Cloud using an email and password, you won’t be able to log in using Google or Microsoft single sign-on.

If you prefer SSO, ask your Account Owner to delete your current user and send you a new invitation.
During re-invitation, be sure to sign up using your preferred authentication method.

Use the [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users) endpoint to invite a user.

```
POST /cloud/users
```

The request body includes a `spec` with the following fields:

- `spec.email` — The email address of the user to invite.
- `spec.access.account_access.role` — The account-level role to assign.
- `spec.access.namespace_accesses` — A map of Namespace names to permissions.

Available roles: `ROLE_ADMIN` | `ROLE_DEVELOPER` | `ROLE_READ` | `ROLE_OWNER` | `ROLE_FINANCE_ADMIN`.

Available Namespace permissions: `PERMISSION_ADMIN` | `PERMISSION_WRITE` | `PERMISSION_READ`.

The new users receive an email with a link to accept the invitation and complete their setup. The new user must use this
link to sign up to be added to your account unless the account has a SAML configuration. If your account has a SAML
configuration, the new user can sign in using their existing SAML credentials and be included in the account
automatically.

> **⚠️ Caution:**
>
> The new user must use the same authentication method they originally signed up with to sign in to Temporal Cloud. If
> they used single sign-on (SSO), they must use the same SSO provider to sign in to Temporal Cloud. If they used email and
> password authentication, they must use the same email and password to sign in to Temporal Cloud, and cannot use SSO,
> even if the underlying email address is the same.
>

---

# High Availability

Source: https://docs.temporal.io/cloud/high-availability

> Temporal Cloud's Namespace with High Availability features offers automatic failover, synchronized data, and replication for workloads requiring disaster-tolerant deployment and a higher uptime SLA.

Temporal keeps your Workflows running even when a Worker crashes. But what happens when a whole data center crashes? Or a region?

In the cloud, outages are commonplace. An outage can bring down a whole data center, cluster, region, or cloud provider. To be durable in the cloud, Workflows and applications must handle these outages smoothly, just like Temporal handles a Worker crash.

Temporal Cloud's High Availability features add extra reliability to Temporal Cloud Namespaces by handling cloud outages. Using asynchronous replication between multiple regions or cloud providers, combined with automatic outage detection and failover, High Availability keeps your Workflows running even during a cloud region outage.
This extra availability comes with an enhanced [SLA](https://temporal.io/sla).

> **💡 Tip:**
> White paper
>
> For an in-depth guide covering everything from why you need High Availability to setting it up in production and advanced options, read the [High Availability White Paper](https://temporal.io/pages/high-availability-white-paper).
>

> **⚠️ Caution:**
> GCP capacity restriction
>
> Before creating a GCP Namespace or adding a GCP replica, check [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction) for regional availability and exception requirements.
>

## Built-in reliability

Even without High Availability features, Temporal Cloud provides robust reliability and a contractual [Service Level Agreement (SLA)](https://temporal.io/sla) against service errors.

Each standard Temporal Namespace uses replication across three [Availability Zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones) (AZs) to ensure high availability.
An Availability Zone is akin to an isolated data center managed by a cloud hyperscaler, with independent power, networking, and cooling infrastructure.

Replication across AZs makes sure that any changes to Workflow state or History are saved in all three AZs _before_ the Temporal Service acknowledges a change back to the Client.
As a result, your standard Temporal Namespace stays operational even if one of its three AZs becomes unavailable.
This provides the basis of the SLA for standard Temporal Cloud Namespaces.

However some critical use cases--such as customer-facing applications--require even better availability. That is where Temporal Cloud's High Availability features come in.

## High Availability features 

High Availability features extend Temporal Cloud's replication across regions and cloud providers, so your Namespace keeps running even when a whole region or cloud provider goes down:

| **Deployment**                          | **Description**                                            |
| --------------------------------------- | ---------------------------------------------------------- |
| **Multi&#8209;region&nbsp;Replication** | Namespace is replicated across two cloud regions           |
| **Multi&#8209;cloud&nbsp;Replication**  | Namespace is replicated across different cloud providers   |

### Key features

- **Real-time replication** — Temporal replicates your Namespace across distant regions or cloud providers with no performance impact to your Workers or Workflows.
- **Automatic failover with 20-minute RTO** — Temporal manages failover with a 20-minute [RTO](/cloud/rpo-rto). You can also [trigger failover](/cloud/high-availability/failovers) manually at any time, for example for testing.
- **Transparent DNS routing** — On failover, DNS reroutes your [Namespace Endpoint](/cloud/namespaces#access-namespaces) to the active region. [Requests that reach the replica are forwarded to the active region automatically](#request-forwarding).
- **Sub-1-minute RPO** — In a failover during an outage, the [Recovery Point Objective](/cloud/rpo-rto) is under one minute.
- **Real-time lag monitoring** — Monitor your Namespace's replication lag in real time to understand your current RPO.
- **Conflict resolution** — If the two regions are not fully in sync at the time of failover, Temporal's conflict resolution process reconciles discrepancies and ensures data integrity.

> **ℹ️ Info:**
> Region availability
>
> You can usually choose your replica region, but the replica must be on the same continent as the primary region.
> This means that a few Temporal Cloud regions do not yet support Multi-region Replication or Multi-cloud Replication.
> See [Regions](/evaluate/cloud/regions) for a full list of supported replica regions.
>
> You can't enable both Multi-region Replication and Multi-cloud Replication on the same Namespace at the same time.
>

### Multi-cloud Replication

Multi-cloud Replication spreads a Namespace across entirely different cloud providers, keeping your Namespace running even during a cloud-wide outage.
If a provider outage, service disruption, or network issue occurs, traffic automatically shifts to the replica.

Replicated data is encrypted and transmitted across the public internet between cloud providers.
This internet connectivity also allows Workers in one cloud to reach the replica in a different cloud during failover.
If you use [private connectivity](/cloud/high-availability/ha-connectivity), additional architecture work may be required to ensure your Workers can reach the replica region.

> **ℹ️ Info:**
>
> When you adopt Temporal's High Availability features, don't forget to consider the reliability of your own workers, infrastructure, and dependencies.
> Issues like network outages, hardware failures, or misconfigurations in your own systems can affect your application performance.
>
> For the highest level of reliability, distribute your dependencies across regions, and use our Multi-region or Multi-cloud replication features.
> Using physically separated regions improves the fault tolerance of your application.
>

## Request forwarding 

A Namespace with High Availability features replicates across two regions, with one replica active and one passive at any moment. The active replica accepts reads and writes; the passive replica receives replicated state asynchronously and stands ready for failover.

When a request reaches the passive replica — for example, through the passive region's Regional Endpoint — Temporal Cloud forwards the request transparently to the active replica and the response back to the Worker. This allows Workers and Clients to connect to the passive region during healthy times, and when an outage hits, start processing Workflows immediately after a failover.

Forwarding adds a cross-region hop, so requests that travel through the passive replica complete with higher average latency than requests that reach the active replica directly.

You can turn off forwarding for Worker poll requests, which keeps the passive region's Workers from processing tasks. Client requests such as Start Workflow, Signal, Query, Cancel, and Terminate are always forwarded, so a Client that reaches the passive replica keeps working either way.

To route Workers to the passive region's replica, see [How requests reach the replica](/cloud/high-availability/ha-connectivity#how-requests-reach-the-replica).

To stop forwarding Worker polls that reach the passive replica, see [Change the forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior).

To see which Client requests keep forwarding either way, see [Client requests are forwarded regardless of this setting](/cloud/high-availability/enable#client-requests-still-forwarded).

To run Worker fleets in both regions that rely on this forwarding, see [Active/Active](/cloud/high-availability/architecture-patterns#active-active).

To keep passive-region Workers on standby until failover by disabling this forwarding, see [Active/Hot-Passive](/cloud/high-availability/architecture-patterns#active-hot).

## Service levels and recovery objectives

Namespaces using High Availability have a higher [uptime SLA](https://temporal.io/sla), with sub-1-minute [RPO](/cloud/rpo-rto) and 20-minute [RTO](/cloud/rpo-rto). For detailed information:

- [Temporal Cloud Service Level Agreement (SLA)](https://temporal.io/sla)
- [Uptime, latency, and SLOs](/cloud/slo)
- [Recovery Point Objective (RPO) and Recovery Time Objective (RTO)](/cloud/rpo-rto)

## Failover

High Availability Namespaces can automatically or manually [fail over](/cloud/high-availability/failovers) to the replica if the primary is unavailable or unhealthy.
The Namespace fails over automatically, but your Workers and the rest of your architecture need their own plan — see [Worker deployment patterns for Active-Passive and Active-Active HA/DR](/cloud/high-availability/architecture-patterns).

## Target workloads

High Availability Namespaces are a great solution for Workloads where an outage would cause:

- Revenue loss
- Poor customer experience
- Problems stemming from policy/legal requirements that demand high availability

These are often major concerns for financial services, e-commerce, gaming, global SaaS platforms, bookings & reservations, delivery & shipping, and order management.

## Same-region Replication

In selected regions, you can add a replica to a Namespace in the same region.
Temporal operates a "cell architecture" and will replicate the Namespace across multiple cells in that region.
This feature is currently in [Public Preview](/evaluate/product-release-stages) in selected regions.

Failovers between cells are always managed automatically by Temporal. Unlike Multi-region and Multi-cloud Replication, you cannot disable automatic failovers and you cannot trigger a manual failover for a Same-region Replication Namespace. See [Failovers](/cloud/high-availability/failovers) for details.

## Related considerations

A few other Temporal features interact with High Availability in ways worth understanding before you enable it.

### Serverless Workers

[Serverless Workers](/serverless-workers) do not support Multi-cloud Replication. Serverless Workers must be invoked
from the same cloud provider where the Namespace is currently active. After a failover to a different cloud provider,
Serverless Workers can no longer be invoked. This is a temporary restriction during Public Preview.

You can use Serverless Workers with a Namespace that has Multi-region Replication within the same cloud provider,
but without manual intervention, the workloads that failed over to the new region will keep invoking Workers in the old
region. If the compute provider's region is experiencing degraded performance, this may impact your workloads even
though your Temporal Cloud Namespace has failed over successfully.

This is because each compute provider configuration (for example, a Lambda ARN) is scoped to a single region. The
[Worker Controller Instance](/serverless-workers#worker-controller-instance) has no mechanism to detect a failover or
redirect invocations to a Worker in the new active region. To keep processing Workflows after a failover, you must
manually reconfigure the Worker Deployment Version's compute provider to point at a Worker in the new region. See
[Constraints - Serverless Workers](/serverless-workers#constraints).

### External Storage

If your Workflows use [External Storage](/external-storage) to offload large payloads, durability of the external store
is a separate concern from Namespace replication. 

- For S3, see [Durable External Storage](/external-storage#durable-external-storage) to see how CRR + MRAP can be used.
- For external storage providers other than what is listed above, we do not yet provide guidance. Consult the providers documentation for more information.

---

# Architecture patterns for Active-Passive and Active-Active HA/DR

Source: https://docs.temporal.io/cloud/high-availability/architecture-patterns

> Choose a Worker deployment pattern for high availability and disaster recovery (HA/DR) on Temporal Cloud — Active/Passive, Active/Hot-Passive, and Active/Active — and plan multi-region failover for Workers, Clients, Codec Servers, and databases to meet your RTO and RPO.

When a cloud outage strikes, a Temporal Cloud Namespace with [High Availability](/cloud/high-availability) fails over to another region automatically, providing high availability and disaster recovery for your most important Workflows. But Workers, Workflow starters, Codec Servers, databases, network load balancers, and other external systems that touch your Workflows each need their own failover story.

In a real outage, your [recovery time](/cloud/rpo-rto) depends on your **Worker deployment pattern**: where Worker fleets run, how they failover, and which region (or regions) processes Workflows at each step. This page describes the Active/Passive, Active/Hot-Passive, and Active/Active patterns for deploying Workers and the other systems in your cloud architecture to achieve your business goals for high availability and disaster recovery (HA/DR).

## Pieces of a highly available cloud architecture 

To keep your Workflows running during a cloud outage, these components need to failover to a healthy region:

- **Temporal Cloud Namespace** - achieve this simply by enabling [High Availability](/cloud/high-availability) on the Namespace.
- **Workers** (the focus of this page) — the compute resources that execute Workflows and Activities.
- **Workflow starters and Clients** — the applications that start and Signal Workflows.
- **Codec Servers** — a critical dependency for Workers, the Web UI, and the CLI.
- **Proxies between Workers and Temporal Cloud** — any forward proxy or mTLS terminator in the connection path between Workers / Starters / Clients → Namespace.
- **Datastores** — databases, queues, and any other systems that Activities read and write.

```mermaid
flowchart LR
    classDef app stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef endpoint stroke-width:1px;
    classDef env stroke-width:1.5px;
    subgraph WFENV["<b>Application environment</b>"]
      WFW["<b>Workers</b>"]:::app
      WFCL["<b>Workflow starters / Clients</b>"]:::app
      WFCODEC["<b>Codec Server</b>"]:::app
      WFDB[("<b>DB / queue</b>")]:::app
      WFPROXY["<b>Proxy</b>"]:::endpoint
    end
    WFNS["<b>Namespace</b><br/>(Temporal Cloud)"]:::ns
    WFW <--> WFCODEC
    WFW <--> WFDB
    WFW --> WFPROXY
    WFCL --> WFPROXY
    WFPROXY --> WFNS
    class WFENV env
```

## Highly available Worker patterns 

A Worker deployment pattern pairs with Namespace High Availability to achieve a disaster recovery (DR) and business continuity plan for your full Temporal architecture.

This page covers three main patterns: **Active/Passive**, **Active/Hot-Passive**, and **Active/Active**.

They trade off **recovery time** after an outage, **cost during normal operation**, and **operational complexity**. They are defined by where the Workers run and where Workflows process:

- **[Active/Passive](#active-cold)** — Workflows process in one region at a time, the "active" region. The other region is "passively" waiting, without any Workers. On a failover, the passive region becomes active, and new Workers are launched (from a "cold" start) to process Workflows.
- **[Active/Hot-Passive](#active-hot)** — Workflows process in one region at a time, the "active" region. However, Workers run in **both regions** simultaneously: processing Workflows in the "active" region, and on "hot" standby in the passive region. This achieves a faster failover and lower recovery time.
- **[Active/Active](#active-active)** — Workflows process in both/multiple regions at the same time, and Workers run in all regions at all times.

```mermaid
---
title: Active/Passive
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef empty stroke-width:1px,stroke-dasharray:4 3;
    classDef region stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph ICPRIM["<b>Primary</b>"]
      subgraph ICWP["Worker Pool"]
        ICW1["<b>Worker</b>"]:::worker
        ICW2["<b>Worker</b>"]:::worker
        ICW3["<b>Worker</b>"]:::worker
      end
      ICNS["<b>Namespace</b>"]:::ns
      ICWP <-->|Workflows| ICNS
    end
    subgraph ICSEC["<b>Secondary</b>"]
      ICR["<b>Replica</b>"]:::ns
      subgraph ICWP2["Worker Pool"]
        ICE["&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Empty&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]:::empty
      end
      ICR ~~~ ICWP2
    end
    ICNS --> ICR
    class ICPRIM,ICSEC region
    class ICWP,ICWP2 pool
```

```mermaid
---
title: Active/Hot-Passive
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef standby stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef region stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph IHPRIM["<b>Primary</b>"]
      subgraph IHWP["Worker Pool"]
        IHW1["<b>Worker</b><br/>Active"]:::worker
        IHW2["<b>Worker</b><br/>Active"]:::worker
        IHW3["<b>Worker</b><br/>Active"]:::worker
      end
      IHNS["<b>Namespace</b>"]:::ns
      IHWP <-->|Workflows| IHNS
    end
    subgraph IHSEC["<b>Secondary</b>"]
      IHR["<b>Replica</b>"]:::ns
      subgraph IHWP2["Worker Pool"]
        IHS1["<b>Worker</b><br/>Standby"]:::standby
        IHS2["<b>Worker</b><br/>Standby"]:::standby
        IHS3["<b>Worker</b><br/>Standby"]:::standby
      end
      IHR <-.-> IHWP2
    end
    IHNS --> IHR
    class IHPRIM,IHSEC region
    class IHWP,IHWP2 pool
```

```mermaid
---
title: Active/Active
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef region stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph IAPRIM["<b>Primary</b>"]
      subgraph IAWP["Worker Pool"]
        IAW1["<b>Worker</b><br/>Active"]:::worker
        IAW2["<b>Worker</b><br/>Active"]:::worker
      end
      IANS["<b>Namespace</b>"]:::ns
      IAWP <-->|Workflows| IANS
    end
    subgraph IASEC["<b>Secondary</b>"]
      IAR["<b>Replica</b>"]:::ns
      subgraph IAWP2["Worker Pool"]
        IAS1["<b>Worker</b><br/>Active"]:::worker
        IAS2["<b>Worker</b><br/>Active"]:::worker
      end
      IAR <-->|Workflows| IAWP2
    end
    IANS --> IAR
    class IAPRIM,IASEC region
    class IAWP,IAWP2 pool
```

> **ℹ️ Info:**
>
> **Namespaces always have a single active region, but can support an Active/Active Worker deployment pattern.**
>
> A Temporal Cloud Namespace with High Availability has exactly one active region at a time. The other region holds a replica that passively receives replicated state.
>
> However, since **Workers don't need to run in the same region as the active Namespace replica**, Temporal Cloud Namespaces can still fit into an Active/Active HA/DR strategy, as described below.
>

These patterns work across two cloud regions, which could be in the same cloud provider ("multi-region") or different cloud providers ("multi-cloud"):

- **Primary region** — the region where the Namespace is active during normal operation, also called the "preferred region."
- **Secondary region** — the region the Namespace fails over to. It can be any [Temporal Cloud region](/evaluate/cloud/regions) that supports replication from the primary region. Multi-region and multi-cloud architectures use the same Worker deployment patterns.

### Compare patterns at a glance 

| Pattern | Where Workers run | Best for and benefits | Major tradeoffs |
| --- | --- | --- | --- |
| **[Active/Passive](#active-cold)** | One region at a time | Easy initial deployment; acts like a single region with no special setup | Failing over Workers is your responsibility; highest recovery time of the three |
| **[Active/Hot-Passive](#active-hot)** | Both regions; secondary on warm standby | Low RTO with all task processing in the active region; fast Worker failover with no cold start | More configuration and the cost of a full standby fleet |
| **[Active/Active](#active-active)** | All regions, all processing Workflows | Low RTO with Workers active in every region; fast failover that uses fleet capacity instead of a standby fleet | Cross-region requests add Workflow latency; external systems need a cross-region consistency story |

## Active/Passive 

Workers run in only one region at a time. The secondary region stays empty until a failover, when you bring up a fresh Worker fleet there from a cold start.

```mermaid
---
title: Normal operation
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef empty stroke-width:1px,stroke-dasharray:4 3;
    classDef region stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph CPRIM["<b>Primary</b>"]
      subgraph CWP["Worker Pool"]
        CW1["<b>Worker</b>"]:::worker
        CW2["<b>Worker</b>"]:::worker
        CW3["<b>Worker</b>"]:::worker
      end
      CNS["<b>Namespace</b>"]:::ns
      CWP <-->|Workflows| CNS
    end
    subgraph CSEC["<b>Secondary</b>"]
      CR["<b>Replica</b>"]:::ns
      subgraph CWP2["Worker Pool"]
        CE["&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Empty&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]:::empty
      end
      CR ~~~ CWP2
    end
    CNS --> CR
    class CPRIM,CSEC region
    class CWP,CWP2 pool
```

```mermaid
---
title: After failover
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef down stroke-width:1px,stroke-dasharray:3 3;
    classDef region stroke-width:1.5px;
    classDef regiondown stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph FPRIM["<b>Primary</b> <span class='dp-outage-text'>(outage)</span>"]
      subgraph FPP["Worker Pool"]
        FPW1["Unavailable"]:::down
      end
      FPN["<b>Namespace</b>"]:::down
      FPP ~~~ FPN
    end
    subgraph FSEC["<b>Secondary</b>"]
      FSN["<b>Namespace</b><br>(Active)"]:::ns
      subgraph FSP["Worker Pool"]
        FSW1["<b>Worker</b><br/>Cold start"]:::worker
        FSW2["<b>Worker</b><br/>Cold start"]:::worker
        FSW3["<b>Worker</b><br/>Cold start"]:::worker
      end
      FSN <-->|Workflows| FSP
    end
    FPN -->|"Failover"| FSN
    class FPRIM regiondown
    class FSEC region
    class FPP,FSP pool
```

Here's how each component behaves during normal operation and after a failover:

| Component | Normal operation | On failover |
| --- | --- | --- |
| **Workers** | Run only in the primary region, processing all Workflows. No Workers run in the secondary region. | Brought up from nothing in the secondary region — a "cold" start. No Workflows progress until they're running. |
| **Namespace** | Active replica in the primary region; passive replica in the secondary region, continuously receiving replicated Workflow state. | Temporal Cloud promotes the secondary region's replica to active automatically. To trigger or test a failover yourself, see [Failovers](/cloud/high-availability/failovers). |
| **Workflow starters and Clients** | Run with the Workers in the primary region. | Brought up in the secondary region along with the Workers. |
| **Codec Servers and proxies** | Run alongside the active Workers in the primary region. | Scaled up in the secondary region as part of the failover. |
| **Databases and queues** | Replicate to the secondary region, if your Workflows depend on that data. | Promote the secondary region's copy to active, if needed, so the new Workers can read and write it. |

Setup is minimal: turn on Replication for your Namespace (see [Enable and manage High Availability](/cloud/high-availability/enable)) and enable replication on any databases or queues your Workflows use. At that point you're technically already running Active/Passive: the secondary region holds a ready replica, and failing over is a matter of bringing your Workers up there.

Recovery time is dominated by two things: how quickly you detect the outage, since Workflows make no progress until you respond (see [Detect a failover or an outage](/cloud/high-availability/monitoring#detect-failover-or-outage)), and how long the secondary-region Worker fleet takes to cold start — container or VM startup, image pulls, and application warm-up. The [Active/Hot-Passive](#active-hot) pattern removes both by keeping Workers already running and warm in the secondary region.

### Benefits 

Active/Passive offers the simplest operational model of the three patterns:

- **Easy to reason about.**
   - Only one region is active at a time, so traffic routing and interactions with systems (such as databases and queues) are simpler to understand, and the pattern pairs naturally with other active / passive systems. Active/Active, by contrast, requires deciding how Workers reach an active database: either a local active database in each region, or a single active / passive database that some Workers must reach cross-region.
- **Simple to operate.**
   - During normal operation it resembles a single-region deployment.
- **Lowest overall architecture cost.**
   - The size of the Worker fleet is simply the capacity needed to operate in one region. There are no standby Workers during steady state.

### Tradeoffs 

That simplicity comes at the cost of recovery time:

- Highest overall recovery time of the three patterns, due to cold starting the Worker fleet after failover.
- Depends on tested automation to bring up the secondary-region fleet quickly.

### Recommendations and important constraints 

Keep these in mind when setting up Active/Passive:

- **Failing over the Workers is the operator's responsibility.** The Namespace fails over automatically, but bringing up the Workers in the secondary region is up to you. Plan for these sub-considerations:
   - **How do you detect an outage and decide to fail over?** Define the failover conditions and the signals (alerts, health checks) that trigger them. Because Workflows make no progress until you detect the outage and respond, detection is on the critical path of your recovery time. To monitor for an outage and a failover, see [Detect a failover or an outage](/cloud/high-availability/monitoring#detect-failover-or-outage).
   - **How do you scale up the Workers?** Bring up the secondary-region fleet, ideally with tested automation, and scale down the primary region's fleet so Workers run in only one region at a time.
   - **Do you need to enforce single-region task processing?** This pattern relies on the operator to keep Workers in one region. To have Temporal enforce that Workflow and Activity Tasks are processed only in the active region, use the [Active/Hot-Passive](#active-hot) pattern.

```mermaid
flowchart LR
    FS1["Detect<br/>outage"]
    FS3["Failover<br/>DBs / queues"]
    FS4["Scale down<br/>Primary region<br/>Workers"]
    FS5["Scale up<br/>Secondary region<br/>Workers"]
    FS6["Confirm<br/>Workflows run<br/>normally"]
    FS1 --> FS3 --> FS4 --> FS5 --> FS6
```

- **Use the Namespace Endpoint.**
   - Connect Workers through the [Namespace Endpoint](/cloud/namespaces#access-namespaces), which always connects to the Namespace in its active region and automatically fails over to the new region.
   - **Rationale:** If a Temporal Cloud incident requires the Namespace to fail over while the rest of the primary region is healthy, the Workers in the primary region can still connect through the Namespace Endpoint and process Workflows. If the Workers use the Regional Endpoint for the primary region, they will not reliably connect to the Namespace during a Temporal Cloud incident in the primary region.

```mermaid
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef endpoint stroke-width:1px;
    classDef region stroke-width:1.5px;
    NEW["<b>Worker</b>"]:::worker
    NEEP["<b>Namespace Endpoint</b>"]:::endpoint
    NEW --> NEEP
    subgraph NEPRIM["<b>Primary</b>"]
      NEPNS["<b>Namespace</b>"]:::ns
    end
    subgraph NESEC["<b>Secondary</b>"]
      NESNS["<b>Namespace</b>"]:::ns
    end
    NEEP -->|"normal operation"| NEPNS
    NEEP -.->|"after failover"| NESNS
    class NEPRIM,NESEC region
```

- **Set up cross-region private connectivity.**
   - If you use private connectivity, give the primary region's Workers a network route to the VPC Endpoint in the other region, so they can reach the active replica after a Namespace-only failover. If you can't provide that cross-region route, use the [Active/Hot-Passive](#active-hot) pattern instead, where each region's Workers connect to their local replica.
   - For the full setup of Regional Endpoints, VPC Endpoints, and cross-region routing, see [Connectivity for High Availability](/cloud/high-availability/ha-connectivity).

```mermaid
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef endpoint stroke-width:1px;
    classDef region stroke-width:1.5px;
    subgraph CRCPRIM["<b>Primary Region</b>"]
      CRCPW["<b>Worker</b>"]:::worker
      CRCPEP["<b>VPC Endpoint</b>"]:::endpoint
      CRCPNS["<b>Namespace</b>"]:::ns
    end
    subgraph CRCSEC["<b>Secondary Region</b>"]
      CRCSEP["<b>VPC Endpoint</b>"]:::endpoint
      CRCSNS["<b>Replica</b>"]:::ns
    end
    CRCPW -->|"normal operation"| CRCPEP
    CRCPEP --> CRCPNS
    CRCSEP --> CRCSNS
    CRCPW -.->|"after a Namespace failover"| CRCSEP
    class CRCPRIM,CRCSEC region
```

- **Route Workers to the active region's Codec Server.** Two common approaches:
   - Put DNS or a load balancer in front of the Codec Server address, and update it on failover to point at the new region's instance.
   - Pass each Worker the Codec Server address for its own region as configuration, so a Worker always uses the service local to it. This is common in Kubernetes or with service discovery.
- **Route Workers to the active region's proxy.** Two common approaches:
   - Put DNS or a load balancer in front of the proxy address, and update it on failover to point at the new region's instance.
   - Pass each Worker the proxy address for its own region as configuration, so a Worker always uses the service local to it. This is common in Kubernetes or with service discovery.

## Active/Hot-Passive 

A full Worker fleet runs in both regions, but only the primary region's fleet processes Workflows. The secondary region's fleet stays warm on standby, ready to take over the moment a failover promotes it, with no cold start.

```mermaid
---
title: Normal operation
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef standby stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef region stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph HPRIM["<b>Primary</b>"]
      subgraph HWP["Worker Pool"]
        HW1["<b>Worker</b><br/>Active"]:::worker
        HW2["<b>Worker</b><br/>Active"]:::worker
        HW3["<b>Worker</b><br/>Active"]:::worker
      end
      HNS["<b>Namespace</b>"]:::ns
      HWP <-->|Workflows| HNS
    end
    subgraph HSEC["<b>Secondary</b>"]
      HR["<b>Replica</b>"]:::ns
      subgraph HWP2["Worker Pool"]
        HS1["<b>Worker</b><br/>Standby"]:::standby
        HS2["<b>Worker</b><br/>Standby"]:::standby
        HS3["<b>Worker</b><br/>Standby"]:::standby
      end
      HR <-.-> HWP2
    end
    HNS --> HR
    class HPRIM,HSEC region
    class HWP,HWP2 pool
```

```mermaid
---
title: After failover
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef down stroke-width:1px,stroke-dasharray:3 3;
    classDef region stroke-width:1.5px;
    classDef regiondown stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph HFPRIM["<b>Primary</b> <span class='dp-outage-text'>(outage)</span>"]
      subgraph HFPP["Worker Pool"]
        HFPW1["Unavailable"]:::down
      end
      HFPN["<b>Namespace</b>"]:::down
      HFPP ~~~ HFPN
    end
    subgraph HFSEC["<b>Secondary</b>"]
      HFSN["<b>Namespace</b><br>(Active)"]:::ns
      subgraph HFSP["Worker Pool"]
        HFSW1["<b>Worker</b><br/>Active"]:::worker
        HFSW2["<b>Worker</b><br/>Active"]:::worker
        HFSW3["<b>Worker</b><br/>Active"]:::worker
      end
      HFSN <-->|Workflows| HFSP
    end
    HFPN -->|"Failover"| HFSN
    class HFPRIM regiondown
    class HFSEC region
    class HFPP,HFSP pool
```

Here's how each component behaves during normal operation and after a failover:

| Component | Normal operation | On failover |
| --- | --- | --- |
| **Workers** | Run in both regions. The primary region's Workers are active and process all Workflows; the secondary region's Workers stay connected and warm on standby, doing no work. Forwarding is disabled for Worker polls, so the standby fleet adds no cross-region overhead. | The secondary region's standby Workers — already connected and warm — begin processing immediately. No cold start and no DNS wait. |
| **Namespace** | Active replica in the primary region; passive replica in the secondary region, continuously receiving replicated Workflow state. | The Namespace and Workers fail over together, automatically: Temporal Cloud promotes the secondary replica to active. |
| **Workflow starters and Clients** | Run in both regions alongside the Workers. Disabling forwarding does not apply to them: their requests, such as Start Workflow, Signal, and Query, are still forwarded from the secondary region to the active region and succeed. | No changes needed — already running in both regions. |
| **Codec Servers and proxies** | Run in both regions continuously, not just after a failover. | No changes needed — already running in the secondary region. |
| **Databases and queues** | Workers in each region typically read and write only their local, active copy. | Promote the secondary region's copy to active, if needed, so the now-active Workers can read and write it. |

Because a full Worker fleet is already running and warm in the secondary region, there's nothing to start or scale up before processing resumes. Along with Active/Active, this gives Active/Hot-Passive the lowest recovery time of the three patterns; because the standby fleet is already sized for full load, failover needs no scale-up.

### Benefits 

Active/Hot-Passive trades steady-state cost for a faster, more predictable failover:

- **Easy to reason about.**
   - Only one region is active at a time, so traffic routing and interactions with systems (such as databases and queues) are simpler to understand, and the pattern pairs naturally with other active / passive systems. Active/Active, by contrast, requires deciding how Workers reach an active database: either a local active database in each region, or a single active / passive database that some Workers must reach cross-region.
- **Lowest recovery time, tied with Active/Active.**
   - The secondary-region Workers are already connected and warm, so failover involves no cold start. Because the standby fleet is already sized for full load, it also needs no scale-up.
- **Low latency during normal operation.**
   - Workflow and Activity Tasks are processed only in the active region, with no cross-region forwarding of Worker polls. Client requests that originate in the secondary region are still forwarded, and pay the cross-region hop.

### Tradeoffs 

That speed comes at a steady-state cost:

- Highest overall architecture cost: a full standby Worker fleet runs in the secondary region at all times, even during steady state.

### Recommendations and important constraints 

Keep this in mind when setting up Active/Hot-Passive:

- **Use Regional or VPC Endpoints and disable forwarding.**
   - Connect each Worker fleet through its region's [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) (or VPC Endpoint) and [disable forwarding](/cloud/high-availability/enable#change-forwarding-behavior) for Worker polls. Using the Namespace Endpoint by mistake routes the standby Workers to the active region and defeats the pattern.
   - **Disabling forwarding applies to Worker polls only.** Client requests, such as Start Workflow, Signal, Query, Cancel, and Terminate, are always forwarded to the active region. A Workflow starter in the secondary region keeps working, and the Workflows it starts are processed by the active region's Workers. For the full list of APIs that keep forwarding, see [Client requests are forwarded regardless of this setting](/cloud/high-availability/enable#client-requests-still-forwarded).

```mermaid
flowchart LR
    classDef worker stroke-width:1px;
    classDef standby stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef endpoint stroke-width:1px;
    classDef region stroke-width:1.5px;
    subgraph HEPRIM["<b>Primary</b>"]
      HEPW["<b>Workers</b><br/>Active"]:::worker
      HEPEP["<b>Regional / VPC Endpoint</b>"]:::endpoint
      HEPNS["<b>Namespace</b>"]:::ns
      HEPW --> HEPEP
      HEPEP --> HEPNS
    end
    subgraph HESEC["<b>Secondary</b>"]
      HESW["<b>Workers</b><br/>Standby"]:::standby
      HESEP["<b>Regional / VPC Endpoint</b>"]:::endpoint
      HESNS["<b>Replica</b>"]:::ns
      HESW --> HESEP
      HESEP --> HESNS
    end
    HEPNS -. replicates .-> HESNS
    class HEPRIM,HESEC region
```

## Active/Active 

Worker fleets run in as many regions as you want, all processing Workflows against the Namespace's single active replica through the Namespace Endpoint. If one region goes down, the others keep processing without interruption.

```mermaid
---
title: Normal operation
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef region stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph APRIM["<b>Primary</b>"]
      subgraph AWP["Worker Pool"]
        AW1["<b>Worker</b><br/>Active"]:::worker
        AW2["<b>Worker</b><br/>Active"]:::worker
      end
      ANS["<b>Namespace</b>"]:::ns
      AWP <-->|Workflows| ANS
    end
    subgraph ASEC["<b>Secondary</b>"]
      AR["<b>Replica</b>"]:::ns
      subgraph AWP2["Worker Pool"]
        AS1["<b>Worker</b><br/>Active"]:::worker
        AS2["<b>Worker</b><br/>Active"]:::worker
      end
      AR <-->|Workflows| AWP2
    end
    ANS --> AR
    class APRIM,ASEC region
    class AWP,AWP2 pool
```

```mermaid
---
title: After failover
---
flowchart LR
    classDef worker stroke-width:1px;
    classDef workerhollow stroke-width:1px,stroke-dasharray:4 3;
    classDef ns stroke-width:1px;
    classDef down stroke-width:1px,stroke-dasharray:3 3;
    classDef region stroke-width:1.5px;
    classDef regiondown stroke-width:1.5px;
    classDef pool stroke-width:1px;
    subgraph AFPRIM["<b>Primary</b> <span class='dp-outage-text'>(outage)</span>"]
      subgraph AFPP["Worker Pool"]
        AFPW1["Unavailable"]:::down
      end
      AFPN["<b>Namespace</b>"]:::down
      AFPP ~~~ AFPN
    end
    subgraph AFSEC["<b>Secondary</b>"]
      AFSN["<b>Namespace</b><br>(Active)"]:::ns
      subgraph AFSP["Worker Pool"]
        AFSW1["<b>Worker</b><br/>Active"]:::worker
        AFSW2["<b>Worker</b><br/>Active"]:::worker
        AFSW3["<b>Worker</b><br/>Scaled up<br/>(as needed)"]:::workerhollow
      end
      AFSN <-->|Workflows| AFSP
    end
    AFPN -->|"Failover"| AFSN
    class AFPRIM regiondown
    class AFSEC region
    class AFPP,AFSP pool
```

Here's how each component behaves during normal operation and after a failover:

The automatic Namespace Endpoint failover described below applies to public connectivity and correctly configured AWS PrivateLink. GCP Private Service Connect does not support automatic DNS failover; see [GCP PSC guidance](/cloud/high-availability/ha-connectivity#single-cloud-ha-on-gcp-private-service-connect) for manual DNS or endpoint updates, or use [Active/Hot-Passive](#active-hot) so each Worker fleet connects to its local replica.

| Component | Normal operation | On failover |
| --- | --- | --- |
| **Workers** | Run in as many regions as you want — fleets don't have to match the Namespace's regions. Every fleet connects through the Namespace Endpoint. | Fleets in unaffected regions keep processing with no cold-start gap. Scale them up if needed to carry the full load. |
| **Namespace** | One active replica; every other region holds a passive replica that receives replicated state. The Namespace Endpoint always redirects via DNS to whichever region currently holds the active Namespace. | Temporal Cloud promotes a passive replica in another region to active. Every fleet follows automatically — no reconfiguration, nothing to bring up. |
| **Workflow starters and Clients** | Run wherever convenient and connect through the Namespace Endpoint, like the Workers. | Automatically follow the Namespace Endpoint to the new active region, like the Workers. |
| **Codec Servers and proxies** | Run in every region where Workers run. | Already running in every surviving region — no action needed. |
| **Databases and queues** | Accessed from every Worker region, so you need a cross-region consistency story. | Promote the active region's copy, if needed, so the Workers there can read and write it. |

Because surviving regions keep processing without a cold start, recovery time depends mainly on how quickly you can scale them up to absorb the extra load, not on starting new Workers.

### Benefits 

Active/Active spreads capacity across regions instead of parking it in a standby fleet:

- **Hands-off Worker failover.**
   - With public connectivity or correctly configured AWS PrivateLink, Workers in every region follow the Namespace Endpoint to the new active region automatically — there's no Worker failover step to run.
- **Lowest recovery time, no standby fleet.**
   - Surviving regions keep processing, so there's no cold start, giving the same low recovery time as Active/Hot-Passive while spreading capacity across regions instead of parking it in a dedicated standby fleet. Surviving regions may need to scale up to absorb the failed region's load.
- **Resilient to losing a region.**
   - Like spreading across Availability Zones, losing one region's fleet leaves the others running.

### Tradeoffs 

Running Workers active in multiple regions introduces new considerations:

- Workers outside the active region reach it across regions (directly or via forwarding), which adds latency that can matter for latency-sensitive Workflows.
- External systems are harder: Workers are active in multiple regions at once, so any databases and queues they touch need a cross-region consistency story.

### Recommendations and important constraints 

Keep these in mind when setting up Active/Active:

- **Default to the Namespace Endpoint.**
   - With public connectivity or correctly configured AWS PrivateLink, all fleets in any region can connect through the single Namespace Endpoint, which routes to the active region and follows failovers automatically. For GCP PSC, see the [manual-update or local-replica options](/cloud/high-availability/ha-connectivity#single-cloud-ha-on-gcp-private-service-connect).
- **Use a Regional Endpoint only when you need the lowest recovery time.**
   - Connecting each fleet to its region's [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) (or VPC Endpoint) removes the DNS step from the connection path, which can shave time off failover for the lowest possible RTO. The tradeoffs: more setup, and a real risk of misconfiguration (such as routing a fleet to the wrong region). Reach for it only when you absolutely need low recovery time. With Regional Endpoints, keep forwarding enabled so passive-region polls still reach the active replica.

```mermaid
flowchart LR
    classDef worker stroke-width:1px;
    classDef ns stroke-width:1px;
    classDef endpoint stroke-width:1px;
    classDef region stroke-width:1.5px;
    subgraph REPR1["<b>Region 1</b>"]
      REPW1["<b>Workers</b>"]:::worker
      REPE1["<b>Regional Endpoint</b>"]:::endpoint
      REPN1["<b>Namespace</b><br/>(Active)"]:::ns
      REPW1 --> REPE1
      REPE1 --> REPN1
    end
    subgraph REPR2["<b>Region 2</b>"]
      REPW2["<b>Workers</b>"]:::worker
      REPE2["<b>Regional Endpoint</b>"]:::endpoint
      REPN2["<b>Replica</b>"]:::ns
      REPW2 --> REPE2
      REPE2 --> REPN2
    end
    REPN2 -.->|"forwards polls"| REPN1
    class REPR1,REPR2 region
```

## Clients, Codec Servers, and databases 

The Worker deployment pattern sets the approach; the supporting pieces follow it.

- **Workflow starters and Clients.** Deploy these with the same regional pattern as the Workers, since a starter or Client often shares the same in-region dependencies (databases, queues, upstream services) and should fail over alongside them. With public connectivity or correctly configured AWS PrivateLink, point Clients at the Namespace Endpoint so they follow the active region automatically with no configuration change on failover. For GCP PSC, see the [manual-update or local-replica options](/cloud/high-availability/ha-connectivity#single-cloud-ha-on-gcp-private-service-connect). Use a [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) only when a Client must be pinned to a region.
- **Codec Servers and proxies.** Anything in the connection path between Workers and Temporal Cloud must be reachable from every region where Workers connect. In Active/Passive, scale them up in the secondary region as part of a failover; in the Active/Hot-Passive and Active/Active patterns, run them in both regions at all times.
- **Databases and queues.** These remain the application's responsibility, and the right approach depends on the Worker deployment pattern: a single-region-active datastore pairs naturally with the Active/Passive and Active/Hot-Passive patterns, while running Workers active in both regions (as in Active/Active) raises consistency questions that must be designed for. Detailed guidance is out of scope for this page.

## Frequently asked questions 

### What is the difference between active-passive and active-active patterns? 

Both **Active/Passive** and **Active/Hot-Passive** keep Workflows processing in one region at a time, with the other region standing by for failover. **Active/Active** runs Workers in every region and processes Workflows in all of them at once. See [Worker deployment patterns](#ha-worker-patterns) for the full comparison.

### How do I fail over Workers to another region? 

A Namespace with High Availability fails over automatically, but bringing up or activating Workers in the secondary region is your responsibility. The exact steps depend on your pattern; see [Active/Passive](#active-cold), [Active/Hot-Passive](#active-hot), and [Active/Active](#active-active).

### Which pattern has the lowest recovery time (RTO)? 

**Active/Hot-Passive** and **Active/Active** both achieve the lowest recovery time, because neither requires a cold start after failover. In Active/Hot-Passive, a warm standby Worker fleet in the secondary region begins processing the moment it becomes active. In Active/Active, Workers are already processing in every region, so the surviving regions keep running with no gap. By contrast, **Active/Passive** must cold start a Worker fleet in the secondary region, giving it the highest recovery time. See [Active/Hot-Passive](#active-hot), [Active/Active](#active-active), and [RPO and RTO](/cloud/rpo-rto).

### Do I have to run Workers in both regions for high availability? 

No. **Active/Passive** runs Workers in one region at a time and is the simplest starting point for disaster recovery. Running Workers in both regions — **Active/Hot-Passive** or **Active/Active** — lowers recovery time at higher cost.

### Does Temporal Cloud support Active/Active for HA/DR? 

Yes, as a Worker deployment pattern — not as a database-level active/active. A Temporal Cloud Namespace with High Availability always has exactly one active region and one passive replica underneath, no matter which [Worker deployment pattern](#ha-worker-patterns) you choose. But because Workers in any region reach the active replica through the Namespace Endpoint, you can run Worker fleets in every region and process Workflows in all of them at once. See [Active/Active](#active-active).

### What special patterns are needed for multi-cloud HA/DR? 

None specific to multi-cloud. Multi-region and multi-cloud HA/DR use the same [Worker deployment patterns](#ha-worker-patterns) — the secondary region can be any [Temporal Cloud region](/evaluate/cloud/regions) that supports replication from the primary, whether in the same cloud provider or a different one. The same considerations apply: route Workers through the right [endpoints and private connectivity](/cloud/high-availability/ha-connectivity), and give any databases and queues a cross-region — here, cross-cloud — consistency story.

If you use the Active/Passive pattern, it is highly recommended that you ensure Workers in the active cloud can reach the Namespace if it fails over to the passive cloud due to a Temporal-specific outage, so they can keep processing Tasks across the cross-cloud path.

### What special considerations are there for private networking? 

The patterns are the same. But if you use the Active/Passive pattern, it is highly recommended that you provide a network path for Workers in the active region to reach the VPC Endpoint in the passive region, so they can keep processing Tasks if the Namespace fails over due to a Temporal-specific outage. See [Connectivity for High Availability](/cloud/high-availability/ha-connectivity).

---

# Enable and manage High Availability

Source: https://docs.temporal.io/cloud/high-availability/enable

> Add a replica to a Namespace to enable High Availability, then manage forwarding behavior and automatic failover settings.

You can enable [High Availability](/cloud/high-availability) features for a new or existing Namespace by adding a
replica. When you add a replica, Temporal Cloud begins asynchronously replicating ongoing and existing Workflow
Executions.

Adding a replica fails the Namespace over automatically; to plan how your Workers fail over with it, see [Worker deployment patterns for high availability and disaster recovery](/cloud/high-availability/architecture-patterns).

 The replica region must be on the same continent as the primary region. Because of that, not all replication options are available in all Temporal Cloud regions. See the [Service regions](/evaluate/cloud/regions) page for the supported replica regions for each active region.

Using private network connectivity with a HA namespace requires extra setup. See
[Connectivity for HA](/cloud/high-availability/ha-connectivity).

There are charges associated with Replication and enabling High Availability features. For pricing details, visit
Temporal Cloud's [Pricing](/cloud/pricing) page.

> **⚠️ Caution:**
> GCP capacity restriction
>
> Before creating a GCP Namespace or adding a GCP replica, check [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction) for regional availability and exception requirements.
>

## Create a Namespace with High Availability features 

To create a new Namespace with High Availability features, you can use the Temporal Cloud UI or the command line
utility.

**Web UI**

    1. Visit Temporal Cloud in your Web browser.
    1. During Namespace creation, specify the primary [region](/evaluate/cloud/regions) for the Namespace.
    1. Select "Add a replica".
    1. Choose the [region](/evaluate/cloud/regions) for the replica.

    The web interface will present an estimated time for replication to complete.
    This time is based on your selection and the size and scale of the Workflows in your Namespace.

**Temporal CLI**

At the command line, enter:

```
temporal cloud namespace create \
   --name <namespace_id> \
   --region <primary_region> \
   --region <replica_region>
```

Specify the [region codes](/evaluate/cloud/regions) as arguments to the two `--region` flags.

**tcld**

At the command line, enter:

```
tcld namespace create \
   --namespace <namespace_id>.<account_id> \
   --region <primary_region> \
   --region <replica_region>
```

Specify the [region codes](/evaluate/cloud/regions) as arguments to the two `--region` flags.

If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before
`namespace create`.

Temporal Cloud sends an email alert to all Namespace Admins once your Namespace replica is ready for use.

## Add High Availability to an existing Namespace 

A replica can be added after a namespace has already been created.

**Web UI**

1. Visit Temporal Cloud Namespaces in your Web browser.
1. Navigate to the Namespace details page.
1. Select the “Add a replica” button.
1. Choose the [region](/evaluate/cloud/regions) for the replica.

The web interface will present an estimated time for replication to complete. This time is based on your selection and
the size and scale of the Workflows in your Namespace.

Temporal Cloud sends an email alert to all Namespace Admins once your Namespace replica is ready for use.

**Temporal CLI**

At the command line, enter:

```
temporal cloud namespace ha region add \
   --namespace <namespace_id>.<account_id> \
   --region <replica_region>
```

Specify the region ID (for example, `aws-us-east-1`) of the region where you want to create the replica as an argument to
the `--region` flag. See [Regions](/evaluate/cloud/regions) for available region names.

Temporal Cloud sends an email alert once your Namespace is ready for use.

**tcld**

At the command line, enter:

```
tcld namespace add-region \
   --namespace <namespace_id>.<account_id> \
   --region <replica_region>
```

Specify the region name (for example, `us-east-1`) of the region where you want to create the replica as an argument to the
`--region` flag. See [Regions](/evaluate/cloud/regions) for available region names.

If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before
`namespace add-region`.

Temporal Cloud sends an email alert once your Namespace is ready for use.

## Change a replica location 

Temporal Cloud can't change replica locations directly. To change a replica's location, you need to remove the replica
and add a new one.

> **⚠️ Caution:**
>
> We discourage changing the location of your replica for deployed applications, except under exceptional circumstances.
> If you remove your replica, you lose the availability guarantees of the Namespace, and it can take time to add another
> replica.
>
> If you remove a replica from a region, you must wait seven days before you can re-enable High Availability (HA) in that
> same location. During this period, you may add a replica to a different region, provided you have not had one
> there within the last seven days.
>

Follow these steps to change the replica location:

1. [Remove your replica](#disable). This disables High Availability for your Namespace.
2. [Add a new replica](#upgrade) to your Namespace.

You will receive an email alert once your Namespace is ready for use.

## Change the forwarding behavior 

Requests that reach the passive replica can be [forwarded](/cloud/high-availability/#request-forwarding) to the active region, and responses sent back to the Worker or Client. The `disablePassivePollerForwarding` Namespace setting controls this behavior for Worker poll traffic only.

With `disablePassivePollerForwarding` enabled, Worker polls that reach a passive replica are not forwarded, and these Workers do not execute Workflows or Activities. Workers connected to such a passive replica receive a `NamespaceNotActive` error on poll requests. These Workers stay connected and will start executing Workflows and Activities if the replica becomes active.

Same-region replicas are not affected by this setting.

To deploy Worker fleets in both regions that stay on standby in the passive region until failover, see [Active/Hot-Passive](/cloud/high-availability/architecture-patterns#active-hot).

Set `disablePassivePollerForwarding` through the [Cloud Ops API](/ops), the [cloud-api SDK](https://github.com/temporalio/cloud-sdk-go), or the [`temporal cloud` CLI extension](/cli/cloud), using one of the recipes in this section.

> **ℹ️ Info:**
>
> To see which endpoints route to which replica, see [How requests reach the replica](/cloud/high-availability/ha-connectivity#how-requests-reach-the-replica).
>

### Client requests are forwarded regardless of this setting 

`disablePassivePollerForwarding` stops Worker polls. It does not stop Client requests. The following APIs are forwarded from the passive replica to the active region whether or not the setting is enabled, with responses returned to the Client:

| API group | Forwarded operations |
| --- | --- |
| Workflow | Start Workflow, Signal-with-Start, Signal, Cancel, Terminate, Delete, Query |
| [Standalone Activity](/standalone-activity) | Start, Cancel, Terminate, Delete, Pause, Unpause, Reset, Update Activity Options |
| [Standalone Nexus Operation](/standalone-nexus-operation) | Start, Cancel, Terminate, Delete |

For the current list, see [`selectedAPIsForwardingRedirectionPolicyWhitelistedAPIs`](https://github.com/temporalio/temporal/blob/main/common/rpc/interceptor/dc_redirection_policy.go#L59-L71) in the Temporal Server source.

A Client that reaches the passive replica, for example through the passive region's Regional Endpoint, keeps working while the setting is enabled: it can start Workflows, send Signals, and run Queries successfully. Workers in the active region process the resulting Workflow and Activity Tasks, because the passive region's Workers receive none.

Plan for this if you disable forwarding to keep a region's traffic out of the active region. The setting stops that region's Workers from processing tasks. It does not stop Workflow starters, Signal senders, or other Clients in that region from reaching the Namespace.

### Set the forwarding behavior with the `temporal cloud` CLI 

Use the [`temporal cloud namespace ha update`](/cli/command-reference/cloud/namespace#ha-update) command:

```bash
temporal cloud namespace ha update \
    --namespace <namespace>.<account> \
    --passive-poller-forwarding disabled
```

Set the flag to `enabled` to re-enable forwarding.

### Set the forwarding behavior with the Cloud Ops API 

This recipe uses `curl` against the Cloud Ops API. Because `UpdateNamespace` replaces the entire Namespace spec, it fetches the current spec, merges in the new value, and posts it back with the current `resourceVersion`. The `jq` filter preserves any other High Availability fields you have set (such as `disableManagedFailover`).

Set the Cloud Ops API key and Namespace ID. The Namespace ID is in `<namespace>.<account>` format (the full identifier shown in the Cloud Web UI):

```bash
export TEMPORAL_CLOUD_OPS_API_KEY='<your-api-key>'
export NS='<namespace>.<account>'
```

Fetch the current spec:

```bash
curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \
  -H "Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY" > /tmp/ns.json
```

Build the update payload. Set the value to `true` to disable forwarding, or `false` to restore the default:

```bash
jq --arg rv "$(jq -r '.namespace.resourceVersion' /tmp/ns.json)" '{
  spec: (.namespace.spec
    | .highAvailability = ((.highAvailability // {}) + {disablePassivePollerForwarding: true})),
  resourceVersion: $rv
}' /tmp/ns.json > /tmp/ns-update.json
```

Post the update. The response contains an `asyncOperation` ID; the change is complete when `GET /cloud/operations/<id>` reports a terminal state.

```bash
curl -sS -X POST "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \
  -H "Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY" \
  -H "Content-Type: application/json" \
  -d @/tmp/ns-update.json
```

Verify the current value:

```bash
curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \
  -H "Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY" \
  | jq '.namespace.spec.highAvailability.disablePassivePollerForwarding'
```

A result of `null` means the field has never been set, which is equivalent to `false` — proto3 JSON omits default-`false` values from responses.

## Enable or disable automatic failovers 

When a Temporal Cloud Namespace has a replica in a different region or cloud, Temporal Cloud automatically fails over the Namespace to its replica in the event of an outage. _This is the recommended and default option._

If you prefer to disable automatic failovers and handle your own failovers, follow these instructions:

> **⚠️ Warning:**
> Disabling automatic failovers voids Temporal's RTO
>
> With automatic failovers disabled, Temporal Cloud cannot fail your Namespace over to its replica during an outage. You take responsibility for detecting outages and [triggering a failover](/cloud/high-availability/failovers/manage#trigger-failover) yourself. Temporal's [20-minute RTO](/cloud/rpo-rto) does not apply while this setting is disabled.
>

**Web UI**

1. Navigate to the Namespace detail page in Temporal Cloud.
1. Choose the "Disable Temporal-initiated failovers" option.

**Temporal CLI**

To disable automatic failovers, run the following command in your terminal:

```
temporal cloud namespace ha update \
    --namespace <namespace_id>.<account_id> \
    --auto-failover disabled
```

**tcld**

To disable automatic failovers, run the following command in your terminal:

```
tcld namespace update-high-availability \
    --namespace <namespace_id>.<account_id> \
    --disable-auto-failover=true
```

If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before
`namespace update-high-availability`.

To restore the default behavior, unselect the option in the Web UI, pass `--auto-failover enabled` with the Temporal CLI,
or pass `--disable-auto-failover=false` with `tcld`.

> **📝 Note:**
> Automatic failovers are always enabled for Same-region Replication
>
> This setting applies only to Multi-region and Multi-cloud Replication. You cannot disable automatic failovers for a [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace, because same-region failovers between cells are always managed by Temporal.
>

## Disable High Availability (remove a replica) 

To disable High Availability features on a Namespace, remove the replica from that Namespace. Removing a replica
disables all High Availability features:

- Discontinues replication of the Workflows in the Namespace.
- Disables the Namespace's ability to trigger a failover to a different region or cloud.
- Ends High Availability charges.

> **⚠️ Caution:**
>
> After removing a Namespace's replica, you cannot add a new replica to that same region for seven days.
> During that time, you can still add a replica to any other region.
>

### Prepare connectivity before removing a replica 

Before removing the replica, update any Workers and Clients that reach the Namespace through the region being removed:

- **Regional Endpoint:** Change the Worker and Client endpoint configuration to use the [Namespace Endpoint](/cloud/high-availability/ha-connectivity#namespace-endpoint-recommended) or the remaining region's Regional Endpoint. After deletion, the removed region's Regional Endpoint can no longer reach the Namespace or forward requests to the remaining region. Workers and Clients configured with the Namespace Endpoint do not need an endpoint configuration change, but private routing and DNS might still need updates as described below.
- **Private connectivity:** Move traffic using a PrivateLink service in the removed region to private connectivity in the remaining region. After deletion, the removed region's private connection can no longer reach the Namespace or forward requests. Check your own DNS and network routing to determine which AWS VPC Endpoints (or equivalent private endpoints on GCP or Azure) your Workers and Clients use. A useful clue is whether the Namespace is attached to any [Connectivity Rules](/cloud/connectivity#connectivity-rules) from the region being removed; this does not replace checking your routing. Update Worker and Client endpoint configurations if they target the removed region's private endpoint directly.
- **Private DNS:** If you followed the [HA private DNS setup](/cloud/high-availability/ha-connectivity) and override regional FQDNs such as `aws-us-east-1.region.tmprl.cloud`, change to the single-region private DNS setup for the remaining region. Follow the [AWS PrivateLink private DNS instructions](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink) or [GCP Private Service Connect private DNS instructions](/cloud/connectivity/gcp-connectivity#configuring-private-dns-for-gcp-private-service-connect).
  - Configure private DNS for the endpoint your Workers and Clients use: for the Namespace Endpoint, create an A record for each Namespace or a wildcard record for `*.<account>.tmprl.cloud`; for the Regional Endpoint, configure a single record for the remaining region's Regional Endpoint. Resolve these records to the remaining region's private endpoint. Both endpoint choices support mTLS and API key authentication. See [Update DNS or clients to use private connectivity](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity).
  - After HA is disabled, do not rely on the Namespace Endpoint continuing to resolve through the HA regional CNAME records.
  - Keep regional DNS records that other HA Namespaces still use.

Verify that Workers and Clients can connect through the remaining region before deleting the replica.

### Remove the replica 

Follow these steps to remove a replica from a Namespace:

**Web UI**

1. Navigate to the Namespace details page in Temporal Cloud
1. Select the option to "Remove Replica" on the "Region" card.

**Temporal CLI**

Run the following command to remove the replica:

```
temporal cloud namespace ha region delete \
    --namespace <namespace_id>.<account_id> \
    --region <replica_region_name>
```

See [Regions](/evaluate/cloud/regions) for available region names.

**tcld**

Run the following command to remove the replica:

```
tcld namespace delete-region \
    --api-key <api_key> \
    --namespace <namespace_id>.<account_id> \
    --region <replica_region_name>
```

See [Regions](/evaluate/cloud/regions) for available region names.

---

# Failovers

Source: https://docs.temporal.io/cloud/high-availability/failovers

> How automatic and manual failovers work with Temporal Cloud High Availability.

When a Namespace with [High Availability](/cloud/high-availability) is disrupted by an outage, Temporal Cloud can fail
over the Namespace from the primary to the replica. This lets in-flight Workflow Executions continue, new Workflow
Executions start, and closed Workflow Executions be inspected, all with minimal interruptions or data loss.

Returning control from the replica to the primary is called a failback. After an automatic
failover, Temporal automatically fails back to the original region once it is healthy, unless you
[opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). See
[Failbacks](/cloud/high-availability/failovers/manage#failbacks) for details.

## Automatic failover

Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. These
automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage
and complete a failover in minutes from when the outage began, according to the stated
[Recovery Time Objective (RTO)](/cloud/rpo-rto).

After an automatic failover, the Namespace will have a replica in its original region. Once the original region is
healthy again, Temporal Cloud automatically performs a [failback](/cloud/high-availability/failovers/manage#failbacks),
moving the Namespace back to its original region.

![On failover, the replica becomes active and the Namespace endpoint directs access to it.](https://docs.temporal.io/img/cloud/high-availability/failover.png)

To opt out of automatic failovers and their RTO, you can
[disable automatic failovers](/cloud/high-availability/enable#automatic-failovers).

### Conditions that trigger an automatic failover

While the failover operation itself usually completes in seconds, the bulk of the Recovery Time in an outage is spent
detecting the disruption and deciding to trigger a failover. See [The failover process](#failover-process) for a
detailed breakdown.

Temporal Cloud runs automated Workflows that detect outages and trigger failovers. These Workflows continuously monitor
the health of Temporal Cloud in every region and every cell.

If any of the monitored conditions are failing for too long, Temporal Cloud automatically triggers a failover on any
Namespaces with High Availability that have a healthy replica.

Temporal's on-call engineers may also trigger a failover at their discretion, for example, if they see early signs of a
regional outage.

> **ℹ️ Info:**
>
> The following list gives a general idea of the conditions that trigger an automatic failover. This is not an exhaustive
> list, and it may change over time.
>

- Whether Temporal Cloud's services in the cell are reachable from the Control Plane.
- The average latency of inbound RPC calls (excluding long-polling APIs) to Temporal services in the cell.
- The percentage of inbound RPC calls that returned errors related to server health.
- The average latency of calls from Temporal Cloud's services in the cell to its persistence layer.
- The percentage of calls to the persistence layer that returned errors related to persistence health.

## Manual failover

You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own
monitoring or for failover testing.

Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (that is,
a failover triggered by a user) is warranted are:

- **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover
  process with your Clients and Workers, or to move a Namespace to a different region.
- **An outage that affects only your systems.** If an outage is contained to your application, Workers, or other
  infrastructure, and Temporal Cloud is not affected, Temporal will not initiate a failover on your behalf. Detect the
  outage with your own monitoring and trigger a failover yourself.
- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a
  failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect,
  and the later one is a no-op. A manual failover does not conflict with Temporal's automatic failover.

> **📝 Note:**
> Same-region Replication
>
> Manual failovers apply only to Multi-region and Multi-cloud Replication. A
> [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between
> cells and cannot be failed over manually or have its automatic failovers disabled.
>

## The failover process 

The failover process is the same whether it is triggered automatically by Temporal or manually by a user.

1. **During normal operation**, the primary asynchronously replicates data to the replica, keeping them in sync.
2. **A failover is triggered.** For automatic failovers, the majority of time is spent on outage detection. Temporal's
   automated health checks must confirm the disruption before initiating a failover. For the overall timing target, see
   the [Recovery Time Objective (RTO)](/cloud/rpo-rto).
3. **The Namespace becomes active in the replica's region.**
   1. Temporal Cloud first attempts a _graceful failover_: it pauses traffic, drains in-flight replication, and switches
      to the replica with no data conflicts.
   2. If the graceful attempt does not complete within 10 seconds, Temporal Cloud falls back to a _forced failover_,
      which immediately activates the replica. In a forced failover, any events not yet replicated undergo
      [conflict resolution](#conflict-resolution) once the original region comes back.
   3. This hybrid strategy balances consistency and availability. During the switch, Workflow operations are briefly
      paused, and Temporal Cloud returns a retryable "Service unavailable" error to SDKs.

4. **The Namespace Endpoint redirects via DNS to the active region.** This change can take a few minutes to fully propagate
   to all Clients and Workers. If your application has an extremely demanding Recovery Time, you can eliminate this
   stage by connecting through a [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) instead
   of the Namespace Endpoint.
5. **Failback.** If the failover was triggered by Temporal, Temporal automatically triggers a failback to the original
   region once the region is healthy. If the failover was triggered by a user, the Namespace continues as-is until a
   user triggers another failover. See [failback options](/cloud/high-availability/failovers/manage#failbacks) for
   details.

## Post-failover events 

After any failover, whether triggered by you or by Temporal, an event appears in both the
[Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The
audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud
[notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs.

After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless
you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover,
the Namespace stays in the replica region until a user triggers another failover. See
[failback options](/cloud/high-availability/failovers/manage#failbacks) for details.

## Split-brain scenario

At any time, only the primary or the replica should be active. However, if a network partition separates the two
regions, the regions cannot communicate with each other. If you promote the replica to active during a network
partition, both regions will be active simultaneously, accepting writes independently. This is known as a split-brain
scenario.

When the network partition resolves and the regions can communicate again, Temporal's
[conflict resolution](#conflict-resolution) process reconciles the divergent histories and determines which region
remains active.

## Conflict resolution 

Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be
reflected in the replica due to replication lag, particularly during failovers. In the event of a
non-graceful failover, replication lag causes a temporary setback in Workflow progress. At the moment of non-graceful
failover:

- Operations that had already replicated remain durable in the replica.
- Operations that had not yet replicated (that is, that are still in the replication backlog) are reconciled when the
  region recovers, according to the conflict resolution process.

> **⚠️ Caution:**
> Conflict resolution requires a recoverable region
>
> Conflict resolution can only recover data from a functioning Temporal Service. If the previously active region never
> recovers, Workflow API calls that fall within the [RPO](/cloud/rpo-rto) — under one minute — may be permanently lost.
> Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history
> of Temporal Cloud.
>

In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching
regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero.

Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a
retry policy's [maximum attempts](/encyclopedia/retry-policies#maximum-attempts) is set to 0. High
Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be
re-dispatched in a newly active Namespace, leading to repeated executions.

The same durability boundary applies to Workflow starts, Signals, and Updates: a `StartWorkflowExecution`,
`SignalWorkflowExecution`, `SignalWithStartWorkflowExecution`, or `UpdateWorkflowExecution` call that returns success is
durably committed in the active region, and replicated asynchronously to the replica.

### How Workflow Id uniqueness is preserved after a forced failover

The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid#workflow-id) — at most one Open Workflow
Execution per Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across
a failover. This guarantee limits how many Executions are _Open_ at the same time; reuse of a Workflow Id after an
Execution Closes is governed separately by the
[Workflow Id Reuse Policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy), and a start request that
collides with an already-Open Execution is governed by the
[Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy). Because the guarantee
constrains only concurrency, and not how many [Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id
accumulates over its lifetime, conflict resolution can reconcile a divergence without ever running the same Workflow Id
twice concurrently.

1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event
   History to the replica.
2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently
   append events under the same Workflow Id. When the regions reconnect, their Event Histories have diverged for that
   Workflow Id.
3. **One Execution stays Open.** Temporal Cloud does not interleave the divergent histories. Events from the previously
   active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History
   into a new branch. Its conflict resolution process then keeps a single Workflow Execution
   Open. The competing Execution in the previously active region becomes a
   [zombie Workflow Execution](/temporal-service/multi-cluster-replication#zombie-workflows) — an Execution that region
   can no longer mutate on its own — and is terminated there once replication informs it of the competing Workflow Id.
   The Temporal Service ensures the resulting Event Histories remain valid and replayable by SDKs.

---

# Manage failovers

Source: https://docs.temporal.io/cloud/high-availability/failovers/manage

> Trigger, configure, and test failovers for Temporal Cloud High Availability Namespaces.

## Trigger a failover 

You can trigger a failover manually using the Temporal Cloud Web UI, the CLI, or the Cloud Ops API.

Manual failovers apply only to Multi-region and Multi-cloud Replication. A
[Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between
cells and cannot be failed over manually.

> **⚠️ Warning:**
> Check your replication lag
>
> Always check the replication lag before initiating a failover. A forced failover when there is a
> significant replication lag has a higher likelihood of rolling back Workflow progress.
>

**Web UI**

1. Visit the [Namespace page](https://cloud.temporal.io/namespaces) on the Temporal Cloud Web UI.
1. Navigate to your Namespace details page and select the **Trigger a failover** option from the menu.
1. Confirm your action. After confirmation, Temporal initiates the failover.

**Temporal CLI**

To manually trigger a failover, run the following command in your terminal:

```
temporal cloud namespace ha failover \
    --namespace <namespace_id>.<account_id> \
    --region <target_region>
```

The `<target_region>` must be the ID of a region (example: `aws-us-east-1`) where the Namespace has a replica that is
ready to be failed over to (replica state is `Activated`).

**tcld**

To manually trigger a failover, run the following command in your terminal:

```
tcld namespace failover \
    --namespace <namespace_id>.<account_id> \
    --region <target_region>
```

The `<target_region>` must be the name of a region (example: `us-east-1`) where the Namespace has a replica that is
ready to be failed over to (replica state is `Activated`).

If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before
`namespace failover`.

**Cloud Ops API**

You can trigger a failover programmatically using the [Cloud Ops API](/ops). The API is available via both HTTP and
gRPC.

**Using HTTP**

Send a POST request to the
[`FailoverNamespaceRegion`](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/{namespace}/failover-region)
endpoint:

```
POST https://saas-api.tmprl.cloud/cloud/namespaces/<namespace>/failover-region
```

Request body:

```json
{
  "region": "<target_region>",
  "asyncOperationId": "<optional_async_operation_id>"
}
```

- `region` (required): The [region code](/evaluate/cloud/regions) of the region to failover to. Must be a region where the
  Namespace has a replica in `Activated` replica state, indicating the replica is ready to be failed over to. Example:
  `aws-us-east-1`
- `asyncOperationId` (optional): A user-defined ID for tracking the async operation. If not set, the server will assign
  one.

**Using gRPC**

Use the
[`FailoverNamespaceRegion`](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CloudService.FailoverNamespaceRegion)
RPC with a
[`FailoverNamespaceRegionRequest`](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.FailoverNamespaceRegionRequest):

```protobuf
message FailoverNamespaceRegionRequest {
    // The namespace to failover.
    string namespace = 1;
    // The id of the region to failover to.
    // Must be a region that the namespace is currently available in.
    string region = 2;
    // The id to use for this async operation - optional.
    string async_operation_id = 3;
}
```

Both methods return a
[`FailoverNamespaceRegionResponse`](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.FailoverNamespaceRegionResponse)
containing an async operation that you can use to track the failover status.

> **ℹ️ Info:**
> Terraform not supported
>
> The [Temporal Cloud Terraform provider](https://registry.terraform.io/providers/temporalio/temporalcloud/latest) does
> not support triggering failovers. You must use the Web UI, the CLI, or the Cloud Ops API.
>

Once the failover async operation returns successfully, the Namespace will be failed over. Temporal manages retries for
the failover Workflow. In the rare event that an internal error prevents the failover from completing, the Temporal
on-call team is automatically paged to intervene and force the failover to completion.

## Return to the primary with failbacks 

Failback behavior depends on whether the failover was automatic or manually triggered.

### After an automatic failover 

After an automatic failover, Temporal Cloud automatically fails back to the original region once the region is
healthy. No action is required from you. Follow [Temporal's status page](https://status.temporal.io) for updates on the
original region's health.

If you prefer to manage failback yourself, you have two options:

- **Opt out of automatic failback (manage failback manually):** After the automatic failover has completed,
  [disable automatic failovers](/cloud/high-availability/enable#automatic-failovers) on the Namespace to prevent
  Temporal from automatically failing back. When you're ready to return to the original region,
  [trigger a failover](#trigger-failover) to that region and then re-enable automatic failovers.

- **Stay on the new region permanently ("fail forward"):** After the automatic failover has completed,
  [trigger a failover](#trigger-failover) to the region that is already active. This tells Temporal that you want to
  treat the new region as your primary for as long as it's healthy. Automatic failovers remain enabled,
  so Temporal will still protect you if the new region has an outage.

### After a user-triggered failover

If you triggered a failover yourself during an outage (instead of relying on an automatic failover), Temporal will
_not_ automatically fail back for you. You must [trigger a failover](#trigger-failover) back to the original region when
it is healthy. Monitor [Temporal's status page](https://status.temporal.io) for updates on region health.

Automatic failback is only available when the most recent failover was automatic.

### How to check whether your Namespace will be automatically failed back

If you are not sure whether your Namespace will be automatically failed back, check the list of failovers in the
Temporal Cloud Web UI on your Namespace's detail page:

- If the most recent failover was **automatic**, then Temporal will fail the Namespace back when
  the original region is healthy.
- If the most recent failover was **user-triggered**, then the Namespace will _not_ be automatically failed back. You
  must trigger the failback yourself.

## Workers and failovers 

Enabling High Availability for Namespaces does not require specific Worker configuration. When a Namespace fails over to
the replica, the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the
Namespace without interruption. Temporal Cloud forwards their requests from the passive replica to the active region and
the responses back, so Workers keep running through a failover.

To choose where your Worker fleets run across regions, see [Deployment patterns for High Availability](/cloud/high-availability/architecture-patterns).

To route Workers to the passive region's replica, see [How requests reach the replica](/cloud/high-availability/ha-connectivity#how-requests-reach-the-replica).

To stop forwarding Worker polls to the active region, see [Change the forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior).

To disable automatic failovers, see [Enable or disable automatic failovers](/cloud/high-availability/enable#automatic-failovers).

When a Namespace fails over to a replica in a different region, Workers will be communicating cross-region.

- If your application cannot tolerate this latency, deploy a second set of Workers in the replica's region or opt for a
  replica in the same region.
- In the case of a complete regional outage, Workers in the original region may fail alongside the original Namespace.
  To keep Workflows moving during this level of outage, deploy a second set of Workers to the secondary region.

Temporal Cloud enforces a maximum connection lifetime of 5 minutes, which gives your Workers an opportunity to
re-resolve the DNS.

## Test failovers 

Temporal recommends regular failover testing for mission-critical applications in production. By testing in
non-emergency conditions, you verify that your application continues to function even when parts of the infrastructure
fail.

Because failover testing relies on manually triggering a failover, it applies to Multi-region and Multi-cloud
Replication. A [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over
automatically between cells and cannot be failed over manually for testing.

> **💡 Tip:**
>
> If this is your first time performing a failover test, run it with a test-specific Namespace and application. Practice
> runs help ensure the process runs smoothly during real incidents in production.
>

Failover testing (also known as "trigger testing") can:

- **Validate replicated deployments:** In multi-region setups, failover testing ensures your application can run from
  another region when the primary region experiences outages.

- **Assess replication lag:** In multi-region deployments, monitoring
  [replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) between regions
  is important. Check the lag before initiating a failover to avoid rolling back Workflow progress.

- **Assess recovery time:** Manual testing helps you measure actual recovery time and check if it meets your expected
  [Recovery Time Objective (RTO)](/cloud/rpo-rto).

- **Identify potential issues:** Failover testing uncovers problems not visible during normal operation, including
  issues like
  [backlogs and capacity planning](https://temporal.io/blog/workers-in-production#testing-failure-paths-2438) and how
  external dependencies behave during a failover event.

- **Operational readiness:** Regular testing familiarizes your team with the failover process, improving their ability
  to handle real incidents.

---

# Connectivity for High Availability

Source: https://docs.temporal.io/cloud/high-availability/ha-connectivity

> How to choose endpoints and configure private network connectivity for Namespaces with Temporal Cloud High Availability features.

A Namespace with High Availability features spans two regions, and the endpoint your Workers and Clients connect through determines how they behave before, during, and after a failover.
This page covers:

- How to choose between the Namespace Endpoint and a Regional Endpoint for a Namespace with High Availability features.
- How to configure PrivateLink so that failover remains transparent to Workers on private networks.

## How to choose an endpoint for a Namespace with High Availability features

Temporal Cloud exposes two kinds of gRPC endpoints for a Namespace.
See [How to access a Namespace](/cloud/namespaces#access-namespaces) for the general definitions; this section focuses on how each behaves with replication and failover.

### Namespace Endpoint (recommended)

Format: `<namespace>.<account>.tmprl.cloud:7233`

The Namespace Endpoint always connects to whichever region is currently active.
Under the hood, it is a CNAME that points at the active region's Regional Endpoint.
When Temporal Cloud fails the Namespace over, it updates the CNAME to point at the new active region.
The DNS <a href="https://en.wikipedia.org/wiki/Time_to_live">TTL</a> is 15 seconds, so Clients converge within about 30 seconds with no configuration change on your side.
For private connections, this behavior depends on your DNS configuration and provider; [GCP Private Service Connect does not support automatic failover through Temporal Cloud DNS](#single-cloud-ha-on-gcp-private-service-connect).

Use the Namespace Endpoint unless you have a specific reason to pin traffic to a region.

### Regional Endpoint

Format: `<cloud>-<region>.region.tmprl.cloud:7233` (for example, `aws-us-west-2.region.tmprl.cloud` or `gcp-us-central1.region.tmprl.cloud`).
See [regions](/evaluate/cloud/regions) for the full list.

A Regional Endpoint is shared across every Namespace that is active or replicated in that region.
Unlike the Namespace Endpoint, a Regional Endpoint stays pinned to the region in its name — if that region holds the passive replica of your Namespace, the Regional Endpoint connects to the passive replica.

Use a Regional Endpoint only when you need explicit control over which replica a Client or Worker reaches.

Trade-offs to consider:

- **Faster recovery.** A Worker connecting through a Regional Endpoint skips the DNS step that Clients on the Namespace Endpoint wait for during a failover. This removes the ~30-second DNS convergence window from the recovery path, which is useful for Workloads that must minimize [Recovery Time](/cloud/rpo-rto) at all costs.
- **You are responsible for regional coverage.** A Worker using the Regional Endpoint of a region cannot reach the Namespace if that region is in an outage. To stay available through a failover, you must run Workers that use the **replica** region's Regional Endpoint — Workers pointed only at the outage region's Regional Endpoint will not reconnect automatically.

When authenticating with mTLS, set the Client's `server_name` / `serverNameOverride` config equal to the Namespace Endpoint.
This overrides the SNI that the Client will expect during the TLS Handshake with Temporal Cloud.
The Regional Endpoint forwards the request to your Namespace, so the Client must expect the Namespace's certificate during the TLS handshake.

For example, in TypeScript, the Client's config would be set like this:

```typescript
await Connection.connect({
  address: 'aws-us-east-1.region.tmprl.cloud:7233',
  tls: {
    serverNameOverride: 'my-namespace.my-account.tmprl.cloud',
    clientCertPair: { crt: clientCert, key: clientKey },
  },
  ...
});
```

### How endpoints route on failover

Consider a Namespace replicated across `us-east-1` (initially active) and `us-west-2` (initially the replica), with a failover that swaps the two.

| Client connects via                         | Before failover | After failover          |
| ------------------------------------------- | --------------- | ----------------------- |
| Namespace Endpoint                          | `us-east-1`     | `us-west-2` (automatic) |
| Regional Endpoint `aws-us-east-1.region...` | `us-east-1`     | `us-east-1`             |
| Regional Endpoint `aws-us-west-2.region...` | `us-west-2`     | `us-west-2`             |

The Namespace Endpoint moves with the active region via an updated CNAME — no Client changes required.
The Regional Endpoints do not change their targets on failover: each continues to route to the replica that lives in its region.

## How requests reach the replica 

A request can reach the passive replica in three ways:

- **Through the passive region's Regional Endpoint.** A [Regional Endpoint](#regional-endpoint) is pinned to its region, so the Regional Endpoint of the region that currently holds the passive replica connects to the passive replica.
- **Through a PrivateLink or Private Service Connect endpoint in the passive region.** A VPC Endpoint or PSC endpoint in the passive region routes to the passive replica.
- **Through the Namespace Endpoint during a failover.** When a Namespace fails over, two things happen in parallel:
  1. The replica becomes active, and the former active becomes a replica.
  2. The Namespace Endpoint changes to point at the replica's region, and the Worker re-resolves the Namespace Endpoint to connect to the new active region via DNS.

  If #1 completes before #2, a Worker that was connected to the former active before the failover will stay connected to it even after it becomes the replica. The Worker will change to point at the new active when the DNS changes propagate and the Client re-resolves DNS (typically 30 seconds, though up to 5 minutes, bounded by Temporal Cloud's maximum connection lifetime).

By default, Temporal Cloud transparently forwards any request that reaches the passive replica to the active region, and the response back.
You can turn forwarding off for Worker polls, but Client requests such as Start Workflow, Signal, and Query are always forwarded.

To learn what forwarding does, see [Request forwarding](/cloud/high-availability/#request-forwarding).

To stop forwarding Worker polls on a Namespace, see [Change the forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior).

To run Worker fleets in both regions that rely on this forwarding, see [Active/Active](/cloud/high-availability/architecture-patterns#active-active).

To keep passive-region Workers on standby until failover by disabling this forwarding, see [Active/Hot-Passive](/cloud/high-availability/architecture-patterns#active-hot).

## How to use PrivateLink with High Availability features

> **💡 Tip:**
>
> Proper networking configuration is required for failover to be transparent to Clients and Workers when using PrivateLink.
> This section describes how to configure routing for Namespaces with High Availability features on AWS PrivateLink.
>

> **⚠️ Warning:**
> GCP PSC does not support automatic DNS failover
>
> The regional DNS override pattern below applies to AWS PrivateLink only.
> GCP Cloud DNS cannot combine public and private DNS resolution in the way this pattern requires, so it does not provide automatic failover with GCP Private Service Connect, including for GCP-only HA.
> See [GCP PSC failover options](#single-cloud-ha-on-gcp-private-service-connect) and [multi-cloud connectivity options](#multi-cloud-ha-aws-privatelink--gcp-private-service-connect).
>

These instructions assume you already have the AWS private connections in place. If not, follow the [AWS PrivateLink](/cloud/connectivity/aws-connectivity) creation guide first.

## How HA + private connectivity works

A Namespace with High Availability features has two replicas — a primary and a secondary, in different regions or different cloud providers. At any moment, one is **active** and one is **passive**. On failover, Temporal Cloud changes the active replica.

Temporal Cloud expresses the active replica through DNS:

- The Namespace DNS record (`<ns>.<account>.tmprl.cloud`) is a CNAME.
- It points to the active region's regional record (`<provider>-<region>.region.tmprl.cloud`).
- On failover, Temporal Cloud rewrites the CNAME target.

Namespace DNS records have a 15-second TTL. Clients should converge to the new region within roughly 30 seconds (about twice the TTL) once their resolver cache expires.

> **📝 Note:**
> Deterministic DNS behavior is unique to HA Namespace Endpoints
>
> This is the **only** place in Temporal Cloud where you can depend on Temporal-managed DNS to behave in a specific, deterministic way — a Namespace Endpoint on an HA Namespace CNAMEing to its active region's regional record. Everywhere else, take a dependency on Temporal's published endpoints (the hostnames themselves), not on what they resolve to. The underlying IP addresses, CNAME chains, and resolution behavior of non-HA endpoints can change at any time without notice.
>

For AWS PrivateLink, your job is to make sure that:

- Override the Regional Endpoint's DNS zone to resolve to a VPC Endpoint.
- Ensure network connectivity between the two regions.

> **⚠️ Warning:**
> Do not override the Namespace Endpoint in your private hosted zone
>
> For HA Namespaces, the PHZ must override only the regional records (`<provider>-<region>.region.tmprl.cloud`) — never the Namespace Endpoint itself (`<ns>.<account>.tmprl.cloud`).
>
> If the PHZ holds a record for the Namespace Endpoint, the resolver answers from the PHZ before consulting public DNS, so Temporal Cloud's active-region CNAME is never followed. On failover, Workers keep resolving to the old (now passive) region's VPC Endpoint and never reach the new active region.
>
> This matters most when **enabling HA on a Namespace that previously used the [single-region PHZ pattern](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink)**, where the Namespace Endpoint itself was the overridden name. See [How to enable HA on a Namespace using Private Connectivity](#how-to-enable-ha-on-a-namespace-using-private-connectivity) below for the migration steps.
>

> **⚠️ Warning:**
> Do not attach a Stable IPs public Connectivity Rule
>
> If you attach a public [Connectivity Rule with Stable IPs](/cloud/connectivity/ip-addresses#stable-ip-addresses) to a Namespace, the Namespace Endpoint resolves to a public Stable IP instead of to `<provider>-<region>.region.tmprl.cloud`. Stable IPs DNS behavior supersedes the regional DNS behavior described here, so the Namespace Endpoint's DNS resolution will not work in the way the Private Hosted Zone needs. To keep HA + Private Connectivity working, do not attach a Stable IPs public Connectivity Rule to that Namespace.
>

## How to enable HA on a Namespace using Private Connectivity: changing private DNS overrides from single-region to multi-region 

These steps apply to **AWS PrivateLink only**. For GCP Private Service Connect, use the [GCP PSC failover options](#single-cloud-ha-on-gcp-private-service-connect) instead; regional DNS overrides do not enable automatic failover.

If you are turning on [High Availability features](/cloud/high-availability/enable) on a Namespace that already uses AWS PrivateLink, the existing private DNS setup almost certainly needs to change before DNS-based failover will work.

The common single-region private DNS pattern (described in the [AWS PrivateLink guide](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink)) overrides the **Namespace Endpoint** directly. That pattern short-circuits Temporal Cloud's regional CNAME and prevents DNS-based failover from working — see the warning above.

Follow these steps in order to update your private DNS overrides without interrupting traffic:

1. **Inventory the existing PHZ records.** List the records in your private hosted zone for `tmprl.cloud`. Note any CNAME (or A record) for `<ns>.<account>.tmprl.cloud` — that is the single-region override you'll be removing.

2. **Add regional records for both the source and target HA regions.** Before removing anything, create:
   - `aws-<source-region>.region.tmprl.cloud` → source-region VPC Endpoint
   - `aws-<target-region>.region.tmprl.cloud` → target-region VPC Endpoint

   These records are additive and do not yet affect resolution of `<ns>.<account>.tmprl.cloud`, because the Namespace-endpoint override still short-circuits the chain.

3. **Confirm both VPC Endpoints are reachable from your Worker VPCs.** From a Worker host, `dig` the new regional names and confirm they resolve to the right VPC Endpoint. Also verify the network path actually works in both regions (security groups, route tables, cross-region connectivity).

4. **Enable HA on the Namespace.** Follow [Enable High Availability features](/cloud/high-availability/enable). Temporal Cloud creates the replica and starts replicating.

5. **Remove the Namespace-endpoint PHZ record.** Delete the `<ns>.<account>.tmprl.cloud` record from the PHZ. With it gone, Workers resolve the name through public DNS → regional CNAME → PHZ regional override → VPC Endpoint, which is the correct HA chain. **Do not skip this step.** If the Namespace-endpoint override remains, failover does not work.

6. **Test failover end-to-end.** Trigger a [manual failover](/cloud/high-availability/failovers#manual-failover) in a staging environment to confirm Workers converge to the new active region within the expected window (about 30 seconds after the public CNAME update, given the 15-second TTL).

> **⚠️ Caution:**
> Order matters
>
> Adding the regional records first (step 2) and removing the Namespace-endpoint record last (step 5) means Workers always have a working DNS resolution. Reversing the order leaves a window where Workers cannot resolve the Namespace Endpoint at all.
>

To reverse this change when you disable HA, [change private DNS back to the single-region setup](/cloud/high-availability/enable#prepare-connectivity-before-removing-a-replica) before removing the replica.

## How to migrate to another Temporal Cloud Region when using Private Connectivity 

To move a Namespace to a new Temporal Cloud Region while keeping Private Connectivity in place, the recommended pattern is to use a **separate private hosted zone in each region**, each overriding the Namespace Endpoint to point at that region's own VPC Endpoint. Because a PHZ is scoped only to the VPCs it is associated with, Workers in each region resolve the Namespace Endpoint to their local VPC Endpoint and traffic stays in-region throughout the move.

This is not the only way to handle a region change with Private Connectivity, but it is the most commonly used.

> **⚠️ Caution:**
> GCP capacity restriction
>
> Before creating a GCP Namespace or adding a GCP replica, check [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction) for regional availability and exception requirements.
>

Steps:

1. **Keep the existing Namespace Endpoint PHZ override in the original region.** Do not change anything in the original region's private DNS setup yet.

2. **Create a new VPC Endpoint in the new region.** Follow the [AWS PrivateLink](/cloud/connectivity/aws-connectivity) (or [GCP PSC](/cloud/connectivity/gcp-connectivity)) creation steps in the new region's VPC.

3. **In the new region, create a PHZ that overrides the Namespace Endpoint to point at the new VPC Endpoint.** Use the [single-region PHZ pattern](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink), scoped only to the new region's Worker VPCs. With one PHZ per region, traffic in each region routes through that region's own VPC Endpoint.

4. **Add a replica in the new region.** Follow [Enable High Availability features](/cloud/high-availability/enable). The new region comes up as a passive replica.

5. **Start Workers in the new region.** They begin processing tasks immediately, even though they are connecting to the passive replica, because Temporal Cloud forwards Workflow and Activity tasks across regions transparently. This is what keeps the region change zero-downtime.

6. **Failover to the new region.** Trigger a [manual failover](/cloud/high-availability/failovers#manual-failover) to make the new region active. Workers in the old region keep running and keep using the old VPC Endpoint — they now connect to the passive replica that lives in the old region, and Temporal Cloud forwards their tasks to the new active region.

7. **Drain and remove Workers and the VPC Endpoint in the old region.** Once you are confident the new region is handling all new work and you no longer need old-region Workers, stop them. Complete the [replica-removal connectivity checklist](/cloud/high-availability/enable#prepare-connectivity-before-removing-a-replica) to confirm no other Workers or Clients still depend on the old region's Regional Endpoint or private connection, then tear down the old region's VPC Endpoint and PHZ.

8. **Remove the replica in the old region.** See [Migrate between regions or cloud providers](/cloud/migrate/between-regions-or-cloud-providers) for the replica-removal step. The Namespace is now single-region in the new location, and the HA pricing surcharge no longer applies.

> **📝 Note:**
> Per-region PHZ versus shared regional-record PHZ
>
> This pattern is **different** from the long-term HA setup described in [Single-cloud HA on AWS PrivateLink](#single-cloud-ha-on-aws-privatelink), which uses one shared PHZ holding regional records (`aws-<region>.region.tmprl.cloud`) and relies on DNS-based failover to switch Workers between regions. The region-change pattern instead uses one PHZ per region, each overriding the Namespace Endpoint itself, and relies on Temporal Cloud's cross-region task forwarding rather than DNS to keep Workers productive across the cutover.
>

## Single-cloud HA on AWS PrivateLink

### How Namespace DNS records work with PrivateLink

When using PrivateLink, you connect to Temporal Cloud through a VPC Endpoint, which uses addresses local to your network.
Temporal treats each `region.tmprl.cloud` zone as a separate zone, so you override resolution per region — this routes traffic to your VPC Endpoint internally for the regions you're using.

A Namespace's active region is reflected in the target of the Namespace Endpoint's CNAME record.
For example, if the active region of a Namespace is AWS us-east-1, the DNS configuration would look like this:

| Record name                         | Record type | Value                              |
| ----------------------------------- | ----------- | ---------------------------------- |
| ha-namespace.account-id.tmprl.cloud | CNAME       | aws-us-east-1.region.tmprl.cloud   |

After a failover, the CNAME record is updated to point to the failover region, for example:

| Record name                         | Record type | Value                              |
| ----------------------------------- | ----------- | ---------------------------------- |
| ha-namespace.account-id.tmprl.cloud | CNAME       | aws-us-west-2.region.tmprl.cloud   |

The Temporal domain did not change, but the CNAME updated from us-east-1 to us-west-2.

![Customer side solution example](https://docs.temporal.io/img/cloud/high-availability/private-link.png)

### How to set up the DNS override

In AWS, use a Route 53 private hosted zone for `region.tmprl.cloud` to override resolution per region:

| Record name                          | Record type | Value (your VPC Endpoint DNS)                                |
| ------------------------------------ | ----------- | ------------------------------------------------------------ |
| `aws-us-west-2.region.tmprl.cloud`   | CNAME       | `vpce-...-us-west-2.vpce.amazonaws.com`                      |
| `aws-us-east-1.region.tmprl.cloud`   | CNAME       | `vpce-...-us-east-1.vpce.amazonaws.com`                      |

Link the private zone to every VPC where Workers run.

When your Workers connect to the Namespace, they first resolve `<ns>.<account>.tmprl.cloud`, which CNAMEs to `aws-<active-region>.region.tmprl.cloud`, which then resolves to your local VPC Endpoint.

You also need to decide how Workers reach whichever region becomes active. Either:

- Run Workers in **both** regions continuously (recommended), or
- Establish cross-region connectivity (Transit Gateway, VPC Peering) so Workers in one region can reach the VPC Endpoint in the other.

## Single-cloud HA on GCP Private Service Connect

Automatic failover through Temporal Cloud DNS is not supported with GCP Private Service Connect.
GCP Cloud DNS cannot combine the public Namespace CNAME with private regional DNS overrides in the way the AWS pattern requires.
Creating a PSC endpoint and a regional DNS override in each GCP region does not make Workers or Clients follow the active region automatically.

To reconnect through PSC after failover, manually update your private DNS override or Worker and Client endpoint configuration to use the active region's PSC endpoint.
Create a PSC endpoint in each replica region and ensure your Workers and Clients can reach the endpoint they will use after failover.
See the [GCP PSC limitations and setup instructions](/cloud/connectivity/gcp-connectivity).

A Connectivity Rule is required for each PSC connection — see [GCP PSC setup](/cloud/connectivity/gcp-connectivity) and [Connectivity Rules](/cloud/connectivity#connectivity-rules).

## Multi-cloud HA (AWS PrivateLink + GCP Private Service Connect)

If your replicas span clouds — for example, AWS `us-east-1` (active) and GCP `us-east4` (passive) — your Workers need a way to reach the active replica regardless of which cloud it's in.
The GCP PSC DNS limitation above also applies to multi-cloud HA. Regional DNS overrides in GCP do not provide automatic failover to an AWS replica.

For GCP-resident Workers and Clients that use PSC, plan how they will reach an AWS replica:

- **Use public connectivity after failover.** Remove the private DNS override that directs the Namespace Endpoint to PSC so the endpoint resolves through public DNS. Allow public egress from your Workers and Clients, and attach a [public Connectivity Rule](/cloud/connectivity#connectivity-rules) to the Namespace. If you configured Clients to use the PSC IP address directly, change them to use the Namespace Endpoint as well. To avoid this manual connectivity change during failover, use public connectivity in both clouds from the start.
- **Provide your own private network path.** Temporal Cloud does not provide a GCP PSC endpoint for an AWS-hosted replica. To reach it privately from GCP, provide network connectivity to an AWS VPC with an AWS PrivateLink endpoint, and configure DNS or Clients to use that endpoint. Private network connectivity alone does not remove the GCP DNS limitation.
- **Run Worker fleets in both clouds.** Configure each fleet to reach its local replica through the appropriate private endpoint. See [Worker architecture patterns](/cloud/high-availability/architecture-patterns) for forwarding and failover behavior. Clients also need a working connection to a replica.

> **⚠️ Caution:**
> Alpine/musl + GCP PSC: missing AAAA records can break Workers
>
> GCP Private Service Connect endpoints return only A (IPv4) records — there is no AAAA (IPv6) record. Most Linux distributions handle a missing AAAA gracefully, but **Alpine Linux's musl resolver returns a SERVFAIL** when AAAA is missing, which can cause Temporal SDK clients to fail name resolution after a failover from AWS to GCP.
>
> If you run Workers on Alpine and use multi-cloud HA, either:
>
> - Switch the Worker base image to a glibc-based distribution (Debian, Ubuntu, distroless), or
> - Configure your application/runtime to disable AAAA lookups (for example, set `GODEBUG=netdns=go+v4` for Go, or prefer IPv4 in the Java/Node/Python runtimes you use).
>

### Available regions, PrivateLink endpoints, and DNS record overrides

> **⚠️ Caution:**
>
> The `sa-east-1` region is not yet available for use with Multi-region Namespaces. Currently, it is the only region on the continent.
>

The cloud-specific connectivity pages list the available Temporal regions and the DNS record overrides used for High Availability and private connectivity.
GCP regional DNS record overrides do not enable automatic failover with PSC.

### AWS regions and PrivateLink endpoints

See [Available AWS regions, PrivateLink endpoints, and DNS record overrides](/cloud/connectivity/aws-connectivity#available-aws-regions-privatelink-endpoints-and-dns-record-overrides).

### GCP regions and Private Service Connect endpoints

See [Available GCP regions, PSC endpoints, and DNS record overrides](/cloud/connectivity/gcp-connectivity#available-gcp-regions-psc-endpoints-and-dns-record-overrides).

When using a Namespace with High Availability features, the Namespace's DNS record `<ns>.<account>.tmprl.cloud` points to a regional DNS record in the format `<provider>-<region>.region.tmprl.cloud`, where `<provider>-<region>` is the currently active region for your Namespace.

During failover, Temporal Cloud changes the target of the Namespace DNS record from one region to another. Namespace DNS records are configured with a 15-second <a href="https://en.wikipedia.org/wiki/Time_to_live">TTL</a>. Any DNS cache should re-resolve the record within this time. For public connectivity and correctly configured AWS PrivateLink, receiving an updated DNS record takes about twice (2x) the TTL — clients should converge to the newly targeted region within, at most, a 30-second delay, assuming their resolver and language runtime honor the TTL. This DNS-based failover behavior does not apply to GCP PSC connections.

---

# Monitoring High Availability

Source: https://docs.temporal.io/cloud/high-availability/monitoring

> How to track the health and performance of your High Availability Namespaces.

Temporal Cloud offers several ways for you to track the health and performance of your
[High Availability](/cloud/high-availability) namespaces.

## Detect a failover or an outage 

With some [Worker deployment patterns](/cloud/high-availability/architecture-patterns) — most notably [Active/Passive](/cloud/high-availability/architecture-patterns#active-cold) — detecting an outage is your responsibility, and your Workflows make no progress until you detect it and bring up Workers in the new active region. Fast, reliable detection therefore directly determines your recovery time, so it is worth monitoring for both of the following.

### Detect a failover 

The clearest way to detect that a failover has happened is to watch whether your Namespace's active region changed. When Temporal Cloud promotes the replica in the secondary region to active, the active region reported for the Namespace changes — a reliable, unambiguous signal that a failover occurred. To track failovers as they happen, look for the `FailoverNamespace` operation described in [Failover audit log](#failover-audit-log).

### Detect an outage 

A failover is not the only signal worth watching. You may want to detect a regional outage directly, before or independently of a Namespace failover, so you can begin your own response. Watch for:

- **A spike in replication lag** between the primary and the replica. See [Monitoring replication](#monitoring-replication).
- **A drop in Workflow throughput**, such as a sudden decline in the rate of Workflows started, completed, or Tasks processed.
- **A spike in errors across your overall stack**, not just Temporal — for example, application errors, failed Activities, or connection failures.
- **A drop in throughput across your overall stack**, such as fewer requests reaching your services or fewer Activities executing.
- **Errors or failovers in other cloud systems you depend on**, such as databases, queues, or other regional services, which often signal a broader regional outage.

## Replication status

You can monitor your replica status with the Temporal Cloud UI. If the replica is unhealthy, Temporal Cloud disables the
"Trigger a failover" option to prevent failing over to an unhealthy replica. An unhealthy replica might be due to:

- **Data synchronization issues:** The replica fails to remain in sync with the primary due to network or performance
  problems.
- **Replication lag:** The replica falls behind the primary, causing it to be out of sync.
- **Network issues:** Loss of communication between the replica and the primary causes problems.
- **Failed health checks:** If the replica fails health checks, it's marked as unhealthy.

These issues prevent the replica from being used during a failover, ensuring system stability and consistency.

## Monitoring replication

Temporal Cloud's High Availability features use asynchronous replication between the primary and the replica. Workflow
updates in the primary, along with associated History Events, are transmitted to the replica. Replication lag refers to
the transmission delay of Workflow updates and history events from the primary to the replica.

> **💡 Tip:**
>
> Temporal Cloud strives to maintain a P95 replication lag of less than 1 minute. In this context,
> P95 means 95% of updates are processed faster than this limit.
>

A forced failover, when there is significant replication lag, increases the likelihood of rolling back Workflow
progress. Always check the replication lag metrics before initiating a failover.

Temporal Cloud emits replication lag [metrics](/cloud/metrics/openmetrics/metrics-reference#replication-metrics)
as pre-computed percentiles (p50, p95, p99) that are labeled with `temporal_namespace`.

When a Namespace is using a replica, you may notice that the Action count in `temporal_cloud_v1_total_action_count` is
2x what it was before adding a replica. This happens because Actions are replicated; they occur on both the primary and
the replica.

## Failover audit log

When Temporal triggers failovers, the [audit log](/cloud/audit-logs) will update with details.

Look for `"operation": "FailoverNamespace"` in the logs.

---

# Account access

Source: https://docs.temporal.io/cloud/manage-access

> Manage access to your Temporal Cloud account

Access to Temporal Cloud is governed by role-based access control (RBAC). Within an account, each access principal, such
as user, user group or service account, has one account-level role and optionally, one or more Namespace-level
permissions. Each principal can only perform actions that are allowed by their assigned roles and permissions.

Temporal Cloud supports [Security Assertion Markup Language (SAML)](/cloud/manage-access/saml) and
[System for Cross-domain Identity Management (SCIM)](/cloud/manage-access/scim) for integration with your organization's
identity provider (IdP). SAML enables single sign-on (SSO) by allowing your identity provider to authenticate users into
Temporal Cloud. SCIM automatically creates, updates, and removes users and groups in Temporal Cloud based on changes in
your identity provider.

To enforce shorter sessions for SAML-authenticated Cloud UI users, see
[Restrict Cloud UI session lifetimes](/best-practices/security-controls#restrict-cloud-ui-session-lifetimes).

## Temporal Cloud accounts

Accounts are the top-level container for access control. Each account has at least one user assigned the Account Owner
role, which has full administrative permissions across the account, including users, billing and usage. An account is
**not** an access principal itself.

When you sign up for Temporal Cloud without joining an existing account, you are automatically assigned the Account
Owner role for a new account. You can then invite other users to join the account and assign them roles. If your
organization has an IdP, we recommend using [a SAML integration](/cloud/manage-access/saml) for enterprise identity management.

> **ℹ️ Info:**
>
> Multiple accounts can coexist on the same email domain. Each account can have its own independent SAML configuration,
> tied to its unique Account ID.
>
> However, each email address can only be associated with a single Temporal Cloud account. If you need access to multiple
> accounts, you’ll need a separate invite for each one using a different email address.
>

## Access principals

Temporal Cloud offers the following principals for access control:

- [**Users**](/cloud/manage-access/users) - Manage individual user accounts and permissions
- [**User Groups**](/cloud/manage-access/user-groups) - Organize users into groups for simplified access management
- [**Service Accounts**](/cloud/manage-access/service-accounts) - Configure service accounts for automated access
- [**Custom Roles**](/cloud/manage-access/custom-roles) - Define custom permissions for specific use cases
- [**SAML**](/cloud/manage-access/saml) - Configure SAML-based single sign-on integration
- [**SCIM**](/cloud/manage-access/scim) - Use your IDP to manage Temporal Cloud users and access via SCIM integration

## Troubleshoot account access issues

### Recover your account after losing access to your authenticator app 

Accounts registered with email and password require multi-factor authentication (MFA) with an authenticator app. If you
lose access to your authenticator app, you can still log in by clicking **Try another method** on the MFA screen. From
there, you can either:

- Enter your recovery code (provided when you first set up MFA)
- Receive a verification code through email

Once you're logged in, you can reset your authenticator app by navigating to **My Profile** > **Password and
Authentication** and then clicking **Authenticator App** > **Remove method**.

### Reset your password 

If you're currently logged in and would like to change your password, click your profile icon at the top right of the
Temporal Cloud UI, navigate to **My Profile** > **Password and Authentication**, and then click **Reset Password**.

If you're not currently logged in, navigate to the login page of the Temporal Cloud UI, enter your email address, click
**Continue**, and then select **Forgot password**. In both cases, you will receive an email with instructions on how to
reset your password.

### Sign in after email domain changes 

If your organization changed its email domain (for example, from `@oldcompany.com` to `@newcompany.com`), you may be
unable to sign in to Temporal Cloud with your existing account.

**Why this happens:** When you sign in using "Continue with Google" or "Continue with Microsoft", Temporal Cloud
identifies your account by your email address. If your email address changes, Temporal Cloud sees this as a different
identity and cannot match it to your existing account.

**How to resolve this:** [Create a support ticket](/evaluate/cloud/support#support-ticket) with the following information:

- Your previous email address (the one originally used to access Temporal Cloud)
- Your new email address
- Your Temporal Cloud Account Id (if known)

Temporal Support can update your account to use your new email address.

> **💡 Tip:**
> Use SAML for enterprise identity management
>
> If your organization frequently changes email domains or wants centralized control over user authentication, consider
> using [SAML authentication](/cloud/manage-access/saml). With SAML, your identity provider (IdP) manages user identities, and email
> domain changes can be handled within your IdP without affecting Temporal Cloud access.
>

---

# Manage custom roles

Source: https://docs.temporal.io/cloud/manage-access/custom-roles

> **Pre-release**

With Custom Roles, you can define granular permissions in Temporal Cloud, giving your team precise control over who can
perform specific actions within your account.

## What are Custom Roles?

Custom Roles are user-defined collections of permissions that grant access to specific Temporal Cloud resources
([Account](/cloud/manage-access#temporal-cloud-accounts), [Namespace](/namespaces), [Nexus Endpoint](/nexus/endpoints),
or [Connectivity Rule](/cloud/connectivity#connectivity-rules)). They allow you to define custom permission sets that
are more granular than the predefined roles, and assign them to any principal (user, group, service account).

## Why use Custom Roles?

Use Custom Roles when you need more granular access control than the
[predefined roles](/cloud/manage-access/users#account-level-roles) provide. Common use cases include:

- **Least-privilege access**: Grant users only the permissions they need to perform their job functions.
- **Delegated administration**: Allow teams to manage specific Temporal Cloud resources without granting full account
  administration privileges.
- **Service account security**: Create narrowly scoped permissions for automation and integrations, reducing risk if
  credentials are compromised.

## Defining Custom Roles

When defining a Custom Role, you select the permission actions to include and the resources each action applies to.
Every principal must still have a predefined account role (such as Developer or Read-Only). The Custom Role is applied
alongside the predefined role, and the effective access is the union of both. Custom Roles cannot narrow or remove
permissions granted by the predefined role.

For example, you might want users with the Account Developer role to view all Namespaces for troubleshooting, without
allowing them to modify Namespace configuration. To do this, create a Custom Role named `NamespaceGlobalReadOnly` that
grants:

- `cloud.namespace.list`, scoped to the Cloud Account
- `cloud.namespace.get`, scoped to all Namespace resources

## Delegating Custom Roles

Custom role administration defaults to the Account Owner, but can be delegated. Account Owners are the primary
principals that can create, list, update, delete, or assign a Custom Role.

Account Owners can delegate Custom Role administration to other roles such as Global Admin, but this comes with risk. If
the Account Owner creates a Custom Role that includes the ability to create, update, or assign Custom Roles, and assigns
that role to any other principal, such as Global Admin or Developer, users with that new role can create or modify roles
with other operations and assign them to themselves or others.

> **⚠️ Warning:**
> Use caution when delegating Custom Role operations
>
> Delegating Custom Role administration to another user will give that user the ability to create or modify roles. This
> leads to privilege escalation risk where a user would have the ability to assign themselves or others to a role with any
> operations, even ones they're not approved for.
>

The following operations present privilege escalation risk when delegated:

- `cloud.customrole.create`
- `cloud.customrole.update`
- `cloud.customrole.assign`

Receiving `cloud.customrole.assign` alone does not grant the ability to update a principal's access on its own.
Assigning a Custom Role to a user, group, or service account also requires the relevant principal-update permission such
as [update a user's access](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/users/{userId}).
Developer and Read-only predefined roles do not include those permissions.
Global Admin can perform full Custom Role delegation after receiving `cloud.customrole.assign` because it has the
required principal-update permissions by default.

## Available permissions

Most of the operations listed in the Cloud Ops API references
([HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction),
[GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/)) can be assigned to a Custom Role. For the operations that are
supported by Custom Roles, see
[the Custom Role Permissions table](/cloud/manage-access/permissions-reference#custom-role-permissions-reference).

## Create Custom Roles

**Cloud UI**

To create a Custom Role from the Web UI, select **Identities** in the left sidebar, and then select the **Custom Roles** tab.

On the Custom Roles tab, you'll see a list of the roles that have already been defined for your account, 50 to a page.
Click the three dots menu to view details about an existing Custom Role, or to edit or delete that role.

Click the Create Custom Role button to create a new role. On the Create Custom Role page, give the Custom Role a name,
and optionally a description.

In the Permissions section, you'll assign the appropriate resources and its permissions. The following resources are
available for Custom Roles:

- Account: Permissions scoped to the current account, listed by type.
- Namespace: Permissions scoped to the selected Namespace, listed by type. You can only assign permissions to Namespaces
  to which you have access.
- Nexus Endpoint: Permissions scoped to the selected Nexus Endpoint, listed by type. You must have at least one Nexus
  Endpoint enabled before you can assign Nexus Endpoint permissions to a Custom Role.
- Connectivity Rules: Permissions scoped to the selected Connectivity Rules, listed by type.

Account-level permissions apply to the current account. For each of the other resources, you'll need to first select the
resource that you want to grant permissions for. Once you select a resource, the list of available permissions will
appear, and you can turn them on or off as desired.

At the bottom of the Account tab are permissions that relate to Custom Roles. Here, you can define whether members of
the Custom Role you're creating will be able to create, update, or delete Custom Roles.

When you're done assigning permissions to resources, click Create Custom Role at the bottom of the page.

**Cloud Operations API**

Use the
[`CreateCustomRoleRequest` API call](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CreateCustomRoleRequest)
to create a Custom Role. The structure of the `CreateCustomRoleRequest` is defined by a JSON spec, as follows:

```
{
  "name": "my-role-name"
  "description": "Optional description",
  "permissions": [
    {
      "resources": {
        "resource_type": "namespace",
        "allow_all": true
      },
      "actions": ["cloud.namespace.get"]
    }
  ]
}
```

This example creates a Custom Role with the defined `spec`. The example shown here creates a role named `my-role-name`
and grants the ability to get all Namespaces. The request returns a `RoleID`.

The following constraints apply to the request:

- **Name constraints:**
  - A name is required, with a maximum of 64 characters
  - The following characters are allowed: letters (a-z, A-Z), numbers (0-9), hyphens (-), underscores (\_)
  - Names are not unique per account, meaning two different roles can share the same name
- **Description constraints:**
  - A description is optional, with a maximum of 256 characters
- **Permissions constraints:**
  - At least one permission is required
  - A Custom Role can contain up to 20 permissions. See [Custom Roles limits](/evaluate/cloud/limits#custom-role-limits).

**Permission Structure**

Each permission has two parts: actions and resources.

- **Resource Types & Scoping Behavior**

| **Resource Type**  | **Scoping Behavior**                                                                      |
| ------------------ | ----------------------------------------------------------------------------------------- |
| account            | Account-level; no specific resource ID needed                                             |
| Namespaces         | Scoped to specific Namespace IDs, or `allow_all: true` for all Namespaces                 |
| Nexus endpoints    | Scoped to specific Nexus endpoint IDs, or `allow_all: true` for all Nexus endpoints       |
| connectivity rules | Scoped to specific connectivity rule IDs, or `allow_all: true` for all connectivity rules |

Resource IDs must exist in your account. Specifying an unknown resource ID will return an error.

Either `allow_all: true` or `resource_ids` must be set, not both.

For a list of available actions, see
[the list of Custom Role permissions](/cloud/manage-access/permissions-reference#custom-role-permissions-reference). -
Account-scoped actions (resource_type: "accounts") - Namespace-scoped actions (resource_type: "namespaces") - Nexus
endpoint-scoped actions (resource_type: "nexus_endpoints") - Connectivity rule-scoped actions (resource_type:
"connectivity_rules")

The following example allows read access to specific Namespaces. The action is `cloud.namespace.get`, the resource type
is `namespaces`, and the list of resource IDs is specified.

```jsx
{
    "actions": ["cloud.namespace.get"],
    "resources": {
      "resource_type": "namespaces",
      "resource_ids": ["my-namespace.account-id"]
    }
  }
```

The following example assigns multiple permissions to one Role. The permissions are assigned in an array, with the
action `cloud.namespace.get` granted for all Namespaces, and `cloud.user.list` granted for all accounts.

```jsx
 {
    "spec": {
      "name": "ns-reader-user-lister",
      "description": "Can read namespaces and list users",
      "permissions": [
        {
          "actions": ["cloud.namespace.get"],
          "resources": { "resource_type": "namespaces", "allow_all": true }
        },
        {
          "actions": ["cloud.user.list"],
          "resources": { "resource_type": "accounts", "allow_all": true }
        }
      ]
    }
  }
```

For more information about the `Custom Role` and `CustomRoleSpec` definitions, see the
[HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles) and/or
[GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/) API reference doc.

**Terraform**

For more detailed examples on how to manage Custom Roles via Terraform, see the
[Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/custom_role)
for usage guidance.

**Cloud CLI**

The a Custom Role is defined by a JSON spec, as follows:

```
{
  "name": "my-role-name"
  "description": "Optional description",
  "permissions": [
    {
      "resources": {
        "resource_type": "namespace",
        "allow_all": true
      },
      "actions": ["cloud.namespace.get"]
    }
  ]
}
```

This example creates a Custom Role with the defined `spec`. The example shown here creates a role named `my-role-name`
and grants the ability to get all Namespaces.

Save the spec to a file (such as `role.json`) so it can be passed via `@file`.

The following constraints apply to the request:

- **Name constraints:**
  - A name is required, with a maximum of 64 characters
  - The following characters are allowed: letters (a-z, A-Z), numbers (0-9), hyphens (-), underscores (\_)
  - Names are not unique per account, meaning two different roles can share the same name
- **Description constraints:**
  - A description is optional, with a maximum of 256 characters
- **Permissions constraints:**
  - At least one permission is required
  - You can assign up to 25 permissions per role (account default, may vary)

**Permission Structure**

Each permission has two parts: actions and resources.

- **Resource Types & Scoping Behavior**

| **Resource Type**  | **Scoping Behavior**                                                                      |
| ------------------ | ----------------------------------------------------------------------------------------- |
| account            | Account-level; no specific resource ID needed                                             |
| Namespaces         | Scoped to specific Namespace IDs, or `allow_all: true` for all Namespaces                 |
| Nexus endpoints    | Scoped to specific Nexus endpoint IDs, or `allow_all: true` for all Nexus endpoints       |
| connectivity rules | Scoped to specific connectivity rule IDs, or `allow_all: true` for all connectivity rules |

Resource IDs must exist in your account. Specifying an unknown resource ID will return an error.

Either `allow_all: true` or `resource_ids` must be set, not both.

For a list of available actions, see
[the list of Custom Role permissions](/cloud/manage-access/permissions-reference#custom-role-permissions-reference). -
Account-scoped actions (resource_type: "accounts") - Namespace-scoped actions (resource_type: "namespaces") - Nexus
endpoint-scoped actions (resource_type: "nexus_endpoints") - Connectivity rule-scoped actions (resource_type:
"connectivity_rules")

To create a custom role:

```
temporal cloud custom-role create --spec @role.json
```

## Assigning Custom Roles to users

Once you have created a Custom Role, it is available on the Identities page to assign to a user or group, the same as
the pre-defined Temporal permissions. See
[How to update an account-level role in Temporal Cloud](/cloud/manage-access/users#update-roles) for more information.

## Modifying a Custom Role

**Cloud UI**

To modify a Custom Role from the Web UI, select **Identities** in the left sidebar, and then select the **Custom Roles** tab.

On the Custom Roles tab, you'll see a list of the roles that have already been defined for your account, 50 to a page.
Click the three dots menu of the Custom Role you want to modify and select Edit.

The Edit Custom Role page has the same options as the Create Custom Role page. You can change the Custom Role's name or
description, or you can modify any of the current permissions assigned to that Role.

When finished, click Update Custom Role.

**Cloud Operations API**

Use the UpdateCustomRole API
([HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/POST/cloud/custom-roles/{roleId}),
[GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/#updatecustomrole))to modify the Custom Role.

**Terraform**

To update a Custom Role, modify the `temporalcloud_custom_role` resource in your Terraform configuration and run
`terraform apply`. Updating a Custom Role replaces the full permissions list. Include all permissions you want to
retain. Any permission omitted from the update will be removed.

For more details see the
[Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/custom_role).

**Cloud CLI**

To update a role spec by ID, use `update`:

```
temporal cloud custom-role update --role-id my-role-id --spec @role.json
```

You can override the resource version explicitly with `--resource-version`; otherwise the latest version is fetched and
used.

## Delete a Custom Role

**Cloud UI**

To delete a Custom Role from the Web UI, select **Identities** in the left sidebar, and then select the **Custom Roles** tab.

On the Custom Roles tab, you'll see a list of the roles that have already been defined for your account, 50 to a page.
Click the three dots menu of the Custom Role you want to delete and select Delete. A pop-up notification will let you
know that the Custom Role has been deleted.

**Cloud Operations API**

Use the DeleteCustomRole API
([HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/DELETE/cloud/custom-roles/{roleId}),
[GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/#deletecustomrole))to delete the Custom Role.

**Terraform**

To delete a Custom Role, remove the `temporalcloud_custom_role` resource block from your Terraform configuration and run
`terraform apply`. You can also target a specific resource:

Deleting a Custom Role permanently removes it and automatically revokes it from all principals it was assigned to. This
action cannot be undone.

For more details see the
[Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/custom_role).

**Cloud CLI**

To delete a Custom Role, use `delete`:

```
temporal cloud custom-role delete --role-id my-role-id
```

## Custom Roles limits

For more information about the limits of Custom Roles, such as the maximum number of custom roles per Account, see
[Custom Roles limits](/evaluate/cloud/limits#custom-role-limits).

---

# Permissions reference

Source: https://docs.temporal.io/cloud/manage-access/permissions-reference

> Reference for permissions in Temporal Cloud

Temporal Cloud access controls are organized across three scopes:

- Account-level role permissions
- Project-level permissions
- Namespace-level permissions

Within each scope, permissions apply to publicly documented [Temporal Cloud Ops API](/ops)
endpoints and to additional non-Cloud Ops capabilities, such as Temporal Cloud UI and internal automation behaviors.

## Account-level access 

Account-level access is granted to users and service accounts by assigning them an account-level role. Temporal Cloud
supports the following account-level roles:

- Account Owner
- Global Admin
- Developer
- Finance Admin
- Read-Only
- Metrics Read-Only

### Cloud Ops API permissions

The Metrics Read-Only role grants no Cloud Ops API permissions, so it is not included in the following table. For its
permissions and API key requirements, see [Metrics Read-Only role](/cloud/metrics/openmetrics#metrics-read-only-role).

This table provides API-level details for permissions granted through account-level roles. These permissions are
configured per user.

| Permission                  | Read-only | Developer | Finance Admin | Global Admin | Account Owner |
| --------------------------- | :-------: | :-------: | :-----------: | :----------: | :-----------: |
| [AddUserGroupMember](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D/members)          |           |           |               |      ✅       |       ✅       |
| [CreateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sinks)   |           |           |               |      ✅       |       ✅       |
| [CreateApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/POST/cloud/api-keys)                |    ✅\*    |    ✅\*    |      ✅\*      |     ✅\*      |      ✅\*      |
| [CreateBillingReport](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/billing/POST/cloud/billing-reports)         |           |           |       ✅       |              |       ✅       |
| [CreateConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/POST/cloud/connectivity-rules)      |           |           |               |      ✅       |       ✅       |
| [CreateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces)             |           |     ✅     |               |      ✅       |       ✅       |
| [CreateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints)         |           |     ✅     |               |      ✅       |       ✅       |
| [CreateProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects)               |           |     ✅     |               |      ✅       |       ✅       |
| [CreateServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/manage-access/service-accounts)        |    ✅†     |    ✅†     |      ✅†       |      ✅†      |      ✅†       |
| [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/users)                  |           |           |               |      ✅       |       ✅       |
| [CreateUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups)             |           |           |               |      ✅       |       ✅       |
| [DeleteAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/DELETE/cloud/audit-log-sinks/%7Bname%7D)   |           |           |               |      ✅       |       ✅       |
| [DeleteApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/DELETE/cloud/api-keys/%7BkeyId%7D)                |    ✅\*    |    ✅\*    |      ✅\*      |     ✅\*      |      ✅\*      |
| [DeleteConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/DELETE/cloud/connectivity-rules/%7BconnectivityRuleId%7D)      |           |           |               |      ✅       |       ✅       |
| [DeleteNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/DELETE/cloud/nexus/endpoints/%7BendpointId%7D)         |           |     ✅     |               |      ✅       |       ✅       |
| [DeleteServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/DELETE/cloud/manage-access/service-accounts/%7BserviceAccountId%7D)        |    ✅†     |    ✅†     |      ✅†       |      ✅†      |      ✅†       |
| [DeleteUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/DELETE/cloud/users/%7BuserId%7D)                  |           |           |               |      ✅       |       ✅       |
| [DeleteUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/DELETE/cloud/user-groups/%7BgroupId%7D)             |           |           |               |      ✅       |       ✅       |
| [GetAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/account)                  |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-log-sinks/%7Bname%7D)      |           |           |               |      ✅       |       ✅       |
| [GetAccountAuditLogSinks](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-log-sinks)     |           |           |               |      ✅       |       ✅       |
| [GetApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/GET/cloud/api-keys/%7BkeyId%7D)                   |    ✅\*    |    ✅\*    |      ✅\*      |     ✅\*      |      ✅\*      |
| [GetApiKeys](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/GET/cloud/api-keys)                  |    ✅\*    |    ✅\*    |      ✅\*      |     ✅\*      |      ✅\*      |
| [GetAsyncOperation](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/operations/GET/cloud/operations/%7BasyncOperationId%7D)           |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetAuditLogs](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-logs)                |           |           |               |      ✅       |       ✅       |
| [GetBillingReport](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/billing/GET/cloud/billing-reports/%7BbillingReportId%7D)            |           |           |       ✅       |              |       ✅       |
| [GetConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules/%7BconnectivityRuleId%7D)         |           |     ✅     |               |      ✅       |       ✅       |
| [GetConnectivityRules](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules)        |           |     ✅     |               |      ✅       |       ✅       |
| [GetCurrentIdentity](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/current-identity)          |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetNamespaces](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces)               |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints/%7BendpointId%7D)            |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetNexusEndpoints](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints)           |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetProjects](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects)                 |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/regions/GET/cloud/regions/%7Bregion%7D)                   |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetRegions](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/regions/GET/cloud/regions)                  |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/manage-access/service-accounts/%7BserviceAccountId%7D)           |    ✅†     |    ✅†     |      ✅†       |      ✅†      |      ✅†       |
| [GetServiceAccounts](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/manage-access/service-accounts)          |    ✅†     |    ✅†     |      ✅†       |      ✅†      |      ✅†       |
| [GetUsage](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/usage)                    |           |           |       ✅       |      ✅       |       ✅       |
| [GetUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/users/%7BuserId%7D)                     |    ✅‡     |    ✅‡     |      ✅‡       |     ✅‡       |      ✅‡       |
| [GetUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups/%7BgroupId%7D)                |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetUserGroupMembers](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups/%7BgroupId%7D/members)         |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetUserGroups](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups)               |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| [GetUsers](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/users)                    |    ✅‡     |    ✅‡     |      ✅‡       |     ✅‡       |      ✅‡       |
| [MoveNamespaceToProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/namespaces/%7Bnamespace%7D/move-to-project) |           |           |               |     ✅§      |      ✅§      |
| [RemoveUserGroupMember](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D/remove-member)       |           |           |               |      ✅       |       ✅       |
| [UpdateAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/account)               |           |           |               |      ✅       |       ✅       |
| [UpdateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sinks/%7Bspec.name%7D)   |           |           |               |      ✅       |       ✅       |
| [UpdateApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/POST/cloud/api-keys/%7BkeyId%7D)                |    ✅\*    |    ✅\*    |      ✅\*      |     ✅\*      |      ✅\*      |
| [UpdateNamespaceTags](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D/update-tags)         |           |           |               |      ✅       |       ✅       |
| [UpdateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints/%7BendpointId%7D)         |           |     ✅     |               |      ✅       |       ✅       |
| [UpdateServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/manage-access/service-accounts/%7BserviceAccountId%7D)        |    ✅†     |    ✅†     |      ✅†       |      ✅†      |      ✅†       |
| [UpdateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/users/%7BuserId%7D)                  |           |           |               |      ✅       |       ✅       |
| [UpdateUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D)             |           |           |               |      ✅       |       ✅       |
| [ValidateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sink-validate) |           |           |               |      ✅       |       ✅       |

- \* See [API Key Authorization Behavior](#api-key-authorization-behavior)
- † See [Service Account Authorization Behavior](#service-account-authorization-behavior)
- ‡ See [User authorization behavior](#user-authorization-behavior)
- § Requires Project Admin on both the source Project and the destination Project. Account Owner and Global Admin meet this requirement because they are Project Admin on every Project. Project Admin on only one of the two Projects is not sufficient. See [Move a Namespace to another Project](/cloud/projects#move-a-namespace).

## Namespace-level permissions

Namespace-level permissions are granted to users and service accounts by assigning them a Namespace-level permission.
Temporal Cloud supports the following Namespace-level permissions:

- Namespace Admin
- Write
- Read

Users with the Global Admin and Account Owner roles automatically have Namespace Admin permissions on all Namespaces in
the account.

### Cloud Ops API permissions

This table provides API-level details for permissions granted through Namespace-level permissions. These permissions are
configured per Namespace per user.

| Permission                       | Read | Write | Namespace Admin |
| -------------------------------- | :--: | :---: | :-------------: |
| [AddNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/%7Bnamespace%7D/add-region)               |      |       |        ✅        |
| [CreateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sinks)        |      |       |        ✅        |
| [DeleteNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/DELETE/cloud/namespaces/%7Bnamespace%7D)                  |      |       |        ✅        |
| [DeleteNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/DELETE/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bname%7D)        |      |       |        ✅        |
| [DeleteNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/DELETE/cloud/namespaces/%7Bnamespace%7D/regions/%7Bregion%7D)            |      |       |        ✅        |
| [FailoverNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/%7Bnamespace%7D/failover-region)          |      |       |        ✅        |
| [GetNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/%7Bnamespace%7D)                     |  ✅   |   ✅   |        ✅        |
| [GetNamespaceCapacityInfo](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/%7Bnamespace%7D/capacity-info)         |  ✅   |   ✅   |        ✅        |
| [GetNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/GET/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bname%7D)           |  ✅   |   ✅   |        ✅        |
| [GetNamespaceExportSinks](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/GET/cloud/namespaces/%7Bnamespace%7D/export-sinks)          |  ✅   |   ✅   |        ✅        |
| [GetUserNamespaceAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/namespaces/%7Bnamespace%7D/user-assignments) |      |       |        ✅        |
| [RenameCustomSearchAttribute](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D/rename-custom-search-attribute)      |      |       |        ✅        |
| [SetServiceAccountNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/namespaces/%7Bnamespace%7D/service-accounts/%7BserviceAccountId%7D/access) |      |       |        ✅        |
| [SetUserGroupNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/namespaces/%7Bnamespace%7D/user-groups/%7BgroupId%7D/access)      |      |       |        ✅        |
| [SetUserNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/namespaces/%7Bnamespace%7D/users/%7BuserId%7D/access)           |      |       |        ✅        |
| [UpdateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D)                  |      |       |        ✅        |
| [UpdateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bspec.name%7D)        |      |       |        ✅        |
| [ValidateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sink-validate)      |      |       |        ✅        |

### Workflow-level permissions

This table provides API-level details for Workflow-level Data Plane permissions granted through Namespace-level
permissions. These permissions are configured per Namespace per user.

| Permission                            | Read | Write | Namespace Admin |
| ------------------------------------- | :--: | :---: | :-------------: |
| CountActivityExecutions               |  ✅   |   ✅   |        ✅        |
| CountSchedules                        |  ✅   |   ✅   |        ✅        |
| CountWorkflowExecutions               |  ✅   |   ✅   |        ✅        |
| CreateSchedule                        |      |   ✅   |        ✅        |
| CreateWorkflowRule                    |      |   ✅   |        ✅        |
| DeleteActivityExecution               |      |   ✅   |        ✅        |
| DeleteSchedule                        |      |   ✅   |        ✅        |
| DeleteWorkerDeployment                |      |   ✅   |        ✅        |
| DeleteWorkerDeploymentVersion         |      |   ✅   |        ✅        |
| DeleteWorkflowExecution               |      |   ✅   |        ✅        |
| DeleteWorkflowRule                    |      |   ✅   |        ✅        |
| DescribeActivityExecution             |  ✅   |   ✅   |        ✅        |
| DescribeBatchOperation                |  ✅   |   ✅   |        ✅        |
| DescribeNamespace                     |  ✅   |   ✅   |        ✅        |
| DescribeSchedule                      |  ✅   |   ✅   |        ✅        |
| DescribeTaskQueue                     |  ✅   |   ✅   |        ✅        |
| DescribeWorker                        |  ✅   |   ✅   |        ✅        |
| DescribeWorkerDeployment              |  ✅   |   ✅   |        ✅        |
| DescribeWorkerDeploymentVersion       |  ✅   |   ✅   |        ✅        |
| DescribeWorkflowExecution             |  ✅   |   ✅   |        ✅        |
| DescribeWorkflowRule                  |  ✅   |   ✅   |        ✅        |
| ExecuteMultiOperation                 |      |   ✅   |        ✅        |
| FetchWorkerConfig                     |  ✅   |   ✅   |        ✅        |
| GetSearchAttributes                   |  ✅   |   ✅   |        ✅        |
| GetWorkerBuildIdCompatibility         |  ✅   |   ✅   |        ✅        |
| GetWorkerTaskReachability             |  ✅   |   ✅   |        ✅        |
| GetWorkerVersioningRules              |  ✅   |   ✅   |        ✅        |
| GetWorkflowExecutionHistory           |  ✅   |   ✅   |        ✅        |
| GetWorkflowExecutionHistoryReverse    |  ✅   |   ✅   |        ✅        |
| ListActivityExecutions                |  ✅   |   ✅   |        ✅        |
| ListBatchOperations                   |  ✅   |   ✅   |        ✅        |
| ListClosedWorkflowExecutions          |  ✅   |   ✅   |        ✅        |
| ListOpenWorkflowExecutions            |  ✅   |   ✅   |        ✅        |
| ListScheduleMatchingTimes             |  ✅   |   ✅   |        ✅        |
| ListSchedules                         |  ✅   |   ✅   |        ✅        |
| ListTaskQueuePartitions               |  ✅   |   ✅   |        ✅        |
| ListWorkerDeployments                 |  ✅   |   ✅   |        ✅        |
| ListWorkers                           |  ✅   |   ✅   |        ✅        |
| ListWorkflowExecutions                |  ✅   |   ✅   |        ✅        |
| ListWorkflowRules                     |  ✅   |   ✅   |        ✅        |
| PatchSchedule                         |      |   ✅   |        ✅        |
| PauseActivity                         |      |   ✅   |        ✅        |
| PauseWorkflowExecution                |      |   ✅   |        ✅        |
| PollActivityExecution                 |      |   ✅   |        ✅        |
| PollActivityTaskQueue                 |      |   ✅   |        ✅        |
| PollNexusTaskQueue                    |      |   ✅   |        ✅        |
| PollWorkflowExecutionUpdate           |      |   ✅   |        ✅        |
| PollWorkflowTaskQueue                 |      |   ✅   |        ✅        |
| QueryWorkflow                         |  ✅   |   ✅   |        ✅        |
| RecordActivityTaskHeartbeat           |      |   ✅   |        ✅        |
| RecordActivityTaskHeartbeatById       |      |   ✅   |        ✅        |
| RecordWorkerHeartbeat                 |      |   ✅   |        ✅        |
| RequestCancelActivityExecution        |      |   ✅   |        ✅        |
| RequestCancelWorkflowExecution        |      |   ✅   |        ✅        |
| ResetActivity                         |      |   ✅   |        ✅        |
| ResetStickyTaskQueue                  |      |   ✅   |        ✅        |
| ResetWorkflowExecution                |      |   ✅   |        ✅        |
| RespondActivityTaskCanceled           |      |   ✅   |        ✅        |
| RespondActivityTaskCanceledById       |      |   ✅   |        ✅        |
| RespondActivityTaskCompleted          |      |   ✅   |        ✅        |
| RespondActivityTaskCompletedById      |      |   ✅   |        ✅        |
| RespondActivityTaskFailed             |      |   ✅   |        ✅        |
| RespondActivityTaskFailedById         |      |   ✅   |        ✅        |
| RespondNexusTaskCompleted             |      |   ✅   |        ✅        |
| RespondNexusTaskFailed                |      |   ✅   |        ✅        |
| RespondQueryTaskCompleted             |      |   ✅   |        ✅        |
| RespondWorkflowTaskCompleted          |      |   ✅   |        ✅        |
| RespondWorkflowTaskFailed             |      |   ✅   |        ✅        |
| SetWorkerDeploymentCurrentVersion     |      |   ✅   |        ✅        |
| SetWorkerDeploymentManager            |      |   ✅   |        ✅        |
| SetWorkerDeploymentRampingVersion     |      |   ✅   |        ✅        |
| ShutdownWorker                        |      |   ✅   |        ✅        |
| SignalWithStartWorkflowExecution      |      |   ✅   |        ✅        |
| SignalWorkflowExecution               |      |   ✅   |        ✅        |
| StartActivityExecution                |      |   ✅   |        ✅        |
| StartBatchOperation                   |      |   ✅   |        ✅        |
| StartWorkflowExecution                |      |   ✅   |        ✅        |
| StopBatchOperation                    |      |   ✅   |        ✅        |
| TerminateActivityExecution            |      |   ✅   |        ✅        |
| TerminateWorkflowExecution            |      |   ✅   |        ✅        |
| TriggerWorkflowRule                   |      |   ✅   |        ✅        |
| UnpauseActivity                       |      |   ✅   |        ✅        |
| UnpauseWorkflowExecution              |      |   ✅   |        ✅        |
| UpdateActivityOptions                 |      |   ✅   |        ✅        |
| UpdateSchedule                        |      |   ✅   |        ✅        |
| UpdateTaskQueueConfig                 |      |   ✅   |        ✅        |
| UpdateWorkerBuildIdCompatibility      |      |   ✅   |        ✅        |
| UpdateWorkerConfig                    |      |   ✅   |        ✅        |
| UpdateWorkerDeploymentVersionMetadata |      |   ✅   |        ✅        |
| UpdateWorkerVersioningRules           |      |   ✅   |        ✅        |
| UpdateWorkflowExecution               |      |   ✅   |        ✅        |
| UpdateWorkflowExecutionOptions        |      |   ✅   |        ✅        |

## Project-level permissions

[Project-level permissions](/cloud/manage-access/permissions-reference#project-level-permissions) are granted to users, groups, and service accounts by assigning them a Project-level role.
Temporal Cloud supports the following Project-level roles:

- Project Admin
- Project Write
- Project Read
- Project Contribute
- Project List
- Project Member

Users with the Global Admin and Account Owner roles automatically have Project Admin on all Projects in the account.

Project Admin, Project Write, and Project Read also grant Namespace Admin, Write, and Read respectively on every
Namespace in the Project. Project Contribute, Project List, and Project Member grant no Namespace-level access. This is
why Project Read and Project List grant the same Cloud Ops API permissions, as do Project Write and Project Contribute:
they differ in the Namespace-level and [Workflow-level permissions](#workflow-level-permissions) they carry, not in the
Cloud Ops API endpoints they can call.

### Cloud Ops API permissions

This table provides API-level details for permissions granted through Project-level roles. These permissions are
configured per Project per user.

These permissions control who can manage Nexus Endpoints in the Project.
They do not restrict which Namespaces can call an Endpoint at runtime, which continues to be controlled by the Endpoint [allowlist](/nexus/security#runtime-access-controls).
See [Use Nexus across Projects](/cloud/projects#use-nexus-across-projects).

| Permission                          | Member | List | Read | Contribute | Write | Project Admin |
| ----------------------------------- | :----: | :--: | :--: | :--------: | :---: | :-----------: |
| [CreateConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/POST/cloud/connectivity-rules) |        |      |      |     ✅      |   ✅   |       ✅       |
| [CreateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces) |        |      |      |     ✅      |   ✅   |       ✅       |
| [CreateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints) |        |      |      |     ✅      |   ✅   |       ✅       |
| [CreateServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/service-accounts) |        |      |      |            |       |      ✅†       |
| [DeleteConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/DELETE/cloud/connectivity-rules/%7BconnectivityRuleId%7D) |        |      |      |     ✅      |   ✅   |       ✅       |
| [DeleteNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/DELETE/cloud/nexus/endpoints/%7BendpointId%7D) |        |      |      |     ✅      |   ✅   |       ✅       |
| [DeleteProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/DELETE/cloud/projects/%7BprojectId%7D) |        |      |      |            |       |       ✅       |
| [GetConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules/%7BconnectivityRuleId%7D) |        |  ✅   |  ✅   |     ✅      |   ✅   |       ✅       |
| [GetConnectivityRules](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules) |        |  ✅   |  ✅   |     ✅      |   ✅   |       ✅       |
| [GetNamespaces](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces) |   ✅    |  ✅   |  ✅   |     ✅      |   ✅   |       ✅       |
| [GetNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints/%7BendpointId%7D) |        |  ✅   |  ✅   |     ✅      |   ✅   |       ✅       |
| [GetNexusEndpoints](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints) |        |  ✅   |  ✅   |     ✅      |   ✅   |       ✅       |
| [GetProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D) |   ✅    |  ✅   |  ✅   |     ✅      |   ✅   |       ✅       |
| [GetProjectScopedServiceAccounts](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/service-accounts) |        |      |      |            |       |       ✅       |
| [GetServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/service-accounts/%7BserviceAccountId%7D) |        |  ✅†  |  ✅†  |     ✅†     |  ✅†   |      ✅†       |
| [GetServiceAccountProjectAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/service-account-assignments) |        |      |      |            |       |       ✅       |
| [GetServiceAccounts](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/service-accounts) |        |  ✅†  |  ✅†  |     ✅†     |  ✅†   |      ✅†       |
| [GetUserGroupProjectAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/user-group-assignments) |        |      |      |            |       |       ✅       |
| [GetUserProjectAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/user-assignments) |        |      |      |            |       |       ✅       |
| [MoveNamespaceToProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/namespaces/%7Bnamespace%7D/move-to-project) |        |      |      |            |       |      ✅§       |
| [SetServiceAccountProjectAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D/service-accounts/%7BserviceAccountId%7D/access) |        |      |      |            |       |       ✅       |
| [SetUserGroupProjectAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D/user-groups/%7BgroupId%7D/access) |        |      |      |            |       |       ✅       |
| [SetUserProjectAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D/users/%7BuserId%7D/access) |        |      |      |            |       |       ✅       |
| [UpdateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints/%7BendpointId%7D) |        |      |      |     ✅      |   ✅   |       ✅       |
| [UpdateProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D) |        |      |      |            |       |       ✅       |

- † See [Service Account Authorization Behavior](#service-account-authorization-behavior)
- § Requires Project Admin on both the source Project and the destination Project. Account Owner and Global Admin meet this requirement because they are Project Admin on every Project. Project Admin on only one of the two Projects is not sufficient. See [Move a Namespace to another Project](/cloud/projects#move-a-namespace).

Project Developer is a compatibility role that users with the Account Developer role inherit automatically. It cannot be
assigned directly. It grants everything Project Contribute grants except CreateConnectivityRule and
DeleteConnectivityRule, and carries no Namespace-level access. For details, see
[Project roles](/cloud/projects#project-roles).

## Custom Role permissions reference

The following tables list the permission action strings available when defining [Custom Roles](/cloud/manage-access/custom-roles).
Use these strings in the `actions` field of a Custom Role permission grant.

Not all Cloud Ops API operations can be assigned to a Custom Role. For details on which operations are excluded and why, see [Available permissions](/cloud/manage-access/custom-roles#available-permissions).

### Account permissions

| Permission | Cloud Ops API | Resource type |
| ---------- | ------------- | ------------- |
| `cloud.account.get` | [GetAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/account) | Account |
| `cloud.account.update` | [UpdateAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/account) | Account |
| `cloud.asyncoperation.get` | [GetAsyncOperation](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/operations/GET/cloud/operations/%7BasyncOperationId%7D) | Account |
| `cloud.auditlog.createSink` | [CreateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sinks) | Account |
| `cloud.auditlog.deleteSink` | [DeleteAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/DELETE/cloud/audit-log-sinks/%7Bname%7D) | Account |
| `cloud.auditlog.getSink` | [GetAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-log-sinks/%7Bname%7D) | Account |
| `cloud.auditlog.list` | [GetAuditLogs](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-logs) | Account |
| `cloud.auditlog.listSinks` | [GetAccountAuditLogSinks](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-log-sinks) | Account |
| `cloud.auditlog.updateSink` | [UpdateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sinks/%7Bspec.name%7D) | Account |
| `cloud.auditlog.validateSink` | [ValidateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sink-validate) | Account |
| `cloud.billingreport.create` | [CreateBillingReport](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/billing/POST/cloud/billing-reports) | Account |
| `cloud.billingreport.get` | [GetBillingReport](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/billing/GET/cloud/billing-reports/%7BbillingReportId%7D) | Account |
| `cloud.connectivityrule.list` | [GetConnectivityRules](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules) | Account, Project |
| `cloud.customrole.create` | [CreateCustomRole](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/POST/cloud/custom-roles) | Account, Project |
| `cloud.customrole.list` | [GetCustomRoles](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/GET/cloud/custom-roles) | Account, Project |
| `cloud.namespace.list` | [GetNamespaces](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces) | Account, Project |
| `cloud.nexusendpoint.list` | [GetNexusEndpoints](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints) | Account, Project |
| `cloud.region.get` | [GetRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/regions/GET/cloud/regions/%7Bregion%7D) | Account |
| `cloud.region.list` | [GetRegions](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/regions/GET/cloud/regions) | Account |
| `cloud.usage.get` | GetUsage | Account |
| `cloud.user.create` | [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/users) | Account |
| `cloud.user.delete` | [DeleteUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/DELETE/cloud/users/%7BuserId%7D) | Account |
| `cloud.user.get` | [GetUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/users/%7BuserId%7D) | Account |
| `cloud.user.list` | [GetUsers](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/users) | Account |
| `cloud.usergroup.addMember` | [AddUserGroupMember](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D/members) | Account |
| `cloud.usergroup.create` | [CreateUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups) | Account |
| `cloud.usergroup.delete` | [DeleteUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/DELETE/cloud/user-groups/%7BgroupId%7D) | Account |
| `cloud.usergroup.get` | [GetUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups/%7BgroupId%7D) | Account |
| `cloud.usergroup.getMembers` | [GetUserGroupMembers](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups/%7BgroupId%7D/members) | Account |
| `cloud.usergroup.list` | [GetUserGroups](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups) | Account |
| `cloud.usergroup.removeMember` | [RemoveUserGroupMember](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D/remove-member) | Account |

### Namespace permissions

| Permission | Cloud Ops API | Resource type |
| ---------- | ------------- | ------------- |
| `cloud.namespace.addRegion` | [AddNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/%7Bnamespace%7D/add-region) | Namespace |
| `cloud.namespace.capacityinfo.get` | [GetNamespaceCapacityInfo](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/%7Bnamespace%7D/capacity-info) | Namespace |
| `cloud.namespace.delete` | [DeleteNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/DELETE/cloud/namespaces/%7Bnamespace%7D) | Namespace |
| `cloud.namespace.deleteRegion` | [DeleteNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/DELETE/cloud/namespaces/%7Bnamespace%7D/regions/%7Bregion%7D) | Namespace |
| `cloud.namespace.exportsink.create` | [CreateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sinks) | Namespace |
| `cloud.namespace.exportsink.delete` | [DeleteNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/DELETE/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bname%7D) | Namespace |
| `cloud.namespace.exportsink.get` | [GetNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/GET/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bname%7D) | Namespace |
| `cloud.namespace.exportsink.list` | [GetNamespaceExportSinks](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/GET/cloud/namespaces/%7Bnamespace%7D/export-sinks) | Namespace |
| `cloud.namespace.exportsink.update` | [UpdateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bspec.name%7D) | Namespace |
| `cloud.namespace.exportsink.validate` | [ValidateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sink-validate) | Namespace |
| `cloud.namespace.failoverRegion` | [FailoverNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/%7Bnamespace%7D/failover-region) | Namespace |
| `cloud.namespace.get` | [GetNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/%7Bnamespace%7D) | Namespace |
| `cloud.namespace.getServiceAccountNamespaceAssignments` | [GetServiceAccountNamespaceAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/namespaces/%7Bnamespace%7D/service-account-assignments) | Namespace |
| `cloud.namespace.getUserGroupNamespaceAssignments` | [GetUserGroupNamespaceAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/namespaces/%7Bnamespace%7D/user-group-assignments) | Namespace |
| `cloud.namespace.getUserNamespaceAssignments` | [GetUserNamespaceAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/namespaces/%7Bnamespace%7D/user-assignments) | Namespace |
| `cloud.namespace.renameCustomSearchAttribute` | [RenameCustomSearchAttribute](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D/rename-custom-search-attribute) | Namespace |
| `cloud.namespace.setUserAccess` | [SetUserNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/namespaces/%7Bnamespace%7D/users/%7BuserId%7D/access) | Namespace |
| `cloud.namespace.setUserGroupAccess` | [SetUserGroupNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/namespaces/%7Bnamespace%7D/user-groups/%7BgroupId%7D/access) | Namespace |
| `cloud.namespace.update` | [UpdateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D) | Namespace |
| `cloud.namespace.updateTags` | [UpdateNamespaceTags](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D/update-tags) | Namespace |

### Project permissions

| Permission | Cloud Ops API | Resource type |
| ---------- | ------------- | ------------- |
| `cloud.connectivityrule.create` | [CreateConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/POST/cloud/connectivity-rules) | Project |
| `cloud.connectivityrule.list` | [GetConnectivityRules](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules) | Account, Project |
| `cloud.customrole.create` | [CreateCustomRole](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/POST/cloud/custom-roles) | Account, Project |
| `cloud.customrole.list` | [GetCustomRoles](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/GET/cloud/custom-roles) | Account, Project |
| `cloud.namespace.create` | [CreateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces) | Project |
| `cloud.namespace.list` | [GetNamespaces](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces) | Account, Project |
| `cloud.nexusendpoint.create` | [CreateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints) | Project |
| `cloud.nexusendpoint.list` | [GetNexusEndpoints](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints) | Account, Project |

### Nexus Endpoint permissions

| Permission | Cloud Ops API | Resource type |
| ---------- | ------------- | ------------- |
| `cloud.nexusendpoint.delete` | [DeleteNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/DELETE/cloud/nexus/endpoints/%7BendpointId%7D) | Nexus Endpoint |
| `cloud.nexusendpoint.get` | [GetNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints/%7BendpointId%7D) | Nexus Endpoint |
| `cloud.nexusendpoint.update` | [UpdateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints/%7BendpointId%7D) | Nexus Endpoint |

### Connectivity Rule permissions

| Permission | Cloud Ops API | Resource type |
| ---------- | ------------- | ------------- |
| `cloud.connectivityrule.delete` | [DeleteConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/DELETE/cloud/connectivity-rules/%7BconnectivityRuleId%7D) | Connectivity Rule |
| `cloud.connectivityrule.get` | [GetConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules/%7BconnectivityRuleId%7D) | Connectivity Rule |

### Custom Role permissions

| Permission | Cloud Ops API | Resource type |
| ---------- | ------------- | ------------- |
| `cloud.customrole.delete` | [DeleteCustomRole](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/DELETE/cloud/custom-roles/%7BroleId%7D) | Custom Role |
| `cloud.customrole.get` | [GetCustomRole](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/GET/cloud/custom-roles/%7BroleId%7D) | Custom Role |
| `cloud.customrole.update` | [UpdateCustomRole](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/POST/cloud/custom-roles/%7BroleId%7D) | Custom Role |

## Additional authorization behaviors

Some APIs are granted to all account-level roles but enforce additional authorization rules at runtime. The action group
grants access to call the API, but the scope of what the caller can interact with depends on their role.

### API key authorization behavior

All roles can create and manage their **own** API keys. An API key inherits the permissions of its owner — it cannot
grant access beyond what the owning user or service account already has.

| Behavior                                            | Read-only | Developer | Finance Admin | Global Admin | Account Owner |
| --------------------------------------------------- | :-------: | :-------: | :-----------: | :----------: | :-----------: |
| Create, view, update, and delete own API keys       |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| View, update, and delete any API key in the account |           |           |               |      ✅       |       ✅       |

**Affected APIs:** CreateApiKey, GetApiKey, GetApiKeys, UpdateApiKey, DeleteApiKey

### Service account authorization behavior

All roles can list service accounts within their account. However, the ability to create, update, and delete service
accounts depends on the scope of the service account and the caller's role.

| Behavior                                         | Read-only | Developer | Finance Admin | Global Admin | Account Owner |
| ------------------------------------------------ | :-------: | :-------: | :-----------: | :----------: | :-----------: |
| List all service accounts in the account         |     ✅     |     ✅     |       ✅       |      ✅       |       ✅       |
| Manage unscoped (account-level) service accounts |           |           |               |      ✅       |       ✅       |
| Manage Namespace-scoped service accounts         |     §     |     §     |       §       |      ✅       |       ✅       |
| Manage Project-scoped service accounts           |     †     |     †     |       †       |      ✅       |       ✅       |

§ Requires Namespace Admin permission on the target Namespace. Any role can manage Namespace-scoped service accounts if
they hold Namespace Admin on that Namespace.

† Requires Project Admin permission on the target Project. Any role can manage Project-scoped service accounts if
they hold Project Admin on that Project.

**Affected APIs:** CreateServiceAccount, GetServiceAccount, GetServiceAccounts, UpdateServiceAccount,
DeleteServiceAccount

### User authorization behavior

Every account-level role can call GetUser and GetUsers. The response includes the user's ID, email, and state.

Another user's access (account role, Namespace permissions, Project access, and Custom Roles) and invitation are
returned only to Account Owner and Global Admin. Namespace Admin does not change this.

GetUser of the caller always includes that caller's own access.

To list who has access to a Namespace, use GetUserNamespaceAssignments.

**Affected APIs:** GetUser, GetUsers

---

# Roles and permissions

Source: https://docs.temporal.io/cloud/manage-access/roles-and-permissions

> Temporal Cloud RBAC gives every access principal an account-level role, with Namespace-level permissions and optional Custom Roles for granular access.

Temporal Cloud uses role-based access control (RBAC) to manage access to resources. Access is governed both on the
account-level and within a Namespace. At the account-level, each access principal is assigned one [account-level role](#account-level-roles). At the Namespace-level, each access principal can be assigned one [Namespace-level permissions](#namespace-level-permissions). 
Some account-level roles, such as Account Owner and Global Admin, automatically have Namespace Admin permissions on all Namespaces in the account. 

Each access principal, including users, groups, and Service Accounts, must have a predefined account-level role. For more granular access control, you can also use [Custom Roles](/cloud/manage-access/custom-roles) to define specific permissions for specific Temporal Cloud resources and assign them to any principal.

## Account-level roles 

Account-level roles are assigned to access principals at the account level. They control access to account resources,
such as:

- Users and Service Accounts
- Billing and usage
- Namespaces. This includes creating and managing Namespaces only, not access to resources within a Namespace, which is
  controlled by [Namespace-level permissions](#namespace-level-permissions).
- Nexus Endpoints

The following table provides a summary of the account-level roles and their primary purpose. Refer to the
[Permissions reference](/cloud/manage-access/permissions-reference#account-level-access) for API-level details.

| Role              | Primary purpose                                | Can create Namespaces | Automatic Namespace Admin               | Billing and usage access          |
| ----------------- | ---------------------------------------------- | --------------------- | --------------------------------------- | --------------------------------- |
| Account Owner     | Owns and governs the account                   | Yes                   | All Namespaces (cannot be revoked)      | Full billing, payments, and usage |
| Global Admin      | Administers account configuration and users    | Yes                   | All Namespaces (cannot be revoked)      | Usage only                        |
| Developer         | Creates and manages Namespaces they own        | Yes                   | Namespaces they create (can be revoked) | None                              |
| Finance Admin     | Manages billing and payment information        | No                    | None                                    | Full billing, payments, and usage |
| Read-Only         | Views account configuration and resources      | No                    | None                                    | None                              |
| Metrics Read-Only | Reads account metrics from the OpenMetrics API | No                    | None                                    | None                              |

The Metrics Read-Only role is intended for Service Accounts used by observability integrations. It grants read access to
the [Temporal Cloud OpenMetrics endpoint](/cloud/metrics/openmetrics) but no other permissions. An Account Owner or
Global Admin must manage the Service Account and its API keys.

Account Owner is the only role that can assign Finance Admin. This applies to users, groups, and Service
Accounts. Global Admin can create these principals but cannot assign Finance Admin. Global Admin has usage visibility,
not billing or payment access.

Account-level roles don't govern day-to-day operations within a Namespace. Access to resources inside a Namespace, such
as Workflows and Workflow Executions, is controlled by [Namespace-level permissions](#namespace-level-permissions).

Account Owner and Global Admin roles automatically have Namespace Admin permissions on all Namespaces in the account,
and these permissions cannot be revoked without removing the role. Developers can create Namespaces, and have Namespace
Admin permissions for each Namespace they create. This permission can be revoked. Developer roles also don't have
automatic access to Namespaces that they didn't create.

### Best practice for assigning the Account Owner role

The Account Owner role holds the highest level of access in the system. This role configures account-level parameters
and manages Temporal billing and payment information. It allows users to perform all actions within the Temporal Cloud
account.

We strongly recommend the following precautions when assigning the Account Owner role to users:

- Assign the role to at least two users in your organization. Otherwise, limit the number of users with this role.
- Associate a person’s direct email address to the Account Owner, rather than a shared or generic address, so Temporal
  Support can contact the right person in urgent situations.

This latter rule is useful for anyone on your team who may need to be contacted urgently, regardless of their Account
role.

## Namespace-level permissions 

Namespace-level permissions govern access to resources within a Namespace, such as the following:

- Workflows
- Workflow Executions
- Task Queues
- Activity Executions
- Search Attributes
- History
- Events

Namespace-level permissions are assigned to access principals within a Namespace. Each permission has a set of actions
that grant access to specific resources within the Namespace.

The following table provides a summary of the Namespace-level permissions and their primary purpose. Refer to the
[Permissions reference](/cloud/manage-access/permissions-reference#namespace-level-permissions) for API-level details.

| Permission level | Intended use                      | Human access                                                                                 | Worker runtime access                                     | Namespace administration                                                                                  |
| ---------------- | --------------------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Read             | Observe Namespace activity        | View Workflows, Workflow Executions, Schedules, Task Queues, and metadata                    | None                                                      | None                                                                                                      |
| Write            | Operate Workflows and run Workers | Start, signal, cancel, terminate, and reset Workflows; manage Schedules and batch operations | Poll Task Queues and complete Workflow and Activity Tasks | None                                                                                                      |
| Namespace Admin  | Administer the Namespace          | All Read and Write capabilities                                                              | All Read and Write capabilities                           | Update Namespace settings, manage Search Attributes, Export Sinks, replication, and Namespace user access |

You can grant Namespace Admin, Write, or Read-Only permissions to principals with the account-level roles of Developer,
Finance Admin, or Read-Only. Account Owners and Global Admins already have Namespace Admin permissions on all Namespaces
in the account and do not need to be manually assigned Namespace-level permissions.

---

# SAML authentication

Source: https://docs.temporal.io/cloud/manage-access/saml

To authenticate the users of your Temporal Cloud account, you can connect an identity provider (IdP) to your account by
using Security Assertion Markup Language (SAML) 2.0.

> **ℹ️ Info:**
>
> SAML is included on all Temporal Cloud plans. See the [pricing page](/cloud/pricing) for details.
>

## Integrate SAML with your Temporal Cloud account

1. Locate your [Temporal Cloud Account Id](/cloud/namespaces#temporal-cloud-account-id). Your Account Id can be viewed
   and copied from the Temporal Cloud user profile dropdown menu in the top right corner. Alternatively, find your
   [Namespace Id](/cloud/namespaces#temporal-cloud-namespace-id). The Account Id is the five or six characters following
   the period (.), such as `f45a2`. You will need the Account Id to construct your callback URL and your entity
   identifier.
1. Configure SAML with your IdP:
   - [Microsoft Entra ID example](#configure-saml-with-azure-ad)
   - [Okta example](#configure-saml-with-okta)
   - [Generic SAML 2.0 configuration](#configure-saml-with-other-identity-providers)
1. [Share your connection information with us and test your connection.](#finish-saml-configuration)

## How to configure SAML with Microsoft Entra ID 

If you want to use the general Microsoft login mechanism, you don't need to set up SAML with Entra ID. Just select
**Continue with Microsoft** on the Temporal Cloud sign-in page.

To use Entra ID as your SAML IdP, create a Microsoft Entra ID Enterprise application.

1. Sign in to the [Microsoft Entra ID](https://portal.azure.com/).
1. On the home page, under **Manage Microsoft Entra ID**, select **View**.
1. On the **Overview** page near the top, select **Add > Enterprise application**.
1. On the **Browse Microsoft Entra ID Gallery** page near the top, select **Create your own application**.
1. In the **Create your own application** pane, provide a name for your application (such as `temporal-cloud`) and
   select **Integrate any other application you don't find in the gallery**.
1. Select **Save**.
1. In the **Getting Started** section, select **2. Set up single sign on**.
1. On the **Single sign-on** page, select **SAML**.
1. In the **Basic SAML Configuration** section of the **SAML-based Sign-on** page, select **Edit**.
1. In **Identifier (Entity ID)**, enter the following entity identifier, including your Account Id where indicated:

   ```bash
   urn:auth0:prod-tmprl:ACCOUNT_ID-saml
   ```

   A correctly formed entity identifier looks like this:

   ```bash
   urn:auth0:prod-tmprl:f45a2-saml
   ```

1. In **Reply URL (Assertion Consumer Service URL)**, enter the following callback URL, including your Account Id where
   indicated:

   ```bash
   https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml
   ```

   A correctly formed callback URL looks like this:

   ```bash
   https://login.tmprl.cloud/login/callback?connection=f45a2-saml
   ```

1. In **Sign on URL**, enter the following login url, including your Account Id where indicated:

   ```bash
   https://cloud.temporal.io/login/saml?connection=ACCOUNT_ID-saml
   ```

   A correctly formed login URL looks like this:

   ```bash
   https://cloud.temporal.io/login/saml?connection=f45a2-saml
   ```

1. You can leave the other fields blank. Near the top of the pane, select **Save**.
1. In the **Attributes & Claims** section, select **Edit**. Configure the following settings. Under **Required claim**:

   - Set **Unique User Identifier (NameID)** to `user.userprincipalname`
   - Set the **NameID format** to `emailAddress`

   These are the default settings for Microsoft Entra ID. Then under **Additional claims**, ensure **Email** and
   **Name** are present.

1. Collect information that you need to send to us:
   - In the **SAML Certificates** section of the **SAML-based Sign-on** page, select the download link for **Certificate
     (Base64)**.
   - In the **Set up _APPLICATION_NAME_** section of the **SAML-based Sign-on** page, copy the value of **Login URL**.

To finish setting up Microsoft Entra ID as your SAML IdP, see [Finish SAML configuration](#finish-saml-configuration).

## How to configure SAML with Okta 

To use Okta as your SAML IdP, configure a new Okta application integration.

1. Sign in to the [Okta Admin Console](https://www.okta.com/login/).
1. In the left navigation pane, select **Applications > Applications**.
1. On the **Applications** page, select **Create App Integration**.
1. In the **Create a new app integration** dialog, select **SAML 2.0** and then select **Next**.
1. On the **Create SAML Integration** page in the **General Settings** section, provide a name for your application
   (such as `temporal-cloud`) and then select **Next**.
1. In the **Configure SAML** section in **Single sign on URL**, enter the following callback URL, including your Account
   Id where indicated:

   ```bash
   https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml
   ```

   A correctly formed callback URL looks like this:

   ```bash
   https://login.tmprl.cloud/login/callback?connection=f45a2-saml
   ```

1. In **Audience URI (SP Entity ID)**, enter the following entity identifier, including your Account Id where indicated:

   ```bash
   urn:auth0:prod-tmprl:ACCOUNT_ID-saml
   ```

   A correctly formed entity identifier looks like this:

   ```bash
   urn:auth0:prod-tmprl:f45a2-saml
   ```

1. We require the user's full email address when connecting to Temporal.
   - In **Name ID format**, select `EmailAddress`.
   - In **Attribute Statements**, set **email** and **name**.
1. Select **Next**.
1. In the **Feedback** section, select **Finish**.
1. On the **Applications** page, select the name of the application integration you just created.
1. On the application integration page, select the **Sign On** tab.
1. Under **SAML Setup**, select **View SAML setup instructions**.
1. Collect information that you need to send to us:
   - Copy the IdP settings.
   - Download the active certificate.

To finish setting up Okta as your SAML IdP, see
[Finish SAML configuration](#finish-saml-configuration).

## How to configure SAML with another identity provider 

Temporal Cloud works with identity providers that meet the SAML 2.0 requirements described below. Provider names in
this documentation are configuration examples, not an exhaustive compatibility list. An identity provider does not
need to be listed to work with Temporal Cloud.

Configure your IdP with the following values, replacing `ACCOUNT_ID` with your Temporal Cloud Account Id:

- **Service provider entity identifier:**

  ```text
  urn:auth0:prod-tmprl:ACCOUNT_ID-saml
  ```

- **Assertion Consumer Service URL:**

  ```text
  https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml
  ```

Configure the SAML connection to:

- Use SAML 2.0 with HTTP POST binding.
- Use an RSA X.509 signing certificate and RSA-SHA256 signing.
- Provide a stable subject `NameID` that uniquely identifies each user.
- Include the user's email address and name as attributes.

The following example shows how to apply the same configuration to an additional identity provider. It does not
represent the complete set of compatible providers.

### PingFederate configuration example

Create a SAML 2.0 Browser SSO service provider connection in PingFederate using the Temporal Cloud entity identifier
and Assertion Consumer Service URL shown above. Configure the connection with an RSA signing certificate, a stable
subject `NameID`, and the required `email` and `name` attributes.

For IdP-initiated SSO, publish the connection-specific `/idp/startSSO.ping` URL in your organization's application
portal.

For detailed instructions, see:

- [Configure IdP Browser SSO](https://docs.pingidentity.com/pingfederate/13.1/administrators_reference_guide/help_spconnectionconfigtasklet_spbrowserssostate.html)
- [PingFederate IdP endpoints](https://docs.pingidentity.com/pingfederate/13.1/developers_reference_guide/pf_idp_endpoints.html)

After configuring your IdP, continue to [Finish SAML configuration](#finish-saml-configuration).

## How to finish your SAML configuration 

After you configure SAML with your IdP, we can finish the configuration on our side.
[Create a support ticket](/evaluate/cloud/support#support-ticket) that includes the following information:

- The sign-in URL from your application
- The X.509 SAML sign-in certificate in PEM format
- One or more IdP domains to map to the SAML connection

Generally, the provided IdP domain is the same as the domain for your email address. You can provide multiple IdP
domains.

When you receive confirmation from us that we have finished configuration, log in to Temporal Cloud. This time, though,
enter your email address in **Enterprise identity** and select **Continue**. Do not select **Continue with Google** or
**Continue with Microsoft**. You will be redirected to the authentication page of your IdP.

---

# SCIM user management

Source: https://docs.temporal.io/cloud/manage-access/scim

> Link your IdP with your Temporal Cloud account to securely automate user and group management.

[SCIM](https://scim.cloud/) lets you integrate your identity provider (IdP) with Temporal Cloud to automate user provisioning and access. Once SCIM is configured, changes in your IdP are automatically reflected in Temporal Cloud, including:

- User creation / onboarding
- User deletion / offboarding
- User membership in groups

You can map SCIM groups to Temporal Cloud [roles and permissions](/cloud/manage-access/users#account-level-roles-and-namespace-level-permissions), so users automatically get the Temporal Cloud access they need based on the groups they belong to.

> **ℹ️ Info:**
>
> SCIM is a paid feature. See the [pricing page](/cloud/pricing) for details.
>

## Supported IdP vendors

Supported upstream IdP vendors include:
* [Okta](#configure-scim-with-okta)
* Microsoft Entra ID (Azure AD)
* Google Workspace
* OneLogin
* CyberArk
* JumpCloud
* PingFederate
* Any SCIM 2.0-compliant provider

## Preparing for SCIM

Before starting your work with SCIM, you'll need to complete this checklist:

1. Configure [SAML](/cloud/manage-access/saml) SSO.
1. Identify your organization's **IdP administrator**, who is responsible for configuring and managing your SCIM integration.
   Specify their contact details when you reach out to support in the next stage of this process.

After completing these steps, you're ready to submit your [support ticket](/evaluate/cloud/support#support-ticket) to enable SCIM.

> **💡 Tip:**
> Adding and removing users
>
> When SCIM is enabled for user management, you can still add and remove users outside of SCIM using the Temporal Cloud interface, until you disable user lifecycle management.
> You can always change a user's or group's Account Role from the Temporal Cloud interface.
>

## Onboarding with SCIM and Okta 

1. Temporal Support enables the SCIM integration on your account.
   Enabling integration automatically emails a configuration link to your Okta administrator.
   This authorizes them to set up the integration.
1. Your Okta administrator opens the supplied link.
   The link leads to step-by-step instructions for configuring the integration.
1. Once configured in Okta, Temporal Cloud will begin to receive SCIM messages and automatically onboard and offboard the users and groups configured in Okta.

Some points to note:

- User and group change events are applied within 10 minutes of them being made in Okta.
- User lifecycle management with SCIM also allows user roles to be derived from group membership.
- Once a group has been synced in Temporal Cloud, you can use `tcld` to assign roles to the group.
  For instructions, see the [User Group Management](https://github.com/temporalio/tcld?tab=readme-ov-file#user-group-management) page.

---

# Manage service accounts

Source: https://docs.temporal.io/cloud/manage-access/service-accounts

> Service Accounts are machine identities that authenticate to Temporal Cloud with API Keys, managed via the Cloud UI or tcld.

Temporal Cloud provides Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) with the option to create machine identities named Service Accounts.

Service Accounts are a type of identity in Temporal Cloud.
Temporal Cloud supports User identities as a representation of a human user who uses Temporal Cloud.
Service Accounts afford Temporal Cloud Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) the ability to create an identity for machine authentication, an identity not associated with a human user.

With the addition of Service Accounts, Temporal Cloud now supports 2 identity types:

- Users (tied to a human, identified by email address or ID)
- Service Accounts (not tied to a human, email address optional, identified by name or ID)

Service Accounts use API Keys as the authentication mechanism to connect to Temporal Cloud.
You should use Service Accounts to represent a non-human identity when authenticating to Temporal Cloud for operations automation or the Temporal SDKs and the Temporal CLI for Workflow Execution and management.

> **💡 Tip:**
>
> Namespace Admins can now manage and create [Namespace-scoped Service Accounts](/cloud/manage-access/service-accounts#scoped), regardless of their Account Role.
>

## Manage Service Accounts

Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) can manage Service Accounts by creating, viewing, updating, deleting Service Accounts using the following tools:

- Temporal Cloud UI
- Temporal CLI with the [Temporal Cloud extension](/cli/cloud)
  - Use `temporal cloud service-account --help` for a list of all service-account commands
- Temporal Cloud CLI (tcld)
  - Use `tcld service-account --help` for a list of all service-account commands

Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) also have the ability to manage API Keys for Service Accounts.

For permissions and API key requirements for Metrics Read-Only Service Accounts, see
[Metrics Read-Only role](/cloud/metrics/openmetrics#metrics-read-only-role).

### Prerequisites

- A Cloud user account with Account Owner or Global Admin [role](/cloud/manage-access/users#account-level-roles) permissions
- Access to the Temporal Cloud UI, the Temporal CLI with the [Temporal Cloud extension](/cli/cloud), or the Temporal Cloud CLI (tcld)
- Enable access to API Keys for your Account
- To manage Service Accounts using the Temporal Cloud CLI (tcld), upgrade to the latest version of tcld (v0.18.0 or higher) using `brew upgrade tcld`.
  - If using a version of tcld less than v0.31.0, enable Service Account commands with `tcld feature toggle-service-account`.

### Create a Service Account

Create a Service Account using the Temporal Cloud UI or the CLI.
While User identities are invited to Temporal Cloud, Service Accounts are created in Temporal Cloud.

**Using the Cloud UI**

1. Go to [Identities page](https://cloud.temporal.io/identities?tab=service-accounts&type=service_account)
2. Click the `Create Service Account` button located near the top of the `Identities` page
3. Provide the following information:
   - **Name** (required)
   - **Description** (optional)
   - **Account Level Role** (required)
   - **Namespace Permissions** (optional)
     - Use this section of the Create Service Account page to grant the Service Account access to individual Namespaces
4. Click `Create Service Account` at the bottom of the page
   - A status message is displayed at the bottom right corner of the screen and on the next screen
   - You will be prompted to create an API Key for the Service Account (optional)
5. (Optional) Create API Key
   - It is recommended to create an API Key for the Service Account right after you create the Service Account, though you can create/manage API Keys for Service Accounts at any time
   - See the API Key [documentation](/cloud/api-keys) for more information on creating and managing API Keys

**Using the Temporal CLI**

To create a Service Account, use the `temporal cloud service-account create` command:

```
temporal cloud service-account create --name "sa_test" --description "this is a test SA" --account-role read
```

This example creates a Service Account with the name `sa_test`, description `this is a test SA`, and a `read` Account Role.

Creating a Service Account requires `--name`. You can optionally assign an Account Role with `--account-role` or grant
Namespace Permissions with `--namespace-access`, in the format `namespace.account=permission`. Repeat
`--namespace-access` to grant access to more than one Namespace.
Creating a Service Account returns the Service Account ID, which you use to retrieve, update, or delete the Service Account.

**Using tcld**

To create a Service Account using tcld, use the `tcld service-account create` command:

```
tcld service-account create -n "sa_test" -d "this is a test SA" --ar "Read"
```

This example creates a Service Account with the name `"sa_test"`, description `"this is a test SA"`, and a `Read` Account Role.

Creating a Service Account requires the following attributes: `name` and `account-role` (as above).
You can also provide the Namespace Permissions for the Service Account using the `—-np` flag.
Creating a Service Account returns the `ServiceAccountId` which is used to retrieve, update, or delete a Service Account.

### View Service Accounts

View a single or all Service Account(s) using the Temporal Cloud UI or the CLI.

**Using the Cloud UI**

Service Accounts are listed on the Identities page, alongside Users.
To locate a Service Account:

1. Go to [Identities page](https://cloud.temporal.io/identities?tab=service-accounts&type=service_account)
2. Select the `Service Accounts` filter

**Using the Temporal CLI**

To view all Service Accounts in your account, use the `temporal cloud service-account list` command:

```
temporal cloud service-account list
```

**Using tcld**

To view all Service Accounts in your account using tcld, use the `tcld service-account list` command:

```
tcld service-account list
```

### Delete a Service Account

Delete a Service Account using the Temporal Cloud UI or the CLI. When you delete a Service Account, all associated API keys are automatically deleted as well.
Therefore, you don't need to manually remove API keys after deleting a Service Account.

**Using the Cloud UI**

1. Go to [Identities page](https://cloud.temporal.io/identities?tab=service-accounts&type=service_account)
2. Find the relevant Service Account
3. Select the vertical ellipsis menu in the Service Account row
4. Select `Delete`
5. Confirm the delete action when prompted

**Using the Temporal CLI**

To delete a Service Account, use the `temporal cloud service-account delete` command:

```
temporal cloud service-account delete --service-account-id "e9d87418221548"
```

Run the Service Account list command to confirm the Service Account has been removed from the account.
The Service Account is deleted when it is no longer visible in the output of the list command.

**Using tcld**

To delete a Service Account using tcld, use the `tcld service-account delete` command:

```
tcld service-account delete --service-account-id "e9d87418221548"
```

Use the tcld Service Account list command to validate the Service Account has been removed from the account.
The Service Account is deleted when it is no longer visible in the output of the list command.

### Update a Service Account 

Update a Service Account's description using the Temporal Cloud UI or the CLI.

**Using the Cloud UI**

1. Go to [Identities page](https://cloud.temporal.io/identities?tab=service-accounts&type=service_account)
2. Find the relevant Service Account
3. Select the vertical ellipsis menu in the Service Account row
4. Select `Edit`
5. Make changes to the Service Account
   - You can change the Service Account's name, description, Account Level Role, and Namespace Permissions
6. Click the `Save` button located in the bottom left of the screen
   - A status message is displayed at the bottom right corner of the screen

**Using the Temporal CLI**

The `temporal cloud service-account update` command changes the name, description, Account Role, and Namespace access of a Service Account:

```
temporal cloud service-account update --service-account-id "2f68507677904e09b9bcdbf93380bb95" \
  --description "new description"
```

Use `--account-role` to change the Account Role and `--namespace-access` to change Namespace access. For a Namespace-scoped Service Account, use `--namespace-permission` instead.

**Using tcld**

Three different commands exist to help users update a Service Account using tcld:

- `tcld service-account update`: to update a Service Account's name or description field
- `tcld service-account set-account-role`: to update a Service Account's Account Role
- `tcld service-account set-namespace-permissions`: to update a Service Account's Namespace Permissions

Example:

```
tcld service-account update --id "2f68507677904e09b9bcdbf93380bb95" -d "new description"
```

## Namespace-scoped Service Accounts 

There is a special type of Service Account, called a Namespace-scoped Service Account, which shares the
same functionality as the Service Accounts above, but is limited (or scoped) to a single namespace.

In particular, a Namespace-scoped Service Account must _always_ have:

- A `Read` Account Role
- A single Namespace Permission

Note that a Namespace-scoped Service Account cannot be reassigned to a different Namespace after creation, but its Namespace permission can be modified (for example, from `Read` to `Write`).

Namespace-scoped Service Accounts are useful in situations when you need to restrict a client's access to a single Namespace.

You can retrieve, update, and delete a Namespace-scoped Service Account using the same process and commands as above, but creation is slightly different.

### Permissions
Unlike regular Service Accounts, which require a Global Admin or Account Owner role, Namespace-scoped Service Accounts can be created and managed by Namespace Admins.
For example, an Account Developer with Namespace Admin for `test_ns` can create a Service Account scoped to `test_ns`.

Global Admins and Account Owners can also create Namespace-scoped Service Accounts, as they implicitly have Namespace Admin rights for all Namespaces.

### Create a Namespace-scoped Service Account

As with regular Service Accounts, Namespace-scoped Service Accounts can be created using the Temporal Cloud UI or the CLI.

#### Using the Cloud UI 

Currently, creating a Namespace-scoped Service Account from the Temporal Cloud UI happens on an individual [Namespace](/cloud/namespaces#manage-namespaces) page.
If the current Namespace has API key authentication enabled, then there will be a `Generate API Key` button as a banner on the top of the Namespace page or in the `Authentication` section.

By clicking on the `Generate API Key` button, a Namespace-scoped Service Account will be automatically created for the given Namespace (if one does not already exist) and an associated API key will be displayed. This key will have the maximum expiration time, which is 2 years.

The resulting Namespace-scoped Service Account will be named `<namespace>-service-account` and will have an `Admin` Namespace permission by default.

#### Using the Temporal CLI

To create a Namespace-scoped Service Account, use the `temporal cloud service-account create-namespace-scoped` command:

```
temporal cloud service-account create-namespace-scoped --name "test-scoped-sa" \
  --namespace "test-ns.<account-id>" --namespace-permission admin
```

#### Using tcld

To create a Namespace-scoped Service Account with tcld, use the `tcld service-account create-scoped` command:

```
tcld service-account create-scoped -n "test-scoped-sa" --np "test-ns=Admin"
```

This example creates a Namespace-scoped Service Account for the Namespace `test-ns`, named `test-scoped-sa`, with `Admin` Namespace Permission.
Note that the Account Role is omitted, since Namespace-scoped Service Accounts always have a `Read` Account Role.

### Lifecycle

When a Namespace is deleted, all associated Namespace-scoped Service Accounts and their associated API keys are automatically deleted as well.
Therefore, you do not need to manually remove Namespace-scoped Service Accounts and their API keys after deleting a Namespace.

## Project-scoped Service Accounts

> **Pre-release**

Project-scoped Service Accounts can be created and managed within a Project. Unlike Account-scoped Service Accounts, they can only access resources within their Project, making them useful for workloads that shouldn't have Account-wide permissions. For more information, see [Add resources to a Project](/cloud/projects/#add-resources-to-a-project).

---

# Manage user groups

Source: https://docs.temporal.io/cloud/manage-access/user-groups

> Assign account-level roles and Namespace permissions to a group of users at once, and see how SCIM groups interact with them.

## What are user groups?

User groups can be used to help manage sets of users that should have the same
access. Instead of separately assigning the same role to individual users, a user group can be
created, assigned the desired roles, and then users added to the user group. This
eases the toil of managing individual user permissions and can simplify access management. When
a new role is needed, it can be added to the group once and all users' access will reflect the
new role.

User groups can be assigned both [account-level roles](/cloud/manage-access/users#account-level-roles) and [namespace-level permissions](/cloud/manage-access/users#namespace-level-permissions).

One user can be assigned to many groups. In the event that a user's group memberships have multiple roles for the same resource, the user will have an effective role of the most permissive of the permissions. For example if `Group A` grants a read-only role to a namespace, but `Group B` grants a write role to a namespace then a user that belongs to both `Group A` and `Group B` would have the write role to the namespace.

[Service accounts](/cloud/manage-access/service-accounts) cannot be assigned to user groups.

Only users with the Account Owner or Global Admin account-level [role](/cloud/manage-access/users#account-level-roles) can manage user groups.

## How SCIM groups work with user groups 

[SCIM groups](/cloud/manage-access/scim) work similarly to user groups with respect to role assignment. Unlike a user group, the lifecycle of a SCIM group is fully managed by the SCIM integration which means:

1. SCIM groups cannot be created except through the SCIM integration
1. SCIM groups cannot be deleted except through the SCIM integration
1. SCIM group membership is managed through the SCIM integration

User groups and SCIM groups can be used simultaneously in a single Temporal Cloud account. One user may belong to multiple SCIM groups and to multiple user groups.

Using user group and SCIM groups together can be useful when the groups defined in the identity provider (IDP) don't map cleanly to the access you need to grant in Temporal Cloud. Instead of having to update the IDP (which is often sensitive and time-consuming), you can use Temporal Cloud user groups to manage access.

> **ℹ️ Info:**
>
> All user group administration requires an Account Owner or Global Admin account-level [role](/cloud/manage-access/users#account-level-roles).
>

## How to create a user group in your Temporal Cloud account 

User group names must be 3-64 characters long and can only contain lowercase letters, numbers, hyphens, and underscores.

**Web UI**

1. Navigate to the [identities page](https://cloud.temporal.io/identities?tab=groups&type=group)
1. Click the Create Group button
1. Name the group
1. Assign an account-level role to the group (you can assign namespace-level permissions after the group is created)
1. Click Save

**Temporal CLI**

Use the command that matches how the group is backed:

- [`temporal cloud user-group create-cloud-group`](/cli/command-reference/cloud/user-group#create-cloud-group) for a group managed in Temporal Cloud
- [`temporal cloud user-group create-google-group`](/cli/command-reference/cloud/user-group#create-google-group) for a group backed by a Google group
- [`temporal cloud user-group create-scim-group`](/cli/command-reference/cloud/user-group#create-scim-group) for a group backed by SCIM

**tcld**

See the [`tcld` user-group create](/cloud/tcld/user-group/#create) command reference for details.

**Terraform**

See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.

## How to assign roles to a user group 

**Web UI**

To edit the account role of a group:
  1. Navigate to the [identities page](https://cloud.temporal.io/identities?tab=groups&type=group)
  1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
  1. Click Edit Group
  1. Click the Account Role dropdown
  1. Select a new account role
  1. Click Save

To add namespace permissions to a group:
  1. Navigate to the [identities page](https://cloud.temporal.io/identities?tab=groups&type=group)
  1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
  1. Click Edit Group
  1. Click Add Namespaces
  1. Under Grant Access to a Namespace, search for the namespace you’d like to add permissions for
  1. Select the namespace
  1. Click the pencil to edit the permissions for the selected namespace
  1. Click Save

To edit or remove namespace permissions from a group:
  1. Click Edit Group
  1. Click the pencil on a permission to edit it, or the trash can to delete it
  1. Click Save

**Temporal CLI**

Account-level roles and Namespace-level permissions are set separately:

- [`temporal cloud user-group set-account-role`](/cli/command-reference/cloud/user-group#set-account-role)
- [`temporal cloud user-group set-namespace-permissions`](/cli/command-reference/cloud/user-group#set-namespace-permissions)

**tcld**

See the [`tcld` user-group set-access](/cloud/tcld/user-group/#set-access) command reference for details.

**Terraform**

See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.

## How to manage users in a group 

**Web UI**

To add users to the group:
  1. Navigate to the [identities page](https://cloud.temporal.io/identities?tab=groups&type=group)
  1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
  1. Click Edit Group
  1. Under Members, search for the user you’d like to add 
  1. Select the user
  1. Click Save
To remove a user from the group:
  1. Click Edit Group
  1. Under Members, click the X next to the user you’d like to remove
  1. Click Save

**Temporal CLI**

See the [`temporal cloud user-group members add`](/cli/command-reference/cloud/user-group#members-add),
[`temporal cloud user-group members remove`](/cli/command-reference/cloud/user-group#members-remove), and
[`temporal cloud user-group members list`](/cli/command-reference/cloud/user-group#members-list) command reference for details.

**tcld**

See the [`tcld` user-group add-users](/cloud/tcld/user-group/#add-users) and the [`tcld` user-group remove-users](/cloud/tcld/user-group/#remove-users) command reference for details.

**Terraform**

See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.

## Delete a user group

**Web UI**

  1. Navigate to the [identities page](https://cloud.temporal.io/identities?tab=groups&type=group)
  1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
  1. Click the dropdown next to the edit button
  1. Click Delete
  1. Confirm by clicking Delete

**Temporal CLI**

See the [`temporal cloud user-group delete`](/cli/command-reference/cloud/user-group#delete) command reference for details.

**tcld**

See the [`tcld` user-group delete](/cloud/tcld/user-group/#delete) command reference for details.

**Terraform**

See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.

---

# Manage users

Source: https://docs.temporal.io/cloud/manage-access/users

> Invite users, set account-level roles and Namespace-level permissions, and remove users from a Temporal Cloud account with the Web UI, the CLI, or the Cloud Ops API.

- [How to invite users to your Temporal Cloud account](#invite-users)
- [What are the account-level roles?](#account-level-roles)
- [What are the Namespace-level permissions?](#namespace-level-permissions)
- [How to update an account-level Role in Temporal Cloud](#update-roles)
- [How to update Namespace-level permissions in Temporal Cloud](#update-permissions)
- [How to delete a user from your Temporal Cloud account](#delete-users)
- [How to troubleshoot account access issues](#troubleshoot-access)

## How to invite users to your Temporal Cloud account 

# User management

> Learn how to manage user invitations for Temporal Cloud

**Web UI**

To invite users using the Temporal Cloud UI:

1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page, select **Create Users** in the upper-right portion of the window.
1. On the **Create Users** page in the **Email Addresses** box, type or paste one or more email addresses.
1. In **Account-Level Role**, select a [Role](/cloud/manage-access/roles-and-permissions#account-level-roles). The Role
   applies to all users whose email addresses appear in **Email Addresses**.
1. If the account has any Namespaces, they are listed under **Grant access to Namespaces**. To add a permission, select
   the checkbox next to a Namespace, and then select a
   [permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions). Repeat as needed.
1. When all permissions are assigned, select **Send Invite**.

**Temporal CLI**

Use the [`temporal cloud user invite`](/cli/command-reference/cloud/user#invite) command. Specify the user's email, an
account-level role, and optionally one or more Namespace permissions.

Available account roles: `owner` | `admin` | `developer` | `finance-admin` | `read` | `metrics-read`.

Available Namespace permissions: `admin` | `write` | `read`.

```command
temporal cloud user invite \
  --email <user@example.com> \
  --account-role <role> \
  --namespace-access <namespace.account>=<permission>
```

Repeat `--namespace-access` to grant permissions on more than one Namespace. `--email` takes a single address, so invite
one user per command:

```command
temporal cloud user invite \
  --email user1@example.com \
  --account-role developer \
  --namespace-access ns1.my-account=admin \
  --namespace-access ns2.my-account=write
```

**tcld**

Use the [`tcld user invite`](/cloud/tcld/user/#invite) command. Specify the user's email, an account-level role, and
optionally one or more Namespace permissions.

Available account roles: `admin` | `developer` | `read`.

Available Namespace permissions: `Admin` | `Write` | `Read`.

```command
tcld user invite \
  --user-email <user@example.com> \
  --account-role <role> \
  --namespace-permission <namespace>=<permission>
```

You can invite multiple users and assign multiple Namespace permissions in a single command:

```command
tcld user invite \
  --user-email user1@example.com \
  --user-email user2@example.com \
  --account-role developer \
  --namespace-permission ns1=Admin \
  --namespace-permission ns2=Write
```

### Frequently asked questions

#### Can multiple Temporal Cloud accounts share the same email domain?

Yes. Multiple Temporal Cloud accounts can coexist with users from the same email domain.
Each account has its own independent SAML configuration, tied to its unique Account Id.
We recommend configuring [SAML](/cloud/manage-access/saml) for each account independently.
For the smoother login experience, you can configure SAML for each account separately and use IdP-initiated login: you click the relevant app tile in your identity provider's portal to access the Temporal Cloud account associated with your email address directly.

#### Can the same email be used across different Temporal Cloud accounts?

No. Each email address can only be associated with a single Temporal Cloud account.
If you need access to multiple accounts, you’ll need a separate invite for each one using a different email address.

#### Can I use Google or Microsoft SSO after signing up with email and password?

If you originally signed up for Temporal Cloud using an email and password, you won’t be able to log in using Google or Microsoft single sign-on.

If you prefer SSO, ask your Account Owner to delete your current user and send you a new invitation.
During re-invitation, be sure to sign up using your preferred authentication method.

Use the [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users) endpoint to invite a user.

```
POST /cloud/users
```

The request body includes a `spec` with the following fields:

- `spec.email` — The email address of the user to invite.
- `spec.access.account_access.role` — The account-level role to assign.
- `spec.access.namespace_accesses` — A map of Namespace names to permissions.

Available roles: `ROLE_ADMIN` | `ROLE_DEVELOPER` | `ROLE_READ` | `ROLE_OWNER` | `ROLE_FINANCE_ADMIN`.

Available Namespace permissions: `PERMISSION_ADMIN` | `PERMISSION_WRITE` | `PERMISSION_READ`.

The new users receive an email with a link to accept the invitation and complete their setup. The new user must use this
link to sign up to be added to your account unless the account has a SAML configuration. If your account has a SAML
configuration, the new user can sign in using their existing SAML credentials and be included in the account
automatically.

> **⚠️ Caution:**
>
> The new user must use the same authentication method they originally signed up with to sign in to Temporal Cloud. If
> they used single sign-on (SSO), they must use the same SSO provider to sign in to Temporal Cloud. If they used email and
> password authentication, they must use the same email and password to sign in to Temporal Cloud, and cannot use SSO,
> even if the underlying email address is the same.
>

## What are the account-level roles for users in Temporal Cloud? 

When an Account Owner or Global Admin invites a user to join an account, they select one of the following roles for that user:

- **Global Admin**
  - Has full administrative permissions across the account, including users and usage
  - Can create and manage [Namespaces](/namespaces) and [Nexus Endpoints](/nexus/endpoints)
  - Has Namespace Admin [permissions](#namespace-level-permissions) on all Namespaces in the account.
    This permission cannot be revoked
- **Developer**
  - Can create Namespaces
  - Is granted [Namespace Admin](/cloud/manage-access/users#namespace-level-permissions) permission for each Namespace they create.
    This permission can be revoked
  - Can create and manage Nexus Endpoints where they are a [Namespace Admin](/cloud/manage-access/users#namespace-level-permissions) on the Endpoint's target Namespace.
    Changing an Endpoint's target Namespace requires Namespace Admin on both the current target Namespace and the new target Namespace.
- **Read-Only**
  - Can read information
  - Can be granted Namespace [permissions](#namespace-level-permissions), for example to read or write Workflow state in a given Namespace
  - Can view all Nexus Endpoints in the account, which have separate [runtime access controls](/nexus/security#runtime-access-controls)

In addition, there are two roles that the Global Admin cannot assign:

- **Account Owner**
  - Has full administrative permissions across the account, including users, usage and [billing](/cloud/billing-and-usage)
  - Can create and manage Namespaces and Nexus Endpoints
  - Has Namespace Admin [permissions](#namespace-level-permissions) on all [Namespaces](/namespaces) in the account.
    This permission cannot be revoked
- **Finance Admin**
  - Has permissions to view [billing](/cloud/billing-and-usage) information and update payment information
  - Otherwise, has the same permissions as Account Read-only users
  - Only an Account Owner can assign Finance Admin to a user, group, or Service Account

> **📝 Note:**
> Default Role
>
> When the account is created, the initial user who logs in is automatically assigned the Account Owner role.
> If your account does not have an Account Owner, please reach out to [Support](https://temporalsupport.zendesk.com/) to assign the appropriate individual to this role.
>

## Using the Account Owner role

The Account Owner role (that is, users with the Account Owner system role) holds the highest level of access in the system.
This role configures account-level parameters and manages Temporal billing and payment information.
It allows users to perform all actions within the Temporal Cloud account.

> **💡 Tip:**
> Best Practices
>
> Temporal strongly recommends the following precautions when assigning the Account Owner role to users:
>
> - Assign the role to at least two users in your organization.
>   Otherwise, limit the number of users with this role.
> - Associate a person’s direct email address to the Account Owner, rather than a shared or generic address, so Temporal Support can contact the right person in urgent situations.
>
> This latter rule is useful for anyone on your team who may need to be contacted urgently, regardless of their Account role.
>

## What are the Namespace-level permissions for users in Temporal Cloud? 

An Account Owner or Global Admin can assign permissions for any [Namespace](/namespaces) in an account.
A Developer can assign permissions for a Namespace they create.

For a Namespace, a user can have one of the following permissions:

- **Namespace Admin:**
  - Can [manage the Namespace](/cloud/namespaces#manage-namespaces) including identities and permissions
  - Can create, rename, update, and delete [Workflows](/workflows) within the Namespace
- **Write:**
  - Can create, rename, update, and delete [Workflows](/workflows) within the Namespace
- **Read-Only:**
  - Can only read information from the Namespace

## How to update an account-level role in Temporal Cloud 

With Global Admin or Account Owner privileges, you can update any user's account-level [role](#account-level-roles) using either the Web UI or the CLI.
The Account Owner role can only be granted by existing Account Owners.

For security reasons, changes to the Account Owner role must be made through Temporal Support.
To change or delete an Account Owner, you must submit a [support ticket](https://temporalsupport.zendesk.com/).

### How to update an account-level role using Web UI

1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page, select the user.
1. On the user profile page, select **Edit User**.
1. On the **Edit User** page in **Account Level Role**, select the role.
1. Select **Save**.

### How to update an account-level role using the CLI

**Temporal CLI**

For details, see the [`temporal cloud user set-account-role`](/cli/command-reference/cloud/user#set-account-role) command.

**tcld**

For details, see the [tcld user set-account-role](/cloud/tcld/user/#set-account-role) command.

## How to update Namespace-level permissions in Temporal Cloud 

You can update Namespace-level [permissions](#namespace-level-permissions) by using either the Web UI or the CLI.

### How to use the Web UI to update a user's permissions across multiple Namespaces

1. In Temporal Web UI, select **Namespaces** in the left portion of the window.
1. On the **Namespaces** page, select the Namespace.
1. If necessary, scroll down to the list of permissions
1. On the user profile page in **Namespace permissions**, select the Namespace.
1. On the Namespace page in **Account Level Role**, select the role.
1. Select **Save**.

### How to use the Web UI to update permissions for multiple users within a single Namespace

> **📝 Note:**
>
> A user with the Account Owner or Global Admin account-level [role](#account-level-roles) has Namespace Admin permissions for all Namespaces.
>

1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page in the **Users** tab, select the user.
1. On the user profile page, select **Edit User**.
1. On the **Edit User** page in **Namespace permissions**, change the permissions for one or more Namespaces.
1. Select **Save**.

### How to use the CLI to update Namespace-level permissions

**Temporal CLI**

For details, see the [`temporal cloud user set-namespace-permissions`](/cli/command-reference/cloud/user#set-namespace-permissions) command.

**tcld**

For details, see the [tcld user set-namespace-permissions](/cloud/tcld/user/#set-namespace-permissions) command.

## How to delete a user from your Temporal Cloud account 

You can delete a user from your Temporal Cloud Account by using either the Web UI or the CLI.

> **ℹ️ Info:**
>
> To delete a user, a user must have the Account Owner or Global Admin account-level [role](#account-level-roles).
>

### How to delete a user using Web UI

1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page, find the user and, on the right end of the row, select **Delete**.
1. In the **Delete User** dialog, select **Delete**.

You can delete a user in two other ways in Web UI:

- User profile page: Select the down arrow next to **Edit User** and then select **Delete**.
- **Edit User** page: Select **Delete User**.

### How to delete a user using the CLI

**Temporal CLI**

For details, see the [`temporal cloud user delete`](/cli/command-reference/cloud/user#delete) command.

**tcld**

For details, see the [tcld user delete](/cloud/tcld/user/#delete) command.

## Account-level roles and Namespace-level permissions 

Temporal account-level roles and Namespace-level permissions provide access to specific Temporal Workflow and Temporal Cloud operational APIs.
The following table provides the API details associated with each account-level role and Namespace-level permission.

#### Account-level role details

This table provides API-level details for the permissions granted to a user through account-level roles. These permissions are configured per user.

| Permission                        | Read-only | Developer | Finance Admin | Global Admin | Account Owner |
| --------------------------------- | --------- | --------- | ------------- | ------------ | ------------- |
| CountIdentities                   | ✅         | ✅         | ✅             | ✅            | ✅             |
| CreateAccountAuditLogSink         |           |           |               | ✅            | ✅             |
| CreateAPIKey                      | ✅         | ✅         | ✅             | ✅            | ✅             |
| CreateNamespace                   |           | ✅         |               | ✅            | ✅             |
| CreateNexusEndpoint               |           | ✅         |               | ✅            | ✅             |
| CreateServiceAccount              |           |           |               | ✅            | ✅             |
| CreateServiceAccountAPIKey        |           |           |               | ✅            | ✅             |
| CreateStripeCustomerPortalSession |           |           | ✅             |              | ✅             |
| CreateUser                        |           |           |               | ✅            | ✅             |
| DeleteAccountAuditLogSink         |           |           |               | ✅            | ✅             |
| DeleteAPIKey                      | ✅         | ✅         | ✅             | ✅            | ✅             |
| DeleteNexusEndpoint               |           | ✅         |               | ✅            | ✅             |
| DeleteServiceAccount              |           |           |               | ✅            | ✅             |
| DeleteUser                        |           |           |               | ✅            | ✅             |
| GetAccount                        | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetAccountAuditLogSink            |           |           |               | ✅            | ✅             |
| GetAccountAuditLogSinks           |           |           |               | ✅            | ✅             |
| GetAccountFeatureFlags            | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetAccountLimits                  | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetAccountSettings                | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetAccountUsage                   |           |           |               | ✅            | ✅             |
| GetAPIKey                         | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetAPIKeys                        | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetAsyncOperation                 | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetAuditLogs                      |           |           |               | ✅            | ✅             |
| GetDecodedCertificate             | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetIdentities                     | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetIdentity                       | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetNamespaces                     | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetNamespacesUsage                |           |           |               | ✅            | ✅             |
| GetNexusEndpoint                  | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetNexusEndpoints                 | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetRegion                         | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetRegions                        | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetRequestStatus                  | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetRequestStatuses                |           |           |               | ✅            | ✅             |
| GetRequestStatusesForNamespace    | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetRequestStatusesForUser         | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetRoles                          | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetRolesByPermissions             | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetServiceAccount                 | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetServiceAccounts                | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetStripeInvoice                  |           |           | ✅             |              | ✅             |
| GetUser[‡](/cloud/manage-access/permissions-reference#user-authorization-behavior) | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetUsers[‡](/cloud/manage-access/permissions-reference#user-authorization-behavior) | ✅         | ✅         | ✅             | ✅            | ✅             |
| GetUsersWithAccountRoles          | ✅         | ✅         | ✅             | ✅            | ✅             |
| InviteUsers                       |           |           |               | ✅            | ✅             |
| ListCreditLedgerEntries           |           |           | ✅             |              | ✅             |
| ListGrants                        |           |           | ✅             |              | ✅             |
| ListMetronomeInvoices             |           |           | ✅             |              | ✅             |
| ListMetronomeInvoicesForNamespace |           |           | ✅             |              | ✅             |
| ListNamespaces                    | ✅         | ✅         | ✅             | ✅            | ✅             |
| ListPromotionGrantBalances        |           |           | ✅             |              | ✅             |
| ResendUserInvite                  |           |           |               | ✅            | ✅             |
| SetAccountSettings                |           |           |               | ✅            | ✅             |
| SyncCurrentUserInvite             | ✅         | ✅         | ✅             | ✅            | ✅             |
| UpdateAccount                     |           |           |               | ✅            | ✅             |
| UpdateAccountAuditLogSink         |           |           |               | ✅            | ✅             |
| UpdateAPIKey                      | ✅         | ✅         | ✅             | ✅            | ✅             |
| UpdateNexusEndpoint               |           | ✅         |               | ✅            | ✅             |
| UpdateServiceAccount              |           |           |               | ✅            | ✅             |
| UpdateUser                        |           |           |               | ✅            | ✅             |
| ValidateAccountAuditLogSink       |           |           |               | ✅            | ✅             |

#### Namespace-level permissions details

This table provides API-level details for the permissions granted to a user through Namespace-level permissions.
These permissions are configured per Namespace per user.

> **📝 Note:**
>
> Account Owners and Global Admins inherit Namespace Admin permissions on all Namespaces.
>

| Permission                         | Read | Write | Namespace Admin |
| ---------------------------------- | ---- | ----- | --------------- |
| CountWorkflowExecutions            | ✅    | ✅     | ✅               |
| CreateExportSink                   |      |       | ✅               |
| CreateSchedule                     |      | ✅     | ✅               |
| DeleteExportSink                   |      |       | ✅               |
| DeleteNamespace                    |      |       | ✅               |
| DeleteSchedule                     |      | ✅     | ✅               |
| DescribeBatchOperation             | ✅    | ✅     | ✅               |
| DescribeNamespace                  | ✅    | ✅     | ✅               |
| DescribeSchedule                   | ✅    | ✅     | ✅               |
| DescribeTaskQueue                  | ✅    | ✅     | ✅               |
| DescribeWorkflowExecution          | ✅    | ✅     | ✅               |
| FailoverNamespace                  |      |       | ✅               |
| GetExportSink                      | ✅    | ✅     | ✅               |
| GetExportSinks                     | ✅    | ✅     | ✅               |
| GetNamespace                       | ✅    | ✅     | ✅               |
| GetNamespaceUsage                  | ✅    | ✅     | ✅               |
| GetReplicationStatus               | ✅    | ✅     | ✅               |
| GetSearchAttributes                | ✅    | ✅     | ✅               |
| GetUsersForNamespace               | ✅    | ✅     | ✅               |
| GetWorkerBuildIdCompatibility      | ✅    | ✅     | ✅               |
| GetWorkerTaskReachability          | ✅    | ✅     | ✅               |
| GetWorkflowExecutionHistory        | ✅    | ✅     | ✅               |
| GetWorkflowExecutionHistoryReverse | ✅    | ✅     | ✅               |
| GlobalizeNamespace                 |      |       | ✅               |
| ListBatchOperations                | ✅    | ✅     | ✅               |
| ListClosedWorkflowExecutions       | ✅    | ✅     | ✅               |
| ListExportSinks                    | ✅    | ✅     | ✅               |
| ListFailoverHistoryByNamespace     | ✅    | ✅     | ✅               |
| ListOpenWorkflowExecutions         | ✅    | ✅     | ✅               |
| ListReplicaStatus                  | ✅    | ✅     | ✅               |
| ListScheduleMatchingTimes          | ✅    | ✅     | ✅               |
| ListSchedules                      | ✅    | ✅     | ✅               |
| ListTaskQueuePartitions            | ✅    | ✅     | ✅               |
| ListWorkflowExecutions             | ✅    | ✅     | ✅               |
| PatchSchedule                      |      | ✅     | ✅               |
| PollActivityTaskQueue              |      | ✅     | ✅               |
| PollWorkflowTaskQueue              |      | ✅     | ✅               |
| QueryWorkflow                      | ✅    | ✅     | ✅               |
| RecordActivityTaskHeartbeat        |      | ✅     | ✅               |
| RecordActivityTaskHeartbeatById    |      | ✅     | ✅               |
| RenameCustomSearchAttribute        |      |       | ✅               |
| RequestCancelWorkflowExecution     |      | ✅     | ✅               |
| ResetStickyTaskQueue               |      | ✅     | ✅               |
| ResetWorkflowExecution             |      | ✅     | ✅               |
| RespondActivityTaskCanceled        |      | ✅     | ✅               |
| RespondActivityTaskCanceledById    |      | ✅     | ✅               |
| RespondActivityTaskCompleted       |      | ✅     | ✅               |
| RespondActivityTaskCompletedById   |      | ✅     | ✅               |
| RespondActivityTaskFailed          |      | ✅     | ✅               |
| RespondActivityTaskFailedById      |      | ✅     | ✅               |
| RespondQueryTaskCompleted          |      | ✅     | ✅               |
| RespondWorkflowTaskCompleted       |      | ✅     | ✅               |
| RespondWorkflowTaskFailed          |      | ✅     | ✅               |
| SetUserNamespaceAccess             |      |       | ✅               |
| SignalWithStartWorkflowExecution   |      | ✅     | ✅               |
| SignalWorkflowExecution            |      | ✅     | ✅               |
| StartBatchOperation                |      | ✅     | ✅               |
| StartWorkflowExecution             |      | ✅     | ✅               |
| StopBatchOperation                 |      | ✅     | ✅               |
| TerminateWorkflowExecution         |      | ✅     | ✅               |
| UpdateExportSink                   |      |       | ✅               |
| UpdateNamespace                    |      |       | ✅               |
| UpdateSchedule                     |      | ✅     | ✅               |
| UpdateSearchAttributes             |      |       | ✅               |
| UpdateUserNamespacePermissions     |      |       | ✅               |
| ValidateExportSink                 |      |       | ✅               |
| ValidateGlobalizeNamespace         |      |       | ✅               |

> **📝 Note:**
> UpdateNamespace settings
>
> `UpdateNamespace` requires Namespace Admin permission and covers these settings:
> - [Retention period](/temporal-service/temporal-server#retention-period)
> - [API key auth](/cloud/api-keys#namespace-authentication)
> - [mTLS certificates](/cloud/certificates)
> - [Certificate filters](/cloud/certificates#manage-certificate-filters)
> - [Codec server](/production-deployment/data-encryption)
> - [Connectivity rules](/cloud/connectivity)
> - [Custom Search Attributes](/search-attribute#custom-search-attribute)
> - [Provisioned capacity (TRUs)](/cloud/capacity-modes#provisioned-capacity)
> - [High Availability](/cloud/high-availability)
>

## How to troubleshoot account access issues 

### Why can't I sign in after my email domain changed? 

If your organization changed its email domain (for example, from `@oldcompany.com` to `@newcompany.com`), you may be unable to sign in to Temporal Cloud with your existing account.

**Why this happens:**
When you sign in using "Continue with Google" or "Continue with Microsoft", Temporal Cloud identifies your account by your email address.
If your email address changes, Temporal Cloud sees this as a different identity and cannot match it to your existing account.

**How to resolve this:**
[Create a support ticket](/evaluate/cloud/support#support-ticket) with the following information:

- Your previous email address (the one originally used to access Temporal Cloud)
- Your new email address
- Your Temporal Cloud Account Id (if known)

Temporal Support can update your account to use your new email address.

> **💡 Tip:**
> Use SAML for enterprise identity management
>
> If your organization frequently changes email domains or wants centralized control over user authentication, consider using [SAML authentication](/cloud/manage-access/saml).
> With SAML, your identity provider (IdP) manages user identities, and email domain changes can be handled within your IdP without affecting Temporal Cloud access.
>

---

# Set up Cloud metrics with OpenMetrics

Source: https://docs.temporal.io/cloud/metrics/openmetrics

> Export metrics from Temporal Cloud using the OpenMetrics standard and third party integrations.

> **💡 Tip:**
> PRICING
>
> Future pricing may apply to high-volume usage that exceeds standard [limits](/cloud/metrics/openmetrics/api-reference#api-limits).
>

Temporal Cloud's [OpenMetrics](https://openmetrics.io/) endpoint provides operational metrics for your Temporal Cloud workloads in industry-standard Prometheus format, enabling comprehensive monitoring across Namespaces, Workflows, and Task Queues with your existing observability stack.

## Metrics Read-Only role 

Metrics Read-Only is an account-level role intended for Service Accounts that scrape the OpenMetrics endpoint. It grants
read access to the OpenMetrics endpoint but no other permissions.

An Account Owner or Global Admin must create and manage the Service Account and all of its API keys. A Metrics Read-Only
Service Account cannot create, update, or delete API keys, including its own.

## Quickstart

Stream metrics from Temporal Cloud into your observability tool in about 5 minutes.

**Prerequisites**

- An **Account Owner** or **Global Admin** role on the Temporal Cloud account. The [Metrics Read-Only role](#metrics-read-only-role) can only be granted by these roles. A Namespace Admin cannot complete these steps.
- An account in the observability tool you want to use, such as Datadog, Grafana Cloud, New Relic, ClickStack, or self-hosted Prometheus.

**Steps**

1. **Create a Service Account with the Metrics Read-Only role.**

   In the Temporal Cloud UI, go to **Settings → Service Accounts → Create Service Account** and assign the **Metrics Read-Only** account-level role.

2. **Generate an API key for the Service Account.**

   Open the Service Account and create an API key. Copy the key and store it somewhere secure. It is shown only once.

3. **Verify the endpoint is reachable.**

   ```shell
   curl -H "Authorization: Bearer <API_KEY>" https://metrics.temporal.io/v1/metrics
   ```

   You should see OpenMetrics-formatted output beginning with `# TYPE temporal_cloud_v1_...`.

> **📝 Note:**
> `metrics.temporal.io` is for scrapers, not browsers
>
>    The endpoint requires an `Authorization: Bearer <API key>` header on every request. There is no browser UI. Opening `https://metrics.temporal.io` or `https://metrics.temporal.io/v1/metrics` directly in a browser returns `Jwt is missing`. Configure the endpoint inside your observability tool instead.
>

4. **Configure your observability tool.**

   Paste the API key into the integration for your tool of choice. See [Metrics integrations](/cloud/metrics/openmetrics/metrics-integrations) for tool-specific setup:

   - [Datadog](/cloud/metrics/openmetrics/metrics-integrations#datadog)
   - [Elastic](/cloud/metrics/openmetrics/metrics-integrations#elastic)
   - [Grafana Cloud](/cloud/metrics/openmetrics/metrics-integrations#grafana-cloud)
   - [New Relic](/cloud/metrics/openmetrics/metrics-integrations#new-relic)
   - [ClickStack](/cloud/metrics/openmetrics/metrics-integrations#clickstack)
   - [SigNoz](/cloud/metrics/openmetrics/metrics-integrations#signoz)
   - [Parseable](/cloud/metrics/openmetrics/metrics-integrations#parseable)
   - [Self-hosted Prometheus or OpenTelemetry Collector](/cloud/metrics/openmetrics/metrics-integrations#prometheus-grafana)

Metrics begin populating in your tool within a few minutes.

## Quick links
* [Integrations](/cloud/metrics/openmetrics/metrics-integrations) - Get started exporting metrics with common integrations
* [API Documentation](/cloud/metrics/openmetrics/api-reference) - Endpoint specification and advanced configuration
* [Metrics Reference](/cloud/metrics/openmetrics/metrics-reference) - Complete catalog of all metrics with descriptions and labels
* [OpenMetrics FAQ](/cloud/metrics/openmetrics/faq) - Answers about querying, scrape limits, and missing metrics

## Overview
Temporal Cloud OpenMetrics exposes 50+ metrics covering workflow lifecycles, task queue operations, service performance, and system limits. All metrics are aggregated over one-minute windows and available for scraping within three minutes. Each scrape returns only the most recently completed one-minute window—configure your monitoring system to retain what it scrapes.

* [Set up authentication and scraping](/cloud/metrics/openmetrics/api-reference#authentication) with the API documentation. 
* Browse the [complete metrics catalog](/cloud/metrics/openmetrics/metrics-reference) for descriptions and labels. 

## API key authentication
Create a Service Account with the "Metrics Read-Only" role and generate an API key. See the [Quickstart](#quickstart) above for step-by-step instructions. API keys work with standard HTTPS, with no certificate rotation or distribution required.

## Global endpoint
This is a single endpoint at `metrics.temporal.io` which serves all metrics across your entire account with API key authentication and standard HTTPS.

## Namespace and metric filtering
You can use query parameters to enable selective scraping to manage data volume and costs, which support wildcards for flexible namespace selection and specific metric filtering.

## Dashboard templates
Production-ready [Grafana dashboards](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) provide immediate visibility with pre-built queries and visualizations.

---

# OpenMetrics API reference

Source: https://docs.temporal.io/cloud/metrics/openmetrics/api-reference

> Detailed API documentation for the Temporal Cloud OpenMetrics endpoint.

The Temporal Cloud OpenMetrics API provides actionable operational metrics about your Temporal Cloud deployment. This is a scrapable HTTP API that returns metrics in OpenMetrics format, suitable for ingestion by Prometheus-compatible monitoring systems.

## Available metrics reference

Metrics descriptions are also available programmatically via the `/v1/descriptors` endpoint. You can see the Metrics Reference for a list of available metrics.

## Authentication

Temporal uses API keys for integrating with the OpenMetrics endpoint. Applications must be authorized and authenticated before they can access metrics from Temporal Cloud.

An API key is owned by a Service Account and inherits the permissions granted to the owner.

### Creating API keys

API keys can be created using the [Temporal Cloud UI](https://cloud.temporal.io):

1. Navigate to Settings → Service Accounts  
2. Create a service account with **"Metrics Read-Only"** Account Level Role
3. Generate an API key within the service account

> **ℹ️ Info:**
>
> See the [docs](/cloud/api-keys#serviceaccount-api-keys) for more details on generating API keys.
>

### Using API keys

All API requests must be made over HTTPS. Calls made over plain HTTP will fail. API requests without authentication will also fail.

```shell
curl -H "Authorization: Bearer <API_KEY>" https://metrics.temporal.io/v1/metrics
```

## Object model

The object model for the Metrics API follows the [OpenMetrics](https://openmetrics.io/) standard.

### Metrics

A metric is a numeric attribute measured at a specific point in time, labeled with contextual metadata gathered at the point of instrumentation.

### Metric types

All Temporal Cloud metrics are exposed as *gauges* in OpenMetrics format, but represent different measurement types:

* **Rate metrics**: Pre-computed per-second rates with delta temporality (for example, `temporal_cloud_v1_workflow_success_count` \- workflows completed per second)  
* **Value metrics**: Current or instantaneous values (for example, `temporal_cloud_v1_approximate_backlog_count` \- current number of tasks in queue)

The list of metrics and their labels are available via the [List Descriptors](/cloud/metrics/openmetrics/api-reference#list-metric-descriptors) endpoint or in the [Metrics Reference](/cloud/metrics/openmetrics/metrics-reference).

### Labels

A label is a key-value attribute associated with a metric data point. Labels can be used to filter or aggregate metrics.
Temporal SDKs and the Temporal Service call this same concept a [tag](/glossary#tag); it's called a label once scraped in OpenMetrics format.

Common labels include:

* `temporal_namespace`: The Temporal namespace  
* `temporal_account`: The Temporal account  
* `region`: The cloud region where the metric originated  
* `temporal_workflow_type`: The workflow type (where applicable)  
* `temporal_task_queue`: The task queue name (where applicable)

Each metric has its own set of applicable labels. See the Metrics Reference for complete details.

### Metric family

A [Metric Family](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#metricfamily) may have zero or more metrics.  The set of metrics returned will vary based on actual system activity.  Metrics only appear in a Metric Family if they were reported during the aggregation window.

## Client considerations

### Rate limiting

To protect the stability of the API and keep it available to all users, Temporal employs multiple safeguards.

When a rate limit is breached, an HTTP `429 Too Many Requests` error is returned with the following headers:

| Header | Description |
| ----- | ----- |
| `Retry-After` | The time in seconds until the rate limit window resets |

#### Rate limit scopes
> **📝 Note:**
> Rate limit scopes are subject to change.
>

| Scope | Limit |
| ----- | ----- |
| Account | 180 requests per hour |

> **⚠️ Caution:**
> Practical enforcement of this limit may allow bursts that exceed the limit at various times.
> Do not rely on this headroom: design production clients against the documented limit.
>

### Response completeness

The `X-Completeness` header indicates whether the response contains all available data:

* `complete`: The response contains all metrics requested  
* `limited`: Response truncated due to size limits (50k metric data points max). Use namespace or metric filtering to reduce the response size.
* `unknown`: Completeness cannot be determined (possibly due to regional issues or timeouts). Clients are encouraged to retry.

### Retry logic

Implement retry logic in your client to gracefully handle transient API failures. Use exponential backoff with jitter to avoid retry storms with reasonable retry intervals to avoid reaching rate limits.

### Data latency

Metric data points are available for query within 3 minutes of their origination. This is in line with the freshest metrics [available from any major service provider](https://docs.datadoghq.com/integrations/guide/cloud-metric-delay/). This latency should be accounted for when setting up monitoring alerts.

### Scrape window

The endpoint exposes only the most recently completed one-minute aggregation window. Each scrape returns a snapshot of that window—there is no query interface for historical data. To retain historical metrics, configure your monitoring system to store what it scrapes.

## Endpoints

> **ℹ️ Info:**
>
> All endpoints are served from: `metrics.temporal.io`
>

### Get metrics

`GET /v1/metrics`

Returns metrics in OpenMetrics format suitable for scraping by Prometheus-compatible systems.

#### Timestamp offset

To account for metric data latency, this endpoint returns metrics from the current timestamp minus a fixed offset.  The current offset is 3 minutes rounded down to the start of the minute. To accommodate this offset, the timestamps in the response should be honored when importing the metrics. For example, in Prometheus this can be controlled using the `honor\_timestamps` flag.

#### Query parameters

| Parameter | Type | Description |
| ----- | ----- | ----- |
| `namespaces` | string array | Filter to specific Namespaces. Supports wildcards (for example, `production-*`) |
| `metrics` | string array | Filter to specific metrics |

Array parameters use repeated keys. To pass multiple values, repeat the parameter name once per value:

```shell
/v1/metrics?namespaces=prod-payments&namespaces=prod-orders
```

The `namespaces` and `metrics` parameters can be combined, and each may be repeated independently:

```shell
/v1/metrics?namespaces=prod-payments&namespaces=prod-orders&metrics=temporal_cloud_v1_workflow_success_count&metrics=temporal_cloud_v1_approximate_backlog_count
```

#### Response headers

| Header | Description |
| ----- | ----- |
| `X-Completeness` | Indicates the response status: `complete`, `limited`, or `unknown` |
| `Content-Type` | `application/openmetrics-text` |

> **ℹ️ Info:**
> Example
>
> Request:
>
> ```shell
> curl -H "Authorization: Bearer <API_KEY>" \
>   "https://metrics.temporal.io/v1/metrics?namespaces=production-*"
> ```
>
> Response:
> ```
> # TYPE temporal_cloud_v1_workflow_success_count gauge
> # HELP temporal_cloud_v1_workflow_success_count The number of successful workflows per second
> temporal_cloud_v1_workflow_success_count{temporal_namespace="production",temporal_workflow_type="payment-processing",region="aws-us-west-2"} 42.0 1609459200000
> temporal_cloud_v1_workflow_success_count{temporal_namespace="production",temporal_workflow_type="order-fulfillment",region="aws-us-west-2"} 128.0 1609459200000
>
> # TYPE temporal_cloud_v1_approximate_backlog_count gauge  
> # HELP temporal_cloud_v1_approximate_backlog_count Approximate number of tasks in a task queue
> temporal_cloud_v1_approximate_backlog_count{temporal_namespace="production",temporal_task_queue="critical-queue",task_type="workflow", region="aws-us-west-2"} 15.0 1609459200000
> ```
>

#### Summary of best practices

* *Honor timestamps*: Set `honor_timestamps: true` in Prometheus  
* *Scrape interval*: Use 30s. Intervals longer than 60s may skip datapoints because metrics update once per minute.  
* *Timeout*: Set scrape timeout to 10 seconds for large responses  
* *Filtering*: Use query parameters to reduce response size

### List metric descriptors

`GET /v1/descriptors`

Lists all metric descriptors, including help text, available dimensions (labels), and release stages.

Each descriptor includes a `release_stage` field that reports the metric's lifecycle status. The supported values are:

| Value | Description |
| ----- | ----- |
| `public-preview` | The metric is in [Public Preview](/evaluate/product-release-stages#public-preview) |
| `general-availability` | The metric is in [General Availability](/evaluate/product-release-stages#general-availability) |
| `deprecated` | The metric is deprecated |

#### Query parameters

| Parameter | Type | Description |
| ----- | ----- | ----- |
| `limit` | integer | Page size (1-100, default: 100\) |
| `offset` | integer | Page offset |

> **ℹ️ Info:**
> Example
>
> Request:
>
> ```shell
> curl -H "Authorization: Bearer <API_KEY>" \
>   "https://metrics.temporal.io/v1/descriptors"
> ```
>
> Response:
>
> ```json
> {
>   "meta": {
>     "pagination": {
>       "total": 35,
>       "limit": 100,
>       "offset": 0
>     }
>   },
>   "descriptors": [
>     {
>       "name": "temporal_cloud_v1_workflow_success_count",
>       "help": "The number of successful workflows per second",
>       "dimensions": [
>         "temporal_namespace",
>         "temporal_workflow_type", 
>         "temporal_task_queue",
>         "region"
>       ],
>       "release_stage": "general-availability"
>     }
>   ]
> }
> ```
>

## Managing high cardinality

> **⚠️ Caution:**
>
> High-cardinality labels like `temporal_task_queue` and `temporal_workflow_type` can significantly increase metric volume and impact performance of your monitoring system. 
>

### Cardinality estimation

To estimate your metric cardinality and see if this is an issue:

```
Total series = Base metrics × Namespaces × Task queues × Workflow types
```

Example:

* 6 workflow metrics with both labels  
* 10 namespaces  
* 50 task queues  
* 20 workflow types  
* \= 6 × 10 × 50 × 20 \= 60,000 time series

> **📝 Note:**
>
> 60,000 time series in the above example results in exceeding the 30,000 data points per scrape limit.
>

If the cardinality is too high or you are hitting API limits, consider the following strategies.

### Filtering at scrape time

You can isolate only the metrics/namespaces you need.  For example, the following shows examples of filtering by modifying the `metrics_path.`

```shell
# Only specific namespaces matching the wildcard pattern
/v1/metrics?namespaces=production-*

# Multiple namespaces
/v1/metrics?namespaces=prod-payments&namespaces=prod-orders

# Only specific metrics
/v1/metrics?metrics=temporal_cloud_v1_workflow_success_count

# Combined filtering
/v1/metrics?namespaces=prod-*&metrics=temporal_cloud_v1_approximate_backlog_count
```

> **ℹ️ Info:**
>
> In Prometheus, the `params` config can be set to match the same behavior as above.
>
> ```yaml
> scrape_configs:
> - job_name: 'temporal-cloud'
>   ...
>   static_configs:
>     - targets: ['metrics.temporal.io']
>   metrics_path: '/v1/metrics'
>   params:
>     namespaces: ['prod-*']
>     metrics: ['temporal_cloud_v1_approximate_backlog_count']
>
> ```
>

### Label management

#### Prometheus

If using Prometheus, you can configure it to drop metrics with a specific label or even rename specific label values to reduce the cardinality.

```yaml
metric_relabel_configs:
# Consolidate non-critical task queues
- source_labels: [temporal_task_queue]
  regex: '(critical-queue|payment-queue)'
  target_label: __tmp_keep_original
  replacement: 'true'
  
- source_labels: [__tmp_keep_original]
  regex: ''
  target_label: temporal_task_queue
  replacement: 'other'
  
- regex: '__tmp_keep_original'
  action: labeldrop
```

#### OpenTelemetry collector

To accomplish the same as Prometheus, a filter can be used in the collector along with any other processors.

```
processors:
  filter:
    metrics:
      include:
        match_type: regexp
        expressions:
          # Only keep metrics with critical-queue or payment-queue
          - Label("temporal_task_queue") == nil or IsMatch(Label("temporal_task_queue"), "^(critical-queue|payment-queue)$")
```

### Monitoring cardinality

Cardinality can be monitored using this PromQL query.

```shell
# Count the total number of series
count({__name__=~"temporal_cloud_v1_.*"})

# Count the total number of series by metric
count({__name__=~"temporal_cloud_v1_.*"}) by (__name__)
```

## API limits

| Limit | Impact | Mitigation |
| ----- | ----- | ----- |
| 50k total datapoints per scrape | Response may be truncated | Use namespace/metric filtering |
| 180 requests per account per hour (~3 requests per minute) | HTTP 429 returned | Set scrape interval to 30s |

> **⚠️ Caution:**
> Practical enforcement of this limit may allow bursts that exceed the limit at various times.
> Do not rely on this headroom: design production clients against the documented limit.
>

---

# OpenMetrics FAQ

Source: https://docs.temporal.io/cloud/metrics/openmetrics/faq

> Answers to common questions about querying, scraping limits, and missing Temporal Cloud OpenMetrics data.

## Which Cloud metrics and labels are available?

OpenMetrics includes metrics for Activity outcomes and latency, Task Queue backlog, Namespace usage and limits, service requests, and Workflow outcomes. Metrics can include labels such as `region`, `temporal_task_queue`, and `temporal_workflow_type`. Some high-cardinality labels, including `temporal_activity_type`, require you to opt in when scraping.

For the current list of metric names, descriptions, and labels, see the [OpenMetrics metrics reference](/cloud/metrics/openmetrics/metrics-reference).

## Can I query Cloud metrics directly?

The OpenMetrics endpoint returns the most recently completed aggregation window for scraping. It does not provide a time-series query API. Configure an [observability tool](/cloud/metrics/openmetrics/metrics-integrations) to collect and retain the metrics, then query them in that tool.

## How often can I scrape, and how much data can I retrieve?

See [OpenMetrics API limits](/cloud/metrics/openmetrics/api-reference#api-limits) for the request and datapoint limits. If a scrape exceeds the datapoint limit, use [Namespace or metric filtering](/cloud/metrics/openmetrics/api-reference#managing-high-cardinality) to reduce the response size.

## Why are some metrics missing from a scrape?

The OpenMetrics endpoint returns only metrics reported during its one-minute aggregation window. For example, `temporal_cloud_v1_workflow_failed_count` is absent if no Workflows failed in that window. A Task Queue with no activity can also have no metrics in a scrape. The set of returned metrics can therefore vary between scrapes.

If a dashboard or alert should treat an absent series as zero, handle that in your observability tool. For example, in Prometheus:

```promql
sum(temporal_cloud_v1_workflow_failed_count) or vector(0)
```

---

# Metrics integrations

Source: https://docs.temporal.io/cloud/metrics/openmetrics/metrics-integrations

> Integrating with the Temporal Cloud OpenMetrics endpoint.

Metrics can be exported from Temporal Cloud using the OpenMetrics endpoint. This document describes configuring integrations that have third party support or are based on open standards.
This document is for basic configuration only. For advanced concepts such as label management and high cardinality scenarios see the 
[general API reference](/cloud/metrics/openmetrics/api-reference).

## Integrations

Before configuring any integration, complete the [Quickstart](/cloud/metrics/openmetrics#quickstart) to create a Service Account with the **Metrics Read-Only** role and generate an API key. This requires the **Account Owner** or **Global Admin** role - a Namespace Admin cannot grant the Metrics Read-Only role.

### Datadog

Datadog provides a serverless integration with the OpenMetrics endpoint. It scrapes metrics, stores them in Datadog, and ships a default dashboard with built-in monitors.

1. In Datadog, open the [Integrations catalog](https://app.datadoghq.com/integrations) and search for **Temporal Cloud OpenMetrics**. Install the integration.
2. Click **Add Account** in the integration tile and paste your Temporal Cloud API key into the **API Key** field.
3. Save the configuration. The default Temporal Cloud dashboard appears in **Dashboards → Dashboards List** once data starts flowing (typically within a few minutes).

For Datadog-side details, see the [Datadog integration page](https://docs.datadoghq.com/integrations/temporal-cloud-openmetrics/).

Datadog also offers a separate [Cost Management integration](https://docs.datadoghq.com/integrations/temporal-cloud-costs/) that sends Temporal Cloud billing data to Datadog Cloud Cost Management for spend attribution by namespace and service type.

For Datadog users, treat the OpenMetrics integration as the Cloud-side half of your observability setup:

- Use OpenMetrics in Datadog to monitor Temporal Cloud behavior such as Task Queue backlog, poll success, and rate limiting.
- Collect [SDK metrics](/cloud/metrics/sdk-metrics-setup) from your Workers separately to monitor saturation, Schedule-To-Start latency, slot availability, and sticky cache behavior.

If you only ingest Cloud metrics, you will miss many worker-side bottlenecks. For recommended Worker monitors, see
[Monitor worker health](/cloud/worker-health).

### Elastic

The Elastic [Temporal Integration](https://www.elastic.co/docs/reference/integrations/temporal) scrapes the OpenMetrics endpoint, stores metrics in Elasticsearch, and automatically installs dashboards, alert templates, and SLO templates in Kibana.

Setup

1. In Kibana, go to Management → Integrations and search for Temporal (OpenTelemetry).
2. Add the integration. The default Cloud endpoint (metrics.temporal.io:443) and path (/v1/metrics) are pre-configured.
3. Enter your Temporal Cloud API key in the Temporal Cloud API Key field.
4. Verify metrics are flowing in Discover using the filter data_stream.dataset: "temporal.cloud_metrics.otel".

Dashboards, alert rules, and SLO templates are installed automatically.

### Grafana Cloud

Grafana Cloud provides a serverless integration that scrapes the OpenMetrics endpoint, stores metrics in Grafana Cloud, and ships a prebuilt **Temporal overview** dashboard.

1. In Grafana Cloud, go to **Connections** and select the **Temporal Cloud** tile.
2. On the **Configuration** page, add a scrape job: give it a name, paste your Temporal Cloud API key, and set the scrape interval (default 1 minute - lower intervals increase data points per minute and cost).
3. Click **Test Connection** to verify authentication, then **Save** to start collecting metrics.
4. Click **Install** to deploy the prebuilt dashboards.

Add a separate scrape job per account to monitor multiple accounts from one Grafana Cloud instance.

If the dashboard shows no data after a few minutes, confirm the API key's Service Account has the **Metrics Read-Only** role and that the endpoint is reachable using the `curl` check from the [Quickstart](/cloud/metrics/openmetrics#quickstart).

For full configuration, dashboards, and changelog, see the [Grafana Cloud integration page](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-temporal/).

### ClickStack

ClickHouse provides an integration with the OpenMetrics endpoint for ClickStack. This integration uses an OpenTelemetry collector to read from the OpenMetrics endpoint, ingest data into ClickHouse, and
includes a default dashboard to visualize the data with HyperDX. See the [integration page](https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/temporal-metrics) for more details.

1. Save your Temporal Cloud API key to a local file named `temporal.key` (no trailing newline or spaces).
2. Create an OpenTelemetry collector config named `temporal-metrics.yaml` that uses a Prometheus receiver against `metrics.temporal.io` with Bearer token auth, a 60-second scrape interval, the `service.name: "temporal"` resource attribute, and the ClickHouse exporter. Copy the full template from the [ClickStack integration page](https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/temporal-metrics).
3. Mount both files into your ClickStack collector and set the custom config env var. With Docker Compose:

   ```yaml
   volumes:
     - ./temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml
     - ./temporal.key:/etc/otelcol-contrib/temporal.key
   environment:
     CUSTOM_OTELCOL_CONFIG_FILE: /etc/otelcol-contrib/custom.config.yaml
   ```

4. In HyperDX, open the **Metrics explorer** and confirm metrics with the `temporal` prefix are arriving.
5. Import the pre-built dashboard: in HyperDX click **Import Dashboard**, upload `temporal-metrics-dashboard.json` from the ClickStack integration page, then click **Finish Import**.

### New Relic

The New Relic integration pulls metrics from the OpenMetrics endpoint via the `nri-flex` integration that runs alongside the New Relic infrastructure agent.

> **📝 Note:**
> Requires a host
>
> The integration runs on a host (Linux, Windows, or Kubernetes) with the New Relic infrastructure agent installed. The agent scrapes the endpoint and forwards metrics to New Relic.
>

1. Install the **New Relic infrastructure agent** on a host. See the [agent install docs](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent/get-started/install-infrastructure-agent/) for platform-specific instructions.
2. Create `/etc/newrelic-infra/integrations.d/nri-flex-temporal-cloud-config.yml` using the template from the [New Relic integration page](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/temporal-cloud-integration/), and replace the `${TEMPORAL_API_KEY}` placeholder with your Temporal Cloud API key.
3. Restart the agent so the new config is picked up:

   ```shell
   sudo systemctl restart newrelic-infra.service
   ```

4. In **one.newrelic.com**, go to **Integrations & Agents → Dashboards**, search for **Temporal Cloud**, and install the pre-built dashboard. Data appears within a few minutes.

For New Relic-side details, see the [New Relic integration page](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/temporal-cloud-integration/).

### SigNoz

SigNoz is an OpenTelemetry-native observability platform, available as a cloud service or self-hosted. The integration runs an OpenTelemetry Collector that scrapes the OpenMetrics endpoint and forwards metrics to SigNoz over OTLP.

1. Configure an OpenTelemetry Collector with a Prometheus receiver against `metrics.temporal.io` using Bearer token auth with your Temporal Cloud API key, and an OTLP exporter that sends to your SigNoz ingestion endpoint (using your SigNoz ingestion key and region). Copy the full template - including the Kubernetes Secret-based variant - from the [SigNoz integration page](https://signoz.io/docs/integrations/temporal-cloud-metrics/).
2. Deploy the collector (VM/binary or Kubernetes) and confirm `temporal_cloud_v1_*` metrics arrive in the SigNoz **Metrics Explorer**, typically within a few minutes.
3. Import the pre-built dashboard JSON from the [SigNoz integration page](https://signoz.io/docs/integrations/temporal-cloud-metrics/) to visualize Temporal Cloud metrics.

For SigNoz-side details, see the [SigNoz integration page](https://signoz.io/docs/integrations/temporal-cloud-metrics/).

### Parseable

Parseable uses an OpenTelemetry Collector to scrape Temporal Cloud's OpenMetrics endpoint and export metrics to Parseable over OTLP/HTTP.
Follow the [Parseable Temporal Cloud metrics guide](https://www.parseable.com/docs/ingest-data/ai-agents/temporal?tab=temporal-cloud-metrics) to configure the Collector and verify that metrics appear in Parseable.

### Prometheus \+ Grafana 

Self hosted Prometheus can be used to scrape the OpenMetrics endpoint.

1. Add a new scrape job for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys).

```yaml
scrape_configs:
  - job_name: 'temporal-cloud'
    scrape_interval: 30s
    scrape_timeout: 30s
    honor_timestamps: true
    scheme: https
    authorization:
      type: Bearer
      credentials: '<API_KEY>'
    static_configs:
      - targets: ['metrics.temporal.io']
    metrics_path: '/v1/metrics'
```

2. Import the [Grafana dashboard](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) and configure your Prometheus datasource.

### OpenTelemetry Collector configuration

Collect metrics with a self-hosted OpenTelemetry Collector to ingest into the system of your choosing.

1. Add a new prometheus receiver for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys).

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'temporal-cloud'
        scrape_interval: 30s
        scrape_timeout: 30s
        honor_timestamps: true
        scheme: https
        authorization:
          type: Bearer
          credentials_file: <API_KEY_FILE>
        static_configs:
          - targets: ['metrics.temporal.io']
        metrics_path: '/v1/metrics'

processors:
  batch:

exporters:
  otlphttp:
    endpoint: <ENDPOINT>

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlphttp]
```

> **ℹ️ Info:**
>
> Examples for these integrations and more are [here](https://github.com/temporal-community/cloud-metrics-scrape-examples).
>

---

# OpenMetrics metrics reference

Source: https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference

> Detailed API documentation for the Temporal Cloud OpenMetrics endpoint.

This document describes all metrics available from the Temporal Cloud OpenMetrics endpoint.

## Metric Conventions

### Metric Types

All metrics are exposed as OpenMetrics gauges, but represent different measurement types:

* *Rate Metrics*: per-second rate of the aggregated values  
* *Value Metrics*: The most recent aggregate value within a look-back window (for example, backlogs, limits)  
* *Percentile Metrics*: Pre-calculated aggregated latency percentiles in seconds

> **📝 Note:**
>
> All metrics are stored as 1 minute aggregates. Rate metrics are therefore per-second rates averaged over each minute, which smooths sub-minute bursts. A short spike can read below your limit even when it triggered throttling. See [Why does throttling occur when count metrics stay below the limit?](/cloud/service-health#why-does-throttling-occur-when-count-metrics-stay-below-the-limit) for a worked example.
>

> **📝 Note:**
> Percentile metrics on low-traffic namespaces
>
> Percentile metrics (`*_p50` / `_p95` / `_p99`) are calculated from the requests observed in each 1-minute aggregation window. On a namespace with few requests per minute, that sample is small, so a single slow request dominates every percentile and p50, p95, and p99 converge toward the slowest observed request. Tail percentiles generally need roughly 20 or more samples per window before they are statistically meaningful; below that, values vary widely.
>
> For example, a low-volume namespace that starts one Workflow every few minutes can report a several-hundred-millisecond `StartWorkflowExecution` `temporal_cloud_v1_service_latency_p95` that reflects a single request, not systemic latency. When alerting on percentile latency for low-traffic namespaces, gate the alert on a minimum request count (for example, [`temporal_cloud_v1_service_request_count`](#temporal_cloud_v1_service_request_count)) so that windows with too few samples don't trigger it. These percentiles are pre-calculated per 1-minute window and cannot be re-aggregated into an accurate longer-window percentile, so widening your evaluation window does not by itself make a sparse sample meaningful.
>

### Common Labels

All metrics include these base labels:

| Label | Description |
| ----- | ----- |
| `temporal_namespace` | The Temporal namespace |
| `temporal_account` | The Temporal account identifier |
| `region` | Cloud region where the metric originated |

### Opt-in Labels

Some labels are **opt-in** due to their high cardinality.
These labels are not included by default when you scrape the OpenMetrics endpoint.
To enable an opt-in label, add it to the `labels` query parameter on your scrape URL.
When an opt-in label is enabled, it is populated on **all metrics** that support it.

| Label | Available on | Description |
| ----- | ----- | ----- |
| `temporal_activity_type` | Activity metrics | The activity type name |
| `temporal_worker_deployment_name` | `temporal_cloud_v1_approximate_backlog_count`, `temporal_cloud_v1_approximate_backlog_age` | The Worker Deployment name |
| `temporal_worker_build_id` | `temporal_cloud_v1_approximate_backlog_count`, `temporal_cloud_v1_approximate_backlog_age` | The Worker Deployment Version Build ID |
| `nexus_service` | Nexus metrics | The Nexus Service name |
| `nexus_operation` | Nexus metrics | The Nexus Operation name |

For example, to include `temporal_activity_type` in your scrape results:

```
/v1/metrics?labels=temporal_activity_type
```

Enable multiple labels at the same time by concatenating multiple `labels` query parameters:

```
/v1/metrics?labels=temporal_worker_build_id&labels=temporal_worker_deployment_name
```

## Metrics Catalog

### Frontend Service Metrics

#### temporal\_cloud\_v1\_service\_request\_count

gRPC requests received per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_request\_throttled\_count

gRPC requests throttled per second. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) for guidance on setting alert thresholds against the corresponding limit metric.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_error\_count

gRPC errors per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_pending\_requests

The number of pollers that are actively long polling for a task. Use this to track against ``temporal_cloud_v1_poller_limit``

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Value

#### temporal\_cloud\_v1\_resource\_exhausted\_error\_count

Resource exhaustion errors per second, incremented when a single resource receives a burst larger than it can absorb. SDKs retry these errors gracefully. This metric does not include throttling due to Namespace limits - see [`temporal_cloud_v1_total_action_throttled_count`](#temporal_cloud_v1_total_action_throttled_count) and related throttle metrics for rate limiting against account limits.

See [Detecting Resource Exhaustion](/cloud/service-health#detecting-resource-exhaustion) for guidance on investigating non-zero values.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile latency of service requests in seconds

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Latency  

#### temporal\_cloud\_v1\_service\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile latency of service requests in seconds

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Latency  

#### temporal\_cloud\_v1\_service\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions as the percentile won't be accurate.
>

The 99th percentile latency of service requests in seconds

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Latency  

### Workflow Completion Metrics

> **⚠️ Caution:**
> High Cardinality
>
> These metrics could have high cardinality depending on number of workflow types and task queues.
>

#### temporal\_cloud\_v1\_workflow\_success\_count

Successful workflow completions per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate  

#### temporal\_cloud\_v1\_workflow\_failed\_count

Workflow failures per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_timeout\_count

Workflow timeouts per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_cancel\_count

Workflow cancellations per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_terminate\_count

Workflow terminations per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_continued\_as\_new\_count

Workflows continued as new per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile workflow schedule-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_workflow_type` | The workflow type |

**Type**: Latency

#### temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile workflow schedule-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_workflow_type` | The workflow type |

**Type**: Latency

#### temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 99th percentile workflow schedule-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_workflow_type` | The workflow type |

**Type**: Latency

### Activity Metrics

> **⚠️ Caution:**
> High Cardinality
>
> These metrics could have high cardinality depending on number of activity types, workflow types, and task queues. The `temporal_activity_type` label is [opt-in](#opt-in-labels) to help manage cardinality.
>

> **📝 Note:**
> Standalone Activities
>
> [Standalone Activities](/standalone-activity) are Activity Executions that are started independently, without an associated Workflow. For Activity metrics that include the `temporal_workflow_type` label, Standalone Activities use the placeholder value `"__standalone_activity"`.
>

#### temporal\_cloud\_v1\_activity\_success\_count

Successful activity completions per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_activity\_fail\_count

Activity failures per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_activity\_timeout\_count

Activity timeouts per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
| `temporal_activity_type` | The activity type (opt-in) |
| `timeout_type` | The timeout type |

**Type**: Rate

#### temporal\_cloud\_v1\_activity\_task\_fail\_count

Activity task failures per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_activity\_task\_timeout\_count

Activity task timeouts per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
| `temporal_activity_type` | The activity type (opt-in) |
| `timeout_type` | The timeout type |

**Type**: Rate

#### temporal\_cloud\_v1\_activity\_cancel\_count

Activity cancellations per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_activity\_terminate\_count

Activity terminations per second.  This metric only applies to Standalone Activities. Regular Activities that run within a Workflow cannot be terminated independently.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Rate

> **ℹ️ Info:**
> Activity latency labels
>
> Activity latency metrics include only the `temporal_activity_type` label.
> Labels such as `temporal_task_queue` and `temporal_workflow_type` are intentionally excluded because pre-calculated percentile values cannot be accurately aggregated across additional dimensions.
>

#### temporal\_cloud\_v1\_activity\_start\_to\_close\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile activity start-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_activity\_start\_to\_close\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile activity start-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_activity\_start\_to\_close\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 99th percentile activity start-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_activity\_schedule\_to\_close\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile activity schedule-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_activity\_schedule\_to\_close\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile activity schedule-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_activity\_schedule\_to\_close\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 99th percentile activity schedule-to-close latency in seconds.

| Label | Description |
| ----- | ----- |
| `temporal_activity_type` | The activity type (opt-in) |

**Type**: Latency

### Nexus metrics

> **📝 Note:**
> Public Preview
>
> The Temporal Cloud Nexus metrics in this section are in
> [Public Preview](/evaluate/product-release-stages#public-preview).
>

Nexus operation metrics measure calls from the caller side. For these metrics, `temporal_namespace` identifies the caller Namespace.
Nexus request metrics measure requests on the handler side. For these metrics, `temporal_namespace` identifies the handler Namespace.

> **📝 Note:**
> Standalone Nexus Operations
>
> Standalone Nexus Operations are Nexus Operation Executions that are started independently, without a caller Workflow. For Nexus Operation metrics that include the `temporal_workflow_type` label, Standalone Nexus Operations use the placeholder value `__temporal_standalone_nexus_operation__`.
>

> **📝 Note:**
> Nexus latency intervals
>
> - Request latency measures how long the handler takes to process one Nexus request. For a synchronous start request, it ends when the handler returns the result. For an asynchronous start request, it ends when the handler finishes the start request and doesn't continue until the Operation completes.
> - Schedule-to-close latency measures from when the caller schedules an Operation until it reaches a terminal state. It includes Operations that complete synchronously and asynchronously.
> - Schedule-to-start latency measures from when the caller schedules an Operation until it starts asynchronously. For an Operation that completes synchronously or closes without starting, Temporal records this latency when the Operation closes, and it equals schedule-to-close latency.
> - Start-to-close latency measures from when an asynchronous Operation starts until it reaches a terminal state. Temporal doesn't record this latency for an Operation that completes synchronously or closes without starting.
>
> The Operation metric labels don't distinguish synchronous from asynchronous Operations. Schedule-to-close and schedule-to-start series can contain both, so you can't filter their samples by completion type.
>

> **⚠️ Caution:**
> High Cardinality
>
> Nexus metrics could have high cardinality depending on the number of Nexus Endpoints, Services, Operations, and callers.
> The `nexus_service` and `nexus_operation` labels are [opt-in](#opt-in-labels) to help manage cardinality.
>

#### temporal\_cloud\_v1\_nexus\_operation\_schedule\_to\_close\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile time from when a Nexus Operation is scheduled until it reaches a terminal state, in seconds. This metric includes Operations that complete synchronously and asynchronously.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_schedule\_to\_close\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile time from when a Nexus Operation is scheduled until it reaches a terminal state, in seconds. This metric includes Operations that complete synchronously and asynchronously.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_schedule\_to\_close\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 99th percentile time from when a Nexus Operation is scheduled until it reaches a terminal state, in seconds. This metric includes Operations that complete synchronously and asynchronously.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_schedule\_to\_start\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile time from when a Nexus Operation is scheduled until it starts asynchronously, in seconds. For an Operation that completes synchronously or closes without starting, this duration equals its schedule-to-close latency.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_schedule\_to\_start\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile time from when a Nexus Operation is scheduled until it starts asynchronously, in seconds. For an Operation that completes synchronously or closes without starting, this duration equals its schedule-to-close latency.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_schedule\_to\_start\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 99th percentile time from when a Nexus Operation is scheduled until it starts asynchronously, in seconds. For an Operation that completes synchronously or closes without starting, this duration equals its schedule-to-close latency.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_start\_to\_close\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile time from when an asynchronous Nexus Operation starts until it reaches a terminal state, in seconds. This metric isn't recorded for an Operation that completes synchronously or closes without starting.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_start\_to\_close\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile time from when an asynchronous Nexus Operation starts until it reaches a terminal state, in seconds. This metric isn't recorded for an Operation that completes synchronously or closes without starting.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_start\_to\_close\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 99th percentile time from when an asynchronous Nexus Operation starts until it reaches a terminal state, in seconds. This metric isn't recorded for an Operation that completes synchronously or closes without starting.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_operation\_success\_count

Successfully completed Nexus Operations per second.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_nexus\_operation\_cancel\_count

Canceled Nexus Operations per second.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_nexus\_operation\_terminate\_count

Nexus Operations terminated before successful completion per second.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_nexus\_operation\_fail\_count

Failed Nexus Operations per second.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_nexus\_operation\_timeout\_count

Nexus Operations that timed out before completing execution per second.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `temporal_workflow_type` | The Workflow Type |
| `timeout_type` | The timeout type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_nexus\_request\_count

Nexus requests received by the Nexus Endpoint per second.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `nexus_caller_type` | The Nexus caller type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_nexus\_request\_error\_count

Failed Nexus requests per second.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `nexus_caller_type` | The Nexus caller type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Rate

#### temporal\_cloud\_v1\_nexus\_request\_latency\_p50

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 50th percentile time the handler takes to process one Nexus request, in seconds. For an asynchronous start request, the measurement ends when the handler finishes the request and doesn't continue until the Operation completes.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `nexus_caller_type` | The Nexus caller type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_request\_latency\_p95

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 95th percentile time the handler takes to process one Nexus request, in seconds. For an asynchronous start request, the measurement ends when the handler finishes the request and doesn't continue until the Operation completes.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `nexus_caller_type` | The Nexus caller type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

#### temporal\_cloud\_v1\_nexus\_request\_latency\_p99

> **⚠️ Caution:**
>
> Avoid aggregating this metric across dimensions because the percentile won't be accurate.
>

The 99th percentile time the handler takes to process one Nexus request, in seconds. For an asynchronous start request, the measurement ends when the handler finishes the request and doesn't continue until the Operation completes.

| Label | Description |
| ----- | ----- |
| `nexus_endpoint` | The Nexus Endpoint name |
| `nexus_caller_type` | The Nexus caller type |
| `nexus_service` | The Nexus Service name (opt-in) |
| `nexus_operation` | The Nexus Operation name (opt-in) |

**Type**: Latency

### Task Queue Metrics

> **⚠️ Caution:**
> High Cardinality
>
> These metrics could have high cardinality depending on number of task queues present.
>

#### temporal\_cloud\_v1\_approximate\_backlog\_count

The approximate number of tasks pending in a task queue. Started Activities are not included in the count as they have been dequeued from the task queue.

> **📝 Note:**
> Known accuracy limitations
> This metric is approximate.
> It can overcount because invalid or expired Tasks, like from cancelled, terminated, completed, or timed out Workflows, remain in the count until they reach the head of the queue and are processed and discarded.
>
> It can also reset to zero on an idle Task Queue. If no Worker polls, no new Tasks are added, and no other Task Queue calls occur (such as `DescribeTaskQueue` or `UpdateTaskQueueConfig`) for approximately 5 minutes. The Task Queue is unloaded from memory.
> Infrequent metadata updates and database time-to-live settings can also cause this metric to drift at a smaller magnitude.
> See [backlog accuracy limitations](/develop/worker-performance/task-queues#backlog-accuracy-limitations) for details.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `task_type` | Type of task: `workflow` or `activity` |
| `task_priority` | The task priority |
| `temporal_worker_deployment_name` | The Worker Deployment name (opt-in) |
| `temporal_worker_build_id` | The Worker Deployment Version Build ID (opt-in) |

**Type**: Value

#### temporal\_cloud\_v1\_approximate\_backlog\_age

The approximate age, in seconds, of the oldest task in a task queue backlog.

> **📝 Note:**
> Known accuracy limitations
> This metric is approximate and is subject to the same limitations as `temporal_cloud_v1_approximate_backlog_count`.
> See [backlog accuracy limitations](/develop/worker-performance/task-queues#backlog-accuracy-limitations) for details.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_worker_deployment_name` | The Worker Deployment name (opt-in) |
| `temporal_worker_build_id` | The Worker Deployment Version Build ID (opt-in) |

**Type**: Value

#### temporal\_cloud\_v1\_poll\_success\_count

Successfully matched tasks per second.

| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |

**Type**: Rate

#### temporal\_cloud\_v1\_poll\_success\_sync\_count

Tasks matched synchronously per second (no polling wait).

| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |

**Type**: Rate

#### temporal\_cloud\_v1\_poll\_timeout\_count

The rate of poll requests that timed out without receiving a task.

| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |

**Type**: Rate

#### temporal\_cloud\_v1\_no\_poller\_tasks\_count

The rate of tasks added to queues with no active pollers.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `task_type` | Type of task: `workflow` or `activity` |

**Type**: Rate

### Namespace Metrics

#### temporal\_cloud\_v1\_namespace\_open\_workflows

The current number of open workflows in a namespace.

**Type**: Value

#### temporal\_cloud\_v1\_total\_action\_count

The total number of actions performed per second. Actions with `is_background=false` are counted toward the ``temporal_cloud_v1_action_limit``.

| Label | Description |
| ----- | ----- |
| `is_background` | Whether the action was background: `true` or `false`. Background actions do not count toward the action rate limit |
| `namespace_mode` | Indicates if actions are produced by an `active` or a `standby` Namespace |

> **📝 Note:**
>
> Does not include the `region` label. Actions are scoped to the Namespace level.
>

#### temporal\_cloud\_v1\_billable\_action\_count

The number of billable actions per second, broken down by action type and Workflow Type. Not all billable actions are included in this metric; see [Actions](/evaluate/cloud/actions) for details on exceptions.

This metric is an estimate and can differ by up to 5% from the absolute Action counts recorded by metering. See
[Compare Action metrics with metering](/evaluate/cloud/actions#compare-action-metrics-with-metering) for guidance on
metered totals.

| Label | Description |
| ----- | ----- |
| `action_type` | The [action](/evaluate/cloud/actions) type |
| `temporal_workflow_type` | The workflow type |

> **⚠️ Caution:**
> High Cardinality
>
> This metric could have high cardinality depending on number of action types and workflow types.
>

**Type**: Rate

#### temporal\_cloud\_v1\_total\_action\_throttled\_count

The total number of actions throttled per second. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) for guidance on setting alert thresholds against the corresponding limit metric.

**Type**: Rate

#### temporal\_cloud\_v1\_operations\_count

Operations performed per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
| `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit |
| `namespace_mode` | Indicates if operations are produced by an `active` or a `standby` Namespace |

**Type**: Rate

#### temporal\_cloud\_v1\_operations\_throttled\_count

Operations throttled due to rate limits per second. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) for guidance on setting alert thresholds against the corresponding limit metric.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
| `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit |
| `namespace_mode` | Indicates if actions are throttled in an `active` or a `standby` Namespace |

**Type**: Rate

### Schedule Metrics

#### temporal\_cloud\_v1\_schedule\_action\_success\_count

Successful scheduled executions per second.

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_buffer\_overruns\_count

Scheduled executions dropped per second because the Schedule's internal Action buffer is full.

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_missed\_catchup\_window\_count

Scheduled executions permanently dropped per second because they fell outside the Catchup Window. This metric is not emitted for backfills.

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_rate\_limited\_count

[DEPRECATED] Scheduled executions delayed per second due to rate limiting. This metric is in the process of being replaced by standard request per second rate limit monitoring with dynamic scaling. This metric will be removed when this change is rolled out to all of Temporal Cloud.

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_overlap\_skipped\_count

Schedule executions dropped per second due to the overlap policy. This metric is emitted for `SKIP` when another execution is running or already selected, and for `BUFFER_ONE` when its single buffered slot is occupied. It can also be emitted during backfills that use `SKIP` or `BUFFER_ONE`. A sustained rate can indicate that executions are generated faster than they complete.

| Label | Description |
| ----- | ----- |
| `schedule_overlap_policy` | The overlap policy that caused the execution to be dropped |

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_action\_e2e\_delay\_p50

The 50th percentile total delay from an execution's scheduled time to actual start, in seconds. Includes overlap-policy wait. Not emitted for backfills.

**Type**: Latency

#### temporal\_cloud\_v1\_schedule\_action\_e2e\_delay\_p95

The 95th percentile total delay from an execution's scheduled time to actual start, in seconds. Includes overlap-policy wait. Not emitted for backfills.

**Type**: Latency

#### temporal\_cloud\_v1\_schedule\_action\_e2e\_delay\_p99

The 99th percentile total delay from an execution's scheduled time to actual start, in seconds. Includes overlap-policy wait. Not emitted for backfills.

**Type**: Latency

### Replication Metrics

#### temporal\_cloud\_v1\_replication\_lag\_p50

The 50th percentile cross-region replication lag in seconds.

**Type**: Latency

#### temporal\_cloud\_v1\_replication\_lag\_p95

The 95th percentile cross-region replication lag in seconds.

**Type**: Latency

#### temporal\_cloud\_v1\_replication\_lag\_p99

The 99th percentile cross-region replication lag in seconds.

**Type**: Latency

### Limit Metrics

#### temporal\_cloud\_v1\_operations\_limit

The current configured operations per second limit for a namespace.

**Type**: Value

#### temporal\_cloud\_v1\_action\_limit

The current configured actions per second limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_total_action_count`` and `is_background=false`.

**Type**: Value

#### temporal\_cloud\_v1\_service\_request\_limit

The current configured frontend service RPS limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_request_count``

**Type**: Value

#### temporal\_cloud\_v1\_poller\_limit

The current configured poller limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_pending_requests``.

**Type**: Value

#### temporal\_cloud\_v1\_action\_on\_demand\_envelope\_limit

The on-demand envelope limit for actions per second. For Namespaces in provisioned capacity mode, this shows what the action limit would be if operating in on-demand mode. For Namespaces already in on-demand mode, this tracks the same value as `temporal_cloud_v1_action_limit`.

> **📝 Note:**
>
> Does not include the `region` label. Limits are scoped to the Namespace level.
>

**Type**: Value

#### temporal\_cloud\_v1\_operations\_on\_demand\_envelope\_limit

The on-demand envelope limit for operations per second. For Namespaces in provisioned capacity mode, this shows what the operations limit would be if operating in on-demand mode. For Namespaces already in on-demand mode, this tracks the same value as `temporal_cloud_v1_operations_limit`.

> **📝 Note:**
>
> Does not include the `region` label. Limits are scoped to the Namespace level.
>

**Type**: Value

#### temporal\_cloud\_v1\_service\_request\_on\_demand\_envelope\_limit

The on-demand envelope limit for service requests per second. For Namespaces in provisioned capacity mode, this shows what the service request limit would be if operating in on-demand mode. For Namespaces already in on-demand mode, this tracks the same value as `temporal_cloud_v1_service_request_limit`.

> **📝 Note:**
>
> Does not include the `region` label. Limits are scoped to the Namespace level.
>

**Type**: Value

#### temporal\_cloud\_v1\_provisioned\_capacity\_tru\_count

The number of Temporal Resource Units (TRUs) provisioned for a Namespace. This is `0` for Namespaces that are not in [provisioned capacity mode](/cloud/capacity-modes).

> **📝 Note:**
>
> Does not include the `region` label. Provisioned capacity is scoped to the Namespace level.
>

**Type**: Value

---

# Set up SDK metrics with Prometheus and Grafana

Source: https://docs.temporal.io/cloud/metrics/sdk-metrics-setup

> Set up Temporal SDK metrics with Prometheus and Grafana for monitoring Workers and Client performance.

SDK metrics are emitted by SDK Clients used to start your Workers and to start, signal, or query your Workflow Executions.
Unlike [Temporal Cloud metrics](/cloud/metrics/openmetrics), which are exposed through the OpenMetrics endpoint, SDK metrics require you to set up a scrape endpoint in your application code for Prometheus to collect and aggregate.

For a full list of available SDK metrics and their descriptions, see the [SDK metrics reference](/references/sdk-metrics).

The process for setting up SDK metrics includes the following steps:

1. [Expose a metrics endpoint](#sdk-metrics-setup) in your application code where Prometheus can scrape SDK metrics.
2. [Configure Prometheus](#prometheus-configuration) to scrape your SDK metrics endpoints.
3. [Add an SDK metrics data source](#grafana-data-source-configuration) in Grafana.
4. [Set up dashboards](#grafana-dashboards-setup) to visualize SDK metrics.

Set up your connections to Temporal Cloud using an SDK of your choice and have some Workflows running on Temporal Cloud.
Ensure Prometheus and Grafana are installed.

- [Go](/develop/go/client/temporal-client#connect-to-temporal-cloud)
- [Java](/develop/java/client/temporal-client#connect-to-temporal-cloud)
- [Python](/develop/python/client/temporal-client#connect-to-temporal-cloud)
- [TypeScript](/develop/typescript/client/temporal-client#connect-to-temporal-cloud)
- [.NET](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud)

## Expose a metrics endpoint 

You must configure a Prometheus scrape endpoint for Prometheus to collect and aggregate your SDK metrics.
Each language development guide has details on how to set this up.

- [Go SDK](/develop/go/platform/observability#metrics)
- [Java SDK](/develop/java/platform/observability#metrics)
- [TypeScript SDK](/develop/typescript/platform/observability#metrics)
- [Python](/develop/python/platform/observability#metrics)
- [.NET](/develop/dotnet/platform/observability#metrics)

For working examples of how to configure metrics in each SDK, see the metrics samples:

- [Go SDK Samples](https://github.com/temporalio/samples-go/tree/main/metrics)
- [Java SDK Samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics)
- [TypeScript SDK Samples](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry)
- [Python SDK Samples](https://github.com/temporalio/samples-python/tree/main/custom_metric)
- [.NET SDK Samples](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry/DotNetMetrics)

Some examples use OpenTelemtry to instrument metrics. It is useful to use a
[Prometheus exporter with OpenTelemetry](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter) to expose metrics for scraping.

## Configure Prometheus 

For Temporal SDKs, you must have Prometheus running and configured to listen on the scrape endpoints exposed in your application code.

For this example, you can run Prometheus locally or as a Docker container.
In either case, ensure that you set the listen targets to the ports where you expose your scrape endpoints.
This configuration assumes the scrape endpoint is set to port 8077 as in the [SDK metrics setup](#sdk-metrics-setup) example.

```yaml
global:
  scrape_interval: 30s # Set the scrape interval to every 30 seconds. Default is every 1 minute.
#...

# Set your scrape configuration targets to the ports exposed on your endpoints in the SDK.
scrape_configs:
  - job_name: 'temporalsdkmetrics'
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets:
          # This is the scrape endpoint where Prometheus listens for SDK metrics.
          - localhost:8077
        # You can have multiple targets here, provided they are set up in your application code.
```

See the [Prometheus documentation](https://prometheus.io/docs/introduction/first_steps/) for more details on how you can run Prometheus locally or using Docker.

To check whether Prometheus is receiving metrics from your SDK target, go to [http://localhost:9090](http://localhost:9090) and navigate to **Status&nbsp;> Targets**.
The status of your target endpoint defined in your configuration appears here.

## Add an SDK metrics data source in Grafana 

Depending on how you use Grafana, you can either install and run it locally, run it as a Docker container, or log in to Grafana Cloud to set up your data sources.

If you have installed and are running Grafana locally, go to [http://localhost:3000](http://localhost:3000) and sign in.

To add the SDK metrics Prometheus endpoint as a data source, do the following:

1. Go to **Configuration&nbsp;> Data sources**.
2. Select **Add data source&nbsp;> Prometheus**.
3. Enter a name for your SDK metrics data source, such as _Temporal SDK metrics_.
4. In the **HTTP** section, enter your Prometheus endpoint in the URL field.
   If running Prometheus locally as described in the examples in this article, enter `http://localhost:9090`.
5. For this example, enable **Skip TLS Verify** in the **Auth** section.
6. Click **Save and test** to verify that the data source is working.

If you see issues in setting this data source, check whether the endpoints set in your SDKs are showing metrics.
If you don't see your SDK metrics at the scrape endpoints defined, check whether your Workers and Workflow Executions are running.
If you see metrics on the scrape endpoints, but Prometheus shows your targets are down, then there is an issue with connecting to the targets set in your SDKs.
Verify your Prometheus configuration and restart Prometheus.

If you're running Grafana as a container, you can set your SDK metrics Prometheus data source in your Grafana configuration.
See the example Grafana configuration described in the [Prometheus and Grafana setup for open-source Temporal Service](/self-hosted-guide/monitoring#grafana) article.

## Set up Grafana dashboards 

To set up SDK metrics dashboards in Grafana, you can use the UI or configure them directly in your Grafana deployment.

> **💡 Tip:**
>
> Temporal provides community-driven [example dashboards for Temporal SDKs](https://github.com/temporalio/dashboards/tree/master/sdk) that you can customize to meet your needs.
>

To import a dashboard in Grafana:

1. In the navigation bar, select **Dashboards** > **Import dashboard**.
2. You can either copy and paste the JSON from the [Temporal SDK sample dashboards](https://github.com/temporalio/dashboards/tree/master/sdk), or import the JSON files into Grafana.
3. Save the dashboard and review the metrics data in the graphs.

To configure dashboards with the UI:

1. Go to **Create > Dashboard** and add an empty panel.
2. On the **Panel configuration** page, in the **Query** tab, select the "Temporal SDK metrics" data source that you configured earlier.
3. Expand the **Metrics browser** and select the metrics you want.
   A list of Worker performance metrics is described in the [Developer's Guide - Worker performance](/develop/worker-performance).
   All SDK-related metrics are listed in the [SDK metrics](/references/sdk-metrics) reference.
4. The graph should now display data based on your selected queries.
   Note that SDK metrics will only show if you have Workflow Execution data and running Workers.
   If you don't see SDK metrics, run your Worker and Workflow Executions, then monitor the dashboard.

---

# Migrate

Source: https://docs.temporal.io/cloud/migrate

> Migrate Temporal workflows with zero downtime, whether it's from self-hosted Temporal Server to Temporal Cloud, or between different regions or cloud providers within Temporal Cloud.

Learn how to migrate your Temporal workflows with zero downtime:

- [Automated Migration](/cloud/migrate/automated) - This process automates the transition from self-hosted Temporal instances to Temporal Cloud.
- [Manual Migration](/cloud/migrate/manual) - This process enables transitions from self-hosted Temporal instances to Temporal Cloud by updating clients
  and workflows to utilize new resources within Temporal Cloud.
- [Migrate between regions or cloud providers](/cloud/migrate/between-regions-or-cloud-providers) - Move a Temporal Cloud Namespace to another region or cloud provider by adding a replica and failing over.
- [Estimate Actions](/cloud/migrate/estimate-actions) - This process helps you estimate Actions and Actions per second from a self-hosted Temporal Service before migrating to Temporal Cloud.

---

# Automated migration

Source: https://docs.temporal.io/cloud/migrate/automated

> **Pre-release**
> Please contact your Temporal account executive prior to planning your migration project.

## Process overview

Automated migration is designed to provide a zero-downtime, secure means of migrating to Temporal Cloud. This guide
outlines the current process for transitioning Workflows from self-hosted Temporal Server to a namespace hosted within
Temporal Cloud.

### Namespace migration schedule

When planning a migration it is highly recommended to migrate in the order of least-critical to most-critical Namespace.
Ideally, the project will begin with Namespaces designated as "testing", where downtime is an acceptable outcome of the
testing process. From there, prioritize migrations based on order of least potential impact.

![Temporal automated migration components](https://docs.temporal.io/img/cloud/migration/ns_migration_schedule.png)

### Project phases

The migration process is separated into several phases, part of which involves coordinating with Temporal to create
necessary cloud-side resources.

Migration involves the following phases:

1. Prepare - Initial preparation involves collecting and evaluating key data points from the self-hosted clusters.
   Collected data will be evaluated by Temporal to ensure eligibility for migration.
2. Setup - Once eligibility has been verified, configurations for the self-hosted clusters will be modified to support
   the migration. All cloud-side components will be provisioned and the self-hosted
   [S2S Proxy](https://hub.docker.com/r/temporalio/s2s-proxy/tags) will be deployed.
3. Test - Once all required components are in place, the migration process will be validated using a test Namespace.
4. Initiate - At the conclusion of a successful testing process, migration of production Namespaces will begin.
5. Finalize - After the Namespace has been transferred to Temporal Cloud and validated, the migration will be finalized.

Please review the [additional notes](#additional-notes) section prior to planning the migration.

## Phase 1: Prepare

In preparation for migrating to Temporal Cloud, several data points must be collected and provided to Temporal via
support ticket. 

### Capture cluster configurations
Depending on your server version, use one of the following methods to capture the configuration of each cluster.

For server versions **above 1.28.1**, run the following command against each of your clusters:

```
temporal operator cluster describe --address <frontend:7233> --output json
```

For server versions **1.28.1 and prior**, use one of the following alternate methods:

```
tctl --address <frontend:7233> admin cluster describe
```

or

```
grpcurl -v -plaintext <your temporal address and port> temporal.server.api.adminservice.v1.AdminService.DescribeCluster
```

### Capture custom search attributes

Temporal must ensure that your custom Search Attributes are compatible with Temporal Cloud.
Capture any custom Search Attributes using one of the following commands:

For Elasticsearch/OpenSearch
```
temporal operator search-attribute list
```

For SQL
```
temporal operator search-attribute list --namespace="your_namespace"
```

### Capture Namespace metrics

Metrics are used for cloud-side resource planning. For each Namespace, collect the following:

- Total number of open/closed Workflows
- Total storage used
- Current retention policy. Note that this may differ from the
  [default retention policy](/evaluate/cloud/limits#default-retention-period) in Temporal Cloud.
- Peak [action per second](/glossary#actions-per-second-aps) (APS). For instructions on collecting these metrics, see
  [Estimate Actions for migration](/cloud/migrate/estimate-actions).

When planning an automated migration, share the following estimates with your Temporal account team:

- Peak APS for each Namespace.
- Average APS for each Namespace, if available.
- Fixed-range Action counts, such as total Actions over the last 30 days.
- Representative Workflow Types and their estimated Actions per Workflow Execution.
- Retention Periods and storage estimates.

These estimates help Temporal plan capacity, Namespace limits, and migration timing.

### Capture schedules
Capture a list of schedules on the system. This helps to ensure backwards compatibility.

```
temporal schedule list
```

### Prepare mTLS certificates

mTLS is used to secure the [S2S Proxy](#configure-the-proxy) communications channel. Provide a single certificate using the process outlined
[here](/cloud/certificates#issue-certificates).

Verify your end-entity certificate with the following command.

```
openssl verify -CAfile ca.pem client-cert.pem
```

In the example above, the ca.pem file will be provided to Temporal where the client-cert.pem will be used by your S2S Proxy.

### Create Cloud Namespaces

1. Create your cloud-side Namespaces noting current [naming requirements](/cloud/namespaces#temporal-cloud-namespace-name).
2. Apply any required custom Search Attributes and adjust the [rate limits](/cloud/capacity-modes) as needed.

Migration cannot proceed into a Namespace that is already in use. Please ensure that these Namespaces remain empty (no Workflows).

If you are new to Temporal Cloud, consider your connectivity path to cloud. You may connect over the
public internet or via [private connectivity](/cloud/connectivity).

### Report collected data
Provide all collected data to Temporal via a [support ticket](/evaluate/cloud/support#ticketing). 

In your ticket, please provide:
* mTLS certificate - base64 encoded pem file works well for easy submission in the ticket
* cluster configurations -  json output for each cluster
* custom search attributes - CLI output for each cluster/Namespace
* list of schedules used
* Namespace metrics
* cluster/Namespace mappings - use CSV format (see below)

Sample CSV file for reporting cluster/Namespace mappings. Use a separate file for each Temporal Cloud account.

```
cluster_name, cloud_region, source_namespace, cloud_namespace
cluster1,     us-east-1,    default,          use1.nnnnn
cluster1,     us-east-1,    ns2,              use2.nnnnn
cluster2,     us-central1,  default,          usc1.nnnnn
```

## Phase 2: Setup

Once the migration has been approved, the next step is to prepare both the self-hosted clusters and Temporal Cloud
resources for the migration.

> **⚠️ Warning:**
>
> Proceed only when your request has been approved by Temporal.
>

### Plan migration endpoints

A migration endpoint is a cloud-side resource that connects one of your self-hosted servers to Temporal Cloud. Temporal
creates the endpoints and provides the `endpoint-id` values used with the `tcld migration` commands in later phases.

Each self-hosted server you migrate from needs its own S2S Proxy deployment and its own migration endpoint. You can
migrate from more than one self-hosted server at the same time, such as separate development and production servers,
provided each self-hosted server has a unique cluster name and Initial Failover Version. Temporal uses `active` as the
default cluster name, so servers that still use the default will collide. See
[Modify cluster configuration](#modify-cluster-configuration) for setting these values.

If your self-hosted servers cannot use unique cluster names, migrate one server at a time, or contact Temporal to
discuss the available options.

Namespaces migrating from the same self-hosted server share that server's proxy and endpoint. You can migrate more than
one Namespace from a server at the same time, provided each source Namespace and its cloud-side target are listed in
the `namespaceTranslation` mappings of that server's proxy configuration. See
[Configure the proxy](#configure-the-proxy).

Tell Temporal how many self-hosted servers you plan to migrate from, and whether you intend to migrate from them in
parallel, so that the required cloud-side resources are in place.

### Configure the proxy

The [S2S Proxy](https://github.com/temporalio/s2s-proxy) requires a cloud-side inbound endpoint. Proceed with deployment
only after receiving the endpoint from Temporal.

The proxy provides API forwarding over a secure 2-way tunnel to Temporal Cloud. The self-hosted proxy will initiate an
outbound connection (TCP 8233) to its cloud-side counterpart and establish the 2-way tunnel. If there are firewalls
in-path, ensure that they permit this outbound connection.

![Temporal automated migration components](https://docs.temporal.io/img/cloud/migration/auto-migration-components.png)

Use the following procedure to deploy the proxy:

1. Obtain the latest Docker image from the
   [temporalio/s2s-proxy repository](https://hub.docker.com/r/temporalio/s2s-proxy/tags).
2. Gather the mTLS certs generated in the previous step.
3. Deploy **3 replicas** of the s2s-proxy (minimum 4 CPU and 512mb memory). For Kubernetes users, use this
   [helm chart example](https://github.com/temporalio/s2s-proxy/blob/main/charts/s2s-proxy/README.md) as a reference.
   See the [example](https://github.com/temporalio/s2s-proxy/blob/main/charts/s2s-proxy/example.yaml) configuration file
   as a reference. Note that the number of replicas must match on both sides of the connection. If your replica count differs
   from 3, update Temporal with the actual number of replicas so that the same count can be configured on the cloud side.
4. Verify that you have included your self-hosted namespaces and their translations in your proxy configuration. Example:
```yaml
aclPolicy:
  allowedNamespaces:
   - temporal-system  # required - do not remove
   - namespace1
   - namespace2
   - etc...
namespaceTranslation:
  mappings:
  - local: "namespace1"
    remote: "cloud-namespace1.wxyz"
  - local: "namespace2"
    remote: "cloud-namespace2.wxyz"
  - etc...
```
5. Test access using the command below. It should display the information of the cloud-side migration server.

   The address is your own proxy's external address, not a cloud-side address. It's the same value you set as
   `replicationEndpoint` under `clusterConnections` in your proxy configuration file.

```
temporal operator cluster describe --address {the-outbound-external-address-of-your-proxy}
```

There are multiple metrics available on the S2S proxy (prometheus endpoint: _proxy-pod-ip_:9090/metrics). These are
helpful for monitoring the overall health of the proxy. In particular, the metric

`temporal_s2s_proxy_mux_connection_active`

will monitor connectivity to the cloud-side proxy.

### Modify cluster configuration

> **⚠️ Warning:**
>
> Coordinate closely with Temporal before completing this process.
>

The [dynamic configuration](/references/dynamic-configuration) of your self-hosted cluster must be modified to
facilitate the migration.

Complete the following process.

1. Adjust the maximum connection keepalive time to match the setting in cloud.

```yaml
frontend.keepAliveMaxConnectionAge:
  - value: '2h'
```

2. For server versions 1.22.6 - 1.23.x, apply this [extra required setting](#special-dynamic-configuration-for-version-122---123).
```yaml
history.enableReplicationStream:
  - value: true
```

3. If [Global Namespace](/global-namespace) **is already enabled**, then skip to step 5.

4. If [Global Namespace](/global-namespace) **is not enabled**, then enable it and set _failoverVersionIncrement_ and _initialFailoverVersion_
   to the values provided by Temporal. Pay close attention when setting these values. They **cannot be changed** once Global Namespace has 
   been enabled. See the sample configuration below for a reference.

```yaml
dcRedirectionPolicy:
  policy: 'all-apis-forwarding'

clusterMetadata:
enableGlobalNamespace: true # add this
failoverVersionIncrement: CHANGEME # use value provided by Temporal
masterClusterName: _NO_CHANGE_
currentClusterName: _NO_CHANGE_
clusterInformation:
  _NO_CHANGE_:
    enabled: true
    initialFailoverVersion: CHANGEME # use value provided by Temporal
    rpcName: _NO_CHANGE_
    rpcAddress: _NO_CHANGE_

# for versions 1.22 - 1.23 only
#history.enableReplicationStream:
#  - value: true
```

5. Verify your configuration and restart all Temporal services (frontend, history, matching, worker), starting with the frontend.

6. After all services have restarted, verify the configuration using:

```
temporal operator cluster describe
```

The following sample output is expected:

```yaml
"failoverVersionIncrement": "nnn",
"initialFailoverVersion": "nnn"
"isGlobalNamespaceEnabled": true
```

### Verify current cluster utilization
It is important to ensure that your production cluster has been allocated enough resources to support the migration. In particular, 
it is important to verify that your persistence/database layer has plenty of CPU and I/O capacity.

## Phase 3: Testing
Once the proxy is deployed and the cluster configuration changes have been applied, then final testing may begin.

### Verifying the proxy

Temporal will validate the proxy from the cloud side of the connection.  Temporal will validate
1. the stability of the proxy connection
2. that all required permissions are allowed on the self-hosted proxy
3. that all cluster configuration changes have been applied
4. that all to-be-migrated Namespaces are visible
5. the presence of Custom Search Attributes

Once the self-hosted setup has been verified, the next step will be to perform test migrations.

### Complete test migrations
Migration testing should use either a newly created Namespace or else one that is considered to be non-production. 
It is ideal to have a mix of completed and running Workflows to use during testing.

Testing uses the following process:

1. Create or identify a non-production Namespace that can tolerate data loss in the event of issues.
2. Create target cloud-side Namespace and add the Namespace definition to the S2S Proxy configuration.
3. Run test Workflows against the Namespace.
4. Perform a complete end-to-end migration for the Namespace (see remaining phases for full process).
5. Optionally, test [transferring clients](#transfer-clients-to-cloud) to the cloud.

Testing is considered successful if all data from the self-hosted deployment is migrated to cloud.

## Phase 4: Initiate
Review the section on [transferring clients](#transfer-clients-to-cloud) before proceeding.

The sections below outline the process for initiating the migration.

### Migration start

Temporal will generate the endpoint-id and initiate the migration. During this process, the self-hosted Namespace remains active 
while the cloud Namespace becomes passive. Workflows are replicated from the self-hosted Namespace to the cloud Namespace. Once the
cloud Namespace has fully synced with self-hosted Namespace, migration is ready for handover.

> **📝 Note:**
> Billing
>
> Billing for the cloud Namespace does not begin until the migration is [confirmed](#confirm-complete).
>

The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-start-a-migration) is used to start the
migration:

```
tcld migration start --endpoint-id <endpoint-id> --source-namespace <source-namespace> --target-namespace <target-namespace>
```

### Monitor

During the initial sync, it is important to monitor the overall process to ensure progress is being made. While Temporal
will monitor from the cloud-side, progress may also be monitored from the self-hosted side using the
_replication_stream_stuck_ metric.

The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-get-a-migration) may also be used to
monitor the migration progress:

```
tcld migration get --id <migration-id>
```

### Handover-to-cloud

Once the sync process has completed, Temporal will flip the roles of the self-hosted and cloud Namespace. At this point,
the cloud becomes active and the self-hosted Namespace becomes passive. Workflows are then replicated from the cloud to
the self-hosted server.

The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-perform-handover-during-a-migration) is
used to trigger the handover:

```
tcld migration handover --id <migration-id> --to-replica-id <to-replica-id>
```

When using this command, replace `<to-replica-id>` with `cloud` when handing over to Temporal Cloud. Replace
`<to-replica-id>` with `on-prem` when handing back to the self-hosted setup.

## Phase 5: Finalize
If you have not done so already, complete the process of [transferring clients](#transfer-clients-to-cloud) to the cloud Namespace.

### Final validation

Use the following checklist prior to finalizing the migration:

- Confirm that workers can access Namespaces. Either via public internet or [private connectivity](/cloud/connectivity).
- Understand how to access metrics for your Namespace on Temporal Cloud.
- Monitor general Workflow metrics (schedule to start latency, start v.s. completion rate, sync match rate, etc).
- Learn how [capacity management](/cloud/capacity-modes) works in Temporal Cloud.
- Plan for a worker tuning session - performance change between Temporal Cloud v.s. self-hosted cluster, which could
  lead to unexpected symptoms and optimizations.
- Know how to reach out to your Temporal Solutions Architect (SA) and Account Executive (AE) for assistance.

### Confirm complete 

Once a Namespace has been transferred to the cloud and validated, the migration will be completed. Note that this step
is final and may not be undone. Once performed, Workflow replication from the cloud Namespace to the self-hosted server
is halted.

The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-confirm-a-migration) is used to confirm the
migration:

```
tcld migration confirm --id <migration-id>
```

or to [abort](https://pkg.go.dev/github.com/temporalio/tcld#readme-abort-a-migration) and roll-back changes without
impacting your Workflows, if needed:

```
tcld migration abort --id <migration-id>
```

## Transfer clients to cloud

There are two options for switching Temporal clients to the cloud.

### Option 1 (recommended)

Deploy two sets of Temporal clients: one pointing to your Temporal server and one to the Cloud Namespace endpoint. This
is the recommended option since your Workflows will continue to make progress during the handover, even if your cloud
Temporal client is unable to access the cloud (due to misconfiguration, for example). The process is as follows:

1. Direct your cloud-based Temporal clients to the cloud Namespace endpoint. Initially, these clients will connect and
   send Poll requests but will not receive any tasks.
2. Start migration. Your self-hosted Namespace is active while your cloud Namespace is passive (or standby). Your cloud
   Temporal clients can begin receiving tasks, but all requests from cloud clients to the Cloud Namespace will
   automatically forward from the cloud to your self-hosted server.
3. Hand over Namespace to the cloud. Your cloud Namespace becomes active and your self-hosted Namespace becomes passive.
   All requests from your self-hosted Temporal clients will automatically forward from your server to the cloud.
4. Complete migration. Your self-hosted Temporal clients will no longer receive any tasks from your server, allowing you
   to stop these clients.

### Option 2

Deploy one set of Temporal clients and switch the Namespace endpoint during migration. With this option, if your workers
are misconfigured during the switch, then it is possible that Workflows can stop making progress. It is important to
ensure that all workers maintain connectivity to cloud to avoid this scenario. The process is as follows:

1. Start migration.
2. Switch your Temporal clients to point to the cloud Namespace endpoint. Requests from your Temporal clients will
   automatically forward from the cloud to your server. Alternatively, you may switch Temporal clients to the cloud
   Namespace endpoint after handover.
3. Hand over Namespace to the cloud. Requests from your Temporal clients will now be served by the cloud and will not be
   forwarded to your server.
4. Confirm migration completion.

## Additional notes

### Limitations

The following are known limitations.

- OSS server versions 1.22.6 or newer are required. Refer to the
  [upgrade](/self-hosted-guide/upgrade-server#upgrade-server) procedure as needed.
- History shard counts must be a power of two (for example, 512 or 1024).
- If you have multiple self-hosted servers and they are all configured with the same cluster name (by default Temporal
  uses 'active' as cluster name), they cannot be connected to a single migration server simultaneously due to cluster
  name collision. There are 2 available options:
  1. Migrate one server at a time using a single migration server.
  2. Create multiple migration servers (one for each self-hosted server) if you need to migrate all servers
     simultaneously.
- If you are using multi-cluster replication in your self-hosted setup and have previously failed over Namespaces, then
  this may impact your eligibility for automated migration. Specifically, whenever Global Namespace has been previously
  enabled the following restrictions apply:
  1. Initial Failover Version must be less than or equal to 1,000,000
  2. Failover Version Increment must be a divisor of 1,000,000 (for example, 10)
- OSS supports cross-Namespace commands (for example, parent-child, SignalExternal, CancelExternal) through the
  `system.enableCrossNamespaceCommands` configuration. This configuration is disabled on Temporal Cloud. The
  `system.enableCrossNamespaceCommands` configuration must be disabled, and code using cross-Namespace calls must be
  updated or removed prior to migration.

### Special dynamic configuration for Version 1.22 - 1.23

Temporal versions 1.22 and 1.23 include support for stream-based replication, but it is disabled by default. Since
those releases, stream-based replication has been validated as more reliable than the poll-based replication that
remained the default in 1.22 and 1.23.

When preparing for an S2C migration on these versions, configure the following dynamic settings to enable stream-based
replication:

```yaml
history.enableReplicationStream:
  - value: true
```

Enabling this configuration will require a restart of your history pods.

## Frequently asked questions

### How many Temporal Cloud accounts do I need?

If you are new to Temporal Cloud, then the most common recommendation is to create a single Account that contains your Namespaces.

### When should I opt for auto migration?

The answer depends on your specific situation. However, automated migration is most likely to help if any of the following apply to you:

* When safe transfer of workers is a concern - automated migration allows workers to run against both self-hosted and cloud
  environments simultaneously, allowing a gradual and lower-risk transition.
* When there are a high number of long-running Workflows.
* When there is a need to migrate closed Workflows.
* When there is a need to migrate Schedules.

In contrast, automated migration may not be the best solution if your self-hosted clusters do not meet the [minimum requirements](#limitations).

### Can I migrate from multiple self-hosted servers at the same time?

Yes. Each self-hosted server needs its own S2S Proxy deployment and its own migration endpoint, and each must have a
unique cluster name and Initial Failover Version. See [Plan migration endpoints](#plan-migration-endpoints).

### Can I migrate multiple Namespaces from the same self-hosted server at the same time?

Yes. Those Namespaces share the server's proxy and endpoint, so no additional cloud-side resources are needed. Include
each source Namespace and its cloud-side target in the `namespaceTranslation` mappings of that server's proxy
configuration. See [Configure the proxy](#configure-the-proxy).

### Can I split Workflows from a single source Namespace into multiple cloud-side Namespaces?
No. All Workflows will be migrated.

### Can I combine manual and auto migration?
No. Auto migration requires a "fresh" target cloud-side Namespace (one that has never had a running Workflow). 
If Workflows were manually migrated to a cloud-side Namespace, then this Namespace would not be suitable as an auto-migration target.

### Why does it matter if custom search attributes are used?
Custom search attributes must be mapped to a Namespace in Temporal Cloud. They matter because configurations in a self-hosted environment 
may not be directly compatible with Temporal Cloud, potentially requiring additional migration work. The exact process can also differ 
depending on the type of visibility data store used.

### What Workflows are migrated by default?
All Workflows are migrated by default. For closed Workflows, you may specify a date range to be migrated. Your cloud-side Namespace must
be configured with your desired retention period prior to starting the migration.

### What can I do to speed up an automated migration?
The #1 speed optimization is to limit the time range for closed Workflows. This will reduce the amount of data required to be 
migrated and in many cases will dramatically reduce overall migration time. 

### Is the migration of Schedules supported?
Yes. Under the hood, Schedules are essentially Workflows.

### I have a long retention period for my Workflows. Is this compatible with Temporal Cloud?
Occasionally, self-hosted [retention periods](/temporal-service/temporal-server#retention-period) are in excess of what
is [supported](/evaluate/cloud/limits#default-retention-period) in Temporal Cloud. In these cases it is recommended to utilize
[archival](/temporal-service/archival) to store closed Workflows that cannot be migrated. In general, archival is
recommended over large retention periods since the extra data can stress the persistence layer of the system.

### I am using payload encryption in my self-hosted Temporal cluster. Is this supported in cloud?
Yes. If payloads are already [encrypted](/payload-codec#encryption) in your self-hosted server via data converter, then 
they will remain encrypted during and after migration.

### I would like to enable payload encryption as part of the migration. Is this supported?
The automated migration tooling cannot add payload encryption. To encrypt payloads sent to Temporal Cloud, you must encrypt
payloads in your cluster before starting the automated migration process.

---

# Migrate between regions or cloud providers

Source: https://docs.temporal.io/cloud/migrate/between-regions-or-cloud-providers

> Move a Temporal Cloud Namespace to another region or cloud provider with zero downtime by adding a replica, waiting for it to activate, and failing over.

Temporal Cloud's [High Availability features](/cloud/high-availability) allow you to migrate a Temporal Cloud Namespace from one region or cloud provider to another with zero downtime.

## Preparing to migrate

Namespaces using Export will need to stop Export and migrate the region configuration to the new region for Export jobs to continue after migration. 
See [failover scenarios](/cloud/export#failover-scenarios) for details.

[Using High Availability features affects pricing](/cloud/pricing#high-availability-features).

> **ℹ️ Info:**
> Using AWS PrivateLink or GCP Private Service Connect?
>
> If the Namespace uses Private Connectivity, the steps below need additional DNS and VPC Endpoint work. Follow [How to migrate to another Temporal Cloud Region when using Private Connectivity](/cloud/high-availability/ha-connectivity#how-to-migrate-regions-with-private-connectivity) instead of (or alongside) the public steps below.
>

## Steps to migrate

> **⚠️ Caution:**
> GCP capacity restriction
>
> Before creating a GCP Namespace or adding a GCP replica, check [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction) for regional availability and exception requirements.
>

1. Add a Namespace replica in the region you want to migrate to. See [regions](/evaluate/cloud/regions) for a list of available regions and supported multi-region and multi-cloud configurations.

![Add a namespace replica](https://docs.temporal.io/img/cloud/high-availability/migrate/1-add-replica.png)

![Choose the region for the replica](https://docs.temporal.io/img/cloud/high-availability/migrate/2-choose-region.png)

2. Wait for the replica to become active. The Cloud UI will display a time estimate, and namespace admins will receive an email when the replica is active.
3. If your Workers are using [Regional Endpoints](/cloud/namespaces#access-namespaces), ensure they are updated to use the Regional Endpoint of the [replica's region](/evaluate/cloud/regions).
4. Trigger a failover to the new region.

![Initiate failover to the new region](https://docs.temporal.io/img/cloud/high-availability/migrate/3-failover.png)

5. [Prepare connectivity before removing the replica](/cloud/high-availability/enable#prepare-connectivity-before-removing-a-replica), then remove the Namespace replica in the region you are migrating from.

![Remove the replica for the original region](https://docs.temporal.io/img/cloud/high-availability/migrate/4-remove-replica.png)

> **📝 Note:**
> All replica changes are subject to a [cooldown period](/cloud/high-availability/enable#changing) before further replica changes can be made.
>

---

# Estimate Actions for migration

Source: https://docs.temporal.io/cloud/migrate/estimate-actions

> Estimate Temporal Cloud Actions and Actions per second from a self-hosted Temporal Service before migrating to Temporal Cloud.

Before you migrate from a self-hosted Temporal Service to Temporal Cloud, you can measure your current Action usage to help predict your Cloud usage and costs.

Use server metrics to estimate Actions per second (APS) and peak load.
Use representative Workflow Execution Histories to understand how many Actions a typical Workflow Execution creates.
Estimate storage separately from Actions.

For details about which operations count as Actions in Temporal Cloud, see [Temporal Cloud Actions](/evaluate/cloud/actions).

## Choose an estimation method

Use one or more of the following methods depending on what you need to estimate.

| Method | Use it to estimate | Notes |
| --- | --- | --- |
| Self-hosted server metrics | APS, peak APS, and Action counts over a fixed time range | Best for sizing Namespace limits and understanding usage patterns. |
| Workflow Execution History samples | Actions per representative Workflow Execution | Best for understanding workload shape by Workflow Type. Some [Actions do not appear](/evaluate/cloud/actions) in Event History. |
| Storage estimates | Event History storage | Storage is [priced separately](/cloud/pricing#pricing-model) from Actions. |

## Estimate APS from self-hosted server metrics

Temporal Server versions later than 1.17 provide an `action` metric.
Use this metric to estimate Actions per second from a self-hosted Temporal Service.

Temporal Server versions 1.22.3 and later provide an `action` metric that more closely reflects current Temporal Cloud Action pricing, including Local Activity metering.
For Temporal Server versions from 1.17 through 1.22.2, use the `action` metric to understand server load, but do not treat it as a precise billable Action estimate.

To calculate total APS, use the following PromQL query:

```promql
sum(rate(action{service_name="frontend"}[1m]))
```

To calculate APS by Namespace, use the following PromQL query:

```promql
sum(rate(action{service_name="frontend"}[1m])) by (exported_namespace)
```

Depending on your metrics exporter setup, the Namespace label might be `namespace` instead of `exported_namespace`.

For a Grafana dashboard example, see the [`server-general.json`](https://github.com/temporalio/dashboards/blob/master/server/server-general.json) dashboard in the Temporal dashboards repository.

For Datadog, use a query like the following to calculate Actions per second:

```text
sum:io.temporal.server.action.count{$server-name}.as_rate()
```

## Estimate Action counts over a fixed time range

To estimate total Actions during a fixed time range, use `increase()` over the range you want to measure.

For example, to estimate total Actions over 30 days, use the following PromQL query:

```promql
sum(increase(action{service_name="frontend"}[30d]))
```

To estimate Actions for one Namespace, add the Namespace label:

```promql
sum(increase(action{service_name="frontend", exported_namespace="default"}[30d]))
```

When you run fixed-range queries in Grafana, set the end of the dashboard time window to the end of the date range that you want to measure.
For example, to measure Actions for the 30-day period ending March 31, set the dashboard end time to March 31.

## Estimate Actions from Workflow Execution Histories

You can estimate Actions per Workflow Execution by counting billable events in representative Workflow Execution Histories.
This method is useful when you need to understand how much Action usage a Workflow Type creates.

1. Choose representative Workflow Executions for each Workflow Type.
2. Download the Event History for each Workflow Execution from the Web UI or API.
3. Count the events that map to Temporal Cloud Actions.
   For the list of Action types and corresponding History Event types, see [Temporal Cloud Actions](/evaluate/cloud/actions).
4. Multiply the Action count by the expected number of Workflow Executions in the period you want to estimate.

This method produces an estimate.
Some Actions do not appear directly in Event History, including Queries and some Activity Heartbeat Actions.
If you use Global Namespaces, account for the additional Action and storage cost.
Estimate storage separately from Workflow Action counts.

## Estimate storage separately

Temporal Cloud storage is priced separately from Actions.
To estimate storage, collect the following information:

- Event History size for representative Workflow Executions.
- Retention Period for closed Workflow Executions.
- Duration for open Workflow Executions.
- Expected number of Workflow Executions.

For current storage pricing, see [Temporal Cloud pricing](/cloud/pricing).

---

# Manual migration

Source: https://docs.temporal.io/cloud/migrate/manual

> Migrating to Temporal Cloud from self-hosted Temporal Service varies by Workflow requirements. This guide covers changing Client code, Workflow migration strategies, and necessary code adjustments.

Migrating to Temporal Cloud from a self-hosted Temporal Service will have different requirements depending on your usage.
This guide provides some guidance based on our experience helping customers of all sizes successfully migrate.

### What to expect from a migration

Depending on your Workflows' requirements, the migration process may be as simple as changing a few parameters, or may require more extensive code changes.
There are two aspects to consider when migrating: your Temporal Client connection code and your Workflow Executions.
Here's a high-level overview of what you can expect:

- **Introduce another Temporal Client to your Starter and Worker Processes:** Configure and deploy a new Temporal Client so that Temporal Cloud becomes responsible for new Workflow Executions.
- **Migrate Workflow Executions:** There are different approaches for new, running, and completed Workflow Executions.
  - **New Workflow Executions:** When you no longer need to send Signals or Queries to your self-hosted Temporal Service, you can deprecate your old Client code. Until then, your self-hosted Temporal Service can receive relevant traffic, while new Workflow Executions are sent to Temporal Cloud.
  - **Running Workflow Executions:** Short-running Workflows can often be drained and then started again on Temporal Cloud. Long-running Workflows that cannot be drained might require you to implement more code changes to pass the state of the currently running Workflow to Temporal Cloud.
  - **Completed Workflow Executions:** Completed Workflow Execution History cannot be automatically migrated to Temporal Cloud. Refer to [Multi-Cluster Replication](#multi-cluster-replication) for more information.

### Updating Client connection code in your Workers

Whether you're self-hosting Temporal or using Temporal Cloud, you manage runtime of your code.
To migrate your Workflows to Temporal Cloud, you need to change some parameters in the Client connection code, such as updating the namespace and gRPC endpoint.

The changes needed to direct your Workflow to your Temporal Cloud
Namespace are only a few lines of code, including:

- Add your [SSL certificate and private key](/cloud/certificates) or [API key](/cloud/api-keys) associated with your Namespace.
- [Copy the Cloud-hosted endpoint](/cloud/namespaces#temporal-cloud-grpc-endpoint) from the Namespace detail Web page.
  The endpoint uses this format: `<namespace_id>.<account_id>.tmprl.cloud:port`.
- [Connect to Temporal Cloud](/cloud/get-started) with your Client.
- [Configure tcld, the Cloud CLI](/cloud/tcld), with the same address, Namespace, and
  certificate used to create a Client through code.

### Migrating your Workflow Executions

A Temporal Service stores the complete Event History for the entire lifecycle of a
Workflow Execution.
To migrate from a self-hosted Temporal Service to Temporal Cloud, take into account the current state, Event History, and any future expectations of your Workflow Executions.

**New Workflows are automatically executed on Temporal Cloud.**
Once you've made the code changes in Step 1, and your new code is deployed, new Workflow Executions will be sent to Temporal Cloud.
Existing Workflows must receive Signals to migrate and re-execute on Cloud.
If you maintain your self-hosted instance, you will still be able to use it to access any execution history from before your migration.
You can also export JSON from your previous execution history, that you can then import into your own analytics system.

**Running Workflows can either be drained or migrated.**
If your Workflow can be completed before any compelling event which drives a move to Temporal Cloud, those Workflows can be automatically restarted on Temporal Cloud.
If your Workflows need to run continuously, you must migrate Workflows while they are running.
To accomplish this migration, cancel your current Workflow and pass the current state to a new Workflow in Temporal Cloud.
Refer to [this repository](https://github.com/temporalio/temporal-migration) for an example of migrating running Workflows in Java.

When performing a live migration, make sure your Worker capacity can support the migration load.
Both a [Signal](/sending-messages#sending-signals) and a [Query](/sending-messages#sending-queries) will be executed during the course of the migration.
Also, the Query API loads the entire history of Workflows into Workers to compute the result (if they are not already cached).
That means that your self-hosted Temporal Service Worker capacity will need to support having those executions in memory to serve those requests.
The volume of these requests might be high to execute against all the matches to a `ListFilter`.

### Considerations when resuming Workflows on a new Temporal Service or Namespace

- **Skipping Steps:** If your Workflow steps cannot guarantee idempotency, determine whether you need to skip those steps when resuming the execution in the target Namespace.
- **Elapsed Time:** If your Workflow is “resuming sleep” when in the target Namespace, determine how you will calculate the delta for the sleep invocation in the new execution.
- **Child Relationships:** If your Workflow has Child Workflow relationships (other than Detached Parent Close Policy children), determine how you can pass the state of those children into the parent to execute the child in a resumed state.
- **Heartbeat state:** If you have long running activities relying on heartbeat state, determine how you can resume these activities in the target Namespace.
- Child Workflows with the same type as their Parent types are returned in List Filters used to gather relevant executions. Unless these are Detached `ParentClosePolicy` children, this is not what you want since the Parent/Child relationship will not be carried over to the target Namespace.
- Long running activities that use heartbeat details will not receive the latest details in the target Namespace.
- Duration between Awaitables inside a Workflow definition needs to be considered for elapsed time accuracy when resuming in the target Namespace.
- When Signaling directly from one Workflow to another, make sure to handle `NotFound` executions in the target Namespace. The Workflows may resume out of order.

### Other considerations when migrating

- Have you added an mTLS certificate to your Temporal Namespace? Review our [documentation for adding a certificate to your Temporal Cloud account](/cloud/certificates) for more information.
- There are differences in how metrics are generated in self-hosted Temporal and Temporal Cloud. Review the [Cloud metrics reference](/cloud/metrics/openmetrics/metrics-reference) for more information.
- Consider the implications for [security and access to your Temporal Service](/evaluate/cloud/security).
- Review your current load with [Action estimates](/cloud/migrate/estimate-actions) and speak to your Account Executive and Solutions Architect so we can set appropriate [Namespace limits](/evaluate/cloud/limits).

### Multi-Cluster Replication

[Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) is an experimental feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters for backup and state reconstruction.
Migrating Execution History from a self-hosted Temporal Service to Temporal Cloud is not currently supported.
However, a migration tool based on Multi-Cluster Replication, which will enable this, is currently in development for Temporal Cloud.
If you have used this feature locally or you are interested in using it to migrate to Temporal Cloud, [create a support ticket](/evaluate/cloud/support) or watch this space for more information about public availability.

---

# Monitor Temporal Cloud

Source: https://docs.temporal.io/cloud/monitor

> Detect Task Queue backlogs, Worker capacity issues, and Temporal Cloud service errors, then route alerts to your monitoring tools.

Monitor Temporal Cloud and the Workers connected to it to detect service issues, Task Queue backlogs, and Worker
capacity problems.

Temporal provides Cloud metrics, SDK metrics, health-monitoring guidance, and notifications. Use the following guides
based on what you need to detect.

## Choose what to monitor

| Goal | Guide | What it helps you detect |
| --- | --- | --- |
| Collect metrics from Temporal Cloud | [Set up Cloud metrics](/cloud/metrics/openmetrics) | Service and Task Queue behavior |
| Collect metrics from Workers and Clients | [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup) | Worker and Client behavior |
| Monitor Temporal Cloud service health | [Monitor service health](/cloud/service-health) | Service latency and errors, Workflow and Activity failure trends, throttling, and capacity limits |
| Monitor Workers connected to Temporal Cloud | [Monitor Worker health](/cloud/worker-health) | Task backlogs, insufficient or excess Worker capacity, configuration problems, and Worker availability |
| Receive operational updates | [Receive notifications](/cloud/notifications) | Temporal Cloud incidents, expiring credentials, billing events, and completed or failed failovers |

## Understand metric sources

Temporal provides two complementary metric sources:

- **Cloud metrics** show what the Temporal Service observes, including request latency, service errors, Task Queue
  behavior, throttling, and Namespace-level activity.
- **SDK metrics** show what your Workers and Clients observe, including Schedule-To-Start latency, available Worker
  slots, cache behavior, and request failures.

Use both sources to distinguish a Temporal Cloud service issue from a Worker or application issue.

To begin collecting metrics:

1. [Set up Cloud metrics with OpenMetrics](/cloud/metrics/openmetrics).
2. [Connect Temporal Cloud to your observability tool](/cloud/metrics/openmetrics/metrics-integrations).
3. [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup).
4. Use the [service health](/cloud/service-health) and [Worker health](/cloud/worker-health) guides to create monitors and
   alerts.

## Monitor Nexus

To monitor Nexus metrics and debug Nexus Operations, see [Temporal Nexus observability](/cloud/nexus/observability).

For records of Control Plane changes, see [Audit Logs](/cloud/audit-logs). Audit Logs answer who changed a Temporal
Cloud resource, what they changed, and when. They are separate from runtime health monitoring.

---

# Namespaces

Source: https://docs.temporal.io/cloud/namespaces

> **ℹ️ Info:**
> Temporal Cloud
> This page covers namespace operations in **Temporal Cloud**.
> For core namespace concepts, see [Temporal Namespace](/namespaces).
> For open source Temporal, see [Managing Namespaces](/self-hosted-guide/namespaces).

A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique
identifiers, and gRPC endpoints in Temporal Cloud.

- [Create a Namespace](#create-a-namespace)
- [Access a Namespace](#access-namespaces)
- [Manage Namespaces](#manage-namespaces)
- [Delete a Namespace](#delete-a-namespace)
- [Tag a Namespace](#tag-a-namespace)
- [Add a Namespace description](#describe-a-namespace)

## What is a Cloud Namespace Name? 

A Cloud Namespace Name is a customer-supplied name for a [Namespace](/namespaces) in Temporal Cloud. Each Namespace
Name, such as `accounting-production`, is unique within the scope of a customer's account. It cannot be changed after
the Namespace is provisioned.

Each Namespace Name must conform to the following rules:

- A Namespace Name must contain at least 2 characters and no more than 39 characters.
- A Namespace Name must begin with a letter, end with a letter or number, and contain only letters, numbers, and the
  hyphen (-) character.
- All letters in a Namespace Name must be lowercase.

> **📝 Note:**
> The Namespace Name alone is not sufficient to connect to or reference a Namespace. All Temporal Cloud operations — SDK connections, CLI commands, and API calls — require the full **Namespace ID**, which combines your Namespace Name with your Account ID:
>
> `<namespace-name>.<account-id>`
>
> For example: `accounting-production.123de`
>
> The full Namespace ID is shown in the Namespace list in the Temporal Cloud UI and in the output of the `namespace list` command. See [Cloud Namespace ID](#temporal-cloud-namespace-id) for details.

## What is a Temporal Cloud Account ID? 

A Temporal Cloud Account ID is a unique customer identifier assigned by Temporal Technologies. Each Id is a short string
of numbers and letters like `f45a2`, at least five characters long. This account identifier is part of every Namespace ID
in your account and is retained while you use Temporal Cloud.

You can retrieve your Account ID from the [Temporal Cloud](https://cloud.temporal.io) Web UI or from the command line.
Follow these steps.

**Web UI**

    Follow these steps to retrieve your Account ID:

    1. Log into Temporal Cloud.
    1. Select your account avatar at the top right of the page.
       A profile dropdown menu appears.
    1. Copy the Cloud Account ID from the menu.
![Temporal Cloud user profile dropdown menu. The dropdown includes a Cloud Account ID and copy button, with role, profile and logout options, and links to community resources.](https://docs.temporal.io/img/cloud/cloud-guide/cloud-account-id.png)

    In this example, the Account ID is `123de`.

**Temporal CLI**

    1. Use the Temporal Cloud extension to log into an account.

       ```
       temporal cloud login
       ```

       Complete the interactive login in the browser window that opens.

    1. Return to the command line and list your Namespaces.

       ```
       temporal cloud namespace list
       ```

       Each Namespace uses an Account ID suffix, the same as in the Temporal Cloud Web UI Namespaces list.

**tcld**

    1. Use the `tcld` utility to log into an account.

       ```
       tcld login
       ```

       The `tcld` output presents a URL with an activation code at the end. Take note of this code. The utility blocks until the login/activation process completes.

       ```
       Login via this url: https://login.tmprl.cloud/activate?user_code=KTGC-ZPWQ
       ```

       A Web page automatically opens for authentication in your default browser.

    1. Visit the browser.
       Ensure the user code shown by the CLI utility matches the code shown in the Web browser.
       Then, click Confirm in the browser to continue.
       After confirmation, Web feedback lets you know that the CLI "device" is now connected.

    1. Return to the command line.
       Issue the following command.

       ```
       tcld namespace list
       ```

       The CLI tool returns a short JSON packet with your namespace information.
       This is the same list found in the Temporal Cloud Web UI Namespaces list.
       Like the browser version, each Namespace uses an Account ID suffix.

       ```
       {
         "namespaces": [
           "your-namespace.123de",
           "another-namespace.123de"
         ],
         "nextPageToken": ""
       }
       ```

    Each Namespace automatically appends an Account ID suffix to its customer-supplied identifier.
    This five-character-or-longer string appears after the name, separated by a period.
    In this Namespace listing sample, the Account ID is 123de.

## What is a Cloud Namespace Id? 

A Cloud Namespace Id is a globally unique identifier for a [Namespace](/namespaces) in Temporal Cloud. A Namespace Id is
formed by concatenating the following:

1. A [Namespace Name](#temporal-cloud-namespace-name)
1. A period (.)
1. The [Account ID](#temporal-cloud-account-id) to which the Namespace belongs

For example, for the Account ID `123de` and Namespace Name `accounting-production`, the Namespace Id is
`accounting-production.123de`.

## What is a Cloud gRPC Endpoint? 

Temporal Clients communicate between application code and a Temporal Server by sending and receiving messages via the
gRPC protocol. gRPC is a Remote Procedure Call framework featuring low latency and high performance. gRPC provides
Temporal with an efficient, language-agnostic communication framework.

Every Temporal Namespace uses a gRPC endpoint for communication. When migrating to Temporal Cloud, you'll need to switch
the gRPC endpoint in your code from your current hosting, whether self-hosted or locally-hosted, to Temporal Cloud.

A gRPC endpoint appears on the detail page for each Cloud Namespace. Follow these steps to find it:

1. Log into your account on [cloud.temporal.io](https://cloud.temporal.io/namespaces).
2. Navigate to the Namespace list page from the left-side vertical navigation.
3. Tap or click on the Namespace Name to select and open the page for the Namespace whose endpoint you want to retrieve.
4. On the Namespace detail page, click on the "Connect" button in the top right corner of the page.
5. Click the copy icon next to the gRPC address to copy it to your clipboard.

See [How to access a Namespace in Temporal Cloud](/cloud/namespaces/#access-namespaces) for more information on
different gRPC endpoint types and how to access them.

## How to create a Namespace in Temporal Cloud 

> **⚠️ Caution:**
> GCP capacity restriction
>
> You can create GCP Namespaces in Mumbai (`asia-south1`), Jakarta (`asia-southeast2`), and Oregon (`us-west1`) without an account exception.
> New Namespaces in Iowa (`us-central1`), Northern Virginia (`us-east4`), and Frankfurt (`europe-west3`) require an approved account exception.
> See [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction) for details and how to request an exception through Temporal Dev Success.
>

> **ℹ️ Info:**
>
> The user who creates a [Namespace](/namespaces) is automatically granted
> [Namespace Admin](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) permission for that Namespace.
>
> To create a Namespace, a user must have the Developer, Account Owner, or Global Admin account-level
> [Role](/cloud/manage-access/roles-and-permissions#account-level-roles).
>

> **💡 Tip:**
>
> By default, each account starts with 10 Namespaces. This limit increases automatically when your existing Namespaces have scheduled or running Workflow Executions. For large-scale needs, open a [support ticket](/evaluate/cloud/support#support-ticket).
>

### Information needed to create a Namespace

To create a Namespace in Temporal Cloud, gather the following information:

- [Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name), region, and Cloud Provider
- Optional [Namespace description](#describe-a-namespace)
- [Retention Period](/temporal-service/temporal-server#retention-period) for the
  [Event History](/workflow-execution/event#event-history) of closed [Workflow Executions](/workflow-execution).
- [CA certificate](/cloud/certificates#certificate-requirements) for the Namespace, if you are using mTLS
  authentication.
- [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) to
  show decoded payloads to users in the Event History for Workflow Executions in the Namespace. For details, see
  [Securing your data](/production-deployment/data-encryption).
- [Permissions](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) for each user.

**Web UI**

### Create a Namespace using Temporal Cloud UI

1. Gather the information listed earlier in
   [Information needed to create a Namespace](#information-needed-to-create-a-namespace).
1. Go to the Temporal Cloud UI and log in.
1. On the left side of the window, click **Namespaces**.
1. On the **Namespaces** page, click **Create Namespace** in the upper-right portion of the window.
1. On the **Create Namespace** page in **Name**, enter the Namespace Name.
1. Optional: In **Description**, enter a short purpose note (at most 255 printable ASCII characters plus whitespace).
   This is free-text context, not a Tag. Do not include secrets, confidential data, or PII. Descriptions are not
   encrypted.
1. In **Cloud Provider**, select the cloud provider in which to host this Namespace.
1. In **Region**, select the region in which to host this Namespace.
1. In **Retention Period**, specify a value from 1 to 90 days. When choosing this value, consider your needs for Event
   History versus the cost of maintaining that Event History. Typically, a development Namespace has a short retention
   period and a production Namespace has a longer retention period. The retention period of a namespace can be changed
   in the Temporal Cloud UI under the namespace's Settings tab or by using the Temporal CLI.
1. Select your authentication method: [API keys](/cloud/api-keys) or [mTLS](/cloud/certificates).
1. If using mTLS authentication, paste the CA certificate for this Namespace.
1. Optional: In **Codec Server**, enter the HTTPS URL (including the port number) of your Codec Server endpoint. You may
   also enable "Pass the user access token with your endpoint" and "Include cross-origin credentials." For details, see
   [Hosting your Codec Server](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli).
1. Click **Create Namespace**.

**Temporal CLI**

See the [`temporal cloud namespace create`](/cli/command-reference/cloud/namespace#create) command reference for details.

**Cloud Operations API**

Use the
[`CreateNamespace` API](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces)
to create a Namespace. Pass an optional `description` on `spec`.

```
curl -sS -X POST "$BASE/cloud/namespaces" \
  -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \
  -d '{
    "spec": {
      "name": "payments-prod",
      "regions": ["aws-us-west-2"],
      "retentionDays": 30,
      "apiKeyAuth": { "enabled": true },
      "description": "Finance data integrations for Elevate across AWS, OCI, GCP"
    }
  }'
```

For authentication headers and API versioning, see [Cloud Ops API](/ops).

**tcld**

See the [`tcld` namespace create](/cloud/tcld/namespace/#create) command reference for details.

## What are some Namespace best practices? 

For guidance on how many Namespaces to create, how to split workloads across services or domains, and when to isolate
tenants or teams, see [Namespace best practices](/best-practices/managing-namespace).

This page focuses on Temporal Cloud Namespace mechanics such as naming rules, provisioning, authentication, tagging,
description, and accessing Namespace endpoints.

## How to access a Namespace in Temporal Cloud 

Temporal Cloud supports authentication to Namespaces using [API keys](/cloud/api-keys) or
[mTLS](/cloud/certificates). A Namespace uses one method unless both are enabled.

To change a Namespace from one method to the other, or to enable both methods on the same Namespace, contact
[Support](/evaluate/cloud/support#support-ticket).

> **⚠️ Caution:**
>
> Using both API key and mTLS authentication on the same Namespace is in
> [pre-release](/evaluate/product-release-stages). These cases are not supported:
>
> - **API-key Namespaces cannot enable both methods.** If the Namespace already uses API keys, Temporal Cloud rejects the change. To move that Namespace to mTLS, switch it from API keys to mTLS in one change. API key clients stop authenticating as soon as you save. Confirm that mTLS clients connect before you switch. There is no period where the Namespace accepts both methods.
> - **Both methods do not support High Availability.** A Namespace that uses both API key and mTLS cannot use [High Availability features](/cloud/high-availability). To enable High Availability, leave only one method enabled.
>

Connecting to your Namespace requires a specific endpoint that works for the given Namespace.
There are two types of gRPC endpoints for accessing a Namespace in Temporal Cloud: a Namespace endpoint and a regional endpoint.

- **Namespace endpoint** (`<namespace>.<account>.tmprl.cloud:7233`) — **Recommended**
    - This endpoint is unique to each Namespace. It always connects to the Namespace, no matter which region(s) the Namespace is using.
    - A Temporal Client that uses a Namespace endpoint doesn't have to be aware of which region the Namespace is in.
    - For Namespaces with [High Availability](/cloud/high-availability), the Namespace endpoint automatically directs traffic to the active region, so Workers and Clients don't need to change endpoints during a failover.
- Regional endpoint (`<region>.<cloud_provider>.api.temporal.io:7233`)
    - Temporal Cloud has only one regional endpoint for each cloud region. The same regional endpoint can access any Namespace that is active in that region (or that has a [replica](/cloud/high-availability) in that region).
    - A Temporal Client can use a regional endpoint to ensure connection to a Namespace always happens within that region. This can be useful in advanced [High Availability](/cloud/high-availability) setups where you want explicit control over which region handles requests.
    - For a Namespace that uses both API keys and mTLS, use the regional endpoint for API key authentication. API keys cannot authenticate through that Namespace’s Namespace endpoint.
    - When using mTLS to authenticate, the Temporal Client must set the `server_name` property to `<namespace endpoint value>` in its request to the value of the Namespace endpoint. This tells the client to expect a different SNI header during the TLS handshake, since the request to the regional endpoint is redirected to the specific Namespace.

> **⚠️ Caution:**
> Do not take dependencies on Temporal Cloud endpoint DNS resolution
>
> In general, the IP addresses that Temporal Cloud endpoints resolve to are subject to change without notice. Do not configure Workers, Temporal Clients, or firewalls against the specific IPs you observe for an endpoint at a point in time.
>
> Temporal Cloud guarantees the DNS resolution behavior of the Namespace Endpoint in two cases:
>
> 1. **Stable IPs enabled.** The Namespace Endpoint resolves to one of the [Stable IPs](/cloud/connectivity/ip-addresses) for the Namespace's active region.
> 2. **High Availability features with AWS PrivateLink.** The Namespace Endpoint resolves to a regional intermediary (`aws-<region>.region.tmprl.cloud`) that you can override in a Route 53 private hosted zone to point at your VPC Endpoint. This DNS override does not provide automatic failover with GCP Private Service Connect; see [Connectivity for High Availability](/cloud/high-availability/ha-connectivity#single-cloud-ha-on-gcp-private-service-connect) for GCP PSC options.
>

### Configuring a Temporal Client with API keys or mTLS

To use API keys to connect with the [Temporal CLI](/cli), [Client SDK](/develop), [tcld](/cloud/tcld),
  [Cloud Ops API](/ops), and [Terraform](/cloud/terraform-provider), see
  [Use API keys to authenticate](/cloud/api-keys#using-apikeys).

To use mTLS to connect with the [Temporal CLI](/cli) and [Client SDK](/develop), see
  [Configure Clients to use Client certificates](/cloud/certificates#configure-clients-to-use-client-certificates).

### Accessing the Temporal Web UI

For accessing the Temporal Web UI, use the HTTPS endpoint in the form:
`https://cloud.temporal.io/namespaces/<namespace>.<account>`. For example:
`https://cloud.temporal.io/namespaces/accounting-production.f45a2`.

### Access Namespaces with encryption and private connectivity

To ensure the security of your data, all traffic to and from your Namespace is encrypted with TLS 1.2 or higher (TLS 1.3 preferred).

For enhanced protection:
- Set up [private connectivity](/cloud/connectivity#private-network-connectivity-for-namespaces) to the Namespace.
- Set up your allow list for outgoing network requests from your Clients and Workers. You have two options:
  - Use Temporal Cloud's [stable IPs configuration](/cloud/connectivity/ip-addresses) to get non-changing IP addresses for your Namespace endpoint, which you can then allowlist in your firewall rules.
  - Allowlist the entire Cloud Provider IP address ranges for the region in which your Namespace is located:
    - [AWS IP address ranges](https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html)
    - [GCP IP address ranges](https://cloud.google.com/compute/docs/faq#find_ip_range)

## How to manage Namespaces in Temporal Cloud 

### Manage Namespaces in Temporal Cloud using Temporal Cloud UI

To list Namespaces:

- On the left side of the window, select **Namespaces**.

To edit a Namespace (including custom Search Attributes, certificates, certificate filters, Codec Server endpoint,
permissions, and users), find the Namespace and do either of the following:

- On the right end of the Namespace row, select the three vertical dots (⋮). Click **Edit**.
- Select the Namespace name. In the top-right portion of the page, select **Edit**.

On the **Edit** page, you can do the following:

- Add a [custom Search Attribute](/search-attribute#custom-search-attribute).
- [Manage CA certificates](/cloud/certificates).
- [Manage certificate filters](/cloud/certificates#manage-certificate-filters-using-temporal-cloud-ui).
- Set
  [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli)
  for all users on the Namespace. Each user on the Namespace has the option to
  [override this setting](/production-deployment/data-encryption#web-ui) in their browser.
- Manage [Namespace-level permissions](/cloud/manage-access/roles-and-permissions#namespace-level-permissions).
- Add users.

To add a user to a Namespace, scroll to the bottom of the page and select **Add User**.

After you make changes, select **Save** in the top-right or bottom-left portion of the page.

### Manage Namespaces in Temporal Cloud from the CLI

To list Namespaces and get information about them, use the following commands:

- [`temporal cloud namespace list`](/cli/command-reference/cloud/namespace#list) or [tcld namespace list](/cloud/tcld/namespace/#list)
- [`temporal cloud namespace get`](/cli/command-reference/cloud/namespace#get) or [tcld namespace get](/cloud/tcld/namespace/#get)

To manage certificates, use the [`temporal cloud namespace mtls cert-ca`](/cli/command-reference/cloud/namespace#mtls-cert-ca)
or [tcld namespace accepted-client-ca](/cloud/tcld/namespace/#accepted-client-ca) commands.
For more information, see [How to manage certificates in Temporal Cloud](/cloud/certificates).

To manage certificate filters, use the [`temporal cloud namespace mtls cert-filter`](/cli/command-reference/cloud/namespace#mtls-cert-filter)
or [tcld namespace certificate-filters](/cloud/tcld/namespace/#certificate-filters)
commands. For more information, see
[How to manage certificate filters in Temporal Cloud](/cloud/certificates#manage-certificate-filters).

## How to delete a Namespace in Temporal Cloud 

> **ℹ️ Info:**
>
> To delete a Namespace, a user must have Namespace Admin [permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) for that
> Namespace.
>

### Delete a Namespace using Temporal Cloud UI

1. Go to the Temporal Cloud UI and log in.
1. On the left side of the window, select **Namespaces**.
1. On the **Namespaces** page, select a Namespace Name.
1. On the Namespace page, select **Edit** in the upper-right portion of the window.
1. On the **Edit** Namespace page, select **Delete Namespace** in the upper-right portion of the window.
1. In the **Delete Namespace** dialog, type `DELETE` to confirm the deletion of that Namespace.
1. Select **Delete**.

After deleting a Temporal Cloud Namespace, the Temporal Service immediately removes the Namespace's Workflow Executions
and Task Queues. Make sure all Workflows have been completed, canceled, or terminated before removing a Namespace. The
Namespace removal is permanent.

Closed Workflow Histories remain in Temporal storage until the user-defined retention period expires. This period
reflects the policy in effect when the Workflow Execution was closed.

For further questions or concerns, contact [Support](/evaluate/cloud/support#support-ticket).

### Delete a Namespace from the CLI

See the [`temporal cloud namespace delete`](/cli/command-reference/cloud/namespace#delete) or
[tcld namespace delete](/cloud/tcld/namespace/#delete) command reference for details.

### Namespace deletion protection 

To prevent accidental Namespace deletion, Temporal Cloud provides a protection feature. When you enable Deletion
Protection for your production environment Namespace, you ensure that critical data won't be deleted unintentionally.

Follow these steps:

- Visit the [Namespaces page](https://cloud.temporal.io/namespaces) on Temporal Cloud.
- Open your Namespace details page.
- Select the Edit button.
- Scroll down to Security and click the disclosure button (downward-facing caret).
- Enable **Deletion Protection**

![Deletion Protection is enabled by toggling the switch](https://docs.temporal.io/img/cloud/namespace/deletion-protection.png)

To enable or disable this feature from the CLI, use the following command. Set the value to `true` to
enable or `false` to disable:

**Temporal CLI**

```
temporal cloud namespace lifecycle set \
    --namespace <namespace_id.account_id> \
    --enable-delete-protection <Boolean>
```

**tcld**

```
tcld namespace lifecycle set \
    --namespace <namespace_id.account_id> \
    --enable-delete-protection <Boolean>
```

## How to tag a Namespace in Temporal Cloud 

Tags are key-value metadata pairs that can be attached to namespaces in Temporal Cloud to help operators organize,
track, and manage namespaces more easily.

### Tag structure and limits

- Each namespace can have a maximum of 10 tags
- Each key must be unique for a given namespace (for example, a namespace cannot have both `team:foo` and `team:bar` tags)
- Keys and values must be 1-63 characters in length
- Allowed characters: lowercase letters (`a-z`), numbers (`0-9`), periods (`.`), underscores (`_`), hyphens (`-`), and
  at signs (`@`)
- Tags are not a secure storage mechanism and should not store PII or PHI
- Tags will not change the behavior of the tagged resource
- There is a soft limit of 1000 unique tag keys per account

### Permissions

- Only [**Account Admins** and **Account Owners**](/cloud/manage-access/roles-and-permissions#account-level-roles) can create and edit tags
- All users with access to a namespace can view its tags

### Temporal Cloud CLI

See the [`temporal cloud namespace tag`](/cli/command-reference/cloud/namespace#tag) or
[tcld namespace tags](/cloud/tcld/namespace/#tags) command reference for details.

### Terraform

See the
[Terraform provider](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/docs/resources/namespace_tags.md)
for details.

### Web UI

Tags can be viewed and managed through the Temporal Cloud web interface. When viewing a namespace, you'll see tags
displayed and can add, edit, or remove them if you have the appropriate permissions.

![Tags appear in namespace details](https://docs.temporal.io/img/cloud/tags/Namespace-DetailsWithTags.png)

![Tags appear on the list of namespaces](https://docs.temporal.io/img/cloud/tags/Namespaces-IndexWithTags.png)

![Where to add tags during namespace creation](https://docs.temporal.io/img/cloud/tags/CreateNamespace-AddNewTag.png)

![After adding a tag during namespace creation](https://docs.temporal.io/img/cloud/tags/CreateNamespace-AddedTag.png)

## How to add a Namespace description in Temporal Cloud 

A Namespace description is an optional free-text purpose note. Use it to answer "what is this Namespace for?" when the
name is not enough.

Description is not a Tag. Description is a human-readable summary. Tags are structured key-value metadata for filtering,
reporting, and automation. See [How to tag a Namespace](#tag-a-namespace).

> **⚠️ Caution:**
> Do not include sensitive or confidential information
>
> Namespace descriptions are visible to anyone with Namespace Read access through supported management tools. Descriptions are Cloud metadata, not Workflow payloads, and are not processed by a [Payload Codec](/payload-codec#payload-codec). Do not include secrets, credentials, confidential information, or personally identifiable information.
>

### Description structure and limits
- A description is optional and can be added, changed, or removed after Namespace creation.
- Descriptions can contain up to 255 characters total. Supported characters are printable ASCII and whitespace.
- The Temporal Cloud UI can render a limited Markdown subset. The stored value is still the 255-character printable ASCII
- A description does not affect Namespace runtime behavior.
- A Temporal Cloud Namespace description is separate from the open source `NamespaceInfo.Description` field. The temporal operator namespace commands do not manage the Cloud description.

### Permissions

- Account Owners, Account Admins, and Namespace Admins can change a description.
- Anyone with Namespace Read access can view it.

**Web UI**

When creating a Namespace, fill **Description** on the Create Namespace page.
For an existing Namespace, open its overview and select Add description or the description edit control under the Namespace full name. Save an empty value to remove the description.

**Cloud Operations API**

Create: include `spec.description` when calling
[`CreateNamespace`](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces).

Update: UpdateNamespace replaces the complete Namespace specification. Call [GetNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/%7Bnamespace%7D), preserve the returned spec and resourceVersion, change spec.description, and submit the complete specification to [UpdateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D). Omitting description from the updated specification clears the existing description.
Read: `GET /cloud/namespaces/{namespace}` returns `spec.description`.

Namespace descriptions require Cloud Operations API version v0.21.0 or later. See [Cloud Operations API](/ops).

**Temporal CLI**

See [`temporal cloud namespace create --description`](/cli/command-reference/cloud/namespace#create) and
[`temporal cloud namespace description`](/cli/command-reference/cloud/namespace#description).

```
temporal cloud namespace description set \
  --namespace payments-prod.account \
  --value "Elevate project migration from AWS to GCP"
```

Pass an empty `--value` to clear the description.

---

# Nexus

Source: https://docs.temporal.io/cloud/nexus

> Temporal Cloud adds global Nexus Registry, built-in access controls, audit logging, and multi-region connectivity on top of core Nexus.

Temporal Cloud builds on the [core Nexus experience](/nexus) with:

- **Global [Nexus Registry](/nexus/registry)** - Scoped to your entire Account across all Namespaces. Workers in any Namespace can host Nexus Services for others to use.
- **Built-in [access controls](/nexus/registry#configure-runtime-access-controls)** - Restrict which caller Namespaces can use a Nexus Endpoint at runtime.
- **[Audit logging](/cloud/audit-logs)** - Stream Nexus Registry actions (create, update, delete Endpoints) to your audit log integration.
- **Multi-region connectivity** - Nexus requests route across Namespaces within and across AWS and GCP using a global mTLS-secured Envoy mesh. Compatible with Namespaces that have [High Availability](/cloud/high-availability) as Endpoint targets.
- **[Terraform support](/cloud/terraform-provider#manage-temporal-cloud-nexus-endpoints-with-terraform)** - Manage Nexus Endpoints with the Temporal Cloud Terraform provider.

![Nexus Overview](https://docs.temporal.io/img/cloud/nexus/nexus-overview-short.png)

## Learn more

- [Evaluate Nexus](/evaluate/features/nexus) | [Keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082)
- [How Nexus works](/nexus) | [Deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4&t=934s)

---

# Latency and Availability - Temporal Nexus

Source: https://docs.temporal.io/cloud/nexus/latency-availability

> Learn about Nexus latency and availability

Nexus latency and availability in Temporal Cloud:

- **SLOs and the SLA** - Nexus operations (for example, `RespondWorkflowTaskCompleted`, `PollNexusTaskQueue`, `RespondNexusTaskCompleted`, and `RespondNexusTaskFailed`) have the same [latency SLOs](/cloud/slo#latency) as other Worker requests in both caller and handler Namespaces. For contractual coverage, see the [Temporal Cloud SLA](https://temporal.io/sla).
- **[Nexus metrics](/nexus/metrics)** - SDK and Cloud latency metrics for monitoring Nexus performance.
- **Cross-Namespace connectivity** - Traffic routes through a global mTLS-secured Envoy mesh. Same-region Namespaces have low latency; cross-region latency varies by provider. See [secure connectivity](/nexus/security#secure-connectivity).

---

# Limits - Temporal Nexus

Source: https://docs.temporal.io/cloud/nexus/limits

> Learn about Nexus limits including rate limits, the maximum number of Endpoints, and handler request timeouts.

Nexus limits are documented in [Temporal Cloud limits](/evaluate/cloud/limits):

- [Nexus rate limits](/evaluate/cloud/limits#nexus-rate-limits) - Nexus requests count toward the Namespace RPS limit.
- [Nexus Endpoint limits](/evaluate/cloud/limits#nexus-endpoints-limits) - 100 Endpoints per Account (default).
- [Nexus caller Namespace limits](/evaluate/cloud/limits#nexus-endpoint-access-policy-limits) - 1,000 caller Namespaces per Endpoint (default).
- [Per-Workflow Nexus Operation limits](/evaluate/cloud/limits#per-workflow-nexus-operation-limits) - 30 in-flight Operations per Workflow.
- [Nexus Operation request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout) - Less than 10 seconds for a handler to process a start or cancel request.
- [Nexus Operation duration limits](/evaluate/cloud/limits#nexus-operation-duration-limits) - 60-day maximum ScheduleToClose duration.
- [Per-Workflow callback limits](/evaluate/cloud/limits#per-workflow-callback-limits) - 2000 callbacks per Workflow. Governs how many Nexus callers can attach to a handler Workflow.

---

# Observability - Temporal Nexus

Source: https://docs.temporal.io/cloud/nexus/observability

> Learn about integrated observability in Temporal Nexus including cloud metrics and audit log streaming.

Nexus observability in Temporal Cloud:

- **[Nexus metrics](/nexus/metrics)** - [SDK metrics](/nexus/metrics#sdk-metrics) emitted by Workers and [Cloud metrics](/nexus/metrics#cloud-metrics) emitted by Temporal Cloud.
- **[Execution debugging](/nexus/execution-debugging)** - Bi-directional linking, pending Operations, pending callbacks, and tracing across Namespaces.
- **[Audit logging](/cloud/audit-logs)** - `CreateNexusEndpoint`, `UpdateNexusEndpoint`, and `DeleteNexusEndpoint` actions streamed to your audit log integration.

---

# Pricing for Temporal Nexus

Source: https://docs.temporal.io/cloud/nexus/pricing

> Learn about the pricing structure for using Nexus.

Nexus pricing:

- **One Action to start or cancel a Nexus Operation** in the caller Namespace.
  Underlying primitives (Workflows, Activities, Signals) and their retries created by the handler result in normal Actions.
- **No Action for handling or retrying the Nexus Operation itself**.
  However, billable actions initiated by the handler (such as Activities) are charged if they fail and retry.

See [Pricing](/cloud/pricing) for details.

---

# Security - Temporal Nexus

Source: https://docs.temporal.io/cloud/nexus/security

> Learn about security in Temporal Nexus

Nexus security in Temporal Cloud:

- **[Runtime access controls](/nexus/security#runtime-access-controls)** - Endpoint allowlists restrict which caller Namespaces can use an Endpoint. See [configuring access controls](/nexus/registry#configure-runtime-access-controls).
- **[Secure connectivity](/nexus/security#secure-connectivity)** - mTLS for all Nexus communication across cells and regions. Endpoints are only accessible within a Temporal Cloud Account.
- **[Payload encryption](/nexus/security#payload-encryption-data-converter)** - Same Data Converter as Workflows and Activities, with three approaches for cross-Namespace encryption.
- **[Registry roles and permissions](/nexus/registry#roles-and-permissions)** - Controls who can view, create, edit, and delete Endpoints.

---

# Receive Temporal Cloud notifications

Source: https://docs.temporal.io/cloud/notifications

> Receive updates about Temporal Cloud including when certificates will expire, billing updates, and when a failover has completed.

## Get notified about Temporal Cloud status 

In the event of an incident, Temporal updates the [Temporal Cloud status page](https://status.temporal.io/) with important updates.
Users can subscribe to updates in their preferred mode, such as email, Slack, or SMS, by visiting this page.

## Get notified about administrative events 

Temporal Cloud sends emails to notify users of important administrative events.

| Reason for email | Who receives email |
|------------------ | -------------------|
| Certificate Expiring in 15 days | Global Administrator, Namespace Administrator, Account Owner |
| Certificate Expiring in 10 days | Global Administrator, Namespace Administrator, Account Owner |
| Certificate Expiring in 5 days  | Global Administrator, Namespace Administrator, Account Owner |
| API Key Expiring in 30 days     | Global Administrator, Account Owner, individual user (if API Key has an owner) |
| API Key Expiring in 20 days     | Global Administrator, Account Owner, individual user (if API Key has an owner) |
| API Key Expiring in 10 days     | Global Administrator, Account Owner, individual user (if API Key has an owner) |
| Sign up credit expiring in 30 days | Account Owner, Finance Administrator |
| Sign up credit expiring in 14 days | Account Owner, Finance Administrator |
| Sign up credit expiring in  7 days | Account Owner, Finance Administrator |
| Sign up credit expiring in  1 days | Account Owner, Finance Administrator |
| Sign up credit is 50% consumed     | Account Owner, Finance Administrator |
| Sign up credit is 90% consumed     | Account Owner, Finance Administrator |
| Account plan type changed          | Global Administrator, Account Owner, Finance Administrator |
| Namespace Failover Completed/Failed | Global Administrator, Namespace Administrator, Account Owner |

To ensure that you receive email notifications, configure your junk-email filters to permit email from
`noreply@temporal.io`.

To provide feedback on notifications or request changes, [create a support ticket](/evaluate/cloud/support#support-ticket).

---

# Temporal Cloud pricing

Source: https://docs.temporal.io/cloud/pricing

> Temporal Cloud offers flexible, predictable pricing for Workflows, Activities, Workers, and Storage. Pay for what you use with volume discounts and credit savings.

Temporal Cloud is a consumption-based service.
You pay only for what you use.
Our pricing reflects your use of [_Actions_](#action), [_Storage_](#storage), and [_Support_](/evaluate/cloud/support#support).
It is flexible, transparent, and predictable, so you know your costs.

This page describes the elements of Temporal Cloud pricing.
It gives you the information you need to understand and estimate costs for your implementation.
For more exact estimates, please reach out to [our team](https://pages.temporal.io/ask-an-expert).

Billing and cost information is available directly in the Temporal Cloud UI.
For more information, visit the [Billing](/cloud/billing) page.

## Temporal Cloud pricing model 

This section explains the basis of the Temporal Cloud pricing model and how it works.
Your total invoice each calendar month is the combination of Temporal Cloud consumption ([Actions](#action) and [Storage](#storage)), and a [Temporal Cloud Support Plan](#base_plans).

### Temporal Cloud Support plans 

**How plans work**

Each Temporal Cloud account includes a plan with Support. Higher tiers include Actions, Active Storage, Retained Storage and add-on features for the platform.
Base allocations help you get started with the Temporal platform, so you can better estimate costs.

- Temporal Cloud Plans are charged monthly.
- Action and Storage allocations are reset each calendar month.

Temporal offers four support plans: Developer, Business, Enterprise, Mission Critical.
Prices are outlined in the following table:

|                   | Developer                                                                      | Business                                                                                                                          | Enterprise                                                                                                                                                 | Mission Critical                                                                                                                                           |
| ----------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Support Targeting | Basic use                                                                      | Production deployments<br />that scale                                                                                            | Enterprise deployments<br/>w/ stringent uptime demands                                                                                                     | Mission-critical applications<br />w/ the highest support needs                                                                                            |
| Support Features  | Access to Support                                                              | - P0 Response Times                                                                                                               | - P0: \<30 Min, 24/7<br />- Private Slack                                                                                                                  | - P0: \<15 Min, 24/7<br />- Private Slack<br />- Dedicated Platform Architect                                                                              |
| Product Features  | - All Core Features<br />- SAML included<br />- SCIM (Add-on)                  | - Commit discounts<br />- SAML included<br />- SCIM (Add-on)<br />- 2.5M Actions<br />- 2.5 GB Active Storage<br />- 100 GB Retained Storage | - Commit discounts<br />- SAML included<br />- SCIM included<br />- 10M Actions<br />- 10 GB Active Storage<br />- 400 GB Retained Storage      | - Commit discounts<br />- SAML included<br />- SCIM included<br />- 10M Actions<br />- 10 GB Active Storage<br />- 400 GB Retained Storage                 |
| Plan Pricing      | No base fee<br />- 10% of Usage Spend                                          | Greater of<br />- &#36;500/mo or<br />- 10% of Usage Spend                                                                        | Priced annually:<br /> [contact Sales](mailto:sales@temporal.io) for details                                                                               | Priced annually:<br /> [contact Sales](mailto:sales@temporal.io) for details                                                                               |
| Usage Pricing     | [Pay-As-You-Go](#payg) Pricing                                                 | Choose from <br /> [Pay-As-You-Go](#payg) or [Commitment Pricing](#commitment-pricing)                                            | Choose from <br /> [Pay-As-You-Go](#payg) or [Commitment Pricing](#commitment-pricing)                                                                     | Choose from <br /> [Pay-As-You-Go](#payg) or [Commitment Pricing](#commitment-pricing)                                                                     |

Please note, partial months are prorated to the day.
Find a complete description of Support offerings and response times in the [Support](/evaluate/cloud/support) documentation.

> **📝 Note:**
> Included storage
>
> A gigabyte-hour (GBh) is 1 GB stored for one hour. Each GB of Active or Retained Storage included with a plan covers 744 GBh of usage per calendar month. For example, storing 1 GB for 24 hours uses 24 GBh.
>

### Actions 

**What are Temporal Actions?**

Actions are the primary unit of consumption-based pricing for Temporal Cloud.
They track billable operations within the Temporal Cloud Service, such as starting Workflows, recording a Heartbeat or sending messages.
**Specific Billable Actions are discussed on the [Actions](/evaluate/cloud/actions) page.**

[Reach out](https://pages.temporal.io/contact-us) to our team for more information or to help size your number of Actions.

### Storage 

**How Workflow Storage works**

A Workflow's execution might exist for a few seconds, a day, month, or even forever.
The Temporal Service stores the Workflow Execution's [Event History](/workflow-execution/event#event-history).
Under this framework, a Workflow Execution has only two states, open (Active Storage) or closed (Retained Storage).

- _Active Storage_ measures the amount of storage used by active Workflows.

- When the execution of a Workflow finishes, Temporal Cloud stores Event History for a defined [Retention Period](/temporal-service/temporal-server#retention-period), which is set by the user per Namespace.
  This is _Retained Storage_.
  Typical uses of Retained Storage include compliance, debugging, workload refresh, and business analytics.
  When closed Workflow Histories need to be retained for more than the 90-day maximum period on Temporal Cloud, we recommend using our [**Export**](/cloud/export) feature.

Storage usage is measured in gigabyte-hours (GBh).

### Pricing options 

**How to Pay for Temporal Cloud**

After you exceed your Actions and Storage allocations in your base tier, Temporal Cloud offers two payment options: Pay-As-You-Go and Commitments.
Both models meter and bill for three primary components: [Actions](#action), [Storage](#storage), and [your Temporal Cloud Plan](/evaluate/cloud/support#support).

- With Pay-As-You-Go, you are invoiced each calendar month based on your consumption.
  On the Business plan and above, Pay-As-You-Go pricing automatically applies volume prices as your Actions scale.
- With Commitments, you pre-purchase your Temporal Cloud spend with Temporal Credits.
  Temporal Credits pay for your Temporal Cloud consumption, including Temporal Cloud Plan charges.

## Pay-As-You-Go 

**How does Pay-As-You-Go pricing work?**

Pay-As-You-Go pricing is based on consumption.
This section explains how you're billed each calendar month and gives examples.

### Action pricing 

Actions pricing starts at &#36;50 per million Actions (&#36;0.00005 per Action).
The Developer plan is billed at &#36;50 per million Actions at every volume.
On the Business, Enterprise, and Mission Critical plans, you gain progressive volume discounts as you scale.
Greater discounts are available through commitments.
Discounts are based on your account's total usage, metered and billed for each calendar month:

| Actions (Business, Enterprise, and Mission Critical) | Price per Million Actions                                                                 |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| First 5M                                             | &#36;50                                                                                   |
| Next 5M, up to 10M                                   | &#36;45                                                                                   |
| Next 10M, up to 20M                                  | &#36;40                                                                                   |
| Next 30M, up to 50M                                  | &#36;35                                                                                   |
| Next 50M, up to 100M                                 | &#36;30                                                                                   |
| Next 100M, up to 200M                                | &#36;25                                                                                   |
| Over 200M                                            | Contact [Sales](mailto:sales@temporal.io) for info<br /> _More discounts, helpful humans_ |

**Example**

If you are on the Business plan and consume 11.25M Actions in excess of your plan's allocation in one calendar month, your bill for Actions will be:

```
5M Actions ⨉ $50 Per Million Actions = $250
5M Actions ⨉ $45 Per Million Actions = $225
1.25M Actions ⨉ $40 Per Million Actions = $50
Actions
$250 (First Tier) + $225 (Second Tier) + $50 (Third Tier) = $525
```

### Storage pricing 

You are billed for Active and Retained Storage as follows:
| **Storage** | **Price per GBh (USD)** |
| ----------- | ----------------------- |
| Retained | &#36;0.00105 |
| Active | &#36;0.042 |

> **💡 Tip:**
> Storage costs are also affected by Temporal System Workflows that back features such as:
>
> - [Schedules](/schedule): Each Scheduled Workflow contributes to storage usage. Supplied inputs, outputs, and failures all account for the storage usage incurred from Scheduled Workflows.
> - [Batch jobs](/cli/command-reference/batch): Batch Workflow executions also consume storage.
>
> These Workflow executions contribute to overall active and retained storage consumption.
>

**Example**

If you have 720 GBh of Active Storage and 3,600 GBh of Retained Storage in excess of your Base Tier allocations in one calendar month, your bill will be:

```
720 GBh Active Storage ⨉ $0.042 per GBh = $30.24
3,600 GBh Retained Storage ⨉ $0.00105 per GBh = $3.78
Total Storage Bill: $30.24 Active Storage + $3.78 Retained Storage = $34.02
```

## Temporal Cloud plan pricing

Your Temporal Cloud Plan pricing is the greater of the minimum monthly price or a percent (%) of your consumption spend:

- The Developer tier is priced at the greater of &#36;0/month or 10% of your Temporal Cloud consumption.
- The Business tier is priced at the greater of &#36;500/month or 10% of Temporal Cloud consumption.
- The Enterprise and Mission Critical Support plans must be paid annually. Contact [Sales](mailto:sales@temporal.io) to discuss your needs.

Your Temporal Cloud consumption combines the costs of Actions and Storage.

**Example**

If you are signed up for Business, with &#36;6,000 of monthly spend, your bill will be:

```
Greater of $500 or 10% ⨉ $6,000 = $600, so $600.
```

## Commitment pricing 

**Commitments with Temporal Credits**

Temporal Cloud offers the option to commit to a minimum spend over a given timeframe.
In exchange for this commitment you receive additional discounts.
Key discount levers include:

- Account Action volume over 200M Actions
- Duration of your commitment (1, 2, or 3 years)

Meet your commitments with any Temporal Cloud spend, including Actions, Storage and your Temporal Cloud Plan.
After making a commitment, Temporal locks in your Actions price based on your expected volume and discounts your Active Storage costs.
This price is used to bill your Actions and Active Storage across your account for the timeframe specified in your commitment.

Commitments must be paid for with Temporal Credits.
Temporal Credits are used to pay your Temporal Cloud consumption, including Temporal Cloud Plan charges.
A Temporal Credit is equivalent to &#36;1 USD.
For example, a credit purchase of &#36;20,000 results in 20,000 Temporal Credits.
A minimum credit purchase equivalent to the first year of your commitment is required.
For multi-year deals please contact [Sales](mailto:sales@temporal.io) for the most accurate pricing.

### Commitment pricing Q&A

**How do multi-year commitments work?**

Our sales team works with you to match annual credit purchases in line with your expected spend.
This aligns your payments to annual terms rather than one up-front expense.

**What happens if I exhaust my commitment-based Temporal Credits before the end of my term?**

You continue to receive the negotiated discounted prices for the remainder of your term.
You'll be invoiced for another credit purchase based on your most recent calendar month's spend.
This amount is multiplied by the months remaining in the annual portion of your term.
If your previous month spend was &#36;5,000, and you're 10 months through your annual term, you'll be invoiced for 10,000 Temporal Credits to cover the remaining two months.

**What happens if I have unused Temporal Credits at the end of my term?**

Commitments can be difficult to estimate.
Temporal Cloud offers two ways to roll-over unused credits:

- When you renew a commitment for the same or larger amount, Temporal Cloud rolls over any unused credits into your new commitment.
- Should you need to downsize your commitment, Temporal Cloud rolls over up to 10% of your initial credit purchase amount into the new commitment.

**How do I make a commitment and purchase Temporal Credits?**

Contact our team at [sales@temporal.io](mailto:sales@temporal.io) or reach out to your dedicated account manager.
You can also purchase Temporal Cloud commitment credits through [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) and [GCP Marketplace](https://console.cloud.google.com/marketplace/product/temporal-public/temporal-cloud-pay-as-you-go).

### Credit balance

Your Credit Balance is adjusted each calendar month based on your Credit Usage.

**Example**

An account has purchased credits with an annual spend commitment of &#36;72,000.
In the first calendar month, the account is billed &#36;500 for a Business plan, &#36;5,000 for Actions, &#36;250 for Active Storage and &#36;50 for Retained Storage, a total of &#36;5800.
Their invoice would state:

```
Beginning credit balance of 72,000 - 5800 credits used = 66,200, Temporal Credits remaining.
```

If you have additional questions about credits and volume-based pricing, please contact [sales](mailto:sales@temporal.io) or, if you're already a Temporal Cloud customer, reach out to your dedicated account executive.

## Other pricing 

Temporal Cloud has additional pricing for other elements of the platform.

### Capacity Modes 

**What are Capacity Modes?** 

Temporal offers On-Demand and Provisioned Capacity modes. 
On-Demand capacity is automatically adjusted based on past usage. 
Provisioned Capacity modes lets you define the capacity that is needed by your Workflow and is useful to handle traffic outside of the standard on-demand limits.
See details on how capacity is set and the associated limits at [Capacity Modes](/cloud/capacity-modes).

**How does pricing for Capacity Modes work?**

The number of Actions accrued can be impacted by your capacity mode. 
For On-Demand Capacity Mode, Actions are accrued as usual. 
For Provisioned Capacity there is a minimum number of Actions that must be used per calendar hour for each [Temporal Resource Unit (TRU)](/cloud/capacity-modes) that is provisioned.
If Action volume in a calendar hour does not exceed the minimum allocation, Actions of the subtype `ns_capacity:tru` will be recorded to the required volume. 
The minimum requirement for the first TRU is 0 Actions as it aligns with the default rate limits on Temporal. 
For each additional TRU, there is a minimum hourly requirement of 360,000 Actions Per Hour. 
360,000 Actions per hour represents a 20% utilization of requested resources. 
This can be calculated as 500 APS Per TRU * 3600 Seconds (1 Hour) * 20%= 360,000 Actions. 

For example: 

If you have a namespace that requests 4 TRUs (that is, 2,000 APS) for 4 hours and have usage as follows:
* Hour 1: 2,000,000 Actions
* Hour 2: 5,000,000 Actions
* Hour 3: 4,000,000 Actions
* Hour 4: 500,000 Actions

Then the cost of Provisioned Capacity would be calculated as:

4 TRUs Requested - 1 Default TRU included = 3 TRUs with a minimum usage requirement
3 TRUs * 360,000 Actions per Hour= 1,080,000 minimum Actions per Hour
* Hour 1: 2,000,000 Actions > 1,080,000 minimum Actions = No additional Actions accrued
* Hour 2: 5,000,000 Actions > 1,080,000 minimum Actions = No additional Actions accrued
* Hour 3: 4,000,000 Actions > 1,080,000 minimum Actions = No additional Actions accrued
* Hour 4: 500,000 Actions < 1,080,000 minimum Actions so 1,080,000-500,000 Actions = 580,000 additional actions will be added to the hour

Total for 4 hours: 2,000,000+5,000,000+4,000,000+1,080,000=12,080,000 Actions

If TRUs are changed multiple times within an hour, the highest value within that hour will be used to calculate the minimum actions required.

Temporal’s approach to pricing Provisioned Capacity aligns with our goal to only charge for what you use. 
The minimum hourly allocation will have no impact on your price if you utilize the requested capacity above your default by 20% or more, and will only incur an additional charge if the requested capacity is not used. 
To avoid being charged for provisioned capacity it is advised to "return" capacity to the pool and switch back to the On-demand mode when usage is stable. 

### High Availability feature pricing 

**How does the pricing for High Availability (HA) features work?**

For workloads with stringent high availability requirements, Temporal Cloud provides same-region, multi-region, and multi-cloud replicas, which add a failover capability.
Enabling HA features for Namespace automatically replicates Workflow Execution data and metadata to a replica in the same region or in a different region.
This lets a replica take over automatically when incidents or outages occur.

The pricing for High Availability features aligns with the volume of your workloads.
Actions and Storage in your Namespace contribute to your Actions and Storage consumption.
To estimate costs for this deployment model, apply a 2x multiplier to the Actions and Storage in the Namespace you are replicating and include this scaling in your account's consumption.
For pre-existing Namespaces that add a replica, the 2x multiplier only applies to Actions and Storage after the replica is active.
Storage used and Actions completed before the replica's creation, or while it was being created but not yet in the 'active' status, do not incur the additional charges.

> **💡 Tip:**
> Future Pricing Update
>
> In the future, for Namespaces that incur high volumes of cross-cloud data transfer, Temporal will charge extra to cover the cost of egress charged by the cloud providers.
> All Namespaces will receive a generous amount of network transfer for free, so the vast majority of Namespaces will have no charge.
>

When upgrading an existing Namespace, some points to consider:

- Temporal won't charge for historical Actions completed prior to upgrading to a Namespace with High Availability features.
  Only ongoing (in-flight) and new Workflow Executions will generate consumption.
- Temporal charges for all Actions of existing (ongoing) and new Workflows from the point of adding a replica.
- Temporal charges for Replicated Storage of retained (historical), running (ongoing), and new Workflow Executions from the point of adding a new replica.

### Fairness pricing 

**How does pricing for Fairness work?**

When [Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) is enabled on a Namespace, an additional `0.1` Action is charged per Action in that Namespace during each hour the feature is on, regardless of whether individual Workflows or Activities use fairness keys.

The examples below use a Namespace, `your-namespace`, that normally generates 10,000 Actions per hour.

**Fairness enabled, but no fairness keys in use**

If Fairness is enabled but no Workflows or Activities send `fairness_key` metadata, all Tasks continue to route as before. Fairness Actions are still applied: billing is determined by whether Fairness is enabled on the Namespace.

| Hour          | Actions in Namespace | Fairness enabled? | Fairness Actions | Total billed Actions |
|---------------|----------------------|-------------------|--------------------|----------------------|
| 2:00–3:00 PM  | 10,000               | Yes               | +1,000             | 11,000               |

**Mix of Workflows with and without fairness keys**

Suppose `your-namespace` has two Task Queues:
- `payment-queue`, where Workflows pass `fairness_key = "customer-id"`
- `notification-queue`, where Workflows have no fairness keys and behave exactly as before

If each queue generates 5,000 Actions in a given hour (10,000 total), Fairness Actions apply to the full 10,000, not only to the 5,000 from `payment-queue`.

| Hour          | Actions from payment-queue | Actions from notification-queue | Total Namespace Actions | Fairness | Total billed Actions |
|---------------|----------------------------|---------------------------------|-------------------------|-----------------|----------------------|
| 2:00–3:00 PM  | 5,000                      | 5,000                           | 10,000                  | +1,000          | 11,000               |

### SCIM and SSO via SAML pricing 

**What costs are associated with SSO/SAML use?**

Single sign-on (SSO) integration using SAML is included for all customers on the Developer, Business, Enterprise, and Mission Critical Plans.

**What costs are associated with SCIM?**

SCIM (System for Cross-domain Identity Management) is included in the Enterprise or Mission Critical support plans. SCIM is available as an add-on for the Developer or Business plan (+$500/month).
Please note that you must configure SSO via SAML to use SCIM.

### Use case cost estimates 

Temporal Cloud uses a consumption-based pricing model based primarily on [Actions](#action) and [Storage](#storage).
Each workload is different.
You can estimate the cost of a specific Workflow by running it at a low volume.
Use the resulting Storage and compute measurements to project your production scale cost.

The examples below provide general estimates based on workload size.
You can also use our calculator on the pricing page to build your estimate.
Our team is always happy to [help you estimate costs](https://pages.temporal.io/contact-us) for your specific workloads and requirements.

| Workload size | Cost (monthly) | Characteristics                            | Actions                                            | Typical use cases                                                                                             |
| ------------- | -------------- | ------------------------------------------ | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| Small         | < &#36;50.00   | Modest / transient throughput              | < 1M / month  <br /> _(< 0.38 actions per second)_ | General automation <br /> Human dependent processes <br /> Data pipelines <br /> Nightly batch processes      |
| Medium        | < &#36;2K      | Steady or burst throughput                 | < 40M / month <br /> _(< 15 actions per second)_   | Transaction & order systems <br /> Infrastructure automation <br /> Payment Processing <br /> Batch processes |
| Large         | < &#36;15K     | Sustained throughput or multiple use cases | < 400M / month <br /> _(< 150 actions per second)_ | Data processing / sync <br /> Retail order system <br /> KYC & fraud detection                                |
| Web Scale     | &#36;20K+      | "Web scale" and / or numerous use cases    | 1B+ / month <br /> _(400+ actions per second)_     | Social media application <br /> SaaS application service                                                      |

## Billing questions FAQs 

**What payment methods does Temporal accept?**

You can pay with a credit card, ACH, or wire transfer.
To pay with an AWS or Google Cloud account, sign up in the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) or [GCP Marketplace](https://console.cloud.google.com/marketplace/product/temporal-public/temporal-cloud-pay-as-you-go).

**How often will I be billed?**

Temporal Cloud issues invoices for the previous month’s usage and costs.
Invoices are issued on the 3rd of the month for the previous month.
For example, invoices for May will be issued at midnight UTC on June 3rd.

**Where can I view my usage and billing information?**

Account Owners and Finance Admins can view their detailed billing data at any time.
Visit the [Billing dashboards](/cloud/billing) in Temporal Cloud.

**How do I purchase Temporal Cloud credits?**

You can purchase Temporal Cloud credits by contacting our team at [sales@temporal.io](mailto:sales@temporal.io).

**What's the minimum cost to run Temporal Cloud?**

The Developer plan starts at &#36;0/month.
This means you only pay for what you use.
For example, running a test Workflow with 10,000 Actions costs 10,000 ⨉ &#36;50 per million Actions = &#36;0.50, plus 10% for support, for a total of &#36;0.55.
For tiers with included Actions and Storage, consumption in excess of your plan's allocations is billed on a consumption basis.

**Can I purchase Temporal Cloud through AWS Marketplace or Google Cloud Marketplace?**

Yes.
Both marketplaces offer the same two purchase paths:

- **Pay-As-You-Go**: subscribe directly in [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) or [GCP Marketplace](https://console.cloud.google.com/marketplace/product/temporal-public/temporal-cloud-pay-as-you-go).
- **Credits**: available via private offer. Contact our team at [sales@temporal.io](mailto:sales@temporal.io).

Billing goes through your AWS or Google Cloud account.

**How do I see how many Temporal Cloud credits are remaining?**

To view remaining Temporal Cloud credits, Account Owners and Finance Admins can log in to Temporal Cloud and go to Settings > Billing.
You need appropriate administrative permissions to access this section.

**What happens if I exceed my available credits under a promotion such as the startup program?**

Customers with free credits from the startup program or from a promotion are invoiced once their credit balance is exhausted at the end of that month.

**Do promotional credits expire?**

Credits received through the startup program or an offer have an expiry date.
This date is stated as part of the sign-up process.

**How do I update my payment information?**

Account Owners and Finance Admins can update payment information at any time on the Temporal Cloud [Billing](https://cloud.temporal.io/billing) page under the Plan tab.
You need appropriate administrative permissions to access this section.
Select the "Manage Payment Method" button.

See [this overview](/cloud/billing) for more details.

**What happens if my payment fails?**

Temporal will periodically send you email reminders to complete the payment.

**How do I view my invoices and billing history?**

Invoices are emailed to Account Owners or the designated billing contacts.
Account Owners and Finance Admins can view their [detailed billing information](https://cloud.temporal.io/billing) at any time.
See our [billing](/cloud/billing) page for details.
You need appropriate administrative permissions to access this section.
Alternatively, to view invoices and billing history, contact Temporal Finance at [ar@temporal.io](mailto: ar@temporal.io).

**Does Temporal charge sales tax/VAT?**

We charge applicable sales tax in US jurisdictions as required.

**How do I cancel my account?**

Account Owners can delete their account and cancel their subscription in the Plans tab in the billing center.
See the [billing and cost](/cloud/billing) page for details on how to access the billing center.

**Will I lose access immediately if I cancel my account?**

Customers lose access to Temporal Cloud once Temporal completes the off-boarding process.
Billing is independent of this process.

**Can I reactivate my account after cancellation?**

No.
When your account is canceled, your account data is deleted and cannot be restored.
To return to Temporal Cloud, you must sign up again.
We will assign you a new Temporal account and consider you as a new customer.

---

# Projects

Source: https://docs.temporal.io/cloud/projects

> Organize Temporal Cloud resources including Namespaces and Nexus Endpoints around your organizational structure

> **Pre-release**
> Features in pre-release are experimental. While Project-specific APIs and functionality may evolve, enabling Projects is designed to be backward compatible with existing resources, permissions, and workflows through the Default Project.

Projects are an organizational layer within a Temporal Cloud Account that groups related resources into a single operational boundary. With Projects, you can organize [Namespaces](/cloud/namespaces), [Nexus endpoints](/nexus/endpoints), and [Connectivity Rules](/cloud/connectivity#connectivity-rules) around how your organization operates: by team, environment, service, or business unit.

Projects make it easier to delegate ownership, manage access at scale, and organize resources without creating additional Cloud Accounts. As your organization grows, Projects help align your Temporal Cloud resources with your organizational structure while reducing operational overhead.

Projects are for organization and authorization. They don't change how Workflows execute inside a Namespace.
They also don't restrict which Namespaces can call a Nexus Endpoint. See [Use Nexus across Projects](#use-nexus-across-projects).

![Diagram of the Temporal Cloud resource hierarchy. An Account is the global container for all resources. Each Account contains one or more Projects, which organize and delegate access across teams, apps, or environments; the diagram shows a default Project created by Temporal alongside custom Projects named Agents and Evals, plus a control to create additional Projects. Each Project contains Namespaces, which are isolated, secure runtime environments for Workflow and other executions; the default Project contains NS-prod and NS-staging, Agents contains customer-support and sales-assistant, and Evals contains benchmarks and regression.](https://docs.temporal.io/img/cloud/projects/resource-hierarchy.png)

## Enable Projects on your Account

While Projects are in pre-release, they are not enabled by default. [Contact Support](/evaluate/cloud/support#support-ticket) to request enabling Projects for your account.

Once Projects are enabled, existing Accounts receive a Default Project, and new Accounts are created with one automatically. For existing Accounts, all existing resources are automatically placed in the Default Project. Existing role assignments and effective permissions remain unchanged. For new Accounts, the Default Project is created automatically. New resources are automatically placed in the Default Project unless you specify otherwise.

## Default Project and Project ID

Every Temporal Cloud Account includes a Default Project. The Default Project is not a special type of Project. It behaves like any other Project for managing resources. The main difference between the Default Project and any Projects you create is that the Default Project is automatically created for you as a starting point and provides backward-compatible behavior for existing resources and operations that do not specify a Project. Otherwise, it supports the same resource-management capabilities as other Projects.

If you don't need the organizational boundaries offered by Projects, continue to use the Default Project for all your resources. You can always create a new Project later if you need it.

Every Project has a unique, immutable `Project ID` and an editable `display name`. The Project ID is used by the Cloud Ops API, CLI, and Terraform to identify the Project when creating and managing Project-scoped resources.

You can move an existing Namespace to another Project. See [Move a Namespace to another Project](#move-a-namespace). You can also create a new Project and place a new Namespace in that Project.

For limitations on the number of Projects per Account and number of resources per Project, see [Project-level limitations](/evaluate/cloud/limits#project-level).

## Create a new Project

Create a Project when a set of resources has a distinct owner or access model, separate from the Default Project. A good Project represents a stable ownership boundary, not just a folder.

Start with one Project for a team or application with clear ownership. Add more only when needed.

**Cloud UI**

To create a Project using the Temporal Cloud UI:

1. Select **Projects** at the top of the left navigation bar. 
2. Click the **Create Project** button to create a new Project.
3. On the **Create Project** page that appears, give the Project a name (limited to 64 characters) and an optional description (limited to 255 characters).
4. Click **Create**.

**Cloud Operations API**

Use the
[`CreateProject` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects)
to create a Project. The structure of the `CreateProjectRequest` is defined by a JSON spec, as follows:

```
"spec": {
  "description": "",
  "displayName": "",
  "lifecycle": {
    "enableDeleteProtection": true
  }
}
```

In this call, `description` is a string description of the Project, `displayName` is the Project name to be displayed in the UI, and `enableDeleteProtection` is a Boolean that indicates whether the Project can be deleted.

After creating the Project, add new Namespaces and other supported resources within it. A Namespace can be moved to another Project. See [Move a Namespace to another Project](#move-a-namespace). Other resources cannot yet be moved from one Project to another.

## Add Resources to a Project 

After creating a Project, you can create Project-scoped resources such as Namespaces, Nexus Endpoints, or Connectivity Rules within it.

**Cloud UI**

In Temporal Cloud UI:
1. Open the Project. 
2. Select **Namespaces** (or **Nexus Endpoints**) in the left navigation. 
3. Click **Create Namespace** (or **Nexus Endpoints**) to add to the project. 

**Connectivity rules** cannot currently be created via the UI. Connectivity rules and the Namespace they attach to must reside in the same Project.

**Cloud Operations API**

The process for creating a [Namespace](/cloud/namespaces#create-a-namespace), [Nexus Endpoint](/nexus/registry#view-and-manage-nexus-endpoints), or [Connectivity Rule](/cloud/connectivity#creating-a-connectivity-rule) with the Cloud Operations API is the same as the standard process for creating those resources, with the addition of the `projectId` for the Project in which you're creating the resource.

Use the
[`CreateNamespace` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces)
to create a new Namespace within a Project. For example:

```
curl -sS -X POST "$BASE/cloud/namespaces" \
  -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \
  -d '{
    "projectId": "'"$PROJECT_ID"'",
    "spec": {
      "name": "payments-prod",
      "regions": ["aws-us-west-2"],
      "retentionDays": 7,
      "apiKeyAuth": { "enabled": true }
    }
  }'
```

Use the
[`CreateCreateNexusEndpoint` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints)
to create a new Nexus Endpoint within a Project. For example:

```
curl -sS -X POST "$BASE/cloud/nexus/endpoints" \
  -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \
  -d '{
    "projectId": "'"$PROJECT_ID"'",
    "spec": {
      "name": "order-fulfillment",
      "targetSpec": {
        "workerTargetSpec": {
          "namespaceId": "payments-prod.a1b2c3",
          "taskQueue": "nexus-handler"
        }
      },
      "policySpecs": [
        {
          "allowedCloudNamespacePolicySpec": {
            "namespaceId": "checkout-prod.a1b2c3"
          }
        }
      ]
    }
  }'
```

In this example, `payments-prod.a1b2c3` is the target Namespace (handler) and `checkout-prod.a1b2c3` is an allowed caller Namespace.
Those Namespaces can be in different Projects from each other and from the Endpoint. Runtime access is the caller allowlist, not Project membership.
See [Use Nexus across Projects](#use-nexus-across-projects).

Use the
[`CreateConnectivityRule` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/POST/cloud/connectivity-rules)
to create a new Connectivity Rule within a Project. For example, for a private Connectivity Rule on AWS:

```
curl -sS -X POST "$BASE/cloud/connectivity-rules" \
  -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \
  -d '{
    "projectId": "'"$PROJECT_ID"'",
    "spec": {
      "privateRule": {
        "connectionId": "vpce-0123456789abcdef0",
        "region": "aws-us-west-2"
      }
    }
  }'
```

Or for a public Connectivity Rule:

```
curl -sS -X POST "$BASE/cloud/connectivity-rules" \
  -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \
  -d '{
    "projectId": "'"$PROJECT_ID"'",
    "spec": {
      "publicRule": { "enableStableIps": true }
    }
  }'
```

Then attach the Connectivity Rule to a Namespace within the same project via [UpdateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/{namespace}). Creating the Connectivity Rule alone does not bind it to a Namespace.

## Move a Namespace to another Project 

You can move a Namespace to another Project in the same Temporal Cloud Account. The Namespace metadata and settings stay the same. That includes its name, full Namespace ID, gRPC endpoint, tags, and its other settings.

A move changes which Project contains the Namespace and recalculates Project-level access:

- Explicit Namespace-level role assignments and Namespace-scoped Service Accounts stay with the Namespace.
- Access that comes only from the source Project does not carry over. A role on the destination Project applies after the move.
- Project-scoped Service Accounts remain in their Project. A Service Account scoped to the source Project does not move with the Namespace.

Running Workflows continue, but subsequent requests from workers or clients that relied on source-Project access can fail authorization. Review the identities used by your workers, clients, and automation before moving a production Namespace. See [Manage Project access](#manage-project-access).

The destination Project must be operational and under its Namespace limit. The Namespace must be active. A suspended Namespace, a Namespace that is still updating, or a Namespace with a migration in progress cannot be moved.

If **Move to Project** is not available, [contact Support](/evaluate/cloud/support#support-ticket).

### Prepare the destination Project

1. Confirm that you are an Account Owner or Global Admin, or a Project Admin on both Projects.
2. Find the source and destination Project IDs in **Project Settings**, or run `temporal cloud project list`.
3. Grant any destination-Project roles that workers, clients, and administrators need after the move.
4. Identify users who will have Project Admin on the destination Project and an explicit role on the Namespace. Plan to remove those Namespace roles immediately after the move completes. Keep them in place until then so existing automations retain their access.
5. Check the Namespace's [Connectivity Rules](/cloud/connectivity). A rule belongs to a Project and does not move with the Namespace. Create the replacement rules in the destination Project before the move.

A Namespace with no Connectivity Rules accepts connections without rule-based network restrictions. Detach rules only when that is the intended result.

The Cloud UI blocks the move when the Namespace has Connectivity Rules attached. To keep rule-based network restrictions, use the CLI or the Cloud Operations API and name rules that already exist in the destination Project.

**Cloud UI**

1. Open the Project and select **Namespaces**.
2. Scroll right to the **Capabilities** column.
3. Select the vertical ellipsis (⋮) on the Namespace row, then select **Move to Project**.
4. Select the destination Project.
5. Review the access warning and select **Move Namespace**.

You can also open the Namespace, select **Settings**, select **Danger Zone**, and select **Move Namespace**.

**Cloud Operations API**

Call
[`MoveNamespaceToProject`](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/namespaces/{namespace}/move-to-project).
Read the Namespace's current `resourceVersion` from
[`GetNamespace`](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/{namespace}).
`ruleIds` is the full set of destination-Project rule IDs the Namespace should have after the move.

```
curl -sS -X POST "$BASE/cloud/namespaces/<namespace>.<account>/move-to-project" \
  -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \
  -d '{
    "destinationProjectId": "<destination-project-id>",
    "expectedSourceProjectId": "<source-project-id>",
    "resourceVersion": "<current-namespace-resource-version>",
    "ruleIds": {
      "connectivityRuleIds": [
        "<destination-rule-id>"
      ]
    }
  }'
```

For authentication headers and API versioning, see [Cloud Ops API](/ops).

To leave the Namespace with no rules, send `"unrestricted": {}` instead of `ruleIds`. Omit both only when the Namespace has no rules before the move. The response contains an async operation. Wait for that operation to finish before you treat the move as complete.

**CLI**

Install a Temporal Cloud CLI release that supports `namespace move-to-project`, then sign in with `temporal cloud login`. Use `temporal cloud whoami` to confirm the account.

Check the Namespace's current rules and the rules available in the destination Project:

```bash
temporal cloud namespace connectivity list \
  --namespace <namespace>.<account>

temporal cloud connectivity list \
  --project-id <destination-project-id>
```

Move the Namespace and attach destination-Project rules. Repeat `--connectivity-rule-id` for each rule. Those rules replace the Namespace's current rules.

```bash
temporal cloud namespace move-to-project \
  --namespace <namespace>.<account> \
  --source-project-id <source-project-id> \
  --destination-project-id <destination-project-id> \
  --connectivity-rule-id <destination-rule-id>
```

Use `--no-connectivity-rules` instead of `--connectivity-rule-id` when the Namespace should have no rules after the move. Omit both only when the Namespace has no rules before the move.

The source Project ID is a safety check: `--source-project-id` is required. The move fails when the Namespace is no longer in that Project. The CLI asks for confirmation and waits until the move finishes. Add `--auto-confirm` to skip the prompt. Add `--async` to return an operation ID, then wait with:

```bash
temporal cloud async-operation await \
  --async-operation-id <operation-id>
```

### Verify the move

Confirm that the Namespace is in the destination Project and has the Connectivity Rules you intended. Test the identities used by your workers and clients, including any that previously relied on a source-Project role or a Project-scoped Service Account.

### Resolve overlapping roles after the move

A Namespace move keeps its explicit Namespace roles. If a user becomes a Project Admin in the destination Project after the move, that user has a role conflict between the Project role and the Namespace role because of [role precedence](#role-precedence). The conflict doesn't block the move, but any later update to that user fails until you remove the overlapping role.

**Immediately after the move**, remove the user's redundant roles on the moved Namespace and keep their Project Admin role. Do this before any other update to that user. Keep the Namespace role in place until the move completes so automations that depend on it continue to work.

## Use Nexus across Projects 

Projects group Nexus Endpoints for organization and authorization. They do not change how Nexus Operations run.

- A Nexus Endpoint lives in one Project. That Project controls who can [create, list, update, and delete](/cloud/manage-access/permissions-reference#cloud-ops-api-permissions-2) the Endpoint.
- Runtime access is unchanged: the Endpoint [allowlist of caller Namespaces](/nexus/security#runtime-access-controls). A caller Namespace in another Project can invoke the Endpoint if it is on that allowlist.
- The target Namespace (handler) does not have to be in the same Project as the Endpoint. Unlike [Connectivity Rules](/cloud/connectivity#connectivity-rules), Nexus does not require the Endpoint and its Namespaces to share a Project.

Example: an Endpoint in Project `payments`, a target Namespace in `payments`, and a caller Namespace in `checkout`.
That call is allowed when the `checkout` Namespace is on the Endpoint allowlist.
A user who can only list Endpoints in `checkout` will not see the Endpoint in `payments`.

## Manage Project access

Users, groups, and Account-scoped Service Accounts can be granted Project-level roles. Each [Project role](#project-roles) applies to the Project and, depending on the role, may grant inherited access to resources within it.

**Cloud UI**

To manage Project access in the Temporal Cloud UI:

1. Open the Project.
2. Click the **Project Identities** tab in the left navigation. The Project Identities page has three tabs, for Users, Service Accounts, and Groups.
3. Click the **Manage Identities** button to add (or remove) users, groups or service accounts, and assign desired Project Roles from the drop-down list. You can also modify or remove an existing principal's role in the same view.

**Cloud Operations API**

Use the
[`SetUserProjectAccess` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId}/users/{userId}/access)
to add Users to a Project. 
Use the
[`SetUserGroupProjectAccess` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId}/user-groups/{groupId}/access)
to add User Groups to a Project. 
Both calls take two required parameters:

- `projectId` - The ID of the Project to set permissions for.
- `userId` or `groupId` - The ID of the User or Group to set permissions for.

Use the `role` attribute to set the role for the User or Group. See [Project roles](#project-roles) for more information.

```
"access": {
  "role": ""
},
```

Similar to [Namespace-scoped Service Accounts](/cloud/manage-access/service-accounts#scoped), a Project-scoped Service Account is an identity bound (or scoped) to a single Project. Use it for workers, CI/CD pipelines, Terraform, or operational automation that should not access resources outside that Project. Project-scoped Service Accounts are not created automatically with Project creation. Create them only when needed. 

**Cloud UI**

To create a Project-scoped service account in this Project, click **Project Identities** in the left navigation. Then click the  **Service Accounts** tab to create a Project-scoped service account.

**Cloud Operations API**

Use the
[`SetServiceAccountProjectAccess` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId}/service-accounts/{serviceAccountId}/access)
to add Service Accounts to a Project. 

The call takes two required parameters:

- `projectId` - The ID of the Project to set permissions for.
- `serviceAccountId` - The ID of the Service Account to set permissions for.

Use the `role` attribute to set the role for the User or Group. See [Project roles](#project-roles) for more information.

```
"access": {
  "role": ""
},
```

After creating the Service Account, you can optionally create one or more API keys for it. Each API key inherits only the permissions of its associated Service Account. If the Service Account is deleted, all of its API keys are deleted automatically.Project-scoped Service Accounts count toward the Project's Service Account limit.

A Project-scope Service Account:

- Can only bound (or scope to) exactly one Project. Each project can contain multiple Project-scoped Service Accounts. 
- Cannot be moved to another Project after it is created.
- Counts toward the Project's Service Account limit.
- Must be deleted before the Project can be deleted.

Unlike an [Account-scoped Service Account](/cloud/manage-access/service-accounts), a Project-scoped Service Account cannot access resources outside its Project. For automation that needs access across multiple Projects, use an Account-scoped Service Account and grant it the appropriate Project roles.

## Project roles 

The following roles are available at the Project level for users, groups, and service accounts:

- **Project Admin:** Complete access to the Project resource itself and all resources in it.
- **Project Write:** Write access to all the Namespaces and other resources in a Project, but cannot add other users.
- **Project Read:** Read-only access to all the resources in a Project, including viewing the actual contents of a Workflow.
- **Project Contribute:** Can create new Namespaces, and list all Project resources, but without access to Workflow contents.
- **Project List:** List-only access to metadata for all the resources in a Project, but without access to Workflow contents.
- **Project Member:** Minimal Project-level access; used when a user already has explicit Namespace-level access and only needs to be represented at the Project level

> **⚠️ Warning:**
> Project Developer
>
> Project Developer is a compatibility role that is automatically inherited by users with the Account Developer role. It allows them to create Namespaces (and manage them) without automatically granting access to all existing Namespaces in the Project. This role cannot be assigned directly. For new role assignments, assign Project Write or Project Contribute as appropriate.
>

A principal automatically inherits a Project-level access from their Account-level role. For example, a Global Admin automatically inherits the Project Admin access for every Project in the Account.

Project-level permissions are additive. Granting a Project-level role does not reduce permissions inherited from Account-level or Namespace-level roles. A principal's effective permissions are the union of all applicable Account-, Project-, and Namespace-level role assignments.

### Role precedence 

Role-precedence validation rejects a redundant lower-scope role when a broader-scope role already grants equal or greater access. This keeps role assignments aligned with effective access.

When the principal's Account role is:

| Existing Account role | Project role you can assign |
| --- | --- |
| Account Owner or Global Admin | None. These roles already include Project Admin on every Project. |
| Developer | Project Admin, Project Write, Project Read, or Project Contribute. |
| Finance Admin or Read-Only | Any Project role. |

When the principal's Project role is:

| Existing Project role | Namespace role you can assign in that Project |
| --- | --- |
| Project Admin | None. Project Admin already includes Namespace Admin on every Namespace in the Project. |
| Project Write | Namespace Admin. |
| Project Read | Namespace Admin or Namespace Write. |
| Project Contribute, Project List, or Project Member | Any Namespace role. |

The Cloud UI removes redundant Namespace roles when you upgrade an Account role. A [Namespace move](#move-a-namespace) does not. After a move, remove an explicit Namespace role that overlaps with Project Admin on the destination Project before you update that user again.

## Modify an existing Project

**Cloud UI**

To modify an existing Project, click **Project Settings** in the left navigation. The Project Settings page appears, with the General tab selected by default. Here, you can change the name or description of your Project. You can also access your Project's unique ID. You can copy the ID, but you can't change it.

**Cloud Operations API**

Use the
[`UpdateProject` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId})
to add Service Accounts to a Project. 

The call takes one required parameter:

- `projectId` - The ID of the Project to set permissions for.

The structure of the `UpdateProjectRequest` is defined by a JSON spec, as follows:

```
"spec": {
  "description": "",
  "displayName": "",
  "lifecycle": {
    "enableDeleteProtection": true
  }
}
```

In this call, `description` is a string description of the Project, `displayName` is the Project name to be displayed in the UI, and `enableDeleteProtection` is a Boolean that indicates whether the Project can be deleted.

## Delete a Project

A Project must be empty, meaning it contains zero Namespaces, Nexus Endpoints, Connectivity Rules, or Project-scoped Service Accounts, to be deleted. 

An Account must always contain at least one Project. If the Account contains two or more Projects, you can delete the Default Project. If the Default Project is deleted, Temporal does not automatically designate a new Default Project. If the Default Project is deleted, API calls that do not specify a Project (by providing a Project ID) will fail. 

**Cloud UI**

To delete a Project, click **Project Settings** in the left navigation and then click the **Delete Project** tab. Click the **Delete** button to permanently delete the Project.

Use the **Deletion Prevention** slider to set the Deletion Protection flag for this Project. Once set, the Project can't be deleted unless the flag is deliberately disabled.

**Cloud Operations API**

Use the
[`DeleteProject` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/DELETE/cloud/projects/{projectId})
to delete a Project. 

The call takes one required parameter: 

- `projectId` - The ID of the Project to delete.

This call will not succeed if the delete protection flag is enabled.

Projects support a delete protection flag, similar to Namespaces. When this flag is enabled, the Project cannot be deleted until the flag is disabled. This flag is enabled for the Default Project. The flag is disabled for any new Project created by a user. This flag is not inherited by the Namespaces within the Project. Namespace delete protection can be managed per-Namespace. If a Project is the only one remaining in an Account, that project cannot be deleted, regardless of the status of the Deletion Protection Flag.

---

# Outages and Recovery Objectives (RTO / RPO)

Source: https://docs.temporal.io/cloud/rpo-rto

When a cloud outage disrupts a Namespace, Temporal Cloud takes measures to maintain the Namespace's availability and
data durability. The time it takes to recover from the outage is called the _recovery time_. The _recovery point_ is how
far back in time data must be recovered from after an outage. A durable system should have a low recovery time and a
near recovery point.

Temporal Cloud publishes goals for the recovery time and recovery point for each kind of outage. These goals are called
the Recovery Time Objective (RTO) and Recovery Point Objective (RPO). For details on how each is measured, see
[How RTO and RPO are measured](#how-rto-and-rpo-are-measured). These objectives are complementary to Temporal Cloud's
[Service Level Agreement (SLA)](https://temporal.io/sla).

The RTO and RPO for a Namespace depend on the type of outage and which [High Availability](/cloud/high-availability)
features the Namespace has enabled.
Your real-world recovery time also depends on your [Worker deployment pattern](/cloud/high-availability/architecture-patterns) — how quickly Workers resume processing in the new region after the Namespace fails over.

## RTO and RPO summary

The following table summarizes the RTO and RPO targets for each type of outage. These targets apply to Namespaces that
have automatic failovers enabled, which is the default. Automatic failovers are triggered by
Temporal's tooling and on-call engineers without user action. Users can always initiate a failover independently. In an
outage, a user-initiated failover will not cancel out or reverse an automatic failover.

These targets are for unplanned cloud outages and do not apply to user-initiated failovers during healthy periods, such
as DR drills. Read about [triggering a failover](/cloud/high-availability/failovers/manage#trigger-failover) to see how a Namespace failover
performs during healthy periods.

| Outage type                                           | Applicable Namespaces                                                 | RPO            | RTO              |
| ----------------------------------------------------- | --------------------------------------------------------------------- | -------------- | ---------------- |
| [Availability Zone outage](#availability-zone-outage) | All Namespaces                                                        | Zero           | Near-zero        |
| [Cell outage](#cell-outage)                           | Namespaces with Same-region, Multi-region, or Multi-cloud Replication | Under 1 minute | Under 20 minutes |
| [Cloud Region outage](#cloud-region-outage)           | Namespaces with Multi-region or Multi-cloud Replication               | Under 1 minute | Under 20 minutes |
| [Cloud-wide outage](#cloud-wide-outage)               | Namespaces with Multi-cloud Replication                               | Under 1 minute | Under 20 minutes |

> **💡 Tip:**
>
> Temporal highly recommends keeping automatic failovers enabled. When automatic failovers are
> _disabled,_ Temporal Cloud cannot set an RPO and RTO for that Namespace, because it cannot control when or if the user
> will trigger a failover.
>

As soon as a cloud outage resolves, Temporal's on-call engineers work to restore service to Namespaces that were not
protected by High Availability. A cloud outage can leave lingering effects in Temporal's systems and applications, even
after the cloud provider restores the underlying service. An affected Namespace's outage may last longer than the cloud
provider's outage.

All Namespaces are backed up every 4 hours. If an outage causes data loss on a Namespace that was not protected by High
Availability, Temporal uses the backup to restore as much data as feasible.

## Outage types and their RTO/RPO

The following sections explain each type of outage in more detail, including the blast radius, Temporal Cloud features
that mitigate the outage, and whether the outage is included in the SLA calculation.

### Availability Zone outage 

An
[Availability Zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones)
(AZ) is akin to an isolated datacenter managed by a cloud hyperscaler, with independent power, networking, and cooling
infrastructure. Each cloud region contains multiple AZs, and an individual AZ can fail due to events such as hardware
failure, power loss, or a localized network partition.

AZ outages are the most common type of outage, and Temporal Cloud has weathered many of them transparently.

**Blast Radius:** A single Availability Zone within a single cloud region. Because every Namespace's components are
spread across at least three AZs, the blast radius to Temporal Cloud users is typically zero — Namespaces stay
operational with little to no downtime.

> **⚠️ Caution:**
>
> While Temporal Cloud can withstand single AZ outages without disruption, if you have Workers that are deployed in the
> impacted AZ, those Workers may be disrupted. To mitigate this risk, Temporal recommends deploying your Workers across
> multiple AZs.
>

**Mitigation:** Every Namespace is automatically spread across at least three Availability Zones, and any Namespace can
handle a single AZ failure without disruption to end-user Temporal operations.
[High Availability](/cloud/high-availability) features are _not_ required to keep Temporal Cloud operations running
through an AZ outage.

**SLA inclusion:** Included in the [SLA](https://temporal.io/sla) calculation. Any errors during an AZ outage count toward SLA
credits, since AZ resilience is within Temporal's responsibility.

If two AZs fail simultaneously, Temporal Cloud treats the event as a [Cloud Region outage](#cloud-region-outage). In
that case, Namespaces in the region may be impacted, including those using
[Same-region Replication](/cloud/high-availability#same-region-replication).

> **ℹ️ Info:**
>
> When an AZ fails, Temporal may also trigger a failover on Namespaces that have High Availability enabled, as a
> precaution in case the outage scope expands. For Multi-region and Multi-cloud Replication, you can opt out of this
> behavior by [disabling automatic failovers](/cloud/high-availability/enable#automatic-failovers) on the Namespace.
> Same-region Replication Namespaces always fail over automatically and cannot opt out.
>

#### RTO and RPO

When using Temporal Cloud (no additional features required):

- **Near-zero RTO.** When a single AZ fails, the remaining two AZs continue serving requests without a failover, so end
  users see little to no disruption.
- **Zero RPO.** Writes to Workflow state are synchronously replicated across all three AZs before being acknowledged
  back to the Client, so an AZ failure cannot cause data loss.

### Cell outage 

Temporal Cloud runs on a
[cell architecture](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html).
Each cell contains the software and services necessary to host a Namespace, and components within a cell are distributed
across at least three Availability Zones. Cells provide a strong unit of isolation: a problem inside one cell does not
propagate to other cells. A cell outage occurs when a cell becomes degraded or unavailable, disrupting the Namespaces
hosted within it.

**Blast Radius:** One cell, and the Namespaces within that cell, within a single region. Even though your Workers will
remain healthy, they will not be able to process Workflows because the Namespace is down.

**Mitigation:** [Multi-region Replication](/cloud/high-availability) and
[Multi-cloud Replication](/cloud/high-availability) replicate a Namespace into another cell in a different region or
different cloud provider. [Same-region Replication](/cloud/high-availability) replicates a Namespace into another cell
within the same region. When any of these features are enabled for a Namespace, an outage that disrupts a single cell
can be mitigated by failing the Namespace over to its replica.

**SLA inclusion:** Included in the [SLA](https://temporal.io/sla) calculation. Any errors during a cell outage count toward SLA
credits, since mitigating cell outages is within Temporal's responsibility.

Cell-level disruptions occur from time to time, and Temporal's replication and failover tooling has restored affected
Namespaces in real-world incidents.

#### RTO and RPO

When using Same-region Replication, Multi-region Replication, or Multi-cloud Replication for automatic failover:

- **RTO under 20 minutes.** Temporal detects the disruption and fails the Namespace over to its replica cell.
- **RPO under 1 minute.** Asynchronous replication keeps the replica close to the active cell.

Even though the RPO target is under 1 minute, data is virtually never "lost" thanks to Temporal's built-in Recovery and
Conflict Resolution process, which reconciles state between the active and replica when the outage is over.

### Cloud Region outage 

A cloud region as a whole can become degraded, with effects that span beyond any single cell or Availability Zone.

**Blast Radius:** All Namespaces and Workers within a single cloud region are potentially affected.

**Mitigation:** [Multi-region Replication](/cloud/high-availability) and
[Multi-cloud Replication](/cloud/high-availability) place the replica outside the affected region, so a Namespace can
fail over and continue serving Workflows. Same-region Replication does not protect against a Cloud Region outage, since
the replica resides in the same region.

**SLA inclusion:** Included in the [SLA](https://temporal.io/sla) calculation only for Namespaces that have Multi-region Replication
or Multi-cloud Replication enabled with automatic failovers — in those cases, Temporal can mitigate the outage.
For Namespaces without these features, a Cloud Region outage is excluded from the SLA calculation, as it is beyond
Temporal's control to mitigate.

If two or more regions in the same cloud provider experience an outage simultaneously, Temporal Cloud treats the event
as a [Cloud-wide outage](#cloud-wide-outage).

Regional outages are less common than cell or AZ outages, but they do happen. During the
[AWS us-east-1 incident on October 20, 2025](https://temporal.io/blog/how-devs-kept-running-during-the-aws-us-east-1-oct-20-2025),
Temporal Cloud's regional failover kept customer Namespaces running.

#### RTO and RPO

When using Multi-region Replication or Multi-cloud Replication for automatic failover:

- **RTO under 20 minutes.** Temporal detects the regional disruption and fails the Namespace over to its replica in
  another region.
- **RPO under 1 minute.** Asynchronous replication keeps the replica close to the active region.

Even though the RPO target is under 1 minute, data is virtually never "lost" thanks to Temporal's built-in Recovery and
Conflict Resolution process, which reconciles state between the active and replica when a failover occurs.

### Cloud-wide outage 

On rare occasions, an issue affects two or more regions of a single cloud provider at once. Any simultaneous outage of
two or more regions in the same cloud provider is treated as a cloud-wide outage.

**Example causes:** a software bug rolled out to every region of a cloud provider that triggers cascading failures
across the provider's infrastructure, or two or more regions in the same cloud experiencing independent regional outages
at the same time.

**Blast Radius:** Most or all regions of a single cloud provider. Every Namespace and every Worker hosted in that cloud
is potentially affected.

**Mitigation:** [Multi-cloud Replication](/cloud/high-availability) places the replica in a different cloud provider
entirely, so the Namespace can fail over even when an entire cloud provider goes down.

**SLA inclusion:** Included in the [SLA](https://temporal.io/sla) calculation only for Namespaces that have Multi-cloud Replication
enabled with automatic failovers — in those cases, Temporal can mitigate the outage. For Namespaces without this
feature, a cloud-wide outage is excluded from the SLA calculation, as it is beyond Temporal's control to mitigate.

Cloud-wide outages are the rarest category, but they
[have occurred](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Multi-cloud Replication is designed to
keep Namespaces running through such events.

#### RTO and RPO

When using Multi-cloud Replication for automatic failover:

- **RTO under 20 minutes.** Temporal detects the cloud-wide disruption and fails the Namespace over to its replica in a
  different cloud provider.
- **RPO under 1 minute.** Asynchronous replication keeps the replica close to the active region, even across cloud
  providers.

Even though the RPO target is under 1 minute, data is virtually never "lost" thanks to Temporal's built-in Recovery and
Conflict Resolution process, which reconciles state between the active and replica when a failover occurs.

## How RTO and RPO are measured 

Temporal Cloud achieves its RTO and RPO targets through [High Availability](/cloud/high-availability) replication. The
following sections explain how each metric is measured and what factors can affect them.

### RPO 

Unlike a traditional database where data within the recovery point window may be permanently lost, Temporal Cloud
durably persists all acknowledged data. After an outage resolves, Temporal's Recovery and Conflict Resolution process
automatically syncs data back into the Namespace. The RPO therefore reflects the maximum data that may be _temporarily
unavailable_ in the replica at the moment of failover, not data that is permanently lost.

Temporal keeps replicas up to date using
[asynchronous replication](https://youtu.be/mULBvv83dYM?si=RDeWb3gVsEtgGM4z&t=334), with monitoring, alerting, and
internal SLOs on replication lag for every Namespace.

User actions on a Namespace can affect the recovery point. For example, suddenly spiking into much higher throughput
than a Namespace has seen before could create a period of replication lag where the replica falls behind the active.

Temporal provides a
[replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) metric for each
Namespace. This metric approximates the recovery point the Namespace would achieve in a worst-case failure at that
moment. Temporal recommends monitoring the replication lag and alerting if it rises above 1 minute.

### RTO 

The Recovery Time for a given incident is measured from the moment the incident begins to cause abnormal Namespace
operation — for example, when unavailability or error rates rise above an acceptable level — to the moment the Namespace
is restored to full functionality.

For most incidents, the vast majority of the Recovery Time is spent detecting the incident, determining the affected
boundary (a single cell, a region, or an entire cloud), and deciding to fail Namespaces over to their replicas. The
actual time to complete the failover is usually a very small piece of the Recovery Time.

This Recovery Time covers only the Temporal Namespace. Your application's overall Recovery Time also depends on having
enough healthy Workers that can reach the Namespace and process Workflows. Maintaining sufficient Worker capacity that
can reach the replica region (or replica cloud) during a failover is your responsibility. You are also responsible for
failing over any other regional dependencies your application relies on, such as replicated application databases.

## Tips for a lower recovery time

To achieve the lowest possible recovery times, Temporal recommends that you:

- Keep automatic failovers enabled on your Namespace (the default)
- Invest in a process to detect outages and trigger a manual failover.

You can trigger manual failovers on your Namespaces even if automatic failovers are enabled. There are several
benefits to combining a manual failover process with automatic failovers:

- You can detect outages that Temporal doesn't. In the cloud, regional outages don't affect all services equally. It's
  possible that Temporal--and the services it depends on--are unaffected by the outage, even while your Workers or other
  cloud infrastructure are disrupted. If you
  [monitor services in your critical path](https://sre.google/sre-book/monitoring-distributed-systems/) and alert on
  unusual error rates, you may catch outages before Temporal Cloud does.

- You can sequence your failovers in a particular order. Your cloud infrastructure probably contains more pieces than
  just your Temporal Namespace: Temporal Workers, compute pools, data stores, and other cloud services. If you manually
  fail over, you can choose the order in which these pieces switch to the replica region. You can then test that
  ordering with failover drills and ensure it executes smoothly without data consistency issues or bottlenecks.

- You can proactively fail over more aggressively than Temporal. While the 20-minute RTO should be sufficient for most
  use cases, some may strive to hit an even lower RTO. For workloads like high frequency trading, auctions, or popular
  sporting events, an outage at the wrong time could cause tremendous lost revenue per minute. You can adopt a posture
  that fails over more eagerly than Temporal does. For example, you could trigger a manual failover at the first sign of
  a possible disruption, before knowing whether there's a true regional outage.

- Even if you have robust tooling to detect an outage and trigger a failover, leaving automatic failovers
  enabled provides a "safety net" in case your automation misses an outage. It also gives Temporal leeway to
  preemptively fail over your Namespace if we detect that it may be disrupted soon, for example, by a rolling failure that has
  impacted other Namespaces but not yours, yet.

## Comparing RTO and SLA

Temporal has both a Recovery Time Objective (RTO) and a Service Level Agreement (SLA). They serve complementary purposes
and apply in different situations.

| Aspect                            | RTO                                                                                                                                                                                                                                                          | SLA                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| What is it?                       | An objective, or high-priority goal, for the total time that an outage disrupts a Namespace.                                                                                                                                                                 | A contractual agreement that sets an upper bound on the service error rate, with financial repercussions.                                                                                                                                                                                                                                                                                                                                                                      |
| How is it measured?               | The achieved recovery time is measured in terms of minutes per outage.                                                                                                                                                                                       | The achieved service error rate is measured in terms of error rate per month.                                                                                                                                                                                                                                                                                                                                                                                                  |
| How is the calculation performed? | The achieved recovery time in a given outage is the total time between when a disruption to a Namespace began and when the Namespace was restored to full functionality, either after a failover to a healthy region or after the outage has been mitigated. | Temporal measures the percentage of requests to Temporal Cloud that fail, and applies a [formula](https://temporal.io/sla) to get the final percentage for the month.                                                                                                                                                                                                                                                                                                                       |
| Do partial degradations count?    | Most outages contain periods of **partial degradation** where some percentage of Namespace operations fail while the rest complete as normal. When they disrupt a Namespace, periods of partial degradation count in the calculation of the recovery time.   | Partial degradations only partially count for the service error rate calculation. A 5-minute window with a 10% error rate would count less than a 5-minute window with a 100% error rate.                                                                                                                                                                                                                                                                                      |
| What is excluded?                 | For partial degradations, what counts as a disruption to a Namespace is subject to Temporal's expert judgment, but a good rule of thumb is a service error rate >=10%.                                                                                       | We exclude outages that are out of Temporal's control to mitigate, for example, a failure of the underlying cloud provider infrastructure that affects a Namespace without High Availability and automatic failovers enabled. If a Namespace has the relevant High Availability feature and has automatic failovers enabled, then Temporal can act to mitigate the outage and it does usually count against the SLA. Full exclusions in the [SLA](https://temporal.io/sla). |

The following examples illustrate the RTO and SLA calculations for different types of outages in a regional outage. These
hypothetical Namespaces are based on actual Temporal Cloud performance in a
[real-world outage](https://temporal.io/blog/how-devs-kept-running-during-the-aws-us-east-1-oct-20-2025).

Suppose that region `middle-earth-1` experienced a cascading failure starting at 10:00:00 UTC, causing various instances
and machines to fail over time. Temporal's automatic failover triggered for all Namespaces and completed at 10:15:00
UTC.

- Namespace 0 was in the region but its cell was not affected by the outage. The only downtime it had was for a few
  seconds during the failover operation. It experienced a near-zero Recovery Time, and its service error rate was
  negligible. Graceful failover was successful, and this Namespace achieved a recovery point of 0.

- Namespace 1_A was in the region and its cell experienced a partial degradation that caused 10% of requests to fail in
  the first 5 minutes, 25% in the second five minutes, and 50% in the third five minutes. Since it was significantly
  impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If it had no other service errors that month,
  then its service error rate for the month would be: ( (1 - 10%) + (1 - 25%) + (1 - 50%) + 8925 \* 100% ) / 8928 =
  99.990%. (Note: there are 8928 5-minute periods in a 31-day month.) Graceful failover was successful, and this
  Namespace achieved a recovery point of 0.

- Namespace 1*B was in the same cell as Namespace 2_A, so it also experienced a partial degradation that caused 10% of
  requests to fail. However, its owner detected the outage via their own tooling and decided to manually fail over at
  10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 * (1 - 10%) + 8927 \_
  100% ) / 8928 = 99.998%. Graceful failover was successful, and this Namespace achieved a recovery point of 0.

- Namespace 2*A was in the region and its cell was fully network partitioned at the start of the outage, causing 100% of
  requests to fail. Since it was significantly impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If
  it had no other service errors that month, then its service error rate for the month would be: ( 3 * (1 - 100%) + 8928
  \_ 100% ) / 8640 5-minute periods per month = 99.97%. Because the Namespace was network partitioned, graceful failover
  did not succeed, and forced failover was used. The recovery point achieved was equal to the replication lag at the
  time of the network partition, which was a few seconds.

- Namespace 2*B was in the region and was fully network partitioned, causing 100% of requests to fail. However, its
  owner detected the outage via their own tooling and decided to manually fail over at 10:05:00. This Namespace achieved
  a recovery time of 5 minutes and a service error rate of ( 1 5-minute periods * (1 - 100%) + 8639 5-minute periods \_
  100% ) / 8640 5-minute periods per month = 99.99%. Because the Namespace was network partitioned, graceful failover
  did not succeed, and forced failover was used. The recovery point achieved was equal to the replication lag at the
  time of the network partition, which was a few seconds.

All of the above Namespaces were in the affected region and beat the 1-minute RPO. But they achieved varying recovery
times and service error rates.

- Notice how Namespace 1_A and Namespace 2_A were both automatically failed over with **the same recovery time but
  different service error rates**. Notice how Namespace 2_B and Namespace 1_A happen to have the **same service error
  rate but different recovery times**. This illustrates how RTO and SLA can differ, even in the same outage. Both are
  valuable tools for Temporal Cloud users to measure the availability of their Namespaces.

- Notice how the Namespaces that were manually failed over (Namespace 1_B and Namespace 2_B) achieved lower recovery
  times than the Namespaces that were automatically failed over (Namespace 1_A and Namespace 2_A). This illustrates how
  **proactive, aggressive manual failover can achieve a better recovery time than automatic failover**.

---

# Monitor Temporal Cloud service health

Source: https://docs.temporal.io/cloud/service-health

> Use Temporal Cloud metrics to monitor service latency, errors, failures, throttling, and capacity limits.

Temporal Cloud metrics help you monitor the service health of production deployments.
This page covers recommended signals for monitoring Temporal Cloud.

## Monitor availability issues

When you see a sudden drop in Worker resource utilization, verify whether Temporal Cloud's API is showing increased latency and error rates.

### Reference Metrics

- [temporal\_cloud\_v1\_service\_latency\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_latency_p99)

This metric measures latency for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations.
These operations are mission critical and never [throttled](/cloud/slo#throughput).
This metric is a good indicator of your lowest possible latency for the 99th percentile of requests.

### Workflow execution latency

To monitor end-to-end Workflow execution time (not just the service API latency above), use the workflow schedule-to-close latency metrics:

- [temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p50](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_workflow_schedule_to_close_latency_p50)
- [temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p95](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_workflow_schedule_to_close_latency_p95)
- [temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_workflow_schedule_to_close_latency_p99)

These measure the time from when a Workflow is scheduled until it closes, including all Activity execution time. A sudden increase may indicate Worker capacity issues, downstream service degradation, or retry storms.

## Monitor Temporal Service errors

Check for Temporal Service gRPC API errors.
Note that Service API errors are not equivalent to guarantees mentioned in the [Temporal Cloud SLA](https://temporal.io/sla).

### Reference Metrics

- [temporal\_cloud\_v1\_service\_error\_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_error_count)
- [temporal\_cloud\_v1\_service\_request\_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_request_count)

### Prometheus Query for this Metric

Measure your daily average success rate over 10-minute windows.

OpenMetrics v1 metrics are pre-computed rates. Use `sum()` to aggregate across dimensions rather than `increase()` or `rate()`.

```
avg_over_time((
    (
        (
            sum(temporal_cloud_v1_service_request_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"})
            -
            sum(temporal_cloud_v1_service_error_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"})
        )
        /
        sum(temporal_cloud_v1_service_request_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"})
    )

    or vector(1)

    )[1d:1m])
```

## Detecting Activity and Workflow Failures

The metrics `temporal_cloud_v1_activity_fail_count` and `temporal_cloud_v1_workflow_failed_count` together provide failure detection for Temporal applications. These metrics work in tandem to give you both granular component-level visibility and high-level workflow health insights.

### Activity failure cascade

If not using infinite retry policies, Activity failures can lead to Workflow failures:

```
Activity Failure --> Retry Logic --> More Activity Failures --> Workflow Decision --> Potential Workflow Failure
```

Activity failures are often recoverable and expected. Workflow failures represent terminal states requiring immediate attention.
A spike in activity failures may precede workflow failures.
Generally Temporal recommends that Workflows should be designed to always succeed. If an Activity fails more than its retry policy allows, we suggest having the Workflow handle Activity failure and take action to notify a human to take corrective action or be aware of the error.

### Ratio-based monitoring

#### Failure conversion rate

Monitor the ratio of workflow failures to activity failures:

```
workflow_failure_rate = temporal_cloud_v1_workflow_failed_count / temporal_cloud_v1_activity_fail_count
```

What to watch for:
- High ratio (greater than 0.1): Poor error handling - activities failing are causing workflow failures
- Low ratio (less than 0.01): Good resilience - activities fail but workflows recover
- Sudden spikes: May indicate systematic issues

#### Activity success rate

```
activity_success_rate = temporal_cloud_v1_activity_success_count / (temporal_cloud_v1_activity_success_count + temporal_cloud_v1_activity_fail_count)
```

Target: >95% for most applications. Lower success rate can be a sign of system troubles.
See also:
- [Crafting an Error Handling Strategy](https://learn.temporal.io/courses/errstrat/)
- [Temporal Failures reference](/references/failures)
- [Detecting Workflow failures](/encyclopedia/detecting-workflow-failures)

## Monitor replication lag for Namespaces with High Availability features

Replication lag refers to the transmission delay of Workflow updates and history events from the primary Namespace to the replica.
Always check the [metric replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) before initiating a failover.
A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress.

**Who owns the replication lag?**
Temporal owns replication lag.

**What guarantees are available?**
There is no SLA for replication lag.
Temporal recommends that customers do not trigger failovers except for testing or emergency situations.
By default, Temporal fails over Namespaces that use High Availability automatically during an outage. For recovery targets, see [Outages and Recovery Objectives](/cloud/rpo-rto).
Temporal also monitors replication lag.
Customers who decide to trigger failovers should look at this metric before moving forward.

**If the lag is high, what should you do?**
We don't expect users to failover.
Please contact Temporal support if you feel you have a pressing need.

**Where can you read more?**
See [operations and metrics](/cloud/high-availability) for Namespaces with High Availability features.

### Reference Metrics

- [temporal\_cloud\_v1\_replication\_lag\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99)
- [temporal\_cloud\_v1\_replication\_lag\_p95](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p95)
- [temporal\_cloud\_v1\_replication\_lag\_p50](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p50)

## Monitoring trends against limits 

Tracking trends against your account limits is the most important throttling signal to monitor. Unlike [Resource Exhaustion](#detecting-resource-exhaustion), which usually self-heals through retries, hitting a limit slows or stalls progress until the workload backs off or your capacity is increased.

The set of [limit metrics](/cloud/metrics/openmetrics/metrics-reference#limit-metrics) provide a time series of values for limits. Use these
metrics with their corresponding count metrics to monitor general trends against limits and set alerts when limits are exceeded. Use the corresponding throttle metrics
to determine the severity of any active rate limiting.
| Limit Metric | Count Metric | Throttle Metric |
| ------------ | ------------ | --------------- |
| `temporal_cloud_v1_action_limit` | `temporal_cloud_v1_total_action_count` | `temporal_cloud_v1_total_action_throttled_count` |
| `temporal_cloud_v1_service_request_limit` | `temporal_cloud_v1_service_request_count` | `temporal_cloud_v1_service_request_throttled_count` |
| `temporal_cloud_v1_operations_limit` | `temporal_cloud_v1_operations_count` | `temporal_cloud_v1_operations_throttled_count` |

### On-demand envelope limits

For Namespaces using provisioned capacity, the following metrics show what your limits would be under on-demand mode.
Compare these against your current provisioned limits to evaluate capacity mode choices:

| On-Demand Envelope Metric | Equivalent Limit Metric |
| ------------------------- | ----------------------- |
| `temporal_cloud_v1_action_on_demand_envelope_limit` | `temporal_cloud_v1_action_limit` |
| `temporal_cloud_v1_operations_on_demand_envelope_limit` | `temporal_cloud_v1_operations_limit` |
| `temporal_cloud_v1_service_request_on_demand_envelope_limit` | `temporal_cloud_v1_service_request_limit` |

For Namespaces already in on-demand mode, these metrics track the same values as their equivalent limit metrics.

The [Grafana dashboard example](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) includes a Usage & Quotas section
that creates demo charts for these limits and count metrics respectively.

The limit metrics, throttle metrics, and count metrics are already directly comparable as per second rates. Keep in mind that each `count` metric is represented as a per second rate averaged
over each minute. For example, to get the total count of Actions, you must multiply this metric by 60.
When setting alerts against limits, consider if your workload is spiky or sensitive to throttling (for example, does latency matter?). If your workload is sensitive, consider alerting
for `temporal_cloud_v1_total_action_count` at a 50% threshold of the `temporal_cloud_v1_action_limit`. If your workload is not sensitive, consider an alert at 90% of this threshold
or directly when throttling is detected as a value greater than zero for `temporal_cloud_v1_total_action_throttled_count`. This logic can also be used to automatically scale [Temporal
Resource Units](/cloud/capacity-modes#provisioned-capacity) up or down as needed. Some workloads choose to exceed limits and accept throttling because they are not latency sensitive.

### Provisioned capacity utilization

Each TRU provides 500 Actions per second. The first TRU has an hourly minimum of 0 Actions. Each TRU after the first has an hourly minimum of 360,000 Actions, equal to 20% of its capacity. Billing uses the highest TRU count configured during the hour. If workload Actions fall below the hourly minimum Actions, Temporal adds Actions equal to the shortfall.

To estimate the additional billed Actions caused when workload Actions fall below the hourly minimum Actions, calculate the shortfall for each calendar hour by comparing workload Actions with hourly minimum Actions.

```text
additional billed Actions = max(0, hourly minimum Actions - workload Actions)
```

Calculate workload Actions by summing [`temporal_cloud_v1_total_action_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_total_action_count) with `is_background=false` across the Namespace's series for each one-minute interval, multiplying each rate by 60, and summing the results for the calendar hour.

Calculate hourly minimum Actions by subtracting one from the maximum observed TRU count during the hour and multiplying by 360,000 Actions. Use [`temporal_cloud_v1_provisioned_capacity_tru_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_provisioned_capacity_tru_count) to track the number of TRUs configured for each Namespace. A value of `0` means the Namespace uses On-Demand Capacity.

Add the hourly shortfalls to estimate additional billed Actions over a longer period.

The [Grafana dashboard example](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) includes panels for provisioned capacity utilization and limits.

OpenMetrics estimates usage. Use [Usage](/cloud/actions-usage#usage) and [Billing](/cloud/billing) as the source of truth for billed Actions.

Some workloads intentionally run at low utilization to preserve capacity for traffic spikes or unpredictable demand. The hourly minimum Actions still apply.

### Why does throttling occur when count metrics stay below the limit?

For spiky workloads, the throttle metric can be non-zero even though the count metric never rises above the limit. This looks contradictory, but both values are correct. They describe the workload at different time resolutions.

Count and limit metrics are per-second rates **averaged over a 1-minute window** (see [Metric conventions](/cloud/metrics/openmetrics/metrics-reference#metric-types)). A short burst is smoothed across all 60 seconds, so the count metric can sit well below the limit even though the instantaneous request rate exceeded it and triggered throttling. The throttle metric, not the count-versus-limit comparison, is what tells you throttling actually occurred.

#### Example: a spiky Actions workload

Assume an Actions per second (APS) limit of 2,000 (`temporal_cloud_v1_action_limit` = 2000).

A workload submits 60,000 Actions in a single second, then stays idle for the rest of the minute:

- The rate limiter admits about 2,000 Actions in that second and throttles the remaining ~58,000. The SDK retries throttled Actions, which drain through at the 2,000 APS limit over the next ~30 seconds. They complete, but delayed. When throttling persists, delayed completions such as `RespondWorkflowTaskCompleted` can push Workflow Tasks and Activities past their timeouts and cause retries.
- `temporal_cloud_v1_total_action_throttled_count` reflects the throttling: ~58,000 Actions throttled over the minute, or ~967 per second.
- `temporal_cloud_v1_total_action_count` reports 60,000 Actions / 60 seconds = **1,000 APS** — half the 2,000 limit.

Read in isolation, the count metric (1,000) against the limit (2,000) suggests plenty of headroom and no throttling. The throttle metric tells the true story: the burst exceeded the limit and ~58,000 Actions were delayed.

#### Monitor count, limit, and throttle together

To understand a spiky workload, always read all three metrics in the row as a set:

| Metric | What it tells you |
| ------ | ----------------- |
| Count (for example, `temporal_cloud_v1_total_action_count`) | Average demand over the minute |
| Limit (for example, `temporal_cloud_v1_action_limit`) | Your provisioned ceiling |
| Throttle (for example, `temporal_cloud_v1_total_action_throttled_count`) | Whether the limit was actually hit |

A non-zero throttle value means throttling occurred during that window, even when the count sits comfortably below the limit. Most often this reflects a sub-minute burst in your own workload. If you cannot find a matching burst in your SDK metrics, the cause may be a shared limit or another Cloud-side condition rather than your namespace. In that case, contact [Temporal Support](/evaluate/cloud/support#support-ticket). For spiky or latency-sensitive workloads, alert on the throttle metric directly (any value greater than zero) rather than relying only on a count-versus-limit threshold, which can hide sub-minute bursts. The same logic applies to all three limit types — Actions (APS), service requests (RPS), and operations — using each row of the [limit / count / throttle table](#rps-aps-rate-limits) above.

## Detecting resource exhaustion

Resource exhaustion happens when a single resource (a Namespace, Task Queue, or Workflow ID) receives a burst of operations larger than that resource can absorb in the moment. The Cloud metric `temporal_cloud_v1_resource_exhausted_error_count` increments and `ResourceExhausted` gRPC errors are returned to the client. SDKs retry these errors gracefully, so workflow progress is rarely impacted.

Persistent non-zero values are unexpected and indicate a hot resource. Use the `operation` label to identify which RPC is hitting the burst limit. For example, `StartWorkflowExecution` increments here when the same Workflow Id is started more than once per second. See [Per-primitive Id reuse rate limits](/evaluate/cloud/limits#per-primitive-id-reuse-rate-limits).

Resource exhaustion is distinct from rate limiting against your account limits. For workloads that are throttled because they exceed their provisioned capacity, see [Monitoring Trends Against Limits](#rps-aps-rate-limits). Limits-driven throttling slows or stalls a workload, so it is generally the more important signal to monitor.

### Workflow lock contention (BusyWorkflow)

The most common cause of resource exhaustion is Workflow lock contention. Every operation that mutates a single Workflow Execution (starting it, sending a Signal, etc.) is serialized under a per-Workflow lock. When operations reach one Execution faster than that lock can be acquired, the Service rejects the excess with a `ResourceExhausted` error. In Service logs this appears as `Workflow is busy.`

This is contention on a single Execution, not an account limit. Increasing your Actions, Requests, or Operations per second limits does not resolve it.

To confirm lock contention:

1. Rule out account-limit throttling first. If the throttle metrics described in [Monitoring Trends Against Limits](#rps-aps-rate-limits) are elevated, address that throttling first.
2. If you are within your limits but `temporal_cloud_v1_resource_exhausted_error_count` is still non-zero, break it down by the `operation` label. Lock contention concentrates on operations that target individual executions.
3. Match the operation to the guidance below.

| `operation` | What it indicates | What to do |
| ----------- | ----------------- | ---------- |
| `StartWorkflowExecution`, `SignalWithStartWorkflowExecution` | The same Workflow ID was started again within a short de-duplication window (about one second). The first start succeeded; the duplicate was rejected. | Usually safe to ignore. Do not aggressively retry. There may be a client path firing the duplicate start. |
| `SignalWorkflowExecution` | Too high of a rate of Signals to one execution. | Batch or coalesce Signals (for example, one Signal per N events), shard work across more executions, or buffer Signals and drain them in the main Workflow loop. |
| `UpdateWorkflowExecution` | More than the per-execution in-flight Update limit (10) are outstanding. | Cap concurrent in-flight Updates on the client, then back off and retry. |
| `RecordActivityTaskHeartbeat` | Too many Activities are heartbeating to the same execution. | Increase the heartbeat timeout and interval, and reduce the number of Activities heartbeating into one execution concurrently. |
| `RespondWorkflowTaskCompleted` | A single Workflow schedules a large batch of Activities or Child Workflows in parallel, each taking the lock. | Limit concurrent operations to 500 or fewer per execution. Process the batch in smaller groups using a sliding-window or plain batching pattern instead of scheduling everything at once. |
| `QueryWorkflow` | Too many concurrent Queries against one execution, or a side effect of repeated Workflow Task retries. | Reduce concurrent Queries to that execution. If it correlates with Workflow Task failures or timeouts, resolve those first. |

At low, brief rates this error is benign because clients retry it and no progress is lost. Investigate when the rate is sustained or correlates with rising latency on the affected operations. For the per-execution limits referenced above, see [Per Workflow Execution concurrency limits](/evaluate/cloud/limits#per-workflow-execution-concurrency-limits).

---

# Uptime, latency, and SLOs

Source: https://docs.temporal.io/cloud/slo

> Temporal Cloud's 99.99% uptime objective, the 200ms p99 latency SLO, Visibility availability, and measured latency for starting and signaling Workflows.

> **📝 Note:**
>
> This page describes Temporal's current operational practices and does not create any additional commitment, representation, or warranty.
>

Temporal maintains multiple internal [service level objectives](https://en.wikipedia.org/wiki/Service-level_objective) (SLOs) and an internal alerting system based on those SLOs.
The alerting system accounts for all errors, not only the service errors that count against the [Temporal Cloud SLA](https://temporal.io/sla), and also monitors latency.
Temporal personnel receive an alert when an SLO is not being met, and on-call engineers are paged, which often means that issues are resolved before you notice them.

For current system status and recent incidents, see [Temporal Status](https://status.temporal.io).
For the regions you can run in, see [Service regions](/evaluate/cloud/regions); for the full set of rate, resource, and configuration limits, see [System limits](/evaluate/cloud/limits).

## Service levels at a glance 

| Measure                                                 | Applies to                                                                                                                                      | Target                                                | Type            |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | --------------- |
| [Uptime](https://temporal.io/sla)                       | Covered APIs, per Namespace                                                                                                                     | See the [Temporal Cloud SLA](https://temporal.io/sla) | Contractual SLA |
| [Uptime](#uptime)                                       | All Namespaces                                                                                                                                  | 99.99%                                                | SLO             |
| [Latency](#latency)                                     | Worker requests, p99 per region                                                                                                                 | 200ms                                                 | SLO             |
| [Visibility API availability](#visibility-availability) | List and Count requests, rolling 30-day window                                                                                                  | 99.5%                                                 | SLO             |
| [Cloud Ops API availability](/ops#availability)         | HTTP and gRPC requests, monthly                                                                                                                 | 99.99%                                                | SLO             |
| [Recovery Time Objective (RTO)](/cloud/rpo-rto)         | Namespaces using High Availability with automatic failover, for the [outage types their replication covers](/cloud/rpo-rto#rto-and-rpo-summary) | Under 20 minutes                                      | Objective       |
| [Recovery Point Objective (RPO)](/cloud/rpo-rto)        | Namespaces using High Availability with automatic failover, for the [outage types their replication covers](/cloud/rpo-rto#rto-and-rpo-summary) | Under 1 minute                                        | Objective       |

## Uptime 

Temporal Cloud operates every Namespace to a 99.99% [service availability](https://en.wikipedia.org/wiki/Reliability,_availability_and_serviceability) objective, whether or not the Namespace uses [High Availability](/cloud/high-availability).

The contractual uptime commitment is the [Temporal Cloud Service Level Agreement (SLA)](https://temporal.io/sla).
It defines which APIs are covered, how availability is calculated, what is excluded, and the service credits that apply.
Namespaces using High Availability have a higher SLA than standard single-region Namespaces.

Temporal Cloud runs a cell architecture.
Each cell holds the software and services needed to host a Namespace, and the components in a cell are distributed across at least three availability zones per region.
For how Temporal Cloud handles each type of outage, see [Outages and Recovery Objectives](/cloud/rpo-rto).

## Latency 

Temporal Cloud has a p99 latency SLO of 200ms per region.

That SLO covers normal Worker requests, meaning commands and polling, and it applies to Nexus in both the caller and handler Namespaces.

### Measured latency 

Measured latency over a week-long period for starting and signaling Workflow Executions, and for starting [Standalone Activities](/standalone-activity) (`StartActivityExecution`),
was as follows:

#### August 2026

| Operation                          | p50  | p90  |   p99 |
| :--------------------------------- | :--: | :--: | ----: |
| `StartWorkflowExecution`           | 20ms | 32ms |  78ms |
| `SignalWorkflowExecution`          | 19ms | 42ms |  91ms |
| `SignalWithStartWorkflowExecution` | 30ms | 47ms | 109ms |
| `StartActivityExecution`           | 13ms | 19ms |  45ms |

#### Earlier measurements

**January 2026**

| Operation                          | p50  | p90  |  p99 |
| :--------------------------------- | :--: | :--: | ---: |
| `StartWorkflowExecution`           | 14ms | 21ms | 69ms |
| `SignalWorkflowExecution`          | 11ms | 19ms | 46ms |
| `SignalWithStartWorkflowExecution` | 19ms | 37ms | 95ms |

**March 2024**

| Operation                          | p90  |  p99 |
| :--------------------------------- | :--: | ---: |
| `StartWorkflowExecution`           | 24ms | 54ms |
| `SignalWorkflowExecution`          | 14ms | 40ms |
| `SignalWithStartWorkflowExecution` | 24ms | 61ms |

Latency observed from the Temporal Client also reflects other components in the path, such as the Codec Server, an egress proxy, and the network itself.
Concurrent operations on the same Workflow Execution can raise latency as well.

### Custom persistence layer

Temporal Cloud runs a custom persistence layer rather than the persistence stores available to self-hosted deployments.
Three components of that layer account for most of the difference in latency:

- **Sharding:** Distributes load across multiple databases and resizes them independently, so a traffic spike in one shard doesn't become a bottleneck for the rest.
- **Write-ahead log (WAL):** Batches updates in an append-only log before writing them to the database, which reduces write latency and database size.
- **Tiered storage of Event History:** Moves the Event History of closed Workflow Executions to cheaper storage, which keeps the primary database smaller and faster for running Executions.

## Visibility API availability 

Temporal Cloud targets 99.5% availability for Visibility read requests, measured over a rolling 30-day window.
This covers List and Count operations against Workflow Executions, Activity Executions, Nexus Operations, Schedules, and Batch Operations.

This target is deliberately lower than the availability guaranteed for core Workflow Service APIs.
Visibility is a search index intended for operational discovery, not a critical path for application logic.

This is a service level objective rather than a contractual commitment.
Visibility API requests are not covered by the [Temporal Cloud SLA](https://temporal.io/sla), and no service credits are associated with this objective.

The objective measures whether a request returns a valid response, not how current the results are.
For what it excludes and how to interpret it alongside Visibility's eventual consistency, see [Visibility availability in Temporal Cloud](/visibility#availability).

## Throughput 

Each Namespace has a rate limit measured in [Actions](/cloud/pricing#action) per second, and two capacity modes control it:

- **On-Demand Capacity** raises the Namespace limit automatically as usage grows.
- **Provisioned Capacity** sets the limit from the Temporal Resource Units (TRUs) you request, which suits predictable spikes and guaranteed throughput.

See [Capacity Modes](/cloud/capacity-modes) for how each mode sets your limit.

## Related

- [Temporal Cloud SLA](https://temporal.io/sla): The contractual uptime commitment, including covered APIs, calculation, exclusions, and service credits.
- [Outages and Recovery Objectives (RTO / RPO)](/cloud/rpo-rto): Recovery targets for each type of outage.
- [High Availability](/cloud/high-availability): Replication and failover options for a higher contractual SLA.
- [Higher throughput and lower latency: Temporal Cloud's custom persistence layer](https://temporal.io/blog/higher-throughput-and-lower-latency-temporal-clouds-custom-persistence-layer): How sharding, the write-ahead log, and tiered storage are built.
- [Benchmarking latency: Temporal Cloud versus self-hosted Temporal](https://temporal.io/blog/benchmarking-latency-temporal-cloud-vs-self-hosted-temporal): Measured comparison of the two deployment options.
- [Replay conference talk: Custom persistence layer](https://www.youtube.com/watch?v=SQv9ot-jB6o&list=PLl9kRkvFJrlREHL7fiEKBWTp5QuFeYS2r&index=5): Walkthrough of the persistence design.

---

# tcld command reference

Source: https://docs.temporal.io/cloud/tcld

> The Temporal Cloud CLI (tcld) is a command-line tool for interacting with Temporal Cloud, offering commands for account management, login, namespace, and more. Install via Homebrew or build from source.

The Temporal Cloud CLI (tcld) is a command-line tool that you can use to interact with Temporal Cloud.

> **💡 Tip:**
>
> The Temporal CLI also manages Temporal Cloud through its
> [Temporal Cloud extension](/cli/cloud), which adds `temporal cloud` commands for Namespaces, users, API keys, and Nexus
> Endpoints. See the [`temporal cloud` command reference](/cli/command-reference/cloud) for the equivalent commands.
>

- [How to install tcld](#install-tcld)

### tcld commands

- [tcld account](/cloud/tcld/account)
- [tcld apikey](/cloud/tcld/apikey)
- [tcld connectivity-rule](/cloud/tcld/connectivity-rule)
- [tcld feature](/cloud/tcld/feature)
- [tcld generate-certificates](/cloud/tcld/generate-certificates)
- [tcld login](/cloud/tcld/login)
- [tcld logout](/cloud/tcld/logout)
- [tcld migration](/cloud/tcld/migration)
- [tcld namespace](/cloud/tcld/namespace)
- [tcld nexus](/cloud/tcld/nexus)
- [tcld request](/cloud/tcld/request)
- [tcld service-account](/cloud/tcld/service-account)
- [tcld user](/cloud/tcld/user)
- [tcld user-group](/cloud/tcld/user-group)
- [tcld version](/cloud/tcld/version)

### Global modifiers

#### --auto_confirm

Automatically confirm all prompts.

You can specify the value for this modifier by setting the AUTO_CONFIRM environment variable.
The default value is `false`.

## How to install tcld 

You can install [tcld](/cloud/tcld) in two ways.

### Install tcld by using Homebrew

```bash
brew install temporalio/brew/tcld
```

### Build tcld from source

1. Verify that you have the minimum Go version listed in the tcld
   [`go.mod`](https://github.com/temporalio/tcld/blob/main/go.mod) file.

   ```bash
   go version
   ```

   If that version of Go is not installed, follow the [Download and install](https://go.dev/doc/install) instructions on the Go website.

1. Clone the tcld repository and run make.

   ```bash
   git clone https://github.com/temporalio/tcld.git
   cd tcld
   make
   ```

1. Copy the tcld executable to any directory that appears in the PATH environment variable, such as `/usr/local/bin`.

   ```bash
   cp tcld /usr/local/bin/tcld
   ```

1. Verify that tcld is installed.

   ```bash
   tcld version
   ```

---

# tcld account command reference

Source: https://docs.temporal.io/cloud/tcld/account

> Account operations

`tcld account`: Account operations.

Alias: `a`

- [tcld account get](#get)
- [tcld account list-regions](#list-regions)
- [tcld account audit-log](#audit-log)

### get

`tcld account get`: Get account information.

Alias: `g`

### list-regions

`tcld account list-regions`: Lists all regions where the account can provision namespaces.

Alias: `l`

### audit-log

`tcld account audit-log`: audit log commands.

Alias: `al`

- [tcld account audit-log kinesis](#kinesis)
- [tcld account audit-log pubsub](#pubsub)

#### kinesis

`tcld account audit-log kinesis`: Manage Kinesis audit log sink.

Alias: `k`

- [tcld account audit-log kinesis create](#create)
- [tcld account audit-log kinesis validate](#validate)
- [tcld account audit-log kinesis update](#update)
- [tcld account audit-log kinesis get](#get)
- [tcld account audit-log kinesis delete](#delete)
- [tcld account audit-log kinesis list](#list)

##### create

`tcld account audit-log kinesis create`: Create a kinesis audit log sink.

Alias: `c`

###### --sink-name

Provide a name for the sink

###### --role-name

The role name to use to write to the sink

Alias: `rn`

###### --destination-uri

The destination URI of the audit log sink

Alias: `du`

###### --region

The region to use for the request

Alias: `re`

##### validate

`tcld account audit-log kinesis validate`: Validate kinesis audit log sink.

Alias: `v`

###### --sink-name

Provide a name for the sink

###### --role-name

The role name to use to write to the sink

Alias: `rn`

###### --destination-uri

The destination URI of the audit log sink

Alias: `du`

###### --region

The region to use for the request

Alias: `re`

##### update

`tcld account audit-log kinesis update`: Update a kinesis audit log sink.

Alias: `u`

###### --sink-name

Provide a name for the sink

###### --enabled

Whether the sink is enabled

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --role-name

The role name to use to write to the sink

Alias: `rn`

###### --destination-uri

The destination URI of the audit log sink

Alias: `du`

###### --region

The region to use for the request

Alias: `re`

##### get

`tcld account audit-log kinesis get`: Get audit log sink.

Alias: `g`

###### --sink-name

Provide a name for the sink

##### delete

`tcld account audit-log kinesis delete`: Delete audit log sink.

Alias: `d`

###### --sink-name

Provide a name for the sink

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### list

`tcld account audit-log kinesis list`: List audit log sinks.

Alias: `l`

###### --page-size

The page size for list operations

###### --page-token

The page token for list operations

#### pubsub

`tcld account audit-log pubsub`: Manage PubSub audit log sink.

Alias: `ps`

- [tcld account audit-log pubsub create](#create)
- [tcld account audit-log pubsub validate](#validate)
- [tcld account audit-log pubsub update](#update)
- [tcld account audit-log pubsub get](#get)
- [tcld account audit-log pubsub delete](#delete)
- [tcld account audit-log pubsub list](#list)

##### create

`tcld account audit-log pubsub create`: Create a pubsub audit log sink.

Alias: `c`

###### --sink-name

Provide a name for the sink

###### --service-account-email

The service account email to impersonate to write to the sink

Alias: `sae`

###### --topic-name

The topic name to write to the sink

Alias: `tn`

##### validate

`tcld account audit-log pubsub validate`: Validate pubsub audit log sink.

Alias: `v`

###### --sink-name

Provide a name for the sink

###### --service-account-email

The service account email to impersonate to write to the sink

Alias: `sae`

###### --topic-name

The topic name to write to the sink

Alias: `tn`

##### update

`tcld account audit-log pubsub update`: Update a pubsub audit log sink.

Alias: `u`

###### --sink-name

Provide a name for the sink

###### --enabled

Whether the sink is enabled

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --service-account-email

The service account email to impersonate to write to the sink

Alias: `sae`

###### --topic-name

The topic name to write to the sink

Alias: `tn`

##### get

`tcld account audit-log pubsub get`: Get audit log sink.

Alias: `g`

###### --sink-name

Provide a name for the sink

##### delete

`tcld account audit-log pubsub delete`: Delete audit log sink.

Alias: `d`

###### --sink-name

Provide a name for the sink

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### list

`tcld account audit-log pubsub list`: List audit log sinks.

Alias: `l`

###### --page-size

The page size for list operations

###### --page-token

The page token for list operations

---

# tcld apikey command reference

Source: https://docs.temporal.io/cloud/tcld/apikey

> APIKey operations

`tcld apikey`: APIKey operations.

Alias: `ak`

- [tcld apikey create](#create)
- [tcld apikey get](#get)
- [tcld apikey list](#list)
- [tcld apikey delete](#delete)
- [tcld apikey disable](#disable)
- [tcld apikey enable](#enable)

### create

`tcld apikey create`: Create an apikey. Make sure to copy the secret or else you will not be able to retrieve it again.

Alias: `c`

#### --name

the display name of the apikey

Alias: `n`

#### --description

the description of the apikey

Alias: `desc`

#### --duration

the duration from now when the apikey will expire, will be ignored if expiry flag is set, examples: '1.5y', '30d', '4d12h'

Alias: `d`

#### --expiry

Alias: `e`

#### --service-account-id

setting this flag will create an api key for a service account, not a user

Alias: `si`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### get

`tcld apikey get`: Get an apikey.

Alias: `g`

#### --id

The id of the apikey to get

Alias: `i`

### list

`tcld apikey list`: List API keys.

Alias: `l`

#### --owner-id

Filter API keys by owner ID

Alias: `oid`

#### --owner-type

Filter API keys by owner type (that is, 'user', 'service-account')

Alias: `ot`

### delete

`tcld apikey delete`: Delete an apikey.

Alias: `d`

#### --id

The id of the apikey to delete

Alias: `i`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### disable

`tcld apikey disable`: Disable an apikey.

Alias: `da`

#### --id

The id of the apikey to disable

Alias: `i`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### enable

`tcld apikey enable`: Enable a disabled apikey.

Alias: `ea`

#### --id

The id of the apikey to enable

Alias: `i`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

---

# tcld connectivity-rule command reference

Source: https://docs.temporal.io/cloud/tcld/connectivity-rule

> Connectivity rule operations

`tcld connectivity-rule`: Connectivity rule operations.

Alias: `cr`

- [tcld connectivity-rule create](#create)
- [tcld connectivity-rule get](#get)
- [tcld connectivity-rule list](#list)
- [tcld connectivity-rule delete](#delete)

### create

`tcld connectivity-rule create`: Create a connectivity rule.

Alias: `c`

#### --connectivity-type

The type of connectivity, currently only support 'private' and 'public'

Alias: `ct`

#### --connection-id

The connection ID of the private connection

Alias: `ci`

#### --region

The region of the connection

Alias: `r`

#### --gcp-project-id

The GCP project ID of the connection, required if the cloud provider is 'gcp'

Alias: `gpi`

### get

`tcld connectivity-rule get`: Get a connectivity rule.

Alias: `g`

#### --connectivity-rule-id

The connectivity rule ID

Alias: `id`

### list

`tcld connectivity-rule list`: list connectivity rules.

Alias: `l`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

### delete

`tcld connectivity-rule delete`: Delete a connectivity rule.

Alias: `d`

#### --connectivity-rule-id

The connectivity rule ID

Alias: `id`

---

# tcld feature command reference

Source: https://docs.temporal.io/cloud/tcld/feature

> feature commands

`tcld feature`: feature commands.

Alias: `f`

- [tcld feature get](#get)

### get

`tcld feature get`: get all feature flags Value.

Alias: `g`

---

# tcld generate-certificates command reference

Source: https://docs.temporal.io/cloud/tcld/generate-certificates

> Commands for generating certificate authority and end-entity TLS certificates

`tcld generate-certificates`: Commands for generating certificate authority and end-entity TLS certificates.

Alias: `gen`

- [tcld generate-certificates certificate-authority-certificate](#certificate-authority-certificate)
- [tcld generate-certificates end-entity-certificate](#end-entity-certificate)

### certificate-authority-certificate

`tcld generate-certificates certificate-authority-certificate`: Generate a certificate authority certificate.

Alias: `ca`

#### --organization

The name of the organization

Alias: `org`

#### --validity-period

The duration for which the certificate is valid for. example: 30d10h (30 days and 10 hrs)

Alias: `d`

#### --ca-certificate-file

The path where the generated x509 certificate will be stored

Alias: `ca-cert`

#### --ca-key-file

The path where the certificate's private key will be stored

Alias: `ca-key`

#### --rsa-algorithm

Generates a 4096-bit RSA keypair instead of an ECDSA P-384 keypair (the recommended default) for the certificate (optional)

Alias: `rsa`

### end-entity-certificate

`tcld generate-certificates end-entity-certificate`: Generate an end-entity certificate.

Alias: `leaf`

#### --organization

The name of the organization

Alias: `org`

#### --organization-unit

The name of the organizational unit (optional)

#### --common-name

The common name (optional)

#### --validity-period

The duration for which the end entity certificate is valid for. example: 30d10h (30 days and 10 hrs). By default the generated certificate expires 24 hours before the certificate authority expires (optional)

Alias: `d`

#### --ca-certificate-file

The path of the x509 certificate for the certificate authority

Alias: `ca-cert`

#### --ca-key-file

The path of the private key for the certificate authority

Alias: `ca-key`

#### --certificate-file

The path where the generated x509 certificate will be stored

Alias: `cert`

#### --key-file

The path where the certificate's private key will be stored

Alias: `key`

---

# tcld login command reference

Source: https://docs.temporal.io/cloud/tcld/login

> Login as user

`tcld login`: Login as user.

Alias: `l`

### --disable-pop-up

disable browser pop-up

---

# tcld logout command reference

Source: https://docs.temporal.io/cloud/tcld/logout

> Logout current user

`tcld logout`: Logout current user.

Alias: `lo`

### --disable-pop-up

disable browser pop-up

---

# tcld migration command reference

Source: https://docs.temporal.io/cloud/tcld/migration

> (private preview) Manage migrations between self-hosted Temporal and Temporal cloud

`tcld migration`: (private preview) Manage migrations between self-hosted Temporal and Temporal cloud.

Alias: `m`

- [tcld migration get](#get)
- [tcld migration list](#list)
- [tcld migration start](#start)
- [tcld migration handover](#handover)
- [tcld migration confirm](#confirm)
- [tcld migration abort](#abort)

### get

`tcld migration get`: Get a migration.

Alias: `g`

#### --id

Migration id

Alias: `i`

### list

`tcld migration list`: List migrations.

Alias: `l`

### start

`tcld migration start`: Start a new migration.

Alias: `s`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --endpoint-id

Migration endpoint id

Alias: `e`

#### --source-namespace

Source namespace name

Alias: `s`

#### --target-namespace

Target namespace name

Alias: `t`

### handover

`tcld migration handover`: Handover the namespace from on-prem to cloud, or from cloud back to on-prem.

Alias: `s`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --id

Migration id

Alias: `i`

#### --to-replica-id

The id of the replica to make active

Alias: `rp`

### confirm

`tcld migration confirm`: Confirm the migration.

Alias: `c`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --id

Migration id

Alias: `i`

### abort

`tcld migration abort`: Abort the migration.

Alias: `a`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --id

Migration id

Alias: `i`

---

# tcld namespace command reference

Source: https://docs.temporal.io/cloud/tcld/namespace

> Namespace operations

`tcld namespace`: Namespace operations.

Alias: `n`

- [tcld namespace create](#create)
- [tcld namespace add-region](#add-region)
- [tcld namespace delete-region](#delete-region)
- [tcld namespace lifecycle](#lifecycle)
- [tcld namespace delete](#delete)
- [tcld namespace list](#list)
- [tcld namespace get](#get)
- [tcld namespace accepted-client-ca](#accepted-client-ca)
- [tcld namespace auth-method](#auth-method)
- [tcld namespace certificate-filters](#certificate-filters)
- [tcld namespace update-codec-server](#update-codec-server)
- [tcld namespace retention](#retention)
- [tcld namespace search-attributes](#search-attributes)
- [tcld namespace failover](#failover)
- [tcld namespace update-high-availability](#update-high-availability)
- [tcld namespace tags](#tags)
- [tcld namespace capacity](#capacity)
- [tcld namespace export](#export)
- [tcld namespace set-connectivity-rules](#set-connectivity-rules)

### create

`tcld namespace create`: Create a temporal namespace.

Alias: `c`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --ca-certificate

The base64 encoded ca certificate

Alias: `c`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### --region

Create namespace in specified regions; if multiple regions are selected, the first one will be the active region. See 'tcld account list-regions' to get a list of available regions for your account

Alias: `re`

#### --retention-days

The retention of the namespace in days

Alias: `rd`

#### --auth-method

The authentication method to use for the namespace (for example, 'mtls', 'api_key')

#### --ca-certificate-file

The path to the ca pem file

Alias: `cf`

#### --certificate-filter-file

Path to a JSON file that defines the certificate filters that will be added to the namespace. Sample JSON: &#123; "filters": [ &#123; "commonName": "test1" &#125; ] &#125;

Alias: `cff`

#### --certificate-filter-input

JSON that defines the certificate filters that will be added to the namespace. Sample JSON: &#123; "filters": [ &#123; "commonName": "test1" &#125; ] &#125;

Alias: `cfi`

#### --search-attribute

Flag can be used multiple times; value must be "name=type"; valid types are: [Keyword Text Int Double Datetime Bool KeywordList]

Alias: `sa`

#### --user-namespace-permission

Flag can be used multiple times; value must be "email=permission"; valid permissions are: [Admin Write Read]

Alias: `p`

#### --enable-delete-protection

Enable delete protection on the namespace

Alias: `edp`

#### --endpoint

The codec server endpoint to decode payloads for all users interacting with this Namespace, must be https

Alias: `e`

#### --pass-access-token

Pass the user access token to the remote endpoint

Alias: `pat`

#### --include-credentials

Include cross-origin credentials

Alias: `ic`

#### --cloud-provider

Cloud provider for the namespace to be created for, currently support [aws, gcp].  For this version, if not specified, we default to aws

Alias: `cp`

#### --tag

Add tags to the namespace (format: key=value). Flag can be used multiple times.

Alias: `t`

#### --connectivity-rule-ids

The list of connectivity rule IDs, can be used in create namespace and update namespace. example: --ids id1 --ids id2 --ids id3

Alias: `ids`

### add-region

`tcld namespace add-region`: Add a new region to a namespace.

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### --region

New region to add to the namespace.

Alias: `re`

#### --cloud-provider

The cloud provider of the region. Default: aws

### delete-region

`tcld namespace delete-region`: Delete a region from a namespace.

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### --region

The region to remove from a namespace.

Alias: `re`

#### --cloud-provider

The cloud provider of the region. Default: aws

### lifecycle

`tcld namespace lifecycle`: Enable delete protection on a temporal namespace.

Alias: `lc`

- [tcld namespace lifecycle get](#get)
- [tcld namespace lifecycle set](#set)

#### get

`tcld namespace lifecycle get`: Get the lifecycle spec for the namespace.

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### set

`tcld namespace lifecycle set`: Set the lifecycle spec for the namespace.

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --enable-delete-protection

Enable delete protection on the namespace, value must be true or false

Alias: `edp`

### delete

`tcld namespace delete`: Delete a temporal namespace.

Alias: `d`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

### list

`tcld namespace list`: List all known namespaces.

Alias: `l`

#### --page-token

The page token for list operations

#### --page-size

Number of namespaces to list per page

### get

`tcld namespace get`: Get namespace information.

Alias: `g`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

### accepted-client-ca

`tcld namespace accepted-client-ca`: Manage client ca certificate used to verify client connections.

Alias: `ca`

- [tcld namespace accepted-client-ca list](#list)
- [tcld namespace accepted-client-ca add](#add)
- [tcld namespace accepted-client-ca remove](#remove)
- [tcld namespace accepted-client-ca set](#set)

#### list

`tcld namespace accepted-client-ca list`: List the accepted client ca certificates currently configured for the namespace.

Alias: `l`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### add

`tcld namespace accepted-client-ca add`: Add a new ca accepted client ca certificate.

Alias: `a`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --ca-certificate

The base64 encoded ca certificate

Alias: `c`

##### --ca-certificate-file

The path to the ca pem file

Alias: `f`

#### remove

`tcld namespace accepted-client-ca remove`: Remove existing certificates.

Alias: `r`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --ca-certificate

The base64 encoded ca certificate

Alias: `c`

##### --ca-certificate-file

The path to the ca pem file

Alias: `f`

##### --ca-certificate-fingerprint

The fingerprint of to the ca certificate

Alias: `fp`

##### --all

If set, all existing certificates will be removed

#### set

`tcld namespace accepted-client-ca set`: Set the accepted client ca certificate.

Alias: `s`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --ca-certificate

The base64 encoded ca certificate

Alias: `c`

##### --ca-certificate-file

The path to the ca pem file

Alias: `f`

### auth-method

`tcld namespace auth-method`: Manage the authentication method for the namespace.

Alias: `am`

- [tcld namespace auth-method set](#set)
- [tcld namespace auth-method get](#get)

#### set

`tcld namespace auth-method set`: Set the authentication method for the namespace.

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --auth-method

The authentication method used for the namespace (that is, 'restricted', 'mtls', 'api_key', 'api_key_or_mtls')

Alias: `am`

#### get

`tcld namespace auth-method get`: Retrieve the authentication method for namespace.

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

### certificate-filters

`tcld namespace certificate-filters`: Manage optional certificate filters used by namespace to authorize client certificates based on distinguished name fields.

Alias: `cf`

- [tcld namespace certificate-filters import](#import)
- [tcld namespace certificate-filters export](#export)
- [tcld namespace certificate-filters clear](#clear)
- [tcld namespace certificate-filters add](#add)

#### import

`tcld namespace certificate-filters import`: Sets the certificate filters on the namespace. Existing filters will be replaced.

Alias: `imp`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --certificate-filter-file

Path to a JSON file that defines the certificate filters that will be configured on the namespace. This will replace the existing filter configuration. Sample JSON: &#123; "filters": [ &#123; "commonName": "test1" &#125; ] &#125;

Alias: `file`, `f`

##### --certificate-filter-input

JSON that defines the certificate filters that will be configured on the namespace. This will replace the existing filter configuration. Sample JSON: &#123; "filters": [ &#123; "commonName": "test1" &#125; ] &#125;

Alias: `input`, `i`

#### export

`tcld namespace certificate-filters export`: Exports existing certificate filters on the namespace.

Alias: `exp`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --certificate-filter-file

Path to a JSON file where tcld will export the certificate filter configuration to

Alias: `file`, `f`

#### clear

`tcld namespace certificate-filters clear`: Clears all certificate filters on the namespace. Note that this will allow *any* client certificate that chains up to a configured CA in the bundle to connect to the namespace.

Alias: `c`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### add

`tcld namespace certificate-filters add`: Adds additional certificate filters to the namespace.

Alias: `a`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --certificate-filter-file

Path to a JSON file that defines the certificate filters that will be added to the namespace. Sample JSON: &#123; "filters": [ &#123; "commonName": "test1" &#125; ] &#125;

Alias: `file`, `f`

##### --certificate-filter-input

JSON that defines the certificate filters that will be added to the namespace. Sample JSON: &#123; "filters": [ &#123; "commonName": "test1" &#125; ] &#125;

Alias: `input`, `i`

### update-codec-server

`tcld namespace update-codec-server`: Update codec server config used to decode encoded payloads through remote endpoint.

Alias: `ucs`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### --endpoint

The codec server endpoint to decode payloads for all users interacting with this Namespace, must be https

Alias: `e`

#### --pass-access-token

Pass the user access token to the remote endpoint

Alias: `pat`

#### --include-credentials

Include cross-origin credentials

Alias: `ic`

### retention

`tcld namespace retention`: Manages configuration of the length of time (in days) a closed workflow will be preserved before deletion.

Alias: `r`

- [tcld namespace retention set](#set)
- [tcld namespace retention get](#get)

#### set

`tcld namespace retention set`: Set the length of time (in days) a closed workflow will be preserved before deletion for a given namespace.

Alias: `s`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --retention-days

The retention of the namespace in days

Alias: `rd`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### get

`tcld namespace retention get`: Retrieve the length of time (in days) a closed workflow will be preserved before deletion for a given namespace.

Alias: `g`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

### search-attributes

`tcld namespace search-attributes`: Manage search attributes used by namespace.

Alias: `sa`

- [tcld namespace search-attributes add](#add)
- [tcld namespace search-attributes remove](#remove)
- [tcld namespace search-attributes rename](#rename)

#### add

`tcld namespace search-attributes add`: Add a new namespace custom search attribute.

Alias: `a`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --search-attribute

Flag can be used multiple times; value must be "name=type"; valid types are: [Keyword Text Int Double Datetime Bool KeywordList]

Alias: `sa`

#### remove

`tcld namespace search-attributes remove`: Remove an existing namespace custom search attribute.

Alias: `rm`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --search-attribute

The name of the search attribute to remove

Alias: `sa`

#### rename

`tcld namespace search-attributes rename`: Update the name of an existing custom search attribute.

Alias: `rn`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --existing-name

The name of an existing search attribute

Alias: `en`

##### --new-name

The new name for the search attribute

Alias: `nn`

### failover

`tcld namespace failover`: Failover a temporal namespace.

Alias: `fo`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### --region

The region to failover to

Alias: `re`

#### --cloud-provider

The cloud provider of the region. Default: aws

### update-high-availability

`tcld namespace update-high-availability`: Update Temporal namespace high availability setting.

Alias: `uha`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### --disable-auto-failover

Disable Temporal-managed failover on a replicated namespace (use --disable-auto-failover=false to enable)

### tags

`tcld namespace tags`: Manage namespace tags.

Alias: `t`

- [tcld namespace tags upsert](#upsert)
- [tcld namespace tags remove](#remove)

#### upsert

`tcld namespace tags upsert`: Add new tags or update existing tag values.

Alias: `u`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --tag

Add new or update existing namespace tags (format: key=value). Flag can be used multiple times.

Alias: `t`

#### remove

`tcld namespace tags remove`: Remove existing tags by key.

Alias: `rm`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --tag-key

Remove namespace tags by key. Flag can be used multiple times.

Alias: `tk`

### capacity

`tcld namespace capacity`: Manage namespace capacity.

Alias: `cap`

- [tcld namespace capacity get](#get)
- [tcld namespace capacity update](#update)

#### get

`tcld namespace capacity get`: Get namespace capacity information.

Alias: `g`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### update

`tcld namespace capacity update`: Set the capacity of a given namespace.

Alias: `u`

##### --namespace

The namespace hosted on temporal cloud

Alias: `n`

##### --capacity-mode

The capacity mode to use for the namespace. Valid values are 'on_demand' and 'provisioned'

Alias: `cm`

##### --capacity-value

The capacity value to use for the namespace. Required if capacity mode is 'provisioned', ignored otherwise

Alias: `cv`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

### export

`tcld namespace export`: Manage export.

Alias: `es`

- [tcld namespace export s3](#s3)
- [tcld namespace export gcs](#gcs)

#### s3

`tcld namespace export s3`: Manage S3 export sink.

- [tcld namespace export s3 create](#create)
- [tcld namespace export s3 validate](#validate)
- [tcld namespace export s3 update](#update)
- [tcld namespace export s3 get](#get)
- [tcld namespace export s3 delete](#delete)
- [tcld namespace export s3 list](#list)

##### create

`tcld namespace export s3 create`: Create export sink.

Alias: `c`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --role-arn

Provide role arn for the IAM Role

###### --s3-bucket-name

Provide the name of an AWS S3 bucket that Temporal will send closed workflow histories to

###### --kms-arn

Provide the ARN of the KMS key to use for encryption. Note: If the KMS ARN needs to be added or updated, user must create the IAM Role with KMS or modify the created IAM Role accordingly.

###### --region

The region to use for the request, if not set the server will use the namespace's region

Alias: `re`

##### validate

`tcld namespace export s3 validate`: Validate export sink.

Alias: `v`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --role-arn

Provide role arn for the IAM Role

###### --s3-bucket-name

Provide the name of an AWS S3 bucket that Temporal will send closed workflow histories to

###### --kms-arn

Provide the ARN of the KMS key to use for encryption. Note: If the KMS ARN needs to be added or updated, user must create the IAM Role with KMS or modify the created IAM Role accordingly.

###### --region

The region to use for the request, if not set the server will use the namespace's region

Alias: `re`

##### update

`tcld namespace export s3 update`: Update export sink.

Alias: `u`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --enabled

Whether the sink is enabled

###### --role-arn

Provide role arn for the IAM Role

###### --s3-bucket-name

Provide the name of an AWS S3 bucket that Temporal will send closed workflow histories to

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --kms-arn

Provide the ARN of the KMS key to use for encryption. Note: If the KMS ARN needs to be added or updated, user must create the IAM Role with KMS or modify the created IAM Role accordingly.

##### get

`tcld namespace export s3 get`: Get export sink.

Alias: `g`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

##### delete

`tcld namespace export s3 delete`: Delete export sink.

Alias: `d`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### list

`tcld namespace export s3 list`: List export sinks.

Alias: `l`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --page-size

The page size for list operations

###### --page-token

The page token for list operations

#### gcs

`tcld namespace export gcs`: Manage GCS export sink.

- [tcld namespace export gcs create](#create)
- [tcld namespace export gcs update](#update)
- [tcld namespace export gcs validate](#validate)
- [tcld namespace export gcs get](#get)
- [tcld namespace export gcs delete](#delete)
- [tcld namespace export gcs list](#list)

##### create

`tcld namespace export gcs create`: Create export sink.

Alias: `c`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --service-account-email

Service account that has access to the sink

###### --gcs-bucket

GCS bucket of the sink

##### update

`tcld namespace export gcs update`: Update export sink.

Alias: `u`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --enabled

Whether the sink is enabled

###### --service-account-email

Service account that has access to the sink

###### --gcs-bucket

GCS bucket of the sink

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### validate

`tcld namespace export gcs validate`: Validate export sink.

Alias: `v`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --service-account-email

Service account that has access to the sink

###### --gcs-bucket

GCS bucket of the sink

##### get

`tcld namespace export gcs get`: Get export sink.

Alias: `g`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

##### delete

`tcld namespace export gcs delete`: Delete export sink.

Alias: `d`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --sink-name

Provide a name for the sink

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### list

`tcld namespace export gcs list`: List export sinks.

Alias: `l`

###### --namespace

The namespace hosted on temporal cloud

Alias: `n`

###### --page-size

The page size for list operations

###### --page-token

The page token for list operations

### set-connectivity-rules

`tcld namespace set-connectivity-rules`: set the connectivity rules for a namespace.

Alias: `scrs`

#### --namespace

The namespace hosted on temporal cloud

Alias: `n`

#### --connectivity-rule-ids

The list of connectivity rule IDs, can be used in create namespace and update namespace. example: --ids id1 --ids id2 --ids id3

Alias: `ids`

#### --remove-all

Acknowledge that all connectivity rules will be removed, enabling connectivity from any source

---

# tcld nexus command reference

Source: https://docs.temporal.io/cloud/tcld/nexus

> Reference for tcld nexus commands

Alias: `nxs`

- [tcld nexus endpoint](#endpoint)

### endpoint

`tcld nexus endpoint`: Commands for managing Nexus Endpoints (EXPERIMENTAL).

Alias: `ep`

- [tcld nexus endpoint get](#get)
- [tcld nexus endpoint list](#list)
- [tcld nexus endpoint create](#create)
- [tcld nexus endpoint update](#update)
- [tcld nexus endpoint allowed-namespace](#allowed-namespace)
- [tcld nexus endpoint delete](#delete)

#### get

`tcld nexus endpoint get`: Get a Nexus Endpoint by name (EXPERIMENTAL).

Alias: `g`

##### --name

Endpoint name

Alias: `n`

#### list

`tcld nexus endpoint list`: List Nexus Endpoints (EXPERIMENTAL).

Alias: `l`

#### create

`tcld nexus endpoint create`: Create a new Nexus Endpoint (EXPERIMENTAL).

Alias: `c`

##### --name

Endpoint name

Alias: `n`

##### --description

Endpoint description in markdown format (optional)

Alias: `d`

##### --description-file

Endpoint description file in markdown format (optional)

Alias: `df`

##### --target-namespace

Namespace in which a handler worker will be polling for Nexus tasks on

Alias: `tns`

##### --target-task-queue

Task Queue in which a handler worker will be polling for Nexus tasks on

Alias: `ttq`

##### --allow-namespace

Namespace that is allowed to call this endpoint (optional)

Alias: `ans`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### update

`tcld nexus endpoint update`: Update an existing Nexus Endpoint (EXPERIMENTAL).

Alias: `u`

##### --name

Endpoint name

Alias: `n`

##### --description

Endpoint description in markdown format (optional)

Alias: `d`

##### --description-file

Endpoint description file in markdown format (optional)

Alias: `df`

##### --unset-description

Unset endpoint description

##### --target-namespace

Namespace in which a handler worker will be polling for Nexus tasks on (optional)

Alias: `tns`

##### --target-task-queue

Task Queue in which a handler worker will be polling for Nexus tasks on (optional)

Alias: `ttq`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### allowed-namespace

`tcld nexus endpoint allowed-namespace`: Allowed namespace operations for a Nexus Endpoint (EXPERIMENTAL).

Alias: `an`

- [tcld nexus endpoint allowed-namespace add](#add)
- [tcld nexus endpoint allowed-namespace list](#list)
- [tcld nexus endpoint allowed-namespace set](#set)
- [tcld nexus endpoint allowed-namespace remove](#remove)

##### add

`tcld nexus endpoint allowed-namespace add`: Add allowed namespaces to a Nexus Endpoint (EXPERIMENTAL).

Alias: `a`

###### --name

Endpoint name

Alias: `n`

###### --namespace

Namespace that is allowed to call this endpoint

Alias: `ns`

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### list

`tcld nexus endpoint allowed-namespace list`: List allowed namespaces of a Nexus Endpoint (EXPERIMENTAL).

Alias: `l`

###### --name

Endpoint name

Alias: `n`

##### set

`tcld nexus endpoint allowed-namespace set`: Set allowed namespaces of a Nexus Endpoint (EXPERIMENTAL).

Alias: `s`

###### --name

Endpoint name

Alias: `n`

###### --namespace

Namespace that is allowed to call this endpoint

Alias: `ns`

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

##### remove

`tcld nexus endpoint allowed-namespace remove`: Remove allowed namespaces from a Nexus Endpoint (EXPERIMENTAL).

Alias: `r`

###### --name

Endpoint name

Alias: `n`

###### --namespace

Namespace that is allowed to call this endpoint

Alias: `ns`

###### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

###### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### delete

`tcld nexus endpoint delete`: Delete a Nexus Endpoint (EXPERIMENTAL).

Alias: `d`

##### --name

Endpoint name

Alias: `n`

##### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

##### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

---

# tcld request command reference

Source: https://docs.temporal.io/cloud/tcld/request

> Manage asynchronous requests

`tcld request`: Manage asynchronous requests.

Alias: `r`

- [tcld request get](#get)

### get

`tcld request get`: Get the request status.

Alias: `g`

#### --request-id

The request-id of the asynchronous request

Alias: `r`

---

# tcld service-account command reference

Source: https://docs.temporal.io/cloud/tcld/service-account

> Service Account management operations

`tcld service-account`: Service Account management operations.

Alias: `sa`

- [tcld service-account create](#create)
- [tcld service-account create-scoped](#create-scoped)
- [tcld service-account list](#list)
- [tcld service-account get](#get)
- [tcld service-account update](#update)
- [tcld service-account delete](#delete)
- [tcld service-account set-account-role](#set-account-role)
- [tcld service-account set-namespace-permissions](#set-namespace-permissions)

### create

`tcld service-account create`: Create a service account.

Alias: `c`

#### --description

The service account description

Alias: `d`

#### --name

The service account name

Alias: `n`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --account-role

The account role to set on the service account; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read]

Alias: `ar`

#### --namespace-permission

Flag can be used multiple times; value must be "&lt;namespace&gt;=&lt;permission&gt;"; valid types are: [Admin Read Write]

Alias: `np`

### create-scoped

`tcld service-account create-scoped`: Create a scoped service account (service account restricted to a single namespace).

Alias: `cs`

#### --description

The service account description

Alias: `d`

#### --name

The service account name

Alias: `n`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --namespace-permission

Value must be "&lt;namespace&gt;=&lt;permission&gt;"; valid types are: [Admin Read Write]

Alias: `np`

### list

`tcld service-account list`: List service accounts.

Alias: `l`

#### --page-token

Page token for paging list service accounts request

Alias: `p`

#### --page-size

Page size for paging list service accounts request

Alias: `s`

### get

`tcld service-account get`: Get service account information.

Alias: `g`

#### --service-account-id

The service account id

Alias: `id`

### update

`tcld service-account update`: Update service account from Temporal Cloud.

Alias: `u`

#### --service-account-id

The service account id

Alias: `id`

#### --description

The service account description

Alias: `d`

#### --name

The service account name

Alias: `n`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### delete

`tcld service-account delete`: Delete service account from Temporal Cloud.

Alias: `d`

#### --service-account-id

The service account id

Alias: `id`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### set-account-role

`tcld service-account set-account-role`: Set account role for a service account.

Alias: `sar`

#### --service-account-id

The service account id

Alias: `id`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --account-role

The account role to set on the service account; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read]

Alias: `ar`

### set-namespace-permissions

`tcld service-account set-namespace-permissions`: Set entirely new set of namespace permissions for a service account.

Alias: `snp`

#### --service-account-id

The service account id

Alias: `id`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --namespace-permission

Flag can be used multiple times; value must be "namespace=permission"; valid types are: [Admin Read Write]

Alias: `p`

---

# tcld user command reference

Source: https://docs.temporal.io/cloud/tcld/user

> User management operations

`tcld user`: User management operations.

Alias: `u`

- [tcld user list](#list)
- [tcld user get](#get)
- [tcld user invite](#invite)
- [tcld user resend-invite](#resend-invite)
- [tcld user delete](#delete)
- [tcld user set-account-role](#set-account-role)
- [tcld user set-namespace-permissions](#set-namespace-permissions)

### list

`tcld user list`: List users.

Alias: `l`

#### --namespace

List users that have permissions to the namespace

Alias: `n`

#### --page-token

Page token for paging list users request

Alias: `p`

#### --page-size

Page size for paging list users request

Alias: `s`

### get

`tcld user get`: Get user information.

Alias: `g`

#### --user-id

The user id

Alias: `id`

#### --user-email

The user email address of the user

Alias: `e`

### invite

`tcld user invite`: Invite users to Temporal Cloud.

Alias: `i`

#### --user-email

The email address of the user, you can supply this flag multiple times to invite multiple users in a single request

Alias: `e`

#### --account-role

The account role to set on the user; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read]

Alias: `ar`

#### --namespace-permission

Flag can be used multiple times; value must be "namespace=permission"; valid types are: [Admin Read Write]

Alias: `p`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### resend-invite

`tcld user resend-invite`: Resend invitation to a user on Temporal Cloud.

Alias: `ri`

#### --user-id

The user id

Alias: `id`

#### --user-email

The user email address of the user

Alias: `e`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### delete

`tcld user delete`: Delete user from Temporal Cloud.

Alias: `d`

#### --user-id

The user id

Alias: `id`

#### --user-email

The user email address of the user

Alias: `e`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### set-account-role

`tcld user set-account-role`: Set account role for a user.

Alias: `sar`

#### --user-id

The user id

Alias: `id`

#### --user-email

The user email address of the user

Alias: `e`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --account-role

The account role to set on the user; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read]

Alias: `ar`

### set-namespace-permissions

`tcld user set-namespace-permissions`: Set entirely new set of namespace permissions for a user.

Alias: `snp`

#### --user-id

The user id

Alias: `id`

#### --user-email

The user email address of the user

Alias: `e`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

#### --resource-version

The resource-version (etag) to update from, if not set the cli will use the latest (optional)

Alias: `v`

#### --namespace-permission

Flag can be used multiple times; value must be "namespace=permission"; valid types are: [Admin Read Write]

Alias: `p`

---

# tcld user-group command reference

Source: https://docs.temporal.io/cloud/tcld/user-group

> User group management operations

`tcld user-group`: User group management operations.

Alias: `ug`

- [tcld user-group list](#list)
- [tcld user-group get](#get)
- [tcld user-group create](#create)
- [tcld user-group set-access](#set-access)
- [tcld user-group add-users](#add-users)
- [tcld user-group remove-users](#remove-users)
- [tcld user-group list-members](#list-members)
- [tcld user-group delete](#delete)

### list

`tcld user-group list`: List groups.

Alias: `l`

#### --page-token

list groups starting from this page token

Alias: `p`

#### --page-size

number of groups to list

Alias: `s`

### get

`tcld user-group get`: Get group.

Alias: `g`

#### --group-id

group ID

Alias: `id`

### create

`tcld user-group create`: Create a new user group.

Alias: `c`

#### --display-name

display name for the group

#### --account-role

account role (admin, read, developer, owner, financeadmin, none)

#### --namespace-role

namespace roles

Alias: `nr`

### set-access

`tcld user-group set-access`: Set group access.

Alias: `sa`

#### --group-id

group ID

Alias: `id`

#### --account-role

account role

Alias: `ar`

#### --namespace-role

namespace roles

Alias: `nr`

#### --append

append namespace roles, cannot be used with remove flag, cannot set account role

Alias: `a`

#### --remove

remove namespace roles, cannot be used with append flag, cannot set account role

Alias: `r`

### add-users

`tcld user-group add-users`: Add users to a group.

Alias: `au`

#### --group-id

group ID

Alias: `id`

#### --user-email

The email address of the user, you can supply this flag multiple times to add multiple users in a single request

Alias: `e`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### remove-users

`tcld user-group remove-users`: Remove users from a group.

Alias: `ru`

#### --group-id

group ID

Alias: `id`

#### --user-email

The email address of the user, you can supply this flag multiple times to remove multiple users in a single request

Alias: `e`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

### list-members

`tcld user-group list-members`: List all members of a group.

Alias: `lm`

#### --group-id

group ID

Alias: `id`

### delete

`tcld user-group delete`: Delete a user group.

Alias: `d`

#### --group-id

group ID

Alias: `id`

#### --request-id

The request-id to use for the asynchronous operation, if not set the server will assign one (optional)

Alias: `r`

---

# tcld version command reference

Source: https://docs.temporal.io/cloud/tcld/version

> Version information

`tcld version`: Version information.

Alias: `v`

---

# Temporal Cloud Terraform provider

Source: https://docs.temporal.io/cloud/terraform-provider

The Terraform Temporal Cloud provider allows you to use Terraform to manage resources for Temporal Cloud. The Terraform
tool manages infrastructure as code (IaC). With this provider, you can use Terraform to automate Temporal Cloud resource
management, including Namespaces, Users, Service Accounts, API Keys and more.

> **📝 Note:**
> Terraform Management
>
> Once a resource is managed by Terraform, you should only use Terraform to manage that resource.
>

Resources:

- The [Temporal Cloud Terraform provider](https://registry.terraform.io/providers/temporalio/temporalcloud/latest) is
  available in the Terraform Registry, where you can find detailed documentation on the Provider's supported resources
  and data sources.
- The GitHub repository for the Terraform provider is
  [terraform-provider-temporalcloud](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main), where
  you can report bugs, provide feature requests, and
  [contribute](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/CONTRIBUTING.md) to the
  provider. We encourage your input as we develop the provider with the community.
- To view the list of available Temporal Cloud resources supported by Terraform provider, visit the resources section of
  the Terraform documentation in Hashi's
  [registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs).

### Prerequisites

To use the Terraform provider, you'll need the following:

- The [Terraform CLI](https://developer.hashicorp.com/terraform/cli)
- An [API Key](/cloud/api-keys): an API Key is required to use the Terraform provider.
  - See [the API docs](/cloud/api-keys#generate-an-api-key) for instructions on generating an
    API Key.

> **📝 Note:**
> OpenTofu Registry
>
> Our Terraform Provider is registered with [OpenTofu](https://opentofu.org), but that registration is not maintained or
> managed by Temporal Technologies.
>

## Setup

Generate an [API Key](/cloud/api-keys#generate-an-api-key) to authenticate Terraform operations
with your Temporal Cloud account or a Service Account. Then, either use an environment variable or pass the API Key into
the provider manually to manage your Temporal Cloud Terraform resources.

Follow these examples to use an environment variable to pass in your API Key to the provider.

**macOS**

Export your environment variable for secure access to the API Keys.

```bash
# Replace <your-secret-key> with the token output from `temporal cloud apikey create-for-me` or `tcld apikey create`.
export TEMPORAL_CLOUD_API_KEY=<your-secret-key>
```

> **💡 Tip:**
> ENVIRONMENT VARIABLES
>
> Do not confuse environment variables, set with your shell, with temporal env options.
>

**Windows**

Export your environment variable for secure access to the API Keys.

```bash
# Replace <your-secret-key> with the token output from `temporal cloud apikey create-for-me` or `tcld apikey create`.
set TEMPORAL_CLOUD_API_KEY=<your-secret-key>
```

> **💡 Tip:**
> ENVIRONMENT VARIABLES
>
> Do not confuse environment variables, set with your shell, with temporal env options.
>

Or, pass it in manually in your .tf file using the provider code block

```yml
provider "temporalcloud" { api_key = "my-temporalcloud-api-key" }
```

## Manage Temporal Cloud Namespaces with Terraform

Terraform is a great way to automate the management of Temporal Namespaces. It doesn't matter whether you want
management to be centralized within a platform team or federated to different product teams. The provider allows you to
import, create, update, and delete Namespaces with Terraform.

You must use an Identity with Temporal Cloud Namespace management privileges. This includes the Account Owner, Global
Admin, or Developer Account Role.

For more detailed examples on how to manage Namespaces via Terraform, check the [Terraform Registry documentation for Namespaces](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace).

**How do I create a Namespace with Terraform?**

1. Create a Terraform configuration file (`terraform.tf`) to define a Namespace.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 14
   }
   ```

   In this example, you create a Temporal Cloud Namespace named `terraform`, specifying the AWS region `aws-us-east-1`,
   and specifying the path to the CA certificate.

1. Initialize the Terraform provider.

   Run the following command to initialize the Terraform provider.

   ```bash
   terraform init
   ```

1. Apply the Terraform configuration.

   Once initialization occurs, apply the Terraform configuration to your Temporal Cloud account.

   ```bash
   terraform apply
   ```

Follow the onscreen prompts.

Upon completion, you'll see a success message indicating your Namespace is created.

```bash
temporalcloud_namespace.terraform: Creation complete after 2m17s [id=<your-namespace>]
```

You can find more examples of Namespace management in the Terraform Provider docs located on HashiCorp's
[Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace).
The Terraform Provider docs show how to generate CA certs within Terraform configuration files and create a Namespace
with API Key based authentication.

**How do I validate the creation of the Namespace?**

You can validate the creation of the Namespace through the Temporal Web UI or through the CLI.

**Using the Temporal Web UI**

1. Log into the Temporal Cloud Web UI.
1. Navigate to the Namespaces page.
1. Search for the Namespace you created.

**Using the CLI**

Validate the creation of your Namespace through the Terraform provider. To validate see your Namespace in the Cloud UI
or through the `namespace get` command. Run the command and pass in your
[Cloud Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name) and
[Cloud Account Id](/cloud/namespaces#temporal-cloud-account-id):

**Temporal CLI**

```bash
temporal cloud namespace get -n "<your-namespace>.<your-account-id>"
```

**tcld**

```bash
tcld namespace get -n "<your-namespace>.<your-account-id>"
```

**How do I update a Temporal Cloud Namespace?**

Terraform automatically recognizes changes made within `.tf` files and applies those changes to Temporal.

For example, change the retention period setting in the Terraform file from the previous example and watch Terraform
apply the change without any additional steps required by you.

1. Set the retention period to 30 days.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source  = "temporalio/temporalcloud"
         version = ">= 0.0.6"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 30
   }
   ```

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

Upon completion, you will see a success message indicating your Namespace has been updated. It may take several minutes
to update a Namespace.

```text
temporalcloud_namespace.namespace: Modifications complete after 10s [id=terraform.a1bb2]
```

**How do I delete a Temporal Cloud Namespace?**

To delete a Namespace, remove the `temporalcloud_namespace` resource and all dependent resource configurations from your
Terraform files and run the `terraform apply` command.

Upon completion, you will see a success message indicating the resource has been destroyed:

```text
temporalcloud_namespace.my_namespace: Destruction complete after 3s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.
```

> **📝 Note:**
> Preventing Deletion
>
> You can prevent deletion of any Terraform resource by including the `prevent_destroy` argument in the Terraform
> configuration file.
>

**How do I import a Temporal Cloud Namespace?**

If you have an existing Namespace in Temporal Cloud, you can import it into Terraform to manage the Namespace from
Terraform using the `terraform import` command.

1. Provide a configuration placeholder in your Terraform configuration.

   ```yml
   resource "temporalcloud_namespace" "namespace" { }
   ```

1. Run the `terraform import` command from the command line and pass in the Namespace ID. Your Namespace ID is available
   at the top of the Namespace's page in the Temporal Cloud UI and is in the format `namespaceid.acctid`.

   ```bash
   terraform import temporalcloud_namespace.terraform namespaceid.acctid
   ```

The Namespace is now a part of the Terraform state and all changes to the Namespace should be managed by Terraform.

> **⚠️ Caution:**
>
> Once a resource has been imported into Terraform, outside changes to the resource will create Terraform "drift" errors
> on subsequent Terraform operations.
>

## Manage Temporal Cloud Nexus Endpoints with Terraform

Terraform provides a great way to automate the management of [Nexus Endpoints](/nexus/endpoints). The provider allows
you to import, create, update, and delete Nexus Endpoints with Terraform.

You must use an Identity with [Developer role (or higher)](/cloud/manage-access/roles-and-permissions#account-level-roles) and
[Namespace Admin permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) on the Endpoint's target Namespace.
Changing an Endpoint's target Namespace requires Namespace Admin on both the current target Namespace and the new target Namespace.

**How do I create a Nexus Endpoint with Terraform?**

1. Create a Terraform configuration file (`terraform.tf`) to define a Nexus Endpoint.

   From the
   [example in the Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/nexus_endpoint):

   ```yml
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "target_namespace" {
     name           = "terraform-target-namespace"
     regions        = ["aws-us-west-2"]
     api_key_auth   = true
     retention_days = 14
     timeouts {
       create = "10m"
       delete = "10m"
     }
   }

   resource "temporalcloud_namespace" "caller_namespace" {
     name           = "terraform-caller-namespace"
     regions        = ["aws-us-east-1"]
     api_key_auth   = true
     retention_days = 14
     timeouts {
       create = "10m"
       delete = "10m"
     }
   }

   resource "temporalcloud_namespace" "caller_namespace_2" {
     name           = "terraform-caller-namespace-2"
     regions        = ["gcp-us-central1"]
     api_key_auth   = true
     retention_days = 14
     timeouts {
       create = "10m"
       delete = "10m"
     }
   }

   resource "temporalcloud_nexus_endpoint" "nexus_endpoint" {
     name        = "terraform-nexus-endpoint"
     description = <<-EOT
       Service Name:
         my-hello-service
       Operation Names:
         echo
         say-hello

       Input / Output arguments are in the following repository:
       https://github.com/temporalio/samples-go/blob/main/nexus/service/api.go
     EOT
     worker_target = {
       namespace_id = temporalcloud_namespace.target_namespace.id
       task_queue   = "terraform-task-queue"
     }
     allowed_caller_namespaces = [
       temporalcloud_namespace.caller_namespace.id,
       temporalcloud_namespace.caller_namespace_2.id,
     ]
   }
   ```

   In this example, 3 Namespaces are created:

   - target Namespace for a Nexus Endpoint - Nexus requests will be routed to a Worker that polls the target Namespace.
   - caller Namespace(s) - Nexus Operations are invoked from caller Namespace, for example from a caller Workflow.

   These Namespaces are referenced in the [Nexus Endpoint](/nexus/endpoints) configuration:

   - `worker_target` (Namespace and Task Queue) - currently only a single worker_target is supported.
   - `allowed_caller_namespaces` - used to enforce Nexus Endpoint
     [runtime access controls](/nexus/security#runtime-access-controls).

1. Initialize the Terraform provider.

   Run the following command to initialize the Terraform provider.

   ```bash
   terraform init
   ```

1. Apply the Terraform configuration.

   Once initialization occurs, apply the Terraform configuration to your Temporal Cloud account.

   ```bash
   terraform apply
   ```

   Follow the onscreen prompts.

   Upon completion, you'll see a success message indicating 3 Namespaces and a Nexus Endpoint are created.

   ```bash
   temporalcloud_nexus_endpoint.nexus_endpoint: Creation complete after 2s [id=b158063be978471fa1d200569b03834d]
   ```

You can find more examples of Nexus Endpoint management in the Terraform Provider docs located on HashiCorp's
[Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/nexus_endpoint).
The Terraform Provider docs show how to generate CA certs within Terraform configuration files and create a Namespace
with API Key based authentication.

**How do I validate the creation of the Nexus Endpoint?**

You can validate the creation of the Nexus Endpoint through the Temporal Web UI or through the CLI.

**Using the Temporal Web UI**

1. Log into the Temporal Cloud Web UI.
1. Navigate to [the Nexus page](https://cloud.temporal.io/nexus).
1. Search for the Nexus Endpoint you created, using only the Nexus Endpoint Name (without an account suffix).

**Using the CLI**

Validate the creation of your Nexus Endpoint through the Terraform provider. To validate see your Nexus Endpoint in the
Cloud UI or through the `nexus endpoint get` command.

Run the below command using your Nexus Endpoint Name. Do not use the account ID suffix with this endpoint name:

**Temporal CLI**

```bash
temporal cloud nexus endpoint get --name "<your-nexus-endpoint-name-without-account-suffix>"
```

**tcld**

```bash
tcld nexus endpoint get -n "<your-nexus-endpoint-name-without-account-suffix>"
```

**How do I update a Nexus Endpoint?**

Terraform automatically recognizes changes made within `.tf` files and applies those changes to Temporal.

For example, to change the allowed caller Namespaces on a Nexus Endpoint:

1. Add or remove allowed caller Namespaces by updating the Nexus Endpoint configuration, for example by removing
   `caller_namespace_2` from the configuration above:

   ```yml
   resource "temporalcloud_nexus_endpoint" "nexus_endpoint" {
     name        = "terraform-nexus-endpoint"
     description = <<-EOT
       Service Name:
         my-hello-service
       Operation Names:
         echo
         say-hello

       Input / Output arguments are in the following repository:
       https://github.com/temporalio/samples-go/blob/main/nexus/service/api.go
     EOT
     worker_target = {
       namespace_id = temporalcloud_namespace.target_namespace.id
       task_queue   = "terraform-task-queue"
     }
     allowed_caller_namespaces = [
       temporalcloud_namespace.caller_namespace.id
     ]
   }
   ```

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

   Upon completion, you will see a success message indicating your Nexus Endpoint has been updated. It may take several
   seconds to update a Nexus Endpoint in the Control Plane which is visible from the Temporal UI or the CLI.
   Propagation of Nexus Endpoint changes to the data plane may take longer, but usually complete in less than one
   minute.

   ```text
   temporalcloud_nexus_endpoint.nexus_endpoint: Modifications complete after 1s [id=b158063be978471fa1d200569b03834d]
   ```

**How do I delete a Nexus Endpoint?**

To delete a Nexus Endpoint, remove the `temporalcloud_nexus_endpoint` resource configuration from your Terraform files
and run the `terraform apply` command.

Upon completion, you will see a success message indicating the resource has been destroyed:

```text
temporalcloud_nexus_endpoint.my_nexus_endpoint: Destruction complete after 3s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.
```

**How do I import a Temporal Cloud Nexus Endpoint?**

If you have an existing Nexus Endpoint in Temporal Cloud, you can import it into Terraform to manage the Nexus Endpoint
from Terraform using the `terraform import` command.

1. Initialize the Terraform provider in a new directory.

   Run the following command to initialize the Terraform provider.

   ```bash
   terraform init
   ```

1. Provide a configuration placeholder in your Terraform configuration and ensure you've included your
   [API key](#setup).

   ```yml
   terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" } } }

   provider "temporalcloud" { }

   resource "temporalcloud_nexus_endpoint" "nexus_endpoint" { }
   ```

1. Run the `terraform import` command from the command line and pass in the Nexus Endpoint ID.

   ```bash
   terraform import temporalcloud_nexus_endpoint <your-nexus-endpoint-ID>
   ```

   Your Nexus Endpoint ID is available at the top of the Nexus Endpoint's page in the
   [Temporal Cloud UI](https://cloud.temporal.io/nexus).

![Nexus Endpoint ID](https://docs.temporal.io/img/cloud/nexus/nexus-endpoint-id.png)

   Upon completion, you will see a success message indicating the Nexus Endpoint was imported.

   ```text
   temporalcloud_nexus_endpoint.nexus_endpoint: Refreshing state... [id=3c0c75ccfa8144b092c13ce632463761]

   Import successful!
   ```

The Nexus Endpoint is now a part of the Terraform state and all changes to the Nexus Endpoint should be managed by
Terraform.

> **⚠️ Caution:**
>
> Once a resource has been imported into Terraform, outside changes to the resource will create Terraform "drift" errors
> on subsequent Terraform operations.
>

## Manage Temporal Cloud Users with Terraform

Manage Temporal Cloud Users with the same process you use to manage Namespaces with Terraform. The following examples
create, update, delete, and import Temporal Cloud Users with `terraform apply` commands on the Terraform configuration
file.

> **📝 Note:**
> User Management
>
> Cautions about Temporal User management:
>
> - Terraform can't manage the Temporal Account Owner role. While you can import an Account Owner to Terraform, you cannot
>   create, update, or delete an Account Owner with Terraform.
> - Right now, you can't manage a user's access to a Namespace from the Namespace resource. You must manage Namespace
>   access from the User resource. This is also true for Service Accounts.
> - Account Owners and Global Admins automatically gain access to all Namespaces in Temporal. Therefore, you cannot
>   specify Namespace access for these roles. This is also true for Service Accounts.
> - Follow Terraform best practices for resource management. Manage a specific user in one and only one .tf file. There's
>   a risk that you may overwrite a user's permissions if you don't.
> - To Import a user, you'll need the User's ID which is currently not available in the Temporal Cloud UI. You can fetch
>   current User ID by running the `temporal cloud user list` or `tcld user list` command.
>

For more detailed examples on how to manage Namespaces via Terraform, check the Terraform Registry documentation for [provisioning a Temporal Cloud user](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/user).

**How do I create a Temporal Cloud User with Terraform?**

1. Add a Terraform User resources configuration to your Terraform file.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 14
   }

   # Global admins automatically have access to all namespaces.
   resource "temporalcloud_user" "global_admin" {
     email          = "admin@example.com"
     account_access = "Admin"
   }

   # Developers can be granted explicit namespace permissions.
   resource "temporalcloud_user" "namespace_admin" {
     email          = "developer@example.com"
     account_access = "Developer"

     namespace_accesses = [{
       namespace_id = temporalcloud_namespace.namespace.id
       permission   = "Write"
     }]
   }
   ```

   Replace the email and domain values with your Temporal Cloud User email and domain.

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

Upon completion, you will see a success message indicating your User has been created.

```text
temporalcloud_user.namespace_admin: Creation complete after 1s [id=12a34bc5678910d38d9e8390636e7412]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
```

**How do I update a Temporal Cloud User with Terraform?**

To update a User with Terraform, follow the same steps used to create a User.

**How do I delete a Temporal Cloud User with Terraform?**

To delete a User with Terraform, remove the Terraform User resources configuration from your Terraform file and run the
`terraform apply` command.

1. Remove the Terraform User resources configuration from your Terraform file.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source  = "temporalio/temporalcloud"
         version = ">= 0.0.6"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 14
   }

   # This user will be deleted after running `terraform apply`
   resource "temporalcloud_user" "global_admin" {
     email          = "admin@example.com"
     account_access = "Admin"
   }
   # The following user resource has been removed (or commented out),
   # so Terraform will delete it.
   # resource "temporalcloud_user" "namespace_admin" {
   #   email          = "developer@example.com"
   #   account_access = "Developer"
   #
   #   namespace_accesses = [{
   #     namespace_id = temporalcloud_namespace.namespace.id
   #     permission   = "Write"
   #   }]
   # }
   ```

1. Run the `terraform apply` command. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

Upon completion, you will see a success message indicating your User has been deleted.

```text
temporalcloud_user.namespace_admin: Destruction complete after 2s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.
```

**How do I import a Temporal User?** If you have an existing User in Temporal Cloud, you can import it into Terraform
using the `terraform import` command.

1. Provide a configuration placeholder in your Terraform configuration.

   ```yml
   resource "temporalcloud_user" "user" { }
   ```

1. Run the `terraform import` command and pass in the User ID. Your User ID is available by running `temporal cloud user list` or `tcld u l`.

 ```bash
 terraform import temporalcloud_user.user 72360058153949edb2f1d47019c1e85f
 ```

The User is now a part of the Terraform state and all changes to the User should be managed by Terraform.

## Manage Temporal Cloud Service Accounts with Terraform

The process and steps to managing a Service Account with Terraform are very similar to managing a User with Terraform
with a few small differences:

- Service Accounts use the Service Account Terraform resource not the User resource.
- Service Accounts do not have email addresses, they have names instead. This means you should specify a name for a
  Service Account instead of an email.

Everything else about managing Services Accounts with Terraform follows the same process, guidance, and limitations of
managing Users with Terraform.

## Manage Temporal Cloud API Keys with Terraform

You can manage your own, personal API Keys and Service Account API Keys with Terraform. The process and steps to
managing an API Key with Terraform are very similar to managing other resources with Terraform. You can create, delete,
update and import API Keys with Terraform. One difference between working with API Keys as a Terraform resource compared
to other Temporal Cloud resources is the need to access an API Keys secure token output from Terraform. Walk through the
process of securely accessing the API Key Token in the Create section of this guide.

> **📝 Note:**
> Limits and Best Practices
>
> - See the API Key [documentation](/cloud/api-keys) for information about the limits and best
>   practices for managing API Keys.
> - See Terraform's documentation on working with
>   [sensitive data](https://www.terraform.io/docs/language/values/variables.html#sensitive-values) for more information
>   on how to manage sensitive data in Terraform.
>

**How do I create a Temporal Cloud API Key with Terraform?**

From the example in the [Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/apikey):
1. Add a Terraform API Key resources configuration to your Terraform file.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_service_account" "global_service_account" {
     name           = "admin"
     account_access = "Admin"
   }

   resource "temporalcloud_apikey" "global_apikey" {
     display_name = "admin"
     owner_type   = "service-account"
     owner_id     = temporalcloud_service_account.global_service_account.id
     expiry_time  = "2024-11-01T00:00:00Z"
     disabled     = false
   }
   ```

   Make sure to:

   - Replace the display_name, expiry_time, and disabled values with your Temporal Cloud API Key configuration.
   - Replace the owner_type and owner_id values with your Temporal Cloud Service Account or other Identity information.

1. Create an output.tf file and add the following code to output the API Key Token.

   ```hcl
   output "apikey_token" {
     value     = temporalcloud_apikey.global_apikey.token
     sensitive = true
   }
   ```

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

   Upon completion, you will see a success message indicating the API Key has been created.

   ```text
   temporalcloud_apikey.global_apikey: Creation complete after 1s [id=kayBf38JIWkMPmnfr59iEIaEk2L7uqR4]
   ```

1. Access the API Key Token securely. You'll notice that if you view the state for the API Key resource, the token value
   is not displayed.

   ```bash
   terraform state show temporalcloud_apikey.global_apikey

   # temporalcloud_apikey.global_apikey:
   resource "temporalcloud_apikey" "global_apikey" {
       disabled     = false
       display_name = "adminKey3"
       expiry_time  = "2024-12-01T00:00:00Z"
       id           = "kayBf38JIWkMPmnfr59iEIaEk2L7uqR4"
       owner_id     = "b81336a6097449cba75c2e5500df3d31"
       owner_type   = "service-account"
       state        = "active"
       token        = (sensitive value)
   }
   ```

To access the token, you can use the Terraform output command.

```bash
terraform output -json apikey_token
```

This will display the token value in the terminal.

> **ℹ️ Info:**
> Security and API Keys
>
> Remember, keep your Terraform state files secure if you're managing API Keys with Terraform. The state file contains
> sensitive information, like the API Key Token, that should not be shared or exposed.
>

**How do I update a Temporal Cloud API Key with Terraform?**

To update an API Key with Terraform, follow the same steps used to create an API Key.

> **📝 Note:**
> Editing Fields
>
> You can only edit an API Key's name or description field. Updating an API Key does not generate a new secure token
>

**How do I delete a Temporal Cloud API Key with Terraform?**

To delete an API Key with Terraform, remove the Terraform API Key resources configuration from your Terraform and
output.tf files and run the `terraform apply` command.

**How do I Import a Temporal API Key?**

You cannot import an API Key into Terraform. Once created, the API Key secret isn't stored and can't be retrieved, so
you can't access it using import.

Instead, Temporal recommends creating a new API Key using Terraform directly.

## Data Sources - Regions and Namespaces

The Terraform provider also supports 2 data sources that provide you access to the available Regions and Namespaces in
your Temporal Cloud account.

> **📝 Note:**
> Terraform Data Sources
>
> See Terraform [documentation](https://developer.hashicorp.com/terraform/language/data-sources) to learn more about
> Terraform Data Sources
>

For example, to retrieve a list of regions available for your account, you can use the regions data_source

```hcl
data "temporalcloud_regions" "regions" {}

output "regions" {
  value = data.temporalcloud_regions.regions.regions
}
```

## Community involvement

Do you have feedback about the provider? Want to report a bug or request a feature? We'd love to hear from you.

- Please reach out to us in the Temporal Community
  [Slack](https://join.slack.com/t/temporalio/shared_invite/zt-2u2ey8ilu-LRxnd3PSoAk9GZ94UuzoBA) in the #terraform
  channel
- Feel free to create issues and contribute PRs in the Temporal Terraform
  [GitHub repository](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main)

---

# Monitor Worker health

Source: https://docs.temporal.io/cloud/worker-health

> Detect and configure for Task backlogs, greedy Worker resources, misconfigured Workers, and Sticky cache settings. Optimize alert systems and get actionable insights on metrics like Schedule-To-Start latency, Sync Match Rate, and Poll Success Rate for improved application health.

This page is a guide to monitoring a Temporal Worker fleet and covers the following scenarios:

- [Configuring minimal observations](#minimal-observations)
- [How to detect a backlog of Tasks](#detect-task-backlog)
- [How to detect greedy Worker resources](#detect-greedy-workers)
- [How to detect misconfigured Workers](#detect-misconfigured-workers)
- [How to configure Sticky cache](#configure-sticky-cache)

This page assumes you are monitoring both Worker-side SDK metrics and Cloud-side metrics. Use SDK metrics to understand
what your Workers are doing, and Cloud metrics to understand what Temporal Cloud is seeing at the Task Queue and service
level. For an overview of how these signals fit together, see [metric sources](/cloud/monitor#understand-metric-sources).

> **💡 Tip:**
>
> You can also inspect Workers and the Workers assigned to a Task Queue directly in the Temporal UI. See [Visualize Workers in the UI](/develop/worker-performance#visualize-workers).
>

## Minimal observations 

These alerts should be configured and understood first to gain intelligence into your application health and behaviors.

1. Create monitors and alerts for Schedule To Start latency SDK metrics (both [Workflow Executions](/references/sdk-metrics#workflow_task_schedule_to_start_latency) and [Activity Executions](/references/sdk-metrics#activity_schedule_to_start_latency)).
   See [Detect Task backlog section](#detect-task-backlog) to explore [sample queries](#prometheus-query-samples) and appropriate responses that accompany these values.

- Alert at >200ms for your p99 value
- Plot >100ms for your p95 value

2. Create a [Grafana](/cloud/metrics/openmetrics/metrics-integrations#prometheus-grafana) panel called Sync Match Rate.
   See the [Sync Match Rate section](#sync-match-rate) to explore example queries and appropriate responses that accompany these values.

- Alert at \<95% for your p99 value
- Plot \<99% for your p95 value

3. Create a [Grafana](/cloud/metrics/openmetrics/metrics-integrations#prometheus-grafana) panel called Poll Success Rate.
   See the [Detect greedy Workers section](#detect-greedy-workers) for example queries and appropriate responses that accompany these values.

- Alert at \<90% for your p99 value
- Plot \<95% for your p95 value

The following alerts build on the above to dive deeper into specific potential causes for Worker related issues you might be experiencing.

1. Create monitors and alerts for the [temporal_worker_task_slots_available](/references/sdk-metrics#worker_task_slots_available) SDK metric.
   See the [Detect misconfigured Workers section](#detect-misconfigured-workers) for appropriate responses based on the value.

- Alert at 0 for your p99 value

2. Create monitors for the [temporal_sticky_cache_size](/references/sdk-metrics#sticky_cache_size) SDK metric.
   See the [Configure Sticky Cache section](#configure-sticky-cache) for more details on this configuration.

- Plot at \{value\} > \{WorkflowCacheSize.Value\}

3. Create monitors for the [temporal_sticky_cache_total_forced_eviction](/references/sdk-metrics#sticky_cache_total_forced_eviction) SDK metric.
   This metric is available in the Go SDK, and the Java SDK only.
   See the [Configure Sticky Cache section](#configure-sticky-cache) for more details and appropriate responses.

- Alert at >\{predetermined_high_number\}

4. Create monitors for the [temporal_cloud_v1_approximate_backlog_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count) Cloud metric.
   This metric provides a server-side view of how many Tasks are waiting in a Task Queue and complements the SDK Schedule To Start latency metrics.

- Alert when the value is growing over time for a given Task Queue

## Detect Task backlog 

### Symptoms of high Task backlog

If the Task backlog is too high, you will find that tasks are waiting to find Workers to run on. This can cause a delay in
Workflow execution. Detecting a growing Task backlog is possible by watching the Schedule To Start latency, sync match rate, and approximate backlog count.

Metrics to monitor:

- **SDK metric**: [workflow_task_schedule_to_start_latency](/references/sdk-metrics#workflow_task_schedule_to_start_latency)
- **SDK metric**: [activity_schedule_to_start_latency](/references/sdk-metrics#activity_schedule_to_start_latency)
- **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_count)
- **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_sync_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_sync_count)
- **Temporal Cloud metric**: [temporal_cloud_v1_approximate_backlog_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count)

### Schedule-To-Start latency

The Schedule To Start metric represents how long Tasks are staying unprocessed in the Task Queues.
It is the time between when a Task is enqueued and when it is started by a Worker.
This time being long (likely) means that your Workers can't keep up - either increase the number of Workers (if the host load is already high) or increase the number of pollers per Worker.

If your Schedule To Start latency alert triggers or is high, check the [Sync Match Rate](#sync-match-rate) to decide if you need to adjust your Worker or fleet, or contact Temporal Cloud support.
If your Sync Match Rate is low, contact [Temporal Cloud support](/evaluate/cloud/support#support-ticket).
If your Sync Match Rate is low, you can contact Temporal Cloud support.

The schedule_to_start_latency SDK metric for both [Workflow Executions](/references/sdk-metrics#workflow_task_schedule_to_start_latency) and [Activity Executions](/references/sdk-metrics#activity_schedule_to_start_latency) should have alerts.

#### Prometheus query samples

**Workflow Task Latency, 99th percentile**

```
histogram_quantile(0.99, sum(rate(temporal_workflow_task_schedule_to_start_latency_seconds_bucket[5m])) by (le, namespace, task_queue))
```

**Workflow Task Latency, average**

```
sum(increase(temporal_workflow_task_schedule_to_start_latency_seconds_sum[5m])) by (namespace, task_queue)
/
sum(increase(temporal_workflow_task_schedule_to_start_latency_seconds_count[5m])) by (namespace, task_queue)
```

**Activity Task Latency, 99th percentile**

```
histogram_quantile(0.99, sum(rate(temporal_activity_schedule_to_start_latency_seconds_bucket[5m])) by (le, namespace, task_queue))
```

**Activity Task Latency, average**

```
sum(increase(temporal_activity_schedule_to_start_latency_seconds_sum[5m])) by (namespace, task_queue)
/
sum(increase(temporal_activity_schedule_to_start_latency_seconds_count[5m])) by (namespace, task_queue)
```

**Target**

This latency should be very low, close to zero.
Any higher value indicates a bottleneck.

### Sync match rate 

The sync match rate measures the rate of Tasks that are delivered to workers without having to be persisted (workers are up and available to pick them up) to the rate of all delivered tasks.

A sync match is when a task is immediately matched to a Worker via the Sticky Queue.

An async match is when a Task cannot be matched to the Sticky Queue for a Worker. This can happen when no Worker has cached the Workflow, or if the Task times out during processing. In this case, the Task returns to the general Task Queue.

**Calculate Sync Match Rate**

```
temporal_cloud_v1_poll_success_sync_count / temporal_cloud_v1_poll_success_count = N
```

#### Prometheus query samples

**sync_match_rate query**

```
sum by(temporal_namespace) (
    temporal_cloud_v1_poll_success_sync_count{temporal_namespace=~"$namespace"}
)
/
sum by(temporal_namespace) (
    temporal_cloud_v1_poll_success_count{temporal_namespace=~"$namespace"}
)
```

**Target**

The Sync Match Rate should be at least >95%, but preferably >99%.

### Handling Task backlog issues 

Once you have detected the condition of a high Task backlog, consider the scenarios below to take action.

#### High Schedule-To-Start latency and high sync match rate

There are three typical causes for this:

- There are not enough workers to perform work
- Each worker is either under resourced, or is misconfigured, to handle enough work
- There is congestion caused by the environment (for example, network) hosting the worker(s) and Temporal Cloud

Consider

- Increasing either the number of available workers
- Verifying that your worker hosts are appropriately resourced
- Increasing the worker configuration value for concurrent pollers for workers/task executions (if your worker resources can accommodate the increased load)
- Doing some combination of these

#### High Schedule-To-Start latency and low sync match rate

Verify that you have not set a value for `ScheduleToStartTimeout` in your Activity Options. This may skew your observations and note that this timeout is non-retryable.

It may be acceptable for your use case to have low sync match rate.
For example, if you have known workloads or you intentionally throttle tasks.

In this case it's also important to understand the fill and drain rates of the async tasks are during these windows:

Successful async polls

```
temporal_cloud_v1_poll_success_count - temporal_cloud_v1_poll_success_sync_count = N
```

```
sum by(temporal_namespace, task_type) (
    temporal_cloud_v1_poll_success_count{temporal_namespace=~"$namespace"}
)
-
sum by(temporal_namespace, task_type) (
    temporal_cloud_v1_poll_success_sync_count{temporal_namespace=~"$namespace"}
)
```

You can also monitor the [approximate backlog count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count) to observe Task Queue depth directly:

```
temporal_cloud_v1_approximate_backlog_count{temporal_namespace=~"$namespace", temporal_task_queue=~"$task_queue"}
```

**Actions**

- Verify that your Worker setup is optimized for your instance:
  - Check the system CPU usage against `task_slots` and adjust `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` settings as necessary.
  - Check the system memory usage against `sticky_cache_size` and adjust sticky cache size as necessary.
  - For a detailed explanation of settings, see the [Worker Performance](/develop/worker-performance/task-queues#task-queues-processing-tuning) section.
- Increase the Worker config for concurrent pollers for Workflow or Activity `task_slots`, if your Worker resources can accommodate the increased load.
  - Reference [Worker Performance > Poller Count](/develop/worker-performance/task-queues#poller-count).
- Increase the number of available Workers.

> **⚠️ Warning:**
>
> Setting the [Schedule To Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) in your Activity Options can skew your observations.
> Avoid setting a Schedule To Start Timeout when load testing for latency.
>

## Detect greedy Worker resources 

You can have too many Workers.
If you see the Poll Success Rate showing low numbers, you might have too many resources polling Temporal Cloud.

Metrics to monitor:

- **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_count)
- **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_sync_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_sync_count)
- **Temporal Cloud metric**: [temporal_cloud_v1_poll_timeout_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_timeout_count)
- **SDK metric**: [temporal_workflow_task_schedule_to_start_latency](/references/sdk-metrics#workflow_task_schedule_to_start_latency)
- **SDK metric**: [temporal_activity_schedule_to_start_latency](/references/sdk-metrics#activity_schedule_to_start_latency)

**Calculate Poll Success Rate**

```
(temporal_cloud_v1_poll_success_count)
/
(temporal_cloud_v1_poll_success_count + temporal_cloud_v1_poll_timeout_count)
```

**Target**

Poll Success Rate should be >90% in most cases of systems with a steady load.
For high volume and low latency, try to target >95%.

**Interpretation**

There may be too many Pollers for the amount of Workers.

If you see all of the following at the same time then you might have too many Workers:

- Low poll success rate
- Low Schedule To Start latency
- Low worker host resource utilization

**Actions**

Consider sizing down your Workers by either:

- Reducing the number of Workers polling the impacted Task Queue, OR
- Reducing the concurrent pollers per Worker, OR
- Both of the above

## Detect misconfigured Workers 

Worker configuration can negatively affect Task processing efficiency.

Metrics to monitor:

- **SDK metric**: [temporal_worker_task_slots_available](/references/sdk-metrics#worker_task_slots_available)
- **SDK metric**: [sticky_cache_size](/references/sdk-metrics#sticky_cache_size)
- **SDK metric**: [sticky_cache_total_forced_eviction](/references/sdk-metrics#sticky_cache_total_forced_eviction)

**Execution Size Configuration**

The `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` define the number of total available slots for the Worker.
If this is set too low, the Worker will not be able to keep up processing Tasks.

**Target**

The `temporal_worker_task_slots_available` metric should always be >0.

#### Prometheus query samples

**Over Time**

```
avg_over_time(temporal_worker_task_slots_available{namespace="$namespace",worker_type="WorkflowWorker"}[10m])
```

**Current Time**

```
temporal_worker_task_slots_available{namespace="default", worker_type="WorkflowWorker", task_queue="$task_queue_name"}
```

**Interpretation**

You are likely experiencing a Task backlog if you are seeing inadequate slot counts frequently.
The work is not getting processed as fast as it should/can.

**Action**

Increase the `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` values and keep an eye on your Worker resource metrics (CPU utilization, etc) to make sure you haven't created a new issue.

### Configure Sticky Execution Cache 

Sticky Execution means that a Worker caches a Workflow Execution Event History and creates a dedicated Task Queue to listen on.
It significantly improves performance because the Temporal Service only sends new events to the Worker instead of entire Event Histories.

**Target**

The `sticky_cache_size` should report less than or equal to your `WorkflowCacheSize` value.
Also, sticky_cache_total_forced_eviction should not be reporting high numbers (relative).

**Action**

If you see a high eviction count, verify there are no other inefficiencies in your Worker configuration or resource provisioning (backlog).
If you see the cache size metric exceed the `WorkflowCacheSize`, increase this value if your Worker resources can accommodate it or provision more Workers.
Finally, take time to review [the Worker performance guide](/develop/worker-performance) and see if it addresses other potential cache issues.

#### Prometheus query samples

**Sticky Cache Size**

```
max_over_time(temporal_sticky_cache_size{namespace="$namespace"}[10m])
```

**Sticky Cache Evictions**

```
rate(temporal_sticky_cache_total_forced_eviction_total{namespace="$namespace"}[5m]))
```

## Manage Worker Heartbeating 

> **Public Preview**

Workers send a heartbeat to Temporal Server every 60 seconds by default. This heartbeat serves to provide liveness and configuration data from the Worker to the Server.
Specific data sent can be found in the [API](https://github.com/temporalio/api/blob/main/temporal/api/worker/v1/message.proto). By providing a consistent heartbeat from
Workers, the Server can obtain an accurate count of Workers, understand Worker performance, and respond to Worker heartbeats with commands. Some examples of how this is useful:

- understanding the difference between a Worker that is down and a Worker that is processing tasks for a long time
- identifying a Worker with high CPU usage from the Server point of view

Use the Temporal CLI to view information about all Workers connected to Temporal Server. Use `temporal worker describe` to see details of a specific Worker. Use `temporal worker list` to get a complete list of all connected Workers.

If you wish to disable Worker heartbeating or set heartbeating to be more frequent than every 60 seconds (allowed range is 1s to 60s), set the configuration relevant to your SDK. The server needs Worker heartbeating to understand Worker status accurately. Disabling the Worker heartbeat will cause features that provide the list of active Workers and information about those Workers to show missing or inaccurate information.
**Go**

_Available since Go SDK v1.41.0_

Set the `WorkerHeartbeatInterval` field on [`client.Options`](https://pkg.go.dev/go.temporal.io/sdk/client#Options) to adjust the heartbeat interval.
Set it to a negative value to disable heartbeating.

#### Enable host resource reporting

By default, the Go SDK reports `0` for CPU and memory usage in Worker heartbeats.
Set `SysInfoProvider` on [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/worker#Options) to enable host resource reporting.
Host resource reporting is not included in the core SDK module. Add the [sysinfo](https://pkg.go.dev/go.temporal.io/sdk/contrib/sysinfo) contrib package to your imports - it provides a [gopsutil](https://github.com/shirou/gopsutil)-based implementation that supports cgroup metrics in containerized Linux environments:

```go
import (
	"go.temporal.io/sdk/contrib/sysinfo"
	"go.temporal.io/sdk/worker"
)

w := worker.New(c, "my-task-queue", worker.Options{
	SysInfoProvider: sysinfo.SysInfoProvider(),
})
```

You can also implement the `worker.SysInfoProvider` interface to provide your own resource metrics.

**Python**

_Available since Python SDK v1.20.0_

Use `TelemetryConfig()` to adjust heartbeat settings. See the [Python SDK documentation](https://python.temporal.io/temporalio.bridge.runtime.RuntimeOptions.html#worker_heartbeat_interval_millis) for more details.

**TypeScript**

_Available since TypeScript SDK v1.14.0_

Set the `workerHeartbeatInterval` property on [`RuntimeOptions`](https://typescript.temporal.io/api/interfaces/worker.RuntimeOptions) to adjust the heartbeat interval.
Set it to `0` to disable heartbeating.

**.NET**

_Available since .NET SDK v1.10.0_

Set the `WorkerHeartbeatInterval` property on [`TemporalRuntimeOptions`](https://dotnet.temporal.io/api/Temporalio.Runtime.TemporalRuntimeOptions.html) to adjust the heartbeat interval.
Set it to `null` to disable heartbeating.

**Java**

_Available since Java SDK v1.35.0_

Set the heartbeat interval on [`WorkflowClientOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClientOptions.Builder.html) with `setWorkerHeartbeatInterval(Duration)`.
Set it to a negative `Duration` to disable heartbeating.

**Ruby**

_Available since Ruby SDK v1.1.0_

Add configurations to `Runtime()` to adjust heartbeat settings. See the [Ruby SDK documentation](https://ruby.temporal.io/Temporalio/Runtime.html) for more details.

---

# Codec Server

Source: https://docs.temporal.io/codec-server

A Codec Server is an HTTP/HTTPS server that you host and operate. It runs your [Payload Codec](/payload-codec) logic to
encode and decode [Payloads](/dataconversion#payload) on behalf of the Temporal CLI and Web UI. The Codec Server is
independent of the Temporal Service. Encryption keys and codec logic remain in your environment.

For setup instructions, see [Codec Server setup](/production-deployment/data-encryption#codec-server-setup).

## Why use a Codec Server

When you apply a custom [Payload Codec](/payload-codec) for encryption or compression, data stored in the Temporal
Service is encoded. The Temporal Service never has access to your encryption keys, so it cannot decode this data.
Without a Codec Server, the Web UI and CLI display raw encoded payloads.

A Codec Server solves this by giving the Web UI and CLI a way to decode payloads on demand, without exposing keys to the
Temporal Service. Common reasons to run a Codec Server include:

- **Debugging Workflows.** View decoded Workflow inputs, outputs, and Event History in the Web UI instead of reading
  base64-encoded or encrypted blobs.
- **Operating from the CLI.** Use commands like `temporal workflow show` and `temporal workflow execute` with readable
  data, even when payloads are encrypted at rest.
- **Encoding inputs from the UI and CLI.** When you start or signal a Workflow from the Web UI or CLI, the Codec Server
  can encode the input before it reaches the Temporal Service, so the Temporal Service never sees plaintext (the input
  still travels from your browser or CLI to the Codec Server, which is why HTTPS matters in any non-loopback
  deployment).
- **Compliance and access control.** Because the Codec Server runs in your environment, you control who can decode
  payloads and under what conditions. You can layer authorization on top of the decode endpoint to restrict access per
  user or per Namespace.

## How a Codec Server works

A Codec Server follows the Temporal
[Codec Server Protocol](https://github.com/temporalio/samples-go/tree/main/codec-server#codec-server-protocol). It
exposes two HTTP POST endpoints:

- **`/encode`** accepts plaintext payloads and returns encoded payloads. Used for sending payloads.
- **`/decode`** accepts encoded payloads and returns decoded payloads. Used for retrieving payloads.

Both endpoints receive and respond with a JSON body containing a `payloads` array of [Payload](/dataconversion#payload)
objects. The Codec Server passes each payload through your [Payload Codec](/payload-codec), which applies the same
encoding or decoding logic that your Workers use.

![Codec Server](https://docs.temporal.io/diagrams/codec-server.svg)

When the Web UI or CLI needs to display decoded data, it sends the encoded payloads to your Codec Server's `/decode`
endpoint. The Codec Server decodes the payloads and returns them to the client. The Temporal Service never sees the
decoded data.

The `/encode` endpoint works in the other direction. When you start a Workflow or send a Signal from the Web UI or CLI,
the input is sent to the Codec Server's `/encode` endpoint first, so data reaches the Temporal Service in its encoded
form.

Your Codec Server should use the same Payload Codec implementation as your Workers to ensure consistent encoding and
decoding.

## Codec Server with External Storage 

When your Workers and Clients use [External Storage](/external-storage), your storage drivers replace some payloads in
the Event History with small references that point to data in an external store like Amazon S3. The Temporal Service and
the Web UI only see these references, not the actual payload data. This is further complicated by setups where you run
Codecs in proxy that encode payloads after the Data Converter has returned on the Worker. Your Codec Server must be able
to handle downloading and decoding in the correct order for you to be able to view the Workflow data in the UI or CLI.

To support External Storage, create a handler using `NewPayloadHTTPHandler` with `PayloadHTTPHandlerOptions`. The options
accept your storage drivers, your pre-storage codecs (the Payload Codecs configured in your Worker's Data Converter),
and any post-storage codecs (codecs applied by a proxy after external storage). The handler applies them in the correct
order across all endpoints automatically. When you configure the handler with storage drivers, the existing endpoints
become storage-aware and a new `/download` endpoint becomes available:

> **⚠️ Caution:**
>
> `NewPayloadHTTPHandler` runs the full encode-store-encode and decode-retrieve-decode pipeline. Do not use it as a target
> for a remote Data Converter or remote codec on your Workers. For remote codecs, use `NewPayloadCodecHTTPHandler`
> separately. If you need both, set up `NewPayloadHTTPHandler` for the Web UI and CLI alongside
> `NewPayloadCodecHTTPHandler` for your Workers, and configure both with the same codecs.
>

- **`/download`** retrieves the actual payload data from external storage and decodes it through the Payload Codec. This
  endpoint is used internally by `/decode` when it encounters storage references, but you can also call it directly from the Web UI 
  to retrieve the decoded payload. The Temporal Web UI uses this endpoint when you click to view the full payload for a
  storage reference.
- **`/decode`** still decodes encoded payloads, but also handles storage references. By default, `/decode` uses the
  download logic internally to retrieve and decode any storage references in the request alongside regular payloads.
  With the `?preserveStorageRefs=true` query parameter, `/decode` skips retrieval and returns storage references as-is.
- **`/encode`** applies the Payload Codec, then uploads payloads that exceed the size threshold to external storage and
  replaces them with reference tokens.

![Codec Server with External Storage](https://docs.temporal.io/diagrams/codec-server-with-external-storage.svg)

The following example walks through how all three endpoints work together:

1. A user starts a Workflow from the CLI with a plaintext input. The CLI sends the input to the Codec Server's `/encode`
   endpoint.
2. The Codec Server encodes the payload through the Payload Codec. The encoded payload exceeds the storage threshold,
   so the Codec Server uploads it to external storage and returns a small reference token.
3. The CLI sends the reference token to the Temporal Service, which stores it in the Event History.
4. Later, a user views the Workflow in the Web UI. The Web UI retrieves the Event History from the Temporal Service and
   sends the payloads to the Codec Server's `/decode` endpoint with the `?preserveStorageRefs=true` query parameter.
5. The Codec Server decodes any non-reference payloads through the Payload Codec, but returns storage references as-is.
   The Web UI displays the reference metadata, indicating the payload is stored externally.
6. The user clicks to view the full payload. The Web UI sends the storage reference to the `/download` endpoint.
7. The Codec Server retrieves the encoded payload from external storage, decodes it through the Payload Codec, and
   returns the plaintext result to the Web UI.

## Codec Server versus Payload Codec 

A Codec Server runs a [Payload Codec](/payload-codec) internally, so the two are directly connected. The difference is
where the codec logic runs and who calls it.

|                                   | Payload Codec                                                                                       | Codec Server                                                                                            |
| --------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Purpose**                       | Encodes and decodes Payloads. Applies encryption, compression, or other byte-level transformations. | Hosts a Payload Codec as an HTTP service so the Web UI and CLI can encode and decode Payloads remotely. |
| **Runs where**                    | In-process, inside your Workers and Clients. Also runs inside the Codec Server.                     | As a standalone HTTP service in your environment, with a Payload Codec inside it.                       |
| **Called by**                     | The Temporal SDK, automatically on every serialization and deserialization.                         | The Web UI and CLI, over HTTP, when a user views or submits Payload data.                               |
| **Has access to encryption keys** | Yes. Keys are available in the Worker or Client process.                                            | Yes. Must be configured with the same keys the Payload Codec uses.                                      |

You implement the transformation logic once in a Payload Codec, then host that logic in a Codec Server so the Web UI and
CLI can use it remotely.

## Securing a Codec Server

Because a Codec Server can decode sensitive data, treat it with the same trust as a Worker. Anyone who can call it has
effective decrypt access. Use HTTPS for any deployment that is not strictly loopback (`localhost`).

### Network-level restrictions

Restrict network access to the Codec Server. The Web UI can communicate with a Codec Server that is only accessible on
`localhost`, so running the Codec Server locally is a viable security pattern. For team access, place the Codec Server
behind a VPN.

### Authentication

When the Codec Server is accessible beyond `localhost`, authenticate requests to verify the identity of the caller. The
Web UI supports two approaches:

**Include cross-origin credentials (recommended).** Enable **Include cross-origin credentials** in the Web UI Codec
Server settings. The browser sends cookies scoped to the Codec Server's domain with each request. Your Codec Server must
have its own authentication mechanism (its own login page and session cookies), so the user must have independently
authenticated with the Codec Server. This is the recommended approach because the Codec Server maintains its own auth
boundary, separate from the Temporal UI.

**Pass access token.** Enable **Pass access token** in the Web UI Codec Server settings. The Web UI includes the same
JSON Web Token (JWT) the user used to log into the Temporal UI in the `Authorization` header of each request. Your Codec
Server validates the token signature against the OpenID Connect (OIDC) provider's JSON Web Key Set (JWKS) endpoint. On
Temporal Cloud, verify against the
[Temporal Cloud JWKS endpoint](https://login.tmprl.cloud/.well-known/jwks.json). On a self-hosted Temporal Service, the
token comes from whatever auth provider you have [configured for the Web UI](/references/web-ui-configuration#auth).
This approach requires less setup but reuses the same token across the Temporal UI and the Codec Server.

> **📝 Note:**
>
> If you have a step in your process for token validation to ensure access isn't granted to the wrong token, you can validate the `audience` claim with:
>
> ```bash
> "aud": [
>   "https://saas-api.tmprl.cloud"
> ]
> ```
>

### Namespace-level authorization

Authentication identifies the caller, but does not confirm the caller is authorized to decode payloads for a specific
Namespace. Each request from the Web UI includes an `X-Namespace` header identifying the Namespace. To enforce
Namespace-level access control, your Codec Server must enforce an additional check on whether the authenticated user has
permissions for the requested Namespace. This applies regardless of which authentication approach you use.

### Key management

You may also need [key management infrastructure](/key-management) to share encryption keys between your Workers and the
Codec Server.

## SDK Codec Server samples

Most Temporal SDKs provide example Codec Server implementations:

- [Go](https://github.com/temporalio/samples-go/tree/main/codec-server)
- [Java](https://github.com/temporalio/sdk-java/tree/main/temporal-remote-data-encoder)
- [Python](https://github.com/temporalio/samples-python/blob/main/encryption/codec_server.py) | [Python with External Storage](https://github.com/temporalio/samples-python/tree/main/external_storage)
- [TypeScript](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/codec-server.ts)
- [.NET](https://github.com/temporalio/samples-dotnet/blob/main/src/Encryption/CodecServer/Program.cs)
- [Ruby](https://github.com/temporalio/samples-ruby/blob/main/encryption/codec_server.rb)

---

# Temporal Cron Job

Source: https://docs.temporal.io/cron-job

> A Cron Schedule repeats a Workflow Execution like a unix cron job; see why Schedules are now the recommended alternative.

This page discusses [Cron Job](#temporal-cron-job) including [Cron Schedules](#cron-schedules), [Time Zones](#cron-job-time-zones), and [how to stop a Cron Schedule](#stop-cron-schedules).

## What is a Temporal Cron Job? 

> **📝 Note:**
>
> We recommend using [Schedules](/schedule) instead of Cron Jobs.
> Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules.
>

A Temporal Cron Job is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution.

- [How to set a Cron Schedule using the Go SDK](/develop/go/workflows/schedules#temporal-cron-jobs)
- [How to set a Cron Schedule using the Java SDK](/develop/java/workflows/schedules#cron-schedule)
- [How to set a Cron Schedule using the PHP SDK](/develop/php/workflows/schedules#temporal-cron-jobs)
- [How to set a Cron Schedule using the Python SDK](/develop/python/workflows/schedules#temporal-cron-jobs)
- [How to set a Cron Schedule using the TypeScript SDK](/develop/typescript/workflows/schedules#temporal-cron-jobs)

![Temporal Cron Job timeline](https://docs.temporal.io/diagrams/temporal-cron-job.svg)

A Temporal Cron Job is similar to a classic unix cron job.
Just as a unix cron job accepts a command and a schedule on which to execute that command, a Cron Schedule can be provided with the call to spawn a Workflow Execution.
If a Cron Schedule is provided, the Temporal Server will spawn an execution for the associated Workflow Type per the schedule.

Each Workflow Execution within the series is considered a Run.

- Each Run receives the same input parameters as the initial Run.
- Each Run inherits the same Workflow Options as the initial Run.

The Temporal Server spawns the first Workflow Execution in the chain of Runs immediately.
However, it calculates and applies a backoff (`firstWorkflowTaskBackoff`) so that the first Workflow Task of the Workflow Execution does not get placed into a Task Queue until the scheduled time.
After each Run Completes, Fails, or reaches the [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout), the same thing happens: the next run will be created immediately with a new `firstWorkflowTaskBackoff` that is calculated based on the current Server time and the defined Cron Schedule.

The Temporal Server spawns the next Run only after the current Run has Completed, Failed, or has reached the Workflow Run Timeout.
This means that, if a Retry Policy has also been provided, and a Run Fails or reaches the Workflow Run Timeout, the Run will first be retried per the Retry Policy until the Run Completes or the Retry Policy has been exhausted.
If the next Run, per the Cron Schedule, is due to spawn while the current Run is still Open (including retries), the Server automatically starts the new Run after the current Run completes successfully.
The start time for this new Run and the Cron definitions are used to calculate the `firstWorkflowTaskBackoff` that is applied to the new Run.

A [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) is used to limit how long a Workflow can be executing (have an Open status), including retries and any usage of Continue As New.
The Cron Schedule runs until the Workflow Execution Timeout is reached or you terminate the Workflow.

![Temporal Cron Job Run Failure with a Retry Policy](https://docs.temporal.io/diagrams/temporal-cron-job-failure-with-retry.svg)

## Cron Schedules 

Cron Schedules are interpreted in UTC time by default.

The Cron Schedule is provided as a string and must follow one of two specifications:

**Classic specification**

This is what the "classic" specification looks like:

```
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
│ │ │ │ │
* * * * *
```

For example, `15 8 * * *` causes a Workflow Execution to spawn daily at 8:15 AM UTC.
Use the [crontab guru site](https://crontab.guru/) to test your cron expressions.

### `robfig` predefined schedules and intervals

You can also pass any of the [predefined schedules](https://pkg.go.dev/github.com/robfig/cron/v3#hdr-Predefined_schedules) or [intervals](https://pkg.go.dev/github.com/robfig/cron/v3#hdr-Intervals) described in the [`robfig/cron` documentation](https://pkg.go.dev/github.com/robfig/cron/v3).

```
| Schedules              | Description                                | Equivalent To |
| ---------------------- | ------------------------------------------ | ------------- |
| @yearly (or @annually) | Run once a year, midnight, Jan. 1st        | 0 0 1 1 *     |
| @monthly               | Run once a month, midnight, first of month | 0 0 1 * *     |
| @weekly                | Run once a week, midnight between Sat/Sun  | 0 0 * * 0     |
| @daily (or @midnight)  | Run once a day, midnight                   | 0 0 * * *     |
| @hourly                | Run once an hour, beginning of hour        | 0 * * * *     |
```

For example, "@weekly" causes a Workflow Execution to spawn once a week at midnight between Saturday and Sunday.

Intervals just take a string that can be accepted by [time.ParseDuration](http://golang.org/pkg/time/#ParseDuration).

```
@every <duration>
```

## Time zones 

_This feature only applies in Temporal 1.15 and up_

You can change the time zone that a Cron Schedule is interpreted in by prefixing the specification with `CRON_TZ=America/New_York` (or your [desired time zone from tz](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)). `CRON_TZ=America/New_York 15 8 * * *` therefore spawns a Workflow Execution every day at 8:15 AM New York time, subject to caveats listed below.

Consider that using time zones in production introduces a surprising amount of complexity and failure modes!
**If at all possible, we recommend specifying Cron Schedules in UTC (the default)**.

If you need to use time zones, here are a few edge cases to keep in mind:

- **Beware Daylight Saving Time:** If a Temporal Cron Job is scheduled around the time when daylight saving time (DST) begins or ends (for example, `30 2 * * *`), **it might run zero, one, or two times in a day**! The Cron library that we use does not do any special handling of DST transitions. Avoid schedules that include times that fall within DST transition periods.
  - For example, in the US, DST begins at 2 AM. When you "fall back," the clock goes `1:59 … 1:00 … 1:01 … 1:59 … 2:00 … 2:01 AM` and any Cron jobs that fall in that 1 AM hour are fired again. The inverse happens when clocks "spring forward" for DST, and Cron jobs that fall in the 2 AM hour are skipped.
  - In other time zones like Chile and Iran, DST "spring forward" is at midnight. 11:59 PM is followed by 1 AM, which means `00:00:00` never happens.
- **Self Hosting note:** If you manage your own Temporal Service, you are responsible for ensuring that it has access to current `tzdata` files. The official Docker images are built with [tzdata](https://docs.w3cub.com/go/time/tzdata/index) installed (provided by Alpine Linux), but ultimately you should be aware of how tzdata is deployed and updated in your infrastructure.
- **Updating Temporal:** If you use the official Docker images, note that an upgrade of the Temporal Service may include an update to the tzdata files, which may change the meaning of your Cron Schedule. You should be aware of upcoming changes to the definitions of the time zones you use, particularly around daylight saving time start/end dates.
- **Absolute Time Fixed at Start:** The absolute start time of the next Run is computed and stored in the database when the previous Run completes, and is not recomputed. This means that if you have a Cron Schedule that runs very infrequently, and the definition of the time zone changes between one Run and the next, the Run might happen at the wrong time. For example, `CRON_TZ=America/Los_Angeles 0 12 11 11 *` means "noon in Los Angeles on November 11" (normally not in DST). If at some point the government makes any changes (for example, move the end of DST one week later, or stay on permanent DST year-round), the meaning of that specification changes. In that first year, the Run happens at the wrong time, because it was computed using the older definition.

## How to stop a Temporal Cron Job 

A Temporal Cron Job does not stop spawning Runs until it has been Terminated or until the [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) is reached.

A Cancellation Request affects only the current Run.

Use the Workflow Id in any requests to Cancel or Terminate.

---

# How does Temporal handle application data?

Source: https://docs.temporal.io/dataconversion

> This guide explores Data Converters in the Temporal Platform, detailing how they handle serialization and encoding for Workflow inputs and outputs, ensuring data stays secure and manageable.

This guide provides an overview of data handling using a Data Converter on the Temporal Platform.

Data Converters in Temporal are SDK components that handle the serialization and encoding of data entering and exiting a Temporal Service.
Workflow inputs and outputs need to be serialized and deserialized so they can be sent as JSON to a Temporal Service.

![Data Converter encodes and decodes data](https://docs.temporal.io/diagrams/default-data-converter.svg)

The Data Converter encodes data from your application to a [Payload](/dataconversion#payload) before it is sent to the Temporal Service in the Client call.
When the Temporal Server sends the encoded data back to the Worker, the Data Converter decodes it for processing within your application.
This ensures that all your sensitive data exists in its original format only on hosts that you control.

Data Converter steps are followed when data is sent to a Temporal Service (as input to a Workflow) and when it is returned from a Workflow (as output).
Due to how Temporal provides access to Workflow output, this implementation is asymmetric:

- Data encoding is performed automatically using the default converter provided by Temporal or your custom Data Converter when passing input to a Temporal Service. For example, plain text input is usually serialized into a JSON object.
- Data decoding may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Service. Instead, they are stored encoded on the Temporal Service, and you need to provide an additional parameter when using [`temporal workflow show`](/cli/command-reference/workflow#show) or when browsing the Web UI to view output.

Each piece of data (like a single argument or return value) is encoded as a [Payload](/dataconversion#payload), which consists of binary data and key-value metadata.

For details, see the API references:

- [Go](https://pkg.go.dev/go.temporal.io/sdk/converter#DataConverter)
- [Java](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html)
- [Python](https://python.temporal.io/temporalio.converter.DataConverter.html)
- [TypeScript](https://typescript.temporal.io/api/interfaces/common.DataConverter)

### What is a Payload? 

A [Payload](https://api-docs.temporal.io/#temporal.api.common.v1.Payload) represents binary data such as input and output from Activities and Workflows.
Payloads also contain metadata that describe their data type or other parameters for use by custom encoders/converters.

When processed through the SDK, the [default Data Converter](/default-custom-data-converters#default-data-converter) serializes your data/value to a Payload before sending it to the Temporal Server.
The default Data Converter processes supported type values to Payloads. You can create a custom [Payload Converter](/payload-converter) to apply different conversion steps.

You can additionally apply [custom codecs](/payload-codec), such as for encryption or compression, on your Payloads.

---

# Default and Custom Data Converters

Source: https://docs.temporal.io/default-custom-data-converters

> Learn about the default Data Converter in Temporal SDKs and how to implement a custom Data Converter for custom serialization and encoding needs.

This page discusses the following:

- [Default Data Converter](#default-data-converter)
- [Custom Data Converter](#custom-data-converter)

## What is a default Data Converter? 

Each Temporal SDK includes and uses a default Data Converter.
The default Data Converter converts objects to bytes using a series of Payload Converters and supports binary, Protobufs, and JSON formats.
It encodes values in the following order:

- Null
- Byte array
- Protobuf JSON
- JSON

In SDKs that cannot determine parameter types at runtime (for example, TypeScript), Protobufs aren't included in the default converter.

For example:

- If a value is an instance of a Protobuf message, it is encoded with [proto3 JSON](https://developers.google.com/protocol-buffers/docs/proto3#json).
- If a value isn't null, binary, or a Protobuf, it is encoded as JSON. Most common input types — including strings, integers, floating point numbers, and booleans — are serializable as JSON. If any part of it is not serializable as JSON,  an error is thrown.

The default Data Converter serializes objects based on their root type, rather than nested types.
The JSON serializers of some SDKs cannot process lists with Protobuf children objects without implementing a [custom Data Converter](#custom-data-converter).

## What is a custom Data Converter? 

A custom Data Converter extends the default Data Converter with custom logic for [Payload](/dataconversion#payload) conversion or encoding.

You can create a custom Data Converter to alter formats (for example, using [MessagePack](https://msgpack.org/) instead of JSON) or add compression and encryption.

A Payload Codec encodes and decodes [Payloads](/dataconversion#payload), with bytes-to-bytes conversion.
To use custom encryption or compression logic, create a custom Payload Codec with your encryption/compression logic in the `encode` function and your decryption/decompression logic in the `decode` function.
To implement a custom Payload Codec, you can override the default Data Converter, or create a customized Data Converter that defines its own Payload Converter.

Custom Data Converters are not applied to all data; for example, [Search Attributes](/search-attribute) are persisted unencoded so they can be indexed for searching.

A customized Data Converter can have the following three components:

- [Payload Converter](/payload-converter)
- [Payload Codec](/payload-codec)
- [Failure Converter](/failure-converter)

For details on how to implement custom encryption and compression in your SDK, see [Data Encryption](/production-deployment/data-encryption).

---

# Visualize an Activity Retry Policy with timeouts

Source: https://docs.temporal.io/demos/activity-retry-simulator

> Visualize Activity Execution times, experiment with timeouts, Retry Policies, and simulate scenarios with our tool. Configure retries and see their impact on success or failure.

Use this tool to visualize total Activity Execution times and experiment with different Activity timeouts and Retry Policies.
For a list of Activity Task Execution times, use [this calculator](https://temporal-time.netlify.app/?initialInterval=1&maxInterval=100&maxReties=10&backoffCoefficient=2).

The simulator is based on a common Activity use-case, which is to call a third party HTTP API and return the results.
See the example code snippets below.

Use the Activity Retries settings to configure how long the API request takes to succeed or fail.
There is an option to generate scenarios.
The _Task Time in Queue_ simulates the time the Activity Task might be waiting in the Task Queue.

Use the Activity Timeouts and Retry Policy settings to see how they impact the success or failure of an Activity Execution.

---

# Task Queue Priority and Fairness - Interactive Walkthrough

Source: https://docs.temporal.io/demos/priority-fairness-walkthrough

> Interactively explore how Task Queue Priority and Fairness control dispatch order. Step through scenarios, see which task gets picked next and why, then grab the SDK code.

---

# Standalone Activities Demo

Source: https://docs.temporal.io/demos/standalone-activities

> An interactive overview of Temporal Standalone Activities.

Standalone Activities let you run a single Activity straight from your application without
writing a Workflow. Your code uses the Temporal Client to send the request to the Server, the Server durably enqueues the request
for a Worker to pick up, and the result comes back through a handle that your code can wait on or
check later.

Try the demo below to walk through the full flow, tweak the retry and timeout settings, and watch
the SDK code and CLI command update as you go.

---

## How it works

When you call `client.execute_activity()` (or the equivalent in your applicable SDK) from your
application, the following happens:

1. **Connect**: Your application opens a connection to the Temporal Server using a Temporal Client
   configured with your namespace and credentials.
2. **Schedule**: The Server durably persists the Activity Task on the specified Task Queue so that
   the request survives Worker restarts and network interruptions.
3. **Poll**: A Worker that is polling that Task Queue picks up the Activity Task and prepares to
   execute it.
4. **Execute**: The Worker runs your Activity function with the provided arguments and reports the
   outcome back to the Server.
5. **Return**: The Server stores the result and returns it to the original caller, either directly
   or via a handle, depending on which SDK method you use.

### Standalone vs Workflow Activities

| | Workflow Activity | Standalone Activity |
|---|---|---|
| Orchestrated by | A Workflow Definition | Your application code (via the Temporal Client) |
| Started with | `workflow.execute_activity()` (or the equivalent in your applicable SDK) from inside a Workflow Definition | `client.execute_activity()` (or the equivalent in your applicable SDK) from your application code |
| Retry policy | Set when calling the Activity from inside a Workflow | Set when calling the Activity from your application |
| Visibility | Shown in the Workflow's Event History | Shown in the Standalone Activity list and count views |
| Use case | Multi-step orchestration with multiple Activities | Single, independent jobs like sending an email or processing a webhook |

The Activity function and Worker registration are **identical** for both approaches, and only the
execution path that triggers the Activity differs between them. If the Activity fails, the Server
automatically retries it according to the Retry Policy you configure.

---

## Next steps

For complete API reference and advanced usage, see the SDK-specific guides:

- [Standalone Activities - Go](/develop/go/activities/standalone-activities)
- [Standalone Activities - Java](/develop/java/activities/standalone-activities)
- [Standalone Activities - Python](/develop/python/activities/standalone-activities)
- [Standalone Activities - Ruby](/develop/ruby/activities/standalone-activities)
- [Standalone Activities - Rust](/develop/rust/activities/standalone-activities)
- [Standalone Activities - TypeScript](/develop/typescript/activities/standalone-activities)
- [Standalone Activities - .NET](/develop/dotnet/activities/standalone-activities)

---

# Worker tuning simulator

Source: https://docs.temporal.io/demos/worker-tuning-simulator

> Change Poller, Slot, Workflow cache, CPU, memory, and I/O settings on a simulated Worker and watch Task throughput and Task Queue backlog respond.

This simulator shows how Worker settings affect Task throughput and Task Queue backlog.
It models [Pollers](/develop/worker-performance#task-pollers), [Slots](/develop/worker-performance#slots), the [Workflow cache](/develop/worker-performance/workflow-cache), and CPU, memory, and I/O capacity on a Worker, and runs entirely in your browser.

The simulator is a teaching tool.
It doesn't connect to Temporal, read metrics from your Workers, or recommend production settings.
The numbers show relative behavior, so use them to see which limit a Worker hits first, not to size a real deployment.

Press **Start continuous** to begin generating Workflows, then change a setting while the simulation runs.
The page works best on a desktop-width screen.

## What to try

Use the **Preset** menu to load a scenario that is limited by one resource, run it until a backlog builds, then open **Custom settings** and raise the limit the preset names.
Each preset causes Task Queue backlog for a different reason:

- **Poller constrained**: Too few Activity Pollers for the generated Activity Task rate. Add Activity Pollers and watch the backlog. On a real Worker, [Poller Autoscaling](/develop/worker-tuning-reference#poller-autoscaling) adjusts the Poller count for you.
- **Slot constrained**: Few [Slots](/develop/worker-performance#slots), so Tasks wait even when Pollers are available. Compare adding Slots with adding Pollers.
- **CPU constrained** and **I/O constrained**: Tasks that demand more CPU or I/O than the Worker has. The resource charts show when the Worker is throttled.
- **Memory constrained**: A small Workflow cache on a Worker with little memory. Watch cache hits, evictions, and replays. See [Sticky Execution](/sticky-execution) for why the cache matters.

Expand **Monitoring** to explore common Temporal metrics for Worker capacity, including Poller busy ratio, Slot utilization, Sync Match rate, and Task Queue backlog.
Compare these signals with Worker resource utilization, Task throughput, and schedule-to-completion latency.

## Learn more

The simulator models behavior you can configure and measure on a real Worker:

- **Pollers and Slots**: [Worker performance concepts](/develop/worker-performance#worker-performance-concepts), [Worker performance options](/develop/worker-performance/configuration#configuring-poller-options), and [Poller Autoscaling](/develop/worker-tuning-reference#poller-autoscaling).
- **Slot suppliers**: [Worker runtime performance tuning](/develop/worker-performance/runtime-tuning).
- **Workflow cache**: [Workflow cache tuning](/develop/worker-performance/workflow-cache) and [Sticky Execution](/sticky-execution).
- **CPU, memory, and I/O**: the [compute](/develop/worker-tuning-reference#compute-settings), [memory](/develop/worker-tuning-reference#memory-settings), and [I/O](/develop/worker-tuning-reference#io-settings) settings in the Worker tuning quick reference.
- **Task Queues and Tasks**: [Task Queues](/task-queue), [Tasks](/tasks), and [Task Queue performance](/develop/worker-performance/task-queues).
- **Metrics**: the monitoring charts are the simulator's own versions of Worker signals, so they don't match SDK metric names one to one. For the real metrics, see [Metrics reference by resource type](/develop/worker-tuning-reference#metrics-reference-by-resource-type), [Performance metrics for tuning](/develop/worker-performance/metrics), and the [SDK metrics reference](/references/sdk-metrics).
- **Diagnosing a running system**: [Troubleshoot Worker capacity](/troubleshooting/worker-capacity), [Alerting on Worker metrics](/best-practices/worker-alerting), and [Worker deployment and performance](/best-practices/worker).

---

# Temporal Design Patterns

Source: https://docs.temporal.io/design-patterns

> A catalog of common, reusable, and proven design patterns for Temporal Workflows, organized by problem domain.

Temporal provides a set of durable execution primitives that you can compose into common, reusable, and proven patterns.
Having these patterns in your toolbox helps you solve recurring problems in a battle-tested way.

## Task orchestration patterns

- [Child Workflows](/design-patterns/child-workflows): Decomposes complex Workflows into smaller, reusable units. Each child has an independent Workflow ID, history, and lifecycle.
- [Parallel Execution](/design-patterns/parallel-execution): Executes multiple Activities concurrently for maximum throughput with error handling and controlled parallelism.
- [Pick First (Race)](/design-patterns/pick-first): Races multiple Activities in parallel, returns the first completed result, and cancels the remaining Activities using SDK cancellation scopes.

## Workflow messaging patterns

- [Signal with Start](/design-patterns/signal-with-start): Starts a Workflow when Signaling it if it does not already exist. If already running, it receives the Signal directly.
- [Request-Response via Updates](/design-patterns/request-response-via-updates): Uses a Workflow Update to validate a request, modify Workflow state, and return a typed result to the caller synchronously, with strong consistency.
- [Event Accumulator](/design-patterns/event-accumulator): Durably collect and deduplicate signals from multiple senders, then process the batch after a sliding inactivity timeout.

## Entity & lifecycle patterns

- [Entity Workflow](/design-patterns/entity-workflow): A long-lived business entity — a user account, device, or order — gets one Workflow per instance, with Signals and Updates driving every state transition.
- [Continue-As-New](/design-patterns/continue-as-new): Prevents unbounded history growth by completing the current execution and starting a new one with fresh history.
- [Updatable Timer](/design-patterns/updatable-timer): Dynamically adjustable timers that respond to Signals or Updates. Extend, shorten, or cancel timers based on external events.

## External interaction patterns

- [Polling External Services](/design-patterns/polling): Strategies for polling external resources with varying frequencies: frequent, infrequent, and periodic patterns.
- [Long-Running Activity](/design-patterns/long-running-activity): Long-running Activities report progress via heartbeats and enable resumption after failures with cancellation support.
- [Delayed Start](/design-patterns/delayed-start): Creates Workflows immediately but defers execution until a specified delay expires. Fits one-time scheduled operations and grace periods.
- [Delayed Callback (Webhooks)](/design-patterns/delayed-callback): Webhooks become durable: inbound calls arrive as Signals, outbound calls fire after a durable sleep, and Activities complete later via task tokens.
- [Approval](/design-patterns/approval): Human-in-the-loop Workflows that block until external approval decisions are made. Uses Signals to capture approval data with metadata.

## Distributed transaction patterns

- [Saga Pattern](/design-patterns/saga-pattern): Manages distributed transactions with compensating actions. Each step has a compensation that undoes its effects if subsequent steps fail.
- [Early Return](/design-patterns/early-return): Synchronous initialization with asynchronous completion. Returns results immediately while processing continues in the background.

## Error handling & retry patterns

- [Fixed Count of Retries](/design-patterns/fixed-count-retries): Cap the number of Activity retry attempts to control cost when each attempt consumes a paid or limited resource.
- [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Bound the total elapsed time across all retry attempts to enforce a business SLA, regardless of how many individual attempts occur.
- [Non-Retryable Errors](/design-patterns/non-retryable-errors): Mark error types that will never succeed — such as validation failures or missing records — so Temporal fails fast instead of retrying indefinitely.
- [Delayed Retry](/design-patterns/delayed-retry): Override one failure's retry interval with nextRetryDelay on ApplicationFailure, matching the wait time the error reports.
- [Fast/Slow Retries](/design-patterns/fast-slow-retries): Retry fast with a short interval first, then shift to a slow, unlimited interval so the Workflow outlasts an extended downstream outage.
- [Retry Alerting via Metrics](/design-patterns/retry-metrics): Emit a metric from the Activity when attempts cross a threshold, so on-call teams see persistent failures before an SLA breach.
- [Resumable Activity](/design-patterns/resumable-activity): Park the Workflow after retries are exhausted and wait for a human to signal a correction, then resume execution from where it left off.

## Batch processing patterns

- [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows): Distributes a large record set across parallel Child Workflows for concurrent processing with automatic scaling.
- [Batch Iterator](/design-patterns/batch-iterator): Pages through unbounded datasets using Continue-As-New to prevent history overflow while maintaining exactly-once processing guarantees.
- [Sliding Window](/design-patterns/sliding-window): Maintains a fixed number of concurrently active Child Workflows, starting a new one each time an existing one completes.
- [MapReduce Tree](/design-patterns/mapreduce-tree): Recursively splits a dataset into a binary tree of Child Workflows, processes leaves in parallel, then aggregates results back up the tree.

## QoS & throughput patterns

- [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting): Rate-limits calls to a downstream service by routing throttled Activities to a dedicated Task Queue with a server-enforced throughput cap.
- [Priority](/design-patterns/priority-task-queues): Assigns a priority level to Workflows and Activities so that time-sensitive work dispatches ahead of lower-priority work within a single Task Queue.
- [Fairness](/design-patterns/fairness): Distributes Task dispatches across tenants or users so that a burst from one caller does not starve the others.

## Performance & latency patterns

- [Local Activities](/design-patterns/local-activities): Local Activities run inside the Worker process, skipping server round-trips for short, idempotent Activities on a latency-sensitive path.
- [Early Return + Local Activities](/design-patterns/early-return-local-activities): Extends Early Return by running Phase 1 as Local Activities, so the client's first response comes from in-process work, not a server call.
- [Eager Workflow Start](/design-patterns/eager-workflow-start): Eager Workflow Start sends the first Workflow Task directly to a co-located Worker, skipping the Matching Service to cut startup latency.

## Worker configuration patterns

- [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue): Routes Activities to specific Workers using unique Task Queues for Worker affinity and host-specific processing.
- [Activity Dependency Injection](/design-patterns/activity-dependency-injection): Injects external dependencies into Activities at Worker startup, keeping Workflow Definitions deterministic and Activities testable.

---

# Activity Dependency Injection

Source: https://docs.temporal.io/design-patterns/activity-dependency-injection

> Injects external dependencies into Activities at Worker startup, keeping Workflow Definitions deterministic and Activities testable.

## Overview

The Activity Dependency Injection pattern separates the creation of external dependencies (database connections, API clients, configuration) from Activity business logic by injecting them at Worker startup.
This approach keeps Workflow Definitions deterministic, makes Activities testable in isolation, and ensures expensive resources are initialized once per Worker process rather than once per Activity execution.

## Problem

Activities often need access to external resources such as database connection pools, HTTP clients, third-party API credentials, or shared caches.
Without a structured approach, you face several challenges:

- **Reinitializing resources per execution.** Creating a new database connection or API client on every Activity invocation wastes resources and increases latency.
- **Hardcoded dependencies.** Embedding connection logic directly inside Activity functions couples business logic to infrastructure, making it difficult to swap implementations across environments.
- **Difficult testing.** When Activities construct their own dependencies internally, you cannot substitute test doubles without modifying production code.
- **Non-determinism risk.** Passing dependencies directly into Workflow code breaks Temporal's determinism guarantees, because dependency state can change between replays.

## Solution

You define Activities as methods on a struct or class that holds dependencies as fields. At Worker startup, you instantiate the struct or class with real implementations and register it with the Worker. The Workflow references Activity methods without knowing about the underlying dependencies.

```mermaid
flowchart LR
    subgraph Worker Startup
        D[Dependencies<br/>DB, API Client, Config] --> S[Activity Struct / Class]
        S --> R[Register with Worker]
    end

    subgraph Workflow Execution
        W[Workflow] -->|"execute activity"| A[Activity Method]
        A -->|"uses"| S
    end

    subgraph Testing
        M[Mock / Stub] --> T[Test Environment]
        T -->|"execute activity"| A2[Activity Method]
    end
```

The following describes each path in the diagram:

1. At Worker startup, you create dependency instances (database pools, API clients) and inject them into an Activity struct or class, which you then register with the Worker.
2. During Workflow execution, the Workflow calls Activity methods by reference. The Temporal runtime routes the call to the registered instance on the Worker, where the method accesses the injected dependencies.
3. During testing, you substitute mock or stub implementations into the same Activity struct or class, allowing you to verify behavior without external services.

## Implementation

### Define Activities with dependencies

Define Activities as methods on a struct or class that accepts dependencies through its constructor or fields. Each method acts as a separate Activity Type.

**Go**

```go
// activities.go
package payment

import (
	"context"

	"go.temporal.io/sdk/activity"
)

type Activities struct {
	DBClient    DBClient
	EmailClient EmailClient
}

func (a *Activities) ChargeCustomer(ctx context.Context, orderID string, amount int) (string, error) {
	logger := activity.GetLogger(ctx)
	logger.Info("Charging customer", "orderID", orderID, "amount", amount)

	receiptID, err := a.DBClient.ProcessPayment(orderID, amount)
	if err != nil {
		return "", err
	}

	return receiptID, nil
}

func (a *Activities) SendReceipt(ctx context.Context, email string, receiptID string) error {
	return a.EmailClient.Send(email, "Payment Receipt", receiptID)
}
```

**Python**

```python
# activities.py
from dataclasses import dataclass
from temporalio import activity

@dataclass
class PaymentActivities:
    db_client: DBClient
    email_client: EmailClient

    @activity.defn
    async def charge_customer(self, order_id: str, amount: int) -> str:
        activity.logger.info(
            "Charging customer", extra={"order_id": order_id, "amount": amount}
        )
        receipt_id = await self.db_client.process_payment(order_id, amount)
        return receipt_id

    @activity.defn
    async def send_receipt(self, email: str, receipt_id: str) -> None:
        await self.email_client.send(email, "Payment Receipt", receipt_id)
```

**Java**

```java
// PaymentActivities.java
@ActivityInterface
public interface PaymentActivities {
    String chargeCustomer(String orderId, int amount);
    void sendReceipt(String email, String receiptId);
}

// PaymentActivitiesImpl.java
public class PaymentActivitiesImpl implements PaymentActivities {
    private final DBClient dbClient;
    private final EmailClient emailClient;

    public PaymentActivitiesImpl(DBClient dbClient, EmailClient emailClient) {
        this.dbClient = dbClient;
        this.emailClient = emailClient;
    }

    @Override
    public String chargeCustomer(String orderId, int amount) {
        return dbClient.processPayment(orderId, amount);
    }

    @Override
    public void sendReceipt(String email, String receiptId) {
        emailClient.send(email, "Payment Receipt", receiptId);
    }
}
```

**TypeScript**

```typescript
// activities.ts
export interface DB {
  processPayment(orderId: string, amount: number): Promise<string>;
}

export interface EmailClient {
  send(to: string, subject: string, body: string): Promise<void>;
}

export const createActivities = (db: DB, emailClient: EmailClient) => ({
  async chargeCustomer(orderId: string, amount: number): Promise<string> {
    const receiptId = await db.processPayment(orderId, amount);
    return receiptId;
  },

  async sendReceipt(email: string, receiptId: string): Promise<void> {
    await emailClient.send(email, 'Payment Receipt', receiptId);
  },
});
```

Each SDK uses a different mechanism to group Activities with their dependencies:

- **Go**: Methods on a struct. The struct fields hold dependencies.
- **Python**: A `@dataclass` with `@activity.defn` methods. Fields hold dependencies.
- **Java**: An `@ActivityInterface` with a separate implementation class. Dependencies are passed through the constructor.
- **TypeScript**: A factory function that closes over dependencies and returns an object of Activity functions.

### Reference Activities from the Workflow

The Workflow references Activity methods without any knowledge of the injected dependencies. Each SDK provides a type-safe way to call Activities.

**Go**

```go
// workflow.go
package payment

import (
	"time"

	"go.temporal.io/sdk/workflow"
)

func PaymentWorkflow(ctx workflow.Context, orderID string, amount int, email string) error {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 30 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	// Use a nil struct pointer to reference Activity methods.
	// This provides compile-time type safety without instantiating the struct.
	var a *Activities
	var receiptID string
	err := workflow.ExecuteActivity(ctx, a.ChargeCustomer, orderID, amount).Get(ctx, &receiptID)
	if err != nil {
		return err
	}

	return workflow.ExecuteActivity(ctx, a.SendReceipt, email, receiptID).Get(ctx, nil)
}
```

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import PaymentActivities

@workflow.defn
class PaymentWorkflow:
    @workflow.run
    async def run(self, order_id: str, amount: int, email: str) -> None:
        receipt_id = await workflow.execute_activity_method(
            PaymentActivities.charge_customer,
            args=[order_id, amount],
            start_to_close_timeout=timedelta(seconds=30),
        )

        await workflow.execute_activity_method(
            PaymentActivities.send_receipt,
            args=[email, receipt_id],
            start_to_close_timeout=timedelta(seconds=30),
        )
```

**Java**

```java
// PaymentWorkflowImpl.java
public class PaymentWorkflowImpl implements PaymentWorkflow {
    private final PaymentActivities activities = Workflow.newActivityStub(
        PaymentActivities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .build()
    );

    @Override
    public void processPayment(String orderId, int amount, String email) {
        String receiptId = activities.chargeCustomer(orderId, amount);
        activities.sendReceipt(email, receiptId);
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities } from '@temporalio/workflow';
import type { createActivities } from './activities';

// Use ReturnType to extract the Activity types from the factory function
const { chargeCustomer, sendReceipt } = proxyActivities<
  ReturnType<typeof createActivities>
>({
  startToCloseTimeout: '30s',
});

export async function paymentWorkflow(
  orderId: string,
  amount: number,
  email: string
): Promise<void> {
  const receiptId = await chargeCustomer(orderId, amount);
  await sendReceipt(email, receiptId);
}
```

Key points for each SDK:

- **Go**: A nil pointer of the Activity struct type (`var a *Activities`) provides compile-time method references without instantiating the struct. The Temporal runtime resolves the actual registered instance at execution time.
- **Python**: `workflow.execute_activity_method` references the class method directly and resolves to the registered instance on the Worker.
- **Java**: `Workflow.newActivityStub` creates a typed proxy from the Activity interface. The Temporal runtime routes calls to the registered implementation.
- **TypeScript**: `proxyActivities<ReturnType<typeof createActivities>>` infers the Activity types from the factory function's return type. Activities are always referenced by name at runtime.

### Register Activities with the Worker

At Worker startup, you instantiate the Activity struct or class with real dependency implementations and register it.

**Go**

```go
// worker/main.go
package main

import (
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"

	"example/payment"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, "payment", worker.Options{})

	w.RegisterWorkflow(payment.PaymentWorkflow)

	// Inject real dependencies at Worker startup
	w.RegisterActivity(&payment.Activities{
		DBClient:    payment.NewPostgresClient("postgres://localhost:5432/payments"),
		EmailClient: payment.NewSMTPClient("smtp://mail.example.com"),
	})

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```

**Python**

```python
# worker.py
import asyncio
from temporalio.client import Client
from temporalio.worker import Worker

from activities import PaymentActivities
from workflows import PaymentWorkflow

async def main():
    client = await Client.connect("localhost:7233")

    # Inject real dependencies at Worker startup
    payment_activities = PaymentActivities(
        db_client=PostgresClient("postgres://localhost:5432/payments"),
        email_client=SMTPClient("smtp://mail.example.com"),
    )

    worker = Worker(
        client,
        task_queue="payment",
        workflows=[PaymentWorkflow],
        activities=[
            payment_activities.charge_customer,
            payment_activities.send_receipt,
        ],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**Java**

```java
// PaymentWorker.java
public class PaymentWorker {
    public static void main(String[] args) {
        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service);
        WorkerFactory factory = WorkerFactory.newInstance(client);

        Worker worker = factory.newWorker("payment");
        worker.registerWorkflowImplementationTypes(PaymentWorkflowImpl.class);

        // Inject real dependencies at Worker startup
        worker.registerActivitiesImplementations(
            new PaymentActivitiesImpl(
                new PostgresClient("postgres://localhost:5432/payments"),
                new SMTPClient("smtp://mail.example.com")
            )
        );

        factory.start();
    }
}
```

**TypeScript**

```typescript
// worker.ts
import { Worker } from '@temporalio/worker';
import { createActivities } from './activities';

async function run() {
  // Initialize dependencies at Worker startup
  const db = new PostgresClient('postgres://localhost:5432/payments');
  const emailClient = new SMTPClient('smtp://mail.example.com');

  const worker = await Worker.create({
    taskQueue: 'payment',
    workflowsPath: require.resolve('./workflows'),
    // Inject dependencies through the factory function
    activities: createActivities(db, emailClient),
  });

  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Dependencies are initialized once when the Worker process starts. All Activity executions on that Worker share the same instances, which is appropriate for thread-safe resources like connection pools and HTTP clients.

### Inject a circuit breaker

A circuit breaker is a natural fit for Worker-level injection because it is a *stateful* dependency. It tracks recent failures for a downstream service and, once the failure rate crosses a threshold, trips to an open state that rejects calls immediately for a cool-down period before allowing a probe. That state only protects the service if it is shared across every Activity execution on the Worker. Constructing a new breaker inside each Activity method would reset the counters on every call, so the breaker would never trip.

By injecting a single breaker instance alongside the client it guards, all executions of the Activity feed the same failure window. When the downstream service degrades, the breaker opens and Activities fail fast instead of piling up latency against an unhealthy dependency, which in turn lets Temporal's retry policy back off rather than hammering the service.

The following examples wrap an outbound payment API call in a circuit breaker, using an established library for each language:

**Go**

```go
// activities.go — github.com/sony/gobreaker/v2
package payment

import (
	"context"

	"github.com/sony/gobreaker/v2"
	"go.temporal.io/sdk/activity"
)

type Activities struct {
	PaymentAPI PaymentAPI
	Breaker    *gobreaker.CircuitBreaker[string]
}

func (a *Activities) ChargeCustomer(ctx context.Context, orderID string, amount int) (string, error) {
	logger := activity.GetLogger(ctx)
	logger.Info("Charging customer", "orderID", orderID, "amount", amount)

	// The breaker rejects the call immediately when it is open,
	// returning gobreaker.ErrOpenState without touching the API.
	return a.Breaker.Execute(func() (string, error) {
		return a.PaymentAPI.Charge(orderID, amount)
	})
}
```

**Python**

```python
# activities.py — pybreaker
import pybreaker
from dataclasses import dataclass
from temporalio import activity

@dataclass
class PaymentActivities:
    payment_api: PaymentAPI
    breaker: pybreaker.CircuitBreaker

    @activity.defn
    async def charge_customer(self, order_id: str, amount: int) -> str:
        activity.logger.info(
            "Charging customer", extra={"order_id": order_id, "amount": amount}
        )
        # Raises pybreaker.CircuitBreakerError when the breaker is open.
        return await self.breaker.call_async(
            self.payment_api.charge, order_id, amount
        )
```

**Java**

```java
// PaymentActivitiesImpl.java — resilience4j
import io.github.resilience4j.circuitbreaker.CircuitBreaker;

public class PaymentActivitiesImpl implements PaymentActivities {
    private final PaymentAPI paymentApi;
    private final CircuitBreaker breaker;

    public PaymentActivitiesImpl(PaymentAPI paymentApi, CircuitBreaker breaker) {
        this.paymentApi = paymentApi;
        this.breaker = breaker;
    }

    @Override
    public String chargeCustomer(String orderId, int amount) {
        // Throws CallNotPermittedException when the breaker is open.
        return breaker.executeSupplier(() -> paymentApi.charge(orderId, amount));
    }
}
```

**TypeScript**

```typescript
// activities.ts — opossum
import CircuitBreaker from 'opossum';

export const createActivities = (breaker: CircuitBreaker) => ({
  async chargeCustomer(orderId: string, amount: number): Promise<string> {
    // Rejects with an EOPENBREAKER error when the breaker is open.
    return breaker.fire(orderId, amount) as Promise<string>;
  },
});
```

You construct the breaker at Worker startup and inject it the same way as any other dependency, so its state lives for the lifetime of the Worker process.

**Go**

```go
// worker/main.go
package main

import (
	"log"
	"time"

	"github.com/sony/gobreaker/v2"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"

	"example/payment"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, "payment", worker.Options{})

	w.RegisterWorkflow(payment.PaymentWorkflow)

	// Construct the breaker once at Worker startup so its failure
	// counters are shared across all Activity executions.
	breaker := gobreaker.NewCircuitBreaker[string](gobreaker.Settings{
		Name:        "payment-api",
		MaxRequests: 3,                // probes allowed while half-open
		Interval:    60 * time.Second, // window for counting failures
		Timeout:     30 * time.Second, // cool-down before half-open
	})

	w.RegisterActivity(&payment.Activities{
		PaymentAPI: payment.NewPaymentAPI("https://api.example.com"),
		Breaker:    breaker,
	})

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```

**Python**

```python
# worker.py
import asyncio

import pybreaker
from temporalio.client import Client
from temporalio.worker import Worker

from activities import PaymentActivities, PaymentAPI
from workflows import PaymentWorkflow

async def main():
    client = await Client.connect("localhost:7233")

    # Construct the breaker once at Worker startup so its failure
    # counters are shared across all Activity executions.
    breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)
    payment_activities = PaymentActivities(
        payment_api=PaymentAPI("https://api.example.com"),
        breaker=breaker,
    )

    worker = Worker(
        client,
        task_queue="payment",
        workflows=[PaymentWorkflow],
        activities=[
            payment_activities.charge_customer,
            payment_activities.send_receipt,
        ],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**Java**

```java
// PaymentWorker.java
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.temporal.client.WorkflowClient;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;
import java.time.Duration;

public class PaymentWorker {
    public static void main(String[] args) {
        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service);
        WorkerFactory factory = WorkerFactory.newInstance(client);

        Worker worker = factory.newWorker("payment");
        worker.registerWorkflowImplementationTypes(PaymentWorkflowImpl.class);

        // Construct the breaker once at Worker startup so its failure
        // counters are shared across all Activity executions.
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
            .failureRateThreshold(50)                        // percent
            .waitDurationInOpenState(Duration.ofSeconds(30)) // cool-down
            .slidingWindowSize(20)
            .build();
        CircuitBreaker breaker = CircuitBreaker.of("payment-api", config);

        worker.registerActivitiesImplementations(
            new PaymentActivitiesImpl(new PaymentAPI("https://api.example.com"), breaker)
        );

        factory.start();
    }
}
```

**TypeScript**

```typescript
// worker.ts
import { Worker } from '@temporalio/worker';
import CircuitBreaker from 'opossum';
import { createActivities } from './activities';
import { PaymentAPI } from './payment-api';

async function run() {
  // Construct the breaker once at Worker startup so its failure
  // counters are shared across all Activity executions.
  const paymentApi = new PaymentAPI('https://api.example.com');
  const breaker = new CircuitBreaker(
    (orderId: string, amount: number) => paymentApi.charge(orderId, amount),
    { errorThresholdPercentage: 50, resetTimeout: 30000 }, // cool-down in ms
  );

  const worker = await Worker.create({
    taskQueue: 'payment',
    workflowsPath: require.resolve('./workflows'),
    activities: createActivities(breaker),
  });

  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Circuit breaker libraries by language:

- **Go**: [`sony/gobreaker`](https://github.com/sony/gobreaker) — the generic (`v2`) API returns the wrapped result type directly.
- **Python**: [`pybreaker`](https://github.com/danielfm/pybreaker) — `call_async` wraps coroutine-based clients.
- **Java**: [`resilience4j`](https://resilience4j.readme.io/docs/circuitbreaker) — the successor to Netflix Hystrix, built around functional decorators.
- **TypeScript**: [`opossum`](https://nodeshift.dev/opossum/) — wraps a single async action bound at construction time.

Because the breaker counts failures, size the Activity retry policy accordingly. A breaker that opens after a burst of failures pairs well with a retry policy that backs off, so an open breaker returns fast errors that Temporal retries later rather than each attempt waiting on a timeout.

## When to use

**Good fit:**

- Activities access external services such as databases, message queues, or third-party APIs
- You want to initialize expensive resources once per Worker process
- You need to test Activity logic without connecting to real services
- You operate in multiple environments (development, staging, production) that require different dependency configurations

**Poor fit:**

- Activities are pure functions with no external dependencies
- Activities only use Temporal-provided context, such as heartbeating and logging

## Benefits and trade-offs

Injecting dependencies at the Worker level provides several advantages. Resources like connection pools are initialized once and shared across all Activity executions, reducing overhead. Substituting mock implementations in tests requires no changes to Activity or Workflow code. Switching between environments involves changing only the Worker configuration.

The trade-off is that all Activity executions on a given Worker share the same dependency instances. If an Activity requires per-execution isolation (for example, a database transaction scoped to a single Activity), you need to manage that within the Activity method itself. Dependencies must also be thread-safe, since multiple Activity executions may run concurrently on the same Worker.

## Best practices

- **Keep dependencies thread-safe.** Multiple Activity executions run concurrently on the same Worker. Use connection pools rather than single connections, and avoid mutable shared state.
- **Define dependencies as interfaces.** In Go, Python, and Java, using interfaces (or protocols in Python) for dependencies makes it possible to swap implementations for testing or different environments.
- **Do not inject dependencies into Workflows.** A Workflow Definition must remain deterministic. If a Workflow needs configuration, retrieve it through a Local Activity so the value gets recorded in the Event History.
- **Initialize dependencies before Worker startup.** Create and validate all connections before calling `worker.Run()` or its equivalent. This ensures that the Worker does not start accepting tasks until all dependencies are ready.
- **Group related Activities on a single struct or class.** Activities that share the same dependencies belong together. If two groups of Activities have different dependencies, use separate structs or classes for each group.

## Common pitfalls

- **Constructing dependencies inside Activity methods.** Creating a new database connection or API client per execution leads to resource exhaustion and increased latency.
- **Injecting dependencies into Workflows.** This breaks determinism because dependency state can change between the original execution and a replay. The Temporal Java SDK documentation explicitly warns against this.
- **Using non-thread-safe dependencies.** A single mutable object shared across concurrent Activity executions causes race conditions. Use connection pools and ensure all injected objects are safe for concurrent use.
- **Registering class methods as static in Python.** If you register `BotService.send_message` (the unbound method) instead of `bot_service.send_message` (a method on an instance), the `self` parameter is not bound, causing a missing argument error at runtime.
- **Forgetting to bind methods in TypeScript.** When using a class instead of a factory function, class methods must be defined as arrow functions or explicitly bound in the constructor. Otherwise, `this` is `undefined` when Temporal invokes the Activity.

## Related

### Patterns

- **[Entity Workflow](/design-patterns/entity-workflow)**: Long-lived Workflows that manage stateful entities, often using Activities with injected dependencies.
- **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Routing Activities to specific Workers, which can have different injected dependencies.

### References

- [Activities (Go)](/develop/go/activities/basics): Struct-based Activities sharing a DB pool, client connection, or other process-level resources.
- [Activities (TypeScript)](/develop/typescript/activities/basics): The factory-function pattern for sharing dependencies between Activities.
- [Worker deployment and performance](/best-practices/worker): A reference-app example of registering an Activity struct with injected configuration.

### Sample code

### Go
- [Greetings](https://github.com/temporalio/samples-go/tree/main/greetings) — Activities as struct methods with injected dependencies.
- [Large Payload Fixture](https://github.com/temporalio/samples-go/tree/main/temporal-fixtures/largepayload) — The reference sample using struct-based Activity dependency injection.

### Python
- [Hello Activity Method](https://github.com/temporalio/samples-python/blob/main/hello/hello_activity_method.py) — Activities defined as class methods with dependency injection.

### Java
- [Hello World](https://github.com/temporalio/hello-world-project-template-java) — Activity interface and implementation with Worker registration.

### TypeScript
- [Activities Dependency Injection](https://github.com/temporalio/samples-typescript/tree/main/activities-dependency-injection) — Factory function pattern for sharing dependencies between Activities.

---

# Approval Pattern

Source: https://docs.temporal.io/design-patterns/approval

> Human-in-the-loop Workflows that block until external approval decisions are made. Uses Signals to capture approval data with metadata.

## Overview

The Approval pattern implements human-in-the-loop Workflows where execution blocks until an external decision is made.
It uses Workflow Signals with custom input data to unblock Workflows, enabling approval processes, manual reviews, and decision gates in automated business processes.

## Problem

In many business processes, you need Workflows that wait for human approval before proceeding.
These Workflows must capture approval decisions along with metadata such as the approver's identity, a reason, and a timestamp.
They must also support multiple outcomes — approval, rejection, or escalation — and handle timeout scenarios when no decision arrives.

Without a structured approval pattern, you are forced to poll external systems for approval status, implement complex state machines by hand, and manage race conditions between timeouts and incoming approvals.
You also risk losing approval context and metadata, and you must build custom audit logging to meet compliance requirements.

## Solution

The Approval pattern uses a blocking wait with timeout to pause execution until a Signal is received.
The Signal carries custom data — the approval decision, approver details, and comments — that the Workflow captures and uses to determine next steps.

```mermaid
sequenceDiagram
    participant Requester
    participant Workflow
    participant Approver

    Requester->>+Workflow: Start approval request
    activate Workflow
    Workflow->>Workflow: Wait with timeout
    Note over Workflow: Waiting for approval...

    alt Approval received
        Approver->>Workflow: Signal: submitApproval(data)
        Workflow->>Workflow: Process approval data
        Workflow-->>Requester: Approved
    else Timeout
        Note over Workflow: Timeout expires
        Workflow-->>Requester: Timeout/Rejected
    end
    deactivate Workflow
```

The following describes each step in the diagram:

1. The requester starts the Workflow with an approval request.
2. The Workflow blocks execution with a timeout — using `Workflow.await()` in Java, `condition()` in TypeScript, `workflow.wait_condition()` in Python, or `workflow.AwaitWithTimeout()` in Go.
3. If an approver sends a Signal before the timeout expires, the Workflow receives the approval data, processes the decision, and returns the result to the requester.
4. If the timeout expires before any Signal arrives, the Workflow unblocks and follows the timeout path, which typically results in rejection or escalation.

## Implementation

### Basic approval with timeout

The basic implementation captures the approval decision in a structured data object rather than a plain boolean.
Define a type to hold the approver's identity, the decision, any comments, and a timestamp:

**Python**

```python
# models.py
from dataclasses import dataclass

@dataclass
class ApprovalData:
    approver: str
    decision: str  # "APPROVED", "REJECTED", "ESCALATED"
    comments: str
    timestamp: int
```

**Go**

```go
// types.go
type ApprovalData struct {
    Approver  string
    Decision  string // "APPROVED", "REJECTED", "ESCALATED"
    Comments  string
    Timestamp int64
}
```

**Java**

```java
// ApprovalData.java
public class ApprovalData {
  private String approver;
  private String decision; // "APPROVED", "REJECTED", "ESCALATED"
  private String comments;
  private long timestamp;

  // Constructor, getters, setters
}
```

**TypeScript**

```typescript
// types.ts
export interface ApprovalData {
  approver: string;
  decision: 'APPROVED' | 'REJECTED' | 'ESCALATED';
  comments: string;
  timestamp: number;
}
```

This type gives you a structured way to pass rich context through the Signal rather than a plain boolean.

The implementation ties everything together.
The Workflow blocks until either the approval data arrives via Signal or the timeout expires:

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow
from models import ApprovalData

@workflow.defn
class ApprovalWorkflow:
    def __init__(self) -> None:
        self.approval_data: ApprovalData | None = None
        self.status = "PENDING"

    @workflow.run
    async def run(self, request_id: str, timeout_seconds: int) -> str:
        try:
            # Block until the Signal sets approval_data, or raise on timeout
            await workflow.wait_condition(
                lambda: self.approval_data is not None,
                timeout=timedelta(seconds=timeout_seconds),
            )
            self.status = self.approval_data.decision
            return f"Request {request_id} {self.status} by {self.approval_data.approver}"
        except asyncio.TimeoutError:
            self.status = "TIMEOUT"
            return f"Request {request_id} timed out"

    # Signal handler: an external approver submits the decision
    @workflow.signal
    def submit_approval(self, data: ApprovalData) -> None:
        self.approval_data = data

    # Query handler: read current status without mutating state
    @workflow.query
    def get_status(self) -> str:
        return self.status
```

**Go**

```go
// workflow.go
func ApprovalWorkflow(ctx workflow.Context, requestId string, timeout time.Duration) (string, error) {
    var approvalData *ApprovalData
    status := "PENDING"

    // Query handler: expose current status without mutating state
    err := workflow.SetQueryHandler(ctx, "getStatus", func() (string, error) {
        return status, nil
    })
    if err != nil {
        return "", err
    }

    // Receive the approval signal in a goroutine
    workflow.Go(ctx, func(ctx workflow.Context) {
        signalChan := workflow.GetSignalChannel(ctx, "submitApproval")
        signalChan.Receive(ctx, &approvalData)
    })

    // Block until the signal sets approvalData, or time out (ok=false)
    approved, err := workflow.AwaitWithTimeout(ctx, timeout, func() bool {
        return approvalData != nil
    })
    if err != nil {
        return "", err
    }

    if approved {
        status = approvalData.Decision
        return fmt.Sprintf("Request %s %s by %s", requestId, status, approvalData.Approver), nil
    }
    status = "TIMEOUT"
    return fmt.Sprintf("Request %s timed out", requestId), nil
}
```

**Java**

```java
// ApprovalWorkflowImpl.java
public class ApprovalWorkflowImpl implements ApprovalWorkflow {
  private ApprovalData approvalData;
  private String status = "PENDING";

  @Override
  public String execute(String requestId, Duration timeout) {
    // Block until the Signal sets approvalData, or time out (returns false)
    boolean approved = Workflow.await(timeout, () -> approvalData != null);

    if (approved) {
      status = approvalData.getDecision();
      return "Request " + requestId + " " + status + " by " + approvalData.getApprover();
    } else {
      status = "TIMEOUT";
      return "Request " + requestId + " timed out";
    }
  }

  // Signal handler: an external approver submits the decision
  @Override
  public void submitApproval(ApprovalData data) {
    this.approvalData = data;
  }

  // Query handler: read current status without mutating state
  @Override
  public String getStatus() {
    return status;
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import { ApprovalData } from './types';

export const submitApprovalSignal = wf.defineSignal<[ApprovalData]>('submitApproval');
export const getStatusQuery = wf.defineQuery<string>('getStatus');

export async function approvalWorkflow(
  requestId: string,
  timeout: string | number, // ms or Duration string
): Promise<string> {
  let approvalData: ApprovalData | undefined;
  let status = 'PENDING';

  // Signal handler: an external approver submits the decision
  wf.setHandler(submitApprovalSignal, (data: ApprovalData) => {
    approvalData = data;
  });

  // Query handler: read current status without mutating state
  wf.setHandler(getStatusQuery, () => status);

  // Block until the Signal sets approvalData, or time out (returns false)
  const approved = await wf.condition(() => approvalData !== undefined, timeout);

  if (approved) {
    status = approvalData!.decision;
    return `Request ${requestId} ${status} by ${approvalData!.approver}`;
  } else {
    status = 'TIMEOUT';
    return `Request ${requestId} timed out`;
  }
}
```

Each SDK uses a different mechanism to block with a timeout, but the core pattern is the same.
In Java, `Workflow.await()` takes a timeout and a condition lambda, returning `false` on timeout.
In TypeScript, `condition()` takes a predicate and a timeout, returning `false` on timeout.
In Python, `workflow.wait_condition()` takes a lambda and a timeout, raising `asyncio.TimeoutError` on timeout.
In Go, `workflow.AwaitWithTimeout()` takes a timeout and a condition function, returning `ok=false` on timeout.

The condition is evaluated on every state transition, so it must not call blocking operations, mutate Workflow state, or use time-based checks.
When the Signal handler sets the approval data, the condition evaluates to `true` and the Workflow unblocks.

### Multi-level approval chain

Some business processes require approvals from multiple levels of authority in sequence.
The following implementation iterates through a list of required approval levels, waiting for a Signal at each level before proceeding to the next:

**Python**

```python
# models.py
from dataclasses import dataclass

@dataclass
class MultiLevelApprovalData:
    level: str  # "L1", "L2", "L3"
    approver: str
    decision: str
    comments: str
```

**Go**

```go
// types.go
type MultiLevelApprovalData struct {
    Level    string // "L1", "L2", "L3"
    Approver string
    Decision string
    Comments string
}
```

**Java**

```java
// MultiLevelApprovalData.java
public class MultiLevelApprovalData {
  private String level; // "L1", "L2", "L3"
  private String approver;
  private String decision;
  private String comments;
}
```

**TypeScript**

```typescript
// types.ts
export interface MultiLevelApprovalData {
  level: 'L1' | 'L2' | 'L3';
  approver: string;
  decision: string;
  comments: string;
}
```

This data type extends the basic approval data with a `level` field that identifies which approval tier the decision belongs to.

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow
from models import MultiLevelApprovalData

@workflow.defn
class MultiLevelApprovalWorkflow:
    def __init__(self) -> None:
        self.approvals: list[MultiLevelApprovalData] = []

    @workflow.run
    async def run(self, request_id: str, timeout_per_level_seconds: int) -> str:
        required_levels = ["L1", "L2", "L3"]
        timeout = timedelta(seconds=timeout_per_level_seconds)

        # Require approval at each level in sequence
        for level in required_levels:
            try:
                # Wait for a Signal carrying this level's approval
                await workflow.wait_condition(
                    lambda lv=level: any(a.level == lv for a in self.approvals),
                    timeout=timeout,
                )
            except asyncio.TimeoutError:
                return f"Timeout at {level}"

            approval = next(a for a in self.approvals if a.level == level)
            # Stop the chain early if any level rejects
            if approval.decision == "REJECTED":
                return f"Rejected at {level} by {approval.approver}"

        return "Fully approved through all levels"

    # Signal handler: collect each level's approval as it arrives
    @workflow.signal
    def submit_approval(self, data: MultiLevelApprovalData) -> None:
        self.approvals.append(data)
```

**Go**

```go
// workflow.go
func MultiLevelApprovalWorkflow(ctx workflow.Context, requestId string, timeoutPerLevel time.Duration) (string, error) {
    var approvals []MultiLevelApprovalData
    requiredLevels := []string{"L1", "L2", "L3"}

    // Collect every approval Signal as it arrives
    workflow.Go(ctx, func(ctx workflow.Context) {
        signalChan := workflow.GetSignalChannel(ctx, "submitApproval")
        for {
            var data MultiLevelApprovalData
            signalChan.Receive(ctx, &data)
            approvals = append(approvals, data)
        }
    })

    // Require approval at each level in sequence
    for _, level := range requiredLevels {
        lv := level
        // Wait for a Signal carrying this level's approval
        ok, err := workflow.AwaitWithTimeout(ctx, timeoutPerLevel, func() bool {
            for _, a := range approvals {
                if a.Level == lv {
                    return true
                }
            }
            return false
        })
        if err != nil {
            return "", err
        }
        if !ok {
            return fmt.Sprintf("Timeout at %s", lv), nil
        }

        var approval MultiLevelApprovalData
        for _, a := range approvals {
            if a.Level == lv {
                approval = a
                break
            }
        }
        // Stop the chain early if any level rejects
        if approval.Decision == "REJECTED" {
            return fmt.Sprintf("Rejected at %s by %s", lv, approval.Approver), nil
        }
    }

    return "Fully approved through all levels", nil
}
```

**Java**

```java
// MultiLevelApprovalWorkflowImpl.java
public class MultiLevelApprovalWorkflowImpl implements ApprovalWorkflow {
  private List<MultiLevelApprovalData> approvals = new ArrayList<>();
  private String[] requiredLevels = {"L1", "L2", "L3"};

  @Override
  public String execute(String requestId, Duration timeoutPerLevel) {
    // Require approval at each level in sequence
    for (String level : requiredLevels) {
      // Wait for a Signal carrying this level's approval
      boolean received = Workflow.await(
          timeoutPerLevel,
          () -> hasApprovalForLevel(level));

      if (!received) {
        return "Timeout at " + level;
      }

      MultiLevelApprovalData approval = getApprovalForLevel(level);
      // Stop the chain early if any level rejects
      if (approval.getDecision().equals("REJECTED")) {
        return "Rejected at " + level + " by " + approval.getApprover();
      }
    }

    return "Fully approved through all levels";
  }

  // Signal handler: collect each level's approval as it arrives
  @Override
  public void submitApproval(MultiLevelApprovalData data) {
    approvals.add(data);
  }

  private boolean hasApprovalForLevel(String level) {
    return approvals.stream().anyMatch(a -> a.getLevel().equals(level));
  }

  private MultiLevelApprovalData getApprovalForLevel(String level) {
    return approvals.stream()
        .filter(a -> a.getLevel().equals(level))
        .findFirst()
        .orElse(null);
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import { MultiLevelApprovalData } from './types';

export const submitApprovalSignal = wf.defineSignal<[MultiLevelApprovalData]>('submitApproval');

export async function multiLevelApprovalWorkflow(
  requestId: string,
  timeoutPerLevelMs: number,
): Promise<string> {
  const approvals: MultiLevelApprovalData[] = [];
  const requiredLevels = ['L1', 'L2', 'L3'] as const;

  // Signal handler: collect each level's approval as it arrives
  wf.setHandler(submitApprovalSignal, (data: MultiLevelApprovalData) => {
    approvals.push(data);
  });

  // Require approval at each level in sequence
  for (const level of requiredLevels) {
    // Wait for a Signal carrying this level's approval
    const received = await wf.condition(
      () => approvals.some((a) => a.level === level),
      timeoutPerLevelMs,
    );

    if (!received) {
      return `Timeout at ${level}`;
    }

    const approval = approvals.find((a) => a.level === level)!;
    // Stop the chain early if any level rejects
    if (approval.decision === 'REJECTED') {
      return `Rejected at ${level} by ${approval.approver}`;
    }
  }

  return 'Fully approved through all levels';
}
```

The Workflow loops through each required level and waits with a per-level timeout.
The helper logic checks whether a Signal has arrived for the current level.
If a timeout occurs at any level, the Workflow exits with a timeout result.
If any level returns a rejection, the Workflow exits immediately without proceeding to subsequent levels.

### Approval with escalation

When an initial approval times out, you may want to escalate the request to a manager rather than rejecting it outright.
The following implementation adds an escalation step with an extended timeout:

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow
from models import ApprovalData

with workflow.unsafe.imports_passed_through():
    from activities import send_escalation_email

@workflow.defn
class EscalatingApprovalWorkflow:
    def __init__(self) -> None:
        self.approval_data: ApprovalData | None = None
        self.escalated = False

    @workflow.run
    async def run(self, request_id: str, initial_timeout_seconds: int) -> str:
        try:
            # Wait for the first approval within the initial timeout
            await workflow.wait_condition(
                lambda: self.approval_data is not None,
                timeout=timedelta(seconds=initial_timeout_seconds),
            )
        except asyncio.TimeoutError:
            # No response in time: escalate to a manager, then wait longer
            self.escalated = True
            await workflow.execute_activity(
                send_escalation_email,
                start_to_close_timeout=timedelta(seconds=10),
            )

            try:
                # Extended wait for the escalated approval
                await workflow.wait_condition(
                    lambda: self.approval_data is not None,
                    timeout=timedelta(hours=24),
                )
            except asyncio.TimeoutError:
                return "Escalation timeout - auto-rejected"

        decision = self.approval_data.decision
        approver = self.approval_data.approver
        escalation_note = " (escalated)" if self.escalated else ""

        return f"{decision} by {approver}{escalation_note}"

    @workflow.signal
    def submit_approval(self, data: ApprovalData) -> None:
        self.approval_data = data
```

**Go**

```go
// workflow.go
func EscalatingApprovalWorkflow(ctx workflow.Context, requestId string, initialTimeout time.Duration) (string, error) {
    var approvalData *ApprovalData
    escalated := false

    workflow.Go(ctx, func(ctx workflow.Context) {
        signalChan := workflow.GetSignalChannel(ctx, "submitApproval")
        signalChan.Receive(ctx, &approvalData)
    })

    // Wait for the first approval within the initial timeout
    ok, err := workflow.AwaitWithTimeout(ctx, initialTimeout, func() bool {
        return approvalData != nil
    })
    if err != nil {
        return "", err
    }

    if !ok {
        // No response in time: escalate to a manager, then wait longer
        escalated = true

        ao := workflow.ActivityOptions{
            StartToCloseTimeout: 10 * time.Second,
        }
        actCtx := workflow.WithActivityOptions(ctx, ao)
        // Notify the manager via an Activity
        err = workflow.ExecuteActivity(actCtx, SendEscalationEmail).Get(ctx, nil)
        if err != nil {
            return "", err
        }

        // Extended wait for the escalated approval
        ok, err = workflow.AwaitWithTimeout(ctx, 24*time.Hour, func() bool {
            return approvalData != nil
        })
        if err != nil {
            return "", err
        }
        if !ok {
            return "Escalation timeout - auto-rejected", nil
        }
    }

    escalationNote := ""
    if escalated {
        escalationNote = " (escalated)"
    }

    return fmt.Sprintf("%s by %s%s", approvalData.Decision, approvalData.Approver, escalationNote), nil
}
```

**Java**

```java
// EscalatingApprovalWorkflowImpl.java
public class EscalatingApprovalWorkflowImpl implements ApprovalWorkflow {
  private ApprovalData approvalData;
  private boolean escalated = false;

  @Override
  public String execute(String requestId, Duration initialTimeout) {
    // Wait for the first approval within the initial timeout
    boolean received = Workflow.await(initialTimeout, () -> approvalData != null);

    if (!received) {
      // No response in time: escalate to a manager, then wait longer
      escalated = true;
      sendEscalationNotification();

      // Extended wait for the escalated approval
      received = Workflow.await(
          Duration.ofHours(24),
          () -> approvalData != null);

      if (!received) {
        return "Escalation timeout - auto-rejected";
      }
    }

    String decision = approvalData.getDecision();
    String approver = approvalData.getApprover();
    String escalationNote = escalated ? " (escalated)" : "";

    return decision + " by " + approver + escalationNote;
  }

  @Override
  public void submitApproval(ApprovalData data) {
    this.approvalData = data;
  }

  private void sendEscalationNotification() {
    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(10))
        .build();
    NotificationActivities activities =
        Workflow.newActivityStub(NotificationActivities.class, options);
    activities.sendEscalationEmail();
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import type * as activities from './activities';
import { ApprovalData } from './types';

const { sendEscalationEmail } = wf.proxyActivities<typeof activities>({
  startToCloseTimeout: '10 seconds',
});

export const submitApprovalSignal = wf.defineSignal<[ApprovalData]>('submitApproval');

export async function escalatingApprovalWorkflow(
  requestId: string,
  initialTimeoutMs: number,
): Promise<string> {
  let approvalData: ApprovalData | undefined;
  let escalated = false;

  wf.setHandler(submitApprovalSignal, (data: ApprovalData) => {
    approvalData = data;
  });

  // Wait for the first approval within the initial timeout
  let received = await wf.condition(
    () => approvalData !== undefined,
    initialTimeoutMs,
  );

  if (!received) {
    // No response in time: escalate to a manager, then wait longer
    escalated = true;
    await sendEscalationEmail();

    // Extended wait for the escalated approval
    received = await wf.condition(
      () => approvalData !== undefined,
      '24 hours',
    );

    if (!received) {
      return 'Escalation timeout - auto-rejected';
    }
  }

  const { decision, approver } = approvalData!;
  const escalationNote = escalated ? ' (escalated)' : '';

  return `${decision} by ${approver}${escalationNote}`;
}
```

The Workflow first waits for the initial timeout.
If no Signal arrives, it sets the `escalated` flag, executes a notification Activity to alert the manager, and then waits again with a 24-hour extended timeout.
The notification Activity uses a short start-to-close timeout, since sending an email should complete quickly.
The final result includes an escalation note so the caller knows the request was escalated before approval.

## When to use

The Approval pattern is a good fit for purchase order approvals, expense report reviews, code deployment gates, contract signing Workflows, manual quality checks, compliance reviews, budget authorization, and access request approvals.

It is not a good fit for fully automated processes that require no human input, real-time decisions that need synchronous API responses, or processes that require sub-second response times.
If you only need a boolean yes/no without any context, a plain boolean Signal may be sufficient.

## Benefits and trade-offs

The Approval pattern captures rich context — approver identity, reasons, and timestamps — alongside each decision.
All approval data is recorded in the Workflow history as Signal events, giving you a built-in audit trail.
Timeout handling is automatic: you define the maximum wait time and the Workflow handles the fallback.
The pattern supports multi-level, conditional, and escalating approval chains, and you can check approval status at any time through Query methods without modifying Workflow state.
Because all decisions are recorded in the event history, the Workflow Definition is replay-safe.

The trade-offs to consider are that the pattern requires an external system to send approval Signals, which means you need a separate approval interface.
The Workflow blocks until the approval arrives or the timeout expires, so you must define a maximum wait time.
Large approval data objects increase the size of the Workflow history.

## Comparison with alternatives

| Approach | Rich data | Built-in wait | Caller gets result | Complexity | Use case |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Signal with data | Yes | Yes | No | Low | Approval Workflows |
| Update | Yes | No | Yes | Low | Synchronous validation with immediate confirmation |
| Boolean Signal | No | Yes | No | Low | Yes/no decisions |
| Polling Activity | Yes | Yes | Yes | High | External approval systems |

Signals are fire-and-forget: the caller receives an acknowledgement from the server but cannot wait for the Workflow to process the Signal or receive a result.
Updates are synchronous: the caller blocks until the handler completes and can receive a return value or error.
If the approver's interface needs immediate confirmation that the approval was accepted and valid, consider using an Update with a validator instead of a Signal.

## Best practices

- **Use custom data objects.** Capture rich approval context — approver identity, comments, timestamps — rather than a plain boolean.
- **Set reasonable timeouts.** Balance responsiveness with the time approvers realistically need to respond.
- **Add Query methods.** Expose the current approval status so external systems can check progress without sending a Signal.
- **Validate Signal data.** Verify approver permissions and data completeness before accepting an approval.
- **Log approval events.** Record each decision for audit trails and compliance.
- **Handle timeouts gracefully.** Define clear timeout behavior such as rejection, escalation, or notification.
- **Support cancellation.** Allow Workflows to be cancelled if the request is withdrawn.
- **Ensure idempotency.** Handle duplicate approval Signals safely so that re-delivery does not corrupt state. Signals [may be duplicated in rare cases](/handling-messages#exactly-once-message-processing), so use idempotency keys when necessary.
- **Include timestamps.** Record when each approval was submitted to support time-based auditing.
- **Expose approval history.** Provide a Query method that returns all approval attempts, not only the final decision.

## Common pitfalls

- **No timeout.** Without a timeout, the Workflow waits indefinitely for an approval that may never arrive.
- **Missing validation.** Accepting approvals from unauthorized users compromises the integrity of the process.
- **Lost context.** Failing to capture the approver's identity or reason makes audit trails incomplete.
- **Assuming non-deterministic races.** Temporal processes events in a deterministic, single-threaded order, so a Signal and a timer cannot truly "race." However, if the Signal arrives after the timer fires in the event history, the wait will have already returned with a timeout result. Design your timeout path to account for late-arriving Signals.
- **No audit trail.** Skipping approval logging makes it difficult to meet compliance requirements.
- **Tight timeouts.** Setting the timeout too short causes legitimate approvals to be rejected.
- **Boolean-only Signals.** Using a plain boolean instead of a rich data object limits your ability to capture decision context.
- **No status Query.** Without a Query method, external systems have no way to check approval progress.
- **No duplicate handling.** Receiving multiple approval Signals without deduplication can overwrite earlier decisions.
- **No escalation path.** Without a fallback when the initial approval times out, requests stall or are silently rejected.

## Related

### Patterns

- [Signal-Based Event Handling](/design-patterns/signal-with-start): Receiving external events through Signals.
- [Updatable Timer](/design-patterns/updatable-timer): Extending approval deadlines dynamically.
- [Saga Pattern](/design-patterns/saga-pattern): Executing compensating actions on rejection.

### Guides

- [Reliable document approvals](/guides/reliable-document-approvals): A complete Python implementation with SLA timers, automatic escalation, resubmission loops, and audit logging.

### Sample code

**Java**
- [Hello Signal](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloSignal.java) — Basic Signal handling in a Workflow.
- [Safe Message Passing](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Concurrent Signal handling with validation.

**TypeScript**
- [Signals and Queries](https://github.com/temporalio/samples-typescript/tree/main/signals-queries) — Signal and Query usage in a Workflow.
- [Message Passing](https://github.com/temporalio/samples-typescript/tree/main/message-passing) — Introduction to message passing with Signals, Queries, and Updates.

**Python**
- [Hello Signal](https://github.com/temporalio/samples-python/tree/main/hello/hello_signal.py) — Basic Signal handling in a Workflow.
- [Message Passing](https://github.com/temporalio/samples-python/tree/main/message_passing/introduction) — Introduction to message passing with Signals, Queries, and Updates.

**Go**
- [Await Signals](https://github.com/temporalio/samples-go/tree/main/await-signals) — Waiting for Signals with timeout using `AwaitWithTimeout`.
- [Message Passing](https://github.com/temporalio/samples-go/tree/message-passing/message-passing-intro) — Introduction to message passing with Signals, Queries, and Updates.

---

# Batch Iterator

Source: https://docs.temporal.io/design-patterns/batch-iterator

> Pages through unbounded datasets using Continue-As-New to prevent history overflow while maintaining exactly-once processing guarantees.

> **ℹ️ TLDR:**
> **Process one page at a time** and call Continue-as-New with the next offset after each page so the Workflow's event history never grows without bound. With this method you can process infinite pages. Use this when your record set is arbitrarily large, you need a durable checkpoint after every page, and sequential page-by-page throughput is acceptable.

## Overview

The Batch Iterator pattern processes a large record set one page at a time. Each Workflow run processes a single page and then calls Continue-as-New with the next offset, producing a chain of short-lived runs that together cover the entire record set without accumulating unbounded event history.

## Problem

A single Workflow run is limited to 51,200 history events (aim for 2,000) and 2,000 in-flight Activities. Processing millions of records in one run is not possible within these bounds.

You need a way to process an arbitrarily large record set reliably, with the ability to resume from a checkpoint if the Workflow is interrupted, and without overwhelming downstream systems with a burst of concurrent requests.

## Solution

Each Workflow run fetches one page of records using a persistent `offset` parameter, processes each record sequentially, and then calls `continueAsNew` with the incremented offset. The next run picks up exactly where the previous one left off.

Because each run processes only a bounded number of records, history stays well within limits. The offset acts as a durable checkpoint: if the Workflow is interrupted mid-page, the next run replays only from the start of the current page.

```mermaid
flowchart TD
    DB[("Data Source\n(paginated)")]
    WF1["Workflow Run 1\n(offset=0)"]
    WF2["Workflow Run 2\n(offset=PAGE_SIZE)"]
    WF3["Workflow Run N\n(offset=N×PAGE_SIZE)"]
    Done(["Complete"])

    DB -->|"fetch page 1"| WF1
    WF1 -->|"processRecord ×PAGE_SIZE"| Acts1["Activities"]
    WF1 -->|"continueAsNew\n(offset=PAGE_SIZE)"| WF2

    DB -->|"fetch page 2"| WF2
    WF2 -->|"processRecord ×PAGE_SIZE"| Acts2["Activities"]
    WF2 -->|"continueAsNew\n(offset=N×PAGE_SIZE)"| WF3

    DB -->|"fetch page N"| WF3
    WF3 -->|"processRecord ×PAGE_SIZE"| Acts3["Activities"]
    WF3 -->|"last page → return"| Done
```

The following describes each step in the diagram:

1. The Workflow starts with `offset=0` and calls `fetchPage(offset, pageSize)` to retrieve the first page of records.
2. It processes each record in the page by executing the `processRecord` Activity.
3. After the page is fully processed, it calls `continueAsNew` with `offset + pageSize`, passing the updated offset to the next run.
4. The next run begins with a clean history and repeats the same steps for the next page.
5. When `fetchPage` returns fewer records than `pageSize`, the Workflow knows it has reached the last page and returns normally.

## Implementation

The following examples show how each SDK implements the Batch Iterator pattern.

**TypeScript**

```typescript
// workflows.ts
import { continueAsNew, log, proxyActivities } from "@temporalio/workflow";
import type * as activities from "./activities";
import { PAGE_SIZE } from "./shared";

const { fetchPage, processRecord } = proxyActivities<typeof activities>({
  startToCloseTimeout: "10 seconds",
});

export async function batchIteratorWorkflow(
  offset: number = 0,
  totalProcessed: number = 0
): Promise<number> {
  const page = await fetchPage(offset, PAGE_SIZE);

  for (const record of page) {
    await processRecord(record);
    totalProcessed++;
  }

  log.info(`Processed page at offset ${offset} (${page.length} records, running total: ${totalProcessed})`);

  if (page.length === PAGE_SIZE) {
    await continueAsNew<typeof batchIteratorWorkflow>(offset + PAGE_SIZE, totalProcessed);
  }

  return totalProcessed;
}
```

**Python**

```python
# workflows.py
from temporalio import workflow
from temporalio.workflow import continue_as_new
from datetime import timedelta
from activities import fetch_page, process_record
from shared import PAGE_SIZE

@workflow.defn
class BatchIteratorWorkflow:
    @workflow.run
    async def run(self, offset: int = 0, total_processed: int = 0) -> int:
        page = await workflow.execute_activity(
            fetch_page,
            args=[offset, PAGE_SIZE],
            start_to_close_timeout=timedelta(seconds=10),
        )

        for record in page:
            await workflow.execute_activity(
                process_record,
                record,
                start_to_close_timeout=timedelta(seconds=10),
            )
            total_processed += 1

        workflow.logger.info(
            f"Processed page at offset {offset} ({len(page)} records, running total: {total_processed})"
        )

        if len(page) == PAGE_SIZE:
            continue_as_new(args=[offset + PAGE_SIZE, total_processed])

        return total_processed
```

**Go**

```go
// workflows.go
package main

import (
	"time"

	"go.temporal.io/sdk/workflow"
)

func BatchIteratorWorkflow(ctx workflow.Context, offset int, totalProcessed int) (int, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	var page []Record
	if err := workflow.ExecuteActivity(ctx, FetchPage, offset, PageSize).Get(ctx, &page); err != nil {
		return totalProcessed, err
	}

	for _, record := range page {
		if err := workflow.ExecuteActivity(ctx, ProcessRecord, record).Get(ctx, nil); err != nil {
			return totalProcessed, err
		}
		totalProcessed++
	}

	workflow.GetLogger(ctx).Info("Processed page",
		"offset", offset,
		"pageSize", len(page),
		"totalProcessed", totalProcessed)

	if len(page) == PageSize {
		return totalProcessed, workflow.NewContinueAsNewError(ctx, BatchIteratorWorkflow, offset+PageSize, totalProcessed)
	}

	return totalProcessed, nil
}
```

**Java**

```java
// BatchIteratorWorkflow.java
import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.*;
import java.time.Duration;
import java.util.List;

@WorkflowInterface
public interface BatchIteratorWorkflow {
    @WorkflowMethod
    int run(int offset, int totalProcessed);
}

// BatchIteratorWorkflowImpl.java
public class BatchIteratorWorkflowImpl implements BatchIteratorWorkflow {
    private final Activities activities = Workflow.newActivityStub(
        Activities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(10))
            .build()
    );

    @Override
    public int run(int offset, int totalProcessed) {
        List<Record> page = activities.fetchPage(offset, Shared.PAGE_SIZE);

        for (Record record : page) {
            activities.processRecord(record);
            totalProcessed++;
        }

        Workflow.getLogger(BatchIteratorWorkflowImpl.class).info(
            "Processed page at offset " + offset + " (" + page.size() + " records, total: " + totalProcessed + ")"
        );

        if (page.size() == Shared.PAGE_SIZE) {
            BatchIteratorWorkflow next = Workflow.newContinueAsNewStub(BatchIteratorWorkflow.class);
            next.run(offset + Shared.PAGE_SIZE, totalProcessed);
        }

        return totalProcessed;
    }
}
```

## Best practices

- **Choose a page size that keeps history under 2,000 events.** Each page produces roughly `3 × pageSize` history events (`ActivityTaskScheduled` + `ActivityTaskStarted` + `ActivityTaskCompleted`). A page size of 500–800 records is a safe target.
- **Include `totalProcessed` (or a similar counter) in the `continueAsNew` args.** This lets you observe overall progress via the Workflow input visible in the UI without querying internal state.
- **Fetch inside an Activity, not the Workflow.** The `fetchPage` call must be an Activity — not inline Workflow code — so it can interact with external systems and be retried independently.
- **Make `processRecord` idempotent.** Activities have at-least-once execution semantics. If a worker crashes after an Activity completes externally but before the completion is recorded in history, Temporal will retry it. Your downstream system must tolerate receiving the same record more than once.
- **Avoid accumulating large local state between pages.** `continueAsNew` does not carry over in-memory state; only the arguments you pass are available in the next run.

## Common pitfalls

- **Forgetting `continueAsNew` on the last page.** If you call `continueAsNew` unconditionally, the Workflow loops forever even when the data source is exhausted. Check whether the returned page is shorter than `pageSize` before continuing.
- **Passing unnecessary state into `continueAsNew`.** All arguments are serialized and stored in history. Pass only the minimal state needed (offset, counters) — not accumulated result lists or large collections that grow with each page.
- **Sequential processing bottlenecks.** The default implementation processes one record at a time per page. You can fan out Activities concurrently within a page using the SDK's async primitives for higher per-page throughput — note this increases per-page event count accordingly. If record-set-wide throughput matters more than rate limiting, consider [Sliding Window](/design-patterns/sliding-window) or [MapReduce Tree](/design-patterns/mapreduce-tree).

## Related

### Patterns

- [Continue-as-New pattern](/design-patterns/continue-as-new) — core concepts for history management via `continueAsNew`
- [Sliding Window](/design-patterns/sliding-window) — bounded concurrency that progresses at the rate of the fastest processor
- [MapReduce Tree](/design-patterns/mapreduce-tree) — fully parallel processing for maximum speed
- [Temporal limits reference](/evaluate/cloud/limits)
- [Batch samples (Java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/iterator)

---

# Batch Processing Patterns

Source: https://docs.temporal.io/design-patterns/batch-processing-patterns

> Compare Fan-Out, Batch Iterator, Sliding Window, and MapReduce Tree patterns for processing large record sets reliably at scale.

These patterns process large volumes of records reliably, at scale, and without overwhelming downstream systems. Choose based on your throughput requirements, record set size, and whether you need rate limiting or maximum parallelism.

## When to use which pattern

| Pattern | Record set size | Parallelism model | Workflow-based rate control |
|---|---|---|---|
| [Basic Workflow](#basic-workflow-single-tier-fan-out) | Small (up to a few hundred records) | Sequential or parallel activities in one Workflow | No |
| [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows) | Up to ~500K records | Fixed concurrency (one child per chunk) | No |
| [Batch Iterator](/design-patterns/batch-iterator) | Unlimited | Limited (activities per page) | Yes — fixed page rate |
| [Sliding Window](/design-patterns/sliding-window) | Unlimited | Bounded window of concurrent children | Yes — configurable window |
| [MapReduce Tree](/design-patterns/mapreduce-tree) | Unlimited | Fully parallel recursive tree | No — maximum speed |

## Patterns in this section

- [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows): Splits a record set into fixed-size chunks and assigns each to an independent child Workflow. Direct to reason about; best for record sets up to ~500K items.
- [Batch Iterator](/design-patterns/batch-iterator): Processes one page of records per Workflow run and continues as new with the next page offset. Handles unlimited record sets while controlling downstream traffic.
- [Sliding Window](/design-patterns/sliding-window): Maintains a fixed-size window of concurrent child Workflows. As each child completes it signals the parent, which immediately starts a replacement — maximizing throughput within a concurrency budget.
- [MapReduce Tree](/design-patterns/mapreduce-tree): Recursively splits a record set into chunks, fans out to leaf Workflows for parallel processing, and signals results back up the tree. Maximizes speed for embarrassingly parallel workloads.

---

## Schedules

Schedules allow Workflows to be executed on a recurring basis — think of them as a more flexible cron with start, pause, stop, update, and backfill controls.

- Supports `start` / `pause` / `stop` / `update` / `backfill` of scheduled Workflow executions
- Configurable **Overlap Policies** control what happens when the previous run is still running
- Full execution history visibility in the Temporal UI
- Schedules can be created via the UI, CLI, or SDK

```bash
temporal schedule create \
  --schedule-id 'your-schedule-id' \
  --workflow-id 'your-workflow-id' \
  --task-queue 'your-task-queue' \
  --workflow-type 'YourWorkflowType'
```

**References:**
- [Temporal Schedules](/schedule)
- [CLI schedule commands](/cli/command-reference/schedule)

---

## Basic Workflow (single-tier fan-out)

The most direct form of batch processing: the Workflow fetches or receives record IDs and executes one Activity per record.

- Activities can be executed sequentially or concurrently (using the SDK's async primitives)
- **Limit: 2,000 in-flight Activities per Workflow run** (aim for 500)
- If total event count is likely to exceed 2,000 (hard limit: 51,200), use the [Batch Iterator](/design-patterns/batch-iterator) instead

**Pros:** Minimal code and orchestration overhead  
**Cons:** Hard cap on concurrent Activities; all-or-nothing failure model; can overwhelm downstream systems

```mermaid
flowchart TD
    Records["📋 Record IDs\n(fetched or passed in)"]
    WF["Workflow"]
    A1["Activity"]
    A2["Activity"]
    AN["Activity ..."]

    Records --> WF
    WF --> A1
    WF --> A2
    WF --> AN
```

---

## Batch signalling

The Temporal CLI lets you signal, reset, cancel, or terminate multiple Workflows with a single command using a visibility query.

- 1 running batch job per namespace
- 50 Workflows per second per batch

```bash
# Signal all running Workflows of a given type
temporal workflow signal \
  --name MySignal \
  --input '{"Input": "As-JSON"}' \
  --query 'ExecutionStatus = "Running" AND WorkflowType="YourWorkflow"' \
  --reason "Testing"

# Terminate all running Workflows of a given type
temporal workflow terminate \
  --query 'ExecutionStatus = "Running" AND WorkflowType="SomeWorkflowType"' \
  --reason "Terminate Test Workflows"
```

**Reference:** [CLI batch commands](/cli/command-reference/batch)

---

## Key limits

Full reference: [Temporal Cloud limits](/evaluate/cloud/limits)

| Limit | Value |
|---|---|
| Unfinished actions per Workflow | 2,000 max (aim for 500). Includes Activities, Signals, Child Workflows, cancellation requests |
| Events per Workflow history | 51,200 events max (aim for a few thousand) **or** 50 MB total history size; warns at 10,240 events / 10 MB |
| Signals per Workflow | 10,000 |
| Updates per Workflow | 10 in-flight, 2,000 total |
| Batch Signalling | 1 batch job per namespace; 50 Workflows/sec per batch |

## Related sections

- [Task Orchestration Patterns](/design-patterns/task-orchestration-patterns) — the child-Workflow and parallel primitives these patterns scale up
- [QoS & Throughput Patterns](/design-patterns/qos-throughput-patterns) — rate-limit and prioritize the work a batch generates
- [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — reduce the latency of each record's processing

---

# Child Workflows Pattern

Source: https://docs.temporal.io/design-patterns/child-workflows

> Decomposes complex Workflows into smaller, reusable units. Each child has an independent Workflow ID, history, and lifecycle.

## Overview

Child Workflows enable decomposition of complex business logic into smaller, reusable Workflow units.
Each child executes as an independent Workflow with its own Workflow ID, event history (51,200 event limit), and lifecycle.
Unlike Activities which execute code, Child Workflows orchestrate processes and provide Workflow-level semantics: independent tracking, querying, timeouts, and the ability to outlive the parent.

Key capabilities:

- **Independent identity**: Each child has a unique Workflow ID visible in the UI for tracking and querying.
- **Separate history**: Each child maintains its own event history, preventing parent history bloat.
- **Flexible invocation**: Synchronous (blocking) or asynchronous (non-blocking) execution.
- **Lifecycle control**: Parent close policies (TERMINATE, ABANDON, REQUEST_CANCEL) determine child behavior when the parent completes.
- **Task Queue routing**: Children can execute on different Task Queues with specialized Workers.
- **Reusability**: The same Child Workflow logic can be invoked by multiple different parent Workflows.

## Problem

In distributed systems, you often need Workflows that break down complex processes into modular, reusable components, execute sub-processes that may outlive the parent Workflow, coordinate multiple independent Workflows with different lifecycles, isolate failure domains while maintaining orchestration control, and reuse Workflow logic across different parent Workflows.

Without Child Workflows, you must implement all logic in a single monolithic Workflow, manually coordinate separate Workflows via Signals and Queries, duplicate Workflow logic across multiple implementations, and manage complex state machines for sub-process coordination.

## Solution

You invoke Child Workflows from within parent Workflows using the SDK's Child Workflow API.
You can call them synchronously (blocking until completion) or asynchronously (fire-and-forget).
The `ParentClosePolicy` determines what happens to children when the parent completes.

```mermaid
sequenceDiagram
    participant Parent
    participant Child1
    participant Child2

    Parent->>+Child1: Start (sync)
    activate Parent
    Child1->>Child1: Execute
    Child1-->>-Parent: Result
    
    Parent->>+Child2: Start (async)
    Note over Parent,Child2: Parent continues immediately
    Parent->>Parent: Do other work
    Child2->>Child2: Execute independently
    
    alt Parent completes first
        Parent->>Parent: Complete
        deactivate Parent
        Note over Child2: Policy: ABANDON<br/>Child continues
        Child2->>Child2: Keep running
        Child2-->>-Child2: Complete
    end
```

The following describes each step in the diagram:

1. The parent starts Child 1 synchronously and blocks until it completes.
2. The parent starts Child 2 asynchronously and continues doing other work immediately.
3. If the parent completes before Child 2, the ABANDON policy allows Child 2 to continue running independently.

### Synchronous Child Workflow

The following example creates a Child Workflow and calls it synchronously.
The parent blocks until the child completes and returns a result:

**Python**

```python
# workflows.py
from temporalio import workflow

from child_workflows import ChildWorkflow

@workflow.defn
class ParentWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        # Synchronous call - awaits until child completes
        result = await workflow.execute_child_workflow(
            ChildWorkflow.run,
            input,
            id=f"child-{workflow.uuid4()}",
        )

        return f"Parent received: {result}"
```

**Go**

```go
// parent_workflow.go
func ParentWorkflow(ctx workflow.Context, input string) (string, error) {
    cwo := workflow.ChildWorkflowOptions{}
    ctx = workflow.WithChildOptions(ctx, cwo)

    // Synchronous call - blocks until child completes
    var result string
    err := workflow.ExecuteChildWorkflow(ctx, ChildWorkflow, input).Get(ctx, &result)
    if err != nil {
        return "", err
    }

    return "Parent received: " + result, nil
}
```

**Java**

```java
// ParentWorkflowImpl.java
@WorkflowInterface
public interface ParentWorkflow {
  @WorkflowMethod
  String execute(String input);
}

public class ParentWorkflowImpl implements ParentWorkflow {
  @Override
  public String execute(String input) {
    ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class);

    // Synchronous call - blocks until child completes
    String result = child.processData(input);

    return "Parent received: " + result;
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { executeChild } from '@temporalio/workflow';
import { childWorkflow } from './child-workflows';

export async function parentWorkflow(input: string): Promise<string> {
  // Synchronous call - awaits until child completes
  const result = await executeChild(childWorkflow, {
    args: [input],
  });

  return `Parent received: ${result}`;
}
```

In Java, `Workflow.newChildWorkflowStub()` creates a typed stub and calling a method on it blocks the parent.
In TypeScript, `executeChild()` starts the child and awaits its completion.
In Python, `workflow.execute_child_workflow()` starts the child and awaits its completion.
In Go, `workflow.ExecuteChildWorkflow()` returns a `ChildWorkflowFuture`, and calling `.Get()` blocks until the child completes.

### Asynchronous Child Workflow

The following example starts a Child Workflow asynchronously with an ABANDON policy.
The parent receives the child's execution info without waiting for completion. Select a concept to highlight the matching lines:

**Python**

  annotations={[
    {
      label: 'Parent Close Policy',
      description:
        'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.',
      lines: [14],
    },
    {
      label: 'Workflow Id',
      description:
        'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.',
      lines: [13],
    },
    {
      label: 'Async start',
      description:
        'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.',
      lines: [10, 11, 12, 13, 14, 15],
    },
  ]}
>
```python
from temporalio import workflow
from temporalio.workflow import ParentClosePolicy

from child_workflows import ChildWorkflow

@workflow.defn
class ParentWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        handle = await workflow.start_child_workflow(
            ChildWorkflow.run,
            input,
            id=f"child-{workflow.uuid4()}",
            parent_close_policy=ParentClosePolicy.ABANDON,
        )

        return handle.id
```

**Go**

  annotations={[
    {
      label: 'Parent Close Policy',
      description:
        'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.',
      lines: [3],
    },
    {
      label: 'Workflow Id',
      description:
        'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.',
      lines: [14],
    },
    {
      label: 'Async start',
      description:
        'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.',
      lines: [7],
    },
  ]}
>
```go
func ParentWorkflow(ctx workflow.Context, input string) (string, error) {
    cwo := workflow.ChildWorkflowOptions{
        ParentClosePolicy: enumspb.PARENT_CLOSE_POLICY_ABANDON,
    }
    ctx = workflow.WithChildOptions(ctx, cwo)

    childFuture := workflow.ExecuteChildWorkflow(ctx, ChildWorkflow, input)

    var childWE workflow.Execution
    if err := childFuture.GetChildWorkflowExecution().Get(ctx, &childWE); err != nil {
        return "", err
    }

    return childWE.ID, nil
}
```

**Java**

  annotations={[
    {
      label: 'Parent Close Policy',
      description:
        'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.',
      lines: [6],
    },
    {
      label: 'Workflow Id',
      description:
        'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.',
      lines: [5],
    },
    {
      label: 'Async start',
      description:
        'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.',
      lines: [11],
    },
  ]}
>
```java
public class ParentWorkflowImpl implements ParentWorkflow {
  @Override
  public WorkflowExecution execute(String input) {
    ChildWorkflowOptions options = ChildWorkflowOptions.newBuilder()
        .setWorkflowId("child-" + Workflow.randomUUID())
        .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON)
        .build();

    ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class, options);

    Async.function(child::processData, input);

    Promise<WorkflowExecution> childExecution = Workflow.getWorkflowExecution(child);
    return childExecution.get();
  }
}
```

**TypeScript**

  annotations={[
    {
      label: 'Parent Close Policy',
      description:
        'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.',
      lines: [7],
    },
    {
      label: 'Workflow Id',
      description:
        'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.',
      lines: [10],
    },
    {
      label: 'Async start',
      description:
        'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.',
      lines: [5, 6, 7, 8],
    },
  ]}
>
```typescript
import { startChild, ParentClosePolicy } from '@temporalio/workflow';
import { childWorkflow } from './child-workflows';

export async function parentWorkflow(input: string): Promise<string> {
  const childHandle = await startChild(childWorkflow, {
    args: [input],
    parentClosePolicy: ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON,
  });

  return childHandle.workflowId;
}
```

In Java, `Async.function()` starts the child asynchronously.
`Workflow.getWorkflowExecution(child)` returns a Promise that resolves when the child starts (not when it completes).
In TypeScript, `startChild()` returns a handle once the child has started.
In Python, `workflow.start_child_workflow()` returns a handle once the child has started, without waiting for completion.
In Go, `childFuture.GetChildWorkflowExecution().Get()` blocks until the child has started.
The ABANDON policy ensures the child continues running even if the parent completes first.

## Parent Close Policy

The `ParentClosePolicy` determines Child Workflow behavior when the parent closes:

| Policy | Behavior | Use case |
| :--- | :--- | :--- |
| `TERMINATE` | Child is terminated when parent closes | Tightly coupled processes |
| `ABANDON` | Child continues independently | Fire-and-forget, long-running tasks |
| `REQUEST_CANCEL` | Child receives cancellation request | Graceful cleanup |

## Implementation

### Parallel Child Workflows

The following example starts multiple Child Workflows in parallel and waits for all of them to complete:

**Python**

```python
# workflows.py
import asyncio
from temporalio import workflow

from child_workflows import ChildWorkflow

@workflow.defn
class ParallelParentWorkflow:
    @workflow.run
    async def run(self, items: list[str]) -> str:
        # Start all children concurrently using asyncio.gather
        results = await asyncio.gather(
            *[
                workflow.execute_child_workflow(
                    ChildWorkflow.run,
                    item,
                    id=f"child-{workflow.uuid4()}",
                )
                for item in items
            ]
        )

        return ", ".join(results)
```

**Go**

```go
// parallel_parent_workflow.go
func ParallelParentWorkflow(ctx workflow.Context, items []string) (string, error) {
    cwo := workflow.ChildWorkflowOptions{}
    ctx = workflow.WithChildOptions(ctx, cwo)

    // Start all children - ExecuteChildWorkflow returns immediately
    var futures []workflow.ChildWorkflowFuture
    for _, item := range items {
        futures = append(futures, workflow.ExecuteChildWorkflow(ctx, ChildWorkflow, item))
    }

    // Wait for all children to complete
    var results []string
    for _, future := range futures {
        var result string
        if err := future.Get(ctx, &result); err != nil {
            return "", err
        }
        results = append(results, result)
    }

    return strings.Join(results, ", "), nil
}
```

**Java**

```java
// ParallelParentWorkflowImpl.java
public class ParallelParentWorkflowImpl implements ParentWorkflow {
  @Override
  public String execute(List<String> items) {
    List<Promise<String>> promises = new ArrayList<>();

    for (String item : items) {
      ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class);
      promises.add(Async.function(child::process, item));
    }

    // Wait for all children to complete
    Promise.allOf(promises).get();

    return promises.stream()
        .map(Promise::get)
        .collect(Collectors.joining(", "));
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { executeChild } from '@temporalio/workflow';
import { childWorkflow } from './child-workflows';

export async function parallelParentWorkflow(items: string[]): Promise<string> {
  // Start all children concurrently using Promise.all
  const results = await Promise.all(
    items.map((item) =>
      executeChild(childWorkflow, {
        args: [item],
      })
    )
  );

  return results.join(', ');
}
```

In Java, each child starts asynchronously via `Async.function()`, and `Promise.allOf(promises).get()` blocks until every child completes.
In TypeScript, `Promise.all()` starts all children concurrently and awaits all results.
In Python, `asyncio.gather()` starts all children concurrently and awaits all results.
In Go, `workflow.ExecuteChildWorkflow()` returns a Future immediately without blocking, so starting all children in a loop launches them in parallel. Calling `.Get()` on each Future afterward collects the results.

### Fire-and-forget

The following example starts a Child Workflow with the ABANDON policy and returns immediately without waiting:

**Python**

```python
# workflows.py
from temporalio import workflow
from temporalio.workflow import ParentClosePolicy

from child_workflows import LongRunningChildWorkflow

@workflow.defn
class FireAndForgetParentWorkflow:
    @workflow.run
    async def run(self, data: str) -> None:
        # Start child with ABANDON policy - child survives parent completion
        await workflow.start_child_workflow(
            LongRunningChildWorkflow.run,
            data,
            id=f"child-{workflow.uuid4()}",
            parent_close_policy=ParentClosePolicy.ABANDON,
        )

        # start_child_workflow resolves once the child has started
        # Parent completes, child continues independently
```

**Go**

```go
// fire_and_forget_workflow.go
import (
    enumspb "go.temporal.io/api/enums/v1"
    "go.temporal.io/sdk/workflow"
)

func FireAndForgetParentWorkflow(ctx workflow.Context, data string) error {
    cwo := workflow.ChildWorkflowOptions{
        ParentClosePolicy: enumspb.PARENT_CLOSE_POLICY_ABANDON,
    }
    ctx = workflow.WithChildOptions(ctx, cwo)

    childFuture := workflow.ExecuteChildWorkflow(ctx, LongRunningChildWorkflow, data)

    // Wait for child to start before parent completes
    if err := childFuture.GetChildWorkflowExecution().Get(ctx, nil); err != nil {
        return err
    }

    // Parent completes, child continues independently
    return nil
}
```

**Java**

```java
// FireAndForgetParentWorkflowImpl.java
public class FireAndForgetParentWorkflowImpl implements ParentWorkflow {
  @Override
  public void execute(String data) {
    ChildWorkflowOptions options = ChildWorkflowOptions.newBuilder()
        .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON)
        .build();

    ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class, options);

    // Start child and don't wait for completion
    Async.function(child::longRunningProcess, data);

    // Wait for child to start before parent completes
    Workflow.getWorkflowExecution(child).get();

    // Parent completes, child continues independently
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { startChild, ParentClosePolicy } from '@temporalio/workflow';
import { longRunningChildWorkflow } from './child-workflows';

export async function fireAndForgetParentWorkflow(data: string): Promise<void> {
  // Start child with ABANDON policy - child survives parent completion
  await startChild(longRunningChildWorkflow, {
    args: [data],
    parentClosePolicy: ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON,
  });

  // startChild resolves once the child has started
  // Parent completes, child continues independently
}
```

You must wait for the child to start before the parent completes.
Without this, the parent could complete before the child is scheduled, and the child would never execute.
The ABANDON policy ensures the child continues running after the parent completes.

### Conditional child execution

The following example conditionally starts different Child Workflows based on business logic:

**Python**

```python
# workflows.py
from temporalio import workflow

from child_workflows import ApprovalWorkflow, FulfillmentWorkflow

@workflow.defn
class ConditionalParentWorkflow:
    @workflow.run
    async def run(self, order: Order) -> str:
        if order.requires_approval:
            approved = await workflow.execute_child_workflow(
                ApprovalWorkflow.run,
                order,
                id=f"approval-{order.id}",
            )

            if not approved:
                return "Order rejected"

        return await workflow.execute_child_workflow(
            FulfillmentWorkflow.run,
            order,
            id=f"fulfillment-{order.id}",
        )
```

**Go**

```go
// conditional_parent_workflow.go
func ConditionalParentWorkflow(ctx workflow.Context, order Order) (string, error) {
    cwo := workflow.ChildWorkflowOptions{}
    ctx = workflow.WithChildOptions(ctx, cwo)

    if order.RequiresApproval {
        var approved bool
        err := workflow.ExecuteChildWorkflow(ctx, ApprovalWorkflow, order).Get(ctx, &approved)
        if err != nil {
            return "", err
        }

        if !approved {
            return "Order rejected", nil
        }
    }

    var result string
    err := workflow.ExecuteChildWorkflow(ctx, FulfillmentWorkflow, order).Get(ctx, &result)
    if err != nil {
        return "", err
    }

    return result, nil
}
```

**Java**

```java
// ConditionalParentWorkflowImpl.java
public class ConditionalParentWorkflowImpl implements ParentWorkflow {
  @Override
  public String execute(Order order) {
    if (order.requiresApproval()) {
      ApprovalWorkflow approval = Workflow.newChildWorkflowStub(ApprovalWorkflow.class);
      boolean approved = approval.requestApproval(order);

      if (!approved) {
        return "Order rejected";
      }
    }

    FulfillmentWorkflow fulfillment = Workflow.newChildWorkflowStub(FulfillmentWorkflow.class);
    return fulfillment.fulfill(order);
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { executeChild } from '@temporalio/workflow';
import { approvalWorkflow, fulfillmentWorkflow } from './child-workflows';

export async function conditionalParentWorkflow(order: Order): Promise<string> {
  if (order.requiresApproval) {
    const approved = await executeChild(approvalWorkflow, {
      args: [order],
    });

    if (!approved) {
      return 'Order rejected';
    }
  }

  return await executeChild(fulfillmentWorkflow, {
    args: [order],
  });
}
```

The parent checks whether the order requires approval and only starts the approval Child Workflow when needed.

## When to use

Child Workflows and Activities serve different purposes.

Use Child Workflows when:

- You need a separate Workflow ID for tracking and querying.
- The operation may outlive the parent Workflow.
- You need to reuse Workflow logic across multiple parents.
- You want to execute Workflows on different Task Queues.
- You need independent history and event limits.
- You want to apply different timeouts or retry policies at the Workflow level.

Use Activities when:

- You are executing external operations (API calls, database queries).
- The operation is short-lived.
- You do not need independent Workflow tracking.
- The operation is tightly coupled to the parent Workflow lifecycle.
- Lower overhead is important.

The key distinction is that Activities are for executing code (especially external operations), while Child Workflows are for orchestrating processes that benefit from independent Workflow semantics.

## Benefits and trade-offs

Child Workflows provide modularity by breaking complex logic into reusable units.
Each child is a first-class Workflow with its own ID for tracking, its own 51,200-event history limit, and its own execution timeout configuration.
Children can outlive parents with the ABANDON policy, and you can start multiple children concurrently.
Child failures do not automatically fail the parent, and the same Child Workflow can be reused by multiple parents.

The trade-offs to consider are that each child is a separate Workflow execution with its own history (overhead).
There are more moving parts than a single Workflow.
Child execution details are not in the parent history (but are queryable independently).
Async children require explicit synchronization if needed.
More Workflow executions mean higher resource usage.
Starting a Child Workflow has more overhead than starting an Activity.

## Comparison with alternatives

| Approach | Modularity | Independent history | Can outlive parent | Overhead | Separate Workflow ID |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Child Workflow | High | Yes | Yes (ABANDON) | Medium | Yes |
| Activity | Medium | No | No | Low | No |
| Separate Workflow + Signals | High | Yes | Yes | High | Yes |
| Async Lambda | Low | No | No | Very Low | No |

## Best practices

- **Use unique Workflow IDs.** Generate unique IDs for Child Workflows to avoid conflicts.
- **Choose the appropriate policy.** Use TERMINATE for tightly coupled children, ABANDON for independent children.
- **Handle child failures.** Catch and handle Child Workflow exceptions appropriately.
- **Limit parallelism.** Do not spawn unlimited children; use batch patterns for large datasets.
- **Consider Activities first.** Use Activities for operations that do not need independent Workflow tracking.
- **Set timeouts.** Configure appropriate Workflow execution timeouts for children.
- **Use typed stubs.** Prefer typed stubs over untyped for compile-time safety.
- **Monitor child executions.** Track Child Workflow IDs for observability and debugging.

## Common pitfalls

- **Treating Child Workflows like Activities.** Child Workflows are for orchestration, not for executing external code. If you only need to call an API or run a function, use an Activity instead.
- **Spawning unbounded children in a loop.** Starting thousands of Child Workflows without batching can overwhelm the Temporal Service and bloat the parent's event history. Temporal enforces a hard limit of 2,000 pending (in-flight) children per parent, but the [recommended cap](/child-workflows#when-to-use-child-workflows) is lower: a single parent should not spawn more than 1,000 Child Workflow Executions in total, since each one adds more history to the parent than an Activity would. Use fixed-size batches or a sliding window.
- **Ignoring the Parent Close Policy.** The default policy is TERMINATE, which kills children when the parent closes. If children must outlive the parent, set the policy to ABANDON explicitly.
- **Using synchronous calls when async is needed.** Calling a Child Workflow synchronously blocks the parent until the child completes. For long-running children, use the async API (`Async.function()` in Java, `startChild()` in TypeScript, `start_child_workflow()` in Python, or collect Futures without calling `.Get()` in Go) to avoid stalling the parent.
- **Omitting Workflow IDs.** Without explicit Workflow IDs, you lose the ability to deduplicate or look up Child Workflows by a meaningful identifier. Generate deterministic IDs based on business keys.
- **Not handling child failures.** Child Workflow failures propagate to the parent as a Child Workflow Failure (`ChildWorkflowFailure` in TypeScript and Java, `ChildWorkflowError` in Python, `ChildWorkflowExecutionError` in Go), with the underlying cause in its `cause` field. If you do not catch and handle them, the parent Workflow fails as well.

## Related

### Patterns

- **[Parallel Execution](/design-patterns/parallel-execution)**: Running multiple children concurrently.
- **[Continue-As-New](/design-patterns/continue-as-new)**: Child Workflows can use Continue-As-New independently.
- **[Saga Pattern](/design-patterns/saga-pattern)**: Children as compensatable transactions.

### References

- [Parent Close Policy](/parent-close-policy): Canonical reference for `TERMINATE`, `ABANDON`, and `REQUEST_CANCEL`, including the default and how each behaves during a Continue-As-New.
- [Child Workflows (Go)](/develop/go/workflows/child-workflows) · [Child Workflows (Java)](/develop/java/workflows/child-workflows) · [Child Workflows (Python)](/develop/python/workflows/child-workflows) · [Child Workflows (TypeScript)](/develop/typescript/workflows/child-workflows): Official per-SDK how-to guides.

### Sample code

**Java:**
- [HelloChild](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloChild.java) — Basic synchronous Child Workflow.
- [Async Child Workflow](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/asyncchild) — Asynchronous child with ABANDON policy.
- [Async Untyped Child](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/asyncuntypedchild) — Untyped async Child Workflow.

**TypeScript:**
- [Child Workflows](https://github.com/temporalio/samples-typescript/tree/main/child-workflows) — Parent and child Workflow using `executeChild` and `startChild`.

**Python:**
- [Child Workflows](https://github.com/temporalio/samples-python/tree/main/hello/hello_child_workflow.py) — Basic Child Workflow using `execute_child_workflow`.

**Go:**
- [Child Workflow](https://github.com/temporalio/samples-go/tree/main/child-workflow) — Synchronous and async Child Workflow patterns.

---

# Continue-As-New Pattern

Source: https://docs.temporal.io/design-patterns/continue-as-new

> Prevents unbounded history growth by completing the current execution and starting a new one with fresh history.

## Overview

The Continue-As-New pattern allows long-running Workflows to reset their event history by completing the current execution and immediately starting a new one with fresh state.
This prevents Workflows from hitting Temporal's event history limits while maintaining logical continuity, making it essential for periodic tasks, infinite loops, and Workflows that process unbounded data streams.
By archiving old event history and starting fresh, Continue-As-New also reduces active storage costs — only the current execution's history remains in active storage while previous runs are moved to cheaper archived storage.

## Problem

Long-running Workflows — periodic tasks that run indefinitely, infinite loops spanning months or years, or Workflows processing an unbounded stream of data — accumulate Event History with every iteration. Left unchecked, that history eventually hits the 51,200-event limit, and the Workflow still needs to keep its state across whatever comes next.

Without Continue-As-New, the alternatives are all worse: manually stop and restart Workflows and lose continuity, build external orchestration to manage the Workflow's lifecycle, or accept degraded performance as history grows — and risk failure once it hits the limit.

## Solution

Continue-As-New completes the current Workflow execution and atomically starts a new one with the same Workflow ID.
The new execution begins with a fresh event history while preserving logical continuity.
You pass state as arguments to the new execution.

```mermaid
flowchart LR
    Start([Start]) --> Exec1[Execution 1<br/>Process batch]
    Exec1 --> Check1{History<br/>large?}
    Check1 -->|Yes| CAN1[Continue-As-New]
    Check1 -->|No| More1{More<br/>data?}
    More1 -->|Yes| Exec1
    More1 -->|No| End1([Complete])
    
    CAN1 -.->|Fresh history<br/>Same Workflow ID| Exec2[Execution 2<br/>Process batch]
    Exec2 --> Check2{History<br/>large?}
    Check2 -->|Yes| CAN2[Continue-As-New]
    Check2 -->|No| More2{More<br/>data?}
    More2 -->|Yes| Exec2
    More2 -->|No| End2([Complete])
    
    CAN2 -.-> Etc[...]
    classDef highlight stroke-width:1px
    class CAN1,CAN2 highlight
```

The following describes each step in the diagram:

1. The Workflow starts Execution 1 and processes a batch of data.
2. After each batch, the Workflow checks whether the history is getting large.
3. If the history is large, the Workflow calls Continue-As-New, which starts Execution 2 with a fresh history and the same Workflow ID.
4. If the history is not large and more data remains, the Workflow loops and processes the next batch.
5. If no more data remains, the Workflow completes normally.

The following implementation shows a data processor that passes a cursor and a running total across executions.
When the batch is full (indicating more data to process), the Workflow calls Continue-As-New with the updated state:

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import fetchBatch, process

BATCH_SIZE = 100

@workflow.defn
class DataProcessorWorkflow:
    @workflow.run
    async def run(self, cursor: str, total_processed: int = 0) -> None:
        batch = await workflow.execute_activity(
            fetchBatch, args=[cursor, BATCH_SIZE],
            start_to_close_timeout=timedelta(seconds=60),
        )

        for record in batch:
            await workflow.execute_activity(
                process, record,
                start_to_close_timeout=timedelta(seconds=60),
            )
            total_processed += 1
            cursor = record.id

        if len(batch) == BATCH_SIZE:
            # More data to process - continue as new with updated state
            workflow.continue_as_new(args=[cursor, total_processed])
        # Otherwise complete normally
```

**Go**

```go
// data_processor_workflow.go
const BatchSize = 100

func DataProcessorWorkflow(ctx workflow.Context, cursor string, totalProcessed int) error {
    ao := workflow.ActivityOptions{StartToCloseTimeout: time.Minute}
    ctx = workflow.WithActivityOptions(ctx, ao)

    var batch []Record
    err := workflow.ExecuteActivity(ctx, FetchBatch, cursor, BatchSize).Get(ctx, &batch)
    if err != nil {
        return err
    }

    for _, record := range batch {
        err = workflow.ExecuteActivity(ctx, Process, record).Get(ctx, nil)
        if err != nil {
            return err
        }
        totalProcessed++
        cursor = record.ID
    }

    if len(batch) == BatchSize {
        // More data to process - continue as new with updated state
        return workflow.NewContinueAsNewError(ctx, DataProcessorWorkflow, cursor, totalProcessed)
    }
    // Otherwise complete normally
    return nil
}
```

**Java**

```java
// DataProcessorWorkflowImpl.java
@WorkflowInterface
public interface DataProcessorWorkflow {
  @WorkflowMethod
  void processData(String cursor, int totalProcessed);
}

public class DataProcessorWorkflowImpl implements DataProcessorWorkflow {
  private static final int BATCH_SIZE = 100;

  @Override
  public void processData(String cursor, int totalProcessed) {
    List<Record> batch = activities.fetchBatch(cursor, BATCH_SIZE);

    for (Record record : batch) {
      activities.process(record);
      totalProcessed++;
      cursor = record.getId();
    }

    if (batch.size() == BATCH_SIZE) {
      // More data to process - continue as new with updated state
      DataProcessorWorkflow continueAsNew =
          Workflow.newContinueAsNewStub(DataProcessorWorkflow.class);
      continueAsNew.processData(cursor, totalProcessed);
    }
    // Otherwise complete normally
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { continueAsNew, proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { fetchBatch, process } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

const BATCH_SIZE = 100;

export async function dataProcessorWorkflow(
  cursor: string,
  totalProcessed: number = 0
): Promise<void> {
  const batch = await fetchBatch(cursor, BATCH_SIZE);

  for (const record of batch) {
    await process(record);
    totalProcessed++;
    cursor = record.id;
  }

  if (batch.length === BATCH_SIZE) {
    // More data to process - continue as new with updated state
    await continueAsNew<typeof dataProcessorWorkflow>(cursor, totalProcessed);
  }
  // Otherwise complete normally
}
```

The Workflow fetches a batch of records, processes each one, and updates the cursor.
If the batch is full, the Workflow triggers Continue-As-New with the updated cursor and total.
In Java, `Workflow.newContinueAsNewStub()` creates a typed stub.
In TypeScript, `continueAsNew()` throws a special error that the runtime intercepts.
In Python, `workflow.continue_as_new()` immediately stops the current execution and starts a new one.
In Go, `workflow.NewContinueAsNewError()` returns a special error that signals the runtime to continue as new.
If the batch is smaller than `BATCH_SIZE`, no more data remains and the Workflow completes.

## Implementation

### Use the Continue-As-New suggestion

Instead of tracking iteration counts manually, you can use the SDK's built-in suggestion to let Temporal tell you when the history is getting large:

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow

@workflow.defn
class DataProcessorWorkflow:
    @workflow.run
    async def run(self, cursor: str, total_processed: int = 0) -> None:
        batch = await workflow.execute_activity(
            fetchBatch, args=[cursor, BATCH_SIZE],
            start_to_close_timeout=timedelta(seconds=60),
        )

        for record in batch:
            await workflow.execute_activity(
                process, record,
                start_to_close_timeout=timedelta(seconds=60),
            )
            total_processed += 1
            cursor = record.id

            # Check if history is getting large
            if workflow.info().is_continue_as_new_suggested():
                workflow.continue_as_new(args=[cursor, total_processed])

        # Continue processing or complete
```

**Go**

```go
// data_processor_workflow.go
func DataProcessorWorkflow(ctx workflow.Context, cursor string, totalProcessed int) error {
    ao := workflow.ActivityOptions{StartToCloseTimeout: time.Minute}
    ctx = workflow.WithActivityOptions(ctx, ao)

    var batch []Record
    err := workflow.ExecuteActivity(ctx, FetchBatch, cursor, BatchSize).Get(ctx, &batch)
    if err != nil {
        return err
    }

    for _, record := range batch {
        err = workflow.ExecuteActivity(ctx, Process, record).Get(ctx, nil)
        if err != nil {
            return err
        }
        totalProcessed++
        cursor = record.ID

        // Check if history is getting large
        if workflow.GetInfo(ctx).GetContinueAsNewSuggested() {
            return workflow.NewContinueAsNewError(ctx, DataProcessorWorkflow, cursor, totalProcessed)
        }
    }

    // Continue processing or complete
    return nil
}
```

**Java**

```java
// DataProcessorWorkflowImpl.java
public class DataProcessorWorkflowImpl implements DataProcessorWorkflow {

  @Override
  public void processData(String cursor, int totalProcessed) {
    List<Record> batch = activities.fetchBatch(cursor, BATCH_SIZE);

    for (Record record : batch) {
      activities.process(record);
      totalProcessed++;
      cursor = record.getId();

      // Check if history is getting large
      if (Workflow.getInfo().isContinueAsNewSuggested()) {
        DataProcessorWorkflow continueAsNew =
            Workflow.newContinueAsNewStub(DataProcessorWorkflow.class);
        continueAsNew.processData(cursor, totalProcessed);
        return;
      }
    }

    // Continue processing or complete
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { continueAsNew, workflowInfo, proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { fetchBatch, process } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

export async function dataProcessorWorkflow(
  cursor: string,
  totalProcessed: number = 0
): Promise<void> {
  const batch = await fetchBatch(cursor, BATCH_SIZE);

  for (const record of batch) {
    await process(record);
    totalProcessed++;
    cursor = record.id;

    // Check if history is getting large
    if (workflowInfo().continueAsNewSuggested) {
      await continueAsNew<typeof dataProcessorWorkflow>(cursor, totalProcessed);
    }
  }

  // Continue processing or complete
}
```

Each SDK provides a method to check if the history is approaching the limit:
- Java: `Workflow.getInfo().isContinueAsNewSuggested()`
- TypeScript: `workflowInfo().continueAsNewSuggested`
- Python: `workflow.info().is_continue_as_new_suggested()`
- Go: `workflow.GetInfo(ctx).GetContinueAsNewSuggested()`

This approach is more reliable than a fixed iteration count because it accounts for the actual number of events generated per iteration.

## When to use

The Continue-As-New pattern is a good fit for periodic Workflows running indefinitely (cron-like behavior), processing unbounded data streams, long-running Workflows with repetitive patterns, Workflows that accumulate state over many iterations, and preventing event history from growing too large.

It is not a good fit for short-lived Workflows (under 1,000 events), Workflows that naturally complete, one-time batch processing, or Workflows that require full history for audit purposes.

## Benefits and trade-offs

Continue-As-New allows you to run Workflows indefinitely without history limits.
Fresh history keeps Workflow execution fast.
It reduces active storage costs by archiving old event history — more aggressive iteration limits mean more frequent archiving, keeping active storage minimal.
The transition is atomic with no gap between old and new execution.
You pass state as arguments to the new execution, and the Workflow ID remains the same, maintaining logical continuity for Queries and Signals.

The trade-offs to consider are that previous execution history is archived separately.
You must explicitly pass state as arguments (manual state management).
Queries only see the current execution's state.
Debugging requires tracing across multiple execution runs.
You cannot undo Continue-As-New once triggered.

## Comparison with alternatives

| Approach | History reset | State continuity | Use case |
| :--- | :--- | :--- | :--- |
| Continue-As-New | Yes | Manual | Long-running periodic |
| Child Workflows | Per child | Automatic | Parallel processing |
| Cron Schedule | Yes | None | Fixed schedule tasks |
| Manual Restart | Yes | None | One-time Workflows |

## Best practices

- **Use the continue-as-new suggestion.** Check the SDK's built-in suggestion (`isContinueAsNewSuggested()` in Java, `continueAsNewSuggested` in TypeScript, `is_continue_as_new_suggested()` in Python, `GetContinueAsNewSuggested()` in Go) to automatically detect when history is large.
- **Set aggressive iteration limits.** Continue as new every 100–1000 iterations to prevent history buildup and reduce storage costs. Balance frequency with the overhead of creating new executions.
- **Pass minimal state.** Only pass necessary state to keep arguments small.
- **Add exit Signals.** Allow graceful termination via Signals.
- **Log transitions.** Log when continuing as new for observability.
- **Version carefully.** Ensure new code can handle state from old executions.
- **Monitor history size.** Track event count and continue before hitting limits.
- **Use typed APIs.** In Java, prefer `newContinueAsNewStub()` over untyped `continueAsNew()`. In TypeScript, use the generic `continueAsNew<typeof myWorkflow>()` for type safety.
- **Consider cron.** For fixed Schedules, use [Temporal Schedules](/schedule) instead.
- **Test state transfer.** Verify state correctly passes between executions.

## Common pitfalls

- **Passing too much state.** Continue-As-New arguments are serialized into the first event of the new execution. Large payloads slow down startup and increase storage costs. Pass only the minimal state needed.
- **Forgetting to drain Signals before continuing.** Any Signals received but not yet processed are lost when Continue-As-New starts a fresh execution. Drain your Signal channel and carry pending Signals forward as arguments.
- **Using a fixed iteration count instead of the built-in suggestion.** Different Workflow paths generate different numbers of events per iteration. A fixed count may continue too early or too late. Use the SDK's built-in continue-as-new suggestion for accurate detection.
- **Not versioning state arguments.** When you change the Workflow method signature or state shape, in-flight executions may continue as new into code that cannot deserialize the old arguments. Use versioning or backward-compatible argument types.
- **Calling Continue-As-New from a Signal handler.** Triggering Continue-As-New inside a Signal handler can cause Signal loss because the handler may preempt other pending Signals. Always set a flag in the Signal handler and call Continue-As-New from the main Workflow thread, where all Signal handlers are guaranteed to have run first.
- **Not accounting for Child Workflows.** Continue-As-New closes the current Workflow Execution, which triggers the Parent Close Policy on all Child Workflows. By default, children are terminated. If children must survive, set `ParentClosePolicy` to `ABANDON` and pass their Workflow IDs to the new execution so you can interact with them via external handles.
- **Caching Run IDs for external interaction.** Continue-As-New creates a new Run ID. If external callers cache the old Run ID for Signals or Queries, they will get a "workflow execution already completed" error. Always use Workflow ID without a Run ID (or an empty Run ID) so the request routes to the currently running execution.
- **Catching the Continue-As-New exception.** In TypeScript and Python, Continue-As-New is implemented by throwing a special exception. Wrapping it in a try-catch or try-except can suppress the transition and cause unexpected behavior. Let the exception propagate unhandled. In Go, return the `ContinueAsNewError` from the Workflow function without wrapping it.

## Related

### Patterns

- **[Entity Workflow](/design-patterns/entity-workflow)**: Long-lived Workflows that model business entities, relying on Continue-As-New to prevent unbounded history.
- **[Child Workflows](/design-patterns/child-workflows)**: Decomposing work into sub-Workflows. Consider Parent Close Policy when combining with Continue-As-New.
- **[Signal with Start](/design-patterns/signal-with-start)**: Idempotent Workflow start with an initial Signal — use Workflow ID without Run ID to interact with continued executions.

### Guides

- [Track customer loyalty points with durable Workflows](/guides/entity-pattern-loyalty-points): Uses Continue-As-New to keep a long-lived customer loyalty account within Event History limits, including an upgrade path at the Continue-As-New boundary for Worker Versioning.
- [Player Sessions That Survive Anything](/guides/durable-gaming-sessions): Uses Continue-As-New so a multiplayer game session can run for weeks without unbounded history growth.

### Sample code

**Java:**
- [Cron Workflow](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloCron.java) — Periodic Workflow using Continue-As-New.
- [Heartbeating Activity Batch](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/heartbeatingactivity) — Batch processing with Continue-As-New for large datasets.

**TypeScript:**
- [Continue-As-New](https://github.com/temporalio/samples-typescript/tree/main/continue-as-new) — Basic Continue-As-New with `continueAsNew()`.
- [Safe Message Handlers](https://github.com/temporalio/samples-typescript/tree/main/message-passing/safe-message-handlers) — Entity Workflow with Continue-As-New and `continueAsNewSuggested`.

**Python:**
- [Safe Message Handlers](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) — Entity Workflow with Continue-As-New and `is_continue_as_new_suggested()`.

**Go:**
- [Safe Message Handlers](https://github.com/temporalio/samples-go/tree/main/safe_message_handler) — Entity Workflow with Continue-As-New and `GetContinueAsNewSuggested()`.

### References

- [Continue-As-New](/workflow-execution/continue-as-new) — Concept reference for the Continue-As-New mechanism.
- [Temporal Blog: How to Keep a Workflow Running Indefinitely Long](https://temporal.io/blog/very-long-running-workflows) — Detailed guidance on managing Workflows that run forever.
- [Temporal Blog: Workflows as Actors](https://temporal.io/blog/workflows-as-actors-is-it-really-possible) — Using Continue-As-New with the Entity Workflow / Actor pattern.

---

# Delayed Callback (Webhooks)

Source: https://docs.temporal.io/design-patterns/delayed-callback

> Webhooks become durable: inbound calls arrive as Signals, outbound calls fire after a durable sleep, and Activities complete later via task tokens.

## Overview
Delayed Callback patterns in Temporal manage delayed completion notification between two systems. They use durable timers, and you define the behavior entirely in code.

### Webhooks
You build and configure [Webhooks](https://www.redhat.com/en/topics/automation/what-is-a-webhook) with the Delayed Callback patterns. There are two recommended patterns for integrating Temporal Workflows to communicate through HTTP callbacks: 
- waiting and receiving inbound webhooks
- firing delayed outbound callbacks

Included is a pattern for completing activities asynchronously via a callback token and some guidance on when to use it.

Temporal lets you build durable, observable webhook-based integrations without ad hoc queues, cron jobs, or fragile state machines.

## Problem

Waiting for another system without Durable Execution is hard. You must implement your own:
- durable timers (for example, with a cron per timer)
- retry queues
- state stores
- reconciliation jobs

All to support a delayed callback.

Webhook-based integrations have several failure modes that are difficult to handle without durable infrastructure:

- An inbound message fires before the target system is ready or at the proper state, causing the message to be ignored or lost.
- An outbound HTTP callback fails midway through a multi-step cross-system process, and there is no record of what was sent, retried, or skipped.
- An external job (payment processor, ML pipeline, etc.) completes and calls back, but the in-process state that was waiting for the callback has been lost to poor state management or an application restart.
- A delayed callback is scheduled via a cron job or message queue, but the scheduling system and the application process have no shared recovery mechanism.

## Solution

Temporal solves these problems with [durable timers](/workflow-execution/timers-delays) in a Workflow. You use [Signals or Updates](/encyclopedia/workflow-message-passing) to send events to a Workflow. All of these are Temporal code — no extra infrastructure to deploy or manage.

- **Pattern 1 — Inbound Callback:** Route the incoming HTTP request to a Temporal Signal-with-Start. The Workflow is the durable recipient; if it is not running yet, Temporal creates it and delivers the Signal atomically.
- **Pattern 2 — Delayed Outbound Callbacks:** Use a durable `workflow.sleep()` to set the proper delay before executing the outbound HTTP activity. The sleep timer survives worker and server restarts; the activity retries automatically on failure.
- **Pattern 3 — Async Activity Completion:** The activity records a task token before returning, and your callback endpoint uses that token to complete the activity from the outside. The Workflow resumes with the result as if the activity had returned normally.

### Pattern 1 — inbound webhooks

```mermaid
sequenceDiagram
    participant E as External Service
    participant A as Your API Handler
    participant T as Temporal Cluster
    participant W as Workflow

    E->>A: POST /webhook (payload)
    A->>T: signal_with_start(workflow_id,<br/>"payment_received", payload)
    Note over T: Atomic: start if not running, then signal
    T->>W: Start workflow (if not running)<br/>+ deliver signal
    W->>W: Wake up, process payload
    W->>W: Execute follow-on activities
```

The following describes each step in the diagram:

1. An external service sends an HTTP POST to your API handler — this is the inbound webhook.
2. Your handler calls `signal_with_start` on the Temporal client with the Workflow ID and payload. The handler can return an HTTP 200 immediately after this call; Temporal takes responsibility for delivery.
3. Temporal atomically starts the Workflow if it is not already running, then delivers the Signal — no race condition between "start" and "signal."
4. The Workflow wakes up exactly where it was blocked waiting (or begins execution if newly created) and processes the payload.

### Pattern 2 — delayed outbound callbacks

```mermaid
sequenceDiagram
    participant C as Client
    participant W as Workflow
    participant T as Temporal Cluster
    participant E as External Service

    C->>T: Start DelayedCallbackWorkflow<br/>(url, data, delay)
    T->>W: Schedule first task
    W->>T: workflow.sleep(delay)
    Note over T: Durable timer (survives restarts)
    T->>W: Timer fires after delay
    W->>W: execute_activity(send_callback, url, data)
    W->>E: POST callback_url (data)
    E-->>W: HTTP 200
    W-->>C: Workflow complete
```

The following describes each step in the diagram:

1. The client starts the Workflow with a target URL, payload, and delay duration.
2. The Workflow calls `workflow.sleep()`. This stores a durable timer in the Temporal cluster — not in process memory.
3. If any worker restarts during the delay, the timer continues. When it fires, Temporal schedules the next Workflow Task on a healthy worker.
4. The Workflow executes an activity that performs the outbound HTTP POST. If the POST fails, Temporal retries it with the configured retry policy.

### Pattern 3 — async Activity Completion

```mermaid
sequenceDiagram
    participant W as Workflow
    participant A as Activity
    participant E as External Service
    participant CB as Your Callback API

    W->>A: execute_activity(submit_job)
    A->>E: Submit job, record task_token
    A-->>W: Return (workflow paused waiting)
    Note over W: Waiting for external completion...
    E->>CB: POST /callback (result)
    CB->>W: complete_async_activity<br/>(task_token, result)
    W->>W: Resume with result
```

The following describes each step in the diagram:

1. The Workflow executes an activity that submits a job to an external system.
2. The activity records its task token (an opaque handle Temporal provides) alongside the submitted job ID — for example, in a database row.
3. The activity returns without waiting; the Workflow is now paused waiting for the activity to complete externally.
4. When the external system finishes, it calls your callback endpoint with the result.
5. Your callback handler retrieves the task token from the database and completes the activity through the Temporal client using that token. The Workflow resumes immediately with the result.

## Implementation

### Pattern 1 — inbound webhooks via Signal-With-Start

The following examples show an `OrderWorkflow` that waits for a `payment_received` Signal. The starter uses Signal-with-Start to atomically create the Workflow and deliver the payment signal in one call — exactly what your HTTP handler would do on a real POST.

**Python**

```python
# workflows.py
import asyncio
from dataclasses import dataclass
from datetime import timedelta
from typing import Optional

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import process_payment
    from shared import OrderInput, PaymentPayload

@workflow.defn
class OrderWorkflow:
    def __init__(self) -> None:
        self._payment: Optional[PaymentPayload] = None

    @workflow.run
    async def run(self, order: OrderInput) -> str:
        workflow.logger.info(f"Order {order.order_id}: waiting for payment webhook")

        # Block until the inbound webhook signal arrives (or timeout after 24 hours)
        await workflow.wait_condition(
            lambda: self._payment is not None,
            timeout=timedelta(hours=24),
        )

        if self._payment is None:
            return f"Order {order.order_id}: timed out waiting for payment"

        result = await workflow.execute_activity(
            process_payment,
            self._payment,
            start_to_close_timeout=timedelta(seconds=30),
        )
        return result

    @workflow.signal
    async def payment_received(self, payload: PaymentPayload) -> None:
        workflow.logger.info(f"Payment signal received: {payload.payment_id}")
        self._payment = payload
```

**Java**

```java
// OrderWorkflow.java
import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.SignalMethod;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;

import java.time.Duration;

@WorkflowInterface
public interface OrderWorkflow {
    @WorkflowMethod
    String run(Shared.OrderInput order);

    @SignalMethod
    void paymentReceived(Shared.PaymentPayload payload);

    final class Impl implements OrderWorkflow {
        private final Activities activities = Workflow.newActivityStub(
                Activities.class,
                ActivityOptions.newBuilder()
                        .setStartToCloseTimeout(Duration.ofSeconds(30))
                        .build());

        private Shared.PaymentPayload payment = null;

        @Override
        public String run(Shared.OrderInput order) {
            System.out.println("Order " + order.orderId() + ": waiting for payment webhook");

            // Block until the inbound webhook signal arrives (or timeout after 24 hours)
            boolean received = Workflow.await(Duration.ofHours(24), () -> payment != null);
            if (!received) {
                return "Order " + order.orderId() + ": timed out waiting for payment";
            }

            return activities.processPayment(payment);
        }

        @Override
        public void paymentReceived(Shared.PaymentPayload payload) {
            System.out.println("Payment signal received: " + payload.paymentId());
            this.payment = payload;
        }
    }
}
```

**Go**

```go
// workflows.go
package main

import (
	"time"

	"go.temporal.io/sdk/workflow"
)

func OrderWorkflow(ctx workflow.Context, order OrderInput) (string, error) {
	workflow.GetLogger(ctx).Info("Order waiting for payment webhook", "order_id", order.OrderID)

	var payment *PaymentPayload

	// Block until the inbound webhook signal arrives (or timeout after 24 hours)
	selector := workflow.NewSelector(ctx)
	timerFired := false

	timerCtx, cancelTimer := workflow.WithCancel(ctx)
	timer := workflow.NewTimer(timerCtx, 24*time.Hour)

	signalCh := workflow.GetSignalChannel(ctx, SignalName)
	selector.AddReceive(signalCh, func(ch workflow.ReceiveChannel, more bool) {
		ch.Receive(ctx, &payment)
		cancelTimer()
	})
	selector.AddFuture(timer, func(f workflow.Future) {
		if err := f.Get(ctx, nil); err == nil {
			timerFired = true
		}
	})

	selector.Select(ctx)

	if timerFired || payment == nil {
		return "Order " + order.OrderID + ": timed out waiting for payment", nil
	}

	ao := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 30 * time.Second,
	})

	var result string
	err := workflow.ExecuteActivity(ao, ProcessPayment, payment).Get(ao, &result)
	return result, err
}
```

The Starter uses Signal-with-Start to atomically create the Workflow (if needed) and deliver the simulated webhook payload:

**Python**

```python
# starter.py
import asyncio
import time

from temporalio.client import Client

from shared import TASK_QUEUE, OrderInput, PaymentPayload
from workflows import OrderWorkflow

async def main() -> None:
    client = await Client.connect("localhost:7233")

    order_id = f"order-{int(time.time() * 1000)}"
    order = OrderInput(order_id=order_id, amount=99.99)
    payment = PaymentPayload(payment_id=f"pay-{int(time.time() * 1000)}", amount=99.99)

    print(f"Sending webhook for order {order_id}")

    # Signal-with-Start: atomically starts the workflow (if not running) and
    # delivers the payment signal — this is exactly what your HTTP handler would do.
    handle = await client.start_workflow(
        OrderWorkflow.run,
        order,
        id=f"order-{order_id}",
        task_queue=TASK_QUEUE,
        start_signal="payment_received",
        start_signal_args=[payment],
    )
    print(f"Webhook signal sent: {payment.payment_id}")

    result = await handle.result()
    print(f"Order completed: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```

**Java**

```java
// Starter.java
import io.temporal.client.BatchRequest;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.client.WorkflowStub;
import io.temporal.serviceclient.WorkflowServiceStubs;

public class Starter {
    public static void main(String[] args) throws Exception {
        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service);

        String orderId = "order-" + System.currentTimeMillis();
        Shared.PaymentPayload payment = new Shared.PaymentPayload(
                "pay-" + System.currentTimeMillis(), 99.99);

        OrderWorkflow workflow = client.newWorkflowStub(
                OrderWorkflow.class,
                WorkflowOptions.newBuilder()
                        .setTaskQueue(Shared.TASK_QUEUE)
                        .setWorkflowId("order-" + orderId)
                        .build());

        System.out.println("Sending webhook for order " + orderId);

        // Signal-with-Start: atomically starts the workflow (if not running) and
        // delivers the payment signal — this is exactly what your HTTP handler would do.
        BatchRequest request = client.newSignalWithStartRequest();
        request.add(workflow::run, new Shared.OrderInput(orderId, 99.99));
        request.add(workflow::paymentReceived, payment);
        client.signalWithStart(request);
        System.out.println("Webhook signal sent: " + payment.paymentId());

        // Wait for the workflow to complete
        String result = WorkflowStub.fromTyped(workflow).getResult(String.class);
        System.out.println("Order completed: " + result);

        System.exit(0);
    }
}
```

**Go**

```go
// starter.go
package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"go.temporal.io/sdk/client"
)

func main() {
	c, err := client.Dial(client.Options{HostPort: "localhost:7233"})
	if err != nil {
		log.Fatalln("Unable to create client:", err)
	}
	defer c.Close()

	ctx := context.Background()
	orderID := fmt.Sprintf("order-%d", time.Now().UnixMilli())
	workflowID := "order-" + orderID

	order := OrderInput{OrderID: orderID, Amount: 99.99}
	payment := PaymentPayload{
		PaymentID: fmt.Sprintf("pay-%d", time.Now().UnixMilli()),
		Amount:    99.99,
	}

	fmt.Printf("Sending webhook for order %s\n", orderID)

	// Signal-with-Start: atomically starts the workflow (if not running) and
	// delivers the payment signal — this is exactly what your HTTP handler would do.
	we, err := c.SignalWithStartWorkflow(
		ctx,
		workflowID,
		SignalName,
		payment,
		client.StartWorkflowOptions{
			ID:        workflowID,
			TaskQueue: TaskQueue,
		},
		OrderWorkflow,
		order,
	)
	if err != nil {
		log.Fatalln("SignalWithStart failed:", err)
	}
	fmt.Printf("Webhook signal sent: %s\n", payment.PaymentID)

	var result string
	if err := we.Get(ctx, &result); err != nil {
		log.Fatalln("Workflow result failed:", err)
	}
	fmt.Printf("Order completed: %s\n", result)
}
```

### Pattern 2 — delayed outbound callbacks

Use a durable `workflow.sleep()` before the outbound activity. The timer is stored in the Temporal cluster, not in process memory — it survives any number of worker restarts.

**Python**

```python
# delayed_callback_workflow.py
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import send_webhook_callback
    from shared import CallbackInput

@workflow.defn
class DelayedCallbackWorkflow:
    @workflow.run
    async def run(self, input: CallbackInput) -> str:
        workflow.logger.info(
            f"Sleeping {input.delay_seconds}s before calling {input.callback_url}"
        )

        # Durable sleep — survives worker restarts, server restarts, everything
        await workflow.sleep(timedelta(seconds=input.delay_seconds))

        # Fire the outbound callback; Temporal retries on HTTP failure
        result = await workflow.execute_activity(
            send_webhook_callback,
            input,
            start_to_close_timeout=timedelta(minutes=5),
        )
        workflow.logger.info(f"Callback delivered to {input.callback_url}")
        return result
```

**Java**

```java
// DelayedCallbackWorkflow.java
import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;

import java.time.Duration;

@WorkflowInterface
public interface DelayedCallbackWorkflow {
    @WorkflowMethod
    void run(Shared.CallbackInput input);

    final class Impl implements DelayedCallbackWorkflow {
        private final Activities activities = Workflow.newActivityStub(
                Activities.class,
                ActivityOptions.newBuilder()
                        .setStartToCloseTimeout(Duration.ofMinutes(5))
                        .build());

        @Override
        public void run(Shared.CallbackInput input) {
            System.out.println("Sleeping " + input.delaySeconds() + "s before calling " + input.callbackUrl());

            // Durable sleep — survives worker restarts, server restarts, everything
            Workflow.sleep(Duration.ofSeconds(input.delaySeconds()));

            // Fire the outbound callback; Temporal retries on HTTP failure
            activities.sendWebhookCallback(input);
            System.out.println("Callback delivered to " + input.callbackUrl());
        }
    }
}
```

**Go**

```go
// delayed_callback.go (add to workflows.go)
package main

import (
	"time"

	"go.temporal.io/sdk/workflow"
)

func DelayedCallbackWorkflow(ctx workflow.Context, input CallbackInput) error {
	workflow.GetLogger(ctx).Info("Sleeping before callback",
		"delay", input.DelaySeconds, "url", input.CallbackURL)

	// Durable sleep — survives worker restarts, server restarts, everything
	if err := workflow.Sleep(ctx, time.Duration(input.DelaySeconds)*time.Second); err != nil {
		return err
	}

	// Fire the outbound callback; Temporal retries on HTTP failure
	ao := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 5 * time.Minute,
	})
	return workflow.ExecuteActivity(ao, SendWebhookCallback, input).Get(ao, nil)
}
```

### Pattern 3 — async Activity Completion

The activity records a task token before returning. Your callback endpoint later uses that token to complete the activity and unblock the Workflow.

**Python**

```python
# async_completion_activities.py
import asyncio
from temporalio import activity
from temporalio.client import Client

from shared import JobInput, JobResult

@activity.defn
async def submit_job(input: JobInput) -> str:
    """Submit job to external system and return immediately.
    The activity completes asynchronously when the callback arrives."""

    # Get the task token — this is the claim ticket
    task_token = activity.info().task_token

    # Submit the job to the external system, persisting the task token
    # so your callback handler can retrieve it later
    job_id = await external_service.submit(
        payload=input.payload,
        callback_url=f"https://your-api.example.com/callback",
        task_token_hex=task_token.hex(),  # store alongside job_id
    )
    activity.logger.info(f"Job {job_id} submitted; waiting for async callback")

    # Tell Temporal not to mark the activity complete on return; the external
    # callback will complete it later using the task token.
    activity.raise_complete_async()

# In your webhook callback handler (e.g., FastAPI route):
async def handle_callback(result: str, task_token_hex: str) -> None:
    token = bytes.fromhex(task_token_hex)
    client = await Client.connect("localhost:7233")
    handle = client.get_async_activity_handle(task_token=token)
    await handle.complete(result)
    # Workflow resumes with `result` immediately
```

**Java**

```java
// AsyncCompletionActivity.java — submit side
import io.temporal.activity.Activity;
import io.temporal.activity.ActivityExecutionContext;
import io.temporal.activity.ActivityInterface;
import io.temporal.activity.ActivityMethod;
import io.temporal.client.ActivityCompletionClient;

@ActivityInterface
public interface AsyncJobActivity {
    @ActivityMethod
    String submitJob(Shared.JobInput input);
}

public class AsyncJobActivityImpl implements AsyncJobActivity {
    private final ActivityCompletionClient completionClient;

    public AsyncJobActivityImpl(ActivityCompletionClient completionClient) {
        this.completionClient = completionClient;
    }

    @Override
    public String submitJob(Shared.JobInput input) {
        ActivityExecutionContext context = Activity.getExecutionContext();

        // Get the task token — this is the claim ticket
        byte[] taskToken = context.getTaskToken();

        // Submit to external system, persisting the task token so the callback can retrieve it
        String jobId = ExternalService.submit(input.payload(), taskToken);
        System.out.println("Job " + jobId + " submitted; waiting for async callback");

        // Tell Temporal not to mark the activity complete yet
        context.doNotCompleteOnReturn();
        return null; // ignored
    }
}

// In your callback handler:
// completionClient.complete(taskToken, result);
```

**Go**

```go
// async_completion.go
package main

import (
	"context"
	"encoding/hex"
	"fmt"

	"go.temporal.io/sdk/activity"
	"go.temporal.io/sdk/client"
)

// SubmitJob submits a job and returns immediately; the activity completes
// when the external callback arrives and calls CompleteAsyncActivity.
func SubmitJob(ctx context.Context, input JobInput) (string, error) {
	info := activity.GetInfo(ctx)

	// Persist the task token so the callback handler can retrieve it by job ID
	taskToken := info.TaskToken
	jobID := fmt.Sprintf("job-%d", info.StartedTime.UnixMilli())

	if err := persistTaskToken(jobID, hex.EncodeToString(taskToken)); err != nil {
		return "", err
	}

	fmt.Printf("Job %s submitted; waiting for async callback\n", jobID)

	// Return ErrResultPending to tell Temporal not to mark the activity complete yet
	return "", activity.ErrResultPending
}

// CompleteJob is called by your webhook callback handler to unblock the workflow.
func CompleteJob(ctx context.Context, c client.Client, jobID string, result string) error {
	tokenHex, err := loadTaskToken(jobID)
	if err != nil {
		return err
	}
	taskToken, _ := hex.DecodeString(tokenHex)
	return c.CompleteActivity(ctx, taskToken, result, nil)
}
```

## When to use

| Scenario | Pattern |
| :--- | :--- |
| External service POSTs a webhook (Workflow may or may not be running) | Signal-with-Start (Pattern 1) |
| Fire an outbound HTTP callback after a delay (seconds to years) | `workflow.sleep()` + activity (Pattern 2) |
| Submit a job to an external system; wait for its completion webhook | Async activity completion (Pattern 3) |
| Poll an external system that does not support webhooks | [Polling External Services](/design-patterns/polling) pattern |

**Do not use** Pattern 2 for delays shorter than one second. Timer durations range from one second to several years, and your Workflows should not rely on [sub-second accuracy for Timers](/workflow-execution/timers-delays).

## Benefits and trade-offs

**Benefits**

- Retries and backoff on outbound HTTP calls come for free via Temporal's retry policy — no custom retry queues needed.
- Workflow state survives worker restarts, deploys, and infrastructure failures; durable timers continue without a running process.
- Every in-flight delayed callback is visible in the Temporal UI with its scheduled time, payload, and retry count.
- Signal-with-Start eliminates the race condition between "does the Workflow exist?" and "deliver the event."
- Async activity completion decouples job submission from job completion without polling.

**Trade-offs**

- Your inbound webhook handler requires a Temporal client; you need the client library in the service receiving webhooks.
- Task tokens for async completion must be persisted outside Temporal (for example, in a database); if that store is unavailable the callback cannot complete.
- Workflow IDs must be deterministic and stable across webhook deliveries (order ID, user ID, etc.) so that Signal-with-Start routes to the correct instance.

## Comparison with alternatives

| Approach | Durability | Retries | Observability | Complexity |
| :--- | :--- | :--- | :--- | :--- |
| Temporal Signals + Workflows | Durable — survives restarts | Built-in, configurable | Full Temporal UI | Low — primitives compose naturally |
| Message queue (SQS, Kafka) | Durable (queue level) | Limited, manual DLQ | Requires external tooling | Medium — must handle ordering, DLQ |
| Redis `SET` + cron job | In-memory/volatile | Manual | None | High — cron + polling + error handling |
| Direct HTTP retry loops | Process lifetime only | Manual with `time.sleep` | None | High — fragile without process supervisor |

## Best practices

- Use stable, business-meaningful Workflow IDs (for example, `order-{order_id}`) so that Signal-with-Start and queries always route to the right Workflow.
- Return HTTP 200 from your inbound webhook handler as soon as you have called `signal_with_start`; do not wait for the Workflow to process the payload.
- Set a realistic `start_to_close_timeout` on outbound callback activities — long enough for the destination to respond, short enough to surface failures quickly.
- For async activity completion, persist the task token in a transactional write alongside the job submission so you never lose the token.
- Add a timeout to the `workflow.wait_condition` / `Workflow.await` call in inbound webhook Workflows so they do not wait indefinitely if the webhook is never delivered.
- For Pattern 3, use `heartbeat` if the external job takes longer than the activity heartbeat timeout to report back — heartbeating keeps the activity lease alive.

## Common pitfalls

- **Sending a plain Signal to a Workflow that does not exist** causes an error. Use Signal-with-Start when the Workflow may not be running.
- **Using `time.sleep()` (non-durable)** in Pattern 2 instead of `workflow.sleep()`. A process sleep disappears on restart; only Temporal's timer is durable.
- **Non-deterministic Workflow IDs** — generating IDs from timestamps or random values means Signal-with-Start creates a new Workflow on every webhook delivery instead of routing to the existing one.
- **Losing the task token** in Pattern 3. If the service storing task tokens is unavailable when the callback arrives, the activity can never complete. Store the token durably (database, not in-process cache).
- **Forgetting to signal async completion** in Pattern 3 (`raise_complete_async()` in Python, `ErrResultPending` in Go, `doNotCompleteOnReturn()` in Java). Without this, Temporal marks the activity as completed immediately when the function returns, before the external callback arrives.

## Related

### Patterns

- [Signal with Start](/design-patterns/signal-with-start) — deeper coverage of the Signal-with-Start API for entity Workflows
- [Approval](/design-patterns/approval) — blocking wait for an external human decision using Signals
- [Polling External Services](/design-patterns/polling) — alternative to callbacks when the external system does not support webhooks
- [Delayed Start](/design-patterns/delayed-start) — defer Workflow execution to a future time without `workflow.sleep()`
- [Long-Running Activity](/design-patterns/long-running-activity) — heartbeating pattern for activities that run for extended periods

### References

- [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) — canonical reference for Pattern 3's task-token mechanism, including when to prefer it over Signals

---

# Delayed Retry

Source: https://docs.temporal.io/design-patterns/delayed-retry

> Override one failure's retry interval with nextRetryDelay on ApplicationFailure, matching the wait time the error reports.

> **ℹ️ TLDR:**
> Throw an `ApplicationFailure` with `nextRetryDelay` set inside the Activity to **delay the next retry for a fixed time.** Use this when an error carries its own timing information — such as an HTTP 429 `Retry-After` header or a known maintenance window — so Temporal waits exactly as long as needed instead of following the generic backoff schedule.

## Overview

The Delayed Retry pattern overrides the next retry interval for a specific failure by throwing an `ApplicationFailure` with a `nextRetryDelay` field set from inside the Activity.
Use it when a particular error carries information about how long to wait before retrying — such as a rate-limit response with a `Retry-After` header, or a known maintenance window with a fixed end time.

## Problem

A `RetryPolicy` applies a single backoff schedule to all failures from an Activity.
This works well for generic transient errors, but some errors carry specific information about how long the caller must wait:

- An HTTP 429 response includes a `Retry-After: 60` header telling you exactly when the quota resets.
- A downstream system returns an error message saying "maintenance until 02:00 UTC" — a precise, known delay.
- A database error includes a lock timeout duration that indicates when the resource will be available.

With a global `RetryPolicy`, you have two options, neither of which is what you need: set a short interval and retry too early (wasting quota and adding load), or set a long interval and wait longer than necessary.
What you need is to set the next retry delay *specific to this failure*, based on the information the error itself provides.

## Solution

Throw an `ApplicationFailure` with the `nextRetryDelay` field set from inside the Activity.
Temporal replaces the RetryPolicy-calculated interval for that single retry with the value you specify.
Subsequent retries (if the next attempt also fails) return to the normal RetryPolicy schedule unless you set `nextRetryDelay` again.

```mermaid
sequenceDiagram
    participant Workflow
    participant Temporal as Temporal Service
    participant API as Rate-Limited API

    Workflow->>Temporal: Schedule activity<br/>(RetryPolicy: initialInterval=1s)
    Temporal->>+API: Attempt 1
    API-->>-Temporal: HTTP 429 — Retry-After: 60s
    Note over Temporal: Activity throws<br/>ApplicationFailure(nextRetryDelay=60s)
    Note over Temporal: Override: wait 60s<br/>(ignoring RetryPolicy interval)
    Temporal->>+API: Attempt 2
    API-->>-Temporal: Success
    Temporal-->>Workflow: Result
```

The following describes each step:

1. The Activity calls the API. It receives an HTTP 429 with a `Retry-After: 60` header.
2. The Activity extracts the retry delay from the response and throws `ApplicationFailure` with `nextRetryDelay=60s`.
3. Temporal ignores the RetryPolicy's calculated interval for this retry and waits exactly 60 seconds instead.
4. The next attempt succeeds and Temporal delivers the result to the Workflow.

## Implementation

### Override the retry delay from the response

Extract the wait duration from the error or response and pass it to `ApplicationFailure`.
The RetryPolicy's `MaximumAttempts` and `ScheduleToCloseTimeout` still apply — only the interval for the next retry is overridden.

**Python**

```python
# activities.py
from datetime import timedelta
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def call_api(endpoint: str) -> str:
    response = await http_client.get(endpoint)

    if response.status_code == 429:
        retry_after = response.headers.get("Retry-After")
        if retry_after is not None:
            raise ApplicationError(
                f"Rate limited — retrying after {retry_after}s",
                type="RateLimitError",
                next_retry_delay=timedelta(seconds=int(retry_after)),
            )
        raise ApplicationError(
            "Rate limited — retrying per RetryPolicy", type="RateLimitError"
        )

    return response.text
```

**Go**

```go
// rate_limited_activity.go
func CallApi(ctx context.Context, endpoint string) (string, error) {
    response, err := httpClient.Get(endpoint)
    if err != nil {
        return "", err
    }

    if response.StatusCode == 429 {
        if retryAfter := response.Header.Get("Retry-After"); retryAfter != "" {
            seconds, _ := strconv.Atoi(retryAfter)
            return "", temporal.NewApplicationErrorWithOptions(
                fmt.Sprintf("Rate limited — retrying after %ds", seconds),
                "RateLimitError",
                temporal.ApplicationErrorOptions{NextRetryDelay: time.Duration(seconds) * time.Second},
            )
        }
        return "", temporal.NewApplicationError("Rate limited — retrying per RetryPolicy", "RateLimitError")
    }

    return response.Body, nil
}
```

**Java**

```java
// RateLimitedActivityImpl.java
import io.temporal.activity.Activity;
import io.temporal.failure.ApplicationFailure;
import java.time.Duration;

public class RateLimitedActivityImpl implements RateLimitedActivity {
    @Override
    public String callApi(String endpoint) {
        ApiResponse response = httpClient.get(endpoint);

        if (response.getStatusCode() == 429) {
            int retryAfterSeconds = response.getHeaderInt("Retry-After", 0);
            if (retryAfterSeconds > 0) {
                throw ApplicationFailure.newFailureWithCauseAndDelay(
                    "Rate limited — retrying after " + retryAfterSeconds + "s",
                    "RateLimitError",
                    null,
                    Duration.ofSeconds(retryAfterSeconds)
                );
            }
            throw ApplicationFailure.newFailure("Rate limited — retrying per RetryPolicy", "RateLimitError");
        }

        return response.getBody();
    }
}
```

**TypeScript**

```typescript
// activities.ts
import { ApplicationFailure } from '@temporalio/activity';

export async function callApi(endpoint: string): Promise<string> {
    const response = await fetch(endpoint);

    if (response.status === 429) {
        const retryAfterHeader = response.headers.get('Retry-After');
        const retryAfterSeconds = retryAfterHeader != null ? parseInt(retryAfterHeader, 10) : undefined;
        throw ApplicationFailure.create({
            message: retryAfterSeconds != null
                ? `Rate limited — retrying after ${retryAfterSeconds}s`
                : 'Rate limited — retrying per RetryPolicy',
            type: 'RateLimitError',
            // Only override the interval when the header is present; fall back to RetryPolicy otherwise
            nextRetryDelay: retryAfterSeconds != null ? `${retryAfterSeconds}s` : undefined,
        });
    }

    return response.text();
}
```

### Attempt-proportional delay

You can also set the delay dynamically based on the attempt number — for example, to implement a custom backoff that differs from exponential, or to add a known base delay on top of the standard backoff.

**Python**

```python
# activities.py
from datetime import timedelta
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def process(input: str) -> str:
    attempt = activity.info().attempt

    try:
        return await downstream_service.call(input)
    except ServiceUnavailableError as e:
        # Custom delay: 3 seconds × attempt number (3s, 6s, 9s, …)
        raise ApplicationError(
            f"Service unavailable on attempt {attempt}",
            type="ServiceUnavailable",
            next_retry_delay=timedelta(seconds=3 * attempt),
        ) from e
```

**Go**

```go
// backoff_activity.go
func Process(ctx context.Context, input string) (string, error) {
    attempt := activity.GetInfo(ctx).Attempt

    result, err := downstreamService.Call(input)
    if err != nil {
        // Custom delay: 3 seconds × attempt number (3s, 6s, 9s, …)
        return "", temporal.NewApplicationErrorWithOptions(
            fmt.Sprintf("Service unavailable on attempt %d", attempt),
            "ServiceUnavailable",
            temporal.ApplicationErrorOptions{
                Cause:          err,
                NextRetryDelay: 3 * time.Second * time.Duration(attempt),
            },
        )
    }
    return result, nil
}
```

**Java**

```java
// BackoffActivityImpl.java
import io.temporal.activity.Activity;
import io.temporal.failure.ApplicationFailure;
import java.time.Duration;

public class BackoffActivityImpl implements BackoffActivity {
    @Override
    public String process(String input) {
        int attempt = Activity.getExecutionContext().getInfo().getAttempt();

        try {
            return downstreamService.call(input);
        } catch (ServiceUnavailableException e) {
            // Custom delay: 3 seconds × attempt number (3s, 6s, 9s, …)
            throw ApplicationFailure.newFailureWithCauseAndDelay(
                "Service unavailable on attempt " + attempt,
                "ServiceUnavailable",
                e,
                Duration.ofSeconds(3L * attempt)
            );
        }
    }
}
```

**TypeScript**

```typescript
// activities.ts
import { ApplicationFailure, activityInfo } from '@temporalio/activity';

export async function process(input: string): Promise<string> {
    const { attempt } = activityInfo();

    try {
        return await downstreamService.call(input);
    } catch (e) {
        // Custom delay: 3 seconds × attempt number (3s, 6s, 9s, …)
        throw ApplicationFailure.create({
            message: `Service unavailable on attempt ${attempt}`,
            type: 'ServiceUnavailable',
            cause: e as Error,
            nextRetryDelay: `${3 * attempt}s`,
        });
    }
}
```

### Workflow configuration

The Workflow sets a normal `RetryPolicy`.
The `nextRetryDelay` set in the Activity overrides the interval only for the retry following that specific failure — subsequent attempts fall back to the RetryPolicy schedule if `nextRetryDelay` is not set again.

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

with workflow.unsafe.imports_passed_through():
    from activities import call_api

@workflow.defn
class ApiWorkflow:
    @workflow.run
    async def run(self, endpoint: str) -> str:
        return await workflow.execute_activity(
            call_api,
            endpoint,
            start_to_close_timeout=timedelta(seconds=10),
            retry_policy=RetryPolicy(
                initial_interval=timedelta(seconds=1),
                backoff_coefficient=2.0,
                maximum_attempts=10,
            ),
        )
```

**Go**

```go
// api_workflow.go
func ApiWorkflow(ctx workflow.Context, endpoint string) (string, error) {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 10 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2.0,
            MaximumAttempts:    10,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var result string
    err := workflow.ExecuteActivity(ctx, CallApi, endpoint).Get(ctx, &result)
    return result, err
}
```

**Java**

```java
// ApiWorkflowImpl.java
public class ApiWorkflowImpl implements ApiWorkflow {
    private final RateLimitedActivity activities = Workflow.newActivityStub(
        RateLimitedActivity.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(10))
            .setRetryOptions(RetryOptions.newBuilder()
                .setInitialInterval(Duration.ofSeconds(1))
                .setBackoffCoefficient(2.0)
                .setMaximumAttempts(10)
                .build())
            .build()
    );

    @Override
    public String run(String endpoint) {
        return activities.callApi(endpoint);
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import type * as activities from './activities';

const { callApi } = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '10s',
    retry: {
        initialInterval: '1s',
        backoffCoefficient: 2,
        maximumAttempts: 10,
    },
});

export async function apiWorkflow(endpoint: string): Promise<string> {
    return await callApi(endpoint);
}
```

## Best practices

- **Use the error's own delay information when available.** HTTP 429 `Retry-After`, database lock timeouts, and API-provided backoff hints are more accurate than any value you could configure statically.
- **Fall back to the RetryPolicy for unknown errors.** Only set `nextRetryDelay` for error types where you have reliable delay information. Let the RetryPolicy handle all other failures normally.
- **Still set a meaningful RetryPolicy.** `nextRetryDelay` overrides the interval for a single retry; the RetryPolicy still governs maximum attempts and the intervals for attempts where `nextRetryDelay` is not set. Also ensure `scheduleToCloseTimeout` is long enough to accommodate the maximum possible `nextRetryDelay` value — a tight budget can cause the Activity to expire before the delayed retry executes.
- **Surface the delay in the failure message.** Include the delay value and its source in the `ApplicationFailure` message (for example, `"Rate limited — retrying after 60s (Retry-After header)"`) so it appears directly in the Workflow history - Activity failure details. This makes it clear why the Activity waited an unusual amount of time without requiring separate log correlation.

## Common pitfalls

- **Assuming `nextRetryDelay` persists across all retries.** It only applies to the immediate next retry. If the following attempt also fails without setting `nextRetryDelay`, the RetryPolicy interval resumes.
- **Setting `nextRetryDelay` longer than `ScheduleToCloseTimeout`.** If the override delay exceeds the remaining `ScheduleToCloseTimeout` budget, the retry will never execute — Temporal will expire the Activity before the delay elapses.

## Related

### Patterns

- [Fixed Count of Retries](/design-patterns/fixed-count-retries): Cap total attempts to prevent unbounded retry cost.
- [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Enforce a total elapsed time budget across all attempts.
- [Fast/Slow Retries](/design-patterns/fast-slow-retries): Shift from a fast retry cadence to a slow one after initial attempts are exhausted.
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns.

### References

- [Per-error next Retry delay](/encyclopedia/retry-policies#per-error-next-retry-delay)

---

# Delayed Start Pattern

Source: https://docs.temporal.io/design-patterns/delayed-start

> Creates Workflows immediately but defers execution until a specified delay expires. Fits one-time scheduled operations and grace periods.

## Overview

The Delayed Start pattern enables Workflows to be created immediately but begin execution after a specified delay.
The Workflow execution is registered in Temporal right away, but the first Workflow Task is scheduled to run only after the delay period expires, making it suitable for scheduled operations, grace periods, and deferred processing.

## Problem

Some business processes need a Workflow that exists and is queryable immediately, but doesn't actually start executing until a specific time in the future — a one-time delay, not a recurring schedule.

Without delayed start, the alternatives are all a worse fit: trigger Workflow creation from an external scheduler, start the Workflow immediately and make `sleep()` its first operation (which runs Workflow code — and processes any regular Signal sent during the wait — before the delay you actually wanted has elapsed), build a custom queueing system for deferred execution, or reach for [Temporal Schedules](/schedule) — built for recurring executions, more machinery than a single delay needs.

## Solution

The Delayed Start uses a start delay option in WorkflowOptions to defer the first Workflow Task.
The Workflow execution is created immediately with a `firstWorkflowTaskBackoff` set to the delay duration, but no Workflow code runs until the delay expires.

```mermaid
sequenceDiagram
    participant Client
    participant Temporal
    participant Workflow

    Client->>Temporal: Start with setStartDelay(30s)
    Temporal->>Temporal: Create execution
    Note over Temporal: Execution visible<br/>but not running
    Temporal-->>Client: Workflow ID
    
    Note over Temporal: Delay period (30s)...
    
    opt During delay
        Client->>Temporal: Signal-With-Start
        Note over Temporal: Bypasses remaining delay
    end
    
    Note over Temporal: Delay expires (if not bypassed)
    Temporal->>+Workflow: Schedule first task
    Workflow->>Workflow: Execute
    Workflow-->>-Temporal: Complete
```

The following describes each step in the diagram:

1. The client starts the Workflow with a 30-second delay. Temporal creates the execution immediately.
2. The execution is visible and queryable, but no Workflow code runs during the delay.
3. If the client sends a Signal-With-Start or Update-With-Start during the delay, the remaining delay is bypassed and a Workflow Task is dispatched immediately. Regular Signals do not interrupt the delay.
4. After the delay expires, Temporal schedules the first Workflow Task and the Workflow begins execution.

The following example creates a Workflow with a 30-second start delay:

**Python**

```python
# client.py
from datetime import timedelta

handle = await client.start_workflow(
    DelayedStartWorkflow.run,
    id=WORKFLOW_ID,
    task_queue=TASK_QUEUE,
    start_delay=timedelta(seconds=30),
)
# Created now, executes in 30 seconds
```

**Go**

```go
// starter/main.go
workflowOptions := client.StartWorkflowOptions{
    ID:         WorkflowID,
    TaskQueue:  TaskQueue,
    StartDelay: 30 * time.Second,
}

we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, DelayedStartWorkflow)
// Created now, executes in 30 seconds
```

**Java**

```java
// Client.java
DelayedStartWorkflow workflow = client.newWorkflowStub(
    DelayedStartWorkflow.class,
    WorkflowOptions.newBuilder()
        .setWorkflowId(WORKFLOW_ID)
        .setTaskQueue(TASK_QUEUE)
        .setStartDelay(Duration.ofSeconds(30))
        .build());

workflow.start(); // Created now, executes in 30 seconds
```

**TypeScript**

```typescript
// client.ts
const handle = await client.workflow.start(delayedStartWorkflow, {
  workflowId: WORKFLOW_ID,
  taskQueue: TASK_QUEUE,
  startDelay: '30 seconds',
});
// Created now, executes in 30 seconds
```

The start delay option sets the `firstWorkflowTaskBackoff` on the execution.
The Workflow is created and visible in the UI immediately, but the Worker does not receive a Task until the delay expires.

## Implementation

### Basic delayed notification

The following implementation sends a notification after a one-hour delay.
The Workflow code runs only after the delay expires:

**Python**

```python
# workflows.py
from temporalio import workflow

@workflow.defn
class NotificationWorkflow:
    @workflow.run
    async def run(self, message: str) -> None:
        workflow.logger.info(f"Sending notification: {message}")

# client.py
from datetime import timedelta

handle = await client.start_workflow(
    NotificationWorkflow.run,
    "Your trial expires soon",
    task_queue=TASK_QUEUE,
    start_delay=timedelta(hours=1),
)
```

**Go**

```go
// workflow.go
func NotificationWorkflow(ctx workflow.Context, message string) error {
    logger := workflow.GetLogger(ctx)
    logger.Info("Sending notification: " + message)
    return nil
}

// starter/main.go
workflowOptions := client.StartWorkflowOptions{
    TaskQueue:  TaskQueue,
    StartDelay: 1 * time.Hour,
}

we, err := c.ExecuteWorkflow(
    context.Background(), workflowOptions, NotificationWorkflow, "Your trial expires soon",
)
```

**Java**

```java
// NotificationWorkflowImpl.java
@WorkflowInterface
public interface NotificationWorkflow {
  @WorkflowMethod
  void sendNotification(String message);
}

public class NotificationWorkflowImpl implements NotificationWorkflow {
  @Override
  public void sendNotification(String message) {
    Workflow.getLogger(NotificationWorkflowImpl.class)
        .info("Sending notification: " + message);
  }
}

// Client.java
NotificationWorkflow workflow = client.newWorkflowStub(
    NotificationWorkflow.class,
    WorkflowOptions.newBuilder()
        .setTaskQueue(TASK_QUEUE)
        .setStartDelay(Duration.ofHours(1))
        .build());

workflow.sendNotification("Your trial expires soon");
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';

export async function notificationWorkflow(message: string): Promise<void> {
  wf.log.info(`Sending notification: ${message}`);
}

// client.ts
const handle = await client.workflow.start(notificationWorkflow, {
  args: ['Your trial expires soon'],
  taskQueue: TASK_QUEUE,
  startDelay: '1 hour',
});
```

The Workflow is created immediately, but the notification logic does not execute until one hour later.

### Cancellable delayed execution

The following implementation adds Signal handlers for cancellation and a Query for status.
You can cancel the Workflow before it runs or check its status during the delay:

**Python**

```python
# workflows.py
from temporalio import workflow

@workflow.defn
class DelayedOrderWorkflow:
    def __init__(self) -> None:
        self._cancelled = False
        self._status = "SCHEDULED"

    @workflow.run
    async def run(self, order_id: str) -> None:
        if self._cancelled:
            self._status = "CANCELLED"
            return

        self._status = "PROCESSING"
        # Process order logic
        self._status = "COMPLETED"

    @workflow.signal
    async def cancel(self) -> None:
        self._cancelled = True

    @workflow.query
    def get_status(self) -> str:
        return self._status
```

**Go**

```go
// workflow.go
func DelayedOrderWorkflow(ctx workflow.Context, orderID string) error {
    logger := workflow.GetLogger(ctx)
    cancelled := false
    status := "SCHEDULED"

    // Register Signal handler for cancellation
    cancelCh := workflow.GetSignalChannel(ctx, "cancel")
    // Drain any pending signals without blocking
    for {
        var signal interface{}
        ok := cancelCh.ReceiveAsync(&signal)
        if !ok {
            break
        }
        cancelled = true
    }

    // Register Query handler for status
    err := workflow.SetQueryHandler(ctx, "getStatus", func() (string, error) {
        return status, nil
    })
    if err != nil {
        return err
    }

    if cancelled {
        logger.Info("Order cancelled before processing", "orderId", orderID)
        return nil
    }

    status = "PROCESSING"
    // Process order logic
    status = "COMPLETED"
    return nil
}
```

**Java**

```java
// DelayedOrderWorkflowImpl.java
@WorkflowInterface
public interface DelayedOrderWorkflow {
  @WorkflowMethod
  void processOrder(String orderId);

  @SignalMethod
  void cancel();

  @QueryMethod
  String getStatus();
}

public class DelayedOrderWorkflowImpl implements DelayedOrderWorkflow {
  private boolean cancelled = false;
  private String status = "SCHEDULED";

  @Override
  public void processOrder(String orderId) {
    if (cancelled) {
      status = "CANCELLED";
      return;
    }

    status = "PROCESSING";
    // Process order logic
    status = "COMPLETED";
  }

  @Override
  public void cancel() {
    cancelled = true;
  }

  @Override
  public String getStatus() {
    return status;
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';

const cancelSignal = wf.defineSignal('cancel');
const getStatusQuery = wf.defineQuery<string>('getStatus');

export async function delayedOrderWorkflow(orderId: string): Promise<void> {
  let cancelled = false;
  let status = 'SCHEDULED';

  wf.setHandler(cancelSignal, () => {
    cancelled = true;
  });

  wf.setHandler(getStatusQuery, () => status);

  if (cancelled) {
    status = 'CANCELLED';
    return;
  }

  status = 'PROCESSING';
  // Process order logic
  status = 'COMPLETED';
}
```

The `cancel` Signal handler sets a flag that the Workflow checks when it starts executing.
Note that Signal handlers and Query handlers only run after the delay expires and the first Workflow Task is dispatched.
To cancel before execution, use `Signal-With-Start` to bypass the delay, or cancel the Workflow Execution directly.

## When to use

The Delayed Start pattern is a good fit for scheduled one-time operations (send a reminder in 24 hours), grace periods before processing (cancel a subscription in 7 days), delayed notifications and alerts, deferred batch processing, and trial expiration Workflows.

It is not a good fit for recurring Schedules (use Temporal Schedules), immediate execution with internal delays (use Workflow sleep — `Workflow.sleep()` in Java, `wf.sleep()` in TypeScript, `workflow.sleep()` in Python, `workflow.Sleep()` in Go), complex scheduling logic (use Schedules with cron), or sub-second delays (minimal benefit).

## Benefits and trade-offs

The Workflow is queryable before execution starts (immediate visibility).
No Worker resources are consumed during the delay.
You can cancel the Workflow Execution before it runs.
A Signal-With-Start or Update-With-Start bypasses the remaining delay.
Regular Signals sent during the delay do not interrupt it.
The API is a single configuration option with no external schedulers needed.
The delay is managed by Temporal, ensuring deterministic behavior.

The trade-offs to consider are that you cannot dynamically adjust the delay after creation (use the Updatable Timer pattern for that).
The pattern is for one-time delays only — for recurring Schedules, use Temporal Schedules.
Very short delays (sub-second) provide minimal benefit — Temporal does not guarantee sub-second timer accuracy, and the delay is rounded up to account for scheduling latency.
The delay is time-based only, not condition-based.
Regular Signals sent during the delay are not delivered until the first Workflow Task fires, so Query and Signal handlers are not available until execution begins.

## Comparison with alternatives

| Approach | Immediate visibility | Resource usage | Cancellable | Use case |
| :--- | :--- | :--- | :--- | :--- |
| Delayed Start | Yes | None during delay | Yes | One-time future execution |
| Workflow sleep | Yes | Worker resources | Yes | Internal delays |
| Temporal Schedules | Yes | None | Yes | Recurring Schedules |
| External Scheduler | No | External system | Depends | Complex scheduling |

## Best practices

- **Use for one-time delays.** For recurring Schedules, use Temporal Schedules instead.
- **Set Workflow ID.** Always set an explicit Workflow ID for tracking and cancellation.
- **Add Query methods.** Expose status via Queries to check state during the delay.
- **Enable cancellation.** Add Signal handlers to cancel before execution.
- **Validate delay duration.** Ensure the delay is reasonable (not too short or too long).
- **Monitor backoff.** Check `firstWorkflowTaskBackoff` in history for verification.
- **Consider time zones.** Use absolute timestamps if the delay depends on a specific time.
- **Document behavior.** Clearly indicate that the Workflow does not execute immediately.

## Common pitfalls

- **Using Signals during the delay.** Regular Signals do not interrupt the Start Delay. Only Signal-With-Start or Update-With-Start bypass the delay. Signals sent to a delayed Workflow are buffered but the Workflow code has not started, so there is no handler to process them until the delay expires.
- **Querying before the Workflow starts.** Queries have no state to return during the delay because no Workflow code has executed yet. Clients may receive errors or empty results.
- **Relying on sub-second delays.** Temporal does not guarantee sub-second timer accuracy, and the delay is rounded up due to scheduling latency. Treat the configured duration as a minimum, not an exact value.
- **Forgetting that the Workflow ID is reserved.** A delayed Workflow reserves its Workflow ID immediately. Starting another Workflow with the same ID will fail depending on the ID reuse policy.

## Related

### Patterns

- **[Temporal Schedules](/schedule)**: For recurring Workflow execution.
- **[Updatable Timer](/design-patterns/updatable-timer)**: For dynamically adjustable delays within Workflows.
- **[Signal with Start](/design-patterns/signal-with-start)**: Interacting with Workflows before execution.

### Sample code

- [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloDelayedStart.java) — Delayed start with `setStartDelay()`.
- [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/start-delay) — Delayed start with `startDelay` option.
- [Python Sample](https://github.com/temporalio/samples-python/tree/main/start_delay) — Delayed start with `start_delay` parameter.
- [Go Sample](https://github.com/temporalio/samples-go/tree/main/start-delay) — Delayed start with `StartDelay` option.

---

# Distributed Transaction Patterns

Source: https://docs.temporal.io/design-patterns/distributed-transaction-patterns

> Pattern selection guide for distributed transactions, with a decision tree for choosing between Saga and Early Return.

Distributed transactions span multiple services that each own their own data, with no shared database transaction to roll back. These patterns coordinate the steps, undo completed work when a later step fails, and keep external side effects correct under retries.

## Patterns in this section

- [Saga Pattern](/design-patterns/saga-pattern): Manages a distributed transaction as a sequence of local steps, where each step defines a compensating action that undoes its effect if a later step fails.
- [Early Return](/design-patterns/early-return): Returns a result to the caller as soon as initialization succeeds, while the remaining work continues asynchronously in the background.

## Choosing a pattern

**You need to undo completed steps when a later step fails**: use the [Saga Pattern](/design-patterns/saga-pattern) and define a compensation for every step that has an external effect.

**You need to respond to the caller before the transaction finishes**: use [Early Return](/design-patterns/early-return) to acknowledge after initialization and continue processing in the background.

## Related sections

- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns) — control how each step retries before a compensation triggers
- [Workflow Messaging Patterns](/design-patterns/workflow-messaging-patterns) — the Update-with-Start mechanism behind Early Return
- [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — combine Early Return with Local Activities for the lowest first-response latency

---

# Downstream Rate Limiting

Source: https://docs.temporal.io/design-patterns/downstream-rate-limiting

> Rate-limits calls to a downstream service by routing throttled Activities to a dedicated Task Queue with a server-enforced throughput cap.

> **ℹ️ TLDR:**
> Used to **rate limit outbound requests to a downstream service**. Use this to limit the rate of requests, such as to a third-party API, or payment processor, or other external system, that concurrent Workflows would otherwise exceed.

## Overview

The Downstream Rate Limiting pattern, also known as Task Queue rate limiting, caps how many Activities execute per second against a downstream service.
You place throttled Activities on a dedicated Task Queue backed by Workers configured with `MaxTaskQueueActivitiesPerSecond`.
The Temporal matching service enforces this limit before dispatching tasks, so the downstream service receives a controlled request rate regardless of how many Worker instances or Workflow executions are running concurrently.

## Problem

Many downstream systems — LLM providers, payment processors, third-party REST APIs — enforce requests-per-second limits.
When many Temporal Workflows schedule Activities concurrently, the resulting burst can saturate those limits, causing request failures, cascading retries, and increased latency for all callers.

Without centralized throttling, each Activity implementation must manage backpressure independently, which scatters policy across the codebase and provides no enforcement at the Temporal scheduling layer.

## Solution

You assign rate-limited Activities to a dedicated Task Queue and Worker set and configure the Workers on that queue with a throughput cap.
Because the limit applies to the Task Queue, it is enforced before any Worker executes an Activity, and it holds across all Worker replicas without coordination.

The Workflow routes the throttled Activity to the dedicated queue by specifying an explicit `task_queue` override in the Activity options.

```mermaid
flowchart LR
    subgraph Workflows
        WA[Workflow A]
        WB[Workflow B]
        WC[Workflow C]
    end

    subgraph Temporal Server
        TQ["rate-limited-tq\n─────────────\ntask 1\ntask 2\ntask 3\ntask 4\n…"]
    end

    subgraph Workers
        WK1["Worker 1\nMaxTaskQueueActivitiesPerSecond\n= 5 RPS"]
        WK2["Worker 2\nMaxTaskQueueActivitiesPerSecond\n= 5 RPS"]
    end

    DS["Downstream API\n(rate limit: 5 RPS)"]

    WA -->|"schedule callApi\ntask_queue=rate-limited-tq"| TQ
    WB -->|"schedule callApi\ntask_queue=rate-limited-tq"| TQ
    WC -->|"schedule callApi\ntask_queue=rate-limited-tq"| TQ
    TQ -->|"dispatch ≤5/sec across both"| WK1
    TQ --> WK2
    WK1 -->|"≤5 req/sec combined"| DS
    WK2 --> DS
```

The following describes each step in the diagram:

1. Any number of Workflows schedule `callApi` Activities to the dedicated `rate-limited-tq` Task Queue via an explicit `task_queue` override in their Activity options.
2. The Temporal server holds tasks in `rate-limited-tq`. The queue depth grows if submission rate exceeds dispatch capacity.
3. Two Workers poll the queue. Each is configured with `MaxTaskQueueActivitiesPerSecond = 5`, so together they dispatch at most 5 Activity tasks per second — matching the downstream API's rate limit.
4. The downstream API receives a steady, controlled request rate regardless of how many Workflows are running concurrently.

## Implementation

### Worker configured with a throughput cap

**Python**

```python
# worker.py
# This is a dedicated worker for rate-limited activities.
# You will also need a separate worker registered on your workflow task queue.
from temporalio.worker import Worker
from activities import call_api

async def run_worker(client):
    worker = Worker(
        client,
        task_queue="rate-limited-tq",
        activities=[call_api],
        max_task_queue_activities_per_second=5.0,
    )
    await worker.run()
```

**Go**

```go
// main.go
// This is a dedicated worker for rate-limited activities.
// You will also need a separate worker registered on your workflow task queue.
w := worker.New(c, "rate-limited-tq", worker.Options{
    TaskQueueActivitiesPerSecond: 5.0,
})
w.RegisterActivity(CallApi)
if err := w.Run(worker.InterruptCh()); err != nil {
    log.Fatalf("worker error: %v", err)
}
```

**Java**

```java
// WorkerSetup.java
// This is a dedicated worker for rate-limited activities.
// You will also need a separate worker registered on your workflow task queue.
WorkerOptions rateLimitedOptions = WorkerOptions.newBuilder()
        .setMaxTaskQueueActivitiesPerSecond(5.0)
        .build();

Worker rateLimitedWorker = factory.newWorker("rate-limited-tq", rateLimitedOptions);
rateLimitedWorker.registerActivitiesImplementations(new RateLimitedActivitiesImpl());
factory.start();
```

### Activity Definition

**Python**

```python
# activities.py
from temporalio import activity

@activity.defn
async def call_api(input: str) -> str:
    return await downstream_api.call(input)
```

**Go**

```go
// activities.go
func CallApi(ctx context.Context, input string) (string, error) {
    return downstreamApi.Call(input)
}
```

**Java**

```java
// RateLimitedActivities.java
@ActivityInterface
public interface RateLimitedActivities {
    @ActivityMethod
    String callApi(String input);
}

public class RateLimitedActivitiesImpl implements RateLimitedActivities {
    @Override
    public String callApi(String input) {
        return downstreamApi.call(input);
    }
}
```

### Workflow routing to the rate-limited queue

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from activities import call_api

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        return await workflow.execute_activity(
            call_api,
            input,
            task_queue="rate-limited-tq",
            start_to_close_timeout=timedelta(seconds=30),
        )
```

**Go**

```go
// workflow.go
func MyWorkflow(ctx workflow.Context, input string) (string, error) {
    ao := workflow.ActivityOptions{
        TaskQueue:           "rate-limited-tq",
        StartToCloseTimeout: 30 * time.Second,
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var result string
    err := workflow.ExecuteActivity(ctx, CallApi, input).Get(ctx, &result)
    return result, err
}
```

**Java**

```java
// MyWorkflowImpl.java
public class MyWorkflowImpl implements MyWorkflow {
    private final RateLimitedActivities rateLimitedActivities =
        Workflow.newActivityStub(RateLimitedActivities.class,
            ActivityOptions.newBuilder()
                .setTaskQueue("rate-limited-tq")
                .setStartToCloseTimeout(Duration.ofSeconds(30))
                .build()
        );

    @Override
    public String run(String input) {
        return rateLimitedActivities.callApi(input);
    }
}
```

## When to use

This pattern is a good fit when your Workflow calls a downstream service with explicit requests-per-second limits, when you need throughput enforcement that holds across many concurrent Workflow instances without per-Activity logic, and when only a subset of Activity types require throttling and others should run without restriction.

It is not a good fit when you need concurrency limits rather than throughput limits, when the downstream system has no rate limit and throughput is bounded only by Workflow logic, or when all Activities require the same limit and a single shared queue suffices.

## Benefits and trade-offs

Centralizing rate limiting at the Task Queue ensures enforcement even when any number of Workflow instances run in parallel.
Because the Temporal server controls dispatch, the limit holds regardless of how many Worker replicas are running — provided you account for Worker count when setting the per-worker cap.

Dedicated Task Queues require operating additional Workers.
If the throughput cap is set too low relative to demand, the queue depth grows and scheduling latency increases.
You must size the Worker pool so that slot availability does not become the bottleneck before the rate limit is reached.

## Comparison with alternatives

| Approach | Enforcement point | Works across Workers | Runtime adjustable | Complexity |
| :--- | :--- | :--- | :--- | :--- |
| `MaxTaskQueueActivitiesPerSecond` | Temporal matching service (server-side) | Yes | No (requires redeploy) | Low |
| `MaxWorkerActivitiesPerSecond` | Worker SDK poller (worker-side) | No — per-worker only | No (requires redeploy) | Low |
| Concurrency slots (`MaxConcurrentActivityExecutionSize`, `MaxConcurrentWorkflowTaskExecutionSize`, `MaxConcurrentLocalActivityExecutionSize`) | Worker executor | No — per-worker only | No (requires redeploy) | Low |
| Sleep-based throttle in Workflow | Workflow scheduler | No | Via signal | Low |
| Client-side token bucket in Activity | Activity execution | Per-worker only | No | Medium |
| API gateway rate limiting | Network layer | Yes | Yes | High |

Three distinct layers of worker-side control exist alongside the server-side queue limit. `MaxWorkerActivitiesPerSecond` instructs the SDK to self-throttle its polling — the Worker will not request a new Activity task if doing so would push it over this rate. Because the limit is per-process, multiple Workers on the same queue each apply it independently, so the effective queue throughput is the per-worker cap multiplied by Worker count. By contrast, `MaxTaskQueueActivitiesPerSecond` is a server-side instruction: the Temporal matching service slows dispatch for the entire queue regardless of how many Workers are polling, making it the correct tool for protecting a shared downstream service.

The concurrency slots (`MaxConcurrentActivityExecutionSize`, `MaxConcurrentWorkflowTaskExecutionSize`, `MaxConcurrentLocalActivityExecutionSize`) are not throughput limits but define the number of execution slots available on a Worker. A Worker will not accept more tasks than it has open slots, so a low slot count acts as an indirect throughput ceiling. 

## Best practices

- **Use a separate Task Queue for each rate limit.** `MaxTaskQueueActivitiesPerSecond` applies to every Activity on the queue. Mixing rate-limited and unrestricted Activities on the same queue will throttle the unrestricted ones too.
- **[Run at least two Worker processes per queue](/best-practices/worker#run-at-least-two-workers-per-task-queue) for availability.** A single Worker process is a single point of failure. Because `MaxTaskQueueActivitiesPerSecond` is a server-side per-queue limit rather than a per-worker one, set the same value on every Worker that polls the queue. Set each Worker to the target RPS — for example, 5 on each of two Workers yields a combined queue limit of 5, not 10. If Workers report different values, the server applies the value from the last Worker that polled.
- **Monitor queue depth and schedule latency.** Track the `temporal_activity_schedule_to_start_latency` metric on the rate-limited queue; sustained growth signals that demand consistently exceeds the configured cap. You can also query the Task Queue's `ApproximateBacklogCount` via the `DescribeTaskQueue` API — a steadily growing backlog count is a direct indicator that the configured RPS cap is too low for the current submission rate.

## Common pitfalls

- **Forgetting to override the task queue in Activity options.** If the Workflow does not explicitly specify `task_queue` in the Activity options, the Activity runs on the Workflow's default queue and bypasses the rate-limited Worker entirely.
- **Setting conflicting MaxTaskQueueActivitiesPerSecond limits in workers.** This setting is set in Workers and sent to the Task Queue when a Worker polls. If you have multiple Workers with conflicting settings, the Workers will overwrite each other as they poll.
- **Confusing throughput limits with concurrency limits.** `MaxTaskQueueActivitiesPerSecond` controls starts per second; `MaxConcurrentActivityExecutionSize` controls simultaneous executions. Long-running Activities that hold slots for minutes may exhaust concurrency before the RPS cap applies.
- **Setting the cap far below actual demand.** A cap much lower than actual submission rate causes the queue to grow unboundedly. Monitor queue depth and raise the cap or add more Workers when throughput requirements grow.
- **Expecting a perfectly even per-second rate.** The limit is enforced across the queue's partitions, default four. The server maintains the configured rate as an average over time but can dispatch a short burst above it, up to roughly the rate divided across partitions. If the downstream service rejects any momentary overshoot, set the cap below the hard limit to leave headroom, or reduce the partition count for the queue.
- **Eager Activity execution bypassing the rate-limited queue.** [Eager Activity Start](/develop/worker-performance#eager-activity-start) lets the server hand an Activity straight back to the Worker that just completed the scheduling Workflow Task, skipping the Task Queue and its rate limit. It applies only when the Activity is scheduled on its Workflow's own Task Queue, and every SDK disables it on a Worker that sets `MaxTaskQueueActivitiesPerSecond`. A dedicated rate-limited queue is never bypassed this way. On a queue that also hosts the Workflows, set the limit on every Worker that polls it.

## Related

### Patterns

- **[Priority](/design-patterns/priority-task-queues)**: Order Task dispatches by urgency within a shared Task Queue.
- **[Fairness](/design-patterns/fairness)**: Distribute Task dispatches among tenants that share a Task Queue.
- **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity.

### Guides

- [Rate-limit downstream APIs with separate Task Queues](/guides/rate-limit-downstream-apis): A Python walkthrough covering multiple rate-limited APIs (SendGrid, Stripe, OpenAI), 429 handling with `Retry-After`, and backlog draining strategies.

### References

- **Python** — [`max_task_queue_activities_per_second`](https://python.temporal.io/temporalio.worker.WorkerConfig.html#max_task_queue_activities_per_second) on [`Worker`](https://python.temporal.io/temporalio.worker.Worker.html)
- **Go** — [`TaskQueueActivitiesPerSecond`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) in [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/worker#Options)
- **Java** — [`setMaxTaskQueueActivitiesPerSecond`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.Builder.html) on `WorkerOptions.Builder`
- **Temporal Community** — [Rate limit configuration and best practices](https://community.temporal.io/t/rate-limit-configuration-and-best-practices/5498)

---

# Eager Workflow Start

Source: https://docs.temporal.io/design-patterns/eager-workflow-start

> Eager Workflow Start sends the first Workflow Task directly to a co-located Worker, skipping the Matching Service to cut startup latency.

> **ℹ️ TLDR:**
> **Bypass the Temporal Matching Service by dispatching the first Workflow Task directly to a co-located Worker.** The Worker and the client that starts the Workflow must share the same process and server connection. Eager Workflow Start eliminates the Matching Service round-trip, saving approximately 30–50 ms per Workflow start. When combined with Local Activities, this pattern achieves ~265 ms total-workflow latency (vs ~850 ms baseline). Supported by the Go, Java, Python, TypeScript, and .NET SDKs.

## Overview

When you call `ExecuteWorkflow`, the Temporal server normally stores the new Workflow execution, then routes the first Workflow Task through its Matching Service to an available Worker. **Eager Workflow Start** short-circuits this routing: the server returns the first Workflow Task inline in the `StartWorkflowExecution` response, and the co-located Worker processes it immediately—without a separate polling round-trip.

```mermaid
sequenceDiagram
    participant CW as Client + Worker (same process)
    participant S as Temporal Server

    rect rgb(230, 235, 250)
        Note over CW,S: Normal Workflow Start
        CW->>S: StartWorkflowExecution
        S->>S: Matching Service routes<br/>task to available Worker
        S-->>CW: WorkflowTask dispatched via polling
        CW->>CW: Execute WorkflowTask
        CW->>S: Complete WorkflowTask
    end

    rect rgb(220, 245, 225)
        Note over CW,S: Eager Workflow Start (co-located Worker)
        CW->>S: StartWorkflowExecution (EnableEagerStart=true)
        S-->>CW: WorkflowTask returned inline<br/>— no Matching step
        CW->>CW: Execute WorkflowTask immediately
        CW->>S: Complete WorkflowTask
    end
```

**Numbered walkthrough:**

1. In a normal start, the server queues the Workflow execution and the Matching Service waits for an available Worker slot. The Worker polls, picks up the task, runs it, and reports back—adding an extra server round-trip.
2. With Eager Workflow Start enabled, the server detects that the requesting client has a co-located Worker with an available slot. Instead of queuing the task, the server attaches the first Workflow Task to the `StartWorkflowExecution` response.
3. The Worker processes the Workflow Task immediately upon receiving the response. No separate poll is required.
4. If the server cannot fulfill the eager request (for example, no local slot is available), it falls back silently to normal dispatch. Your code does not need to handle this case explicitly.

## Problem

Even with Local Activities eliminating per-Activity server round-trips, the Workflow's first Workflow Task still requires a scheduling round-trip through the Temporal Matching Service. This adds latency that is unavoidable in a distributed deployment where Workers are separate from the caller.

For applications where the starter and Worker share the same deployment unit—such as a request-handling service that also runs Workers—this Matching overhead can be eliminated.

## Solution

Start a Worker in the same process as the workflow starter, using the same client connection. Set `EnableEagerStart: true` (Go), `setDisableEagerExecution(false)` (Java), `request_eager_start=True` (Python), `requestEagerStart: true` (TypeScript, on a `NativeConnection` shared between Worker and Client), or `RequestEagerStart` (.NET) on the workflow start options. The SDK signals to the server that a local Worker is available, and the server returns the first Workflow Task inline.

Eager Workflow Start is enabled by default in Temporal Cloud and in self-hosted Temporal Server 1.29.0 and later — no additional server configuration or access request is needed. On self-hosted Temporal Server, an operator can disable it with the dynamic config flag `system.enableEagerWorkflowStart` set to `false`; if you don't observe the expected latency improvement, confirm that flag hasn't been turned off. See [Eager Workflow Start](/develop/worker-performance#eager-workflow-start) for the canonical server-side reference.

**Python**

```python
# starter.py — starts the Worker in the same process, then executes the Workflow eagerly
import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
from workflows import TransactionWorkflow
from activities import validate_transaction, settle_transaction
from shared import TASK_QUEUE, TransactionRequest

async def main():
    client = await Client.connect("localhost:7233")

    # The Worker must share this process and client for eager dispatch to work.
    async with Worker(
        client,
        task_queue=TASK_QUEUE,
        workflows=[TransactionWorkflow],
        activities=[validate_transaction, settle_transaction],
    ):
        result = await client.execute_workflow(
            TransactionWorkflow.run,
            TransactionRequest(amount=100.00, currency="USD"),
            id="eager-workflow-start-demo",
            task_queue=TASK_QUEUE,
            request_eager_start=True,  # Dispatch first WorkflowTask inline
        )
        print(f"Transaction complete: ID={result.id} Status={result.status}")

if __name__ == "__main__":
    asyncio.run(main())
```

**Go**

```go
// starter.go — starts the Worker in the same process, then executes the Workflow eagerly
func main() {
    c, err := client.Dial(client.Options{})
    if err != nil {
        log.Fatalln("Unable to create Temporal client:", err)
    }
    defer c.Close()

    // Start the Worker non-blocking — it must share this process and client.
    w := worker.New(c, TaskQueue, worker.Options{})
    w.RegisterWorkflow(TransactionWorkflow)
    w.RegisterActivity(ValidateTransaction)
    w.RegisterActivity(SettleTransaction)
    if err := w.Start(); err != nil {
        log.Fatalln("Unable to start worker:", err)
    }
    defer w.Stop()

    run, err := c.ExecuteWorkflow(context.Background(), client.StartWorkflowOptions{
        ID:               "eager-workflow-start-demo",
        TaskQueue:        TaskQueue,
        EnableEagerStart: true, // Dispatch first WorkflowTask inline
    }, TransactionWorkflow, TransactionRequest{Amount: 100.00, Currency: "USD"})
    if err != nil {
        log.Fatalln("Failed to start workflow:", err)
    }

    var result Transaction
    if err := run.Get(context.Background(), &result); err != nil {
        log.Fatalln("Workflow failed:", err)
    }
    fmt.Printf("Transaction complete: ID=%s Status=%s\n", result.ID, result.Status)
}
```

**Java**

```java
// Starter.java — starts the Worker in the same process, then executes the Workflow eagerly
public class Starter {
    public static void main(String[] args) {
        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service);

        // The Worker must share this process and client for eager dispatch.
        WorkerFactory factory = WorkerFactory.newInstance(client);
        io.temporal.worker.Worker worker = factory.newWorker(Shared.TASK_QUEUE);
        worker.registerWorkflowImplementationTypes(TransactionWorkflow.Impl.class);
        worker.registerActivitiesImplementations(new Activities.Impl());
        factory.start();

        TransactionWorkflow workflow = client.newWorkflowStub(
            TransactionWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue(Shared.TASK_QUEUE)
                .setWorkflowId("eager-workflow-start-demo")
                .setDisableEagerExecution(false) // false = enable eager dispatch
                .build()
        );

        Shared.Transaction result = workflow.processTransaction(
            new Shared.TransactionRequest(100.00, "USD"));
        System.out.printf("Transaction complete: ID=%s Status=%s%n",
            result.id(), result.status());

        factory.shutdown();
    }
}
```

**TypeScript**

```typescript
// starter.ts — starts the Worker in the same process, then executes the Workflow eagerly
import { NativeConnection, Worker } from '@temporalio/worker';
import { Client } from '@temporalio/client';
import { transactionWorkflow } from './workflows';
import { TASK_QUEUE, TransactionRequest } from './shared';

async function run() {
  // The Client and the Worker must share this NativeConnection for eager dispatch to work.
  const connection = await NativeConnection.connect({ address: 'localhost:7233' });

  const worker = await Worker.create({
    connection,
    taskQueue: TASK_QUEUE,
    workflowsPath: require.resolve('./workflows'),
    activities: { validateTransaction, settleTransaction },
  });

  const client = new Client({ connection });

  await worker.runUntil(async () => {
    const handle = await client.workflow.start(transactionWorkflow, {
      args: [{ amount: 100.0, currency: 'USD' } satisfies TransactionRequest],
      workflowId: 'eager-workflow-start-demo',
      taskQueue: TASK_QUEUE,
      requestEagerStart: true, // Dispatch first WorkflowTask inline
    });

    const result = await handle.result();
    console.log(`Transaction complete: ID=${result.id} Status=${result.status}`);
  });
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

> **ℹ️ .NET SDK:**
> The .NET SDK also supports Eager Workflow Start: set `RequestEagerStart = true` on `WorkflowOptions` when starting the Workflow, with the Worker and Client sharing the same connection.

## When to use

**Good fit:**

- The workflow starter and Worker run in the same deployment unit (for example, a single service that both handles API requests and runs Workers)
- You need the absolute minimum total-workflow latency and are already using Local Activities
- Any of the Go, Java, Python, TypeScript, or .NET SDKs

**Poor fit:**

- Workers are deployed independently from starters (the eager request falls back to normal dispatch, which is harmless but provides no benefit)
- First-response latency matters more than total latency—combine with [Early Return](/design-patterns/early-return) or [Early Return + Local Activities](/design-patterns/early-return-local-activities) for that use case

## Benefits and trade-offs

| | Normal Start | Eager Workflow Start |
|---|---|---|
| Matching Service round-trip | Yes (~30–50 ms) | No (eliminated) |
| Worker co-location required | No | Yes (same process + client) |
| Fallback behavior | N/A | Graceful fallback to normal dispatch |
| SDK support | All | Go, Java, Python, TypeScript, .NET |
| Configuration required | None | `EnableEagerStart`/`request_eager_start`/`setDisableEagerExecution(false)`/`requestEagerStart`/`RequestEagerStart` |
| Self-hosted server flag | N/A | On by default (Server 1.29.0+); disable via `system.enableEagerWorkflowStart=false` |

## Best practices

- **Combine with Local Activities.** Eager Workflow Start eliminates the Matching overhead on the first Workflow Task; Local Activities eliminate server round-trips within each Workflow Task. Together they provide the greatest total latency reduction.
- **Use a non-blocking Worker start.** Start the Worker before executing the Workflow so it has an available slot. In Go, use `w.Start()` and defer `w.Stop()`. In Python, use `async with Worker(...)`. In Java, call `factory.start()` before creating the workflow stub.
- **Do not rely on eager dispatch always firing.** The server falls back to normal dispatch if no local slot is available (for example, the Worker is at capacity). Design the Workflow to work correctly in both cases.
- **Share the same client and connection.** The Worker and the workflow starter must use the same `WorkflowClient` instance (Java), `client.Client` (Go), `Client` (Python), or `NativeConnection` (TypeScript). A Worker using a different connection cannot receive eager tasks from another client.
- **Be mindful of resource sharing in co-located deployments.** When a Worker runs in the same process as a request handler, they share CPU, memory, and failure domains. A spike in activity execution can slow request handling, and vice versa. Monitor Worker CPU, Workflow Task execution latency, and task queue depth to ensure Worker load does not affect client-facing latency.

## Common pitfalls

- **Starting the Worker after `ExecuteWorkflow`.** If the Worker is not registered and running before the eager start call, no local slot exists and the request falls back to normal dispatch.
- **Expecting eager dispatch in distributed deployments.** If the process that calls `ExecuteWorkflow` is not the same process running the Worker, eager dispatch will never succeed. The call still works, but it provides no latency benefit.
- **Assuming the self-hosted server flag is off.** Eager Workflow Start ships enabled by default (Server 1.29.0+). If you don't observe the expected latency improvement, check whether an operator explicitly disabled `system.enableEagerWorkflowStart`, rather than assuming it needs to be turned on.
- **Using a Connection instead of a NativeConnection in TypeScript.** The high-level `Client` and the `Worker` must share a `NativeConnection` object, not just the same server address. A `Worker` created from a separate connection cannot receive eager tasks.

## Related

### Patterns

- [Local Activities](/design-patterns/local-activities) — eliminates per-Activity server round-trips; pairs naturally with Eager Workflow Start
- [Early Return + Local Activities](/design-patterns/early-return-local-activities) — minimum first-response latency via Update-with-Start plus Local Activities
- [Early Return](/design-patterns/early-return) — returns early to the client via Update-with-Start

### References

- [Eager Workflow Start](/develop/worker-performance#eager-workflow-start) — canonical server-side reference, including Eager Activity Start

---

# Early Return (Update with Start)

Source: https://docs.temporal.io/design-patterns/early-return

> Synchronous initialization with asynchronous completion. Returns results immediately while processing continues in the background.

## Overview

The Early Return pattern returns initialization results to the caller immediately while continuing asynchronous processing in the background.

## Problem

Clients need immediate feedback on whether an operation can proceed, but the full operation takes significant time to complete.
Blocking the client for the entire operation duration creates a poor user experience and ties up resources.

## Solution

You use Update-with-Start to split operations into two phases: a fast synchronous initialization phase that validates and returns results immediately, and a slower asynchronous completion phase that runs in the background.
The Workflow uses local Activities for quick initialization, Signals completion via Update handlers, then either completes or cancels the operation based on initialization success.

```mermaid
sequenceDiagram
    participant Client
    participant Workflow
    participant Activity

    Client->>+Workflow: Update-with-Start
    activate Workflow
    Workflow->>+Activity: Phase 1: Init (fast)
    Activity-->>-Workflow: Result
    Workflow-->>Client: Init Result (early return)
    deactivate Workflow
    
    Note over Workflow: Workflow continues executing
    Workflow->>+Activity: Phase 2: Complete (slow)
    Activity-->>-Workflow: Done
    deactivate Workflow
```

The following describes each step in the diagram:

1. The client sends an Update-with-Start request to the Workflow.
2. The Workflow executes a fast initialization Activity (Phase 1) and returns the result to the client immediately.
3. The client receives the initialization result while the Workflow continues executing.
4. The Workflow executes the slower completion Activity (Phase 2) in the background.

## Implementation

The following examples show how each SDK implements this pattern.
The Workflow registers an Update handler that blocks until initialization completes, then returns the result to the caller.
The client receives the initialization result in a single round trip while the Workflow continues processing.

**Python**

```python
# workflow.py
from dataclasses import dataclass
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import init_transaction, complete_transaction, cancel_transaction

@dataclass
class TransactionRequest:
    amount: float
    currency: str

@dataclass
class Transaction:
    id: str
    status: str

@workflow.defn
class TransactionWorkflow:
    def __init__(self) -> None:
        self.tx: Transaction | None = None
        self.init_done = False
        self.init_err: Exception | None = None

    @workflow.run
    async def run(self, tx_request: TransactionRequest) -> Transaction | None:
        # Phase 1: Fast synchronous initialization (local activity)
        try:
            self.tx = await workflow.execute_local_activity(
                init_transaction,
                tx_request,
                schedule_to_close_timeout=timedelta(seconds=5),
            )
        except Exception as e:
            self.init_err = e
        finally:
            self.init_done = True  # Signal update handler

        # Phase 2: Slow asynchronous completion
        if self.init_err is not None:
            await workflow.execute_activity(
                cancel_transaction,
                self.tx,
                start_to_close_timeout=timedelta(seconds=30),
            )
            return None

        await workflow.execute_activity(
            complete_transaction,
            self.tx,
            start_to_close_timeout=timedelta(seconds=30),
        )
        return self.tx

    @workflow.update
    async def return_init_result(self) -> Transaction:
        await workflow.wait_condition(lambda: self.init_done)
        if self.init_err is not None:
            raise self.init_err
        return self.tx

# client.py
from temporalio.client import (
    Client,
    WithStartWorkflowOperation,
    WorkflowUpdateStage,
)
from temporalio.common import WorkflowIDConflictPolicy

client = await Client.connect("localhost:7233")

start_op = WithStartWorkflowOperation(
    TransactionWorkflow.run,
    tx_request,
    id="transaction-123",
    task_queue="transactions",
    id_conflict_policy=WorkflowIDConflictPolicy.FAIL,
)

update_handle = await client.start_update_with_start_workflow(
    TransactionWorkflow.return_init_result,
    wait_for_stage=WorkflowUpdateStage.COMPLETED,
    start_workflow_operation=start_op,
)

# Get initialization result immediately
tx = await update_handle.result()

# Use transaction ID immediately while workflow continues
print(f"Transaction initialized: {tx.id}")
```

**Go**

```go
// workflow.go
func Workflow(ctx workflow.Context, txRequest TransactionRequest) (*Transaction, error) {
    var tx *Transaction
    var initDone bool
    var initErr error

    // Register update handler that waits for initialization
    workflow.SetUpdateHandler(ctx, UpdateName,
        func(ctx workflow.Context) (*Transaction, error) {
            workflow.Await(ctx, func() bool { return initDone })
            return tx, initErr
        },
    )

    // Phase 1: Fast synchronous initialization (local activity)
    localOpts := workflow.WithLocalActivityOptions(ctx, workflow.LocalActivityOptions{
        ScheduleToCloseTimeout: 5 * time.Second,
    })
    initErr = workflow.ExecuteLocalActivity(localOpts, txRequest.Init).Get(ctx, &tx)
    initDone = true // Signal update handler

    // Phase 2: Slow asynchronous completion
    activityCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
    })

    if initErr != nil {
        // Cancel on initialization failure
        return nil, workflow.ExecuteActivity(activityCtx, CancelTransaction, tx).Get(ctx, nil)
    }

    // Complete on initialization success
    return tx, workflow.ExecuteActivity(activityCtx, CompleteTransaction, tx).Get(ctx, nil)
}

// client.go
startOp := client.NewWithStartWorkflowOperation(
    client.StartWorkflowOptions{
        ID:                       "transaction-123",
        TaskQueue:                "transactions",
        WorkflowIDConflictPolicy: enumspb.WORKFLOW_ID_CONFLICT_POLICY_FAIL,
    },
    Workflow,
    txRequest,
)

updateHandle, err := client.UpdateWithStartWorkflow(ctx,
    client.UpdateWithStartWorkflowOptions{
        StartWorkflowOperation: startOp,
        UpdateOptions: client.UpdateWorkflowOptions{
            UpdateName:   UpdateName,
            WaitForStage: client.WorkflowUpdateStageCompleted,
        },
    },
)

// Get initialization result immediately
var tx Transaction
err = updateHandle.Get(ctx, &tx)
if err != nil {
    return err
}

// Use transaction ID immediately while workflow continues
fmt.Printf("Transaction initialized: %s\n", tx.ID)
```

**Java**

```java
// TransactionWorkflowImpl.java
public class TransactionWorkflowImpl implements TransactionWorkflow {
    private boolean initDone = false;
    private Transaction tx;
    private Exception initError = null;

    @Override
    public TxResult processTransaction(TransactionRequest txRequest) {
        this.tx = activities.mintTransactionId(txRequest);

        // Phase 1: Fast synchronous initialization
        try {
            this.tx = activities.initTransaction(this.tx);
        } catch (Exception e) {
            initError = e;
        } finally {
            initDone = true; // Signal update handler
        }

        // Phase 2: Slow asynchronous completion
        if (initError != null) {
            activities.cancelTransaction(this.tx);
            return new TxResult("", "Transaction cancelled.");
        } else {
            activities.completeTransaction(this.tx);
            return new TxResult(this.tx.getId(), "Transaction completed successfully.");
        }
    }

    @Override
    public TxResult returnInitResult() {
        Workflow.await(() -> initDone); // Wait for initialization
        if (initError != null) {
            throw Workflow.wrap(initError);
        }
        return new TxResult(tx.getId(), "Initialization successful");
    }
}

// Client.java
TransactionWorkflow workflow = client.newWorkflowStub(
    TransactionWorkflow.class,
    WorkflowOptions.newBuilder()
        .setWorkflowId("transaction-123")
        .setTaskQueue("transactions")
        .setWorkflowIdConflictPolicy(
            WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_FAIL)
        .build());

WorkflowUpdateHandle<TxResult> updateHandle =
    WorkflowClient.startUpdateWithStart(
        workflow::returnInitResult,
        UpdateOptions.<TxResult>newBuilder().build(),
        new WithStartWorkflowOperation<>(workflow::processTransaction, txRequest));

// Get initialization result immediately
TxResult result = updateHandle.getResultAsync().get();

// Use transaction ID immediately while workflow continues
System.out.println("Transaction initialized: " + result.getId());
```

**TypeScript**

```typescript
// workflow.ts
import { defineUpdate, setHandler, condition } from '@temporalio/workflow';
import * as activities from './activities';

const { initTransaction, completeTransaction, cancelTransaction } =
  proxyLocalActivities<typeof activities>({
    scheduleToCloseTimeout: '5s',
  });

export const returnInitResultUpdate = defineUpdate<Transaction>('returnInitResult');

export async function transactionWorkflow(txRequest: TransactionRequest): Promise<Transaction> {
  let tx: Transaction | undefined;
  let initDone = false;
  let initError: Error | undefined;

  // Register update handler that waits for initialization
  setHandler(returnInitResultUpdate, async () => {
    await condition(() => initDone);
    if (initError) {
      throw initError;
    }
    return tx!;
  });

  // Phase 1: Fast synchronous initialization (local activity)
  try {
    tx = await initTransaction(txRequest);
  } catch (err) {
    initError = err as Error;
  } finally {
    initDone = true; // Signal update handler
  }

  // Phase 2: Slow asynchronous completion
  if (initError) {
    await cancelTransaction(tx!);
    throw initError;
  }

  await completeTransaction(tx);
  return tx;
}

// client.ts
const startWorkflowOperation = new WithStartWorkflowOperation(
  transactionWorkflow,
  {
    workflowId: 'transaction-123',
    args: [txRequest],
    taskQueue: 'transactions',
    workflowIdConflictPolicy: 'FAIL',
  },
);

const tx = await client.workflow.executeUpdateWithStart(
  returnInitResultUpdate,
  { startWorkflowOperation },
);

const wfHandle = await startWorkflowOperation.workflowHandle();

// Use transaction ID immediately while workflow continues
console.log(`Transaction initialized: ${tx.id}`);

// Optionally wait for the workflow to complete
const finalResult = await wfHandle.result();
```

The key points across all SDKs are:

- **Update-with-Start** is a single API call that starts the Workflow and returns the initialization result.
- The Workflow uses an Update handler with a condition/await to block until initialization completes.
- The client receives the result immediately while the Workflow continues executing in the background.
- A `WorkflowIdConflictPolicy` must be specified. For early return, use `FAIL` to assert a new Workflow is created.
- Update-with-Start is **not atomic**. If the Update cannot be delivered (for example, no Worker is available), the Workflow Execution will still start. The SDKs will retry the Update request, but there is no guarantee the Update will succeed.

## When to use

The Early Return pattern is a good fit when clients need immediate feedback but operations take time to complete, validation or initialization can be done quickly (under 5 seconds), the operation can be safely cancelled if initialization fails, and the initialization result determines whether to proceed or abort.
Common use cases include e-commerce payment processing (immediate authorization while settlement runs in the background), user onboarding and KYC verification (quick user ID return while background checks continue), resource provisioning (fast validation results while infrastructure is set up), document processing (immediate receipt confirmation while OCR and content analysis continue), and order processing (fast inventory check while fulfillment runs in the background).

It is not a good fit for fully automated processes that require no intermediate feedback, operations that cannot be split into fast and slow phases, or fire-and-forget operations where no immediate response is needed (use Signals).

## Benefits and trade-offs

The Early Return pattern provides immediate client feedback via Update-with-Start in a single round trip.
Clients do not wait for full operation completion.
Local Activities avoid extra server roundtrips during initialization, and there is a clear separation between validation and execution phases.
Automatic cancellation handling runs on initialization failure.

The trade-offs to consider are that you must tune timeouts carefully for local Activities.
There is a concurrent Update limit (10 per Workflow) that can bottleneck high-throughput scenarios requiring multiple simultaneous Updates.
Clients must handle asynchronous completion separately, and initialization must complete within a single Workflow Task.
The pattern is limited to operations that you can split into fast and slow phases.

## Comparison with alternatives

| Approach | Immediate response | Consistency | Complexity | Use case |
| :--- | :--- | :--- | :--- | :--- |
| Early Return (Update-with-Start) | Yes (typed) | Strong | Medium | Synchronous init + async completion |
| Signal + Query polling | Yes (eventual) | Eventual | High | Fire-and-forget with status checks |
| Child Workflow split | Yes (Workflow ID) | Strong | High | Separate init and completion Workflows |
| Blocking until completion | Yes (final) | Strong | Low | Short operations only |

## Best practices

- **Set WorkflowIdConflictPolicy to FAIL.** For early return, use `FAIL` to assert a new Workflow is created per request. Use `USE_EXISTING` only for lazy initialization patterns.
- **Use Workflow.await in the Update handler.** Keep the Update handler lightweight — block on a condition flag (`workflow.Await` in Go, `Workflow.await` in Java, `condition` in TypeScript, `workflow.wait_condition` in Python) and let the main Workflow method do the real work.
- **Use local Activities for initialization.** Local Activities avoid extra server roundtrips, keeping the synchronous phase fast (under 5 seconds).
- **Handle Update-with-Start non-atomicity.** Update-with-Start is not atomic. The Workflow may start even if the Update fails. Ensure Workers are running and handle the case where the Update is not delivered.
- **Set a timeout on the Update result.** Use a timeout when waiting for the Update result to avoid blocking the client indefinitely if the Worker is unavailable.
- **Be aware of the concurrent Update limit.** The default `maxInFlightUpdates` is 10 per Workflow. If you expect high concurrency, design accordingly or use separate Workflows.
- **Provide a unique Update ID.** Use a unique Update ID for idempotency so retried requests attach to the same Update rather than creating duplicates.
- **Avoid Workflow timeouts.** Do not set Workflow Execution timeouts when using early return, as the background phase may take longer than expected.

## Common pitfalls

- **Assuming Update-with-Start is atomic.** Unlike Signal-with-Start, Update-with-Start is not atomic. The Workflow may start even if the Update fails (for example, if no Worker is available). Handle this by checking Workflow state after the call.
- **Missing WorkflowIdConflictPolicy.** Update-with-Start requires a `WorkflowIdConflictPolicy`. Omitting it causes an error. Use `FAIL` for early return (one Workflow per request) or `USE_EXISTING` for lazy initialization.
- **Blocking too long in the Update handler.** The Update handler should return quickly. Perform long-running work in the main Workflow method and use `Workflow.await` in the Update handler to wait for a result.
- **Swallowing the ContinueAsNew exception.** In TypeScript, `continueAsNew` throws a special exception. Catching it in a try-catch without re-throwing (or returning in a `finally` block) silently prevents Continue-As-New.

## Related

### Patterns

- **[Saga Pattern](/design-patterns/saga-pattern)**: You can combine this with the Saga pattern to add compensation for failed completions.
- **[Signal with Start](/design-patterns/signal-with-start)**: For fire-and-forget operations that do not need an immediate response.
- **[Request-Response via Updates](/design-patterns/request-response-via-updates)**: For synchronous state modifications on already-running Workflows.

### Sample code

- [Python Sample](https://github.com/temporalio/samples-python/tree/main/early_return) — Early return with Update-with-Start.
- [Go Sample](https://github.com/temporalio/samples-go/tree/main/early-return) — Early return with Update-with-Start.
- [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/early-return) — Early return with local Activities.
- [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/earlyreturn) — Early return with Update handler.

---

# Early Return + Local Activities

Source: https://docs.temporal.io/design-patterns/early-return-local-activities

> Extends Early Return by running Phase 1 as Local Activities, so the client's first response comes from in-process work, not a server call.

> **ℹ️ TLDR:**
> **Combine Update-with-Start with Local Activities in the synchronous Phase 1 to reduce first-response latency from ~265 ms to ~160 ms.** Phase 1 (initialization) runs as Local Activities with no server round-trips. Phase 2 (settlement) runs as regular Activities in the background. The client receives its response as soon as Phase 1 completes, so the in-process execution speed of Local Activities directly improves the time-to-first-byte.

## Overview

The [Early Return](/design-patterns/early-return) pattern uses Update-with-Start to send the client an early response after a fast Phase 1 completes, while slow Phase 2 work continues in the background. This pattern extends Early Return by running Phase 1 Activities as **Local Activities**, eliminating all server round-trips on the synchronous hot path.

Because the client waits only for Phase 1, and Phase 1 now runs entirely in-process, the end-to-end first-response time drops from approximately 265 ms (Early Return with regular Activities) to approximately 160 ms.

```mermaid
sequenceDiagram
    participant C as Client
    participant S as Temporal Server
    participant W as Worker

    C->>S: UpdateWithStart (start workflow + deliver update)
    S->>W: Schedule WorkflowTask

    rect rgb(220, 245, 225)
        Note over S,W: Phase 1 — Local Activities<br/>(in-process, zero server round-trips)
        W->>W: validateTransaction (local)
        W->>W: initTransaction (local)
        W->>S: Complete WorkflowTask<br/>(Phase 1 done, Update result ready)
    end

    S-->>C: Update result returned (~160 ms first response)

    rect rgb(230, 235, 250)
        Note over S,W: Phase 2 — Regular Activities (background settlement)
        W->>S: Schedule completeTransaction
        S-->>W: Dispatch completeTransaction
        W->>W: Execute (slow settlement)
        W->>S: Complete ActivityTask
    end

    Note over C,W: Client has its response and Phase 2 continues in the background
```

**Numbered walkthrough:**

1. The client sends a single `UpdateWithStart` RPC, which atomically starts the Workflow and delivers the Update in one server call.
2. The Worker picks up the first Workflow Task and executes Phase 1. Because all Phase 1 Activities are Local Activities, they run in-process with no additional server calls. The entire phase completes inside a single Workflow Task.
3. When the Workflow Task completes, the server marks the Update as fulfilled and returns the result to the waiting client. This happens as soon as Phase 1 finishes—approximately 160 ms after the initial request.
4. The Workflow continues in a new Workflow Task to execute Phase 2 using regular Activities. These run in the background. The client is not blocked by this work.

## Problem

The plain Early Return pattern reduces first-response latency significantly compared to waiting for the full Workflow. However, if Phase 1 uses regular Activities, each Activity still incurs server scheduling overhead (~50 ms per call on Temporal Cloud). With two or three Phase 1 Activities, this overhead alone can account for 100–150 ms of the first-response time.

## Solution

Run all Phase 1 Activities as Local Activities. They execute in-process within the first Workflow Task, so their results are available to the Update handler as soon as the task completes—with no additional server calls. Phase 2 Activities remain regular Activities, which is acceptable because Phase 2 runs in the background after the client has already received its response.

**Python**

```python
# workflows.py
from temporalio import workflow
from datetime import timedelta
from activities import validate_transaction, init_transaction, complete_transaction, cancel_transaction

LOCAL_TIMEOUT = timedelta(seconds=5)
ACTIVITY_TIMEOUT = timedelta(seconds=30)

@workflow.defn
class TransactionWorkflow:
    def __init__(self) -> None:
        self._tx: Transaction | None = None
        self._phase1_done = False
        self._phase1_error: Exception | None = None

    @workflow.update
    async def get_result(self, req: TransactionRequest) -> Transaction:
        # Wait for Phase 1 to finish before returning to the caller.
        await workflow.wait_condition(lambda: self._phase1_done)
        if self._phase1_error:
            raise self._phase1_error
        return self._tx

    @workflow.run
    async def run(self, req: TransactionRequest) -> None:
        try:
            # Phase 1: Local Activities — zero server round-trips on the hot path.
            tx = await workflow.execute_local_activity(
                validate_transaction, req,
                schedule_to_close_timeout=LOCAL_TIMEOUT,
            )
            self._tx = await workflow.execute_local_activity(
                init_transaction, tx,
                schedule_to_close_timeout=LOCAL_TIMEOUT,
            )
        except Exception as e:
            self._phase1_error = e
        finally:
            self._phase1_done = True

        if self._phase1_error:
            if self._tx is not None:
                await workflow.execute_activity(
                    cancel_transaction, self._tx,
                    start_to_close_timeout=ACTIVITY_TIMEOUT,
                )
            return

        # Phase 2: Regular Activities — background settlement (client already has response).
        await workflow.execute_activity(
            complete_transaction, self._tx,
            start_to_close_timeout=ACTIVITY_TIMEOUT,
        )
```

**TypeScript**

```typescript
// workflows.ts
import { proxyLocalActivities, proxyActivities, defineUpdate, setHandler, condition } from "@temporalio/workflow";
import type * as activities from "./activities";
import type { TransactionRequest, Transaction } from "./shared";

// Phase 1: local activities — no server round-trips on the hot path.
const { validateTransaction, initTransaction } =
  proxyLocalActivities<typeof activities>({ scheduleToCloseTimeout: "5s" });

// Phase 2: regular activities — background settlement.
const { completeTransaction, cancelTransaction } =
  proxyActivities<typeof activities>({ startToCloseTimeout: "30s" });

export const getResultUpdate = defineUpdate<Transaction, [TransactionRequest]>("getResult");

export async function transactionWorkflow(req: TransactionRequest): Promise<void> {
  let tx: Transaction | undefined;
  let phase1Done = false;
  let phase1Error: unknown;

  setHandler(getResultUpdate, async () => {
    // The Update handler waits for Phase 1 before returning.
    await condition(() => phase1Done);
    if (phase1Error) throw phase1Error;
    return tx!;
  });

  try {
    // Phase 1: Local Activities run in-process.
    tx = await validateTransaction(req);
    tx = await initTransaction(tx);
  } catch (err) {
    phase1Error = err;
  } finally {
    phase1Done = true;
  }

  if (phase1Error) {
    if (tx !== undefined) {
      await cancelTransaction(tx);
    }
    return;
  }

  // Phase 2: Regular Activity runs in the background.
  await completeTransaction(tx!);
}
```

**Go**

```go
// workflows.go
func TransactionWorkflow(ctx workflow.Context, req TransactionRequest) error {
    var tx Transaction
    var initDone bool
    var initErr error

    // Register Update handler — returns to the client as soon as Phase 1 is done.
    if err := workflow.SetUpdateHandler(ctx, "getResult",
        func(ctx workflow.Context, r TransactionRequest) (Transaction, error) {
            _ = workflow.Await(ctx, func() bool { return initDone })
            return tx, initErr
        }); err != nil {
        return err
    }

    // Phase 1: Local Activities — in-process, zero server round-trips.
    localCtx := workflow.WithLocalActivityOptions(ctx, workflow.LocalActivityOptions{
        ScheduleToCloseTimeout: 5 * time.Second,
    })
    if err := workflow.ExecuteLocalActivity(localCtx, ValidateTransaction, req).Get(localCtx, &tx); err == nil {
        initErr = workflow.ExecuteLocalActivity(localCtx, InitTransaction, tx).Get(localCtx, &tx)
    } else {
        initErr = err
    }
    initDone = true

    activityCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
    })
    if initErr != nil {
        // Phase 2 (cancel): regular Activity runs in the background.
        return workflow.ExecuteActivity(activityCtx, CancelTransaction, tx).Get(activityCtx, nil)
    }
    // Phase 2 (complete): regular Activity runs in the background.
    return workflow.ExecuteActivity(activityCtx, CompleteTransaction, tx).Get(activityCtx, nil)
}
```

**Java**

```java
// TransactionWorkflow.java
public class Impl implements TransactionWorkflow {
    // Phase 1: local activities — zero server round-trips on the hot path.
    private final Activities localActivities = Workflow.newLocalActivityStub(
        Activities.class,
        LocalActivityOptions.newBuilder()
            .setScheduleToCloseTimeout(Duration.ofSeconds(5))
            .build()
    );
    // Phase 2: regular activities — background settlement.
    private final Activities activities = Workflow.newActivityStub(
        Activities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .build()
    );

    private Shared.Transaction tx;
    private boolean phase1Done = false;
    private RuntimeException phase1Error = null;

    @Override
    public Shared.Transaction getResult(Shared.TransactionRequest req) {
        // Update handler: wait for Phase 1 before returning.
        Workflow.await(() -> phase1Done);
        if (phase1Error != null) throw phase1Error;
        return tx;
    }

    @Override
    public void processTransaction(Shared.TransactionRequest req) {
        try {
            tx = localActivities.validateTransaction(req);
            tx = localActivities.initTransaction(tx);
        } catch (RuntimeException e) {
            phase1Error = e;
        } finally {
            phase1Done = true;
        }

        if (phase1Error != null) {
            activities.cancelTransaction(tx);
            return;
        }
        // Phase 2: regular activity in the background.
        activities.completeTransaction(tx);
    }
}
```

## When to use

**Good fit:**

- User-facing workflows where the first response is more latency-critical than total execution time
- Phase 1 consists of short, idempotent validation and initialization steps that fit naturally as Local Activities
- Phase 2 is slow (network I/O, external systems) and does not need to be on the client's critical path
- You already use or plan to use the Early Return pattern

**Poor fit:**

- Phase 1 Activities are long-running or require heartbeating—Local Activities cannot heartbeat
- The Workflow's total latency matters more than first-response latency
- Phase 1 and Phase 2 cannot be cleanly separated

## Benefits and trade-offs

| Pattern | First Response | Total Latency | Complexity |
|---|---|---|---|
| Synchronous workflow | Same as total | ~850 ms | Low |
| Early Return (regular activities) | ~265 ms | ~850 ms | Medium |
| Local Activities only | Same as total | ~275 ms | Medium |
| **Early Return + Local Activities** | **~160 ms** | **~275 ms** | **Medium** |
| Eager Workflow Start + Local Activities | ~160 ms | ~265 ms | High |

## Best practices

- **Keep Phase 1 Local Activities short.** Each must complete well within the Workflow Task timeout (default 10 seconds). Aim for under 5 seconds total for all Phase 1 work.
- **Design Phase 1 for at-least-once execution.** If the Workflow Task that runs Phase 1 fails and retries, all Phase 1 Local Activities re-execute. Phase 1 operations must be idempotent.
- **Separate Phase 1 and Phase 2 concerns cleanly.** The Update handler should wait only on the Phase 1 sentinel flag, not on any Phase 2 state. Phase 2 should be independent enough to proceed without client involvement.
- **Set appropriate timeouts for Phase 2.** Phase 2 regular Activities run in the background and should have a `startToCloseTimeout` that reflects the maximum acceptable settlement time.

## Common pitfalls

- **Putting slow operations in Phase 1.** If any Phase 1 Local Activity takes too long, the Workflow Task times out and retries. The client also waits longer for its early response, defeating the purpose of the pattern.
- **Non-idempotent Phase 1.** A retried Workflow Task re-executes all Local Activities in that task. Ensure Phase 1 operations (for example, creating a record in an external system) are safe to re-run.
- **Ignoring Phase 1 errors in Phase 2.** Always check Phase 1 error state before proceeding to Phase 2. If Phase 1 failed, Phase 2 should run a compensating Activity (cancel, rollback) rather than complete.
- **Mixing Local and regular Activity stubs incorrectly.** In Java, `Workflow.newLocalActivityStub` and `Workflow.newActivityStub` return distinct objects. Make sure Phase 1 uses the local stub and Phase 2 uses the regular stub.

## Related

### Patterns

- [Early Return](/design-patterns/early-return) — the baseline Update-with-Start pattern without Local Activity optimization
- [Local Activities](/design-patterns/local-activities) — using Local Activities for full-workflow latency reduction without early return
- [Eager Workflow Start](/design-patterns/eager-workflow-start) — eliminates the Matching step when starting the Workflow for additional total latency improvement

---

# Entity & Lifecycle Patterns

Source: https://docs.temporal.io/design-patterns/entity-lifecycle-patterns

> Pattern selection guide for modeling long-lived stateful entities and managing Workflow history growth over time.

These patterns model long-lived business entities as Workflows and keep those Workflows healthy as they run for days, months, or indefinitely. They cover how an entity holds and mutates state, how you bound Workflow history growth, and how you manage timers that change over time.

## Patterns in this section

- [Entity Workflow](/design-patterns/entity-workflow): Models a long-lived business entity — a user account, device, or order — as a single Workflow, with Signals and Updates driving every state transition.
- [Continue-As-New](/design-patterns/continue-as-new): Prevents unbounded history growth by completing the current execution and starting a fresh one that carries forward the current state.
- [Updatable Timer](/design-patterns/updatable-timer): Provides a timer you can extend, shorten, or cancel in response to Signals or Updates while the Workflow waits.

## Choosing a pattern

**You are modeling something with an ongoing lifecycle** — an account, a device, a subscription: use an [Entity Workflow](/design-patterns/entity-workflow) as the single source of truth for that entity.

**Your Workflow runs long enough to grow a large history**: apply [Continue-As-New](/design-patterns/continue-as-new) to reset history while preserving state.

**You need a wait that responds to new information**: use an [Updatable Timer](/design-patterns/updatable-timer) instead of a fixed sleep.

## Related sections

- [Workflow Messaging Patterns](/design-patterns/workflow-messaging-patterns) — the Signals and Updates that drive entity state transitions
- [Task Orchestration Patterns](/design-patterns/task-orchestration-patterns) — decompose a large entity into child Workflows

---

# Entity Workflow Pattern

Source: https://docs.temporal.io/design-patterns/entity-workflow

> A long-lived business entity — a user account, device, or order — gets one Workflow per instance, with Signals and Updates driving every state transition.

## Overview

The Entity Workflow pattern models long-lived business entities (users, accounts, devices, orders) as individual Workflows that persist for the entity's entire lifetime — potentially months or years.
Each entity gets its own Workflow instance identified by the entity ID, handling all state transitions and operations through Signals and Updates.

## Problem

Many business domains have entities — accounts, orders, subscriptions — that exist for extended periods and go through many state transitions over their lifetime. Modeling one well means keeping its state consistent across operations, recording an audit trail of every change, and handling concurrent operations safely.

Traditional approaches struggle with these requirements:

- **Database-centric**: Complex locking, race conditions, scattered business logic.
- **Event Sourcing**: Requires rebuilding state from events, complex infrastructure.
- **Stateless Services**: No built-in consistency, must coordinate state externally.
- **Short-lived Workflows**: Do not model the full entity lifecycle.

## Solution

You create one Workflow per entity, using the entity ID as the Workflow ID.
The Workflow runs for the entity's entire lifetime, maintaining state in Workflow variables and handling operations via Signals and Updates.
You use Continue-As-New periodically to prevent unbounded history growth.

```mermaid
sequenceDiagram
    participant Client
    participant UserWorkflow
    participant NotificationWorkflow
    participant Activities

    Client->>UserWorkflow: Start(userId="user-123")
    activate UserWorkflow
    Note over UserWorkflow: State: ACTIVE
    
    Client->>UserWorkflow: Update: updateProfile(data)
    UserWorkflow->>Activities: validateProfile(data)
    Activities-->>UserWorkflow: valid
    UserWorkflow->>Activities: updateDatabase(userId, data)
    Activities-->>UserWorkflow: success
    Note over UserWorkflow: Profile updated
    UserWorkflow-->>Client: Success
    
    Client->>UserWorkflow: Signal: suspend()
    Note over UserWorkflow: State: SUSPENDED
    UserWorkflow->>NotificationWorkflow: Start child workflow
    activate NotificationWorkflow
    NotificationWorkflow->>Activities: sendEmail(userId, "suspended")
    deactivate NotificationWorkflow
    
    Client->>UserWorkflow: Update: reactivate()
    Note over UserWorkflow: State: ACTIVE
    UserWorkflow-->>Client: Success
    
    Note over UserWorkflow: After 1000 operations...
    UserWorkflow->>UserWorkflow: Continue-As-New
    Note over UserWorkflow: Fresh history, same state
    
    Client->>UserWorkflow: Signal: delete()
    Note over UserWorkflow: State: DELETED
    UserWorkflow-->>UserWorkflow: Complete
    deactivate UserWorkflow
```

The following describes each step in the diagram:

1. The client starts the Workflow with a user ID. The Workflow initializes in the ACTIVE state.
2. The client sends an Update to modify the profile. The Workflow validates the data via an Activity, persists the change, and returns success.
3. The client sends a Signal to suspend the account. The Workflow transitions to SUSPENDED and starts a Child Workflow to send a notification email.
4. The client sends an Update to reactivate the account. The Workflow transitions back to ACTIVE.
5. After 1000 operations, the Workflow calls Continue-As-New to reset its history while preserving state.
6. The client sends a Signal to delete the account. The Workflow transitions to DELETED and completes.

## Implementation

The following examples show how each SDK implements the Entity Workflow pattern.
Each implementation defines Update handlers for synchronous operations, Signal handlers for asynchronous events, and Query handlers for state inspection.

**Python**

```python
# workflows.py
from dataclasses import dataclass
from datetime import datetime, timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import validate_profile

@dataclass
class UserState:
    status: str = "ACTIVE"
    profile: ProfileData | None = None
    pending_email: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None

@dataclass
class UserAccountInput:
    user_id: str
    # Unset for the original caller; carries state across Continue-As-New
    state: UserState | None = None

@workflow.defn
class UserAccountWorkflow:
    def __init__(self) -> None:
        self.state = UserState(created_at=datetime.utcnow())
        self.deleted = False
        self.operation_count = 0

    @workflow.run
    async def run(self, input: UserAccountInput) -> None:
        # On Continue-As-New, restore the state carried from the previous run
        if input.state is not None:
            self.state = input.state

        # Block until deleted or Continue-As-New is suggested
        await workflow.wait_condition(
            lambda: self.deleted or workflow.info().is_continue_as_new_suggested()
        )

        if not self.deleted and workflow.info().is_continue_as_new_suggested():
            await workflow.wait_condition(workflow.all_handlers_finished)
            # Carry current state forward so it is not reset on the new run
            workflow.continue_as_new(
                UserAccountInput(user_id=input.user_id, state=self.state)
            )

        self.state.status = "DELETED"

    @workflow.update
    async def update_profile(self, data: ProfileData) -> None:
        if self.deleted:
            raise ValueError("User account is deleted")

        await workflow.execute_activity(
            validate_profile, data,
            start_to_close_timeout=timedelta(seconds=30),
        )

        self.state.profile = data
        self.state.updated_at = datetime.utcnow()
        self.operation_count += 1

    @workflow.update
    async def suspend(self) -> None:
        if not self.deleted and self.state.status != "SUSPENDED":
            self.state.status = "SUSPENDED"
            self.state.updated_at = datetime.utcnow()
            self.operation_count += 1

    @workflow.update
    async def reactivate(self) -> None:
        if not self.deleted and self.state.status == "SUSPENDED":
            self.state.status = "ACTIVE"
            self.state.updated_at = datetime.utcnow()
            self.operation_count += 1

    @workflow.signal
    def delete(self) -> None:
        self.deleted = True

    @workflow.query
    def get_state(self) -> UserState:
        return self.state
```

**Go**

```go
// workflow.go
type UserAccountWorkflow struct{}

type UserState struct {
	Status      string
	Profile     ProfileData
	PendingEmail string
	CreatedAt   time.Time
	UpdatedAt   time.Time
}

type UserAccountInput struct {
	UserID string
	// Nil for the original caller; carries state across Continue-As-New
	State *UserState
}

func (w *UserAccountWorkflow) Run(ctx workflow.Context, input UserAccountInput) error {
	// On Continue-As-New, restore the state carried from the previous run
	var state UserState
	if input.State != nil {
		state = *input.State
	} else {
		state = UserState{
			Status:    "ACTIVE",
			CreatedAt: workflow.Now(ctx),
		}
	}
	deleted := false
	operationCount := 0

	err := workflow.SetUpdateHandler(ctx, "updateProfile", func(ctx workflow.Context, data ProfileData) error {
		if deleted {
			return errors.New("user account is deleted")
		}

		if err := workflow.ExecuteActivity(ctx, ValidateProfile, data).Get(ctx, nil); err != nil {
			return err
		}

		state.Profile = data
		state.UpdatedAt = workflow.Now(ctx)
		operationCount++

		return nil
	})
	if err != nil {
		return err
	}

	err = workflow.SetUpdateHandler(ctx, "suspend", func(ctx workflow.Context) error {
		if !deleted && state.Status != "SUSPENDED" {
			state.Status = "SUSPENDED"
			state.UpdatedAt = workflow.Now(ctx)
			operationCount++
		}
		return nil
	})
	if err != nil {
		return err
	}

	// Block until deleted or Continue-As-New is suggested
	for {
		selector := workflow.NewSelector(ctx)
		selector.AddReceive(workflow.GetSignalChannel(ctx, "delete"), func(c workflow.ReceiveChannel, more bool) {
			c.Receive(ctx, nil)
			deleted = true
		})
		selector.Select(ctx)

		if deleted {
			state.Status = "DELETED"
			return nil
		}
		if workflow.GetInfo(ctx).GetContinueAsNewSuggested() {
			// Carry current state forward so it is not reset on the new run
			return workflow.NewContinueAsNewError(ctx, w.Run, UserAccountInput{UserID: input.UserID, State: &state})
		}
	}
}
```

**Java**

```java
// UserAccountWorkflow.java

// userId is always set; state is empty for the original caller and
// carries state across Continue-As-New.
public record UserAccountInput(String userId, Optional<UserState> state) {}

@WorkflowInterface
public interface UserAccountWorkflow {
  @WorkflowMethod
  void run(UserAccountInput input);

  @UpdateMethod
  void updateProfile(ProfileData data);

  @UpdateMethod
  void changeEmail(String newEmail);

  @SignalMethod
  void suspend();

  @SignalMethod
  void reactivate();

  @SignalMethod
  void delete();

  @QueryMethod
  UserState getState();
}

public class UserAccountWorkflowImpl implements UserAccountWorkflow {
  private String userId;
  private UserState state = new UserState();
  private boolean deleted = false;
  private int operationCount = 0;
  private static final int CONTINUE_AS_NEW_THRESHOLD = 1000;

  @Override
  public void run(UserAccountInput input) {
    this.userId = input.userId();
    // On Continue-As-New, restore the state carried from the previous run
    if (input.state().isPresent()) {
      this.state = input.state().get();
    } else {
      state.setStatus("ACTIVE");
      state.setCreatedAt(Workflow.currentTimeMillis());
    }

    // Run until deleted or Continue-As-New is needed
    Workflow.await(() -> deleted || Workflow.getInfo().isContinueAsNewSuggested());

    if (!deleted && Workflow.getInfo().isContinueAsNewSuggested()) {
      // Carry current state forward so it is not reset on the new run
      Workflow.continueAsNew(new UserAccountInput(userId, Optional.of(state)));
    }

    state.setStatus("DELETED");
    state.setDeletedAt(Workflow.currentTimeMillis());
  }

  @Override
  public void updateProfile(ProfileData data) {
    validateNotDeleted();
    Activities.validateProfile(data);
    state.setProfile(data);
    state.setUpdatedAt(Workflow.currentTimeMillis());
    incrementOperationCount();
  }

  @Override
  public void changeEmail(String newEmail) {
    validateNotDeleted();
    Activities.sendVerificationEmail(userId, newEmail);
    state.setPendingEmail(newEmail);
    state.setUpdatedAt(Workflow.currentTimeMillis());
    incrementOperationCount();
  }

  @Override
  public void suspend() {
    if (!deleted && !"SUSPENDED".equals(state.getStatus())) {
      state.setStatus("SUSPENDED");
      state.setUpdatedAt(Workflow.currentTimeMillis());
      incrementOperationCount();
    }
  }

  @Override
  public void reactivate() {
    if (!deleted && "SUSPENDED".equals(state.getStatus())) {
      state.setStatus("ACTIVE");
      state.setUpdatedAt(Workflow.currentTimeMillis());
      incrementOperationCount();
    }
  }

  @Override
  public void delete() {
    deleted = true;
  }

  @Override
  public UserState getState() {
    return state;
  }

  private void validateNotDeleted() {
    if (deleted) {
      throw new IllegalStateException("User account is deleted");
    }
  }

  private void incrementOperationCount() {
    operationCount++;
  }
}
```

**TypeScript**

```typescript
// workflow.ts
import { condition, allHandlersFinished, defineUpdate, defineSignal, defineQuery, setHandler, continueAsNew, workflowInfo, proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { validateProfile } = proxyActivities<typeof activities>({
  startToCloseTimeout: '30s',
});

interface UserState {
  status: string;
  profile?: ProfileData;
  pendingEmail?: string;
  createdAt: number;
  updatedAt: number;
}

interface UserAccountInput {
  userId: string;
  // Unset for the original caller; carries state across Continue-As-New
  state?: UserState;
}

export const updateProfileUpdate = defineUpdate<void, [ProfileData]>('updateProfile');
export const suspendSignal = defineSignal('suspend');
export const deleteSignal = defineSignal('delete');
export const getStateQuery = defineQuery<UserState>('getState');

export async function userAccountWorkflow(input: UserAccountInput): Promise<void> {
  // On Continue-As-New, restore the state carried from the previous run
  const state: UserState = input.state ?? {
    status: 'ACTIVE',
    createdAt: Date.now(),
    updatedAt: Date.now(),
  };
  
  let deleted = false;
  let operationCount = 0;
  
  setHandler(updateProfileUpdate, async (data: ProfileData) => {
    if (deleted) {
      throw new Error('User account is deleted');
    }
    
    await validateProfile(data);
    
    state.profile = data;
    state.updatedAt = Date.now();
    operationCount++;
    

  });
  
  setHandler(suspendSignal, () => {
    if (!deleted && state.status !== 'SUSPENDED') {
      state.status = 'SUSPENDED';
      state.updatedAt = Date.now();
      operationCount++;
    }
  });
  
  setHandler(deleteSignal, () => {
    deleted = true;
  });
  
  setHandler(getStateQuery, () => state);
  
  // Block until deleted or Continue-As-New is suggested
  await condition(() => deleted || workflowInfo().continueAsNewSuggested);
  
  if (!deleted && workflowInfo().continueAsNewSuggested) {
    await condition(allHandlersFinished);
    // Carry current state forward so it is not reset on the new run
    await continueAsNew<typeof userAccountWorkflow>({ userId: input.userId, state });
  }
  
  state.status = 'DELETED';
}
```

The Workflow blocks on `Workflow.await(() -> deleted)` (Java), `condition(() => deleted)` (TypeScript), `workflow.wait_condition(lambda: self.deleted)` (Python), or `selector.Select(ctx)` (Go) until the delete Signal arrives.
All state transitions happen through Signal and Update handlers, ensuring that every operation on the entity goes through a single Workflow with no race conditions.
Continue-As-New is triggered from the main Workflow method (not from handlers) when `isContinueAsNewSuggested()` returns true.
The Workflow takes a single input object that carries both the entity ID and the current state, so state is passed forward on Continue-As-New rather than reset. The state field is unset for the original caller and populated only on continuation; the new run restores it before processing further operations.
All SDK docs explicitly warn: do not call Continue-As-New from Update or Signal handlers.
Instead, handlers set state, and the main Workflow method checks whether to Continue-As-New.

## When to use

The Entity Workflow pattern is a good fit for user accounts and profiles, IoT devices and sensors, customer relationships (CRM), shopping carts and orders, financial accounts, subscription management, device provisioning and lifecycle, and multi-tenant resources.

It is not a good fit for short-lived processes (use regular Workflows), stateless operations (use Activities), high-frequency updates (more than 100 per second per entity), or entities with only CRUD operations (use a database).

## Benefits and trade-offs

Benefits:

- All operations on an entity go through a single Workflow, eliminating race conditions.
- The Workflow history provides a complete audit trail of all state changes.
- All entity logic lives in one place, and state survives process crashes and restarts.
- Temporal provides exactly-once execution and automatic retries.
- You can inspect current state through Queries without side effects.

Trade-offs:

- You must use Continue-As-New to prevent unbounded history growth.
- A single Workflow handles all operations for one entity, which limits throughput.
- State is kept in Workflow memory, so you should use Activities for large data.
- One Workflow per entity means you should consider costs at scale.
- The first operation after an idle period may have latency.

## Comparison with alternatives

| Approach | Consistency | Audit trail | Complexity | Scalability |
| :--- | :--- | :--- | :--- | :--- |
| Entity Workflow | Strong | Complete | Low | High (per entity) |
| Database + Locks | Eventual | Manual | High | Very High |
| Event Sourcing | Strong | Complete | High | High |
| Stateless Service | Weak | Manual | Medium | Very High |

## Best practices

- **Use entity ID as Workflow ID.** This ensures uniqueness and idempotent starts.
- **Implement Continue-As-New.** Use `isContinueAsNewSuggested()` to check when to continue. Always call Continue-As-New from the main Workflow method, never from handlers. Wait for all handlers to finish before continuing.
- **Validate in Updates.** Use Updates for operations that require validation and a return value.
- **Use Signals for events.** Use Signals for asynchronous notifications that do not need responses.
- **Keep state minimal.** Store large data externally and reference it in the Workflow.
- **Add Queries.** Expose state for monitoring and debugging.
- **Handle deletion.** Implement an explicit deletion or decommission Signal.
- **Version carefully.** Use Worker versioning for Workflow code changes.
- **Set timeouts.** Use Workflow execution timeout as a safety net.
- **Monitor history size.** Alert when approaching the Continue-As-New threshold.

## Common pitfalls

- **Calling Continue-As-New from Signal or Update handlers.** Continue-As-New must be called from the main Workflow method, never from inside a handler. Calling it from a handler causes non-determinism errors.
- **Not waiting for handlers to finish before Continue-As-New.** Use `allHandlersFinished` (TypeScript), `Workflow.isEveryHandlerFinished()` (Java), or `workflow.all_handlers_finished()` (Python) to ensure in-flight handlers complete before transitioning.
- **Losing Update ID deduplication across Continue-As-New.** Update IDs are scoped to a single Workflow Execution. After Continue-As-New, the same Update ID can be accepted again. Carry processed IDs in the Continue-As-New input if deduplication is needed.
- **Exceeding the 2 MB payload limit on Continue-As-New input.** State passed to Continue-As-New is subject to the same 2 MB blob size limit as Workflow inputs. Use external storage for large state.
- **Using a hardcoded counter instead of `isContinueAsNewSuggested`.** The SDK provides `isContinueAsNewSuggested()` which accounts for actual history size. Hardcoded thresholds may be too aggressive or too lenient.

## Related

### Patterns

- **[Continue-As-New](/design-patterns/continue-as-new)**: Essential for preventing unbounded history.
- **[Request-Response via Updates](/design-patterns/request-response-via-updates)**: Synchronous operations with validation.
- **[Signal with Start](/design-patterns/signal-with-start)**: Idempotent Workflow start with an initial Signal.

### Guides

- [Track customer loyalty points with durable Workflows](/guides/entity-pattern-loyalty-points): A complete Python implementation of this pattern, including tier calculation, Continue-As-New, and Worker Versioning for accounts that span years.
- [Player Sessions That Survive Anything](/guides/durable-gaming-sessions): Extends the Entity Workflow into an Actor Workflow that executes game actions — joining rooms, resolving combat — rather than only holding state.

### Sample code

**Python:**
- [Safe Message Handlers](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) — Entity Workflow with Updates, Signals, and Continue-As-New.

**Go:**
- [Safe Message Handlers](https://github.com/temporalio/samples-go/tree/main/safe_message_handler) — Entity Workflow with Updates, Signals, and Continue-As-New.

**Java:**
- [Safe Message Handlers](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Entity Workflow with Updates, Signals, and Continue-As-New.

**TypeScript:**
- [Safe Message Handlers](https://github.com/temporalio/samples-typescript/tree/main/message-passing/safe-message-handlers) — Entity Workflow with Updates, Signals, and Continue-As-New.

### References

- [Temporal Blog: Very Long-Running Workflows](https://temporal.io/blog/very-long-running-workflows) — Guidance on managing Workflows that run for extended periods.
- [Continue-As-New](/workflow-execution/continue-as-new) — Concept reference for the Continue-As-New mechanism.

---

# Error Handling & Retry Patterns

Source: https://docs.temporal.io/design-patterns/error-handling-patterns

> Pattern selection guide and decision tree for choosing the right retry strategy based on your error type, cost constraints, and recovery requirements.

These patterns control how Temporal retries Activities, surfaces persistent failures, and recovers from errors that require human intervention.

## Patterns in this section

- [Fixed Count of Retries](/design-patterns/fixed-count-retries): Caps the number of Activity retry attempts to control cost when each attempt consumes a paid or limited resource.
- [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Bounds the total elapsed time across all retry attempts to enforce a business SLA, regardless of how many attempts occur.
- [Non-Retryable Errors](/design-patterns/non-retryable-errors): Marks error types that will never succeed — such as validation failures or missing records — so Temporal fails fast instead of retrying.
- [Delayed Retry](/design-patterns/delayed-retry): Override one failure's retry interval with nextRetryDelay on ApplicationFailure, matching the wait time the error reports.
- [Fast/Slow Retries](/design-patterns/fast-slow-retries): Retries fast with a short interval first, then shifts to a slow, unlimited interval so the Workflow outlasts an extended downstream outage.
- [Retry Alerting via Metrics](/design-patterns/retry-metrics): Emits a metric from the Activity when attempts cross a threshold, so on-call teams see persistent failures before an SLA breach.
- [Resumable Activity](/design-patterns/resumable-activity): Parks the Workflow after retries are exhausted and waits for a human to correct the data or approve continuing, then resumes from where it left off.

## Choosing a pattern

The following decision tree helps you select the appropriate retry strategy for your use case.

```mermaid
flowchart TD
    Start([Activity failing]) --> Q1{Each attempt\ncosts money or\nconsumes quota?}
    Q1 -->|Yes| FixedCount[Fixed Count of Retries\nCap MaximumAttempts]
    Q1 -->|No| Q2{Will this error\never succeed\nautomatically?}
    Q2 -->|No| Q2a{Can a human\ncorrect and retry?}
    Q2a -->|Yes| Resumable[Resumable Activity\nPark and await signal]
    Q2a -->|No| NonRetryable[Non-Retryable Errors\nFail fast]
    Q2 -->|Yes| Q3{Downstream has\na predictable\nunavailability window?}
    Q3 -->|Yes| Delayed[Delayed Retry\nFixed interval backoff]
    Q3 -->|No| Q4{Must resolve\nwithin a\ntime budget?}
    Q4 -->|Yes| WallTime[Fixed Wall-Time Retries\nScheduleToCloseTimeout]
    Q4 -->|No| Q5{Want aggressive\ninitial retries then\npatient recovery?}
    Q5 -->|Yes| FastSlow[Fast/Slow Retries\nTwo-phase retry policy]
    Q5 -->|No| Metrics[Retry Alerting via Metrics\nEmit metrics at attempt threshold]
```

The following describes each decision point:

1. If each attempt consumes a paid API call, a rate-limited token, or another scarce resource, use **Fixed Count of Retries** to cap total consumption.
2. If the error is structural — a missing record, invalid input, or authorization failure — and cannot be corrected automatically, ask whether a human can fix it: if so, use **Resumable Activity** to park the Workflow and await a correction signal; otherwise use **Non-Retryable Errors** to fail fast.
3. If the downstream system has a scheduled maintenance window and you know approximately how long it will be unavailable, use **Delayed Retry** with a fixed interval.
4. If the process must resolve (one way or another) within a business SLA window such as 24 hours, use **Fixed Wall-Time Retries** with `ScheduleToCloseTimeout`.
5. If you want to recover from transient errors quickly but also wait indefinitely for the downstream system to come back, use **Fast/Slow Retries**.
6. For any long-running retry scenario, add **Retry Alerting via Metrics** to surface persistent failures before they breach an SLA.

## How Temporal retries work

Temporal's default `RetryPolicy` retries Activities indefinitely with exponential backoff.
Unless you configure a policy, a failing Activity will keep retrying until the `ScheduleToCloseTimeout` or the Workflow itself completes.

The key `RetryPolicy` fields are:

| Field | Default | Effect |
| :--- | :--- | :--- |
| `MaximumAttempts` | 0 (unlimited) | Caps total attempts including the first |
| `InitialInterval` | 1 second | Delay before the first retry |
| `BackoffCoefficient` | 2.0 | Multiplier applied after each retry |
| `MaximumInterval` | 100× InitialInterval | Upper bound on the backoff delay |
| `NonRetryableErrorTypes` | `[]` | Error types that skip retries entirely |

`ScheduleToCloseTimeout` is set on the Activity call options, not in `RetryPolicy`.
It caps the total wall-clock time from when the Activity is first scheduled to when it must complete — across all retry attempts.

## Related sections

- [External Interaction Patterns](/design-patterns/external-interaction-patterns) — heartbeating, polling, and approval gates for the external calls that fail
- [Distributed Transaction Patterns](/design-patterns/distributed-transaction-patterns) — compensate completed steps when retries finally give up
- [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — where retry configuration fits in the latency budget

## References

- [Temporal Retry Policies](/encyclopedia/retry-policies)
- [Understanding Workflow Retries and Failures](https://community.temporal.io/t/understanding-workflow-retries-and-failures/122)
- [Failure Handling in Practice](https://temporal.io/blog/failure-handling-in-practice)

---

# Event Accumulator Pattern

Source: https://docs.temporal.io/design-patterns/event-accumulator

> Durably collect and deduplicate signals from multiple senders, then process the batch after a sliding inactivity timeout.

> **ℹ️ TL;DR:**
> Use the Event Accumulator pattern to **durably collect and process events from multiple senders over an unlimited time.** The workflow accumulates signals, deduplicates by a stable item key, and processes the batch after a sliding inactivity timeout — no external coordination, no lost events on retry.

## Overview

The Accumulator pattern groups a stream of incoming events by a key and processes them together as a batch.
A *group key* is a stable, domain-specific identifier — for example, an order ID, customer ID, or session token — that logically binds related events belonging to the same accumulation window.
A single workflow instance per group key receives signals as events arrive, deduplicates them, and waits with a sliding inactivity timer.
When no new events arrive within the timeout window, the workflow calls a batch processing activity and completes.

## Problem

In distributed systems, events for the same logical entity arrive asynchronously from multiple producers at unpredictable rates.
Processing each event individually wastes downstream resources and makes throughput harder to control.
Ensuring exactly one active collector per group key — even under concurrent producers — requires coordination logic that is difficult to build reliably without distributed state.

Without the Accumulator pattern, you must:

- Handle race conditions when multiple producers try to start the same collection workflow simultaneously.
- Implement deduplication externally, because at-least-once delivery is common in event streams.
- Manage a reset timer that extends the collection window each time a new event arrives, without a reliable durable timer primitive.
- Persist collection state externally across restarts and failures.
- Handle gracefully the case where a long accumulation period grows the workflow history beyond safe limits.

## Solution

You assign each group a deterministic workflow ID derived from the group key (for example, `accumulator-order-123`).
Producers call Signal-With-Start so the workflow is created on first use and receives additional signals on subsequent calls — without any client-side coordination.
Inside the workflow, `Workflow.await()` with a timeout implements a sliding inactivity window: each arriving signal resets the countdown.
When the countdown expires — or when an explicit flush signal is sent — the workflow passes all accumulated, deduplicated events to a batch processing activity and completes.
If the accumulation period is long enough to grow the workflow history near its limit, the workflow uses Continue-As-New to carry its state forward into a fresh run.

```mermaid
sequenceDiagram
    participant P as Producers
    participant T as Temporal
    participant W as Accumulator Workflow
    participant A as Batch Activity

    P->>T: SignalWithStart(order-A, item-1)
    T->>W: Start workflow + deliver add-item(item-1)
    activate W
    Note over W: Accumulate item-1 · reset timer

    P->>T: SignalWithStart(order-A, item-2)
    T->>W: Deliver add-item(item-2)
    Note over W: Accumulate item-2 · reset timer

    P->>T: SignalWithStart(order-A, item-1) ← duplicate
    T->>W: Deliver add-item(item-1)
    Note over W: Deduplicate: item-1 already seen

    Note over W: Timer expires — no new signals

    W->>A: processItems(order-A, [item-1, item-2])
    A-->>W: "Order order-A: 2 items fulfilled"
    deactivate W
```

The following describes each step in the diagram:

1. A producer calls Signal-With-Start with `order-A` as the bucket key. Temporal starts `accumulator-order-A` and delivers the first `add-item` signal. The workflow adds `item-1` to its list and starts the inactivity timer.
2. A second producer calls Signal-With-Start for the same order. Temporal finds the workflow already running and delivers the signal — no new instance is created. The workflow adds `item-2` and resets the timer.
3. The first producer retries `item-1` due to at-least-once delivery. The workflow checks its deduplication set, finds `item-1` already recorded, and discards the duplicate. Alternatively, you can replace the existing record with the new payload — useful when a producer may resend an updated version of the same event under the same key.
4. No new signals arrive within the inactivity window. The `Workflow.await()` condition times out.
5. The workflow calls `processItems` with all accumulated items. The activity returns a result, and the workflow completes.

## Implementation

The following examples highlight the key mechanics of the accumulator loop.
Full implementations including the worker, starter, and shared types are available in the runner above.

**TypeScript**

```typescript
// workflows.ts
export async function accumulatorWorkflow(
  bucketKey: string,
  accumulated: OrderItem[] = [],
  seenKeys: string[] = [],
): Promise<string> {
  const seenSet = new Set(seenKeys);
  const items: OrderItem[] = [...accumulated];
  const unprocessed: OrderItem[] = [];
  let flushRequested = false;

  setHandler(addItemSignal, (item: OrderItem) => {
    unprocessed.push(item);
  });
  setHandler(flushSignal, () => {
    flushRequested = true;
  });

  do {
    // Sliding window: wait for a signal or let the inactivity timer fire
    const timedOut = !(await condition(
      () => unprocessed.length > 0 || flushRequested,
      "10 seconds",
    ));

    // Drain and deduplicate incoming signals
    while (unprocessed.length > 0) {
      const item = unprocessed.shift()!;
      if (item.orderId === bucketKey && !seenSet.has(item.itemId)) {
        seenSet.add(item.itemId);
        items.push(item);
      }
    }

    if (timedOut || flushRequested) {
      const result = await processItems(bucketKey, items);
      if (unprocessed.length === 0) return result;
      // More signals arrived after timeout/flush — loop to process them
    }
  } while (unprocessed.length > 0 || !workflowInfo().continueAsNewSuggested);

  // History growing large — continue as new, carrying accumulated state forward
  await continueAsNew<typeof accumulatorWorkflow>(bucketKey, items, [...seenSet]);
  return ""; // unreachable
}
```

**Python**

```python
# workflows.py
@workflow.defn
class AccumulatorWorkflow:
    def __init__(self) -> None:
        self._unprocessed: deque[OrderItem] = deque()
        self._flush_requested = False

    @workflow.signal(name="add-item")
    async def add_item(self, item: OrderItem) -> None:
        self._unprocessed.append(item)

    @workflow.signal(name="flush")
    async def flush(self) -> None:
        self._flush_requested = True

    @workflow.run
    async def run(
        self,
        bucket_key: str,
        accumulated: list[OrderItem] | None = None,
        seen_keys: list[str] | None = None,
    ) -> str:
        items = list(accumulated or [])
        seen_set = set(seen_keys or [])

        while True:
            # Sliding window: wait for a signal or let the inactivity timer fire
            timed_out = not await workflow.wait_condition(
                lambda: bool(self._unprocessed) or self._flush_requested,
                timeout=timedelta(seconds=10),
            )

            # Drain and deduplicate the signal queue
            while self._unprocessed:
                item = self._unprocessed.popleft()
                if item.order_id == bucket_key and item.item_id not in seen_set:
                    seen_set.add(item.item_id)
                    items.append(item)

            if timed_out or self._flush_requested:
                result = await workflow.execute_activity(
                    process_items,
                    args=[bucket_key, items],
                    start_to_close_timeout=timedelta(seconds=10),
                )
                if not self._unprocessed:
                    return result
                # More signals arrived after timeout/flush — loop to process them

            if not self._unprocessed and workflow.info().is_continue_as_new_suggested():
                workflow.continue_as_new(args=[bucket_key, items, sorted(seen_set)])
```

**Go**

```go
// workflows.go
func AccumulatorWorkflow(ctx workflow.Context, bucketKey string, items []OrderItem, seenKeys []string) (string, error) {
    seenSet := make(map[string]bool)
    for _, k := range seenKeys {
        seenSet[k] = true
    }
    accumulated := append([]OrderItem{}, items...)

    addItemCh := workflow.GetSignalChannel(ctx, "add-item")
    flushCh := workflow.GetSignalChannel(ctx, "flush")
    flushRequested := false

    for {
        // Drain any signals buffered before this iteration
        for {
            var item OrderItem
            if !addItemCh.ReceiveAsync(&item) {
                break
            }
            if item.OrderID == bucketKey && !seenSet[item.ItemID] {
                seenSet[item.ItemID] = true
                accumulated = append(accumulated, item)
            }
        }
        var voidFlush interface{}
        if flushCh.ReceiveAsync(&voidFlush) {
            flushRequested = true
        }
        if flushRequested {
            break
        }
        if workflow.GetInfo(ctx).GetContinueAsNewSuggested() {
            keys := make([]string, 0, len(seenSet))
            for k := range seenSet {
                keys = append(keys, k)
            }
            sort.Strings(keys) // deterministic order for replay
            return "", workflow.NewContinueAsNewError(ctx, AccumulatorWorkflow, bucketKey, accumulated, keys)
        }

        // Sliding window: wait for a signal or let the inactivity timer fire
        timedOut := false
        timerCtx, cancelTimer := workflow.WithCancel(ctx)
        timer := workflow.NewTimer(timerCtx, maxAwaitTime)
        selector := workflow.NewSelector(ctx)
        selector.AddFuture(timer, func(f workflow.Future) { timedOut = true })
        selector.AddReceive(addItemCh, func(c workflow.ReceiveChannel, _ bool) {
            var item OrderItem
            c.Receive(ctx, &item)
            if item.OrderID == bucketKey && !seenSet[item.ItemID] {
                seenSet[item.ItemID] = true
                accumulated = append(accumulated, item)
            }
        })
        selector.AddReceive(flushCh, func(c workflow.ReceiveChannel, _ bool) {
            var void interface{}
            c.Receive(ctx, &void)
            flushRequested = true
        })
        selector.Select(ctx)
        cancelTimer() // no-op if timer already fired; cancels timer if a signal arrived

        if timedOut || flushRequested {
            break
        }
    }

    ao := workflow.ActivityOptions{StartToCloseTimeout: 10 * time.Second}
    actCtx := workflow.WithActivityOptions(ctx, ao)
    var result string
    if err := workflow.ExecuteActivity(actCtx, ProcessItems, bucketKey, accumulated).Get(ctx, &result); err != nil {
        return "", err
    }
    workflow.GetLogger(ctx).Info("Processed order batch", "bucketKey", bucketKey, "count", len(accumulated))
    return result, nil
}
```

**Java**

```java
// AccumulatorWorkflow.java
@WorkflowInterface
public interface AccumulatorWorkflow {
    @WorkflowMethod
    String accumulate(String bucketKey, List<Shared.OrderItem> items, List<String> seenKeys);

    @SignalMethod
    void addItem(Shared.OrderItem item);

    @SignalMethod
    void flush();

    class Impl implements AccumulatorWorkflow {
        private final Activities activities = Workflow.newActivityStub(
            Activities.class,
            ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Shared.MAX_AWAIT_TIME.plusSeconds(10))
                .build());

        private final ArrayDeque<Shared.OrderItem> unprocessed = new ArrayDeque<>();
        private boolean flushRequested = false;

        @Override
        public String accumulate(String bucketKey, List<Shared.OrderItem> itemsInput, List<String> seenKeysInput) {
            List<Shared.OrderItem> items = new ArrayList<>(itemsInput);
            Set<String> seenSet = new HashSet<>(seenKeysInput);

            do {
                // Sliding window: wait for a signal or let the inactivity timer fire
                boolean timedOut = !Workflow.await(
                    Shared.MAX_AWAIT_TIME,
                    () -> !unprocessed.isEmpty() || flushRequested);

                // Drain and deduplicate the signal queue
                while (!unprocessed.isEmpty()) {
                    Shared.OrderItem item = unprocessed.removeFirst();
                    if (item.orderId.equals(bucketKey) && seenSet.add(item.itemId)) {
                        items.add(item);
                    }
                }

                if (timedOut || flushRequested) {
                    String result = activities.processItems(bucketKey, items);
                    if (unprocessed.isEmpty()) return result;
                    // More signals arrived after timeout/flush — loop to process them
                }
            } while (!unprocessed.isEmpty() || !Workflow.getInfo().isContinueAsNewSuggested());

            // History growing large — continue as new, carrying accumulated state forward
            AccumulatorWorkflow continueAsNew = Workflow.newContinueAsNewStub(AccumulatorWorkflow.class);
            List<String> seenKeysList = new ArrayList<>(seenSet);
            java.util.Collections.sort(seenKeysList); // deterministic order for replay
            continueAsNew.accumulate(bucketKey, items, seenKeysList);
            return ""; // unreachable
        }

        @Override
        public void addItem(Shared.OrderItem item) {
            unprocessed.add(item);
        }

        @Override
        public void flush() {
            flushRequested = true;
        }
    }
}
```

The key differences between SDKs:

- **TypeScript**: Uses `setHandler` to register signal handlers and `condition()` with a string duration for the sliding window. `continueAsNew<typeof accumulatorWorkflow>()` carries accumulated state to the next run.
- **Python**: Uses `@workflow.signal(name=...)` decorators and `workflow.wait_condition()` with a `timedelta` timeout. `workflow.continue_as_new()` accepts the new run arguments.
- **Go**: Uses `workflow.GetSignalChannel` to obtain named signal channels, then builds a `Selector` per iteration with a `NewTimer` future and two channel receivers. Cancels the old timer whenever a signal arrives before it fires.
- **Java**: Uses `@SignalMethod` annotations and `Workflow.await(Duration, Supplier<Boolean>)`. `Workflow.newContinueAsNewStub` carries state to the next run.

Producers in all SDKs call Signal-With-Start to atomically start the workflow if not running and deliver the first signal without any client-side coordination.

## When to use

The Accumulator pattern is well suited when events related to the same entity or group arrive from multiple producers — including a single consumer (for example, a Kafka consumer) polling a topic that carries events for multiple groups simultaneously — and can be processed together as batches, when downstream systems prefer batched calls rather than one call per event, and when at-least-once event delivery makes deduplication necessary at the collection layer.

It is not a good fit for use cases that require processing every event individually in _strict order_, for cases where the batch size is known in advance and all events arrive within a short deterministic window (a standard workflow is sufficient), or when events for different keys must be correlated at processing time (consider fan-in with for example Child Workflows or multiple levels of Accumulator).

## Benefits and trade-offs

The Accumulator pattern reduces downstream load by consolidating many individual events into a single batch and activity call.
Signal-With-Start eliminates client-side coordination logic for starting or locating the collector workflow.
Temporal's durable execution guarantees that accumulated state survives Worker restarts, and Continue-As-New prevents history growth from becoming a long-term problem.

The trade-offs to consider are that events are not processed until the inactivity timeout fires or a flush signal is sent, introducing intentional latency.
The timeout value is a domain-specific trade-off between latency and batch size.
If producers stop sending signals but never send a flush, the workflow holds open resources until the timeout fires.

## Best practices

- **Use a deterministic workflow ID per bucket key.** Encode the group key and, optionally, an accumulation period (for example, `accumulator-order-123-2026-05-14`) to control when a new window starts.
- **Always use Signal-With-Start from producers.** Calling start and signal separately is not atomic: a signal sent between the two calls can be lost if the workflow has not yet started.
- **Include a deduplication key in every signal payload.** At-least-once delivery is common in event streams; without a dedup key, retried events add duplicate entries to the batch.
- **Size the inactivity timeout to your domain's quiet period.** The timeout should reflect how long you are confident no more events will arrive for this batch. Tune it based on observed producer behavior, not an arbitrary constant.
- **Pass accumulated state as workflow arguments into each Continue-As-New run.** Workflow state does not survive a Continue-As-New transition automatically; always pass items and the seen-keys list as arguments to the new run. If you omit the seen-keys list, any signal delivered to both the old and new run during the Continue-As-New handoff will be processed again in the new run, producing duplicate entries in the batch.
- **Keep signal handlers fast and side-effect free.** Signal handlers must not call activities or yield to the scheduler. Buffer incoming signals and process them in the main workflow loop.
- **Add a flush signal for testing and operational runbooks.** An explicit flush signal lets you trigger early batch processing without waiting for the timeout, which is useful for end-to-end tests and manual intervention.
- **Keep producer signal rate below 5/sec per workflow instance.** Each signal briefly locks the workflow execution. A sustained rate above roughly 5 signals/second causes workflow task backlog, limits throughput, and can eventually prevent Continue-As-New from completing. If your producer rate is higher, partition by a finer-grained key so each accumulator workflow receives a manageable share of the total signal volume.
- **Account for the 10,000-signal-per-run limit on Temporal Cloud.** A single workflow run in Temporal Cloud can receive at most 10,000 signals. If your accumulation window is long and producers are active, ensure your Continue-As-New trigger fires well before the per-run signal count reaches this limit.

## Common pitfalls

- **Non-deterministic or random workflow IDs.** If the workflow ID is not derived deterministically from the group key, multiple accumulator instances are created for the same group, splitting the batch.
- **Calling start followed by a separate signal.** These are not atomic. A signal sent between the two calls will be lost if the workflow has not yet started. Use Signal-With-Start instead.
- **Omitting a deduplication key.** Retried or re-delivered events add duplicate entries to the batch. Every signal payload must carry a stable, unique key.
- **Timeout set too short.** The workflow processes a partial batch while more events are still in flight, forcing producers to re-send unprocessed events. Profile your producer arrival rate before choosing a timeout.
- **Forgetting to pass accumulated state into Continue-As-New.** The new run starts with empty state and re-processes events from scratch, producing duplicate batches.
- **Calling activities inside signal handlers.** Signal handlers run synchronously in the workflow thread and must not block or call activities. Buffer the item and let the main loop handle activity calls.
- **Assuming workflow completion means all events were captured.** Producers that send signals after the workflow completes will start a new accumulator instance. Decide whether this is intentional (a new accumulation window) or an error.
- **Signal rate too high to allow Continue-As-New to complete.** Continue-As-New requires a brief window (~100 ms) with no unhandled signals. If producers send signals continuously without pause, the workflow can never enter that window, history grows without bound, and Temporal will eventually terminate the workflow. Partition by a finer-grained key, throttle producers, or batch multiple events into a single signal payload to keep the per-instance signal rate low enough for CAN to succeed.
- **Not draining the signal queue before calling Continue-As-New.** Any signal that arrives between the CAN decision and the actual CAN execution can be lost if the signal buffer is not empty when CAN fires. All SDK implementations in this pattern guard against this by re-checking the unprocessed queue before continuing; do not remove that guard or call CAN unconditionally on the history-size trigger.

## Related

### Patterns

- **[Signal with Start](/design-patterns/signal-with-start)** — the atomic start-and-signal primitive this pattern uses to create or locate the accumulator workflow.
- **[Continue-As-New](/design-patterns/continue-as-new)** — used to reset workflow history when the accumulation period is long.
- **[Updatable Timer](/design-patterns/updatable-timer)** — an alternative approach for a resettable timer that does not require signals.
- **[Entity Workflow](/design-patterns/entity-workflow)** — a broader pattern for long-lived, keyed workflow instances.

### Sample code

- [Java Sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloAccumulator.java) — the canonical accumulator example from the Temporal Java samples repository.

---

# External Interaction Patterns

Source: https://docs.temporal.io/design-patterns/external-interaction-patterns

> Compares five patterns for waiting on external systems and human decisions: polling, heartbeating Activities, delayed start, webhooks, and approval.

These patterns cover how a Workflow waits on or interacts with the world outside it — external APIs, human decisions, scheduled delays, and inbound or outbound callbacks — while staying durable across failures.

## Patterns in this section

- [Polling External Services](/design-patterns/polling): Checks an external resource on a schedule until it reaches the state you need, with frequent, infrequent, and periodic variants.
- [Long Running Activity](/design-patterns/long-running-activity): Reports progress via heartbeats and resumes after failures, with cancellation support, for Activities that run for minutes to hours.
- [Approval](/design-patterns/approval): Blocks the Workflow until an external decision arrives, capturing the approval and its metadata through a Signal.
- [Delayed Start](/design-patterns/delayed-start): Creates the Workflow immediately but defers execution until a delay expires.
- [Delayed Callback (Webhooks)](/design-patterns/delayed-callback): Webhooks become durable: inbound calls arrive as Signals, outbound calls fire after a durable sleep, and Activities complete later via task tokens.

## Choosing a pattern

**The external system offers no notification**: use [Polling External Services](/design-patterns/polling) and tune the interval to the expected latency.

**One Activity runs for a long time**: use a [Long Running Activity](/design-patterns/long-running-activity) with heartbeats so failures resume instead of restarting.

**A person or external system must decide**: use [Approval](/design-patterns/approval) to block on a Signal.

**Execution should begin later**: use [Delayed Start](/design-patterns/delayed-start).

**You send or receive HTTP callbacks**: use [Delayed Callback (Webhooks)](/design-patterns/delayed-callback).

## Related sections

- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns) — retry strategies for the external calls these patterns make
- [Workflow Messaging Patterns](/design-patterns/workflow-messaging-patterns) — the Signals that deliver external decisions into a Workflow

---

# Fairness

Source: https://docs.temporal.io/design-patterns/fairness

> Distributes Task dispatches across tenants or users so that a burst from one caller does not starve others.

> **ℹ️ TLDR:**
> Assign a Fairness key and weight to Workflows and Activities so each tenant or group receives a proportional share of Task dispatches on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants, without requiring a separate Task Queue per tenant.

## Overview

The Fairness pattern distributes Task dispatches proportionally across tenants or user groups within a shared Task Queue so that a burst from one caller cannot starve others. Each group has a Fairness key and an optional weight. The Matching Service uses weighted fair dispatch to select the next Task within a Priority level.

Fairness applies only to Task dispatch. It does not account for Task duration or resource use.

## Problem

When multiple tenants, such as customers, share a Task Queue, a high-volume tenant can fill the backlog and dominate dispatch. Tasks from other tenants can wait behind that backlog, making their latency unpredictable during bursts.

The classic workaround is one Task Queue per tenant. This adds Task Queue, Worker, and routing configuration for every tenant. It can also strand idle capacity when Workers are dedicated to individual tenants.

## Solution

Temporal's Fairness feature lets you assign a Fairness key, such as a tenant name or tier, and an optional Fairness weight to Workflows, Activities, and Child Workflows. A useful mental model is one virtual queue per Fairness key, with weighted round-robin dispatch across the queues. Temporal approximates this model with stride scheduling and a count-min sketch for larger key sets. A single shared Worker pool serves all keys, with no extra queues or routing logic.

For example, assigning weights of 5.0, 3.0, and 2.0 causes approximately 50% of dispatched Tasks to come from `premium`, 30% from `basic`, and 20% from `free` when all three groups have backlogged Tasks. Within a Fairness key, Tasks at the same priority are dispatched in first-in-first-out (FIFO) order.

```mermaid
flowchart TD
    WA["Workflow\nfairness_key=tenant-big\nweight=1.0"] --> TQ["my-task-queue"]
    WB["Workflow\nfairness_key=tenant-mid\nweight=1.0"] --> TQ
    WC["Workflow\nfairness_key=tenant-small\nweight=1.0"] --> TQ
    TQ --> VQ1["Virtual queue\ntenant-big"]
    TQ --> VQ2["Virtual queue\ntenant-mid"]
    TQ --> VQ3["Virtual queue\ntenant-small"]
    VQ1 -->|weighted dispatch| W["Shared Workers"]
    VQ2 -->|weighted dispatch| W
    VQ3 -->|weighted dispatch| W
```

The following describes each step in the diagram:

1. Workflows start with a Fairness key that identifies their tenant or group.
2. Tasks with the same Fairness key enter the same virtual queue.
3. When multiple virtual queues have backlogged Tasks, the Matching Service uses weighted round robin to choose between them.
4. One virtual queue can use all available dispatches when the others have no backlogged Tasks.

## Implementation

Fairness must be enabled for the Namespace. Set Fairness keys and weights on Workflows, Activities, or Child Workflows. Activities and Child Workflows inherit these values unless you override them.

See [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) for setup, SDK examples, inheritance, Task Queue configuration, and limitations.

## When to use

This pattern is a good fit for multi-tenant applications where large tenants should not block small tenants, for workloads that need proportional Task dispatch across groups without hard rate limits, and when the set of tenants or groups is dynamic. New Fairness keys can be introduced without deploying new Workers. For a broader look at multi-tenancy strategies in Temporal, see [Multi-Tenant Patterns](/best-practices/multi-tenant-patterns).

Fairness does not provide exact dispatch ratios, concurrency limits, or compute isolation. Use Activity Task Queue rate limits for throughput caps. Use separate Task Queues with dedicated Worker pools and compute resources for hard isolation. Use Priority to order urgent work ahead of less urgent work.

## Benefits and trade-offs

A single Worker pool serves all tenants, so idle capacity from a low-traffic tenant is available to high-traffic tenants. New tenants require no Worker deployment. Add a Fairness key and Temporal starts dispatching their Tasks. Fairness weights can be updated through Task Queue configuration without redeploying application code.

Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, Worker Versioning, and short time windows. Accuracy can degrade with a large number of Fairness keys. Fairness weights apply when Tasks are scheduled, so changing a weight does not reorder Tasks already in the backlog. Tasks with different runtimes can consume different amounts of Worker capacity even when their dispatch shares match their weights.

## Comparison with alternatives

| Approach | Per-tenant dispatch | Dynamic tenants | Shares idle capacity | Complexity |
| :--- | :--- | :--- | :--- | :--- |
| Temporal Fairness (native) | Weighted fair dispatch | Yes | Yes | Low |
| Dedicated Task Queue per tenant | Separate | No | No | Medium |
| Single shared Task Queue (no control) | None | Yes | Yes | Lowest |
| External queue with per-tenant consumer groups | Separate | Yes | No | High |

## Best practices

- **Use stable, consistent Fairness keys.** Use account identifiers or tenant slugs instead of display names. Key changes do not reorder Tasks already in the backlog.
- **Combine Priority and Fairness for multi-class, multi-tenant workloads.** Priority separates urgent work from batch work. Fairness prevents a single tenant from dominating within each Priority level.

## Common pitfalls

- **Expecting Fairness to reorder the existing backlog.** Fairness weight is evaluated at schedule time. When Fairness is enabled for a Namespace with an existing backlog, that backlog drains in its original order before fairness-aware dispatch applies to new Tasks.
- **Using Fairness as a hard rate limiter.** Fairness by itself doesn't cap throughput. Use [Fairness key rate limits](/develop/task-queue-priority-fairness#set-rate-limits-at-the-task-queue-level) for per-key limits.
- **Unkeyed Tasks bypassing Fairness.** Tasks without a Fairness key are grouped under an implicit empty-string key and participate in weighted fair dispatch alongside named Fairness keys with a default weight of 1.0. They do not bypass Fairness and compete as one group.
- **Task Queue partitioning reducing accuracy.** Task Queues are internally partitioned, and Tasks are distributed to partitions randomly. This can interfere with fair dispatch proportions. If your workload requires higher accuracy, contact Temporal Support to configure a single-partition Task Queue.
- **Assuming Fairness applies across Worker Versioning boundaries.** Worker Deployment Versions have separate backlogs. Fairness applies within each version's backlog.
- **Expecting consistent Fairness immediately after a server restart.** Fairness ordering is preserved across restarts for the most active keys. Less active keys may briefly dispatch new Tasks ahead of their existing backlog until ordering normalizes.
- **Expecting the running Task mix to immediately reflect fair dispatch.** Fairness governs which Task is dispatched next. It does not account for Tasks already running on Workers. A group with longer-running Tasks can consume more Worker time than its dispatch share suggests.

## Related

### Patterns

- **[Priority](/design-patterns/priority-task-queues)**: Order Task dispatches by urgency within the same Task Queue using a Priority key.
- **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap dispatch throughput to a downstream service with a Task Queue RPS setting.
- **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity.

---

# Fan-Out with Child Workflows

Source: https://docs.temporal.io/design-patterns/fanout-child-workflows

> Distributes a large record set across parallel Child Workflows for concurrent processing with automatic scaling.

> **ℹ️ TLDR:**
> Split your record set into fixed-size chunks and start **one child Workflow per chunk** so that each chunk's history stays within Temporal's limits. Use this when you want maximum concurrency with no rate control and you can pre-compute how many chunks you need before the job starts. Keep the number of in-flight children per parent well under the default limit of 2,000; use [Sliding Window](/design-patterns/sliding-window) or [Batch Iterator](/design-patterns/batch-iterator) for larger workloads.

## Overview

The Fan-Out pattern distributes a large record set across multiple independent child Workflows, each responsible for processing a fixed-size chunk. The parent Workflow assigns work by offset and length so that no record IDs need to be passed over the wire — only two integers per child.

## Problem

A single Workflow run can have at most 2,000 in-flight Activities (aim for 500) and at most 51,200 history events. Processing millions of records in a single Workflow run is therefore not possible.

You need a way to partition a large record set, process each partition independently, and coordinate the overall job while keeping each Workflow's history within safe bounds.

## Solution

You split the total record count into fixed-size chunks and start one child Workflow per chunk. Each child is given an `offset` and a `length` so it knows which slice of the record set to fetch and process independently.

The parent Workflow starts all children concurrently and waits for them all to complete. If a child fails the parent can retry that child without re-processing the records handled by other children.

```mermaid
flowchart TD
    Records["📋 Total record set\n(N records)"]
    Parent["Parent Workflow\n(fanOutWorkflow)"]
    C1["Child Workflow\n(offset=0, length=chunk)"]
    C2["Child Workflow\n(offset=chunk, length=chunk)"]
    C3["Child Workflow\n(offset=2×chunk, length=chunk)"]

    Records --> Parent
    Parent -->|"start child 1"| C1
    Parent -->|"start child 2"| C2
    Parent -->|"start child 3"| C3

    C1 --> A1["processRecord ×chunk"]
    C2 --> A2["processRecord ×chunk"]
    C3 --> A3["processRecord ×chunk"]

    A1 -->|"done"| Parent
    A2 -->|"done"| Parent
    A3 -->|"done"| Parent
```

The following describes each step in the diagram:

1. The parent Workflow receives the total record count and a configured chunk size.
2. It divides the total into chunks and starts one child Workflow per chunk, passing only `offset` and `length`.
3. Each child independently fetches its slice of records (using the offset and length) and calls `processRecord` for each one.
4. Each child completes and returns its result to the parent.
5. The parent blocks until all children have completed, then returns the aggregated result.

## Implementation

The following examples show how each SDK implements the Fan-Out pattern.

**TypeScript**

```typescript
// workflows.ts
import {
  executeChild,
  proxyActivities,
  workflowInfo,
} from "@temporalio/workflow";
import type * as activities from "./activities";
import { TASK_QUEUE, CHUNK_SIZE } from "./shared";

const { processRecord } = proxyActivities<typeof activities>({
  startToCloseTimeout: "10 seconds",
});

export async function fanOutWorkflow(
  totalRecords: number,
  chunkSize: number = CHUNK_SIZE
): Promise<number> {
  const children: Promise<number>[] = [];

  for (let offset = 0; offset < totalRecords; offset += chunkSize) {
    const length = Math.min(chunkSize, totalRecords - offset);
    children.push(
      executeChild(recordBatchWorkflow, {
        args: [offset, length],
        taskQueue: TASK_QUEUE,
        workflowId: `${workflowInfo().workflowId}/batch-${offset}`,
      })
    );
  }

  const results = await Promise.all(children);
  return results.reduce((sum, n) => sum + n, 0);
}

export async function recordBatchWorkflow(
  offset: number,
  length: number
): Promise<number> {
  let processed = 0;
  for (let i = offset; i < offset + length; i++) {
    await processRecord(i);
    processed++;
  }
  return processed;
}
```

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.workflow import ChildWorkflowHandle
import asyncio
from activities import process_record
from shared import TASK_QUEUE, CHUNK_SIZE

@workflow.defn
class RecordBatchWorkflow:
    @workflow.run
    async def run(self, offset: int, length: int) -> int:
        processed = 0
        for i in range(offset, offset + length):
            await workflow.execute_activity(
                process_record,
                i,
                start_to_close_timeout=timedelta(seconds=10),
            )
            processed += 1
        return processed

@workflow.defn
class FanOutWorkflow:
    @workflow.run
    async def run(self, total_records: int, chunk_size: int = CHUNK_SIZE) -> int:
        handles: list[ChildWorkflowHandle] = []
        parent_id = workflow.info().workflow_id

        offset = 0
        while offset < total_records:
            length = min(chunk_size, total_records - offset)
            handle = await workflow.start_child_workflow(
                RecordBatchWorkflow.run,
                args=[offset, length],
                id=f"{parent_id}/batch-{offset}",
                task_queue=TASK_QUEUE,
            )
            handles.append(handle)
            offset += chunk_size

        results = await asyncio.gather(*handles)
        return sum(results)
```

**Go**

```go
// workflows.go
package main

import (
	"fmt"
	"time"

	"go.temporal.io/sdk/workflow"
)

func FanOutWorkflow(ctx workflow.Context, totalRecords int, chunkSize int) (int, error) {
	if chunkSize <= 0 {
		chunkSize = ChunkSize
	}

	var futures []workflow.Future
	parentID := workflow.GetInfo(ctx).WorkflowExecution.ID

	for offset := 0; offset < totalRecords; offset += chunkSize {
		length := chunkSize
		if offset+chunkSize > totalRecords {
			length = totalRecords - offset
		}
		off := offset // capture loop variable
		cwo := workflow.ChildWorkflowOptions{
			WorkflowID: parentID + "/batch-" + fmt.Sprintf("%d", off),
			TaskQueue:  TaskQueue,
		}
		cctx := workflow.WithChildOptions(ctx, cwo)
		futures = append(futures, workflow.ExecuteChildWorkflow(cctx, RecordBatchWorkflow, off, length))
	}

	total := 0
	for _, f := range futures {
		var n int
		if err := f.Get(ctx, &n); err != nil {
			return total, err
		}
		total += n
	}
	return total, nil
}

func RecordBatchWorkflow(ctx workflow.Context, offset int, length int) (int, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	processed := 0
	for i := offset; i < offset+length; i++ {
		if err := workflow.ExecuteActivity(ctx, ProcessRecord, i).Get(ctx, nil); err != nil {
			return processed, err
		}
		processed++
	}
	return processed, nil
}
```

**Java**

```java
// FanOutWorkflow.java
import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.*;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;

@WorkflowInterface
public interface FanOutWorkflow {
    @WorkflowMethod
    int run(int totalRecords, int chunkSize);
}

// FanOutWorkflowImpl.java
public class FanOutWorkflowImpl implements FanOutWorkflow {
    @Override
    public int run(int totalRecords, int chunkSize) {
        if (chunkSize <= 0) chunkSize = Shared.CHUNK_SIZE;

        List<Promise<Integer>> promises = new ArrayList<>();
        String parentId = Workflow.getInfo().getWorkflowId();

        for (int offset = 0; offset < totalRecords; offset += chunkSize) {
            int length = Math.min(chunkSize, totalRecords - offset);
            ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder()
                .setWorkflowId(parentId + "/batch-" + offset)
                .setTaskQueue(Shared.TASK_QUEUE)
                .build();
            RecordBatchWorkflow child = Workflow.newChildWorkflowStub(RecordBatchWorkflow.class, opts);
            promises.add(Async.function(child::run, offset, length));
        }

        int total = 0;
        for (Promise<Integer> p : promises) {
            total += p.get();
        }
        return total;
    }
}

// RecordBatchWorkflow.java
@WorkflowInterface
public interface RecordBatchWorkflow {
    @WorkflowMethod
    int run(int offset, int length);
}

// RecordBatchWorkflowImpl.java
public class RecordBatchWorkflowImpl implements RecordBatchWorkflow {
    private final Activities activities = Workflow.newActivityStub(
        Activities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(10))
            .build()
    );

    @Override
    public int run(int offset, int length) {
        int processed = 0;
        for (int i = offset; i < offset + length; i++) {
            activities.processRecord(i);
            processed++;
        }
        return processed;
    }
}
```

## Best practices

- **Use offset and length, not explicit IDs.** Pass only two integers to each child rather than a full slice of IDs. The child fetches its own records. This keeps history events small.
- **Size chunks to stay under the Activity limit.** Each child Workflow can have at most 2,000 in-flight Activities. Aim for chunks of 500 records or fewer if each record maps to one Activity.
- **Cap concurrent children in the parent.** Starting thousands of child Workflows simultaneously puts pressure on the namespace. Consider batching child starts or using [Sliding Window](/design-patterns/sliding-window) if you need tighter concurrency control.
- **Set `PARENT_CLOSE_POLICY_ABANDON`** for fire-and-forget fan-outs where the parent does not need to collect results. With the default `TERMINATE` policy, cancelling or timing out the parent will terminate all in-flight children.
- **Give each child a deterministic Workflow ID** (`parentId/batch-<offset>`). This makes it safe to re-run the parent: Temporal deduplicates child starts by Workflow ID, so already-completed children are not re-executed.

## Common pitfalls

- **Starting too many children at once.** Each child start adds to the parent's history. Temporal enforces a default limit of 2,000 pending (in-flight) child Workflows per parent, but the recommended cap is lower still: a single parent should not spawn more than 1,000 Child Workflow Executions, since each one adds more history to the parent than an Activity would. See [Temporal guidance](/child-workflows#when-to-use-child-workflows). If you need more children, switch to [MapReduce Tree](/design-patterns/mapreduce-tree) or [Sliding Window](/design-patterns/sliding-window).
- **Passing large lists of IDs.** Workflow inputs are stored in event history. Passing millions of record IDs as a list will blow the history size limit. Use offset + length instead.
- **Ignoring child failures.** A failed child does not automatically fail the parent unless you await all results. Always await child handles and handle errors explicitly.

## Related

### Patterns

- [Child Workflows pattern](/design-patterns/child-workflows) — core concepts for parent/child Workflow coordination
- [Batch Iterator](/design-patterns/batch-iterator) — unbounded record sets with Continue-as-New pagination
- [Sliding Window](/design-patterns/sliding-window) — bounded concurrency with maximum throughput
- [Temporal limits reference](/evaluate/cloud/limits)

---

# Fast/Slow Retries

Source: https://docs.temporal.io/design-patterns/fast-slow-retries

> Retry fast with a short interval first, then shift to a slow, unlimited interval so the Workflow outlasts an extended downstream outage.

> **ℹ️ TLDR:**
> Orchestrate two retry phases in the Workflow: a fast phase with short intervals and bounded attempts for transient errors, followed by a slow phase with long intervals and unlimited retries for extended outages. **Use this when a single `RetryPolicy` should not cover both brief blips and hour-long outages or maintenance windows.**

## Overview

The Fast/Slow Retries pattern runs an Activity through two distinct retry phases: a fast phase with a short interval and bounded attempt count, followed by an unlimited slow phase with a long fixed interval managed by the Temporal Service.
Use it when transient errors are common and worth recovering from quickly, but the downstream system may also suffer extended outages that require patient, indefinite waiting.

## Problem

Conventional retry policies force you to choose between:

- **Low `MaximumAttempts`**: Recovers from transient errors quickly but abandons the request when the downstream system has a longer outage.
- **Unlimited or High `MaximumAttempts` with short interval**: Floods a degraded downstream system with retries and accumulates noisy failures in logs. Incurs processing cost with each attempt.
- **Long fixed interval with unlimited retries**: Recovers from outages eventually, but is too slow to recover from transient errors that would have resolved in seconds.

None of these options handles both scenarios well — a downstream system that sometimes has brief 503 errors *and* occasionally goes down for an extended maintenance window.

## Solution

Use the Workflow itself as a retry orchestrator across two phases:

**Phase 1 — Fast retries**: Execute the Activity with a short `InitialInterval` and a bounded `MaximumAttempts`. This phase recovers from transient errors within seconds or minutes.

**Phase 2 — Slow retries**: When the fast retry policy is exhausted, catch the `ActivityError` in the Workflow and execute the Activity again with a long `InitialInterval` and unlimited `MaximumAttempts`. The Temporal Service owns the slow retry management; the Workflow blocks until the Activity eventually succeeds.

Conventional retry libraries can't implement this two-phase design, because the retry orchestrator needs to survive across the entire fast-then-slow window — hours or days — without staying resident in a process. A Temporal Workflow persists its state between retries, so it can hold that phase transition durably.

```mermaid
flowchart TD
    Start([Workflow starts]) --> Phase1

    subgraph Phase1 [Phase 1 — Fast Retries]
        F1[Execute Activity\ninitialInterval=1s\nmaxAttempts=10] -->|Success| Done
        F1 -->|Failure| FCheck{Attempts\nexhausted?}
        FCheck -->|No| F1
        FCheck -->|Yes| Log[Log: switching to slow phase]
    end

    Log --> Phase2

    subgraph Phase2 [Phase 2 — Slow Retries, Unlimited]
        S1[Execute Activity\ninitialInterval=5m\nunlimited attempts] -->|Success| Done
        S1 -->|Failure| SWait[Temporal waits 5m\nthen retries]
        SWait --> S1
    end

    Done([Return result])
```

The following describes each step:

1. The Workflow first tries the Activity with a fast policy: 1-second initial interval and a maximum of 10 total attempts.
2. If the Activity succeeds during the fast phase, the Workflow returns the result immediately.
3. If all fast attempts are exhausted, the Workflow logs a warning and transitions to the slow phase.
4. In the slow phase, the Workflow executes the Activity with a 5-minute fixed interval and unlimited retries. The Temporal Service manages the wait between attempts.
5. When the Activity eventually succeeds — after the downstream system recovers — the Workflow returns the result.

## Implementation

### Two-phase workflow retry management

The key change between phases is the retry interval and attempt count.
In Phase 1, the Temporal Service manages a fast set of retries: short interval, bounded attempts.
In Phase 2, the Temporal Service manages a slow set of retries: long fixed interval, unlimited attempts.

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError
import activities

@workflow.defn
class FastSlowRetryWorkflow:
    @workflow.run
    async def run(self, request: str) -> str:
        # Phase 1: fast retries 
        fast_policy = RetryPolicy(
            initial_interval=timedelta(seconds=1),
            backoff_coefficient=1.5,
            maximum_interval=timedelta(seconds=30),
            maximum_attempts=10,
        )
        try:
            return await workflow.execute_activity(
                activities.call_downstream,
                request,
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=fast_policy,
            )
        except ActivityError:
            workflow.logger.warning(
                "Fast retries exhausted — switching to slow retry phase",
                extra={"request": request},
            )

            # Phase 2: slow retries 
            slow_policy = RetryPolicy(
                initial_interval=timedelta(minutes=5),
                backoff_coefficient=1.0,
            )
            return await workflow.execute_activity(
                activities.call_downstream,
                request,
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=slow_policy,
            )
```

**Go**

```go
// workflow.go
package downstream

import (
    "time"

    "go.temporal.io/sdk/temporal"
    "go.temporal.io/sdk/workflow"
)

func FastSlowRetryWorkflow(ctx workflow.Context, request string) (string, error) {
    log := workflow.GetLogger(ctx)

    // Phase 1: fast retries 
    fastCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 1.5,
            MaximumInterval:    30 * time.Second,
            MaximumAttempts:    10,
        },
    })

    var result string
    err := workflow.ExecuteActivity(fastCtx, CallDownstream, request).Get(fastCtx, &result)
    if err != nil {
        log.Warn("Fast retries exhausted — switching to slow retry phase",
            "request", request)

        // Phase 2: slow retries 
        slowCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
            StartToCloseTimeout: 30 * time.Second,
            RetryPolicy: &temporal.RetryPolicy{
                InitialInterval:    5 * time.Minute,
                BackoffCoefficient: 1.0,
                // MaximumAttempts defaults to 0 (unlimited)
            },
        })

        err = workflow.ExecuteActivity(slowCtx, CallDownstream, request).Get(slowCtx, &result)
    }
    return result, err
}
```

**Java**

```java
// FastSlowRetryWorkflowImpl.java
import io.temporal.activity.ActivityOptions;
import io.temporal.common.RetryOptions;
import io.temporal.failure.ActivityFailure;
import io.temporal.workflow.Workflow;
import java.time.Duration;

public class FastSlowRetryWorkflowImpl implements FastSlowRetryWorkflow {
    @Override
    public String run(String request) {
        // Phase 1: fast retries 
        DownstreamActivities fastActivities = Workflow.newActivityStub(
            DownstreamActivities.class,
            ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofSeconds(30))
                .setRetryOptions(RetryOptions.newBuilder()
                    .setInitialInterval(Duration.ofSeconds(1))
                    .setBackoffCoefficient(1.5)
                    .setMaximumInterval(Duration.ofSeconds(30))
                    .setMaximumAttempts(10)
                    .build())
                .build()
        );

        try {
            return fastActivities.callDownstream(request);
        } catch (ActivityFailure e) {
            Workflow.getLogger(getClass()).warn(
                "Fast retries exhausted — switching to slow retry phase: " + request
            );

            // Phase 2: slow retries 
            DownstreamActivities slowActivities = Workflow.newActivityStub(
                DownstreamActivities.class,
                ActivityOptions.newBuilder()
                    .setStartToCloseTimeout(Duration.ofSeconds(30))
                    .setRetryOptions(RetryOptions.newBuilder()
                        .setInitialInterval(Duration.ofMinutes(5))
                        .setBackoffCoefficient(1.0)
                        // setMaximumAttempts not set — defaults to unlimited
                        .build())
                    .build()
            );

            return slowActivities.callDownstream(request);
        }
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import type * as activities from './activities';

const fastDownstream = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '30s',
    retry: {
        initialInterval: '1s',
        backoffCoefficient: 1.5,
        maximumInterval: '30s',
        maximumAttempts: 10,
    },
});

const slowDownstream = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '30s',
    retry: {
        initialInterval: '5m',
        backoffCoefficient: 1,
        // maximumAttempts defaults to unlimited
    },
});

export async function fastSlowRetryWorkflow(request: string): Promise<string> {
    // Phase 1: fast retries 
    try {
        return await fastDownstream.callDownstream(request);
    } catch {
        wf.log.warn('Fast retries exhausted — switching to slow retry phase', { request });

        // Phase 2: slow retries 
        return await slowDownstream.callDownstream(request);
    }
}
```

## Tuning the phases

Adjust the phase parameters to match the characteristics of your downstream system:

| Parameter | Typical value | Rationale |
| :--- | :--- | :--- |
| Phase 1 `InitialInterval` | 1–5 seconds | Recover from transient errors within seconds |
| Phase 1 `BackoffCoefficient` | 1.5–2.0 | Spread retries to avoid overwhelming a briefly degraded system |
| Phase 1 `MaximumAttempts` | 5–20 | Enough attempts to cover a short transient period |
| Phase 2 `InitialInterval` | 1–15 minutes | Long enough to avoid hammering a down system; short enough to recover promptly |
| Phase 2 `BackoffCoefficient` | 1.0 | Keeps the interval fixed; the default 2.0 would exponentially increase delays between slow-phase attempts |

Phase 2 runs indefinitely by default.
If the business process has a maximum wait time, add a `ScheduleToCloseTimeout` or use a Workflow execution timeout to impose an outer bound.

## Best practices

- **Log the phase transition.** The transition from fast to slow is a meaningful signal that the downstream system may have a sustained problem. Log it with enough context — request identifier, attempt count, timestamp — to aid diagnosis.
- **Leave `MaximumAttempts` unset in Phase 2.** Omitting `MaximumAttempts` (or setting it to 0) gives the slow phase unlimited retries. The Temporal Service manages the wait between attempts via `InitialInterval`; the Workflow blocks until the Activity eventually succeeds.
- **Combine with Retry Alerting via Metrics.** Add a metric counter inside the Activity to surface slow-phase attempts to on-call teams. See [Retry Alerting via Metrics](/design-patterns/retry-metrics).

## Common pitfalls

- **Catching too broadly in Phase 1.** Catch `ActivityError` specifically. Catching all exceptions in Phase 1 may swallow errors that should propagate immediately (such as `CancelledError` in Python or a `PanicError` in Go).
- **Setting `MaximumAttempts` in Phase 2.** If you set a finite `MaximumAttempts` in the slow phase, it will eventually exhaust and propagate a failure to the Workflow. Only add a limit if the business process has a defined maximum wait time; in that case, pair it with a `ScheduleToCloseTimeout` to make the budget explicit.
- **Using exponential backoff in Phase 2.** The default `BackoffCoefficient` is 2.0, which doubles the interval with each attempt. Set `BackoffCoefficient=1.0` in the slow phase to keep the interval fixed and predictable.

## Related

### Patterns

- [Retry Alerting via Metrics](/design-patterns/retry-metrics): Emit a metric in the slow-phase Activity to surface sustained failures to on-call teams.
- [Delayed Retry](/design-patterns/delayed-retry): Override the retry interval per error type using `nextRetryDelay` on `ApplicationFailure`.
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns.

### References

- [Temporal Retry Policies](/encyclopedia/retry-policies)
- [Understanding Workflow Retries and Failures](https://community.temporal.io/t/understanding-workflow-retries-and-failures/122)
- [Failure Handling in Practice](https://temporal.io/blog/failure-handling-in-practice)

---

# Fixed Count of Retries

Source: https://docs.temporal.io/design-patterns/fixed-count-retries

> Cap the number of Activity retry attempts to control cost when each attempt consumes a paid or limited resource.

> **ℹ️ TLDR:**
> Set `MaximumAttempts` on the `RetryPolicy` to **cap how many times Temporal will attempt an Activity**. Use this when each attempt consumes a paid API call, a rate-limited token, or any scarce resource where unbounded retries translate directly to unbounded cost.

## Overview

The Fixed Count of Retries pattern caps the total number of Activity execution attempts by setting `MaximumAttempts` on the `RetryPolicy`.
Use it when each attempt consumes a paid API call, a rate-limited token, or any scarce resource where unbounded retries translate directly to unbounded cost.

## Problem

Temporal's default retry policy retries Activities indefinitely with exponential backoff.
This is appropriate for most infrastructure failures, but it creates problems when the Activity calls a paid third-party API:

- A credit card authorization that fails due to a transient network error will be retried dozens of times, each attempt charging a per-call fee.
- A generative AI API with a per-token pricing model will accumulate costs silently while the Workflow waits.
- A rate-limited partner API will exhaust its quota across all callers if one Workflow retries without bound.

Without a cap, a single stuck Workflow can generate costs that are orders of magnitude larger than the intended spend.

## Solution

Set `MaximumAttempts` on the `RetryPolicy` passed to the Activity call.
Temporal counts the initial attempt and each retry toward the limit.
When the limit is reached, Temporal stops retrying and delivers an `ActivityError` to the Workflow.
The Workflow can catch that error and decide whether to fail, alert, or escalate.

```mermaid
sequenceDiagram
    participant Workflow
    participant Temporal as Temporal Service
    participant API as Payment API

    Workflow->>Temporal: Schedule activity (MaximumAttempts=3)
    Temporal->>+API: Attempt 1
    API-->>-Temporal: Failure
    Note over Temporal: Retry 1 of 2
    Temporal->>+API: Attempt 2
    API-->>-Temporal: Failure
    Note over Temporal: Retry 2 of 2
    Temporal->>+API: Attempt 3
    API-->>-Temporal: Failure
    Note over Temporal: MaximumAttempts reached — no more retries
    Temporal-->>Workflow: ActivityError
    Workflow->>Workflow: Handle failure (alert, compensate, or escalate)
```

The following describes each step:

1. The Workflow schedules the Activity with a `RetryPolicy` that caps attempts at 3.
2. The Temporal Service executes the Activity. On failure, it schedules a retry.
3. After 3 total attempts (1 initial + 2 retries), Temporal delivers an `ActivityError` to the Workflow.
4. The Workflow catches the error and handles it — logging, compensating, or escalating — rather than accumulating further cost.

## Implementation

### Capping attempts

Set `maximum_attempts` (Python), `MaximumAttempts` (Go / Java), or `maximumAttempts` (TypeScript) on the retry policy.
The count includes the initial attempt, so `maximum_attempts=3` means one attempt plus two retries.

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError, RetryState
import activities

@workflow.defn
class PaymentWorkflow:
    @workflow.run
    async def run(self, order_id: str) -> str:
        try:
            return await workflow.execute_activity(
                activities.charge_payment_api,
                order_id,
                start_to_close_timeout=timedelta(seconds=10),
                retry_policy=RetryPolicy(maximum_attempts=3),
            )
        except ActivityError as e:
            if e.retry_state == RetryState.MAXIMUM_ATTEMPTS_REACHED:
                # All retries exhausted — handle the failure here.
                # Options: alert on-call, trigger a compensation activity, or escalate to a human.
                workflow.logger.error(
                    "Payment failed: all 3 attempts exhausted",
                    extra={"order_id": order_id},
                )
            raise
```

**Go**

```go
// workflow.go
package payments

import (
    "errors"
    "time"

    enumspb "go.temporal.io/api/enums/v1"
    "go.temporal.io/sdk/temporal"
    "go.temporal.io/sdk/workflow"
)

func PaymentWorkflow(ctx workflow.Context, orderID string) (string, error) {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 10 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            MaximumAttempts: 3,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var result string
    err := workflow.ExecuteActivity(ctx, ChargePaymentAPI, orderID).Get(ctx, &result)
    if err != nil {
        var actErr *temporal.ActivityError
        if errors.As(err, &actErr) && actErr.RetryState() == enumspb.RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED {
            // All retries exhausted — handle the failure here.
            // Options: alert on-call, trigger a compensation activity, or escalate to a human.
            workflow.GetLogger(ctx).Error("Payment failed: all 3 attempts exhausted",
                "orderID", orderID)
        }
        return "", err
    }
    return result, nil
}
```

**Java**

```java
// PaymentWorkflowImpl.java
import io.temporal.activity.ActivityOptions;
import io.temporal.api.enums.v1.RetryState;
import io.temporal.common.RetryOptions;
import io.temporal.failure.ActivityFailure;
import io.temporal.workflow.Workflow;
import java.time.Duration;

public class PaymentWorkflowImpl implements PaymentWorkflow {
    private final PaymentActivities activities = Workflow.newActivityStub(
        PaymentActivities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(10))
            .setRetryOptions(RetryOptions.newBuilder()
                .setMaximumAttempts(3)
                .build())
            .build()
    );

    @Override
    public String run(String orderId) {
        try {
            return activities.chargePaymentApi(orderId);
        } catch (ActivityFailure e) {
            if (e.getRetryState() == RetryState.RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED) {
                // All retries exhausted — handle the failure here.
                // Options: alert on-call, trigger a compensation activity, or escalate to a human.
                Workflow.getLogger(getClass()).error(
                    "Payment failed: all 3 attempts exhausted: " + orderId, e);
            }
            throw e;
        }
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import { RetryState } from '@temporalio/common';
import * as wf from '@temporalio/workflow';
import type * as activities from './activities';

const { chargePaymentApi } = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '10s',
    retry: { maximumAttempts: 3 },
});

export async function paymentWorkflow(orderId: string): Promise<string> {
    try {
        return await chargePaymentApi(orderId);
    } catch (err) {
        if (err instanceof wf.ActivityFailure && err.retryState === RetryState.MAXIMUM_ATTEMPTS_REACHED) {
            // All retries exhausted — handle the failure here.
            // Options: alert on-call, trigger a compensation activity, or escalate to a human.
            wf.log.error('Payment failed: all 3 attempts exhausted', { orderId });
        }
        throw err;
    }
}
```

### Disabling retries entirely

Set `maximum_attempts=1` to disable retries.
The Activity starts once and any failure is immediately delivered to the Workflow.
This is appropriate when the operation is not idempotent and a second attempt would cause a duplicate side effect such as a double charge or a duplicate email.

**Python**

```python
# workflows.py
result = await workflow.execute_activity(
    activities.send_welcome_email,
    user_id,
    start_to_close_timeout=timedelta(seconds=10),
    retry_policy=RetryPolicy(maximum_attempts=1),
)
```

**Go**

```go
// workflow.go
ao := workflow.ActivityOptions{
    StartToCloseTimeout: 10 * time.Second,
    RetryPolicy: &temporal.RetryPolicy{
        MaximumAttempts: 1,
    },
}
```

**Java**

```java
// Workflow.java
ActivityOptions.newBuilder()
    .setStartToCloseTimeout(Duration.ofSeconds(10))
    .setRetryOptions(RetryOptions.newBuilder()
        .setMaximumAttempts(1)
        .build())
    .build()
```

**TypeScript**

```typescript
// workflows.ts
const { sendWelcomeEmail } = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '10s',
    retry: { maximumAttempts: 1 },
});
```

If a Worker crashes after the API call succeeds but before the result is recorded, Temporal will not retry — the call is lost.
An idempotency key (a stable identifier derived from the Workflow and Activity IDs) lets the downstream system detect and discard duplicates if a retry is needed in future. If you have idempotency keys, there's little need to cap retries at 1.

## Best practices

- **Match the cap to the cost model.** If the API charges per call, set `maximum_attempts` to the maximum number of calls you are willing to pay for per Workflow execution.
- **Combine with `StartToCloseTimeout`.** A per-attempt timeout prevents a slow response from consuming the entire retry budget on a single hanging call.
- **Catch `ActivityError` in the Workflow.** Handle the exhausted-retries case explicitly — log, alert, compensate, or escalate — rather than letting it fail the Workflow silently.
- **Use idempotency keys.** When retrying, it's vital to have downstream systems detect and discard duplicate calls to avoid duplicate downstream effects.
- **Prefer non-retryable errors for structural failures.** If the failure is not transient (for example, invalid input), mark it as non-retryable rather than relying solely on `maximum_attempts`.

## Common pitfalls

- **Confusing `MaximumAttempts` with allowed retry count.** `MaximumAttempts=3` means 3 total attempts (1 initial + 2 retries), not 3 retries after the initial attempt.
- **Setting no timeout alongside a low attempt cap.** Without `StartToCloseTimeout`, a single hanging attempt can block all retries for minutes or hours.
- **Ignoring the `ActivityError` in the Workflow.** Exhausted retries raise an error in the Workflow. If you do not catch it, the Workflow fails without any compensation or alerting.
- **Disabling retries on operations without safeguards.** `maximum_attempts=1` on a call means any failure — including a Worker crash after the API responded — results in a permanent gap.

## Related

### Patterns

- [Non-Retryable Errors](/design-patterns/non-retryable-errors): Fail immediately for errors that will never succeed regardless of how many times you try.
- [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Bound by total elapsed time rather than attempt count.
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns.

### References

- [Temporal Retry Policies](/encyclopedia/retry-policies)
- [Idempotency and Durable Execution](https://temporal.io/blog/idempotency-and-durable-execution)

---

# Fixed Wall-Time Retries

Source: https://docs.temporal.io/design-patterns/fixed-wall-time-retries

> Bound the total elapsed time across all retry attempts to enforce a business SLA, regardless of how many individual attempts occur.

> **ℹ️ TLDR:**
> Set `ScheduleToCloseTimeout` on the Activity call to enforce a hard time budget across all retry attempts. Use this when a business SLA requires the Activity to **succeed or fail within a defined window**, regardless of how many individual attempts occur.

## Overview

The Fixed Wall-Time Retries pattern enforces a maximum total elapsed time across all Activity retry attempts using `ScheduleToCloseTimeout`.
Use it when a business process must _succeed or fail_ within a defined time budget, regardless of how many individual attempts occur.

## Problem

`StartToCloseTimeout` limits how long a single Activity attempt may run before Temporal cancels it and schedules a retry.
It does not limit how long retries collectively may run.

A process with `StartToCloseTimeout=5m` and the default unlimited retry policy can run for days — each attempt times out at 5 minutes, then Temporal waits for the backoff delay and tries again, indefinitely.

When a business SLA exists and violating that SLA is a failure such as a payment must charge in two minutes or less, an authorization check must complete within 30 seconds — you need a hard outer boundary that Temporal enforces automatically without requiring the Workflow to track elapsed time itself.

## Solution

Set `ScheduleToCloseTimeout` on the Activity call options.
It starts when the Activity is first scheduled and expires when the clock runs out, regardless of how many attempts have occurred.
When the timeout expires, the Temporal Service marks the Activity Execution as timed out and delivers an `ActivityError` to the Workflow.
No further retries are scheduled.
A timeout does not forcibly stop Activity code that is already running, so an Activity that runs past the budget must heartbeat and handle cancellation to stop cooperatively.

```mermaid
sequenceDiagram
    participant Workflow
    participant Temporal as Temporal Service
    participant Service as Downstream Service

    Note over Temporal: ScheduleToCloseTimeout = 2m starts
    Workflow->>Temporal: Schedule activity
    Temporal->>+Service: Attempt 1 (StartToClose = 30s)
    Service-->>-Temporal: Failure
    Note over Temporal: Backoff delay (5s)
    Temporal->>+Service: Attempt 2 (StartToClose = 30s)
    Service-->>-Temporal: Failure
    Note over Temporal: Backoff delay (5s), ...retries continue
    Note over Temporal: 2m elapsed — ScheduleToClose exceeded
    Temporal-->>Workflow: ActivityError (schedule-to-close timeout)
    Workflow->>Workflow: Handle SLA breach
```

The following describes each step:

1. The two minute budget clock starts the moment the Workflow schedules the Activity.
2. Each attempt runs up to 30 seconds (`StartToCloseTimeout`). On failure, Temporal waits the backoff delay and retries.
3. Retries continue until either the Activity succeeds or the two minute budget is exhausted.
4. When the budget expires, Temporal delivers an `ActivityError` to the Workflow, which can log, alert, or compensate.

## Implementation

### Enforcing a 2-minute SLA

Set both `schedule_to_close_timeout` (the total budget) and `start_to_close_timeout` (the per-attempt cap).
The retry policy controls the interval between attempts.
Temporal stops retrying automatically when the budget runs out.

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError, TimeoutError, TimeoutType
import activities

@workflow.defn
class PaymentAuthWorkflow:
    @workflow.run
    async def run(self, transaction_id: str) -> str:
        try:
            return await workflow.execute_activity(
                activities.authorize_transaction,
                transaction_id,
                schedule_to_close_timeout=timedelta(minutes=2),  # total budget
                start_to_close_timeout=timedelta(seconds=30),    # per attempt
                retry_policy=RetryPolicy(
                    initial_interval=timedelta(seconds=5),
                    backoff_coefficient=1.5,
                    maximum_interval=timedelta(seconds=30),
                ),
            )
        except ActivityError as e:
            cause = e.__cause__
            if isinstance(cause, TimeoutError) and cause.type == TimeoutType.SCHEDULE_TO_CLOSE:
                workflow.logger.error(
                    "Authorization failed — 2-minute SLA breached",
                    extra={"transaction_id": transaction_id},
                )
            raise
```

**Go**

```go
// workflow.go
package shipment

import (
    "errors"
    "time"

    enumspb "go.temporal.io/api/enums/v1"
    "go.temporal.io/sdk/temporal"
    "go.temporal.io/sdk/workflow"
)

func PaymentAuthWorkflow(ctx workflow.Context, transactionID string) (string, error) {
    ao := workflow.ActivityOptions{
        ScheduleToCloseTimeout: 2 * time.Minute,   // total budget
        StartToCloseTimeout:    30 * time.Second,  // per attempt
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    5 * time.Second,
            BackoffCoefficient: 1.5,
            MaximumInterval:    30 * time.Second,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var result string
    err := workflow.ExecuteActivity(ctx, AuthorizeTransaction, transactionID).Get(ctx, &result)
    if err != nil {
        var timeoutErr *temporal.TimeoutError
        if errors.As(err, &timeoutErr) && timeoutErr.TimeoutType() == enumspb.TIMEOUT_TYPE_SCHEDULE_TO_CLOSE {
            workflow.GetLogger(ctx).Error(
                "Authorization failed — 2-minute SLA breached",
                "transactionID", transactionID,
            )
        }
        return "", err
    }
    return result, nil
}
```

**Java**

```java
// ShipmentNotificationWorkflowImpl.java
import io.temporal.activity.ActivityOptions;
import io.temporal.api.enums.v1.TimeoutType;
import io.temporal.common.RetryOptions;
import io.temporal.failure.ActivityFailure;
import io.temporal.failure.TimeoutFailure;
import io.temporal.workflow.Workflow;
import java.time.Duration;

public class PaymentAuthWorkflowImpl implements PaymentAuthWorkflow {
    private final PaymentActivities activities = Workflow.newActivityStub(
        PaymentActivities.class,
        ActivityOptions.newBuilder()
            .setScheduleToCloseTimeout(Duration.ofMinutes(2))   // total budget
            .setStartToCloseTimeout(Duration.ofSeconds(30))     // per attempt
            .setRetryOptions(RetryOptions.newBuilder()
                .setInitialInterval(Duration.ofSeconds(5))
                .setBackoffCoefficient(1.5)
                .setMaximumInterval(Duration.ofSeconds(30))
                .build())
            .build()
    );

    @Override
    public String run(String transactionId) {
        try {
            return activities.authorizeTransaction(transactionId);
        } catch (ActivityFailure e) {
            if (e.getCause() instanceof TimeoutFailure tf
                    && tf.getTimeoutType() == TimeoutType.TIMEOUT_TYPE_SCHEDULE_TO_CLOSE) {
                Workflow.getLogger(getClass()).error(
                    "Authorization failed — 2-minute SLA breached: " + transactionId, e
                );
            }
            throw e;
        }
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import { TimeoutType } from '@temporalio/common';
import * as wf from '@temporalio/workflow';
import type * as activities from './activities';

const { authorizeTransaction } = wf.proxyActivities<typeof activities>({
    scheduleToCloseTimeout: '2m',   // total budget
    startToCloseTimeout: '30s',     // per attempt
    retry: {
        initialInterval: '5s',
        backoffCoefficient: 1.5,
        maximumInterval: '30s',
    },
});

export async function paymentAuthWorkflow(transactionId: string): Promise<string> {
    try {
        return await authorizeTransaction(transactionId);
    } catch (err) {
        if (err instanceof wf.ActivityFailure) {
            const cause = err.cause;
            if (cause instanceof wf.TimeoutFailure && cause.timeoutType === TimeoutType.SCHEDULE_TO_CLOSE) {
                wf.log.error('Authorization failed — 2-minute SLA breached', { transactionId });
            }
        }
        throw err;
    }
}
```

### Short SLA without a per-attempt timeout

For tighter budgets — such as a 30 second authorization window — you may omit `StartToCloseTimeout` and let `ScheduleToCloseTimeout` act as the only bound. 
Temporal requires at least one timeout to be set; `ScheduleToCloseTimeout` alone satisfies that requirement.

**Python**

```python
# workflows.py
result = await workflow.execute_activity(
    activities.authorize_transaction,
    transaction_id,
    schedule_to_close_timeout=timedelta(seconds=30),
    retry_policy=RetryPolicy(
        initial_interval=timedelta(seconds=3),
        backoff_coefficient=1.5,
    ),
)
```

**Go**

```go
// workflow.go
ao := workflow.ActivityOptions{
    ScheduleToCloseTimeout: 30 * time.Second,
    RetryPolicy: &temporal.RetryPolicy{
        InitialInterval:    3 * time.Second,
        BackoffCoefficient: 1.5,
    },
}
```

**Java**

```java
// Workflow.java
ActivityOptions.newBuilder()
    .setScheduleToCloseTimeout(Duration.ofSeconds(30))
    .setRetryOptions(RetryOptions.newBuilder()
        .setInitialInterval(Duration.ofSeconds(3))
        .setBackoffCoefficient(1.5)
        .build())
    .build()
```

**TypeScript**

```typescript
// workflows.ts
const { authorizeTransaction } = wf.proxyActivities<typeof activities>({
    scheduleToCloseTimeout: '30s',
    retry: {
        initialInterval: '3s',
        backoffCoefficient: 1.5,
    },
});
```

## Best practices

- **Set both timeouts for clarity.** Use `ScheduleToCloseTimeout` as the total SLA and `StartToCloseTimeout` as a per-attempt safety valve. Omitting `StartToCloseTimeout` means a single slow response can consume the entire budget.
- **Cap `MaximumInterval` well below the SLA.** If `MaximumInterval` is 2 hours and the SLA is 24 hours, only 12 retries are possible. Tune the interval so the backoff plateaus at a value that allows meaningful retries within the budget.
- **Handle `ActivityError` explicitly.** When the SLA expires, Temporal delivers an error to the Workflow. Catch it to send an alert, trigger a compensation, or record a breach in an audit log.
- **Distinguish SLA breaches from transient errors.** Inspect the error cause — check that the `ActivityError`'s cause is a `TimeoutError` with `TimeoutType.SCHEDULE_TO_CLOSE` (Python) or a `TimeoutFailure` with `TimeoutType.SCHEDULE_TO_CLOSE` (TypeScript) or `TIMEOUT_TYPE_SCHEDULE_TO_CLOSE` (Go/Java) to separate an SLA breach from an application failure. This lets you log or alert specifically on SLA violations rather than treating all activity errors the same way.

## Common pitfalls

- **Not accounting for `ScheduleToStart` delay in the budget.** `ScheduleToCloseTimeout` begins when the Activity is first scheduled, which includes the time the task waits in the queue before a Worker picks it up. Under high load or insufficient Worker capacity, tasks can sit in the queue for seconds or minutes before the first attempt starts — consuming SLA budget before any work is done. Provision Workers with enough capacity for peak traffic, or use autoscaling, to keep `ScheduleToStart` latency negligible relative to the SLA window.
- **Using `StartToCloseTimeout` alone for SLA enforcement.** A downstream system that responds slowly but never fully times out can keep resetting the per-attempt clock indefinitely.
- **Setting `ScheduleToCloseTimeout` shorter than `StartToCloseTimeout`.** If the total budget is shorter than a single attempt's maximum, the first attempt cannot finish within the budget — the Temporal Service times out the Activity Execution and returns an error before any attempt can succeed.
- **Ignoring the breach in the Workflow.** Letting the `ActivityError` propagate without handling it means SLA breaches go unlogged and uncompensated.
- **Not accounting for backoff delays in the budget.** The total time includes both attempt durations and the backoff delays between them. A 1-hour budget with a 30-minute initial interval and coefficient 2.0 leaves room for only one or two attempts.

## Related

### Patterns

- [Fixed Count of Retries](/design-patterns/fixed-count-retries): Bound by attempt count rather than elapsed time.
- [Delayed Retry](/design-patterns/delayed-retry): Fixed-interval retry when the downstream unavailability window is known.
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns.

### References

- [Detecting Activity Failures](/encyclopedia/detecting-activity-failures): Canonical reference for `ScheduleToCloseTimeout` and `StartToCloseTimeout`, the mechanism this pattern is built on.
- [Activity Timeouts](https://temporal.io/blog/activity-timeouts)
- [Temporal Retry Policies](/encyclopedia/retry-policies)

---

# Local Activities

Source: https://docs.temporal.io/design-patterns/local-activities

> Local Activities run inside the Worker process, skipping server round-trips for short, idempotent Activities on a latency-sensitive path.

> **ℹ️ TLDR:**
> **Replace regular Activities with Local Activities to eliminate Temporal server round-trips and reduce per-Activity overhead to near zero.** This is best for short-lived, idempotent Activities that complete well within the Workflow Task timeout. Each Activity you convert saves approximately 50 ms of scheduling overhead on Temporal Cloud. If you haven't measured a latency problem, start with regular Activities—they are easier to debug, rate-limit, and monitor.

## Overview

A **Local Activity** executes the Activity function directly inside the Worker process that is currently running the Workflow Task. The result is recorded as part of the same Workflow Task completion event, so no additional server calls occur between Activity invocations.

```mermaid
sequenceDiagram
    participant W as Worker
    participant S as Temporal Server

    rect rgb(230, 235, 250)
        Note over W,S: Regular Activity —<br/>server round-trips on each call
        W->>S: Schedule ActivityTask
        S-->>W: Dispatch ActivityTask
        W->>W: Execute activity
        W->>S: Complete ActivityTask
        S-->>W: Resume WorkflowTask
    end

    rect rgb(220, 245, 225)
        Note over W,S: Local Activity —<br/>runs in-process, zero server calls
        W->>W: Execute activity function directly
        Note over W: Result bundled in WorkflowTask completion
    end
```

**Numbered walkthrough:**

1. A regular Activity follows a five-step exchange with the Temporal server: schedule, dispatch, execute, complete, then resume the Workflow Task. On Temporal Cloud, each round-trip adds approximately 50 ms.
2. A Local Activity bypasses all of that. The Workflow Task scheduler on the Worker invokes the Activity function directly. The result is folded into the WorkflowTask completion event sent at the end of the task.
3. For a Workflow with three serial Activities, switching all three to Local Activities can save 150 ms or more while keeping the exact same business logic.

## Problem

Workflows that call several short Activities in sequence accumulate significant latency from server scheduling overhead. A validation-and-reservation flow with three Activities may complete in 850 ms even when the actual computation takes only milliseconds. Every regular Activity incurs at least two server round-trips (schedule + complete), which becomes a bottleneck on latency-sensitive paths.

## Solution

Use `workflow.execute_local_activity` (Python), `proxyLocalActivities` (TypeScript), `workflow.ExecuteLocalActivity` (Go), or `Workflow.newLocalActivityStub` with `LocalActivityOptions` (Java) to run Activity functions in-process. The Activity executes inside the same Workflow Task and its result is available immediately—no scheduling overhead.

Local Activities are subject to the Workflow Task timeout (default 10 seconds) rather than an independent start-to-close timeout. Always configure the `scheduleToCloseTimeout` option (not `startToCloseTimeout`) to set an upper bound.

**Python**

```python
# workflows.py
from temporalio import workflow
from datetime import timedelta
from activities import validate_transaction, reserve_funds, settle_transaction

LOCAL_ACTIVITY_TIMEOUT = timedelta(seconds=10)

@workflow.defn
class TransactionWorkflow:
    @workflow.run
    async def run(self, req: TransactionRequest) -> Transaction:
        # All three activities run in-process — no server round-trips.
        tx = await workflow.execute_local_activity(
            validate_transaction, req,
            schedule_to_close_timeout=LOCAL_ACTIVITY_TIMEOUT,
        )
        tx = await workflow.execute_local_activity(
            reserve_funds, tx,
            schedule_to_close_timeout=LOCAL_ACTIVITY_TIMEOUT,
        )
        return await workflow.execute_local_activity(
            settle_transaction, tx,
            schedule_to_close_timeout=LOCAL_ACTIVITY_TIMEOUT,
        )
```

**TypeScript**

```typescript
// workflows.ts
import { proxyLocalActivities } from "@temporalio/workflow";
import type * as activities from "./activities";

const { validateTransaction, reserveFunds, settleTransaction } =
  proxyLocalActivities<typeof activities>({ scheduleToCloseTimeout: "10s" });

export async function transactionWorkflow(
  req: TransactionRequest,
): Promise<Transaction> {
  // All three activities run in-process — no server round-trips.
  let tx = await validateTransaction(req);
  tx = await reserveFunds(tx);
  return settleTransaction(tx);
}
```

**Go**

```go
// workflows.go
func TransactionWorkflow(ctx workflow.Context, req TransactionRequest) (Transaction, error) {
    localCtx := workflow.WithLocalActivityOptions(ctx, workflow.LocalActivityOptions{
        ScheduleToCloseTimeout: 10 * time.Second,
    })

    // All three activities run in-process — no server round-trips.
    var tx Transaction
    if err := workflow.ExecuteLocalActivity(localCtx, ValidateTransaction, req).Get(localCtx, &tx); err != nil {
        return Transaction{}, err
    }
    if err := workflow.ExecuteLocalActivity(localCtx, ReserveFunds, tx).Get(localCtx, &tx); err != nil {
        return Transaction{}, err
    }
    if err := workflow.ExecuteLocalActivity(localCtx, SettleTransaction, tx).Get(localCtx, &tx); err != nil {
        return Transaction{}, err
    }
    return tx, nil
}
```

**Java**

```java
// TransactionWorkflow.java
public class Impl implements TransactionWorkflow {
    // All activities run as local — no server round-trips.
    private final Activities activities = Workflow.newLocalActivityStub(
        Activities.class,
        LocalActivityOptions.newBuilder()
            .setScheduleToCloseTimeout(Duration.ofSeconds(10))
            .build()
    );

    @Override
    public Shared.Transaction processTransaction(Shared.TransactionRequest req) {
        Shared.Transaction tx = activities.validateTransaction(req);
        tx = activities.reserveFunds(tx);
        return activities.settleTransaction(tx);
    }
}
```

## When to use

**Good fit:**

- Short-lived Activities that complete in milliseconds or a few seconds
- Idempotent operations safe to re-execute if a Workflow Task fails
- Hot-path Workflows where end-to-end latency is a product requirement
- CPU-bound or in-memory computations that do not need a separate worker pool

**Poor fit:**

- Activities that may run longer than the Workflow Task timeout (default 10 seconds)
- Activities that require heartbeating to detect stuck executions
- Operations with long retry back-off intervals—retry timers still schedule server events, reducing the latency benefit
- Non-idempotent operations where re-execution on Worker crash would cause harm
- Operations that need rate limiting or routing through task queue capacity controls—Local Activities bypass the task queue entirely and cannot be throttled by the server

## Benefits and trade-offs

| | Regular Activity | Local Activity |
|---|---|---|
| Server round-trips | 2–4 per call | 0 |
| Latency overhead | ~50 ms per call (Temporal Cloud) | Near zero |
| Heartbeat support | Yes | No |
| Execution timeout | `StartToCloseTimeout` | `ScheduleToCloseTimeout` |
| Retry semantics | Independent per attempt | Entire Workflow Task re-executes on failure |
| Dedicated worker pool | Yes (separate poller) | No (shares Workflow Task thread) |
| Rate limiting / routing | Yes (task queue capacity) | No (bypasses task queue) |
| Visible in Temporal UI | Full Activity history event | Recorded in Workflow Task event; no standalone Activity task in UI |

## Best practices

- **Design for at-least-once execution.** If a Workflow Task fails after a Local Activity completes but before the task is persisted, all Local Activities in that task re-execute on the next attempt. Your Activity logic must tolerate this.
- **Keep each Local Activity short.** Aim for well under 5 seconds to leave headroom for retries within the same Workflow Task, which has a 10-second default timeout.
- **Avoid blocking signal and update handlers.** While a Local Activity executes, the Workflow Task is occupied. Incoming signals and updates accumulate in the server buffer and are not processed until the next task begins.
- **Set retry policy carefully.** Large `initialInterval` or `maximumInterval` values in a retry policy still cause the SDK to schedule server-side timer events, which partially defeats the latency benefit.

## Common pitfalls

- **Exceeding the Workflow Task timeout.** If a Local Activity takes longer than the Workflow Task timeout (default 10 seconds) and Workflow Task heartbeating doesn't cover it, the entire task times out and retries—including any Local Activities that already completed in memory during that task. The SDK mitigates this automatically: once a running Local Activity passes about 80% of the Workflow Task timeout, the Worker heartbeats by completing the current Workflow Task and requesting a new one, so a single long Local Activity (or a chain of them) can run past one Workflow Task's timeout without the whole task failing. Heartbeating adds Events to history and delays processing of incoming Signals until the Local Activity finishes — see [Local Activity](/local-activity#workflow-task-heartbeating) for the full mechanism.
- **Assuming exactly-once semantics.** Unlike regular Activities, a Local Activity does not get its own persisted history event until the Workflow Task completes. A crashed Worker causes the whole task to re-run. This compounds when Local Activities are chained: if a Worker crashes after the third of five sequential Local Activities, all five re-execute on the next attempt. If you need a durable checkpoint between each step, use regular Activities instead.
- **Long retry intervals.** Each retry attempt with back-off creates a server-side timer event. For truly short Activities, use a tight `scheduleToCloseTimeout` and allow immediate retries rather than spaced-out back-off.

## Related

### Patterns

- [Early Return + Local Activities](/design-patterns/early-return-local-activities) — adds an Update-with-Start early-response path on top of Local Activities for minimum first-response latency
- [Early Return](/design-patterns/early-return) — returns a response to the caller before the Workflow finishes, independent of Local Activities
- [Eager Workflow Start](/design-patterns/eager-workflow-start) — eliminates the server Matching step when starting a Workflow for additional latency reduction
- [Long Running Activity](/design-patterns/long-running-activity) — the right choice when Activities need heartbeating and long execution windows

### References

- [Local Activity](/local-activity) — canonical concept reference, including durability guarantees and Workflow Task heartbeating

---

# Long-Running Activity - Tracking Progress and Handling Cancellation with Heartbeats

Source: https://docs.temporal.io/design-patterns/long-running-activity

> Long-running Activities report progress via heartbeats and enable resumption after failures with cancellation support.

## Overview

The Activity Heartbeat pattern enables long-running Activities to report progress, handle cancellation gracefully, and resume from the last checkpoint after failures.
Heartbeats inform Temporal that the Activity is still alive and allow storing progress details that survive Worker restarts.

## Problem

Activities that process large datasets or run for minutes to hours need a way to report progress — so they don't appear stuck or time out — and to resume from a checkpoint rather than restart from scratch after a Worker crash. They also need to handle cancellation gracefully and avoid redoing work that already finished.

Without heartbeats, none of that is possible. You're left setting very long Activity timeouts that delay failure detection, reprocessing entire batches from the beginning on any failure, and building custom checkpointing and recovery logic yourself — with no visibility into whether an Activity is actually making progress or just a zombie that looks alive.

## Solution

Activity heartbeats periodically report progress to the Temporal Service.
The heartbeat details are persisted and available to retry attempts, enabling resumption from the last checkpoint.
Heartbeat timeouts detect stuck Activities faster than execution timeouts.

```mermaid
sequenceDiagram
    participant Workflow
    participant Activity
    participant Temporal

    Workflow->>+Activity: Start (with heartbeat timeout)
    loop Process items
        Activity->>Activity: Process item
        Activity->>Temporal: heartbeat(progress)
        Note over Temporal: Store progress
    end

    alt Activity completes
        Activity-->>-Workflow: Result
    else Worker crashes
        Note over Activity: Heartbeat timeout expires
        Temporal->>+Activity: Retry on new worker
        Activity->>Temporal: getHeartbeatDetails()
        Temporal-->>Activity: Last progress
        Activity->>Activity: Resume from checkpoint
        Activity-->>-Workflow: Result
    end
```

The following describes each step in the diagram:

1. The Workflow starts the Activity with a heartbeat timeout.
2. The Activity processes items in a loop, heartbeating progress after each batch.
3. If the Activity completes normally, it returns the result to the Workflow.
4. If the Worker crashes, the heartbeat timeout expires and Temporal retries the Activity on a new Worker. The new attempt retrieves the last heartbeat details and resumes from the checkpoint.

## Implementation

### Basic progress tracking

The following implementation processes a large file line by line, heartbeating every 100 lines.
On retry, it retrieves the last processed line number and skips ahead:

**Python**

```python
# activities.py
from temporalio import activity

@activity.defn
async def process_large_file(file_path: str) -> None:
    details = activity.info().heartbeat_details
    start_line = details[0] if details else 0

    with open(file_path, "r") as f:
        for i, line in enumerate(f):
            if i < start_line:
                continue

            process_line(line)

            if (i + 1) % 100 == 0:
                activity.heartbeat(i + 1)
```

**Go**

```go
// activities.go
func ProcessLargeFile(ctx context.Context, filePath string) error {
	startLine := 0
	if activity.HasHeartbeatDetails(ctx) {
		if err := activity.GetHeartbeatDetails(ctx, &startLine); err != nil {
			return err
		}
	}

	file, err := os.Open(filePath)
	if err != nil {
		return err
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	currentLine := 0
	for scanner.Scan() {
		if currentLine < startLine {
			currentLine++
			continue
		}

		processLine(scanner.Text())
		currentLine++

		if currentLine%100 == 0 {
			activity.RecordHeartbeat(ctx, currentLine)
		}
	}
	return scanner.Err()
}
```

**Java**

```java
// FileProcessingActivityImpl.java
@ActivityInterface
public interface FileProcessingActivity {
  void processLargeFile(String filePath);
}

public class FileProcessingActivityImpl implements FileProcessingActivity {
  @Override
  public void processLargeFile(String filePath) {
    ActivityExecutionContext context = Activity.getExecutionContext();
    Optional<Integer> lastProcessedLine = context.getHeartbeatDetails(Integer.class);
    int startLine = lastProcessedLine.orElse(0);

    try (BufferedReader reader = new BufferedReader(new FileReader(filePath))) {
      for (int i = 0; i < startLine; i++) {
        reader.readLine();
      }

      String line;
      int currentLine = startLine;
      while ((line = reader.readLine()) != null) {
        processLine(line);
        currentLine++;

        if (currentLine % 100 == 0) {
          context.heartbeat(currentLine);
        }
      }
    }
  }
}
```

**TypeScript**

```typescript
// activities.ts
import { heartbeat, activityInfo } from '@temporalio/activity';
import { createReadStream } from 'fs';
import { createInterface } from 'readline';

export async function processLargeFile(filePath: string): Promise<void> {
  const startLine = activityInfo().heartbeatDetails ?? 0;

  const rl = createInterface({
    input: createReadStream(filePath),
  });

  let currentLine = 0;
  for await (const line of rl) {
    if (currentLine < startLine) {
      currentLine++;
      continue;
    }

    processLine(line);
    currentLine++;

    if (currentLine % 100 === 0) {
      heartbeat(currentLine);
    }
  }
}
```

The heartbeat details call retrieves the last heartbeat value from a previous attempt.
If this is the first attempt, there are no details and the Activity starts from line 0.
The Activity heartbeats every 100 lines, storing the current line number as the checkpoint.

### Handle cancellation

The following implementation adds cancellation support.
The Activity checks for cancellation on each heartbeat and cleans up resources before exiting:

**Python**

```python
# activities.py
import asyncio
from temporalio import activity

@activity.defn
async def process_large_file(file_path: str) -> None:
    details = activity.info().heartbeat_details
    current_line = details[0] if details else 0

    try:
        with open(file_path, "r") as f:
            for i, line in enumerate(f):
                if i < current_line:
                    continue

                activity.heartbeat(i)
                process_line(line)
                current_line = i + 1
    except asyncio.CancelledError:
        cleanup_resources()
        raise
```

**Go**

```go
// activities.go
func ProcessLargeFile(ctx context.Context, filePath string) error {
	currentLine := 0
	if activity.HasHeartbeatDetails(ctx) {
		if err := activity.GetHeartbeatDetails(ctx, &currentLine); err != nil {
			return err
		}
	}

	file, err := os.Open(filePath)
	if err != nil {
		return err
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		if currentLine > 0 {
			currentLine--
			continue
		}

		activity.RecordHeartbeat(ctx, currentLine)

		// Check if the Activity has been cancelled
		select {
		case <-ctx.Done():
			cleanupResources()
			return ctx.Err()
		default:
		}

		processLine(scanner.Text())
		currentLine++
	}
	return scanner.Err()
}
```

**Java**

```java
// FileProcessingActivityImpl.java
public class FileProcessingActivityImpl implements FileProcessingActivity {
  @Override
  public void processLargeFile(String filePath) {
    ActivityExecutionContext context = Activity.getExecutionContext();
    Optional<Integer> lastProcessedLine = context.getHeartbeatDetails(Integer.class);
    int currentLine = lastProcessedLine.orElse(0);

    try (BufferedReader reader = new BufferedReader(new FileReader(filePath))) {
      for (int i = 0; i < currentLine; i++) {
        reader.readLine();
      }

      String line;
      while ((line = reader.readLine()) != null) {
        context.heartbeat(currentLine);
        processLine(line);
        currentLine++;
      }
    } catch (ActivityCompletionException e) {
      cleanupResources();
      throw e;
    }
  }
}
```

**TypeScript**

```typescript
// activities.ts
import { heartbeat, activityInfo, sleep } from '@temporalio/activity';
import { CancelledFailure } from '@temporalio/common';
import { createReadStream } from 'fs';
import { createInterface } from 'readline';

export async function processLargeFile(filePath: string): Promise<void> {
  const startLine = activityInfo().heartbeatDetails ?? 0;

  const rl = createInterface({
    input: createReadStream(filePath),
  });

  let currentLine = 0;
  try {
    for await (const line of rl) {
      if (currentLine < startLine) {
        currentLine++;
        continue;
      }

      heartbeat(currentLine);
      processLine(line);
      currentLine++;
    }
  } catch (err) {
    if (err instanceof CancelledFailure) {
      cleanupResources();
    }
    throw err;
  }
}
```

Cancellation is delivered to the Activity when it heartbeats.
In Java, the next `heartbeat()` call throws an `ActivityCompletionException` (an `ActivityCanceledException` for cancellation).
In TypeScript, cancellation is delivered as a `CancelledFailure` via `sleep()` or `Context.current().cancelled`.
In Python, cancellation is delivered as an `asyncio.CancelledError`.
In Go, the context is cancelled and `ctx.Done()` becomes readable.
The catch/error handling block performs cleanup before re-throwing the error.

### Complex progress state

The following implementation tracks multiple progress fields -- processed count, failed count, and the last processed ID:

**Python**

```python
# activities.py
from dataclasses import dataclass
from temporalio import activity

@dataclass
class ProgressState:
    processed_count: int = 0
    failed_count: int = 0
    last_processed_id: str = ""

@activity.defn
async def process_batch(item_ids: list[str]) -> dict:
    details = activity.info().heartbeat_details
    progress = details[0] if details else ProgressState()

    start_index = (
        item_ids.index(progress.last_processed_id) + 1
        if progress.last_processed_id
        else 0
    )

    for i in range(start_index, len(item_ids)):
        item_id = item_ids[i]

        try:
            await process_item(item_id)
            progress.processed_count += 1
        except Exception:
            progress.failed_count += 1

        progress.last_processed_id = item_id
        activity.heartbeat(progress)

    return {
        "processed_count": progress.processed_count,
        "failed_count": progress.failed_count,
    }
```

**Go**

```go
// activities.go
type ProgressState struct {
	ProcessedCount  int    `json:"processedCount"`
	FailedCount     int    `json:"failedCount"`
	LastProcessedID string `json:"lastProcessedId"`
}

type BatchResult struct {
	ProcessedCount int `json:"processedCount"`
	FailedCount    int `json:"failedCount"`
}

func ProcessBatch(ctx context.Context, itemIDs []string) (BatchResult, error) {
	progress := ProgressState{}
	if activity.HasHeartbeatDetails(ctx) {
		if err := activity.GetHeartbeatDetails(ctx, &progress); err != nil {
			return BatchResult{}, err
		}
	}

	startIndex := 0
	if progress.LastProcessedID != "" {
		for i, id := range itemIDs {
			if id == progress.LastProcessedID {
				startIndex = i + 1
				break
			}
		}
	}

	for i := startIndex; i < len(itemIDs); i++ {
		itemID := itemIDs[i]

		if err := processItem(ctx, itemID); err != nil {
			progress.FailedCount++
		} else {
			progress.ProcessedCount++
		}

		progress.LastProcessedID = itemID
		activity.RecordHeartbeat(ctx, progress)
	}

	return BatchResult{
		ProcessedCount: progress.ProcessedCount,
		FailedCount:    progress.FailedCount,
	}, nil
}
```

**Java**

```java
// BatchProcessingActivityImpl.java
public class BatchProcessingActivityImpl implements BatchProcessingActivity {

  static class ProgressState {
    int processedCount;
    int failedCount;
    String lastProcessedId;
  }

  @Override
  public BatchResult processBatch(List<String> itemIds) {
    ActivityExecutionContext context = Activity.getExecutionContext();
    Optional<ProgressState> details = context.getHeartbeatDetails(ProgressState.class);
    ProgressState progress = details.orElse(new ProgressState());

    int startIndex = itemIds.indexOf(progress.lastProcessedId) + 1;

    for (int i = startIndex; i < itemIds.size(); i++) {
      String itemId = itemIds.get(i);

      try {
        processItem(itemId);
        progress.processedCount++;
      } catch (Exception e) {
        progress.failedCount++;
      }

      progress.lastProcessedId = itemId;
      context.heartbeat(progress);
    }

    return new BatchResult(progress.processedCount, progress.failedCount);
  }
}
```

**TypeScript**

```typescript
// activities.ts
import { heartbeat, activityInfo } from '@temporalio/activity';

interface ProgressState {
  processedCount: number;
  failedCount: number;
  lastProcessedId: string;
}

export async function processBatch(itemIds: string[]): Promise<BatchResult> {
  const saved: ProgressState = activityInfo().heartbeatDetails ?? {
    processedCount: 0,
    failedCount: 0,
    lastProcessedId: '',
  };

  const startIndex = saved.lastProcessedId
    ? itemIds.indexOf(saved.lastProcessedId) + 1
    : 0;

  const progress = { ...saved };

  for (let i = startIndex; i < itemIds.length; i++) {
    const itemId = itemIds[i];

    try {
      await processItem(itemId);
      progress.processedCount++;
    } catch {
      progress.failedCount++;
    }

    progress.lastProcessedId = itemId;
    heartbeat(progress);
  }

  return { processedCount: progress.processedCount, failedCount: progress.failedCount };
}
```

The progress state object stores all the checkpoint data needed to resume.
On retry, the Activity finds the index of the last processed ID and starts from the next item.
Each heartbeat stores the full progress state, so the next attempt has everything it needs to resume.

## When to use

The Heartbeat pattern is a good fit for batch processing of large datasets, file uploads and downloads with progress tracking, database migrations or bulk operations, long-running computations (ML training, video encoding), external API polling with multiple attempts, and any Activity running longer than 30 seconds.

It is not a good fit for quick operations (under 10 seconds), operations that cannot be checkpointed, Activities requiring exact-once semantics without idempotency, or real-time streaming (use Workflows instead).

## Benefits and trade-offs

Heartbeats enable fault tolerance by resuming from the last checkpoint after failures.
Heartbeat timeouts detect stuck Activities faster than execution timeouts.
You gain visibility into Activity progress in real-time.
Activities can handle cancellation gracefully and clean up resources.
Completed work is not reprocessed, and Activities can move between Workers.

The trade-offs to consider are that frequent heartbeats increase network traffic.
You must implement checkpointing logic and state management.
You must handle partial reprocessing of the last checkpoint (idempotency).
You need to balance heartbeat frequency between responsiveness and overhead.
Heartbeat details have size limits, so you should avoid large objects.

## Comparison with alternatives

| Approach | Progress tracking | Resumable | Cancellation | Complexity |
| :--- | :--- | :--- | :--- | :--- |
| Heartbeat | Yes | Yes | Graceful | Medium |
| Long Timeout | No | No | Delayed | Low |
| Child Workflows | Yes | Yes | Immediate | High |
| Local Activity | No | No | N/A | Low |

## Best practices

- **Set heartbeat timeout.** Configure to 2-3x the expected heartbeat interval.
- **Heartbeat at regular intervals.** Balance between responsiveness (every 10-30 seconds) and overhead.
- **Checkpoint strategically.** Save progress at meaningful boundaries (records, pages, chunks).
- **Keep details small.** Store minimal state (IDs, offsets, counts), not full objects.
- **Handle idempotency.** Ensure reprocessing the last checkpoint is safe.
- **Check cancellation.** Heartbeat regularly to detect cancellation quickly.
- **Clean up on cancel.** Handle cancellation errors appropriately: catch `ActivityCompletionException` (Java), `CancelledFailure` (TypeScript), `asyncio.CancelledError` (Python), or check `ctx.Done()` (Go).
- **Log progress.** Log heartbeat details for debugging and monitoring.
- **Test resumption.** Verify Activities resume correctly after simulated failures.
- **Avoid heartbeat spam.** Do not heartbeat on every iteration of tight loops.

## Common pitfalls

- **Missing HeartbeatTimeout.** Without a HeartbeatTimeout, Temporal cannot detect a stuck or crashed Worker until the StartToCloseTimeout expires. Always set HeartbeatTimeout shorter than StartToCloseTimeout.
- **Heartbeating too infrequently.** Cancellation is only delivered on the next heartbeat. If the Activity heartbeats every 5 minutes, cancellation takes up to 5 minutes to propagate.
- **Not resuming from heartbeat progress on retry.** When an Activity retries, retrieve the last heartbeat details -- `context.getHeartbeatDetails()` (Java), `activityInfo().heartbeatDetails` (TypeScript), `activity.info().heartbeat_details` (Python), or `activity.GetHeartbeatDetails()` (Go) -- and resume from the last checkpoint instead of restarting from scratch.
- **Catching the wrong exception for cancellation.** Cancellation is SDK-specific. Inside the Activity, the `heartbeat()` call throws an `ActivityCompletionException` (Java), cancellation surfaces as a `CancelledFailure` (TypeScript) or an `asyncio.CancelledError` (Python), and the context reports `ctx.Err()` returning `context.Canceled` (Go). The `CanceledFailure` type is what the Workflow observes as the cause of the resulting `ActivityFailure`, not what the Activity body catches.

## Related

### Patterns

- **[Saga Pattern](/design-patterns/saga-pattern)**: Compensating transactions with long-running steps.
- **[Polling](/design-patterns/polling)**: Heartbeating Activity for frequent polling.

### References

- [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat): Canonical reference for heartbeats, heartbeat timeouts, and throttling.

### Sample code

### Java
- [Heartbeating Activity Batch](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/heartbeatingactivity) -- Complete batch processing implementation.
- [Auto-Heartbeating](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/autoheartbeat) -- Automatic heartbeating via interceptor.

### TypeScript
- [Activities Cancellation and Heartbeating](https://github.com/temporalio/samples-typescript/tree/main/activities-cancellation-heartbeating) -- Activity cancellation and heartbeat-based resumption.

### Python
- [Hello Cancellation](https://github.com/temporalio/samples-python/blob/main/hello/hello_cancellation.py) -- Activity heartbeating with cancellation handling.
- [Custom Decorator Heartbeat](https://github.com/temporalio/samples-python/blob/main/custom_decorator/activity_utils.py) -- Automatic heartbeating via decorator.

### Go
- [Cancellation](https://github.com/temporalio/samples-go/tree/main/cancellation) -- Workflow and Activity cancellation with heartbeating.

---

# MapReduce Tree

Source: https://docs.temporal.io/design-patterns/mapreduce-tree

> Recursively splits a dataset into a binary tree of Child Workflows, processes leaves in parallel, then aggregates results back up the tree.

> **ℹ️ TLDR:**
> Recursively split a record set into a tree of child Workflows — each node fans out to N sub-slices (two by default) — process every leaf in parallel, and signal results back up the tree to the root. Use this when you need **maximum throughput** for an embarrassingly parallel workload and downstream systems can absorb an unbounded burst of concurrent requests.

## Overview

The MapReduce Tree pattern processes a large record set with maximum parallelism by recursively splitting it into smaller chunks and distributing each chunk to a child Workflow. Results are signalled back up the tree to the parent. It is best suited for embarrassingly parallel workloads where speed matters more than rate limiting.

## Problem

Both the [Batch Iterator](/design-patterns/batch-iterator) and [Sliding Window](/design-patterns/sliding-window) patterns bound concurrency, which limits throughput. When you need to process a large record set as fast as possible and downstream systems can handle the load, you want to fan out work across as many concurrent processors as possible without a fixed window.

You also need a way to handle record sets larger than what a single Workflow's concurrency limits allow, without pre-partitioning data into fixed chunks before the job starts.

## Solution

A Node Workflow receives a slice of records. If the slice is small enough (at or below a configurable `leafThreshold`), it starts one Leaf Workflow per record. Otherwise it splits the slice into `n` sub-slices and starts `n` Node child Workflows recursively.

Each Leaf Workflow runs the actual processing Activity and signals its result back to its parent Node. Each Node aggregates the results it receives and signals them up to its own parent. The Root Node returns the final aggregated result.

```mermaid
flowchart TD
    Records["📋 Full record set"]
    Root["Root Node Workflow\n(depth=0)"]
    Node1["Node Workflow\n(depth=1, chunk 1)"]
    Node2["Node Workflow\n(depth=1, chunk 2)"]
    L1["Leaf Workflow\n(record A)"]
    L2["Leaf Workflow\n(record B)"]
    L3["Leaf Workflow\n(record C)"]
    L4["Leaf Workflow\n(record D)"]
    L5["Leaf Workflow\n(record E)"]
    L6["Leaf Workflow\n(record F)"]

    Records --> Root
    Root -->|"split → chunk 1"| Node1
    Root -->|"split → chunk 2"| Node2

    Node1 --> L1
    Node1 --> L2
    Node1 --> L3

    Node2 --> L4
    Node2 --> L5
    Node2 --> L6

    L1 -->|"Signal result"| Node1
    L2 -->|"Signal result"| Node1
    L3 -->|"Signal result"| Node1

    L4 -->|"Signal result"| Node2
    L5 -->|"Signal result"| Node2
    L6 -->|"Signal result"| Node2

    Node1 -->|"Signal result"| Root
    Node2 -->|"Signal result"| Root
```

The following describes each step in the diagram:

1. The Root Node Workflow receives the full record set and `depth=0`.
2. Because the record set is larger than `leafThreshold`, the Root splits it into N chunks and starts N child Node Workflows (two in this example, but the split factor is configurable).
3. Each Node Workflow receives its chunk and checks its size against `leafThreshold`. In this example, each chunk is small enough, so each Node starts one Leaf Workflow per record.
4. Each Leaf Workflow calls the `processLeaf` Activity and, when complete, signals its result back to its parent Node using `signalExternalWorkflow`.
5. Each Node collects all leaf results via signal handlers, aggregates them, and signals the aggregated result back to the Root.
6. The Root collects both node results and returns the final aggregate.

## Implementation

The following examples show how each SDK implements the MapReduce Tree pattern.

**TypeScript**

```typescript
// workflows.ts
import {
  condition,
  defineSignal,
  getExternalWorkflowHandle,
  proxyActivities,
  setHandler,
  startChild,
  workflowInfo,
} from "@temporalio/workflow";
import type * as activities from "./activities";
import { TASK_QUEUE, LEAF_THRESHOLD, MAX_DEPTH, RESULT_SIGNAL } from "./shared";

const { processLeaf } = proxyActivities<typeof activities>({
  startToCloseTimeout: "30 seconds",
});

interface ResultPayload {
  id: string;
  results: string[];
}

export const resultSignal = defineSignal<[ResultPayload]>(RESULT_SIGNAL);

export async function leafWorkflow(
  record: string,
  parentWorkflowId: string
): Promise<void> {
  const result = await processLeaf(record);
  // Signal result back to parent node.
  const parent = getExternalWorkflowHandle(parentWorkflowId);
  await parent.signal(resultSignal, { id: record, results: [result] });
}

export async function nodeWorkflow(
  records: string[],
  depth: number = 0,
  parentWorkflowId: string = ""
): Promise<string[]> {
  if (depth > MAX_DEPTH) {
    throw new Error(`Tree depth exceeded ${MAX_DEPTH}`);
  }

  const myId = workflowInfo().workflowId;
  const collectedResults: string[] = [];
  let received = 0;
  let expected = 0;

  setHandler(resultSignal, (payload: ResultPayload) => {
    collectedResults.push(...payload.results);
    received++;
  });

  if (records.length <= LEAF_THRESHOLD) {
    // Start one leaf per record.
    expected = records.length;
    for (const record of records) {
      void startChild(leafWorkflow, {
        args: [record, myId],
        workflowId: `${myId}/leaf-${record}`,
        taskQueue: TASK_QUEUE,
      });
    }
  } else {
    // Split and recurse.
    const mid = Math.floor(records.length / 2);
    const chunks = [records.slice(0, mid), records.slice(mid)];
    expected = chunks.length;
    for (let i = 0; i < chunks.length; i++) {
      void startChild(nodeWorkflow, {
        args: [chunks[i], depth + 1, myId],
        workflowId: `${myId}/node-d${depth + 1}-${i}`,
        taskQueue: TASK_QUEUE,
      });
    }
  }

  // Wait until all expected signals have arrived.
  await condition(() => received >= expected);

  // Signal aggregated results up to parent (if this is not the root).
  if (parentWorkflowId) {
    const parent = getExternalWorkflowHandle(parentWorkflowId);
    await parent.signal(resultSignal, { id: myId, results: collectedResults });
  }

  return collectedResults;
}
```

**Python**

```python
# workflows.py
from dataclasses import dataclass
from datetime import timedelta
from temporalio import workflow
from activities import process_leaf
from shared import TASK_QUEUE, LEAF_THRESHOLD, MAX_DEPTH, RESULT_SIGNAL

@dataclass
class ResultPayload:
    id: str
    results: list[str]

@workflow.defn
class LeafWorkflow:
    @workflow.run
    async def run(self, record: str, parent_workflow_id: str) -> None:
        result = await workflow.execute_activity(
            process_leaf,
            record,
            start_to_close_timeout=timedelta(seconds=30),
        )
        parent = workflow.get_external_workflow_handle(parent_workflow_id)
        await parent.signal(RESULT_SIGNAL, ResultPayload(id=record, results=[result]))

@workflow.defn
class NodeWorkflow:
    def __init__(self) -> None:
        self._collected: list[str] = []
        self._received = 0

    @workflow.signal(name=RESULT_SIGNAL)
    def node_result(self, payload: ResultPayload) -> None:
        self._collected.extend(payload.results)
        self._received += 1

    @workflow.run
    async def run(
        self,
        records: list[str],
        depth: int = 0,
        parent_workflow_id: str = "",
    ) -> list[str]:
        if depth > MAX_DEPTH:
            raise RuntimeError(f"Tree depth exceeded {MAX_DEPTH}")

        my_id = workflow.info().workflow_id
        expected = 0

        if len(records) <= LEAF_THRESHOLD:
            expected = len(records)
            for record in records:
                await workflow.start_child_workflow(
                    LeafWorkflow.run,
                    args=[record, my_id],
                    id=f"{my_id}/leaf-{record}",
                    task_queue=TASK_QUEUE,
                )
        else:
            mid = len(records) // 2
            chunks = [records[:mid], records[mid:]]
            expected = len(chunks)
            for i, chunk in enumerate(chunks):
                await workflow.start_child_workflow(
                    NodeWorkflow.run,
                    args=[chunk, depth + 1, my_id],
                    id=f"{my_id}/node-d{depth+1}-{i}",
                    task_queue=TASK_QUEUE,
                )

        await workflow.wait_condition(lambda: self._received >= expected)

        if parent_workflow_id:
            parent = workflow.get_external_workflow_handle(parent_workflow_id)
            await parent.signal(
                RESULT_SIGNAL, ResultPayload(id=my_id, results=self._collected)
            )

        return self._collected
```

**Go**

```go
// workflows.go
package main

import (
	"fmt"
	"time"

	"go.temporal.io/sdk/workflow"
)

// ResultSignal and ResultPayload are declared in shared.go.

func LeafWorkflow(ctx workflow.Context, record string, parentWorkflowID string) error {
	ao := workflow.ActivityOptions{StartToCloseTimeout: 30 * time.Second}
	ctx = workflow.WithActivityOptions(ctx, ao)

	var result string
	if err := workflow.ExecuteActivity(ctx, ProcessLeaf, record).Get(ctx, &result); err != nil {
		return err
	}

	payload := ResultPayload{ID: record, Results: []string{result}}
	return workflow.SignalExternalWorkflow(ctx, parentWorkflowID, "", ResultSignal, payload).Get(ctx, nil)
}

func NodeWorkflow(ctx workflow.Context, records []string, depth int, parentWorkflowID string) ([]string, error) {
	if depth > MaxDepth {
		return nil, fmt.Errorf("tree depth exceeded %d", MaxDepth)
	}

	myID := workflow.GetInfo(ctx).WorkflowExecution.ID
	resultCh := workflow.GetSignalChannel(ctx, ResultSignal)

	var collected []string
	expected := 0

	if len(records) <= LeafThreshold {
		expected = len(records)
		for _, record := range records {
			cwo := workflow.ChildWorkflowOptions{
				WorkflowID: myID + "/leaf-" + record,
				TaskQueue:  TaskQueue,
			}
			workflow.ExecuteChildWorkflow(workflow.WithChildOptions(ctx, cwo), LeafWorkflow, record, myID)
		}
	} else {
		mid := len(records) / 2
		chunks := [][]string{records[:mid], records[mid:]}
		expected = len(chunks)
		for i, chunk := range chunks {
			cwo := workflow.ChildWorkflowOptions{
				WorkflowID: fmt.Sprintf("%s/node-d%d-%d", myID, depth+1, i),
				TaskQueue:  TaskQueue,
			}
			workflow.ExecuteChildWorkflow(workflow.WithChildOptions(ctx, cwo), NodeWorkflow, chunk, depth+1, myID)
		}
	}

	for i := 0; i < expected; i++ {
		var payload ResultPayload
		resultCh.Receive(ctx, &payload)
		collected = append(collected, payload.Results...)
	}

	if parentWorkflowID != "" {
		payload := ResultPayload{ID: myID, Results: collected}
		if err := workflow.SignalExternalWorkflow(ctx, parentWorkflowID, "", ResultSignal, payload).Get(ctx, nil); err != nil {
			return collected, err
		}
	}

	return collected, nil
}
```

**Java**

```java
// NodeWorkflow.java
import io.temporal.workflow.*;
import java.util.*;

@WorkflowInterface
public interface NodeWorkflow {
    @WorkflowMethod
    List<String> run(List<String> records, int depth, String parentWorkflowId);

    @SignalMethod
    void nodeResult(String id, List<String> results);
}

// NodeWorkflowImpl.java
public class NodeWorkflowImpl implements NodeWorkflow {
    private final List<String> collected = new ArrayList<>();
    private int received = 0;

    @Override
    public void nodeResult(String id, List<String> results) {
        collected.addAll(results);
        received++;
    }

    @Override
    public List<String> run(List<String> records, int depth, String parentWorkflowId) {
        if (depth > Shared.MAX_DEPTH) {
            throw new RuntimeException("Tree depth exceeded " + Shared.MAX_DEPTH);
        }

        String myId = Workflow.getInfo().getWorkflowId();
        int expected;

        if (records.size() <= Shared.LEAF_THRESHOLD) {
            for (String record : records) {
                ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder()
                    .setWorkflowId(myId + "/leaf-" + record)
                    .setTaskQueue(Shared.TASK_QUEUE)
                    .build();
                LeafWorkflow leaf = Workflow.newChildWorkflowStub(LeafWorkflow.class, opts);
                Async.procedure(leaf::run, record, myId);
            }
            expected = records.size();
        } else {
            int mid = records.size() / 2;
            List<List<String>> chunks = List.of(records.subList(0, mid), records.subList(mid, records.size()));
            for (int i = 0; i < chunks.size(); i++) {
                ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder()
                    .setWorkflowId(String.format("%s/node-d%d-%d", myId, depth + 1, i))
                    .setTaskQueue(Shared.TASK_QUEUE)
                    .build();
                NodeWorkflow child = Workflow.newChildWorkflowStub(NodeWorkflow.class, opts);
                Async.function(child::run, chunks.get(i), depth + 1, myId);
            }
            expected = chunks.size();
        }

        final int exp = expected;
        Workflow.await(() -> received >= exp);

        if (parentWorkflowId != null && !parentWorkflowId.isEmpty()) {
            ExternalWorkflowStub parent = Workflow.newUntypedExternalWorkflowStub(parentWorkflowId, "");
            parent.signal(Shared.RESULT_SIGNAL, myId, new ArrayList<>(collected));
        }

        return collected;
    }
}
```

## Best practices

- **Set a `leafThreshold` to control tree depth.** A threshold of 3–10 records per leaf is typical. Too small a threshold creates excessive Workflow overhead; too large prevents full parallelism.
- **Set a `MAX_DEPTH` guard.** Recursive fan-out without a depth limit can produce extremely deep trees for large record sets. Fail fast if depth exceeds your expected maximum (for example, `log2(totalRecords / leafThreshold) + 2`).
- **Avoid external writes in Node Workflows.** Node Workflows only aggregate results from children. Leaf Workflows perform the actual work. Keeping the roles separate prevents duplicate external writes if a Node is retried.
- **Use signals for result aggregation, not return values.** A parent cannot directly await a child started in a previous Workflow run. Signals decouple the result delivery from the parent-child lifetime, making the pattern resilient to replays.
- **Skip the reduce phase if results are not needed.** If you only need the side effects of processing each record (writes to a database, messages sent), omit the signal-back entirely and set `PARENT_CLOSE_POLICY_ABANDON` on all children.
- **Consider replacing Leaf Workflows with Activities for lighter workloads.** Leaf Workflows give each record its own Event History, independent cancellation, and dedicated visibility in the UI — useful when per-record observability matters. If those properties are not required, executing the `processLeaf` Activity directly from a Node Workflow reduces overhead. A successful child workflow and a successful Activity each add three Events to the parent's history, but a Leaf Workflow also maintains a separate Event History of its own and emits an extra signal back to the Node, so child workflows produce more overall Events than Activities. The Temporal documentation recommends starting with Activities and adopting child workflows only when there is a clear need.

## Common pitfalls

- **Too many direct children per Node.** Temporal's [guidance](/child-workflows#when-to-use-child-workflows) is that a single parent should not spawn more than 1,000 Child Workflow Executions. This applies to every Node in the tree, not just the Root — if a Node's own branching factor produces more than that many direct child Workflows, add another tree level rather than widening a single Node.
- **Thundering herd.** The MapReduce Tree fans out exponentially. For large record sets, all leaf Activities start nearly simultaneously. Ensure your downstream system can absorb the burst, or switch to [Sliding Window](/design-patterns/sliding-window) for rate limiting.
- **Signal storms.** If thousands of leaves all signal a single Node at the same time, the Node's signal queue can become a bottleneck. A two-level tree (Root → Nodes → Leaves) distributes this load; a deeper tree helps even more.
- **History bloat in the Root Workflow.** Each child start and signal received adds events to the Root's history. For very large record sets, consider adding an extra tree level to keep the Root from receiving too many direct signals.
- **Attempting external/downstream writes from Node Workflows.** Nodes may be retried. Any external write in a Node Workflow will be executed multiple times. Keep all side effects in Leaf Workflows (or Activities called by Leaves).

## Related

### Patterns

- [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows) — simpler flat fan-out for smaller record sets
- [Sliding Window](/design-patterns/sliding-window) — bounded concurrency with rate limiting
- [Child Workflows pattern](/design-patterns/child-workflows) — core concepts for parent/child coordination
- [Temporal limits reference](/evaluate/cloud/limits)

---

# Non-Retryable Errors

Source: https://docs.temporal.io/design-patterns/non-retryable-errors

> Mark error types that will never succeed — such as validation failures or missing records — so Temporal fails fast instead of retrying indefinitely.

> **ℹ️ TLDR:**
> Raise a non-retryable `ApplicationError` from the Activity — or list error types in the `RetryPolicy` — so **the Temporal Activity fails fast instead of retrying indefinitely**. Use this for permanent failures such as invalid input, missing records, or authorization errors where repeating the same call will never succeed.

## Overview

The Non-Retryable Errors pattern marks specific error types so Temporal stops retrying immediately when one is raised.
Use it for failures where the root cause is structural — invalid input, a missing record, an authorization problem — where repeating the same call will never produce a different result.

## Problem

Temporal retries all Activity failures by default.
For transient infrastructure errors such as network timeouts or service restarts, this is the right behavior.
But some failures are permanent: no amount of retrying will fix them.

Retrying a permanent failure wastes time and resources:

- A transfer to a non-existent account number will fail on attempt 1, 2, and 3 in exactly the same way.
- An API call with a malformed request body will be rejected every time.
- A request from a revoked API key will receive an authorization error on every attempt.

With the default unlimited retry policy, the Workflow waits through exponential backoff delays — minutes to hours — before eventually delivering the error to the Workflow, when it could have failed in milliseconds.

## Solution

Raise a non-retryable error from inside the Activity when the failure is known to be permanent.
Temporal detects the error type and skips all remaining retries, delivering the failure to the Workflow immediately.

There are two complementary mechanisms:

1. **Mark the error as non-retryable at the throw site** — the Activity explicitly signals that this specific failure should not be retried.
2. **Register non-retryable error types in the `RetryPolicy`** — the Workflow declares which error type names should never be retried, regardless of how the Activity raises them.

Both mechanisms can be used together.

```mermaid
flowchart TD
    Workflow -->|Schedule Activity| Activity
    Activity -->|Raises error| Check{Is error type\nnon-retryable?}
    Check -->|Yes — marked at throw site\nor listed in RetryPolicy| Fail([Deliver ActivityError\nto Workflow immediately])
    Check -->|No| Retry[Schedule retry\nwith backoff]
    Retry --> Activity
    Fail --> Handle[Workflow handles error:\nlog, compensate, or escalate]
```

The following describes each path:

1. The Activity raises an error. Temporal inspects whether the error type is non-retryable.
2. If non-retryable — either because the Activity flagged it or the `RetryPolicy` lists the type — Temporal delivers the `ActivityError` to the Workflow without delay.
3. If retryable, Temporal schedules another attempt after the configured backoff.
4. The Workflow catches the `ActivityError` and handles it according to the business logic.

## Implementation

### Marking an error as non-retryable at the throw site

Use the SDK's `ApplicationError` (or equivalent) with the non-retryable flag.
Temporal propagates the error type name and the flag to the Workflow without retrying.

**Python**

```python
# activities.py
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def process_order(order_id: str) -> str:
    response = await http_client.post(f"/orders/{order_id}/process")
    if response.status_code == 404:
        raise ApplicationError(
            f"Order {order_id} not found",
            type="OrderNotFoundError",
            non_retryable=True,
        )
    if response.status_code == 422:
        raise ApplicationError(
            f"Order {order_id} rejected: {response.json().get('detail', 'validation error')}",
            type="ValidationError",
            non_retryable=True,
        )

    response.raise_for_status()
    return response.json()["confirmation_id"]
```

**Go**

```go
// activities.go
package orders

import (
    "context"
    "fmt"

    "go.temporal.io/sdk/temporal"
)

func ProcessOrder(ctx context.Context, orderID string) (string, error) {
    resp, err := httpClient.Post(fmt.Sprintf("/orders/%s/process", orderID))
    if err != nil {
        return "", err
    }
    if resp.StatusCode == 404 {
        return "", temporal.NewNonRetryableApplicationError(
            fmt.Sprintf("order %s not found", orderID),
            "OrderNotFoundError",
            nil,
        )
    }
    if resp.StatusCode == 422 {
        return "", temporal.NewNonRetryableApplicationError(
            fmt.Sprintf("order %s rejected: %s", orderID, resp.ErrorDetail),
            "ValidationError",
            nil,
        )
    }
    return resp.ConfirmationID, nil
}
```

**Java**

```java
// ProcessOrderActivityImpl.java
import io.temporal.failure.ApplicationFailure;

public class ProcessOrderActivityImpl implements ProcessOrderActivity {
    @Override
    public String processOrder(String orderId) {
        HttpResponse response = httpClient.post("/orders/" + orderId + "/process");
        if (response.getStatusCode() == 404) {
            throw ApplicationFailure.newNonRetryableFailure(
                "Order " + orderId + " not found",
                "OrderNotFoundError"
            );
        }
        if (response.getStatusCode() == 422) {
            throw ApplicationFailure.newNonRetryableFailure(
                "Order " + orderId + " rejected: " + response.getErrorDetail(),
                "ValidationError"
            );
        }
        return response.getConfirmationId();
    }
}
```

**TypeScript**

```typescript
// activities.ts
import { ApplicationFailure } from '@temporalio/activity';

export async function processOrder(orderId: string): Promise<string> {
    const response = await fetch(`/orders/${orderId}/process`, { method: 'POST' });
    if (response.status === 404) {
        throw ApplicationFailure.nonRetryable(
            `Order ${orderId} not found`,
            'OrderNotFoundError',
        );
    }
    if (response.status === 422) {
        const body = await response.json();
        throw ApplicationFailure.nonRetryable(
            `Order ${orderId} rejected: ${body.detail ?? 'validation error'}`,
            'ValidationError',
        );
    }

    if (!response.ok) throw new Error(`API error ${response.status}`);
    const body = await response.json();
    return body.confirmation_id;
}
```

### Declaring non-retryable types in the RetryPolicy

Alternatively, list error type names in the `RetryPolicy` at the Workflow call site.
Temporal stops retrying when the Activity raises an error whose type matches any name in the list.
This approach separates the retry decision from the Activity code, which is useful when the Activity is shared and the non-retryable classification depends on the caller's context.

The Activity raises a standard `ApplicationError` with a type name but without the non-retryable flag — the retry decision is delegated to the Workflow's `RetryPolicy`:

**Python**

```python
# activities.py
@activity.defn
async def process_order(order_id: str) -> str:
    response = await http_client.post(f"/orders/{order_id}/process")
    if response.status_code == 404:
        # No non_retryable=True — the RetryPolicy in the Workflow controls retry behavior
        raise ApplicationError(f"Order {order_id} not found", type="OrderNotFoundError")
    if response.status_code == 422:
        raise ApplicationError(
            f"Order {order_id} rejected: {response.json().get('detail', 'validation error')}",
            type="ValidationError",
        )
    response.raise_for_status()
    return response.json()["confirmation_id"]
```

**Go**

```go
// activities.go
func ProcessOrder(ctx context.Context, orderID string) (string, error) {
    resp, err := httpClient.Post(fmt.Sprintf("/orders/%s/process", orderID))
    if err != nil {
        return "", err
    }
    if resp.StatusCode == 404 {
        // Use NewApplicationError, not NewNonRetryableApplicationError
        return "", temporal.NewApplicationError(
            fmt.Sprintf("order %s not found", orderID), "OrderNotFoundError",
        )
    }
    if resp.StatusCode == 422 {
        return "", temporal.NewApplicationError(
            fmt.Sprintf("order %s rejected: %s", orderID, resp.ErrorDetail), "ValidationError",
        )
    }
    return resp.ConfirmationID, nil
}
```

**Java**

```java
// ProcessOrderActivityImpl.java
public String processOrder(String orderId) {
    HttpResponse response = httpClient.post("/orders/" + orderId + "/process");
    if (response.getStatusCode() == 404) {
        // Use newFailure, not newNonRetryableFailure
        throw ApplicationFailure.newFailure("Order " + orderId + " not found", "OrderNotFoundError");
    }
    if (response.getStatusCode() == 422) {
        throw ApplicationFailure.newFailure(
            "Order " + orderId + " rejected: " + response.getErrorDetail(), "ValidationError");
    }
    return response.getConfirmationId();
}
```

**TypeScript**

```typescript
// activities.ts
export async function processOrder(orderId: string): Promise<string> {
    const response = await fetch(`/orders/${orderId}/process`, { method: 'POST' });
    if (response.status === 404) {
        // Use ApplicationFailure.create, not .nonRetryable
        throw ApplicationFailure.create({ message: `Order ${orderId} not found`, type: 'OrderNotFoundError' });
    }
    if (response.status === 422) {
        const body = await response.json();
        throw ApplicationFailure.create({
            message: `Order ${orderId} rejected: ${body.detail ?? 'validation error'}`,
            type: 'ValidationError',
        });
    }
    if (!response.ok) throw new Error(`API error ${response.status}`);
    const body = await response.json();
    return body.confirmation_id;
}
```

The Workflow lists which error type names to never retry:

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
import activities

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order_id: str) -> str:
        return await workflow.execute_activity(
            activities.process_order,
            order_id,
            start_to_close_timeout=timedelta(seconds=10),
            retry_policy=RetryPolicy(
                non_retryable_error_types=["OrderNotFoundError", "ValidationError"],
            ),
        )
```

**Go**

```go
// workflow.go
ao := workflow.ActivityOptions{
    StartToCloseTimeout: 10 * time.Second,
    RetryPolicy: &temporal.RetryPolicy{
        NonRetryableErrorTypes: []string{"OrderNotFoundError", "ValidationError"},
    },
}
ctx = workflow.WithActivityOptions(ctx, ao)
```

**Java**

```java
// OrderWorkflowImpl.java
private final ProcessOrderActivity activities = Workflow.newActivityStub(
    ProcessOrderActivity.class,
    ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(10))
        .setRetryOptions(RetryOptions.newBuilder()
            .setDoNotRetry("OrderNotFoundError", "ValidationError")
            .build())
        .build()
);
```

**TypeScript**

```typescript
// workflows.ts
const { processOrder } = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '10s',
    retry: {
        nonRetryableErrorTypes: ['OrderNotFoundError', 'ValidationError'],
    },
});
```

### Handle non-retryable errors in the Workflow

Catch the `ActivityError` in the Workflow to distinguish between permanent failures and transient ones.
Inspect the underlying cause to route to the appropriate compensation or escalation path.

**Python**

```python
# workflows.py
from temporalio.exceptions import ActivityError, ApplicationError

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order_id: str) -> str:
        try:
            return await workflow.execute_activity(
                activities.process_order,
                order_id,
                start_to_close_timeout=timedelta(seconds=10),
                retry_policy=RetryPolicy(
                    non_retryable_error_types=["OrderNotFoundError", "ValidationError"],
                ),
            )
        except ActivityError as e:
            cause = e.__cause__
            if isinstance(cause, ApplicationError):
                if cause.type == "ValidationError":
                    return f"Order rejected: {cause}"
                if cause.type == "OrderNotFoundError":
                    return f"Order not found: {cause}"
            raise
```

**Go**

```go
// workflow.go
var result string
err := workflow.ExecuteActivity(ctx, ProcessOrder, orderID).Get(ctx, &result)
if err != nil {
    var appErr *temporal.ApplicationError
    if errors.As(err, &appErr) {
        switch appErr.Type() {
        case "ValidationError":
            return "", fmt.Errorf("order rejected: %w", appErr)
        case "OrderNotFoundError":
            return "", fmt.Errorf("order not found: %w", appErr)
        }
    }
    return "", err
}
```

**Java**

```java
// OrderWorkflowImpl.java
try {
    return activities.processOrder(orderId);
} catch (ActivityFailure e) {
    if (e.getCause() instanceof ApplicationFailure appFailure) {
        switch (appFailure.getType()) {
            case "ValidationError" -> { return "Order rejected: " + appFailure.getMessage(); }
            case "OrderNotFoundError" -> { return "Order not found: " + appFailure.getMessage(); }
        }
    }
    throw e;
}
```

**TypeScript**

```typescript
// workflows.ts
try {
    return await processOrder(orderId);
} catch (err) {
    if (err instanceof wf.ActivityFailure && err.cause instanceof wf.ApplicationFailure) {
        if (err.cause.type === 'ValidationError') {
            return `Order rejected: ${err.cause.message}`;
        }
        if (err.cause.type === 'OrderNotFoundError') {
            return `Order not found: ${err.cause.message}`;
        }
    }
    throw err;
}
```

## Best practices

- **Validate input before scheduling the Activity.** If the Workflow can detect invalid input upfront — using an `Update` validator or by inspecting the input data — fail fast in the Workflow rather than paying the cost of an Activity execution.
- **Use specific error type names.** Generic names like `"Error"` or `"Failure"` match too broadly. Use domain-specific names like `"OrderNotFoundError"` or `"InsufficientFundsError"` so the Workflow can distinguish between failure causes.
- **Reserve non-retryable for truly permanent failures.** A rate-limit error (HTTP 429) is transient — the same call will succeed after a delay. A not-found error (HTTP 404) is typically permanent. Match the non-retryable classification to the nature of the error.
- **Combine both mechanisms for defence in depth.** Mark the error as non-retryable at the throw site so the Activity is self-describing, and also list the type in the `RetryPolicy` so the classification is enforced even if the Activity code changes.

## Common pitfalls

- **Marking transient errors as non-retryable.** Network timeouts and service unavailability are transient. Marking them non-retryable removes Temporal's ability to recover automatically.
- **Using the error message instead of a type name.** `RetryPolicy.NonRetryableErrorTypes` matches on type names, not message strings. Without a type name, the policy cannot identify the error.
- **Swallowing the `ActivityError` without logging.** Non-retryable errors fail fast and silently if you do not catch and log them. Always log the failure before re-raising or returning an error result.
- **Confusing non-retryable errors with Workflow failures.** A non-retryable `ActivityError` fails the Activity and delivers the error to the Workflow. The Workflow itself does not fail unless it re-raises the error without catching it.
- **Losing the flag by wrapping the error.** The SDK inspects only the **outermost** error to decide how to represent the failure to the Temporal Service — the Server's retry decision looks only at that top-level failure info, not at the `cause` chain. If your Activity catches a non-retryable `ApplicationFailure` and re-throws it wrapped in a plain language error or exception (for example, Go's `fmt.Errorf("...: %w", err)`), the SDK converts that outer error into a new, retryable failure and the `non_retryable` flag is silently lost. To add context without losing it, wrap the error in another Application Failure that carries the same non-retryable flag — see [The outermost error type determines retryability](/encyclopedia/application-failures#outermost-error-type).

## Related

### Patterns

- [Fixed Count of Retries](/design-patterns/fixed-count-retries): Limit retries for transient errors that are worth retrying a bounded number of times.
- [Resumable Activity](/design-patterns/resumable-activity): Park the Workflow and accept a corrected input via Signal when retries are exhausted.
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns.

### Guides

- [Recover business processes without restarting](/guides/recover-without-restart): Uses `ApplicationFailure.nonRetryable()` throughout a loan pipeline to distinguish permanent failures that need a human fix from transient ones the default Retry Policy already handles.

### References

- [Temporal Retry Policies](/encyclopedia/retry-policies)
- [Failure Handling in Practice](https://temporal.io/blog/failure-handling-in-practice)

---

# Parallel Execution

Source: https://docs.temporal.io/design-patterns/parallel-execution

> Executes multiple Activities concurrently for maximum throughput with error handling and controlled parallelism.

## Overview

The Parallel Execution pattern enables concurrent execution of multiple Activities or Child Workflows to maximize throughput and minimize total execution time.
Using Temporal's async APIs, you can launch multiple operations asynchronously and wait for their completion.

## Problem

In sequential execution, operations run one after another, causing unnecessary delays when multiple independent operations could run simultaneously.
Total execution time equals the sum of all operation durations, resources sit idle while waiting, and batch processing takes hours when it could take minutes.

Without parallel execution, you must accept slow sequential processing, implement complex threading or async logic manually, risk inconsistent state management across threads, and handle thread safety and synchronization issues.

## Solution

Each SDK provides its own mechanism for launching Activities concurrently and waiting for the results:

- **Java**: `Async.function()` schedules Activities that return `Promise` objects. `Promise.allOf()` waits for all of them.
- **TypeScript**: Activity proxy functions return native `Promise` objects. `Promise.all()` waits for all of them.
- **Python**: `workflow.execute_activity()` returns awaitables. `asyncio.gather()` waits for all of them.
- **Go**: `workflow.ExecuteActivity()` returns `Future` objects. You call `.Get()` on each Future to collect results.

```mermaid
sequenceDiagram
    participant Workflow
    participant Activity1
    participant Activity2
    participant Activity3

    Workflow->>+Activity1: Start async
    Workflow->>+Activity2: Start async
    Workflow->>+Activity3: Start async
    Note over Workflow: Returns immediately with Futures/Promises

    par Parallel Execution
        Activity1->>Activity1: Execute
        Activity2->>Activity2: Execute
        Activity3->>Activity3: Execute
    end

    Activity1-->>-Workflow: Result 1
    Activity2-->>-Workflow: Result 2
    Activity3-->>-Workflow: Result 3

    Workflow->>Workflow: Await all results
    Note over Workflow: Collect all results
```

The following describes each step in the diagram:

1. The Workflow starts three Activities asynchronously, which returns immediately with Futures or Promises.
2. All three Activities execute in parallel on available Workers.
3. As each Activity completes, its Future or Promise resolves with the result.
4. The Workflow waits until all results are available, then collects them.

## Implementation

### Basic parallel Activities

The following implementation starts one Activity per item in a list and waits for all of them to complete:

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import process

@workflow.defn
class ParallelWorkflow:
    @workflow.run
    async def run(self, items: list[str]) -> list[str]:
        results = await asyncio.gather(
            *[
                workflow.execute_activity(
                    process,
                    item,
                    start_to_close_timeout=timedelta(seconds=30),
                )
                for item in items
            ]
        )
        return list(results)
```

**Go**

```go
// parallel_workflow.go
func ProcessInParallel(ctx workflow.Context, items []string) ([]string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 30 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	futures := make([]workflow.Future, len(items))
	for i, item := range items {
		futures[i] = workflow.ExecuteActivity(ctx, Process, item)
	}

	results := make([]string, len(items))
	for i, future := range futures {
		if err := future.Get(ctx, &results[i]); err != nil {
			return nil, err
		}
	}
	return results, nil
}
```

**Java**

```java
// ParallelWorkflowImpl.java
@WorkflowInterface
public interface ParallelWorkflow {
  @WorkflowMethod
  List<String> processInParallel(List<String> items);
}

public class ParallelWorkflowImpl implements ParallelWorkflow {
  private final ProcessingActivity activity =
    Workflow.newActivityStub(ProcessingActivity.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .build());

  @Override
  public List<String> processInParallel(List<String> items) {
    List<Promise<String>> promises = new ArrayList<>();

    for (String item : items) {
      Promise<String> promise = Async.function(activity::process, item);
      promises.add(promise);
    }

    Promise.allOf(promises).get();
    return promises.stream().map(Promise::get).collect(Collectors.toList());
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { process } = proxyActivities<typeof activities>({
  startToCloseTimeout: '30s',
});

export async function processInParallel(items: string[]): Promise<string[]> {
  const promises = items.map((item) => process(item));
  return await Promise.all(promises);
}
```

Each SDK schedules all Activities before waiting for any results.
In Java, `Async.function()` returns a `Promise`; in TypeScript, calling the activity proxy without `await` returns a native `Promise`; in Python, `workflow.execute_activity()` returns an awaitable; and in Go, `workflow.ExecuteActivity()` returns a `Future`.
The Workflow then waits for all of them to complete and collects the results.

### Controlled parallelism

The following implementation limits the number of concurrent Activities by processing items in batches:

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import process

@workflow.defn
class BatchWorkflow:
    @workflow.run
    async def run(self, items: list[str], max_parallel: int) -> list[str]:
        results: list[str] = []

        for i in range(0, len(items), max_parallel):
            batch = items[i : i + max_parallel]
            batch_results = await asyncio.gather(
                *[
                    workflow.execute_activity(
                        process,
                        item,
                        start_to_close_timeout=timedelta(seconds=30),
                    )
                    for item in batch
                ]
            )
            results.extend(batch_results)

        return results
```

**Go**

```go
// batch_workflow.go
func ProcessBatch(ctx workflow.Context, items []string, maxParallel int) ([]string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 30 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	var results []string

	for i := 0; i < len(items); i += maxParallel {
		end := i + maxParallel
		if end > len(items) {
			end = len(items)
		}
		batch := items[i:end]

		futures := make([]workflow.Future, len(batch))
		for j, item := range batch {
			futures[j] = workflow.ExecuteActivity(ctx, Process, item)
		}

		for _, future := range futures {
			var result string
			if err := future.Get(ctx, &result); err != nil {
				return nil, err
			}
			results = append(results, result)
		}
	}

	return results, nil
}
```

**Java**

```java
// BatchWorkflowImpl.java
public class BatchWorkflowImpl implements BatchWorkflow {
  private final ProcessingActivity activity =
    Workflow.newActivityStub(ProcessingActivity.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .build());

  @Override
  public BatchResult processBatch(List<String> items, int maxParallel) {
    List<String> results = new ArrayList<>();

    for (int i = 0; i < items.size(); i += maxParallel) {
      int end = Math.min(i + maxParallel, items.size());
      List<String> batch = items.subList(i, end);

      List<Promise<String>> promises = batch.stream()
        .map(item -> Async.function(activity::process, item))
        .collect(Collectors.toList());

      Promise.allOf(promises).get();
      results.addAll(promises.stream().map(Promise::get).collect(Collectors.toList()));
    }

    return new BatchResult(results);
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { process } = proxyActivities<typeof activities>({
  startToCloseTimeout: '30s',
});

export async function processBatch(
  items: string[],
  maxParallel: number
): Promise<string[]> {
  const results: string[] = [];

  for (let i = 0; i < items.length; i += maxParallel) {
    const batch = items.slice(i, i + maxParallel);
    const batchResults = await Promise.all(batch.map((item) => process(item)));
    results.push(...batchResults);
  }

  return results;
}
```

The Workflow processes items in chunks of `maxParallel`.
Each chunk runs in parallel, and the Workflow waits for the entire chunk to complete before starting the next one.
This prevents overwhelming Workers or external services.

### Error handling

The following implementation wraps each Activity in error handling so that individual failures do not prevent other Activities from completing:

**Python**

```python
# workflows.py
import asyncio
from dataclasses import dataclass
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import process

@dataclass
class Result:
    item: str
    output: str | None = None
    error: str | None = None

@workflow.defn
class ResilientParallelWorkflow:
    @workflow.run
    async def run(self, items: list[str]) -> list[Result]:
        tasks = [
            workflow.execute_activity(
                process,
                item,
                start_to_close_timeout=timedelta(seconds=30),
            )
            for item in items
        ]
        outcomes = await asyncio.gather(*tasks, return_exceptions=True)

        results: list[Result] = []
        for item, outcome in zip(items, outcomes):
            if isinstance(outcome, BaseException):
                results.append(Result(item=item, error=str(outcome)))
            else:
                results.append(Result(item=item, output=outcome))
        return results
```

**Go**

```go
// resilient_parallel_workflow.go
func ProcessWithErrorHandling(ctx workflow.Context, items []string) ([]Result, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 30 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	futures := make([]workflow.Future, len(items))
	for i, item := range items {
		futures[i] = workflow.ExecuteActivity(ctx, Process, item)
	}

	results := make([]Result, len(items))
	for i, future := range futures {
		var output string
		if err := future.Get(ctx, &output); err != nil {
			results[i] = Result{Item: items[i], Error: err.Error()}
		} else {
			results[i] = Result{Item: items[i], Output: output}
		}
	}
	return results, nil
}
```

**Java**

```java
// ResilientParallelWorkflowImpl.java
public class ResilientParallelWorkflowImpl implements ParallelWorkflow {
  private final ProcessingActivity activity =
    Workflow.newActivityStub(ProcessingActivity.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .build());

  @Override
  public ProcessingReport processWithErrorHandling(List<String> items) {
    List<Promise<Result>> promises = new ArrayList<>();

    for (String item : items) {
      Promise<Result> promise = Async.function(() -> {
        try {
          return activity.process(item);
        } catch (Exception e) {
          return Result.failed(item, e.getMessage());
        }
      });
      promises.add(promise);
    }

    Promise.allOf(promises).get();

    List<Result> results = promises.stream().map(Promise::get).collect(Collectors.toList());
    return new ProcessingReport(results);
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { process } = proxyActivities<typeof activities>({
  startToCloseTimeout: '30s',
});

interface Result {
  item: string;
  output?: string;
  error?: string;
}

export async function processWithErrorHandling(items: string[]): Promise<Result[]> {
  const settled = await Promise.allSettled(items.map((item) => process(item)));

  return settled.map((outcome, i) => {
    if (outcome.status === 'fulfilled') {
      return { item: items[i], output: outcome.value };
    }
    return { item: items[i], error: String(outcome.reason) };
  });
}
```

Each Activity handles its own errors so that the Workflow can collect results from all Activities, including those that failed.
In TypeScript, `Promise.allSettled()` is especially convenient for this pattern.
In Python, `asyncio.gather()` with `return_exceptions=True` captures errors alongside successes.
In Go, each `Future.Get()` call is checked individually for errors.

## When to use

The Parallel Execution pattern is a good fit for processing independent items in batches, calling multiple external services simultaneously, fan-out/fan-in patterns, parallel data transformations, concurrent API requests, and multi-step pipelines with independent stages.

It is not a good fit for operations with dependencies between them, resource-constrained environments (use controlled parallelism), operations requiring strict ordering, or a single fast operation (the overhead is not worth it).

## Benefits and trade-offs

Parallel execution reduces total execution time dramatically and maximizes Worker and external service usage.
Temporal handles retries and failures per operation, and you do not need manual thread management or synchronization.

The trade-offs to consider are that more concurrent operations require more Workers.
Error handling across parallel operations is harder.
Parallel execution makes tracing more difficult.
You may overwhelm external services without throttling, and storing many Futures or Promises consumes Workflow memory.

## Comparison with alternatives

| Approach | Parallelism | Complexity | Control | Use case |
| :--- | :--- | :--- | :--- | :--- |
| Async Activities | High | Low | Medium | Independent operations |
| Sequential | None | Very Low | Full | Dependent operations |
| Child Workflows | High | Medium | High | Complex sub-processes |
| ContinueAsNew | None | Medium | Full | Large iterations |

## Best practices

- **Limit concurrency.** Use batching to avoid overwhelming Workers or external services.
- **Handle failures.** Wrap operations in error handling or use Activity retry policies.
- **Set timeouts.** Configure appropriate Activity timeouts for parallel operations.
- **Monitor resources.** Ensure sufficient Workers for the desired parallelism.
- **Aggregate carefully.** Consider memory when collecting large result sets.
- **Use Child Workflows.** For complex parallel operations with their own state.
- **Test scalability.** Verify performance with realistic parallel loads.
- **Rate limit.** Implement throttling for external API calls.
- **Support partial results.** Consider returning partial results on some failures.
- **Avoid premature blocking.** Schedule all Activities before waiting for any results.

## Common pitfalls

- **Exceeding the pending Activities limit.** A single Workflow Execution can have at most 2,000 pending (concurrently running) Activities. Scheduling more causes Workflow Task failures. Batch Activities or use child Workflows for higher concurrency.
- **Ignoring errors from individual Activities.** Waiting for all results (for example, `Promise.allOf()` in Java, `Promise.all()` in TypeScript, `asyncio.gather()` in Python) fails on the first error by default. If you need partial results, catch errors inside each async function or use `Promise.allSettled()` / `return_exceptions=True` / per-Future error checking.
- **Blowing the 4 MB gRPC message limit.** Scheduling hundreds of Activities in a single Workflow Task can exceed the 4 MB gRPC message size limit if their combined inputs are large. Batch scheduling across multiple Workflow Tasks.
- **Not using Continue-As-New for large fan-outs.** Each Activity adds events to history. Hundreds of parallel Activities can quickly approach the 51,200 event limit. Use Continue-As-New or child Workflows to partition work.

## Related

### Patterns

- **[Child Workflows](/design-patterns/child-workflows)**: For complex parallel operations with their own state.
- **[Saga Pattern](/design-patterns/saga-pattern)**: Parallel operations with compensation.

### References

- [Blob Size Limit Error](/troubleshooting/blob-size-limit-error): Diagnosing and fixing the 4 MB gRPC message limit this pattern's pitfalls mention.
- [Workflow Execution limits](/workflow-execution/limits): Canonical reference for the pending-operations limits — 2,000 by default for each of Activities, Child Workflows, Signals, and Cancellation requests independently, not a single 2,000 cap shared across all of them.

### Sample code

**Java:**
- [HelloParallelActivity](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloParallelActivity.java) — Basic parallel Activity execution.
- [HelloAsync](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloAsync.java) — Async execution with Promises.
- [Sliding Window Batch](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/slidingwindow) — Controlled parallel Child Workflows.

**TypeScript:**
- [activities-examples](https://github.com/temporalio/samples-typescript/tree/main/activities-examples) — Activity patterns including parallel execution with `Promise.all()`.

**Python:**
- [hello_parallel_activity](https://github.com/temporalio/samples-python/blob/main/hello/hello_parallel_activity.py) — Basic parallel Activity execution with `asyncio.gather()`.

**Go:**
- [splitmerge-future](https://github.com/temporalio/samples-go/tree/main/splitmerge-future) — Parallel Activity execution with Futures.
- [splitmerge-selector](https://github.com/temporalio/samples-go/tree/main/splitmerge-selector) — Parallel Activities with Selector for first-completion handling.

---

# Performance & Latency Patterns

Source: https://docs.temporal.io/design-patterns/performance-latency-patterns

> Pattern selection guide for reducing Workflow latency, with a comparison of the round-trips each pattern removes and their combined effect.

Temporal Workflows are durable and reliable, but a default implementation—using regular Activities scheduled through the Temporal server—carries inherent latency. Each regular Activity incurs multiple server round-trips, and each new Workflow begins with a Matching Service routing step. On Temporal Cloud, this baseline can reach 850 ms or more for a typical three-Activity workflow.

This section covers three complementary patterns that each target a different source of latency. They can be applied individually or combined depending on your requirements.

## Latency sources in a typical Workflow

```mermaid
flowchart LR
    A[Client\nExecuteWorkflow] --> B[Matching Service\nroutes first WFT]
    B --> C[Worker\nexecutes WFT]
    C --> D[Server\nschedules Activity]
    D --> E[Worker\nexecutes Activity]
    E --> F[Server\nrecords completion]
    F --> G[Worker\nresumes WFT]
    G --> H[Repeat per\nActivity]
```

| Source | Overhead | Pattern that removes it |
|---|---|---|
| Matching Service (first Workflow Task) | ~30–50 ms | [Eager Workflow Start](/design-patterns/eager-workflow-start) |
| Activity scheduling round-trip | ~50 ms per Activity | [Local Activities](/design-patterns/local-activities) |
| Client waiting for full workflow | Total duration | [Early Return](/design-patterns/early-return) |

## Pattern comparison

The numbers below are approximate benchmarks based on a three-Activity transaction workflow running on Temporal Cloud. Actual results vary by region, Activity implementation, and server load.

| Pattern | First Response | Total Latency | SDK Support |
|---|---|---|---|
| Baseline (regular Activities) | ~850 ms | ~850 ms | All |
| [Early Return](/design-patterns/early-return) | ~265 ms | ~850 ms | All |
| [Local Activities](/design-patterns/local-activities) | ~275 ms | ~275 ms | All |
| [Early Return + Local Activities](/design-patterns/early-return-local-activities) | ~160 ms | ~275 ms | All |
| [Eager Workflow Start](/design-patterns/eager-workflow-start) + Local Activities | ~265 ms | ~265 ms | Go, Java, Python, TypeScript, .NET |
| Early Return + Local Activities + Eager Start | ~160 ms | ~265 ms | Go, Java, Python, TypeScript, .NET |

**First Response** is the time until the client receives an actionable result. **Total Latency** is the time until the Workflow fully completes.

## Patterns in this section

- [Local Activities](/design-patterns/local-activities): Local Activities run inside the Worker process, skipping server round-trips for short, idempotent Activities on a latency-sensitive path.
- [Early Return + Local Activities](/design-patterns/early-return-local-activities): Extends Early Return by running Phase 1 as Local Activities, so the client's first response comes from in-process work, not a server call.
- [Eager Workflow Start](/design-patterns/eager-workflow-start): Eager Workflow Start sends the first Workflow Task directly to a co-located Worker, skipping the Matching Service to cut startup latency.

## Choosing a pattern

**You only care about total workflow latency** (not first-response time): use [Local Activities](/design-patterns/local-activities). If co-location is feasible, add [Eager Workflow Start](/design-patterns/eager-workflow-start) for the maximum reduction.

**You care most about first-response latency**: use [Early Return + Local Activities](/design-patterns/early-return-local-activities). The client gets its response in ~160 ms; background work continues independently.

**You want to start simple**: begin with [Local Activities](/design-patterns/local-activities). It requires minimal structural change and provides the most straightforward per-Activity improvement.

## Related sections

- [Distributed Transaction Patterns](/design-patterns/distributed-transaction-patterns) — the [Early Return](/design-patterns/early-return) pattern lives there, describing the Update-with-Start mechanism in detail
- [Worker Configuration Patterns](/design-patterns/worker-configuration-patterns) — tuning Worker concurrency and task queue assignments that affect throughput
- [QoS & Throughput Patterns](/design-patterns/qos-throughput-patterns) — rate limiting and fairness patterns for high-volume workloads

---

# Pick First Pattern

Source: https://docs.temporal.io/design-patterns/pick-first

> Races multiple Activities in parallel, returns the first completed result, and cancels the remaining Activities using SDK cancellation scopes.

## Overview

The Pick First pattern executes multiple Activities in parallel and returns the result of whichever completes first, then cancels the remaining Activities.
It is suitable for racing multiple approaches to the same task, implementing timeout alternatives, or optimizing for fastest response when multiple options are available.

## Problem

In distributed systems, you often need Workflows that execute multiple Activities that can accomplish the same goal, return as soon as any one succeeds (fastest wins), cancel remaining Activities to avoid wasted resources, and handle scenarios where speed matters more than trying all options.

Without the Pick First pattern, you must wait for all Activities to complete even when only one result is needed, manually track which Activity finished first, implement complex cancellation logic for remaining Activities, and waste compute resources on Activities whose results will not be used.

## Solution

The Pick First pattern races multiple Activities simultaneously, captures the first result, then cancels remaining Activities using a cancellation mechanism provided by each SDK.

```mermaid
sequenceDiagram
    participant Workflow
    participant Activity1
    participant Activity2
    participant Activity3

    Workflow->>+Activity1: Start (shared ctx)
    Workflow->>+Activity2: Start (shared ctx)
    Workflow->>+Activity3: Start (shared ctx)
    Note over Workflow: Selector waits for first

    par Race
        Activity1->>Activity1: Execute (slow)
        Activity2->>Activity2: Execute (fast)
        Activity3->>Activity3: Execute (medium)
    end

    Activity2-->>Workflow: Result (FIRST!)
    Note over Workflow: Selector returns
    Workflow->>Workflow: cancelHandler()

    Workflow->>Activity1: Cancel
    Workflow->>Activity3: Cancel
    Activity1-->>-Workflow: Cancelled
    Activity3-->>-Workflow: Cancelled
    deactivate Activity2
```

The following describes each step in the diagram:

1. The Workflow starts three Activities in parallel using a shared cancellable context.
2. The Workflow waits for the first Activity to complete.
3. Activity 2 completes first. The Workflow captures its result.
4. The Workflow cancels the shared context, which cancels Activities 1 and 3.

The following implementation shows the core pattern.
The Workflow creates a cancellable context, starts two Activities, and captures the first result:

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import sample_activity

@workflow.defn
class PickFirstWorkflow:
    @workflow.run
    async def run(self) -> str:
        task1 = asyncio.create_task(
            workflow.execute_activity(
                sample_activity,
                "option1",
                start_to_close_timeout=timedelta(minutes=2),
                heartbeat_timeout=timedelta(seconds=10),
            )
        )
        task2 = asyncio.create_task(
            workflow.execute_activity(
                sample_activity,
                "option2",
                start_to_close_timeout=timedelta(minutes=2),
                heartbeat_timeout=timedelta(seconds=10),
            )
        )

        done, pending = await workflow.wait(
            [task1, task2], return_when=asyncio.FIRST_COMPLETED
        )

        for task in pending:
            task.cancel()

        return done.pop().result()
```

**Go**

```go
// workflow.go
func PickFirstWorkflow(ctx workflow.Context) (string, error) {
  selector := workflow.NewSelector(ctx)
  var firstResponse string

  childCtx, cancelHandler := workflow.WithCancel(ctx)
  childCtx = workflow.WithActivityOptions(childCtx, activityOptions)

  f1 := workflow.ExecuteActivity(childCtx, Activity, "option1")
  f2 := workflow.ExecuteActivity(childCtx, Activity, "option2")

  selector.AddFuture(f1, func(f workflow.Future) {
    _ = f.Get(ctx, &firstResponse)
  }).AddFuture(f2, func(f workflow.Future) {
    _ = f.Get(ctx, &firstResponse)
  })

  selector.Select(ctx) // Blocks until first completes
  cancelHandler()      // Cancel remaining activities

  return firstResponse, nil
}
```

**Java**

```java
// PickFirstWorkflow.java
public class PickFirstWorkflowImpl implements PickFirstWorkflow {

    private final SampleActivities activities =
        Workflow.newActivityStub(
            SampleActivities.class,
            ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofMinutes(2))
                .setHeartbeatTimeout(Duration.ofSeconds(10))
                .build());

    @Override
    public String pickFirst() {
        List<Promise<String>> results = new ArrayList<>();

        CancellationScope scope =
            Workflow.newCancellationScope(
                () -> {
                    results.add(Async.function(activities::sampleActivity, "option1"));
                    results.add(Async.function(activities::sampleActivity, "option2"));
                });

        scope.run();

        String firstResponse = Promise.anyOf(results).get();

        scope.cancel();

        return firstResponse;
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities, CancellationScope } from '@temporalio/workflow';
import type * as activities from './activities';

const { sampleActivity } = proxyActivities<typeof activities>({
  startToCloseTimeout: '2m',
  heartbeatTimeout: '10s',
});

export async function pickFirstWorkflow(): Promise<string> {
  return await CancellationScope.cancellable(async () => {
    const p1 = sampleActivity('option1');
    const p2 = sampleActivity('option2');
    const firstResponse = await Promise.race([p1, p2]);
    CancellationScope.current().cancel();
    return firstResponse;
  });
}
```

Each SDK provides a different mechanism for racing Activities and cancelling the rest:

- **Go** uses `workflow.NewSelector()` with `AddFuture()` callbacks and `workflow.WithCancel(ctx)` for cancellation.
- **TypeScript** uses `Promise.race()` within a `CancellationScope.cancellable()`. Calling `CancellationScope.current().cancel()` cancels all Activities started in that scope.
- **Python** uses `asyncio.create_task()` to start Activities concurrently, then `workflow.wait()` with `return_when=asyncio.FIRST_COMPLETED` to get the first result. Pending tasks are cancelled explicitly.
- **Java** uses `Async.function()` to start Activities inside a `CancellationScope`, then `Promise.anyOf()` to wait for the first result. Calling `scope.cancel()` cancels the remaining Activities.

## Implementation

### Activity with cancellation support

For the Pick First pattern to work efficiently, Activities must detect cancellation via heartbeats and respond to the cancellation signal provided by their SDK:

**Python**

```python
# activities.py
import asyncio
from temporalio import activity

@activity.defn
async def sample_activity(branch_id: str) -> str:
    try:
        for elapsed in range(60):
            await asyncio.sleep(1)
            activity.heartbeat("status-report")
        return f"Branch {branch_id} completed"
    except asyncio.CancelledError:
        activity.logger.info(f"Branch {branch_id} cancelled")
        raise
```

**Go**

```go
// activity.go
func SampleActivity(ctx context.Context, branchID int, duration time.Duration) (string, error) {
  logger := activity.GetLogger(ctx)
  elapsed := time.Nanosecond

  for elapsed < duration {
    time.Sleep(time.Second)
    elapsed += time.Second

    activity.RecordHeartbeat(ctx, "status-report")

    select {
    case <-ctx.Done():
      msg := fmt.Sprintf("Branch %d cancelled", branchID)
      logger.Info(msg)
      return msg, ctx.Err()
    default:
      // Continue working
    }
  }

  return fmt.Sprintf("Branch %d completed", branchID), nil
}
```

**Java**

```java
// SampleActivityImpl.java
public class SampleActivityImpl implements SampleActivities {

    @Override
    public String sampleActivity(String branchID) {
        for (int elapsed = 0; elapsed < 60; elapsed++) {
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                throw Activity.wrap(e);
            }
            Activity.getExecutionContext().heartbeat("status-report");
            // Heartbeat throws CanceledFailure if cancellation was requested
        }
        return "Branch " + branchID + " completed";
    }
}
```

**TypeScript**

```typescript
// activities.ts
import { Context, heartbeat } from '@temporalio/activity';

export async function sampleActivity(branchID: string): Promise<string> {
  for (let elapsed = 0; elapsed < 60; elapsed++) {
    await new Promise((resolve) => setTimeout(resolve, 1000));
    heartbeat('status-report');
    Context.current().cancellationSignal.throwIfAborted();
  }
  return `Branch ${branchID} completed`;
}
```

The Activity heartbeats on each iteration, which allows the Temporal Server to deliver cancellation notifications promptly.
When cancellation is detected, the Activity performs any necessary cleanup and exits.

### Wait for cancellation completion

The following implementation waits for all Activities to finish their cleanup before returning:

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

with workflow.unsafe.imports_passed_through():
    from activities import sample_activity

@workflow.defn
class PickFirstWithCleanup:
    @workflow.run
    async def run(self) -> str:
        task1 = asyncio.create_task(
            workflow.execute_activity(
                sample_activity,
                "branch1",
                start_to_close_timeout=timedelta(minutes=2),
                heartbeat_timeout=timedelta(seconds=10),
                cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,
            )
        )
        task2 = asyncio.create_task(
            workflow.execute_activity(
                sample_activity,
                "branch2",
                start_to_close_timeout=timedelta(minutes=2),
                heartbeat_timeout=timedelta(seconds=10),
                cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,
            )
        )

        done, pending = await workflow.wait(
            [task1, task2], return_when=asyncio.FIRST_COMPLETED
        )

        for task in pending:
            task.cancel()

        # Wait for all activities to finish cancellation
        for task in pending:
            try:
                await task
            except asyncio.CancelledError:
                pass

        return done.pop().result()
```

**Go**

```go
// workflow.go
func PickFirstWithCleanup(ctx workflow.Context) (string, error) {
  selector := workflow.NewSelector(ctx)
  var firstResponse string

  childCtx, cancelHandler := workflow.WithCancel(ctx)
  childCtx = workflow.WithActivityOptions(childCtx, workflow.ActivityOptions{
    StartToCloseTimeout: 2 * time.Minute,
    WaitForCancellation: true,
  })

  f1 := workflow.ExecuteActivity(childCtx, Activity, "branch1")
  f2 := workflow.ExecuteActivity(childCtx, Activity, "branch2")
  pendingFutures := []workflow.Future{f1, f2}

  selector.AddFuture(f1, func(f workflow.Future) {
    _ = f.Get(ctx, &firstResponse)
  }).AddFuture(f2, func(f workflow.Future) {
    _ = f.Get(ctx, &firstResponse)
  })

  selector.Select(ctx)
  cancelHandler()

  // Wait for all activities to finish cancellation
  for _, f := range pendingFutures {
    _ = f.Get(ctx, nil)
  }

  return firstResponse, nil
}
```

**Java**

```java
// PickFirstWithCleanupImpl.java
public class PickFirstWithCleanupImpl implements PickFirstWorkflow {

    private final SampleActivities activities =
        Workflow.newActivityStub(
            SampleActivities.class,
            ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofMinutes(2))
                .setHeartbeatTimeout(Duration.ofSeconds(10))
                .setCancellationType(
                    ActivityCancellationType.WAIT_CANCELLATION_COMPLETED)
                .build());

    @Override
    public String pickFirst() {
        List<Promise<String>> results = new ArrayList<>();

        CancellationScope scope =
            Workflow.newCancellationScope(
                () -> {
                    results.add(Async.function(activities::sampleActivity, "branch1"));
                    results.add(Async.function(activities::sampleActivity, "branch2"));
                });

        scope.run();

        String firstResponse = Promise.anyOf(results).get();

        scope.cancel();

        // Wait for all activities to finish cancellation
        for (Promise<String> activityResult : results) {
            try {
                activityResult.get();
            } catch (ActivityFailure e) {
                if (!(e.getCause() instanceof CanceledFailure)) {
                    throw e;
                }
            }
        }

        return firstResponse;
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import {
  proxyActivities,
  CancellationScope,
  ActivityCancellationType,
  isCancellation,
} from '@temporalio/workflow';
import type * as activities from './activities';

const { sampleActivity } = proxyActivities<typeof activities>({
  startToCloseTimeout: '2m',
  heartbeatTimeout: '10s',
  cancellationType: ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,
});

export async function pickFirstWithCleanup(): Promise<string> {
  return await CancellationScope.cancellable(async () => {
    const p1 = sampleActivity('branch1');
    const p2 = sampleActivity('branch2');
    const firstResponse = await Promise.race([p1, p2]);
    CancellationScope.current().cancel();

    // Wait for all activities to finish cancellation
    const results = [p1, p2];
    for (const p of results) {
      try {
        await p;
      } catch (err) {
        if (!isCancellation(err)) throw err;
      }
    }

    return firstResponse;
  });
}
```

Each SDK provides a way to wait for cancelled Activities to finish their cleanup:

- **Go** sets `WaitForCancellation: true` in the Activity options, then calls `Get` on all futures after cancelling.
- **TypeScript** sets `cancellationType: ActivityCancellationType.WAIT_CANCELLATION_COMPLETED` in the Activity options, then awaits all promises while catching cancellation errors with `isCancellation()`.
- **Python** sets `cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED`, then awaits pending tasks while catching `asyncio.CancelledError`.
- **Java** sets `setCancellationType(ActivityCancellationType.WAIT_CANCELLATION_COMPLETED)`, then calls `get()` on all promises while catching `ActivityFailure` with a `CanceledFailure` cause.

## When to use

The Pick First pattern is a good fit for racing multiple data sources (primary vs backup), trying multiple algorithms and picking the fastest, implementing fallback strategies with timeout, optimizing for latency when multiple options exist, and testing multiple service endpoints for fastest response.

It is not a good fit when you need results from all Activities (use parallel execution), Activities have side effects that should not be cancelled, order matters (use sequential execution), or all Activities must complete.

## Benefits and trade-offs

The pattern returns as soon as the fastest option completes, optimizing for latency.
Unnecessary work is cancelled automatically.
Each SDK's race mechanism ensures replay consistency, and cancellation cleanup is handled properly.

The trade-offs to consider are that cancelled Activities may have done partial work.
Activities need heartbeats to detect cancellation quickly.
Activities do not cancel instantly (they wait for the next heartbeat).
You must implement proper cancellation handling in Activities.
Only the first result is used; others are discarded.

## Comparison with alternatives

| Approach | Returns first | Cancels others | Complexity | Use case |
| :--- | :--- | :--- | :--- | :--- |
| Pick First | Yes | Yes | Medium | Race for fastest |
| Parallel Execution | No | No | Low | All must complete |
| Sequential | No | N/A | Low | Order matters |
| Split/Merge | No | No | Medium | Aggregate results |

## Best practices

- **Use heartbeats.** Activities must heartbeat to detect cancellation quickly.
- **Configure cancellation wait behavior.** Decide if the Workflow should wait for cleanup to complete before returning.
- **Handle cancellation in Activities.** Activities must check for cancellation signals and exit cleanly.
- **Use a shared cancellable context.** Use a single cancellable context or scope for all raced Activities.
- **Track futures or tasks.** Keep references to all futures or tasks if waiting for cleanup.
- **Set Activity timeouts.** Configure appropriate StartToCloseTimeout and HeartbeatTimeout.
- **Log cancellations.** Log when Activities are cancelled for observability.
- **Design idempotent Activities.** Ensure Activities handle cancellation safely.

## Common pitfalls

- **Missing heartbeats in Activities.** Activities must heartbeat to detect cancellation. Without heartbeats, cancelled Activities continue running until their StartToCloseTimeout expires, wasting resources.
- **Not waiting for cancellation cleanup.** Without configuring the cancellation type to wait for completion (for example, `WaitForCancellation: true` in Go, `WAIT_CANCELLATION_COMPLETED` in other SDKs), fetching a cancelled Activity's result returns a cancellation error immediately, before the Activity has finished cleanup. Configure this setting if you need to wait for cleanup to complete.
- **Ignoring errors from the winning Activity.** The first Activity to complete might return an error. Always check the result for errors rather than assuming success.
- **Forgetting to cancel remaining Activities.** If you forget to cancel the shared context or scope after receiving the first result, the remaining Activities continue running indefinitely.

## Related

### Patterns

- **[Parallel Execution](/design-patterns/parallel-execution)**: Execute in parallel and combine all results.

### References

- [Selectors (Go)](/develop/go/workflows/selectors): The Go SDK's `Selector` construct for racing multiple Futures and Channels, with the same "cancel the losers" technique.

### Sample code

- [Go Sample](https://github.com/temporalio/samples-go/tree/main/pickfirst) — Complete implementation with Worker and starter.
- [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/activities-cancellation-heartbeating) — Activities with cancellation and heartbeating.
- [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello) — Hello samples including cancellation scope patterns.
- [Python Sample](https://github.com/temporalio/samples-python) — Python SDK samples with async patterns.

---

# Polling External Services

Source: https://docs.temporal.io/design-patterns/polling

> Strategies for polling external resources with varying frequencies: frequent, infrequent, and periodic patterns.

## Overview

The Polling External Services pattern implements strategies for periodically checking external systems until a desired state is reached.
It enables Workflows to wait for asynchronous operations in third-party services that do not support callbacks, making it essential for integrating with REST APIs, job queues, and batch processing systems.

## Problem

Workflows often need to wait on an external system that has no way to push a notification back: a REST API with no webhook, a long-running job in a third-party system, a status check that has to be repeated until it changes. Doing that well means picking the right poll frequency for the situation without overwhelming the external service.

Without a deliberate polling strategy, you're left building retry logic by hand, choosing between responsiveness and resource efficiency with no way to have both, and managing heartbeating and timeouts yourself — while Event history grows with every poll.

## Solution

You can use Temporal to implement three distinct polling strategies, each optimized for different polling frequencies and requirements:

1. **Frequent Polling (1 second or faster)**: Loop inside an Activity with heartbeats.
2. **Infrequent Polling (1 minute or slower)**: Use Activity retries with fixed backoff.
3. **Periodic Sequence**: Use Child Workflows for complex polling sequences.

```mermaid
flowchart TD
    Start([Polling Required]) --> Freq{Polling<br/>Frequency?}

    Freq -->|≤1 second| Fast[Frequent Polling]
    Freq -->|≥1 minute| Slow[Infrequent Polling]
    Freq -->|Complex| Complex[Periodic Sequence]

    Fast --> FastImpl[Activity with loop<br/>+ heartbeats]
    Slow --> SlowImpl[Activity retries<br/>backoffCoefficient=1]
    Complex --> ComplexImpl[Child workflow<br/>+ Continue-As-New]
```

The following describes each path in the diagram:

1. If you need polling at 1-second intervals or faster, use frequent polling with an Activity loop and heartbeats.
2. If you need polling at 1-minute intervals or slower, use infrequent polling with Activity retries and a fixed backoff coefficient.
3. If you need complex multi-step polling or changing parameters between attempts, use a periodic sequence with Child Workflows and Continue-As-New.

## Implementation

### Frequent polling (fast response required)

For polling intervals of 1 second or faster, implement the polling loop inside the Activity with heartbeats.
The heartbeat reports progress and enables Temporal to detect stuck Activities:

**Python**

```python
# activities.py
from temporalio import activity
import asyncio

@activity.defn
async def do_poll() -> str:
    while True:
        activity.heartbeat()

        result = await external_service.check_status()

        if result == "COMPLETED":
            return result

        await asyncio.sleep(1)
```

**Go**

```go
// activities.go
func DoPoll(ctx context.Context) (string, error) {
	for {
		activity.RecordHeartbeat(ctx)

		result, err := externalService.CheckStatus()
		if err != nil {
			return "", err
		}

		if result == "COMPLETED" {
			return result, nil
		}

		select {
		case <-ctx.Done():
			return "", ctx.Err()
		case <-time.After(1 * time.Second):
		}
	}
}
```

**Java**

```java
// FrequentPollingActivityImpl.java
@ActivityInterface
public interface PollingActivities {
  String doPoll();
}

public class FrequentPollingActivityImpl implements PollingActivities {
  @Override
  public String doPoll() {
    while (true) {
      Activity.getExecutionContext().heartbeat(null);

      String result = externalService.checkStatus();

      if (result.equals("COMPLETED")) {
        return result;
      }

      try {
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        throw Activity.wrap(e);
      }
    }
  }
}
```

**TypeScript**

```typescript
// activities.ts
import { heartbeat, sleep } from '@temporalio/activity';

export async function doPoll(): Promise<string> {
  while (true) {
    heartbeat();

    const result = await externalService.checkStatus();

    if (result === 'COMPLETED') {
      return result;
    }

    await sleep('1s');
  }
}
```

The Activity loops indefinitely, heartbeating on each iteration.
If the Worker crashes, the heartbeat timeout expires and Temporal retries the Activity on another Worker.

The Workflow configures the Activity with a heartbeat timeout shorter than the start-to-close timeout:

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import do_poll

@workflow.defn
class FrequentPollingWorkflow:
    @workflow.run
    async def run(self) -> str:
        return await workflow.execute_activity(
            do_poll,
            start_to_close_timeout=timedelta(seconds=60),
            heartbeat_timeout=timedelta(seconds=2),
        )
```

**Go**

```go
// workflow.go
func FrequentPollingWorkflow(ctx workflow.Context) (string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 60 * time.Second,
		HeartbeatTimeout:    2 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	var result string
	err := workflow.ExecuteActivity(ctx, DoPoll).Get(ctx, &result)
	return result, err
}
```

**Java**

```java
// FrequentPollingWorkflowImpl.java
public class FrequentPollingWorkflowImpl implements PollingWorkflow {
  @Override
  public String exec() {
    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(60))
        .setHeartbeatTimeout(Duration.ofSeconds(2))
        .build();

    PollingActivities activities = Workflow.newActivityStub(PollingActivities.class, options);
    return activities.doPoll();
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { doPoll } = proxyActivities<typeof activities>({
  startToCloseTimeout: '60s',
  heartbeatTimeout: '2s',
});

export async function frequentPollingWorkflow(): Promise<string> {
  return await doPoll();
}
```

The heartbeat timeout (2 seconds) is shorter than the start-to-close timeout (60 seconds).
If the Activity misses a heartbeat, Temporal detects the failure and retries the Activity.

### Infrequent polling (resource efficient)

For polling intervals of 1 minute or slower, use Activity retries with a backoff coefficient of 1.
The Activity throws an exception when the external service is not ready, and Temporal retries after the configured interval:

**Python**

```python
# activities.py
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def do_poll() -> str:
    result = await external_service.check_status()

    if result != "COMPLETED":
        raise ApplicationError("Service not ready, will retry")

    return result
```

**Go**

```go
// activities.go
func DoPoll(ctx context.Context) (string, error) {
	result, err := externalService.CheckStatus()
	if err != nil {
		return "", err
	}

	if result != "COMPLETED" {
		return "", fmt.Errorf("service not ready, will retry")
	}

	return result, nil
}
```

**Java**

```java
// InfrequentPollingActivityImpl.java
public class InfrequentPollingActivityImpl implements PollingActivities {
  @Override
  public String doPoll() {
    String result = externalService.checkStatus();

    if (!result.equals("COMPLETED")) {
      throw new RuntimeException("Service not ready, will retry");
    }

    return result;
  }
}
```

**TypeScript**

```typescript
// activities.ts
import { ApplicationFailure } from '@temporalio/activity';

export async function doPoll(): Promise<string> {
  const result = await externalService.checkStatus();

  if (result !== 'COMPLETED') {
    throw ApplicationFailure.retryable('Service not ready, will retry');
  }

  return result;
}
```

The Activity performs a single poll and throws if the service is not ready.
Temporal handles the retry scheduling.

The Workflow configures the retry policy with a fixed interval:

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

with workflow.unsafe.imports_passed_through():
    from activities import do_poll

@workflow.defn
class InfrequentPollingWorkflow:
    @workflow.run
    async def run(self) -> str:
        return await workflow.execute_activity(
            do_poll,
            start_to_close_timeout=timedelta(seconds=2),
            retry_policy=RetryPolicy(
                backoff_coefficient=1,
                initial_interval=timedelta(seconds=60),
            ),
        )
```

**Go**

```go
// workflow.go
func InfrequentPollingWorkflow(ctx workflow.Context) (string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 2 * time.Second,
		RetryPolicy: &temporal.RetryPolicy{
			BackoffCoefficient: 1,
			InitialInterval:    60 * time.Second,
		},
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	var result string
	err := workflow.ExecuteActivity(ctx, DoPoll).Get(ctx, &result)
	return result, err
}
```

**Java**

```java
// InfrequentPollingWorkflowImpl.java
public class InfrequentPollingWorkflowImpl implements PollingWorkflow {
  @Override
  public String exec() {
    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(2))
        .setRetryOptions(
            RetryOptions.newBuilder()
                .setBackoffCoefficient(1)
                .setInitialInterval(Duration.ofSeconds(60))
                .build())
        .build();

    PollingActivities activities = Workflow.newActivityStub(PollingActivities.class, options);
    return activities.doPoll();
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { doPoll } = proxyActivities<typeof activities>({
  startToCloseTimeout: '2s',
  retry: {
    backoffCoefficient: 1,
    initialInterval: '60s',
  },
});

export async function infrequentPollingWorkflow(): Promise<string> {
  return await doPoll();
}
```

Setting the backoff coefficient to 1 creates a fixed retry interval.
The initial interval of 60 seconds sets the polling frequency.
Retries do not add events to the Workflow history, keeping it small.

### Periodic sequence (complex polling)

For polling that requires multiple Activities or changing parameters between attempts, use Child Workflows with Continue-As-New.
The Child Workflow polls in a loop and calls Continue-As-New to prevent unbounded history:

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import do_poll

@workflow.defn
class PollingChildWorkflow:
    @workflow.run
    async def run(self, polling_interval_seconds: int) -> str:
        max_attempts = 10

        for _ in range(max_attempts):
            result = await workflow.execute_activity(
                do_poll,
                start_to_close_timeout=timedelta(seconds=10),
            )

            if result == "COMPLETED":
                return result

            await workflow.sleep(polling_interval_seconds)

        # Continue-as-new to prevent unbounded history
        workflow.continue_as_new(polling_interval_seconds)
```

**Go**

```go
// workflow.go
func PollingChildWorkflow(ctx workflow.Context, pollingIntervalSeconds int) (string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	maxAttempts := 10
	for i := 0; i < maxAttempts; i++ {
		var result string
		err := workflow.ExecuteActivity(ctx, DoPoll).Get(ctx, &result)
		if err != nil {
			return "", err
		}

		if result == "COMPLETED" {
			return result, nil
		}

		workflow.Sleep(ctx, time.Duration(pollingIntervalSeconds)*time.Second)
	}

	// Continue-as-new to prevent unbounded history
	return "", workflow.NewContinueAsNewError(ctx, PollingChildWorkflow, pollingIntervalSeconds)
}
```

**Java**

```java
// PeriodicPollingChildWorkflowImpl.java
@WorkflowInterface
public interface PollingChildWorkflow {
  @WorkflowMethod
  String exec(int pollingIntervalInSeconds);
}

public class PeriodicPollingChildWorkflowImpl implements PollingChildWorkflow {
  @Override
  public String exec(int pollingIntervalInSeconds) {
    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(10))
        .build();

    PollingActivities activities = Workflow.newActivityStub(PollingActivities.class, options);

    int maxAttempts = 10;
    for (int i = 0; i < maxAttempts; i++) {
      String result = activities.doPoll();

      if (result.equals("COMPLETED")) {
        return result;
      }

      Workflow.sleep(Duration.ofSeconds(pollingIntervalInSeconds));
    }

    // Continue-as-new to prevent unbounded history
    PollingChildWorkflow continueAsNew = Workflow.newContinueAsNewStub(PollingChildWorkflow.class);
    continueAsNew.exec(pollingIntervalInSeconds);
    return null;
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities, sleep, continueAsNew } from '@temporalio/workflow';
import type * as activities from './activities';

const { doPoll } = proxyActivities<typeof activities>({
  startToCloseTimeout: '10s',
});

export async function pollingChildWorkflow(
  pollingIntervalSeconds: number
): Promise<string> {
  const maxAttempts = 10;

  for (let i = 0; i < maxAttempts; i++) {
    const result = await doPoll();

    if (result === 'COMPLETED') {
      return result;
    }

    await sleep(`${pollingIntervalSeconds}s`);
  }

  // Continue-as-new to prevent unbounded history
  await continueAsNew<typeof pollingChildWorkflow>(pollingIntervalSeconds);
  return ''; // unreachable
}
```

The Child Workflow polls up to 10 times, sleeping between attempts.
After 10 attempts, it calls Continue-As-New to start a fresh execution with the same parameters.

The parent Workflow starts the Child Workflow and waits for its result:

**Python**

```python
# workflows.py
from temporalio import workflow

@workflow.defn
class PeriodicPollingWorkflow:
    @workflow.run
    async def run(self) -> str:
        return await workflow.execute_child_workflow(
            PollingChildWorkflow.run,
            5,
            id="ChildWorkflowPoll",
        )
```

**Go**

```go
// workflow.go
func PeriodicPollingWorkflow(ctx workflow.Context) (string, error) {
	cwo := workflow.ChildWorkflowOptions{
		WorkflowID: "ChildWorkflowPoll",
	}
	ctx = workflow.WithChildOptions(ctx, cwo)

	var result string
	err := workflow.ExecuteChildWorkflow(ctx, PollingChildWorkflow, 5).Get(ctx, &result)
	return result, err
}
```

**Java**

```java
// PeriodicPollingWorkflowImpl.java
public class PeriodicPollingWorkflowImpl implements PollingWorkflow {
  @Override
  public String exec() {
    PollingChildWorkflow childWorkflow = Workflow.newChildWorkflowStub(
        PollingChildWorkflow.class,
        ChildWorkflowOptions.newBuilder()
            .setWorkflowId("ChildWorkflowPoll")
            .build());

    return childWorkflow.exec(5);
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { executeChild } from '@temporalio/workflow';
import { pollingChildWorkflow } from './polling-child-workflow';

export async function periodicPollingWorkflow(): Promise<string> {
  return await executeChild(pollingChildWorkflow, {
    args: [5],
    workflowId: 'ChildWorkflowPoll',
  });
}
```

The parent remains blocked and is unaware of the child's Continue-As-New calls.
When the child completes, the parent receives the result.

## When to use

### Frequent polling (1 second or faster)

This strategy is a good fit for real-time status checks, high-priority operations requiring fast response, and short-lived external operations (minutes, not hours).
It is not a good fit for long-running operations (hours or days), rate-limited APIs, or resource-constrained external services.

### Infrequent polling (1 minute or slower)

This strategy is a good fit for batch job completion checks, long-running external processes, rate-limited APIs, and operations that may take hours or days.
It is not a good fit for sub-minute polling requirements or operations requiring immediate response.

### Periodic sequence

This strategy is a good fit for multi-step polling sequences, changing Activity parameters between polls, and very long-running polls requiring Continue-As-New.
It is not a good fit when the frequent or infrequent patterns are sufficient.

## Benefits and trade-offs

All three strategies work with any external service that does not support callbacks.
Temporal handles retry scheduling and fault tolerance automatically.
All timing is based on Workflow time, ensuring deterministic behavior.

Frequent polling provides fast response but consumes more resources and requires heartbeating.
Infrequent polling is resource-efficient with minimal history growth but has a minimum 1-minute interval.
Periodic sequence is the most flexible but adds complexity through Child Workflow management.

## Comparison with alternatives

| Strategy | Poll frequency | History impact | Complexity | Best for |
| :--- | :--- | :--- | :--- | :--- |
| Frequent Polling | 1 second or faster | Medium | Low | Real-time checks |
| Infrequent Polling | 1 minute or slower | Minimal | Low | Long operations |
| Periodic Sequence | Any | Low (with CAN) | Medium | Complex sequences |
| Workflow Timer Loop | Any | High | Medium | Avoid this approach |

## Best practices

- **Choose the right strategy.** Match polling frequency to the pattern.
- **Set appropriate timeouts.** HeartbeatTimeout must be shorter than StartToCloseTimeout for frequent polling.
- **Handle failures gracefully.** Distinguish transient from permanent failures.
- **Add exponential backoff.** Use backoff for error cases (not normal polling).
- **Implement circuit breakers.** Protect external services from overload.
- **Use Continue-As-New.** Prevent unbounded history in periodic sequences.
- **Monitor polling metrics.** Track poll attempts, success rates, and durations.
- **Respect rate limits.** Adjust polling frequency to API constraints.
- **Add jitter.** Prevent thundering herd when many Workflows poll simultaneously.
- **Consider webhooks.** If the external service supports callbacks, use async completion instead.

## Common pitfalls

- **Wrong pattern choice.** Using frequent polling for hour-long operations wastes resources.
- **Missing heartbeats.** Frequent polling without heartbeats causes delayed failure detection.
- **Unbounded history.** Not using Continue-As-New in periodic sequences leads to history limit failures.
- **Tight polling loops.** Polling too frequently overwhelms external services.
- **No timeout.** Polling indefinitely without max attempts or a deadline risks runaway Workflows.
- **Ignoring errors.** Not distinguishing between retryable and permanent failures leads to wasted retries.
- **Workflow timer loops.** Using Workflow timers instead of proper polling patterns bloats history.

## Related

### Patterns

- **[Long-Running Activity](/design-patterns/long-running-activity)**: Reporting progress in long Activities.
- **[Continue-As-New](/design-patterns/continue-as-new)**: Managing unbounded Workflow history.

### References

- [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat): Canonical reference for heartbeats, heartbeat timeouts, and throttling — the mechanism the frequent-polling variant relies on.

### Sample code

### Java
- [Frequent Polling](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/polling/frequent) — Fast polling with heartbeats.
- [Infrequent Polling](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/polling/infrequent) — Efficient long-interval polling.
- [Periodic Sequence](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/polling/periodicsequence) — Complex polling with Child Workflows.

### TypeScript
- [Frequent Polling](https://github.com/temporalio/samples-typescript/tree/main/polling/frequent) — Fast polling with heartbeats.
- [Infrequent Polling](https://github.com/temporalio/samples-typescript/tree/main/polling/infrequent) — Efficient long-interval polling.
- [Periodic Sequence](https://github.com/temporalio/samples-typescript/tree/main/polling/periodic-sequence) — Complex polling with Child Workflows.

### Python
- [Frequent Polling](https://github.com/temporalio/samples-python/tree/main/polling/frequent) — Fast polling with heartbeats.
- [Infrequent Polling](https://github.com/temporalio/samples-python/tree/main/polling/infrequent) — Efficient long-interval polling.
- [Periodic Sequence](https://github.com/temporalio/samples-python/tree/main/polling/periodic_sequence) — Complex polling with Child Workflows.

### Go
- [Frequent Polling](https://github.com/temporalio/samples-go/tree/main/polling/frequent) — Fast polling with heartbeats.
- [Infrequent Polling](https://github.com/temporalio/samples-go/tree/main/polling/infrequent) — Efficient long-interval polling.
- [Periodic Sequence](https://github.com/temporalio/samples-go/tree/main/polling/periodicsequence) — Complex polling with Child Workflows.

---

# Priority

Source: https://docs.temporal.io/design-patterns/priority-task-queues

> Assigns a priority level to Workflows and Activities so that time-sensitive work dispatches ahead of lower-priority work within a single Task Queue.

> **ℹ️ TLDR:**
> Assign a Priority key from 1 to 5 to Workflows and Activities so **high-priority work dispatches ahead of lower-priority work** on a shared Task Queue. Use this when a flood of batch or background Tasks would otherwise delay high-urgency requests.

## Overview

The Priority pattern assigns a Priority key to Workflows, Activities, and Child Workflows so that time-sensitive work dispatches ahead of lower-priority work within a single Task Queue, without requiring separate queues or routing logic.

Priority applies to dispatch. It does not preempt running Tasks or reserve Worker capacity.

## Problem

In a shared Task Queue, backlogged Tasks are generally dispatched in first-in-first-out (FIFO) order within a partition. When a large batch of low-priority work, such as nightly reports, bulk imports, or background processing, fills the backlog before time-sensitive requests arrive, the higher-priority requests wait behind the entire batch. A single Task Queue with no ordering mechanism gives the same dispatch preference to all Tasks, regardless of business urgency.

## Solution

Temporal's native Priority feature lets you assign a Priority key (an integer from 1 to 5, where 1 is the highest priority and 5 is the lowest) to any Workflow, Activity, or Child Workflow. The Matching Service maintains a sub-queue for each Priority level and exhausts all backlogged Tasks at a given level before dispatching to the next. Tasks default to Priority `3` when no key is set. Activities and Child Workflows inherit the parent Workflow's Priority unless they set their own.

```mermaid
flowchart TD
    WF1["Workflow\nPriority 1\n(payment)"] --> TQ["my-task-queue"]
    WF2["Workflow\nPriority 3\n(default)"] --> TQ
    WF3["Workflow\nPriority 5\n(batch report)"] --> TQ
    TQ --> P1["Priority 1\nsub-queue"]
    TQ --> P3["Priority 3\nsub-queue"]
    TQ --> P5["Priority 5\nsub-queue"]
    P1 -->|dispatched first| W["Shared Workers"]
    P3 -->|dispatched second| W
    P5 -->|dispatched last| W
    W --> DS["Downstream\nService"]
```

The following describes each step in the diagram. The example assumes all three levels have backlogged Tasks in the same Task Queue partition and Worker Deployment Version.

1. Workflows start with a Priority key in their start options. Payment Workflows use Priority `1`. Routine Workflows default to Priority `3`. Nightly batch reports use Priority `5`.
2. The Matching Service routes each Task to the corresponding Priority sub-queue inside the Task Queue.
3. Workers poll the Task Queue and receive the highest-priority backlogged Tasks first.
4. Activities and Child Workflows inherit the parent Workflow's Priority unless they set their own.

## Implementation

Priority is enabled by default in Temporal Cloud and self-hosted Temporal. Set a Priority key in Workflow start options or in Activity and Child Workflow options.

See [Task Queue Priority](/develop/task-queue-priority-fairness#task-queue-priority) for SDK and command-line examples, inheritance behavior, and self-hosted configuration.

## When to use

This pattern is a good fit when your system mixes time-sensitive operations (payment processing, user-facing requests) with background or batch work (reporting, data imports, inventory management), and you want urgent Tasks to dispatch first during periods of high load. It also works well when you need to mark urgent Tasks that should dispatch ahead of normal processing, for example, triggering immediate reruns of failed critical Tasks.

It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers. Use separate Task Queues with dedicated Worker pools and compute resources for hard capacity isolation. If your concern is prioritizing work among tenants or customers, consider the [Fairness](/design-patterns/fairness) pattern, which distributes dispatches proportionally using weighted Fairness keys rather than strict ordering.

## Benefits and trade-offs

Native Priority requires no extra queues, routing logic, or additional Worker pools. A single pool of Workers serves all Priority levels, so idle Worker capacity is available to every level without additional configuration.

Lower-priority Tasks are blocked while higher-priority Tasks remain backlogged. In an environment with a continuously replenished high-priority backlog, low-priority Tasks may be significantly delayed. The built-in Priority key range is 1 to 5. The feature does not support more than five levels.

## Comparison with alternatives

| Approach | Backlog dispatch | Shares idle capacity |
| :--- | :--- | :--- |
| Priority on a shared Task Queue | Higher-priority Tasks first | Yes |
| [Fairness](/design-patterns/fairness) on a shared Task Queue | Weighted across groups within a Priority level | Yes |
| Separate Task Queues with shared compute | Independent backlogs | Yes |
| Separate Task Queues with dedicated compute | Independent backlogs | No |

## Best practices

- **Use no more than five Priority levels.** The Priority key range is 1 to 5. Keep levels coarse. For example, use `1` for urgent work, `3` for normal work, and `5` for batch work.
- **Reserve Priority `1` for genuinely urgent work.** When every caller uses Priority `1`, the highest level fills with routine work and the feature provides no benefit. The default is `3` when no key is set.
- **Set the initial Priority key at Workflow start.** Set the Priority in the start options before execution begins. Activities and Child Workflows inherit it unless they set their own.
- **Override Activity Priority deliberately.** Activities inherit the parent Workflow's Priority by default. Override it only when a specific Activity must dispatch at a different level than its Workflow.
- **Monitor queue depth per Priority level.** Sustained backlog growth at a level means Tasks are arriving faster than they are being dispatched.

## Common pitfalls

- **Assigning Priority `1` to all work by default.** When every caller sets the highest Priority, the feature provides no ordering benefit. Establish an explicit policy for which work types qualify for each level.
- **Neglecting low-priority starvation.** Under sustained high load, Priority `5` Tasks may wait indefinitely. Use a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) on low-priority Activities to surface starvation as a visible failure.
- **Changing priority after scheduling.** The Priority key is evaluated when a Task enters the queue and cannot be changed while it waits. To re-prioritize an already-queued Task, cancel it and reschedule with the new priority.
- **Assuming hard isolation between Priority levels.** Priority controls dispatch order, not Worker capacity allocation. A Priority `5` Task may still occupy a Worker slot when a Priority `1` Task arrives.

## Related

### Patterns

- **[Fairness](/design-patterns/fairness)**: Distribute dispatches across tenants within a Priority level.
- **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap dispatch throughput to a downstream service.
- **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity.

---

# QoS & Throughput Patterns

Source: https://docs.temporal.io/design-patterns/qos-throughput-patterns

> Pattern selection guide for controlling execution rate, protecting downstream services from overload, and providing fair dispatch across tenants.

These patterns control how fast work executes, protect downstream services from overload, and make sure no single caller or tenant dominates dispatch.

## Patterns in this section

- [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting): Rate-limits calls to a downstream service by routing throttled Activities to a dedicated Task Queue with a server-enforced throughput cap.
- [Priority](/design-patterns/priority-task-queues): Assigns a priority level to Workflows and Activities so time-sensitive work dispatches ahead of lower-priority work on the same Task Queue.
- [Fairness](/design-patterns/fairness): Distributes Task dispatches across tenants or users so a burst from one caller does not starve the others.

## Choosing a pattern

**A downstream dependency has a fixed rate limit**: use [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting) to cap throughput at the Worker.

**Urgent work should move ahead of bulk work**: use [Priority](/design-patterns/priority-task-queues).

**Multiple tenants share the same Workers**: use [Fairness](/design-patterns/fairness) to keep one tenant's burst from starving others.

## Related sections

- [Worker Configuration Patterns](/design-patterns/worker-configuration-patterns) — the Task Queue and Worker setup these patterns route through
- [Batch Processing Patterns](/design-patterns/batch-processing-patterns) — rate-control patterns for large record sets
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns) — back off and retry when a rate limit is hit

---

# Request-Response via Updates

Source: https://docs.temporal.io/design-patterns/request-response-via-updates

> Uses a Workflow Update to validate a request, modify Workflow state, and return a typed result to the caller synchronously, with strong consistency.

## Overview

Workflow Updates enable synchronous request-response interactions where clients receive immediate, typed responses while the Workflow continues processing.
Updates modify Workflow state, validate inputs, and return results directly to the caller with strong consistency guarantees.

## Problem

In distributed systems, you often need Workflows that provide immediate feedback to clients (validation results, confirmation IDs), require strong consistency guarantees for operations, need typed error handling for validation failures, should validate inputs before accepting work, and allow external systems to modify Workflow state synchronously.

Without Updates, clients must use Signals and poll via Queries (complex, eventually consistent), wait for entire Workflow completion (slow), implement complex coordination logic, and handle race conditions between Signals and Queries.

## Solution

Temporal's Update API executes an Update handler that can validate inputs, modify state, and return values synchronously.
The Update is recorded in Workflow history before returning, providing strong consistency.

```mermaid
sequenceDiagram
    participant Client
    participant Workflow
    participant State

    Client->>+Workflow: Update Request
    Workflow->>Workflow: Validate Input
    alt Validation Fails
        Workflow-->>Client: Error Response
    else Success
        Workflow->>State: Modify State
        State-->>Workflow: Updated
        Workflow-->>Client: Typed Response
    end
    deactivate Workflow
```

The following describes each step in the diagram:

1. The client sends an Update request to the Workflow.
2. The Workflow validates the input. If validation fails, it returns a typed error response.
3. If validation succeeds, the Workflow modifies its state and returns a typed response to the client.

## Implementation

The following examples show a task assignment Workflow that accepts tasks via Updates with validation.
The Update validator rejects requests when the task limit is reached, and the Update handler assigns the task and returns a result.

**Python**

```python
# workflows.py
import uuid
from dataclasses import dataclass
from temporalio import workflow

MAX_TASKS = 10

@dataclass
class AssignmentResult:
    assignment_id: str
    task_name: str
    total_tasks: int

@workflow.defn
class TaskWorkflow:
    def __init__(self) -> None:
        self.tasks: list[str] = []

    @workflow.run
    async def run(self) -> None:
        await workflow.wait_condition(lambda: False)

    @workflow.update
    async def assign_task(self, task_name: str) -> AssignmentResult:
        assignment_id = str(uuid.uuid4())
        self.tasks.append(task_name)
        return AssignmentResult(
            assignment_id=assignment_id,
            task_name=task_name,
            total_tasks=len(self.tasks),
        )

    @assign_task.validator
    def validate_assign_task(self, task_name: str) -> None:
        if len(self.tasks) >= MAX_TASKS:
            raise ValueError("Task limit reached")

    @workflow.query
    def get_tasks(self) -> list[str]:
        return list(self.tasks)
```

**Go**

```go
// workflow.go
type TaskWorkflow struct{}

const MaxTasks = 10

func (w *TaskWorkflow) Run(ctx workflow.Context) error {
	tasks := []string{}

	err := workflow.SetUpdateHandlerWithOptions(
		ctx,
		"AssignTask",
		func(ctx workflow.Context, taskName string) (AssignmentResult, error) {
			assignmentID := uuid.New().String()
			tasks = append(tasks, taskName)
			return AssignmentResult{
				AssignmentID: assignmentID,
				TaskName:     taskName,
				TotalTasks:   len(tasks),
			}, nil
		},
		workflow.UpdateHandlerOptions{
			Validator: func(taskName string) error {
				if len(tasks) >= MaxTasks {
					return fmt.Errorf("task limit reached")
				}
				return nil
			},
		},
	)
	if err != nil {
		return err
	}

	err = workflow.SetQueryHandler(ctx, "GetTasks", func() ([]string, error) {
		return tasks, nil
	})
	if err != nil {
		return err
	}

	workflow.GetSignalChannel(ctx, "").Receive(ctx, nil)
	return nil
}
```

**Java**

```java
// TaskWorkflow.java
@WorkflowInterface
public interface TaskWorkflow {
  @WorkflowMethod
  void run();

  @UpdateMethod
  AssignmentResult assignTask(String taskName);

  @QueryMethod
  List<String> getTasks();
}

public class TaskWorkflowImpl implements TaskWorkflow {
  private static final int MAX_TASKS = 10;
  private List<String> tasks = new ArrayList<>();

  @Override
  public void run() {
    Workflow.await(() -> false);
  }

  @UpdateValidatorMethod(updateName = "assignTask")
  protected void validateAssignTask(String taskName) {
    if (tasks.size() >= MAX_TASKS) {
      throw new IllegalStateException("Task limit reached");
    }
  }

  @Override
  public AssignmentResult assignTask(String taskName) {
    String assignmentId = UUID.randomUUID().toString();
    tasks.add(taskName);

    return new AssignmentResult(assignmentId, taskName, tasks.size());
  }

  @Override
  public List<String> getTasks() {
    return new ArrayList<>(tasks);
  }
}
```

**TypeScript**

```typescript
// workflow.ts
import * as wf from '@temporalio/workflow';

interface AssignmentResult {
  assignmentId: string;
  taskName: string;
  totalTasks: number;
}

export const assignTaskUpdate = wf.defineUpdate<AssignmentResult, [string]>('assignTask');
export const getTasksQuery = wf.defineQuery<string[]>('getTasks');

const MAX_TASKS = 10;

export async function taskWorkflow(): Promise<void> {
  const tasks: string[] = [];

  wf.setHandler(
    assignTaskUpdate,
    (taskName: string): AssignmentResult => {
      const assignmentId = wf.uuid4();
      tasks.push(taskName);
      return { assignmentId, taskName, totalTasks: tasks.length };
    },
    {
      validator: (taskName: string): void => {
        if (tasks.length >= MAX_TASKS) {
          throw new Error('Task limit reached');
        }
      },
    }
  );

  wf.setHandler(getTasksQuery, (): string[] => tasks);

  await wf.condition(() => false);
}
```

In all SDKs, the validator runs before the Update handler.
If the validator throws an exception, the Update is rejected and the client receives a typed error.
If the validator passes, the Update handler modifies state and returns a typed result.
The Update is recorded in Workflow history before the response is returned to the client.

## When to use

The Update pattern is a good fit for request-response patterns requiring immediate confirmation, input validation before accepting work, synchronous state modifications with typed responses, operations requiring strong consistency guarantees, and entity Workflows that need external state Updates.

It is not a good fit for fire-and-forget operations (use Signals), read-only operations (use Queries), high-throughput scenarios where latency matters (Updates are slower than Signals), or operations that do not need an immediate response.

## Benefits and trade-offs

Benefits:

- Updates provide a synchronous response — the client receives a typed return value immediately.
- Validation failures return as typed exceptions.
- The Update is recorded in history before returning, providing strong consistency.
- You can modify Workflow state directly from external systems.

Trade-offs:

- Updates are slower than Signals (they require a history write).
- The Update handler blocks Workflow Task execution and consumes Workflow Task execution time.
- For fire-and-forget messages that need no response, Updates require more machinery than Signals.
- Update arguments and return values are limited by the Workflow history event size (typically 2 MB per event).
- Each Update adds events to Workflow history, contributing to the 51,200 event limit.
- There is a maximum of 10 in-flight Updates per Workflow execution and a maximum of 2,000 total Updates in Workflow history.

## Comparison with alternatives

| Approach | Use case | Response type | Latency | Consistency |
| :--- | :--- | :--- | :--- | :--- |
| Update | Request-response | Sync typed value | Higher | Strong |
| Signal | Fire-and-forget | None | Lower | Eventual |
| Query | Read-only | Sync typed value | Lowest | Eventual |

## Best practices

- **Validate early.** Check inputs at the start of the Update handler to fail fast.
- **Handle errors.** Throw typed exceptions for validation failures.
- **Return quickly.** Do not perform long operations in the Update handler.
- **Track Update IDs only across Continue-As-New.** Within a single Workflow Execution, the Server deduplicates retried Updates automatically by Update ID, so a retry does not run the handler twice. You only need to track processed Update IDs in Workflow state when carrying them across a Continue-As-New boundary, because Update ID deduplication is scoped to a single Workflow Run.
- **Set timeouts.** Configure appropriate Update timeouts.
- **Maintain state consistency.** Ensure state modifications are atomic within the handler.

## Common pitfalls

- **Performing long operations in the Update handler.** Update handlers block Workflow Task execution. Offload long-running work to Activities and use `Workflow.await` in the handler to wait for results.
- **Exceeding the 2,000 total Updates limit.** Each accepted Update adds events to history. Use Continue-As-New before reaching the limit. The server sets `SuggestContinueAsNew` at 90% of the limit.
- **Not setting Update timeouts.** Without a client-side timeout, the caller blocks indefinitely if the Worker is unavailable. Always set a context timeout or deadline.
- **Assuming you must set an Update ID for retry safety.** The SDK auto-generates a unique `updateId` when you do not provide one, and retried client calls are deduplicated automatically, so a transient retry will not run the handler twice. Set a stable, business-meaningful `updateId` when you want a retried Update-with-Start to attach to an existing in-flight Update instead of starting duplicate work.
- **Using Updates for fire-and-forget.** Updates require a Worker to be online and responsive. For fire-and-forget operations, use Signals instead.

## Related

### Patterns

- **Signal**: Fire-and-forget state modifications.
- **Query**: Read-only state inspection.
- **[Entity Workflow](/design-patterns/entity-workflow)**: Long-running Workflows representing business entities.
- **[Early Return](/design-patterns/early-return)**: Returning intermediate results before Workflow completion.

### References

- [Sending Messages](/sending-messages): Canonical reference for Updates as a delivery mechanism, alongside Signals and Queries.
- [Handling Messages](/handling-messages): Canonical reference for writing Update handlers, including validators and idempotency.

### Sample code

- [Safe Message Handlers (Python)](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) — Concurrent Update handling with validation.
- [Safe Message Passing (Java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Concurrent Update handling with validation.
- [Update with Start - Shopping Cart (Go)](https://github.com/temporalio/samples-go/tree/main/shoppingcart) — Update-with-Start for lazy initialization.

---

# Resumable Activity (AKA Pause On Failure)

Source: https://docs.temporal.io/design-patterns/resumable-activity

> Park the Workflow after retries are exhausted and wait for a human to signal a correction, then resume execution from where it left off.

> **ℹ️ TLDR:**
> After retries are exhausted, **park the Workflow in a waiting state and block on a Signal that notifies the Workflow to proceed or optionally delivers corrected input from a human operator, then re-execute the Activity.** Use this when failures are caused by bad input that can be fixed externally — so the Workflow resumes exactly where it left off instead of being restarted from scratch.

## Overview

The Resumable Activity pattern parks a Workflow, durably waiting, after Activity retries are exhausted, waits for a corrective Signal from a human operator, then re-executes the Activity with the corrected input.
Use it when failures are caused by bad input data that can be corrected externally — a wrong account number, an invalid reference, a missing record that will be created — and abandoning the Workflow is worse than pausing it.

## Problem

When an Activity fails due to bad input, retrying with the same input will never succeed.
The standard options are:

- **Fail the Workflow immediately.** The client must restart the process from scratch, often re-entering the same data that caused the failure.
- **Mark the error as non-retryable.** Same result — the Workflow fails and the client has no way to inject a correction.
- **Poll for the correction from inside the Workflow.** Wastes resources for a fix that may never come.

What you actually want is for the Workflow to *pause* — consuming zero resources — until an authorized operator provides corrected data, then *resume* exactly where it left off.
Temporal's durable execution model makes this possible without any external database, queue, or polling mechanism.

## Solution

Use a bounded `RetryPolicy` to allow a few automatic retries (in case the failure is transient), then catch the exhausted `ActivityError` in the Workflow.
Transition to an `AWAITING_CORRECTION` state and block on `workflow.wait_condition` (or equivalent).
Register a Signal handler that accepts the corrected input and unblocks the condition.
When the Signal arrives, re-execute the Activity with the corrected input.
A second Signal gates the final approval before completing.

```mermaid
sequenceDiagram
    participant Client
    participant Admin
    participant Workflow
    participant Activity as Transfer Activity

    Client->>+Workflow: Start transfer(from, invalid-account, $500)
    Workflow->>Workflow: status = TRANSFERRING

    loop maxAttempts=3
        Workflow->>+Activity: executeTransfer(invalid-account)
        Activity-->>-Workflow: Failure — account not found
    end

    Note over Workflow: Retries exhausted
    Workflow->>Workflow: status = AWAITING_CORRECTION
    Note over Workflow: Parked in Temporal — zero cost, no polling

    Admin->>Workflow: Signal: retryWithCorrection("account-123")
    Workflow->>Workflow: status = TRANSFERRING

    Workflow->>+Activity: executeTransfer(account-123)
    Activity-->>-Workflow: Success

    Workflow->>Workflow: status = AWAITING_APPROVAL
    Note over Workflow: Parked again — waiting for client approval

    Client->>Workflow: Signal: approve(true)
    Workflow-->>-Client: Transfer completed
```

The following describes each step:

1. The client starts the Workflow with an invalid account number.
2. The Activity fails. Temporal retries automatically up to the configured `maxAttempts`.
3. When retries are exhausted, the Workflow catches the `ActivityError` and sets its status to `AWAITING_CORRECTION`.
4. The Workflow parks itself using `wait_condition` — it consumes no CPU, no polling, no timers. Its state is fully persisted in Temporal.
5. An admin notices the problem (via Temporal UI, an alert, or an operations dashboard) and sends a `retryWithCorrection` Signal with the corrected account number.
6. The Workflow wakes up, applies the correction, and re-executes the Activity — which now succeeds.
7. The Workflow transitions to `AWAITING_APPROVAL` and parks again, waiting for the client to approve the transfer.
8. The client sends an `approve` Signal. The Workflow completes and returns the result.

The Workflow stays alive throughout: it waits indefinitely without polling, accepts an external correction, and completes without restarting. Its state — status, corrected account number, approval decision — is durable in Temporal the entire time.

## Implementation

### Workflow with correction and approval signals

The Workflow maintains state as named fields.
Signal handlers set the fields, and `wait_condition` blocks until they are non-null.

**Python**

```python
# workflows.py
from dataclasses import dataclass
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy, SearchAttributeKey
from temporalio.exceptions import ActivityError
import activities

TRANSFER_STATUS_KEY = SearchAttributeKey.for_keyword("TransferStatus")

@dataclass
class TransferInput:
    from_account: str
    to_account: str
    amount: float

@workflow.defn
class TransferWorkflow:
    def __init__(self) -> None:
        self._status = "PENDING"
        self._corrected_account: str | None = None
        self._approval: bool | None = None

    @workflow.run
    async def run(self, transfer: TransferInput) -> str:
        account = transfer.to_account
        correction_attempts = 0

        while True:
            self._status = "TRANSFERRING"
            try:
                result = await workflow.execute_activity(
                    activities.execute_transfer,
                    TransferInput(transfer.from_account, account, transfer.amount),
                    start_to_close_timeout=timedelta(seconds=30),
                    retry_policy=RetryPolicy(maximum_attempts=3),
                )
                break  # Activity succeeded — exit the correction loop
            except ActivityError:
                correction_attempts += 1
                if correction_attempts > 5:
                    self._status = "FAILED"
                    workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)])
                    raise
                self._status = "AWAITING_CORRECTION"
                workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)])
                workflow.logger.warning(
                    "Transfer failed — waiting for account correction",
                    extra={"to_account": account},
                )
                # Park until the admin sends a correction signal
                await workflow.wait_condition(
                    lambda: self._corrected_account is not None
                )
                account = self._corrected_account
                self._corrected_account = None

        self._status = "AWAITING_APPROVAL"
        await workflow.wait_condition(lambda: self._approval is not None)

        if self._approval:
            self._status = "COMPLETED"
            return f"Transfer of {transfer.amount} to {account} completed"
        self._status = "REJECTED"
        return "Transfer rejected by client"

    @workflow.signal
    def retry_with_correction(self, corrected_account: str) -> None:
        self._corrected_account = corrected_account

    @workflow.signal
    def approve(self, approved: bool) -> None:
        self._approval = approved

    @workflow.query
    def get_status(self) -> str:
        return self._status
```

**Go**

```go
// workflow.go
package transfer

import (
    "fmt"
    "time"

    "go.temporal.io/sdk/temporal"
    "go.temporal.io/sdk/workflow"
)

type TransferInput struct {
    FromAccount string
    ToAccount   string
    Amount      float64
}

func TransferWorkflow(ctx workflow.Context, input TransferInput) (string, error) {
    status := "PENDING"
    if err := workflow.SetQueryHandler(ctx, "getStatus", func() (string, error) {
        return status, nil
    }); err != nil {
        return "", err
    }

    correctionCh := workflow.GetSignalChannel(ctx, "retryWithCorrection")
    approvalCh := workflow.GetSignalChannel(ctx, "approve")

    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{MaximumAttempts: 3},
    }
    actCtx := workflow.WithActivityOptions(ctx, ao)

    account := input.ToAccount
    correctionCount := 0
    for {
        status = "TRANSFERRING"
        err := workflow.ExecuteActivity(actCtx, ExecuteTransfer, TransferInput{
            FromAccount: input.FromAccount,
            ToAccount:   account,
            Amount:      input.Amount,
        }).Get(actCtx, nil)

        if err == nil {
            break // Activity succeeded — exit the correction loop
        }

        correctionCount++
        if correctionCount > 5 { // for long correction cycles, consider Continue As New
            status = "FAILED"
            _ = workflow.UpsertSearchAttributes(ctx, map[string]interface{}{"TransferStatus": status})
            return "", err
        }
        status = "AWAITING_CORRECTION"
        _ = workflow.UpsertSearchAttributes(ctx, map[string]interface{}{"TransferStatus": status})
        workflow.GetLogger(ctx).Warn("Transfer failed — waiting for account correction",
            "to_account", account)

        // Park until the admin sends a correction signal
        var corrected string
        _ = workflow.Await(ctx, func() bool {
            return correctionCh.ReceiveAsync(&corrected)
        })
        account = corrected
    }

    status = "AWAITING_APPROVAL"
    var approved bool
    _ = workflow.Await(ctx, func() bool {
        return approvalCh.ReceiveAsync(&approved)
    })

    if approved {
        status = "COMPLETED"
        return fmt.Sprintf("Transfer of %.2f to %s completed", input.Amount, account), nil
    }
    status = "REJECTED"
    return "Transfer rejected by client", nil
}
```

**Java**

```java
// TransferWorkflowImpl.java
import io.temporal.activity.ActivityOptions;
import io.temporal.common.RetryOptions;
import io.temporal.common.SearchAttributeKey;
import io.temporal.failure.ActivityFailure;
import io.temporal.workflow.SignalMethod;
import io.temporal.workflow.QueryMethod;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import io.temporal.workflow.Workflow;
import java.time.Duration;

@WorkflowInterface
public interface TransferWorkflow {
    @WorkflowMethod
    String run(TransferInput input);

    @SignalMethod
    void retryWithCorrection(String correctedAccount);

    @SignalMethod
    void approve(boolean approved);

    @QueryMethod
    String getStatus();
}

public class TransferWorkflowImpl implements TransferWorkflow {
    private String status = "PENDING";
    private String correctedAccount;
    private Boolean approval;

    private final TransferActivities activities = Workflow.newActivityStub(
        TransferActivities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .setRetryOptions(RetryOptions.newBuilder()
                .setMaximumAttempts(3)
                .build())
            .build()
    );

    @Override
    public String run(TransferInput input) {
        String account = input.getToAccount();
        int correctionCount = 0; 

        while (true) {
            status = "TRANSFERRING";
            try {
                activities.executeTransfer(
                    new TransferInput(input.getFromAccount(), account, input.getAmount())
                );
                break; // Activity succeeded — exit the correction loop
            } catch (ActivityFailure e) {
                correctionCount++;
                if (correctionCount > 5) { // for long correction cycles, consider Continue As New
                    status = "FAILED";
                    Workflow.upsertTypedSearchAttributes(
                        SearchAttributeKey.forKeyword("TransferStatus").valueSet(status)
                    );
                    throw e;
                }
                status = "AWAITING_CORRECTION";
                Workflow.upsertTypedSearchAttributes(
                    SearchAttributeKey.forKeyword("TransferStatus").valueSet(status)
                );
                Workflow.getLogger(getClass()).warn(
                    "Transfer failed — waiting for account correction: " + account
                );
                // Park until the admin sends a correction signal
                Workflow.await(() -> correctedAccount != null);
                account = correctedAccount;
                correctedAccount = null;
            }
        }

        status = "AWAITING_APPROVAL";
        Workflow.await(() -> approval != null);

        if (approval) {
            status = "COMPLETED";
            return String.format("Transfer of %.2f to %s completed", input.getAmount(), account);
        }
        status = "REJECTED";
        return "Transfer rejected by client";
    }

    @Override
    public void retryWithCorrection(String account) {
        this.correctedAccount = account;
    }

    @Override
    public void approve(boolean decision) {
        this.approval = decision;
    }

    @Override
    public String getStatus() {
        return status;
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import type * as activities from './activities';

export interface TransferInput {
    fromAccount: string;
    toAccount: string;
    amount: number;
}

export const retryWithCorrectionSignal = wf.defineSignal<[string]>('retryWithCorrection');
export const approveSignal = wf.defineSignal<[boolean]>('approve');
export const getStatusQuery = wf.defineQuery<string>('getStatus');

const { executeTransfer } = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '30s',
    retry: { maximumAttempts: 3 },
});

export async function transferWorkflow(input: TransferInput): Promise<string> {
    let status = 'PENDING';
    let correctedAccount: string | undefined;
    let approval: boolean | undefined;

    wf.setHandler(retryWithCorrectionSignal, (account: string) => {
        correctedAccount = account;
    });
    wf.setHandler(approveSignal, (decision: boolean) => {
        approval = decision;
    });
    wf.setHandler(getStatusQuery, () => status);

    let account = input.toAccount;
    let correctionCount = 0;

    while (true) {
        status = 'TRANSFERRING';
        try {
            await executeTransfer({ ...input, toAccount: account });
            break; // Activity succeeded — exit the correction loop
        } catch (err) {
            correctionCount++;
            if (correctionCount > 5) {
                status = 'FAILED';
                wf.upsertSearchAttributes({ TransferStatus: [status] });
                throw err;
            }
            status = 'AWAITING_CORRECTION';
            wf.upsertSearchAttributes({ TransferStatus: [status] });
            wf.log.warn('Transfer failed — waiting for account correction', { account });
            // Park until the admin sends a correction signal
            await wf.condition(() => correctedAccount !== undefined);
            account = correctedAccount!;
            correctedAccount = undefined;
        }
    }

    status = 'AWAITING_APPROVAL';
    await wf.condition(() => approval !== undefined);

    if (approval) {
        status = 'COMPLETED';
        return `Transfer of ${input.amount} to ${account} completed`;
    }
    status = 'REJECTED';
    return 'Transfer rejected by client';
}
```

### Send signals

An operator sends the correction Signal using the Temporal CLI or any SDK client.
The Workflow wakes immediately when the Signal is delivered.

```bash
# Correct the account number
temporal workflow signal \
  --workflow-id transfer-wf-001 \
  --name retryWithCorrection \
  --input '"account-123"'

# Approve the transfer
temporal workflow signal \
  --workflow-id transfer-wf-001 \
  --name approve \
  --input 'true'
```

### Activity implementation

The `executeTransfer` Activity must distinguish between permanent failures — such as an invalid account number — and transient failures that Temporal should retry automatically.
Throw a non-retryable `ApplicationFailure` for permanent input errors so the Workflow catches the `ActivityError` immediately and transitions to `AWAITING_CORRECTION` instead of exhausting all retry attempts first.
Let all other exceptions propagate so the RetryPolicy handles transient failures.

**Python**

```python
# activities.py
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def execute_transfer(transfer: TransferInput) -> str:
    # Non-retryable: bad account number requires a human correction, not a retry.
    if not await account_service.exists(transfer.to_account):
        raise ApplicationError(
            f"Account {transfer.to_account} not found",
            type="AccountNotFoundError",
            non_retryable=True,
        )
    # Other exceptions propagate as retryable so the RetryPolicy handles them.
    return await payment_service.transfer(
        transfer.from_account, transfer.to_account, transfer.amount
    )
```

**TypeScript**

```typescript
// activities.ts
import { ApplicationFailure } from '@temporalio/activity';
import type { TransferInput } from './workflows';

export async function executeTransfer(transfer: TransferInput): Promise<string> {
    // Non-retryable: bad account number requires a human correction, not a retry.
    const accountExists = await accountService.exists(transfer.toAccount);
    if (!accountExists) {
        throw ApplicationFailure.nonRetryable(
            `Account ${transfer.toAccount} not found`,
            'AccountNotFoundError',
        );
    }
    // Other exceptions propagate as retryable so the RetryPolicy handles them.
    return paymentService.transfer(transfer.fromAccount, transfer.toAccount, transfer.amount);
}
```

## State diagram

The Workflow transitions through a well-defined set of states.
Query `getStatus` at any time to observe the current state.

```mermaid
stateDiagram-v2
    [*] --> PENDING
    PENDING --> TRANSFERRING : Workflow starts
    TRANSFERRING --> AWAITING_CORRECTION : Activity retries exhausted
    AWAITING_CORRECTION --> TRANSFERRING : retryWithCorrection signal received
    AWAITING_CORRECTION --> FAILED : 5 correction attempts exceeded
    TRANSFERRING --> AWAITING_APPROVAL : Activity succeeds
    AWAITING_APPROVAL --> COMPLETED : approve(true) signal
    AWAITING_APPROVAL --> REJECTED : approve(false) signal
    COMPLETED --> [*]
    REJECTED --> [*]
    FAILED --> [*]
```

## Best practices

- **Use a bounded `MaximumAttempts` before parking.** Allow a few automatic retries to recover from transient failures. Parking immediately on the first failure forces operators to intervene for problems that would have resolved on their own.
- **Manage history growth in long-running correction loops.** Each correction cycle — park, receive signal, re-execute Activity — adds events to the Workflow history (signal received, state transitions, Activity scheduled/completed). For workflows that may receive many corrections over time, use [Continue-As-New](/design-patterns/continue-as-new) to carry the current state into a fresh execution before the history grows too large, rather than relying solely on an arbitrary correction counter.
- **Expose status via a Query method.** The `getStatus` Query gives operations tooling visibility into where the Workflow is parked without requiring access to the Workflow history.
- **Validate the correction in the Signal handler.** Check that the corrected account is non-empty and matches the expected format before setting the state. An invalid correction parks the Workflow again, but a clear error message helps operators.
- **Log and record state at every transition.** The `AWAITING_CORRECTION` and `AWAITING_APPROVAL` states can last hours or days. Structured log lines at each transition make the audit trail clear. For operational visibility, also update a [Search Attribute](/visibility) at each transition (for example, a `Keyword` attribute storing the current status) so operators can filter and query workflows by state directly from the Temporal UI or CLI.
- **Notify the operator proactively.** The `AWAITING_CORRECTION` transition is a good point to send an alert — an email, a Slack message, or a ticket — rather than waiting for the operator to notice in the Temporal UI.
- **Distinguish this from the Approval pattern.** The [Approval](/design-patterns/approval) pattern gates forward progress on a human decision. This pattern recovers from failure with a human-supplied data correction. Both use Signals and `wait_condition`, but serve different roles in a process.

## Common pitfalls

- **Waiting without a timeout.** If operators never send the correction Signal, the Workflow waits indefinitely. Add durable timer if the process must resolve within a time bound.
- **Not clearing the correction state before re-entering the loop.** After applying the correction, set `corrected_account = None` (or equivalent) before the next Activity attempt. Otherwise, if the corrected activity also fails, the Workflow immediately re-uses the previous correction instead of waiting for a new one.
- **Accepting corrections in the wrong state.** If a Signal arrives while the Activity is running (not parked), the correction should be queued and applied after the current attempt completes. The Signal handler always runs — the condition check (`wait_condition`) determines when the Workflow acts on it.
- **Conflating the correction loop with a general retry loop.** This pattern is for correcting *input data*. For retrying the same call against a temporarily unavailable system, use [Fast/Slow Retries](/design-patterns/fast-slow-retries) instead.

## Related

### Patterns

- [Approval](/design-patterns/approval): Human-in-the-loop gate for forward progress rather than failure recovery.
- [Non-Retryable Errors](/design-patterns/non-retryable-errors): Fail immediately without parking when the error is structural and no correction is expected.
- [Fast/Slow Retries](/design-patterns/fast-slow-retries): Infinite patient retries when the downstream system is temporarily unavailable.
- [Signal with Start](/design-patterns/signal-with-start): Start the Workflow and send the correction Signal atomically.
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns.

### Guides

- [Recover business processes without restarting](/guides/recover-without-restart): A `recoverableStep` implementation of this pattern in a six-step loan pipeline, with Search Attribute-based routing so operators can find and fix blocked cases.

### References

- [Temporal Retry Policies](/encyclopedia/retry-policies)

---

# Retry Alerting via Metrics

Source: https://docs.temporal.io/design-patterns/retry-metrics

> Emit a metric from the Activity when attempts cross a threshold, so on-call teams see persistent failures before an SLA breach.

> **ℹ️ TLDR:**
> Emit a counter metric from inside the Activity when the attempt number exceeds a threshold, using the SDK's built-in metrics scope. **Use this to surface silent, persistent failures to on-call teams before they breach an SLA** — without changing retry behavior or adding Workflow-level tracking.

## Overview

The Retry Alerting via Metrics pattern emits a custom metric counter from inside the Activity whenever the attempt number exceeds a threshold.
Use it to surface silent, persistent failures to on-call teams before they breach an SLA — without modifying retry behavior or adding Workflow-level tracking.

## Problem

When an Activity retries indefinitely, failures are invisible at the system level until something breaks.
The Temporal UI shows the current attempt number, but on-call teams do not watch the UI continuously.
Without a metric or alert, a downstream system can be down for hours while the Workflow keeps retrying silently — and the first sign of a problem is an SLA breach or a user complaint.

Common gaps:

- A payment gateway is down. Workflows are retrying every 5 minutes. No alert fires until the merchant escalates.
- An email provider is rejecting requests. Activities are on attempt 50. No metric has been emitted. On-call has no signal.
- A third-party API degraded. Retries are accumulating. The engineering team learns about the problem from a customer, not from their own alerting.

## Solution

Read the current attempt number from the Activity execution context and emit a counter metric when it exceeds a threshold.
The metric is sent through the Temporal SDK's built-in metrics scope — the same pipeline used for SDK-internal metrics — so it flows to whatever metrics backend your Workers are already configured to use, such as Prometheus or StatsD, without additional setup.

```mermaid
sequenceDiagram
    participant Temporal as Temporal Service
    participant Activity
    participant Metrics as Metrics Backend

    loop Each retry attempt
        Temporal->>+Activity: Execute (attempt N)
        Activity->>Activity: Check attempt number
        alt attempt > threshold (e.g. 5)
            Activity->>Metrics: increment high_activity_error_count
        end
        Activity-->>-Temporal: Failure
        Note over Temporal: Wait backoff interval
    end
    Note over Metrics: Alert fires when counter crosses threshold
```

The following describes each step:

1. Temporal executes the Activity, passing the current attempt number in the execution context.
2. The Activity checks whether the attempt number exceeds the threshold.
3. If it does, the Activity increments a counter metric using the SDK's built-in metrics scope.
4. On failure, Temporal waits the backoff interval and retries.
5. The metrics backend accumulates the counter. Your alerting system fires when the counter or rate crosses a configured threshold.

## Implementation

### Emit a counter at high attempt counts

Read the attempt number from the Activity info and emit a counter through the SDK metrics scope.
Configure the retry policy separately — the metric emission does not change retry behavior.

**Python**

```python
# activities.py
from temporalio import activity
from temporalio.exceptions import ApplicationError

ALERT_THRESHOLD = 5

@activity.defn
async def call_downstream_service(endpoint: str) -> str:
    info = activity.info()

    if info.attempt > ALERT_THRESHOLD:
        meter = activity.metric_meter()
        meter.create_counter(
            "high_activity_error_count",
            "Activity has exceeded the failure attempt threshold",
        ).add(1)

    # Attempt the actual work — raises on failure, triggering a retry
    response = await downstream.call(endpoint)
    return response.data
```

**Go**

```go
// activities.go
package downstream

import (
    "context"

    "go.temporal.io/sdk/activity"
)

const alertThreshold = 5

func CallDownstreamService(ctx context.Context, endpoint string) (string, error) {
    info := activity.GetInfo(ctx)

    if info.Attempt > alertThreshold {
        activity.GetMetricsHandler(ctx).
            Counter("high_activity_error_count").
            Inc(1)
    }

    // Attempt the actual work — returns an error on failure, triggering a retry
    response, err := downstream.Call(endpoint)
    if err != nil {
        return "", err
    }
    return response.Data, nil
}
```

**Java**

```java
// CallDownstreamActivityImpl.java
import io.temporal.activity.Activity;
import io.temporal.activity.ActivityExecutionContext;

public class CallDownstreamActivityImpl implements CallDownstreamActivity {
    private static final int ALERT_THRESHOLD = 5;

    @Override
    public String callDownstreamService(String endpoint) {
        ActivityExecutionContext ctx = Activity.getExecutionContext();

        if (ctx.getInfo().getAttempt() > ALERT_THRESHOLD) {
            ctx.getMetricsScope()
               .counter("HighActivityErrorCount")
               .inc(1);
        }

        // Attempt the actual work — throws on failure, triggering a retry
        return downstream.call(endpoint).getData();
    }
}
```

**TypeScript**

```typescript
// activities.ts
import { Context } from '@temporalio/activity';

const ALERT_THRESHOLD = 5;

export async function callDownstreamService(endpoint: string): Promise<string> {
    const ctx = Context.current();

    if (ctx.info.attempt > ALERT_THRESHOLD) {
        ctx.metricMeter
            .createCounter('high_activity_error_count')
            .add(1);
    }

    // Attempt the actual work — throws on failure, triggering a retry
    const response = await downstream.call(endpoint);
    return response.data;
}
```

### Workflow configuration

Configure the Activity in the Workflow with the desired retry policy.
The metric emission inside the Activity is independent of the retry configuration.

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
import activities

@workflow.defn
class MonitoredRetryWorkflow:
    @workflow.run
    async def run(self, endpoint: str) -> str:
        return await workflow.execute_activity(
            activities.call_downstream_service,
            endpoint,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(
                initial_interval=timedelta(seconds=5),
                backoff_coefficient=2.0,
                maximum_interval=timedelta(minutes=5),
                # No maximum_attempts — retries indefinitely until success
            ),
        )
```

**Go**

```go
// workflow.go
func MonitoredRetryWorkflow(ctx workflow.Context, endpoint string) (string, error) {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    5 * time.Second,
            BackoffCoefficient: 2.0,
            MaximumInterval:    5 * time.Minute,
            // No MaximumAttempts — retries indefinitely until success
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var result string
    err := workflow.ExecuteActivity(ctx, CallDownstreamService, endpoint).Get(ctx, &result)
    return result, err
}
```

**Java**

```java
// MonitoredRetryWorkflowImpl.java
public class MonitoredRetryWorkflowImpl implements MonitoredRetryWorkflow {
    private final CallDownstreamActivity activities = Workflow.newActivityStub(
        CallDownstreamActivity.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .setRetryOptions(RetryOptions.newBuilder()
                .setInitialInterval(Duration.ofSeconds(5))
                .setBackoffCoefficient(2.0)
                .setMaximumInterval(Duration.ofMinutes(5))
                // No setMaximumAttempts — retries indefinitely until success
                .build())
            .build()
    );

    @Override
    public String run(String endpoint) {
        return activities.callDownstreamService(endpoint);
    }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import type * as activities from './activities';

const { callDownstreamService } = wf.proxyActivities<typeof activities>({
    startToCloseTimeout: '30s',
    retry: {
        initialInterval: '5s',
        backoffCoefficient: 2,
        maximumInterval: '5m',
        // No maximumAttempts — retries indefinitely until success
    },
});

export async function monitoredRetryWorkflow(endpoint: string): Promise<string> {
    return await callDownstreamService(endpoint);
}
```

### Add dimension tags to the metric

Add tags to the metric to identify which Activity type, endpoint, or Workflow is producing the high attempt counts.
This makes the metric actionable in dashboards and alerts.

**Python**

```python
# activities.py
if info.attempt > ALERT_THRESHOLD:
    meter = activity.metric_meter()
    meter.create_counter(
        "high_activity_error_count",
        "Activity has exceeded the failure attempt threshold",
    ).add(1, {"activity_type": info.activity_type, "endpoint": endpoint})
```

**Go**

```go
// activities.go
if info.Attempt > alertThreshold {
    activity.GetMetricsHandler(ctx).
        WithTags(map[string]string{
            "activity_type": info.ActivityType.Name,
            "endpoint":      endpoint,
        }).
        Counter("high_activity_error_count").
        Inc(1)
}
```

**Java**

```java
// CallDownstreamActivityImpl.java
if (ctx.getInfo().getAttempt() > ALERT_THRESHOLD) {
    ctx.getMetricsScope()
       .tagged(ImmutableMap.of(
           "activity_type", ctx.getInfo().getActivityType(),
           "endpoint", endpoint
       ))
       .counter("HighActivityErrorCount")
       .inc(1);
}
```

**TypeScript**

```typescript
// activities.ts
if (ctx.info.attempt > ALERT_THRESHOLD) {
    ctx.metricMeter
        .createCounter('high_activity_error_count')
        .add(1, { activity_type: ctx.info.activityType, endpoint });
}
```

## Best practices

- **Choose a threshold above normal transient noise.** If your downstream system occasionally has 1–2 retry attempts under normal conditions, set the threshold at 5 or 10 so the metric only fires for genuinely sustained failures.
- **Emit on every attempt above the threshold, not only once.** Incrementing the counter on each high-attempt invocation allows alerting systems to detect both the onset and the duration of a problem by watching the counter rate.
- **Use the SDK metrics scope, not a third-party library.** The SDK scope integrates with your Worker's existing metrics pipeline and adds default tags such as namespace and task queue automatically.
- **Set up rate-based alerts, not count-based.** A count alert requires resetting or remembering the baseline. A rate alert (for example, "more than 3 increments per minute") fires when the problem is active and clears when it resolves.
- **Combine with Fast/Slow Retries.** Emit the metric in the slow-phase Activity of a [Fast/Slow Retries](/design-patterns/fast-slow-retries) pattern to alert when the Workflow has been in the slow phase long enough to be a concern.

## Common pitfalls

- **Emitting the metric in the Workflow instead of the Activity.** The Workflow does not have access to the Activity's attempt number without passing it explicitly. The Activity context always has the current attempt number — use it there.
- **Alerting on the total counter value instead of the rate.** If the counter is cumulative, a single high-attempt event in the past will keep the counter elevated forever. Alert on the increment rate (events per minute) rather than the absolute count.
- **Not resetting alerting context on success.** If the Activity eventually succeeds after 50 attempts, the high-attempt metric has already fired. Ensure your alerting system can resolve the alert when the metric rate drops to zero.
- **Setting the threshold too low.** A threshold of 1 means the metric fires on the very first retry — which is normal behavior. Calibrate the threshold to your system's expected transient error rate.

## Related

### Patterns

- [Fast/Slow Retries](/design-patterns/fast-slow-retries): Combine by emitting this metric inside the slow-phase Activity to alert when patient waiting has gone on too long.
- [Fixed Count of Retries](/design-patterns/fixed-count-retries): Cap attempts at a fixed number instead of alerting at a threshold.
- [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns.

### References

- [Temporal Retry Policies](/encyclopedia/retry-policies)

---

# Saga Pattern

Source: https://docs.temporal.io/design-patterns/saga-pattern

> Manages distributed transactions with compensating actions. Each step has a compensation that undoes its effects if subsequent steps fail.

## Overview

The Saga pattern manages distributed transactions across multiple services by coordinating a sequence of local transactions, each with a compensating action that can undo its effects if subsequent steps fail.

## Problem

In distributed systems, you need to maintain data consistency across multiple services or databases without using traditional ACID transactions.
When a multi-step business process fails partway through, you must undo the effects of completed steps to maintain system consistency.
Traditional two-phase commit does not scale well and creates tight coupling between services.

## Solution

You implement each step as a local transaction with a corresponding compensation transaction.
If any step fails, you execute compensation transactions in reverse order to undo the effects of all completed steps.
You register compensations as each step completes, then automatically trigger them when errors occur to ensure cleanup happens reliably.

The following diagram shows the worked example used by the runner: opening a customer account in four steps, where `addBankAccount` simulates a downstream failure to trigger compensation.

```mermaid
flowchart TD
    Start([Start Saga]) --> Step1[Step 1: createAccount]
    Step1 -->|Success| Step2[Step 2: addAddress]
    Step1 -->|Failure| End([End: Failed])

    Step2 -->|Success| Step3[Step 3: addClient]
    Step2 -->|Failure| Comp1[clearPostalAddresses]

    Step3 -->|Success| Step4[Step 4: addBankAccount]
    Step3 -->|Failure| Comp2[removeClient]

    Step4 -->|Success| Complete([End: Success])
    Step4 -->|Failure| Comp3[disconnectBankAccounts]

    Comp3 --> Comp2
    Comp2 --> Comp1
    Comp1 --> End
    classDef success stroke-width:1px
    classDef compensation stroke-width:1px
    classDef complete stroke-width:1px
    classDef fail stroke-width:1px
    class Step1,Step2,Step3,Step4 success
    class Comp1,Comp2,Comp3 compensation
    class Complete complete
    class End fail
```

The following describes each step in the diagram:

1. The Saga begins by executing Step 1 (`createAccount`).
2. If Step 1 succeeds, the Workflow proceeds to Step 2 (`addAddress`). If it fails, the Saga ends immediately — no compensations are registered yet.
3. If Step 2 succeeds, the Workflow proceeds to Step 3 (`addClient`). If it fails, the Workflow runs `clearPostalAddresses`.
4. If Step 3 succeeds, the Workflow proceeds to Step 4 (`addBankAccount`). If it fails, the Workflow runs `removeClient`, then `clearPostalAddresses`.
5. If Step 4 succeeds, the Saga completes. If it fails, the Workflow runs all three compensations in reverse: `disconnectBankAccounts`, `removeClient`, `clearPostalAddresses`. Note that `disconnectBankAccounts` is registered before `addBankAccount` runs, so it executes even if `addBankAccount` failed mid-flight — its implementation must be idempotent.

## Implementation

The following examples show how each SDK implements the Saga pattern.
Each language uses a different mechanism to register and execute compensations, but the core principle is the same: register a compensation before or after each step, and run all compensations in reverse order on failure.

**Python**

```python
# workflows.py
from temporalio import workflow

@workflow.defn
class OpenAccountWorkflow:
    @workflow.run
    async def run(self, req: OpenAccountRequest) -> str:
        compensations = []

        try:
            # Step 1: createAccount has no compensation — leaving an empty
            # account stub on later failure is acceptable.
            await workflow.execute_activity(
                create_account, req,
                start_to_close_timeout=timedelta(seconds=10),
            )

            # Register compensation for Step 2 BEFORE execution
            compensations.append(
                lambda: workflow.execute_activity(
                    clear_postal_addresses, req,
                    start_to_close_timeout=timedelta(seconds=10),
                )
            )
            # Step 2: Add postal address
            await workflow.execute_activity(
                add_address, req,
                start_to_close_timeout=timedelta(seconds=10),
            )

            # Register compensation for Step 3 BEFORE execution
            compensations.append(
                lambda: workflow.execute_activity(
                    remove_client, req,
                    start_to_close_timeout=timedelta(seconds=10),
                )
            )
            # Step 3: Add client record
            await workflow.execute_activity(
                add_client, req,
                start_to_close_timeout=timedelta(seconds=10),
            )

            # Register compensation for Step 4 BEFORE execution
            compensations.append(
                lambda: workflow.execute_activity(
                    disconnect_bank_accounts, req,
                    start_to_close_timeout=timedelta(seconds=10),
                )
            )
            # Step 4: Link bank account (this step fails in the demo)
            await workflow.execute_activity(
                add_bank_account, req,
                start_to_close_timeout=timedelta(seconds=10),
            )
        except Exception:
            # On error, run compensations in reverse order
            for compensation in reversed(compensations):
                await compensation()
            raise
```

**Go**

```go
// open_account_workflow.go
func OpenAccountWorkflow(ctx workflow.Context, req OpenAccountRequest) error {
    var compensations []func()
    runCompensations := func() {
        for i := len(compensations) - 1; i >= 0; i-- {
            compensations[i]()
        }
    }

    // Step 1: CreateAccount has no compensation — leaving an empty account
    // stub on later failure is acceptable.
    if err := workflow.ExecuteActivity(ctx, CreateAccount, req).Get(ctx, nil); err != nil {
        return err
    }

    // Register compensation for Step 2 BEFORE execution
    compensations = append(compensations, func() {
        _ = workflow.ExecuteActivity(ctx, ClearPostalAddresses, req).Get(ctx, nil)
    })
    if err := workflow.ExecuteActivity(ctx, AddAddress, req).Get(ctx, nil); err != nil {
        runCompensations()
        return err
    }

    // Register compensation for Step 3 BEFORE execution
    compensations = append(compensations, func() {
        _ = workflow.ExecuteActivity(ctx, RemoveClient, req).Get(ctx, nil)
    })
    if err := workflow.ExecuteActivity(ctx, AddClient, req).Get(ctx, nil); err != nil {
        runCompensations()
        return err
    }

    // Register compensation for Step 4 BEFORE execution
    compensations = append(compensations, func() {
        _ = workflow.ExecuteActivity(ctx, DisconnectBankAccounts, req).Get(ctx, nil)
    })
    if err := workflow.ExecuteActivity(ctx, AddBankAccount, req).Get(ctx, nil); err != nil {
        runCompensations()
        return err
    }

    return nil
}
```

**Java**

```java
// OpenAccountWorkflow.java
@WorkflowInterface
public interface OpenAccountWorkflow {
    @WorkflowMethod
    String openAccount(OpenAccountRequest req);
}

public class OpenAccountWorkflowImpl implements OpenAccountWorkflow {
    private final Activities activities = Workflow.newActivityStub(
        Activities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(10))
            .build());

    @Override
    public String openAccount(OpenAccountRequest req) {
        // Create a Saga instance with compensation options
        Saga saga = new Saga(new Saga.Options.Builder()
            .setParallelCompensation(false) // Run compensations sequentially
            .build());

        try {
            // Step 1: createAccount has no compensation — leaving an empty
            // account stub on later failure is acceptable.
            activities.createAccount(req);

            // Register compensation for Step 2 BEFORE execution
            saga.addCompensation(activities::clearPostalAddresses, req);
            activities.addAddress(req);

            // Register compensation for Step 3 BEFORE execution
            saga.addCompensation(activities::removeClient, req);
            activities.addClient(req);

            // Register compensation for Step 4 BEFORE execution
            saga.addCompensation(activities::disconnectBankAccounts, req);
            activities.addBankAccount(req);

            return "Account " + req.accountId() + " opened";

        } catch (Exception e) {
            // On any error, run all registered compensations in reverse order
            saga.compensate();
            throw e;
        }
    }
}
```

**TypeScript**

<!--SNIPSTART typescript-pattern-saga-pattern-workflow-->
[typescript-sandbox/src/design-patterns/saga/workflows.ts](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/typescript-sandbox/src/design-patterns/saga/workflows.ts)
```ts
type Compensation = () => Promise<void>;

export async function openAccount(req: OpenAccountRequest): Promise<string> {
  const compensations: Compensation[] = [];

  try {
    // Step 1: createAccount has no compensation. Leaving an empty account
    // stub on later failure is acceptable.
    await acts.createAccount(req);

    // Register compensation for Step 2 BEFORE execution
    compensations.unshift(() => acts.clearPostalAddresses(req));
    await acts.addAddress(req);

    // Register compensation for Step 3 BEFORE execution
    compensations.unshift(() => acts.removeClient(req));
    await acts.addClient(req);

    // Register compensation for Step 4 BEFORE execution
    compensations.unshift(() => acts.disconnectBankAccounts(req));
    await acts.addBankAccount(req);

    return `Account ${req.accountId} opened`;
  } catch (err) {
    // On error, run all compensations in reverse order (unshift keeps them in LIFO already)
    for (const compensate of compensations) {
      await compensate();
    }
    // Re-throw so the Workflow Execution reports the original failure
    throw err;
  }
}
```
<!--SNIPEND-->

The key differences between SDKs are:

- **Go**: Uses a slice of closures and iterates from the end on error. (Some samples use `defer` instead — both achieve LIFO; the slice form makes the rollback trigger explicit.)
- **Python**: Uses a list with `reversed()` to iterate compensations in LIFO order on error.
- **TypeScript**: Uses an array with `unshift()` to maintain LIFO order, and manually iterates on error.
- **Java**: Uses the SDK's `Saga` helper to track compensations and trigger them with `saga.compensate()`.

In all SDKs, compensations are registered before Activity execution and run in reverse order of registration.
All compensations must be idempotent and able to handle cases where the forward Activity never executed.

### When to register compensations

There are two approaches for when to register compensation Activities:

1. **Register before Activity execution** (recommended for safety): This ensures the compensation runs even if the Activity fails after partial completion. For example, a credit card may be charged but the Activity fails before returning success. The compensation must be idempotent and handle cases where the forward Activity never executed (no-op). This is the safer default when Activities have side effects that may occur before failure.

2. **Register after Activity execution** (appropriate when safe): This only compensates Activities that completed successfully. The compensation logic is simpler because you do not need to check whether the forward action occurred. This approach is appropriate when Activities are truly atomic (all-or-nothing). The risk is partial completion without compensation if the Activity fails mid-execution.

The choice depends on your Activity's failure characteristics and whether the compensation can safely handle cases where the forward Activity never executed.
When in doubt, register compensations before execution and ensure they are idempotent.

## When to use

The Saga pattern is a good fit when you need to maintain consistency across multiple services or databases, traditional distributed transactions (two-phase commit) are too slow or unavailable, you can define compensating actions for each step in your business process, eventual consistency is acceptable for your use case, and you need to handle long-running transactions that may span hours or days.

It is not a good fit for operations that require strong ACID consistency, single-service transactions that can use a local database transaction, processes where compensations cannot be defined, or operations that must appear atomic to external observers.

## Benefits and trade-offs

The Saga pattern maintains eventual consistency without distributed locks, and each service can use its own database and transaction model.
Temporal's durable execution guarantees that compensations will execute even after Worker failures.
The pattern scales better than two-phase commit protocols.

The trade-offs to consider are that only eventual consistency is provided — intermediate states are visible to other processes.
You must design idempotent compensation Activities, and compensation logic must be maintained alongside forward logic.
Some operations may not have meaningful compensations.

## Comparison with alternatives

| Approach | Consistency | Rollback mechanism | Coupling | Scalability |
| :--- | :--- | :--- | :--- | :--- |
| Saga (orchestration) | Eventual | Compensating transactions | Loose | High |
| Two-phase commit | Strong (ACID) | Distributed lock/rollback | Tight | Low |
| Saga (choreography) | Eventual | Event-driven compensations | Very loose | High |
| Local transaction | Strong (ACID) | Database rollback | None | Single service |

## Best practices

- **Make all compensations idempotent.** Compensations may run even when the forward Activity never executed (if registered before execution) or may run multiple times on retry. Use idempotency keys to ensure safe re-execution.
- **Register compensations before Activity execution.** This ensures cleanup runs even if the Activity fails after partial completion. The compensation must handle the case where the forward action never occurred (no-op).
- **Use idempotency keys for forward Activities.** Pass a unique identifier (such as a client ID or Workflow ID) to each Activity so retries do not create duplicate side effects.
- **Set StartToCloseTimeout on compensation Activities.** Set a `StartToCloseTimeout` but avoid `ScheduleToCloseTimeout` on compensations. Do not set Workflow-level timeouts — let compensations retry until they succeed.
- **Use a disconnected context for cancellation compensation.** In Go, use `NewDisconnectedContext` to run compensation Activities after Workflow cancellation, since the original context is already cancelled.
- **Keep compensation payloads small.** Pass references (IDs, URLs) instead of full data objects to avoid exceeding the 2 MB payload limit.
- **Log compensation failures but continue.** If a compensation fails, log the error and continue executing remaining compensations. In production, alert for manual intervention on persistent compensation failures.
- **Re-throw the original error after compensating.** Always re-throw the original exception after running compensations so the Workflow reports the correct failure reason.

## Common pitfalls

- **Non-idempotent compensations.** Compensations may run even when the forward Activity never executed (if registered before execution) or may run multiple times on retry. All compensations must be idempotent.
- **Forgetting to register a compensation.** If a step succeeds but its compensation was never registered, a later failure leaves that step's effects permanently in place.
- **Compensations that can fail permanently.** If a compensation Activity fails with a non-retryable error, the Saga cannot fully roll back. Design compensations with generous retry policies.
- **Large payloads in compensation state.** Passing large objects through the compensation chain can exceed the 2 MB payload limit. Use references (IDs, URLs) instead of full data.
- **Swallowing the ContinueAsNew exception in TypeScript.** In TypeScript, `continueAsNew` works by throwing a special exception. A `catch` block that does not re-throw it, or a `finally` block that returns a value, silently prevents Continue-As-New.

## Related

### Patterns

- **[Error Handling & Retry Patterns](/design-patterns/error-handling-patterns)**: Often combined with the Saga pattern to handle transient failures before compensating.
- **[Child Workflows](/design-patterns/child-workflows)**: You can use Child Workflows to organize complex Sagas with multiple sub-Sagas.
- **[Long-Running Activity](/design-patterns/long-running-activity)**: Heartbeats work well with long-running compensation Activities.
- **[Early Return](/design-patterns/early-return)**: You can combine Early Return with the Saga pattern to return initialization results before compensation runs.

### Guides

- [Recover business processes without restarting](/guides/recover-without-restart): Adds LIFO saga compensation to a multi-step loan pipeline, with pre-registered compensations and a recoverable rollback loop for compensations that fail.

### Sample code

- [Go Sample](https://github.com/temporalio/samples-go/tree/main/saga) — Saga with `defer`-based compensations.
- [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloSaga.java) — Saga with the `Saga` API.
- [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/saga) — Saga with array-based compensations.
- [TypeScript runnable sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/typescript-sandbox/src/design-patterns/saga) — The code on this page, with a Worker, a starter, and tests.
- [Python Sample](https://github.com/temporalio/samples-python) — Saga with list-based compensations.
- [.NET Sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Saga) — Saga with `try`/`catch` compensations.
- [Ruby Sample](https://github.com/temporalio/samples-ruby/tree/main/saga) — Saga with `rescue`-based compensations.
- [Python SDK implementation guide](/develop/python/best-practices/error-handling#implement-saga-pattern) — Full walkthrough of the compensation-list technique, including retry and rollback code.

---

# Signal with Start Pattern

Source: https://docs.temporal.io/design-patterns/signal-with-start

> Starts a Workflow when Signaling it if it does not already exist. If already running, it receives the Signal directly.

## Overview

Signal with Start is a pattern that lazily creates Workflows when Signaling them.
If the Workflow is already running, it receives the Signal; if not, the Workflow starts first and then receives the Signal.
This enables entity Workflows that only exist when needed and can receive operations throughout their lifetime.

## Problem

In distributed systems, you often need Workflows that represent long-lived entities (accounts, shopping carts, user sessions), consume events from streams (Kafka, SQS) and trigger certain behaviors of an aggregate or entity, receive multiple operations over time, should only exist when there is work to do, and need to handle the first operation without special client logic.

Without Signal with Start, clients must check if the Workflow exists before Signaling, start the Workflow if it does not exist and then Signal it, handle race conditions when multiple clients try to start the same Workflow, and write complex coordination logic.

## Solution

Temporal's Signal with Start API atomically starts a Workflow (if not running) and delivers a Signal in a single operation.
The client does not need to know whether the Workflow exists — the platform handles it automatically.

```mermaid
sequenceDiagram
    participant C as Client
    participant T as Temporal
    participant W as Workflow

    C->>T: SignalWithStart(workflowId, signal, args)
    T->>T: Check if workflow exists
    alt Workflow does not exist
        T->>W: Start Workflow
        activate W
        T->>W: Deliver Signal
        W->>W: Process Signal
    else Workflow already running
        T->>W: Deliver Signal only (no start)
        W->>W: Process Signal
    end
    deactivate W
```

The following describes each step in the diagram:

1. The client calls SignalWithStart with a Workflow ID, Signal name, and arguments.
2. Temporal checks whether a Workflow with that ID is already running.
3. If the Workflow does not exist, Temporal starts it and then delivers the Signal.
4. If the Workflow is already running, Temporal delivers the Signal without starting a new instance.

## Implementation

### Basic Signal-With-Start

The following examples show a shopping cart entity Workflow that is created lazily when the first item is added.
Each SDK uses its own API to atomically start the Workflow (if needed) and deliver the Signal.

**Python**

```python
# client.py
from temporalio.client import Client
from workflows import ShoppingCartWorkflow, AddItemSignal

async def add_item(client: Client, cart_id: str, item_id: str, product_id: str, quantity: int) -> None:
    # Atomically start workflow (if needed) and deliver signal
    await client.start_workflow(
        ShoppingCartWorkflow.run,
        id=f"cart-{cart_id}",
        task_queue="carts",
        start_signal="add_item",
        start_signal_args=[AddItemSignal(item_id=item_id, product_id=product_id, quantity=quantity)],
    )

# workflows.py
from dataclasses import dataclass
from temporalio import workflow

@dataclass
class AddItemSignal:
    item_id: str
    product_id: str
    quantity: int

@dataclass
class CartItem:
    product_id: str
    quantity: int

@workflow.defn
class ShoppingCartWorkflow:
    def __init__(self) -> None:
        self.processed_items: set[str] = set()
        self.items: list[CartItem] = []

    @workflow.run
    async def run(self) -> None:
        await workflow.wait_condition(lambda: False)  # Run forever (entity workflow)

    @workflow.signal
    def add_item(self, sig: AddItemSignal) -> None:
        if sig.item_id in self.processed_items:
            return  # Idempotency: ignore duplicate signals
        self.processed_items.add(sig.item_id)
        self.items.append(CartItem(product_id=sig.product_id, quantity=sig.quantity))
```

**Go**

```go
// client.go
func AddItem(ctx context.Context, cartID, itemID, productID string, quantity int) error {
	opts := client.StartWorkflowOptions{
		ID:        "cart-" + cartID,
		TaskQueue: "carts",
	}

	// Atomically start workflow (if needed) and deliver signal
	sig := AddItemSignal{ItemID: itemID, ProductID: productID, Quantity: quantity}
	_, err := c.SignalWithStartWorkflow(ctx, "cart-"+cartID, "addItem", sig, opts, ShoppingCartWorkflow)
	return err
}

// workflow.go
func ShoppingCartWorkflow(ctx workflow.Context) error {
	processedItems := make(map[string]bool)
	var items []CartItem

	addItemCh := workflow.GetSignalChannel(ctx, "addItem")
	workflow.Go(ctx, func(ctx workflow.Context) {
		for {
			var sig AddItemSignal
			addItemCh.Receive(ctx, &sig)
			if processedItems[sig.ItemID] {
				continue // Idempotency: ignore duplicate signals
			}
			processedItems[sig.ItemID] = true
			items = append(items, CartItem{ProductID: sig.ProductID, Quantity: sig.Quantity})
		}
	})

	workflow.Await(ctx, func() bool { return false }) // Run forever (entity workflow)
	return nil
}
```

**Java**

```java
// ShoppingCartManager.java
public class ShoppingCartManager {
  public void addItem(String cartId, String itemId, String productId, int quantity) {
    WorkflowOptions options = WorkflowOptions.newBuilder()
        .setWorkflowId("cart-" + cartId)
        .setTaskQueue("carts")
        .build();

    ShoppingCartWorkflow workflow =
        workflowClient.newWorkflowStub(ShoppingCartWorkflow.class, options);

    // Atomically start workflow (if needed) and deliver signal
    BatchRequest request = workflowClient.newSignalWithStartRequest();
    request.add(workflow::run);
    request.add(workflow::addItem, itemId, productId, quantity);
    workflowClient.signalWithStart(request);
  }
}

// ShoppingCartWorkflow.java
@WorkflowInterface
public interface ShoppingCartWorkflow {
  @WorkflowMethod
  void run();

  @SignalMethod
  void addItem(String itemId, String productId, int quantity);
}

public class ShoppingCartWorkflowImpl implements ShoppingCartWorkflow {
  private Set<String> processedItems = new HashSet<>();
  private List<CartItem> items = new ArrayList<>();

  @Override
  public void run() {
    Workflow.await(() -> false); // Run forever (entity workflow)
  }

  @Override
  public void addItem(String itemId, String productId, int quantity) {
    if (!processedItems.add(itemId)) {
      return; // Idempotency: ignore duplicate signals
    }
    items.add(new CartItem(productId, quantity));
  }
}
```

**TypeScript**

```typescript
// client.ts
export async function addItem(
  cartId: string,
  itemId: string,
  productId: string,
  quantity: number
) {
  // Atomically start workflow (if needed) and deliver signal
  const handle = await client.workflow.signalWithStart(shoppingCartWorkflow, {
    workflowId: `cart-${cartId}`,
    taskQueue: 'carts',
    signal: 'addItem',
    signalArgs: [itemId, productId, quantity],
  });
}

// workflow.ts
export async function shoppingCartWorkflow(): Promise<void> {
  const processedItems = new Set<string>();
  const items: CartItem[] = [];

  setHandler(addItemSignal, (itemId: string, productId: string, quantity: number) => {
    if (processedItems.has(itemId)) {
      return; // Idempotency: ignore duplicate signals
    }
    processedItems.add(itemId);
    items.push({ productId, quantity });
  });

  await condition(() => false); // Run forever (entity workflow)
}

export const addItemSignal = defineSignal<[string, string, number]>('addItem');
```

In all SDKs, the Workflow ID is derived from the business entity (the cart ID), ensuring one Workflow per entity.
The Signal handler checks a set of processed item IDs to prevent duplicate processing.
The Workflow blocks indefinitely, acting as a long-lived entity that receives operations over its lifetime.

## When to use

The Signal with Start pattern is a good fit for entity Workflows (accounts, shopping carts, user sessions, clusters), event-driven architectures (Kafka consumers, message queue processors), Workflows that receive multiple operations over their lifetime, lazy entity creation where you only create when the first operation arrives, and fire-and-forget operations where immediate response is not needed.

It is not a good fit for one-time operations (use REJECT_DUPLICATE policy instead), request-response patterns requiring synchronous confirmation (use Update with Start), or operations that need immediate return values.

## Benefits and trade-offs

Signal with Start provides an atomic operation — start and Signal happen atomically with no race conditions.
Workflows only exist when needed (lazy creation).
The client does not need to check if the Workflow exists.
The operation is safe to retry because duplicate starts are handled by the Workflow ID.
The pattern is a natural fit for long-lived business entities.

The trade-offs to consider are that Signals are fire-and-forget with no immediate confirmation that the Signal was processed.
You still need to track processed operation IDs in the Workflow for Signal idempotency.
Workflows must handle unbounded execution (use Continue-As-New).
Signals do not return values — use Queries or Updates for that.

Both ALLOW_DUPLICATE and ALLOW_DUPLICATE_FAILED_ONLY work well with Signal with Start:

- **ALLOW_DUPLICATE** (default): Allows a new Workflow Execution with the same ID after the previous one has closed (completed, failed, timed out, terminated, or cancelled). Does not affect a currently running Workflow — Signal with Start delivers the Signal to the running execution.
- **ALLOW_DUPLICATE_FAILED_ONLY**: Allows restart only if the previous run failed — prevents accidental restarts of running Workflows.
- **REJECT_DUPLICATE**: Prevents any duplicate starts — useful for one-time operations, not entity Workflows.
- **TERMINATE_IF_RUNNING**: Terminates the running Workflow and starts a new one — use with caution.

## Comparison with alternatives

| Approach | Use case | Response type | Idempotency |
| :--- | :--- | :--- | :--- |
| Signal with Start | Entity Workflows | Fire-and-forget | Signal-level |
| Update with Start | Request-response | Sync return value | Update-level |
| REJECT_DUPLICATE | One-time operations | Async (Workflow ID) | Workflow-level |

## Best practices

- **Derive Workflow ID from entity.** Use stable business identifiers (account ID, user ID).
- **Implement Signal idempotency.** Track processed operation IDs to prevent duplicates.
- **Use WorkflowInit.** Initialize state before Signals are delivered (Java, .NET, and Python's `__init__`).
- **Handle unbounded execution.** Use Continue-As-New for long-running entity Workflows.
- **Choose the right Workflow ID policy.** Use ALLOW_DUPLICATE_FAILED_ONLY for entity Workflows.
- **Include operation IDs.** Every Signal should include a unique operation or reference ID.
- **Return early.** Check for duplicates at the start of Signal handlers.

## Common pitfalls

- **Not implementing Signal idempotency.** Signals can be delivered more than once (for example, client retries). Without tracking processed operation IDs, the Workflow processes duplicates.
- **Unbounded history growth.** Entity Workflows that receive many Signals without calling Continue-As-New will hit the 51,200 event or 10K Signal limit. Use `isContinueAsNewSuggested()` to trigger Continue-As-New.
- **Losing pending Signals on Continue-As-New.** Drain all pending Signals before calling Continue-As-New, and pass unprocessed ones as input to the new execution.
- **Expecting a return value from Signals.** Signals are fire-and-forget. If you need a synchronous response, use Updates or Update-with-Start instead.
- **Race between SignalWithStart and Continue-As-New.** Temporal prevents this race — if a Signal arrives while the Workflow is completing via Continue-As-New, the Workflow rewinds to process the Signal first.

## Related

### Patterns

- **[Entity Workflow](/design-patterns/entity-workflow)**: Long-running Workflows representing business entities.
- **[Continue-As-New](/design-patterns/continue-as-new)**: Managing unbounded Workflow history.
- **[Request-Response via Updates](/design-patterns/request-response-via-updates)**: When you need synchronous responses instead of fire-and-forget.
- **[Early Return](/design-patterns/early-return)**: Update-with-Start for request-response with lazy initialization.

### References

- [Sending Messages](/sending-messages): Canonical reference for Signal-with-Start and Update-with-Start as delivery mechanisms.
- [Handling Messages](/handling-messages): Canonical reference for writing Signal and Update handlers, including idempotency and validation.

### Sample code

**Python**
- [Hello Signal](https://github.com/temporalio/samples-python/tree/main/hello/hello_signal.py) — Basic Signal handling in a Workflow.
- [Message Passing](https://github.com/temporalio/samples-python/tree/main/message_passing/introduction) — Introduction to message passing with Signals, Queries, and Updates.

**Java**
- [Hello Signal](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloSignal.java) — Basic Signal handling in a Workflow.
- [Safe Message Passing](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Concurrent Signal handling with validation.

**TypeScript**
- [Signals and Queries](https://github.com/temporalio/samples-typescript/tree/main/signals-queries) — Signal and Query usage in a Workflow.

**Go**
- [Await Signals](https://github.com/temporalio/samples-go/tree/main/await-signals) — Waiting for Signals with timeout.

---

# Sliding Window

Source: https://docs.temporal.io/design-patterns/sliding-window

> Maintains a fixed number of concurrently active Child Workflows, starting a new one each time an existing one completes.

> **ℹ️ TLDR:**
> Keep exactly `windowSize` child Workflows running at all times — each completion signal triggers the next record to start immediately. Use this when your record set is arbitrarily large, you need **bounded concurrency** to protect downstream systems, and you want higher throughput than a sequential Batch Iterator provides.

## Overview

The Sliding Window pattern maintains a fixed-size pool of concurrently running child Workflows. As each child completes it signals the parent, which immediately starts a replacement — keeping the concurrency level constant and progressing at the rate of the fastest processor. Continue-as-New prevents the parent's history from growing without bound.

## Problem

The [Batch Iterator](/design-patterns/batch-iterator) processes records sequentially — the overall throughput is limited by the slowest record in each page. The [Fan-Out](/design-patterns/fanout-child-workflows) pattern starts all children at once, which can overwhelm downstream systems when the record set is large.

You need a way to process an arbitrarily large record set with bounded concurrency, maximum throughput within that bound, and protection against history bloat.

## Solution

The parent Workflow keeps a live count of in-flight children (`active`) and runs a single loop that starts a child whenever a slot is free. The first `windowSize` slots are free, so those children start immediately; after that, a backpressure condition blocks each start until an in-flight child signals completion and frees a slot. Each child processes one record and, when finished, signals the parent, which decrements `active` and starts the next record's child.

Continue-as-New is called after the parent has started `windowSize` children. Because child Workflows have stable Workflow IDs and Continue-as-New preserves the parent's Workflow ID, children started by a previous run can still signal the current run. The parent carries `active` into the next run so it knows how many carried-over children will still signal it.

```mermaid
flowchart TD
    Records["📋 Record IDs\n[r0, r1, r2, ...]"]
    Parent["Parent Workflow\n(window size = W)"]
    C1["Child r0\n✅ done"]
    C2["Child r1\n⏳ running"]
    C3["Child r2\n⏳ running"]
    C4["Child r3\n🆕 started"]
    CAN["continueAsNew\n(startIndex + W)"]

    Records --> Parent
    Parent <-->|"start W children /<br/>signal: complete"| C1
    Parent --> C2
    Parent --> C3

    Parent -->|"slot free → start next"| C4

    Parent -->|"after W children started"| CAN
```

The following describes each step in the diagram:

1. The parent Workflow starts with a list of record IDs and a configured `windowSize`.
2. It starts the first `windowSize` children concurrently, one per record. Each child reads the parent's Workflow ID from its own Workflow metadata, so it knows where to signal.
3. As each child completes, it sends a completion signal to the parent.
4. The parent receives the signal, decrements its in-flight counter (`active`), and starts the next child (the next record in the list).
5. After starting `windowSize` children in total, the parent calls `continueAsNew` with the updated start index. The window slides forward without gaps because the parent's Workflow ID is preserved across runs.
6. Children from previous runs that have not yet signalled will find the new run when they send the signal, because the parent Workflow ID remains the same.

## Implementation

The following examples show how each SDK implements the Sliding Window pattern.

**TypeScript**

```typescript
// workflows.ts
import {
  ApplicationFailure,
  ParentClosePolicy,
  condition,
  continueAsNew,
  defineSignal,
  getExternalWorkflowHandle,
  proxyActivities,
  setHandler,
  startChild,
  workflowInfo,
} from "@temporalio/workflow";
import type * as activities from "./activities";
import { COMPLETION_SIGNAL, TASK_QUEUE, WINDOW_SIZE, type SlidingWindowInput } from "./shared";

const { processRecord } = proxyActivities<typeof activities>({
  startToCloseTimeout: "30 seconds",
});

export const completionSignal = defineSignal<[string]>(COMPLETION_SIGNAL);

// Child Workflow: processes one record and signals the parent on completion.
export async function recordProcessorWorkflow(recordId: string): Promise<void> {
  await processRecord(recordId);
  // Read the parent's Workflow ID from context. It is stable across the parent's
  // Continue-as-New runs, so signaling by Workflow ID (no run ID) always reaches
  // the current run. Ignore NOT_FOUND — the parent's final run may have completed.
  try {
    const parent = getExternalWorkflowHandle(workflowInfo().parent!.workflowId);
    await parent.signal(completionSignal, recordId);
  } catch (err) {
    if (!(err instanceof ApplicationFailure && err.type === "NOT_FOUND")) throw err;
  }
}

// Parent Workflow: maintains a fixed window of concurrent Child Workflows and calls
// Continue-as-New after dispatching windowSize children so history stays bounded.
export async function slidingWindowWorkflow(input: SlidingWindowInput): Promise<number> {
  const { recordIds, windowSize = WINDOW_SIZE, startIndex = 0 } = input;
  const parentId = workflowInfo().workflowId;
  // Total records completed across all runs, carried over via Continue-as-New.
  let totalProcessed = input.totalProcessed ?? 0;
  // Children started in this run; triggers Continue-as-New once it hits windowSize.
  let dispatched = 0;
  // Live in-flight count: +1 per start, -1 per completion signal, carried across runs.
  let active = input.active ?? 0;

  setHandler(completionSignal, () => {
    active--;
    totalProcessed++;
  });

  // Slide the window: keep it full, starting one child per free slot. The first
  // (windowSize - active) slots are already free, so those children start without
  // waiting; after that, each start waits for an in-flight child to free a slot.
  let nextIndex = startIndex;
  while (nextIndex < recordIds.length) {
    // Backpressure: block until the window has a free slot.
    await condition(() => active < windowSize);
    await startChild(recordProcessorWorkflow, {
      args: [recordIds[nextIndex]],
      workflowId: `${parentId}/record-${recordIds[nextIndex]}`,
      taskQueue: TASK_QUEUE,
      parentClosePolicy: ParentClosePolicy.ABANDON,
    });
    nextIndex++;
    dispatched++;
    active++;

    // Once this run has filled the window with fresh children, Continue-as-New so
    // history stays bounded. Carry active so the next run knows how many children
    // will still signal it.
    if (dispatched >= windowSize) {
      await continueAsNew<typeof slidingWindowWorkflow>({ recordIds, windowSize, startIndex: nextIndex, totalProcessed, active });
    }
  }

  // Wait for all remaining in-flight children to complete.
  await condition(() => active === 0);
  return totalProcessed;
}
```

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow
from temporalio.exceptions import ApplicationError
from temporalio.workflow import ParentClosePolicy, continue_as_new
from activities import process_record
from shared import COMPLETION_SIGNAL, TASK_QUEUE, WINDOW_SIZE, SlidingWindowInput

@workflow.defn
class RecordProcessorWorkflow:
    """Child Workflow: processes one record and signals the parent on completion."""

    @workflow.run
    async def run(self, record_id: str) -> None:
        await workflow.execute_activity(
            process_record,
            record_id,
            start_to_close_timeout=timedelta(seconds=30),
        )
        # Read the parent's Workflow ID from context. It is stable across the parent's
        # continue_as_new runs, so signaling by Workflow ID always reaches the current
        # run. Ignore NOT_FOUND — the parent's final run may have already completed.
        parent = workflow.get_external_workflow_handle(workflow.info().parent.workflow_id)
        try:
            await parent.signal(COMPLETION_SIGNAL, record_id)
        except ApplicationError as e:
            if "not found" not in str(e).lower():
                raise

@workflow.defn
class SlidingWindowWorkflow:
    """Parent Workflow: maintains a fixed window of concurrent Child Workflows."""

    def __init__(self) -> None:
        # Live in-flight count: +1 per start, -1 per completion signal, carried across
        # runs. An instance field (not a run() local) because the signal handler is a
        # separate method and completions can signal before run() starts.
        self._active = 0
        # Total records completed across all runs, carried over via continue_as_new.
        self._total_processed = 0

    @workflow.signal(name=COMPLETION_SIGNAL)
    def record_completed(self, record_id: str) -> None:
        self._active -= 1
        self._total_processed += 1

    @workflow.run
    async def run(self, input: SlidingWindowInput) -> int:
        # Use += so completions that signal before run() starts are preserved.
        self._total_processed += input.total_processed
        self._active += input.active
        record_ids = input.record_ids
        window_size = input.window_size
        parent_id = workflow.info().workflow_id
        next_index = input.start_index
        # Children started in this run; triggers continue_as_new once it hits window_size.
        dispatched = 0

        # Slide the window: keep it full, starting one child per free slot. The first
        # (window_size - active) slots are already free, so those children start without
        # waiting; after that, each start waits for an in-flight child to free a slot.
        while next_index < len(record_ids):
            # Backpressure: block until the window has a free slot.
            await workflow.wait_condition(lambda: self._active < window_size)
            await workflow.start_child_workflow(
                RecordProcessorWorkflow.run,
                record_ids[next_index],
                id=f"{parent_id}/record-{record_ids[next_index]}",
                task_queue=TASK_QUEUE,
                parent_close_policy=ParentClosePolicy.ABANDON,
            )
            next_index += 1
            dispatched += 1
            self._active += 1

            # Once this run has filled the window with fresh children, continue_as_new so
            # history stays bounded. Carry _active so the next run knows how many children
            # will still signal it.
            if dispatched >= window_size:
                continue_as_new(args=[SlidingWindowInput(
                    record_ids=record_ids,
                    window_size=window_size,
                    start_index=next_index,
                    total_processed=self._total_processed,
                    active=self._active,
                )])

        # Wait for all remaining in-flight children to complete.
        await workflow.wait_condition(lambda: self._active == 0)
        return self._total_processed
```

**Go**

```go
// workflows.go
package main

import (
	"fmt"
	"strings"
	"time"

	enums "go.temporal.io/api/enums/v1"
	"go.temporal.io/sdk/workflow"
)

const CompletionSignal = "recordCompleted"

// RecordProcessorWorkflow is the child Workflow: it processes one record and
// signals the parent on completion.
func RecordProcessorWorkflow(ctx workflow.Context, recordID string) error {
	ao := workflow.ActivityOptions{StartToCloseTimeout: 30 * time.Second}
	ctx = workflow.WithActivityOptions(ctx, ao)

	if err := workflow.ExecuteActivity(ctx, ProcessRecord, recordID).Get(ctx, nil); err != nil {
		return err
	}

	// Read the parent's Workflow ID from context. It is stable across the parent's
	// ContinueAsNew runs, so signaling by Workflow ID (empty run ID) always reaches
	// the current run. Ignore not-found — the parent's final run may have completed.
	parentID := workflow.GetInfo(ctx).ParentWorkflowExecution.ID
	err := workflow.SignalExternalWorkflow(ctx, parentID, "", CompletionSignal, recordID).Get(ctx, nil)
	if err != nil && strings.Contains(err.Error(), "not found") {
		return nil
	}
	return err
}

// SlidingWindowWorkflow is the parent Workflow: it maintains a fixed window of
// concurrent Child Workflows and calls ContinueAsNew after dispatching windowSize
// children so history stays bounded.
func SlidingWindowWorkflow(ctx workflow.Context, input SlidingWindowInput) (int, error) {
	windowSize := input.WindowSize
	if windowSize <= 0 {
		windowSize = WindowSize
	}
	recordIDs := input.RecordIDs
	parentID := workflow.GetInfo(ctx).WorkflowExecution.ID
	completedCh := workflow.GetSignalChannel(ctx, CompletionSignal)

	nextIndex := input.StartIndex
	// Total records completed across all runs, carried over via ContinueAsNew.
	totalProcessed := input.TotalProcessed
	// Children started in this run; triggers ContinueAsNew once it hits windowSize.
	dispatched := 0
	// Live in-flight count: +1 per start, -1 per completion signal, carried across runs.
	active := input.Active

	startChild := func(recordID string) error {
		cwo := workflow.ChildWorkflowOptions{
			WorkflowID:        fmt.Sprintf("%s/record-%s", parentID, recordID),
			TaskQueue:         TaskQueue,
			ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON,
		}
		future := workflow.ExecuteChildWorkflow(workflow.WithChildOptions(ctx, cwo), RecordProcessorWorkflow, recordID)
		// Wait for the child to start so the command commits before any ContinueAsNew.
		return future.GetChildWorkflowExecution().Get(ctx, nil)
	}

	// Slide the window: keep it full, starting one child per free slot.
	for nextIndex < len(recordIDs) {
		// Backpressure: if the window is full, block on the completion channel until an
		// in-flight child signals, freeing a slot. Otherwise start without waiting.
		if active >= windowSize {
			completedCh.Receive(ctx, nil)
			totalProcessed++
			active--
		}
		if err := startChild(recordIDs[nextIndex]); err != nil {
			return 0, err
		}
		nextIndex++
		dispatched++
		active++

		// Once this run has filled the window with fresh children, ContinueAsNew so
		// history stays bounded. Carry active so the next run knows how many children
		// will still signal it.
		if dispatched >= windowSize {
			return 0, workflow.NewContinueAsNewError(ctx, SlidingWindowWorkflow, SlidingWindowInput{
				RecordIDs:      recordIDs,
				WindowSize:     windowSize,
				StartIndex:     nextIndex,
				TotalProcessed: totalProcessed,
				Active:         active,
			})
		}
	}

	// Drain all remaining in-flight children.
	for active > 0 {
		completedCh.Receive(ctx, nil)
		totalProcessed++
		active--
	}
	return totalProcessed, nil
}
```

**Java**

```java
// SlidingWindowWorkflow.java
import io.temporal.activity.ActivityOptions;
import io.temporal.api.enums.v1.ParentClosePolicy;
import io.temporal.workflow.*;
import java.time.Duration;
import java.util.List;

public interface SlidingWindowWorkflow {

    /** Parent Workflow: maintains a fixed window of concurrent Child Workflows. */
    @WorkflowInterface
    interface Parent {
        @WorkflowMethod
        int run(Shared.SlidingWindowInput input);

        @SignalMethod
        void recordCompleted(String recordId);
    }

    /** Child Workflow: processes one record and signals the parent on completion. */
    @WorkflowInterface
    interface Child {
        @WorkflowMethod
        void run(String recordId);
    }

    final class ParentImpl implements Parent {
        // Live in-flight count: +1 per start, -1 per completion signal, carried across
        // runs. An instance field (not a run() local) because the signal handler is a
        // separate method and completions can signal before run() starts.
        private int active = 0;
        // Total records completed across all runs, carried over via Continue-as-New.
        private int totalProcessed = 0;

        @Override
        public void recordCompleted(String recordId) {
            active--;
            totalProcessed++;
        }

        @Override
        public int run(Shared.SlidingWindowInput input) {
            // Use += so completions that signal before run() starts are preserved.
            this.totalProcessed += input.totalProcessed;
            this.active += input.active;
            int windowSize = input.windowSize > 0 ? input.windowSize : Shared.WINDOW_SIZE;
            List<String> recordIds = input.recordIds;
            String parentId = Workflow.getInfo().getWorkflowId();
            int nextIndex = input.startIndex;
            // Children started in this run; triggers Continue-as-New once it hits windowSize.
            int dispatched = 0;

            // Slide the window: keep it full, starting one child per free slot. The first
            // (windowSize - active) slots are already free, so those children start without
            // waiting; after that, each start waits for an in-flight child to free a slot.
            while (nextIndex < recordIds.size()) {
                // Backpressure: block until the window has a free slot.
                Workflow.await(() -> active < windowSize);

                String recordId = recordIds.get(nextIndex);
                ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder()
                        .setWorkflowId(parentId + "/record-" + recordId)
                        .setTaskQueue(Shared.TASK_QUEUE)
                        .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON)
                        .build();
                Child child = Workflow.newChildWorkflowStub(Child.class, opts);
                Async.procedure(child::run, recordId);
                // Wait until the child has started before counting it (and before any
                // Continue-as-New, which would otherwise race child startup).
                Workflow.getWorkflowExecution(child).get();
                nextIndex++;
                dispatched++;
                active++;

                // Once this run has filled the window with fresh children, Continue-as-New
                // so history stays bounded. Carry active so the next run knows how many
                // children will still signal it.
                if (dispatched >= windowSize) {
                    Workflow.newContinueAsNewStub(Parent.class)
                            .run(new Shared.SlidingWindowInput(
                                    recordIds, windowSize, nextIndex, this.totalProcessed, active));
                    return 0; // unreachable; Continue-as-New throws
                }
            }

            // Wait for all remaining in-flight children to complete.
            Workflow.await(() -> active == 0);
            return this.totalProcessed;
        }
    }

    final class ChildImpl implements Child {
        private final Activities activities = Workflow.newActivityStub(
                Activities.class,
                ActivityOptions.newBuilder()
                        .setStartToCloseTimeout(Duration.ofSeconds(30))
                        .build());

        @Override
        public void run(String recordId) {
            activities.processRecord(recordId);

            // Read the parent's Workflow ID from context. It is stable across the parent's
            // Continue-as-New runs, so signaling by Workflow ID always reaches the current
            // run. Ignore not-found — the parent's final run may have already completed.
            String parentWorkflowId = Workflow.getInfo().getParentWorkflowId().orElseThrow();
            ExternalWorkflowStub parent = Workflow.newUntypedExternalWorkflowStub(parentWorkflowId);
            try {
                parent.signal(Shared.COMPLETION_SIGNAL, recordId);
            } catch (Exception e) {
                String msg = e.getMessage() != null ? e.getMessage() : "";
                if (!msg.toLowerCase().contains("not found")) {
                    throw e;
                }
            }
        }
    }
}
```

## Best practices

- **Preserve the parent Workflow ID across Continue-as-New.** The parent's Workflow ID is stable across `continueAsNew` runs — do not generate a new one. Children read the parent's Workflow ID from their own Workflow metadata (`workflowInfo().parent` in TypeScript, `workflow.info().parent` in Python, `workflow.GetInfo(ctx).ParentWorkflowExecution` in Go, `Workflow.getInfo().getParentWorkflowId()` in Java) rather than receiving it as an argument, then signal by Workflow ID (with no run ID) so they always reach the current run.
- **Use `PARENT_CLOSE_POLICY_ABANDON` on child Workflows.** This lets children that were started by a previous run complete normally even after the parent has continued as new.
- **Size the window conservatively at first.** Each in-flight child counts toward the 2,000 unfinished-actions limit for the parent, and Temporal's own guidance recommends staying well under that: a single parent should not spawn more than 1,000 Child Workflow Executions in total. A window of 50–200 is a reasonable starting point depending on child duration and downstream capacity.
- **Pass only IDs (not full records) to child Workflows.** Workflow inputs are stored in event history. Keep them small.
- **Carry minimal state into `continueAsNew`.** Pass `windowSize`, `startIndex`, the live in-flight count (`active`), a running `totalProcessed`, and the record ID list (or a reference to it). Do not accumulate results in the parent — collect them out-of-band if needed.

## Common pitfalls

- **Losing signals across Continue-as-New.** If a child signals before the parent's new run has registered the signal handler, the signal can be buffered and delivered correctly — Temporal buffers signals for existing Workflow IDs. However, ensure the signal handler is registered before any await, not conditionally.
- **Race between Continue-as-New and remaining signal draining.** After `continueAsNew`, the new run must handle signals from children started by the previous run. Pass `startIndex` (the next *unstarted* record) and `active` (the live in-flight count at the moment of CAN) to the new run so it knows how many carried-over children to expect signals from, without re-starting them. The new run folds `active` in with `+=`, so a completion that arrives before `run()` executes is still counted correctly.
- **Thundering herd on startup.** Starting hundreds of children simultaneously causes a burst of Activity polls. Ramp up the window gradually or use the [Batch Iterator](/design-patterns/batch-iterator) if rate limiting is more important than throughput.

## Related

### Patterns

- [Continue-as-New pattern](/design-patterns/continue-as-new) — history management fundamentals
- [Batch Iterator](/design-patterns/batch-iterator) — sequential alternative when ordered, one-at-a-time processing is acceptable
- [MapReduce Tree](/design-patterns/mapreduce-tree) — fully parallel alternative when rate limiting is not needed
- [Temporal limits reference](/evaluate/cloud/limits)
- [Sliding window sample (Java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/slidingwindow)

---

# Task Orchestration Patterns

Source: https://docs.temporal.io/design-patterns/task-orchestration-patterns

> Compares Child Workflows, Parallel Execution, and Pick First (Race) for decomposing, running concurrently, and racing work inside a single Workflow.

These patterns compose and coordinate multiple units of work within a Workflow — decomposing large processes into reusable pieces, running work concurrently, and racing alternatives against each other.

## Patterns in this section

- [Child Workflows](/design-patterns/child-workflows): Decomposes a complex Workflow into smaller, reusable units. Each child has its own Workflow ID, history, and lifecycle.
- [Parallel Execution](/design-patterns/parallel-execution): Runs multiple Activities concurrently for higher throughput, with error handling and a bound on how many run at once.
- [Pick First (Race)](/design-patterns/pick-first): Races multiple Activities in parallel, takes the first result to arrive, and cancels the rest.

## Choosing a pattern

**A process is large or reused across Workflows**: break it into [Child Workflows](/design-patterns/child-workflows) with independent histories.

**Independent work can run at the same time**: use [Parallel Execution](/design-patterns/parallel-execution) with a concurrency bound.

**Several approaches compete and you want the first to finish**: use [Pick First (Race)](/design-patterns/pick-first) and cancel the losers.

## Related sections

- [Batch Processing Patterns](/design-patterns/batch-processing-patterns) — fan-out orchestration scaled to large record sets
- [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — reduce the latency of the work each task performs

---

# Updatable / Debounced Timer Pattern

Source: https://docs.temporal.io/design-patterns/updatable-timer

> Dynamically adjustable timers that respond to Signals or Updates. Extend, shorten, or cancel timers based on external events.

## Overview

The Updatable / Debounced Timer pattern implements a sleep operation that can be interrupted and dynamically adjusted via Signals.
It enables Workflows to wait for deadlines that can be extended or shortened based on external events, making it suitable for approval processes, SLA management, and time-sensitive business operations.

## Problem

A Workflow waiting on a deadline — an approval timeout, an SLA expiration, a grace period — often needs that deadline to move: extended, shortened, or reset in reaction to something that happens while it's waiting, without restarting the wait from scratch.

Without an updatable timer, a fixed timeout can't be adjusted once it's set. The alternatives are all worse: cancel and restart the Workflow to change the deadline, poll frequently to check whether it changed, or build a state machine to manage the timing updates yourself.

## Solution

The Updatable / Debounced Timer uses a blocking wait with both a time limit and an update condition.
When a Signal updates the wake-up time, the condition becomes true, the Workflow recalculates the sleep duration, and blocks again with the new deadline.

Each SDK provides a different mechanism for this:
- **Java**: `Workflow.await(Duration, condition)` returns `false` when the duration expires, or `true` when the condition is met.
- **TypeScript**: `wf.condition(fn, timeout)` returns `false` when the timeout expires, or `true` when the function returns `true`.
- **Python**: `workflow.wait_condition(fn, timeout=duration)` returns normally when the condition is met, or raises `asyncio.TimeoutError` on timeout.
- **Go**: `workflow.NewTimer()` combined with `workflow.NewSelector()` to race a timer against a Signal channel.

```mermaid
sequenceDiagram
    participant Client
    participant Workflow
    participant Timer

    Client->>Workflow: Start with deadline
    activate Workflow
    Workflow->>Timer: sleepUntil(deadline)
    activate Timer
    Note over Timer: Waiting...

    Client->>Workflow: Signal: extendDeadline(newTime)
    Workflow->>Timer: Update wakeUpTime
    Timer->>Timer: Recalculate duration
    Note over Timer: Waiting with new deadline...

    Timer-->>Workflow: Timer expired
    deactivate Timer
    Workflow-->>Client: Complete
    deactivate Workflow
```

The following describes each step in the diagram:

1. The client starts the Workflow with an initial deadline.
2. The Workflow calls `sleepUntil(deadline)`, which blocks until the deadline.
3. The client sends a Signal to extend the deadline.
4. The timer recalculates the remaining duration based on the new deadline and continues waiting.
5. When the timer expires, the Workflow completes.

The core of the pattern is a reusable timer helper that loops on a blocking wait, recalculating the sleep duration each time the wake-up time is updated:

**Python**

```python
# updatable_timer.py
import asyncio
from datetime import timedelta
from temporalio import workflow

class UpdatableTimer:
    def __init__(self, wake_up_time: float) -> None:
        self._wake_up_time = wake_up_time
        self._wake_up_time_updated = False

    async def sleep_until(self, wake_up_time: float) -> None:
        self._wake_up_time = wake_up_time
        while True:
            self._wake_up_time_updated = False
            sleep_secs = self._wake_up_time - workflow.time()
            try:
                await workflow.wait_condition(
                    lambda: self._wake_up_time_updated,
                    timeout=timedelta(seconds=max(sleep_secs, 0)),
                )
                # Condition met: wake-up time was updated, loop to recalculate
            except asyncio.TimeoutError:
                break  # Timer expired

    def update_wake_up_time(self, wake_up_time: float) -> None:
        self._wake_up_time = wake_up_time
        self._wake_up_time_updated = True  # Unblocks wait_condition

    @property
    def wake_up_time(self) -> float:
        return self._wake_up_time
```

**Go**

```go
// updatable_timer.go
func sleepUntil(ctx workflow.Context, wakeUpTime time.Time, wakeUpChannel workflow.ReceiveChannel) error {
    for {
        timerCtx, cancelTimer := workflow.WithCancel(ctx)
        duration := wakeUpTime.Sub(workflow.Now(ctx))
        if duration <= 0 {
            cancelTimer()
            break
        }
        timer := workflow.NewTimer(timerCtx, duration)

        selector := workflow.NewSelector(ctx)
        timerFired := false

        selector.AddFuture(timer, func(f workflow.Future) {
            timerFired = true
        })

        selector.AddReceive(wakeUpChannel, func(c workflow.ReceiveChannel, more bool) {
            c.Receive(ctx, &wakeUpTime)
            // Cancel the current timer so it can be recreated with the new deadline
            cancelTimer()
        })

        selector.Select(ctx)

        if timerFired {
            break // Timer expired
        }
        // Signal received with new wakeUpTime, loop to recalculate
    }
    return nil
}
```

**Java**

```java
// UpdatableTimer.java
public class UpdatableTimer {
  private long wakeUpTime;
  private boolean wakeUpTimeUpdated;

  public void sleepUntil(long wakeUpTime) {
    this.wakeUpTime = wakeUpTime;
    while (true) {
      wakeUpTimeUpdated = false;
      Duration sleepInterval = Duration.ofMillis(this.wakeUpTime - Workflow.currentTimeMillis());
      if (!Workflow.await(sleepInterval, () -> wakeUpTimeUpdated)) {
        break; // Timer expired
      }
      // Timer was updated, loop to recalculate
    }
  }

  public void updateWakeUpTime(long wakeUpTime) {
    this.wakeUpTime = wakeUpTime;
    this.wakeUpTimeUpdated = true; // Unblocks await
  }
}
```

**TypeScript**

```typescript
// updatable-timer.ts
import * as wf from '@temporalio/workflow';

export class UpdatableTimer implements PromiseLike<void> {
  deadlineUpdated = false;
  #deadline: number;

  constructor(deadline: number) {
    this.#deadline = deadline;
  }

  private async run(): Promise<void> {
    while (true) {
      this.deadlineUpdated = false;
      if (
        !(await wf.condition(
          () => this.deadlineUpdated,
          this.#deadline - Date.now(),
        ))
      ) {
        break; // Timer expired
      }
      // Timer was updated, loop to recalculate
    }
  }

  then<TResult1 = void, TResult2 = never>(
    onfulfilled?: (value: void) => TResult1 | PromiseLike<TResult1>,
    onrejected?: (reason: any) => TResult2 | PromiseLike<TResult2>,
  ): PromiseLike<TResult1 | TResult2> {
    return this.run().then(onfulfilled, onrejected);
  }

  set deadline(value: number) {
    this.#deadline = value;
    this.deadlineUpdated = true;
  }

  get deadline(): number {
    return this.#deadline;
  }
}
```

In Java and TypeScript, the `sleepUntil` method calculates the sleep interval and calls a blocking wait with both a duration and a condition.
If the duration expires first, the wait returns `false` (Java/TypeScript) or raises `asyncio.TimeoutError` (Python), and the timer completes.
If the update flag is set via a Signal, the condition becomes true, the wait unblocks, and the loop recalculates the interval with the new deadline.
In Go, a `Selector` races a `Timer` against a Signal channel; when the Signal arrives, the current timer is cancelled and a new one is created with the updated deadline.

## Implementation

### Basic approval Workflow

The following implementation combines the updatable timer with an approval flag.
The Workflow waits for either an approval Signal or the deadline to expire:

**Python**

```python
# workflows.py
import asyncio
from datetime import timedelta
from temporalio import workflow

@workflow.defn
class ApprovalWorkflow:
    def __init__(self) -> None:
        self._approved = False
        self._status = "PENDING"

    @workflow.run
    async def run(self, approval_deadline: float) -> None:
        timeout_secs = approval_deadline - workflow.time()
        try:
            await workflow.wait_condition(
                lambda: self._approved,
                timeout=timedelta(seconds=max(timeout_secs, 0)),
            )
            self._status = "APPROVED"
        except asyncio.TimeoutError:
            self._status = "REJECTED"

    @workflow.signal
    def approve(self) -> None:
        self._approved = True

    @workflow.query
    def get_status(self) -> str:
        return self._status
```

**Go**

```go
// workflow.go
func ApprovalWorkflow(ctx workflow.Context, approvalDeadline time.Time) (string, error) {
    logger := workflow.GetLogger(ctx)
    status := "PENDING"
    approved := false

    // Listen for the approve signal in a goroutine
    workflow.Go(ctx, func(ctx workflow.Context) {
        ch := workflow.GetSignalChannel(ctx, "approve")
        ch.Receive(ctx, nil)
        approved = true
    })

    // Wait for approval or timeout
    duration := approvalDeadline.Sub(workflow.Now(ctx))
    ok, _ := workflow.AwaitWithTimeout(ctx, duration, func() bool {
        return approved
    })

    if ok {
        status = "APPROVED"
    } else {
        status = "REJECTED"
    }

    logger.Info("Approval workflow completed", "status", status)
    return status, nil
}
```

**Java**

```java
// ApprovalWorkflowImpl.java
@WorkflowInterface
public interface ApprovalWorkflow {
  @WorkflowMethod
  void execute(long approvalDeadline);

  @SignalMethod
  void extendDeadline(long newDeadline);

  @SignalMethod
  void approve();

  @QueryMethod
  String getStatus();
}

public class ApprovalWorkflowImpl implements ApprovalWorkflow {
  private UpdatableTimer timer = new UpdatableTimer();
  private boolean approved = false;
  private String status = "PENDING";

  @Override
  public void execute(long approvalDeadline) {
    Workflow.await(
        Duration.ofMillis(approvalDeadline - Workflow.currentTimeMillis()),
        () -> approved);

    if (approved) {
      status = "APPROVED";
    } else {
      status = "REJECTED";
    }
  }

  @Override
  public void extendDeadline(long newDeadline) {
    timer.updateWakeUpTime(newDeadline);
  }

  @Override
  public void approve() {
    approved = true;
  }

  @Override
  public String getStatus() {
    return status;
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';

export const extendDeadlineSignal = wf.defineSignal<[number]>('extendDeadline');
export const approveSignal = wf.defineSignal('approve');
export const getStatusQuery = wf.defineQuery<string>('getStatus');

export async function approvalWorkflow(approvalDeadline: number): Promise<void> {
  let approved = false;
  let status = 'PENDING';

  wf.setHandler(approveSignal, () => {
    approved = true;
  });

  wf.setHandler(getStatusQuery, () => status);

  // Wait for approval or deadline expiration
  const approvedBeforeDeadline = await wf.condition(
    () => approved,
    approvalDeadline - Date.now(),
  );

  status = approvedBeforeDeadline ? 'APPROVED' : 'REJECTED';
}
```

The Workflow waits with both a deadline duration and a condition that checks the `approved` flag.
If the `approve` Signal arrives before the deadline, the condition becomes true and the Workflow sets the status to APPROVED.
If the deadline expires first, the Workflow sets the status to REJECTED.

### Multiple deadline extensions

The following implementation uses the `UpdatableTimer` directly to support multiple deadline extensions.
The Workflow blocks on the timer helper and checks the approval flag after the timer completes:

**Python**

```python
# workflows.py
from temporalio import workflow
from .updatable_timer import UpdatableTimer

@workflow.defn
class MultiExtensionApprovalWorkflow:
    def __init__(self) -> None:
        self._timer = UpdatableTimer(0)
        self._approved = False
        self._rejected = False

    @workflow.run
    async def run(self, initial_deadline: float) -> None:
        await self._timer.sleep_until(initial_deadline)

        if not self._approved:
            self._rejected = True

    @workflow.signal
    def extend_deadline(self, new_deadline: float) -> None:
        if not self._approved and not self._rejected:
            self._timer.update_wake_up_time(new_deadline)

    @workflow.signal
    def approve(self) -> None:
        self._approved = True
```

**Go**

```go
// workflow.go
func MultiExtensionApprovalWorkflow(ctx workflow.Context, initialDeadline time.Time) (string, error) {
    approved := false
    rejected := false
    wakeUpTime := initialDeadline
    wakeUpChannel := workflow.NewChannel(ctx)

    // Listen for approval signal
    workflow.Go(ctx, func(ctx workflow.Context) {
        ch := workflow.GetSignalChannel(ctx, "approve")
        ch.Receive(ctx, nil)
        approved = true
    })

    // Listen for deadline extension signals
    workflow.Go(ctx, func(ctx workflow.Context) {
        ch := workflow.GetSignalChannel(ctx, "extendDeadline")
        for {
            var newDeadline time.Time
            ch.Receive(ctx, &newDeadline)
            if !approved && !rejected {
                wakeUpChannel.Send(ctx, newDeadline)
            }
        }
    })

    // Block on the updatable timer
    _ = sleepUntil(ctx, wakeUpTime, wakeUpChannel)

    if !approved {
        rejected = true
    }

    if rejected {
        return "REJECTED", nil
    }
    return "APPROVED", nil
}
```

**Java**

```java
// MultiExtensionApprovalWorkflowImpl.java
public class MultiExtensionApprovalWorkflowImpl implements ApprovalWorkflow {
  private UpdatableTimer timer = new UpdatableTimer();
  private boolean approved = false;
  private boolean rejected = false;

  @Override
  public void execute(long initialDeadline) {
    timer.sleepUntil(initialDeadline);

    if (!approved) {
      rejected = true;
    }
  }

  @Override
  public void extendDeadline(long newDeadline) {
    if (!approved && !rejected) {
      timer.updateWakeUpTime(newDeadline);
    }
  }

  @Override
  public void approve() {
    approved = true;
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import * as wf from '@temporalio/workflow';
import { UpdatableTimer } from './updatable-timer';

export const extendDeadlineSignal = wf.defineSignal<[number]>('extendDeadline');
export const approveSignal = wf.defineSignal('approve');

export async function multiExtensionApprovalWorkflow(
  initialDeadline: number,
): Promise<void> {
  let approved = false;
  let rejected = false;
  const timer = new UpdatableTimer(initialDeadline);

  wf.setHandler(extendDeadlineSignal, (newDeadline: number) => {
    if (!approved && !rejected) {
      timer.deadline = newDeadline;
    }
  });

  wf.setHandler(approveSignal, () => {
    approved = true;
  });

  await timer; // Blocks until the timer expires

  if (!approved) {
    rejected = true;
  }
}
```

The `extendDeadline` Signal handler checks that the Workflow has not already been approved or rejected before updating the timer.
Each update unblocks the timer loop, which recalculates the remaining duration and blocks again.

## When to use

The Updatable Timer pattern is a good fit for approval Workflows with deadline extensions, SLA management with grace periods, time-based escalations that can be postponed, auction bidding with extended closing times, and payment grace periods that can be adjusted.

It is not a good fit for fixed timeouts that never change (use a fixed sleep), immediate cancellation (use cancellation scopes), or complex scheduling (use Temporal Schedules).

## Benefits and trade-offs

The pattern allows you to adjust deadlines without restarting Workflows.
Changes take effect instantly.
The timer helper is reusable across multiple Workflows.
All timing is based on Workflow time, ensuring replay consistency.
You can Query the current deadline at any time.

The trade-offs to consider are that the pattern requires an external process to send update Signals.
Each timer instance manages one deadline.
Previous deadlines are not tracked (add tracking if needed).
You must calculate absolute timestamps rather than relative durations.

## Comparison with alternatives

| Approach | Dynamic updates | Complexity | Use case |
| :--- | :--- | :--- | :--- |
| Updatable / Debounced Timer | Yes | Medium | Adjustable deadlines |
| Fixed sleep | No | Low | Fixed delays |
| Cancellation Scope | Yes (cancel only) | Medium | Abort operations |
| Polling Loop | Yes | High | Frequent checks |

## Best practices

- **Use absolute timestamps.** Store wake-up time as an absolute value (epoch millis in Java/TypeScript, epoch seconds in Python, `time.Time` in Go), not relative durations.
- **Validate updates.** Ensure new deadlines are in the future.
- **Add Queries.** Expose the current deadline via Query methods.
- **Handle edge cases.** Check if the timer already expired before updating.
- **Consider max extensions.** Limit how many times or how far deadlines can be extended.
- **Log changes.** Log each deadline update for observability.
- **Reuse the timer helper.** Extract to a helper class or function for use across Workflows.
- **Combine with conditions.** Use a blocking wait with both time and business conditions.

## Common pitfalls

- **Using time-based conditions without a duration.** A wait without a timeout does not create a timer. The condition is only re-evaluated on state changes (Signals, Activity completions). Always provide a timeout for time-based waits.
- **Expecting the wait to re-evaluate its duration.** The timer duration is set once when the wait is called. Changing the duration variable afterward has no effect. This is why the timer helper loops and recalculates.
- **Not validating new deadlines.** Accepting a deadline in the past causes the timer to expire immediately. Always check that the new deadline is in the future before updating.
- **Accumulating uncancelled timers in Java.** In the Java SDK, `Workflow.await(Duration, condition)` does not automatically cancel its internal timer when the condition is met. Repeated calls in a loop accumulate timers. Wrap in a `CancellationScope` if this is a concern.
- **Not cancelling timers in Go.** In the Go SDK, always cancel the previous timer (via `workflow.WithCancel`) before creating a new one. Uncancelled timers wake up the Workflow unnecessarily, creating extra Worker load.

## Related

### Patterns

- **[Signal with Start](/design-patterns/signal-with-start)**: Receiving external events to modify behavior.
- **[Approval Pattern](/design-patterns/approval)**: Approval Workflows with adjustable deadlines.

### References

- [Timers (TypeScript)](/develop/typescript/workflows/timers): Covers the same `UpdatableTimer` class, built on the [`temporal-time-utils`](https://www.npmjs.com/package/temporal-time-utils) package.

### Sample code

- [Java](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/updatabletimer) -- Complete implementation with starter and updater.
- [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/timer-examples) -- Updatable timer with `condition` and `UpdatableTimer` class.
- [Python](https://github.com/temporalio/samples-python/tree/main/updatable_timer) -- Updatable timer with `wait_condition` and helper class.
- [Go](https://github.com/temporalio/samples-go/tree/main/updatabletimer) -- Timer cancellation with Selector, Signal channel, and `WithCancel`.

---

# Worker Configuration Patterns

Source: https://docs.temporal.io/design-patterns/worker-configuration-patterns

> Pattern selection guide for configuring how Workers are set up, how work is routed, and how Activities access external dependencies.

These patterns cover how you set up Workers, route work to them, and give Activities the external dependencies they need — while keeping Workflow Definitions deterministic and testable.

## Patterns in this section

- [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue): Routes Activities to a specific Worker using a unique Task Queue, for Worker affinity and host-specific processing.
- [Activity Dependency Injection](/design-patterns/activity-dependency-injection): Injects external dependencies — clients, connections, configuration — into Activities at Worker startup, keeping Workflow Definitions deterministic and Activities testable.

## Choosing a pattern

**A sequence of Activities must run on the same Worker host**: use [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue) to pin them to one Worker.

**Activities depend on external resources**: use [Activity Dependency Injection](/design-patterns/activity-dependency-injection) to supply them at startup rather than constructing them inside each Activity.

## Related sections

- [QoS & Throughput Patterns](/design-patterns/qos-throughput-patterns) — Task Queue routing for rate control and fairness
- [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — run Activities in-process with Local Activities

---

# Worker-Specific Task Queues Pattern

Source: https://docs.temporal.io/design-patterns/worker-specific-taskqueue

> Routes Activities to specific Workers using unique Task Queues for Worker affinity and host-specific processing.

## Overview

The Worker-Specific Task Queues pattern enables routing Activities to specific Worker hosts when Activities must execute on the same machine.
This is essential for Workflows where subsequent Activities depend on local state, files, or resources created by previous Activities on a particular host.

## Problem

In distributed systems, you often need Workflows that download a file to a Worker's local disk and then process and upload it from the same location, establish a connection or session that subsequent Activities must reuse, create temporary resources on one host that later Activities need to access, or maintain affinity to a specific Worker for performance or data locality.

Without Worker-specific routing, Activities execute on different hosts and cannot access local files or state.
You must set up complex distributed file systems or shared storage, handle race conditions when multiple Workers access the same resources, and accept that you cannot guarantee Activity colocation.

## Solution

You use a two-tier Task Queue architecture: a default shared Task Queue for initial Activities, and dynamically-named host-specific Task Queues for Activities that must run on the same Worker.
The first Activity returns its host-specific Task Queue name, and subsequent Activities use that queue.

```mermaid
sequenceDiagram
    participant Workflow
    participant Worker1
    participant Worker2
    participant Worker3

    Note over Workflow: Default Task Queue: "FileProcessing"
    Workflow->>Worker2: download() - any worker
    activate Worker2
    Worker2-->>Workflow: {hostQueue: "FileProcessing-host2", file: "/tmp/data"}
    deactivate Worker2

    Note over Workflow: Switch to host-specific queue
    Note over Workflow: Task Queue: "FileProcessing-host2"

    Workflow->>Worker2: process(file) - MUST be Worker2
    activate Worker2
    Worker2-->>Workflow: processed file
    deactivate Worker2

    Workflow->>Worker2: upload(file) - MUST be Worker2
    activate Worker2
    Worker2-->>Workflow: done
    deactivate Worker2

    Note over Worker1,Worker3: Workers 1 & 3 never see<br/>host-specific activities
```

The following describes each step in the diagram:

1. The Workflow dispatches the download Activity on the default Task Queue. Any available Worker picks it up.
2. Worker 2 downloads the file and returns both the local file path and its host-specific Task Queue name.
3. The Workflow creates new Activity options targeting Worker 2's host-specific Task Queue.
4. The process and upload Activities execute on Worker 2, where the file is already on disk.
5. Workers 1 and 3 never see the host-specific Activities.

The following snippet shows how the Workflow switches from the default Task Queue to the host-specific queue:

**Python**

```python
# workflows.py
downloaded = await workflow.execute_activity(
    download,
    source,
    start_to_close_timeout=timedelta(seconds=20),
)

processed = await workflow.execute_activity(
    process,
    downloaded.file_name,
    task_queue=downloaded.host_task_queue,
    schedule_to_start_timeout=timedelta(seconds=10),
    start_to_close_timeout=timedelta(seconds=20),
)
await workflow.execute_activity(
    upload,
    args=[processed, destination],
    task_queue=downloaded.host_task_queue,
    schedule_to_start_timeout=timedelta(seconds=10),
    start_to_close_timeout=timedelta(seconds=20),
)
```

**Go**

```go
// workflow.go
var defaultActivities *StoreActivities
downloaded, err := defaultActivities.Download(ctx, source)
if err != nil {
    return err
}

hostOptions := workflow.ActivityOptions{
    TaskQueue:              downloaded.HostTaskQueue,
    ScheduleToStartTimeout: 10 * time.Second,
    StartToCloseTimeout:    20 * time.Second,
}
hostCtx := workflow.WithActivityOptions(ctx, hostOptions)

var hostActivities *StoreActivities
processed, err := hostActivities.Process(hostCtx, downloaded.FileName)
if err != nil {
    return err
}
err = hostActivities.Upload(hostCtx, processed, destination)
```

**Java**

```java
// FileProcessingWorkflowImpl.java
TaskQueueFileNamePair downloaded = defaultTaskQueueActivities.download(source);

ActivityOptions hostOptions = ActivityOptions.newBuilder()
    .setTaskQueue(downloaded.getHostTaskQueue())
    .setScheduleToStartTimeout(Duration.ofSeconds(10))
    .setStartToCloseTimeout(Duration.ofSeconds(20))
    .build();
StoreActivities hostSpecificActivities =
    Workflow.newActivityStub(StoreActivities.class, hostOptions);

String processed = hostSpecificActivities.process(downloaded.getFileName());
hostSpecificActivities.upload(processed, destination);
```

**TypeScript**

```typescript
// workflows.ts
const { download } = proxyActivities<StoreActivities>({
  startToCloseTimeout: '20s',
});

const downloaded = await download(source);

const hostSpecificActivities = proxyActivities<StoreActivities>({
  taskQueue: downloaded.hostTaskQueue,
  scheduleToStartTimeout: '10s',
  startToCloseTimeout: '20s',
});

const processed = await hostSpecificActivities.process(downloaded.fileName);
await hostSpecificActivities.upload(processed, destination);
```

The `taskQueue` option (or `setTaskQueue()` in Java) routes subsequent Activities to the specific Worker that downloaded the file.
The `scheduleToStartTimeout` (or `setScheduleToStartTimeout()` in Java) is critical — if the specific Worker is unavailable, the Activity Task cannot sit in the host-specific queue indefinitely.
This timeout is non-retryable: when it expires, the Activity fails rather than retrying, because a retry would only place the Task back on the same queue.
The Workflow catches that failure and retries the entire sequence so the work can restart on a different host.

## Implementation

### Activity Definition with host-specific return

The download Activity returns both the file path and the host-specific Task Queue name:

**Python**

```python
# activities.py
from dataclasses import dataclass
from temporalio import activity

@dataclass
class TaskQueueFileNamePair:
    host_task_queue: str
    file_name: str

@activity.defn
async def download(source: str) -> TaskQueueFileNamePair:
    local_file = await download_to_local_disk(source)
    return TaskQueueFileNamePair(
        host_task_queue=host_specific_task_queue,
        file_name=local_file,
    )

@activity.defn
async def process(file_name: str) -> str:
    return await process_local_file(file_name)

@activity.defn
async def upload(file_name: str, destination: str) -> None:
    await upload_from_local_disk(file_name, destination)
```

**Go**

```go
// activities.go
type TaskQueueFileNamePair struct {
    HostTaskQueue string
    FileName      string
}

type StoreActivities struct {
    HostSpecificTaskQueue string
}

func (a *StoreActivities) Download(ctx context.Context, source string) (*TaskQueueFileNamePair, error) {
    localFile, err := downloadToLocalDisk(source)
    if err != nil {
        return nil, err
    }
    return &TaskQueueFileNamePair{
        HostTaskQueue: a.HostSpecificTaskQueue,
        FileName:      localFile,
    }, nil
}

func (a *StoreActivities) Process(ctx context.Context, fileName string) (string, error) {
    return processLocalFile(fileName)
}

func (a *StoreActivities) Upload(ctx context.Context, fileName string, destination string) error {
    return uploadFromLocalDisk(fileName, destination)
}
```

**Java**

```java
// StoreActivities.java
public interface StoreActivities {

  class TaskQueueFileNamePair {
    private final String hostTaskQueue;
    private final String fileName;

    public TaskQueueFileNamePair(String hostTaskQueue, String fileName) {
      this.hostTaskQueue = hostTaskQueue;
      this.fileName = fileName;
    }

    public String getHostTaskQueue() { return hostTaskQueue; }
    public String getFileName() { return fileName; }
  }

  TaskQueueFileNamePair download(URL source);
  String process(String fileName);
  void upload(String fileName, URL destination);
}
```

**TypeScript**

```typescript
// activities.ts
export interface TaskQueueFileNamePair {
  hostTaskQueue: string;
  fileName: string;
}

export async function download(source: string): Promise<TaskQueueFileNamePair> {
  const localFile = await downloadToLocalDisk(source);
  return {
    hostTaskQueue: getHostSpecificTaskQueue(),
    fileName: localFile,
  };
}

export async function process(fileName: string): Promise<string> {
  return await processLocalFile(fileName);
}

export async function upload(fileName: string, destination: string): Promise<void> {
  await uploadFromLocalDisk(fileName, destination);
}
```

The download Activity bundles the local file path with the Task Queue name so the Workflow knows where to route subsequent Activities.

### Activity implementation

The Activity implementation receives the host-specific Task Queue name at construction time and includes it in the download result:

**Python**

```python
# activities.py
# In Python, the host-specific Task Queue name is injected at Worker
# startup and captured by the activity closure or class instance.

host_specific_task_queue: str = ""

@activity.defn
async def download(source: str) -> TaskQueueFileNamePair:
    local_file = await download_to_local_disk(source)
    return TaskQueueFileNamePair(
        host_task_queue=host_specific_task_queue,
        file_name=local_file,
    )

@activity.defn
async def process(file_name: str) -> str:
    processed = await process_local_file(file_name)
    return processed

@activity.defn
async def upload(file_name: str, destination: str) -> None:
    await upload_from_local_disk(file_name, destination)
```

**Go**

```go
// activities.go
// In Go, the host-specific Task Queue name is set on the struct
// at Worker startup and returned by the Download method.

func (a *StoreActivities) Download(ctx context.Context, source string) (*TaskQueueFileNamePair, error) {
    localFile, err := downloadToLocalDisk(source)
    if err != nil {
        return nil, err
    }
    return &TaskQueueFileNamePair{
        HostTaskQueue: a.HostSpecificTaskQueue,
        FileName:      localFile,
    }, nil
}

func (a *StoreActivities) Process(ctx context.Context, fileName string) (string, error) {
    return processLocalFile(fileName)
}

func (a *StoreActivities) Upload(ctx context.Context, fileName string, destination string) error {
    return uploadFromLocalDisk(fileName, destination)
}
```

**Java**

```java
// StoreActivitiesImpl.java
public class StoreActivitiesImpl implements StoreActivities {
  private final String hostSpecificTaskQueue;

  public StoreActivitiesImpl(String hostSpecificTaskQueue) {
    this.hostSpecificTaskQueue = hostSpecificTaskQueue;
  }

  @Override
  public TaskQueueFileNamePair download(URL source) {
    File localFile = downloadToLocalDisk(source);
    return new TaskQueueFileNamePair(
        hostSpecificTaskQueue,
        localFile.getAbsolutePath());
  }

  @Override
  public String process(String fileName) {
    File processed = processLocalFile(new File(fileName));
    return processed.getAbsolutePath();
  }

  @Override
  public void upload(String fileName, URL destination) {
    uploadFromLocalDisk(new File(fileName), destination);
  }
}
```

**TypeScript**

```typescript
// activities.ts
// In TypeScript, the host-specific Task Queue name is captured via
// closure when defining the activity functions. A common approach is
// to initialize it at Worker startup and reference it from activities.

let hostSpecificTaskQueue: string;

export function initActivities(taskQueue: string) {
  hostSpecificTaskQueue = taskQueue;
}

function getHostSpecificTaskQueue(): string {
  return hostSpecificTaskQueue;
}

export async function download(source: string): Promise<TaskQueueFileNamePair> {
  const localFile = await downloadToLocalDisk(source);
  return {
    hostTaskQueue: getHostSpecificTaskQueue(),
    fileName: localFile,
  };
}

export async function process(fileName: string): Promise<string> {
  return await processLocalFile(fileName);
}

export async function upload(fileName: string, destination: string): Promise<void> {
  await uploadFromLocalDisk(fileName, destination);
}
```

The `download` method returns the host-specific Task Queue name alongside the file path.
The `process` and `upload` methods operate on local files, which are guaranteed to exist because they run on the same host.

### Workflow implementation

The Workflow uses the default Task Queue for the initial download and switches to the host-specific queue for subsequent Activities:

**Python**

```python
# workflows.py
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import download, process, upload

@workflow.defn
class FileProcessingWorkflow:
    @workflow.run
    async def run(self, source: str, destination: str) -> None:
        downloaded = await workflow.execute_activity(
            download,
            source,
            start_to_close_timeout=timedelta(seconds=20),
        )

        processed = await workflow.execute_activity(
            process,
            downloaded.file_name,
            task_queue=downloaded.host_task_queue,
            schedule_to_start_timeout=timedelta(seconds=10),
            start_to_close_timeout=timedelta(seconds=20),
        )

        await workflow.execute_activity(
            upload,
            args=[processed, destination],
            task_queue=downloaded.host_task_queue,
            schedule_to_start_timeout=timedelta(seconds=10),
            start_to_close_timeout=timedelta(seconds=20),
        )
```

**Go**

```go
// workflow.go
func FileProcessingWorkflow(ctx workflow.Context, source string, destination string) error {
    defaultOptions := workflow.ActivityOptions{
        StartToCloseTimeout: 20 * time.Second,
    }
    defaultCtx := workflow.WithActivityOptions(ctx, defaultOptions)

    var activities *StoreActivities
    var downloaded TaskQueueFileNamePair
    err := workflow.ExecuteActivity(defaultCtx, activities.Download, source).Get(ctx, &downloaded)
    if err != nil {
        return err
    }

    hostOptions := workflow.ActivityOptions{
        TaskQueue:              downloaded.HostTaskQueue,
        ScheduleToStartTimeout: 10 * time.Second,
        StartToCloseTimeout:    20 * time.Second,
    }
    hostCtx := workflow.WithActivityOptions(ctx, hostOptions)

    var processed string
    err = workflow.ExecuteActivity(hostCtx, activities.Process, downloaded.FileName).Get(ctx, &processed)
    if err != nil {
        return err
    }

    return workflow.ExecuteActivity(hostCtx, activities.Upload, processed, destination).Get(ctx, nil)
}
```

**Java**

```java
// FileProcessingWorkflowImpl.java
public class FileProcessingWorkflowImpl implements FileProcessingWorkflow {
  private final StoreActivities defaultTaskQueueActivities;

  public FileProcessingWorkflowImpl() {
    ActivityOptions defaultOptions = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(20))
        .build();
    this.defaultTaskQueueActivities =
        Workflow.newActivityStub(StoreActivities.class, defaultOptions);
  }

  @Override
  public void processFile(URL source, URL destination) {
    TaskQueueFileNamePair downloaded =
        defaultTaskQueueActivities.download(source);

    ActivityOptions hostOptions = ActivityOptions.newBuilder()
        .setTaskQueue(downloaded.getHostTaskQueue())
        .setScheduleToStartTimeout(Duration.ofSeconds(10))
        .setStartToCloseTimeout(Duration.ofSeconds(20))
        .build();
    StoreActivities hostSpecificActivities =
        Workflow.newActivityStub(StoreActivities.class, hostOptions);

    String processed = hostSpecificActivities.process(downloaded.getFileName());
    hostSpecificActivities.upload(processed, destination);
  }
}
```

**TypeScript**

```typescript
// workflows.ts
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { download } = proxyActivities<typeof activities>({
  startToCloseTimeout: '20s',
});

export async function fileProcessingWorkflow(
  source: string,
  destination: string,
): Promise<void> {
  const downloaded = await download(source);

  const hostSpecificActivities = proxyActivities<typeof activities>({
    taskQueue: downloaded.hostTaskQueue,
    scheduleToStartTimeout: '10s',
    startToCloseTimeout: '20s',
  });

  const processed = await hostSpecificActivities.process(downloaded.fileName);
  await hostSpecificActivities.upload(processed, destination);
}
```

The Workflow creates two sets of Activity options: one for the default Task Queue and one for the host-specific queue returned by the download Activity.

### Worker setup

Each Worker registers with both the default Task Queue and its own host-specific Task Queue:

**Python**

```python
# worker.py
import asyncio
import uuid
import socket
from temporalio.client import Client
from temporalio.worker import Worker
from workflows import FileProcessingWorkflow
from activities import download, process, upload, host_specific_task_queue
import activities as act_module

async def main():
    client = await Client.connect("localhost:7233")

    default_task_queue = "FileProcessing"
    host_task_queue = f"FileProcessing-{socket.gethostname()}-{uuid.uuid4()}"

    act_module.host_specific_task_queue = host_task_queue

    default_worker = Worker(
        client,
        task_queue=default_task_queue,
        workflows=[FileProcessingWorkflow],
        activities=[download, process, upload],
    )
    host_worker = Worker(
        client,
        task_queue=host_task_queue,
        activities=[download, process, upload],
    )

    await asyncio.gather(default_worker.run(), host_worker.run())

if __name__ == "__main__":
    asyncio.run(main())
```

**Go**

```go
// worker/main.go
func main() {
    c, err := client.Dial(client.Options{})
    if err != nil {
        log.Fatalln("Unable to create client", err)
    }
    defer c.Close()

    defaultTaskQueue := "FileProcessing"
    hostTaskQueue := fmt.Sprintf("FileProcessing-%s-%s", getHostName(), uuid.New().String())

    activities := &StoreActivities{HostSpecificTaskQueue: hostTaskQueue}

    defaultWorker := worker.New(c, defaultTaskQueue, worker.Options{})
    defaultWorker.RegisterWorkflow(FileProcessingWorkflow)
    defaultWorker.RegisterActivity(activities)

    hostWorker := worker.New(c, hostTaskQueue, worker.Options{})
    hostWorker.RegisterActivity(activities)

    err = defaultWorker.Start()
    if err != nil {
        log.Fatalln("Unable to start default worker", err)
    }
    err = hostWorker.Start()
    if err != nil {
        log.Fatalln("Unable to start host worker", err)
    }

    // Block until interrupted
    select {}
}
```

**Java**

```java
// FileProcessingWorker.java
public class FileProcessingWorker {
  public static void main(String[] args) {
    WorkflowClient client = WorkflowClient.newInstance(service);

    String defaultTaskQueue = "FileProcessing";
    String hostTaskQueue = "FileProcessing-" + getHostName();

    WorkerFactory factory = WorkerFactory.newInstance(client);

    Worker defaultWorker = factory.newWorker(defaultTaskQueue);
    defaultWorker.registerWorkflowImplementationTypes(
        FileProcessingWorkflowImpl.class);
    defaultWorker.registerActivitiesImplementations(
        new StoreActivitiesImpl(hostTaskQueue));

    Worker hostWorker = factory.newWorker(hostTaskQueue);
    hostWorker.registerActivitiesImplementations(
        new StoreActivitiesImpl(hostTaskQueue));

    factory.start();
  }
}
```

**TypeScript**

```typescript
// worker.ts
import { Worker, NativeConnection } from '@temporalio/worker';
import * as activities from './activities';
import { v4 as uuid } from 'uuid';
import os from 'os';

async function run() {
  const defaultTaskQueue = 'FileProcessing';
  const hostTaskQueue = `FileProcessing-${os.hostname()}-${uuid()}`;

  activities.initActivities(hostTaskQueue);

  const defaultWorker = await Worker.create({
    workflowsPath: require.resolve('./workflows'),
    activities,
    taskQueue: defaultTaskQueue,
  });

  const hostWorker = await Worker.create({
    activities,
    taskQueue: hostTaskQueue,
  });

  await Promise.all([defaultWorker.run(), hostWorker.run()]);
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

The default Worker handles Workflows and initial Activities.
The host-specific Worker handles only Activities that require Worker affinity.
Both Workers receive the same Activity implementation, but only the host-specific Worker receives Activities routed to its queue.

## When to use

**Good fit:**

- File processing Workflows (download, process, upload on the same host)
- Database connection pooling (maintain a connection across Activities)
- GPU-bound operations (route to Workers with specific hardware)
- Session-based external API calls
- Temporary resource management (cache, temp files, locks)

**Poor fit:**

- Stateless Activities that can run anywhere
- Activities that use shared storage (S3, databases)
- High-availability requirements (a host failure blocks the Workflow)
- Workflows without local state dependencies

## Benefits and trade-offs

**Benefits:**

- Activities access local files and state without network overhead
- No distributed file systems or state management needed
- Data transfer between Workers is eliminated
- The first Activity can run on any Worker; only subsequent ones are pinned
- Task Queue routing is recorded in Workflow history, ensuring deterministic behavior

**Trade-offs:**

- If the specific Worker crashes, Activities cannot proceed until the `ScheduleToStartTimeout` expires
- Host-specific queues may have uneven load distribution
- You must manage multiple Task Queues per Worker
- You must set `ScheduleToStartTimeout` to handle Worker unavailability
- You need to handle cleanup if the Workflow fails mid-process

## Comparison with alternatives

| Approach | Locality | Complexity | Availability |
| :--- | :--- | :--- | :--- |
| Worker-Specific Queues | Guaranteed | Medium | Lower |
| Shared Storage (S3) | None | Low | Higher |
| Session Framework (Go) | Guaranteed | Low | Lower |

## Best practices

> **⚠️ Caution:**
> Worker Versioning
>
> Do not enable [Worker Versioning](/production-deployment/worker-deployments/worker-versioning#definition) for worker-specific Task Queues — it will unnecessarily exhaust the [per-version Task Queue limit](/evaluate/cloud/limits#max-task-queues-in-deployment-version-limits). A per-worker Task Queue's lifetime is tied to the Worker itself. When the Worker is updated to a new version, it gets a new Task Queue and the old one is gone. So versioning this queue does not add any value.
>

- **Set ScheduleToStartTimeout.** Always configure this for host-specific queues to handle Worker failures.
- **Implement cleanup.** Use try-finally or cancellation scopes to clean up local resources.
- **Use unique queue names.** Use hostname, IP, or UUID to ensure unique Task Queue names.
- **Monitor queue depth.** Alert on growing host-specific queue backlogs.
- **Drain gracefully.** Drain host-specific queues before stopping Workers.
- **Retry the entire sequence.** Wrap the sequence in retry logic to restart on a different host if needed.
- **Limit concurrent Workflows.** Limit concurrent Workflows per Worker to prevent resource exhaustion.
- **Add health checks.** Verify Worker health before accepting work on host-specific queues.

## Common pitfalls

- **Missing ScheduleToStartTimeout on host-specific queues.** Without this timeout, if the target Worker is down, the Activity waits indefinitely. Always set `ScheduleToStartTimeout` so the Workflow can detect unavailability and retry on a different host.
- **Not registering the Worker on both queues.** Each Worker must listen on both the default shared Task Queue (for Workflows and initial Activities) and its own host-specific queue. Forgetting the host-specific queue means routed Activities are never picked up.
- **Assuming the host-specific Worker is always available.** The pinned Worker can crash or be restarted. Design the Workflow to retry the entire sequence on a different host when the `ScheduleToStartTimeout` expires.
- **Leaking temporary files on failure.** If the Workflow fails after downloading but before uploading, temporary files remain on disk. Use cleanup logic (defer, try-finally, or cancellation scopes) to remove local resources.
- **Using host-specific queues when shared storage suffices.** If all Workers can access the same storage (S3, NFS), Worker-specific routing adds unnecessary complexity and reduces availability.

## Related

### Patterns

- **[Long-Running Activity](/design-patterns/long-running-activity)**: For tracking progress and handling cancellation when the colocated Activities run for minutes or hours.

### Guides

- [Ensure Activity execution on the same Worker](/guides/worker-execution-affinity): A Python walkthrough of the shared-queue-discovers-unique-queue technique, with heartbeat and Schedule-to-Start timeouts for detecting a crashed Worker.
- [Route specialized workloads](/guides/route-specialized-workloads): Routes Activities to dedicated GPU, high-memory, and CPU Worker pools by resource requirement — a related Task Queue routing technique, but by hardware capability rather than by Worker affinity.

### Sample code

- [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/fileprocessing) — Complete file processing implementation.
- [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/worker-specific-task-queues) — Worker-specific Task Queues with file processing.
- [Python Sample](https://github.com/temporalio/samples-python/tree/main/worker_specific_task_queues) — Worker-specific Task Queues with file processing.
- [Go Sample](https://github.com/temporalio/samples-go/tree/main/worker-specific-task-queues) — Worker-specific Task Queues with file processing.

---

# Workflow Messaging Patterns

Source: https://docs.temporal.io/design-patterns/workflow-messaging-patterns

> Pattern selection guide for sending data into running Workflows and receiving responses or triggering behavior changes.

These patterns cover how external callers communicate with running Workflows — starting them on demand, sending data in, reading results back, and collecting streams of events. They build on Temporal's Signals and Updates.

## Patterns in this section

- [Signal with Start](/design-patterns/signal-with-start): Starts a Workflow and delivers a Signal in a single atomic operation. If the Workflow already runs, it receives the Signal directly.
- [Request-Response via Updates](/design-patterns/request-response-via-updates): Sends a request into a running Workflow through an Update, which validates it, modifies state, and returns a typed result on the same call.
- [Event Accumulator](/design-patterns/event-accumulator): Collects a stream of incoming Signals into a buffer and processes them together as a batch, rather than one at a time.

## Choosing a pattern

**You want to send a message without checking whether the Workflow is running**: use [Signal with Start](/design-patterns/signal-with-start).

**You need a result back from the Workflow, with validation**: use [Request-Response via Updates](/design-patterns/request-response-via-updates).

**You receive many events and want to process them in batches**: use the [Event Accumulator](/design-patterns/event-accumulator) to buffer and flush.

## Related sections

- [Entity & Lifecycle Patterns](/design-patterns/entity-lifecycle-patterns) — long-lived Workflows that consume these messages over time
- [Distributed Transaction Patterns](/design-patterns/distributed-transaction-patterns) — Early Return builds on the Update-with-Start mechanism

---

# Develop durable applications with Temporal SDKs

Source: https://docs.temporal.io/develop

> Discover comprehensive Temporal SDK feature guides and API references. Enhance your Temporal Application development with .NET, Go, Java, PHP, Python, Ruby, Rust, and TypeScript.

The Temporal SDK developer guides provide a comprehensive overview of the structures, primitives, and features used in [Temporal Application](/temporal#temporal-application) development.

- **Go** (v1.50.0): [Developer guide](/develop/go) · [API reference](http://t.mp/go-api)
- **Java** (v1.40.0): [Developer guide](/develop/java) · [API reference](http://t.mp/java-api)
- **.NET** (v1.21.0): [Developer guide](/develop/dotnet) · [API reference](https://dotnet.temporal.io/)
- **PHP** (v2.20): [Developer guide](/develop/php) · [API reference](https://php.temporal.io/namespaces/temporal.html)
- **Python** (v1.35.0): [Developer guide](/develop/python) · [API reference](https://python.temporal.io)
- **Ruby** (v1.10.1): [Developer guide](/develop/ruby) · [API reference](https://ruby.temporal.io/)
- **Rust** (v1.1.1): [Developer guide](/develop/rust) · [API reference](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/)
- **TypeScript** (v1.25.0): [Developer guide](/develop/typescript) · [API reference](https://typescript.temporal.io)

---

# .NET SDK developer guide

Source: https://docs.temporal.io/develop/dotnet

![.NET](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Install and get started

You can find detailed installation instructions for the .NET SDK in the [Quickstart](/develop/dotnet/set-up-your-local-dotnet).

There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running.

Once your local Temporal Service is set up, continue building with the following resources:

- [Workflow basics](/develop/dotnet/workflows/basics)
- [Activity basics](/develop/dotnet/activities/basics)
- [Start an Activity Execution](/develop/dotnet/activities/execution)
- [Run a Worker](/develop/dotnet/workers/run-worker-process)

From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases.

## [Workflows](/develop/dotnet/workflows)

- [Workflow basics](/develop/dotnet/workflows/basics)
- [Child Workflows](/develop/dotnet/workflows/child-workflows)
- [Continue-As-New](/develop/dotnet/workflows/continue-as-new)
- [Cancellation](/develop/dotnet/workflows/cancellation)
- [Timeouts](/develop/dotnet/workflows/timeouts)
- [Message passing](/develop/dotnet/workflows/message-passing)
- [Schedules](/develop/dotnet/workflows/schedules)
- [Timers](/develop/dotnet/workflows/timers)
- [Dynamic Workflow](/develop/dotnet/workflows/dynamic-workflow)
- [Versioning](/develop/dotnet/workflows/versioning)
- [Workflow Streams](/develop/dotnet/workflows/workflow-streams)

## [Activities](/develop/dotnet/activities)

- [Activity basics](/develop/dotnet/activities/basics)
- [Activity Execution](/develop/dotnet/activities/execution)
- [Standalone Activities](/develop/dotnet/activities/standalone-activities-quickstart)
- [Timeouts](/develop/dotnet/activities/timeouts)
- [Asynchronous Activity Completion](/develop/dotnet/activities/asynchronous-activity)
- [Dynamic Activity](/develop/dotnet/activities/dynamic-activity)
- [Benign exceptions](/develop/dotnet/activities/benign-exceptions)

## [Workers](/develop/dotnet/workers)

- [Run a Worker](/develop/dotnet/workers/run-worker-process)
- [Interceptors](/develop/dotnet/workers/interceptors)
- [Serverless Workers](/develop/dotnet/workers/serverless-workers)

## [Temporal Client](/develop/dotnet/client)

- [Temporal Client](/develop/dotnet/client/temporal-client)

## [Temporal Nexus](/develop/dotnet/nexus)

- [Quickstart](/develop/dotnet/nexus/quickstart)
- [Feature guide](/develop/dotnet/nexus/feature-guide)

## [Platform](/develop/dotnet/platform)

- [Observability](/develop/dotnet/platform/observability)
- [Enriching the UI](/develop/dotnet/platform/enriching-ui)

## [Best practices](/develop/dotnet/best-practices)

- [Error handling](/develop/dotnet/best-practices/error-handling)
- [Testing](/develop/dotnet/best-practices/testing-suite)
- [Debugging](/develop/dotnet/best-practices/debugging)
- [Converters and encryption](/develop/dotnet/best-practices/data-handling)

## Temporal .NET technical resources

- [.NET Quickstart](/develop/dotnet/set-up-your-local-dotnet)
- [.NET API Documentation](https://dotnet.temporal.io/api/)
- [.NET SDK Code Samples](https://github.com/temporalio/samples-dotnet)
- [.NET SDK GitHub](https://github.com/temporalio/sdk-dotnet)
- [Temporal 101 in .NET Free Course](https://learn.temporal.io/courses/temporal_101/dotnet/)

Get Connected with the Temporal .NET Community

- [Temporal .NET Community Slack](https://temporalio.slack.com/archives/C012SHMPDDZ)
- [.NET SDK Forum](https://community.temporal.io/tag/dotnet-sdk)

---

# Activities - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities

> Activity Definitions, Activity Execution, Timeouts, Standalone Activities, Dynamic Activities, and benign exceptions in the Temporal .NET SDK.

![.NET SDK Banner](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Activities

- [Activity basics](/develop/dotnet/activities/basics)
- [Activity Execution](/develop/dotnet/activities/execution)
- [Standalone Activities quickstart](/develop/dotnet/activities/standalone-activities-quickstart)
- [Standalone Activities feature guide](/develop/dotnet/activities/standalone-activities)
- [Timeouts](/develop/dotnet/activities/timeouts)
- [Asynchronous Activity Completion](/develop/dotnet/activities/asynchronous-activity)
- [Dynamic Activity](/develop/dotnet/activities/dynamic-activity)
- [Benign exceptions](/develop/dotnet/activities/benign-exceptions)

---

# Asynchronous Activity Completion - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/asynchronous-activity

> Asynchronously complete an Activity in Temporal. Follow simple steps to allow an Activity Function to return without the Activity Execution completing.

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The Temporal Client is used to Heartbeat and complete the Activity.

To mark an Activity as completing asynchronously, do the following inside the Activity.

```csharp
// Capture token for later completion
capturedToken = ActivityExecutionContext.Current.Info.TaskToken;

// Throw special exception that says an activity will be completed somewhere else
throw new CompleteAsyncException();
```

To update an Activity outside the Activity, use the [GetAsyncActivityHandle()](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_GetAsyncActivityHandle_System_Byte___) method to get the handle of the Activity.

```csharp
var handle = myClient.GetAsyncActivityHandle(capturedToken);
```

Then, on that handle, you can call the results of the Activity, `HeartbeatAsync`, `CompleteAsync`, `FailAsync`, or `ReportCancellationAsync` method to update the Activity.

```csharp
await handle.CompleteAsync("Completion value.");
```

---

# Activity basics - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/basics

> Define an Activity as a .NET method with the Activity attribute, give it a custom name, and keep argument Payloads inside the 2 MB and 4 MB gRPC limits.

## Develop an Activity 

One of the primary things that Workflows do is orchestrate the execution of Activities.
An Activity is a normal method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file.
An Activity can interact with the world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service.
For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition).

Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/dotnet/activities/basics#develop-activity). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/dotnet/activities/standalone-activities-quickstart).

You can develop an Activity Definition by using the `[Activity]` attribute from the `Temporalio.Activities` namespace on the method.
To register a method as an Activity with a custom name, use an attribute parameter, for example `[Activity("your-activity")]`.
Otherwise, the activity name is the unqualified method name (sans an "Async" suffix if the method is async).

Activities can be asynchronous or synchronous.

```csharp
using Temporalio.Activities;

public class MyActivities
{
    // Activities can be async and/or static too. We just demonstrate instance methods since many
    // use them that way.
    [Activity]
    public string MyActivity(MyActivityParams input) =>
        $"{input.Greeting}, {input.Name}!";
}
```

There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support.
However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB.
And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB.

Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance.
This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task).

Some SDKs require that you pass context objects, others do not.
When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities.
This is so that you can change what data is passed to the Activity without breaking a method signature.

Activity parameters are the method parameters of the method with the `[Activity]` attribute.
These can be any data type Temporal can convert, including records.
Technically this can be multiple parameters, but Temporal strongly encourages a single parameter containing all input fields.

---

# Benign exceptions - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/benign-exceptions

> Mark expected or non-severe Activity errors as benign in .NET to reduce noise in logs, metrics, and OpenTelemetry traces.

When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues.
By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic.

To mark an error as benign, set the `category` parameter to `ApplicationErrorCategory.Benign` when throwing an [`ApplicationFailureException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.ApplicationFailureException.html).

Benign errors:
- Have Activity failure logs downgraded to DEBUG level
- Do not emit Activity failure metrics
- Do not set the OpenTelemetry failure status to ERROR

```csharp
using Temporalio.Activities;
using Temporalio.Api.Enums.V1;
using Temporalio.Exceptions;

public class MyActivities
{
    [Activity]
    public async Task<string> MyActivityAsync()
    {
        try
        {
            return await CallExternalServiceAsync();
        }
        catch (Exception e)
        {
            // Mark this error as benign since it's expected
            throw new ApplicationFailureException(
                "Service is down",
                inner: e,
                category: ApplicationErrorCategory.Benign);
        }
    }
}
```

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried.

---

# Dynamic Activity - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/dynamic-activity

> Register one Dynamic Activity per Worker with Activity(Dynamic = true) to handle Activity Types that match no other registered Activity.

## Set a Dynamic Activity 

A Dynamic Activity in Temporal is an Activity that is invoked dynamically at runtime if no other Activity with the same name is registered.
An Activity can be made dynamic by setting `Dynamic` as `true` on the `[Activity]` attribute.
You must register the Activity with the Worker before it can be invoked.
Only one Dynamic Activity can be present on a Worker.

The Activity Definition must then accept a single argument of type `Temporalio.Converters.IRawValue[]`.
The [PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Activities.ActivityExecutionContext.html#Temporalio_Activities_ActivityExecutionContext_PayloadConverter) property on the `ActivityExecutionContext` is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` namespace.

```csharp
public class MyActivities
{
    [Activity(Dynamic = true)]
    public string DynamicActivity(IRawValue[] args)
    {
        var input = ActivityExecutionContext.Current.PayloadConverter.ToValue<MyActivityParams>(args.Single());
        return $"{input.Greeting}, {input.Name}!";
    }
}
```

---

# Activity Execution - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/execution

> Call an Activity from a .NET Workflow with ExecuteActivityAsync, set the required Schedule-To-Close or Start-To-Close Timeout, and await the result.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition).
The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command.
This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History.

A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations.
Activity implementation code should be _idempotent_.

The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history.
The entire Execution history is transferred from the Temporal Service to Workflow Workers when a Workflow state needs to recover.
A large Execution history can thus adversely impact the performance of your Workflow.

Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values.
Otherwise, no additional limitations exist on Activity implementations.

To spawn an Activity Execution, use the `ExecuteActivityAsync` operation from within your Workflow Definition.

```csharp
using Temporalio.Workflows;

[Workflow]
public class MyWorkflow
{
    public async Task<string> RunAsync(string name)
    {
        var param = MyActivityParams("Hello", name);
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivity(param),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
}
```

Activity Execution semantics rely on several parameters.
The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).
These values are set in the Activity Options.

### Get the results of an Activity Execution 

The Activity result is returned in the Task from the `ExecuteActivityAsync` call.

---

# Standalone Activities feature guide - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/standalone-activities

> Execute Activities independently without a Workflow using the Temporal .NET SDK.

[Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated
by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/dotnet/activities/basics#develop-activity). The only difference is that you execute a
Standalone Activity directly from your Temporal Client.

> **💡 Tip:**
>
> New to Standalone Activities? Start with the [Standalone Activities quickstart](/develop/dotnet/activities/standalone-activities-quickstart).
>

This page covers the following:

- [Prerequisites](#prerequisites)
- [Start a Standalone Activity without waiting for the result](#start-activity)
- [Get a handle to an existing Standalone Activity](#get-activity-handle)
- [Wait for the result of a Standalone Activity](#get-activity-result)
- [List Standalone Activities](#list-activities)
- [Count Standalone Activities](#count-activities)
- [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from the [StandaloneActivity](https://github.com/temporalio/samples-dotnet/tree/main/src/StandaloneActivity) sample project.
>

## Prerequisites 

Standalone Activities require:

- **.NET** 10.0+ for the sample project (the Temporal .NET SDK requires .NET Core 3.1+, Framework 4.6.2+, or Standard 2.0+)
- **Temporal .NET SDK** v1.19.0 or higher
- **[Temporal CLI](/cli/setup-cli)** v1.9.1 or higher

The [Standalone Activities quickstart](/develop/dotnet/activities/standalone-activities-quickstart)
walks through installing these.

## Start a Standalone Activity without waiting for the result 

Use
[`client.StartActivityAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html)
to start a Standalone Activity and get a handle without waiting for the result:

[src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs)

```csharp
using Temporalio.Client;
using Temporalio.Common.EnvConfig;
using TemporalioSamples.StandaloneActivity;

var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.TargetHost ??= "localhost:7233";
var client = await TemporalClient.ConnectAsync(connectOptions);

var handle = await client.StartActivityAsync(
    () => MyActivities.ComposeGreetingAsync(new ComposeGreetingInput("Hello", "World")),
    new("standalone-activity-id", "standalone-activity-sample")
    {
        ScheduleToCloseTimeout = TimeSpan.FromSeconds(10),
    });
Console.WriteLine($"Started activity: {handle.Id}");

// Wait for the result later
var result = await handle.GetResultAsync();
Console.WriteLine($"Activity result: {result}");
```

With the Temporal Server and Worker running, open a new terminal in the `samples-dotnet` directory and run:

```
dotnet run --project src/StandaloneActivity start-activity
```

Or use the Temporal CLI:

```bash
temporal activity start \
  --type ComposeGreeting \
  --activity-id standalone-activity-id \
  --task-queue standalone-activity-sample \
  --schedule-to-close-timeout 10s \
  --input '{"Greeting": "Hello", "Name": "World"}'
```

## Get a handle to an existing Standalone Activity 

Use `client.GetActivityHandle()` to create a handle to a previously started Standalone Activity:

```csharp
// Without a known result type
var handle = client.GetActivityHandle("my-activity-id", runId: "the-run-id");

// With a known result type
var typedHandle = client.GetActivityHandle<string>("my-activity-id", runId: "the-run-id");
```

You can use the handle to wait for the result, describe, cancel, or terminate the Activity.

## Wait for the result of a Standalone Activity 

Under the hood, calling `client.ExecuteActivityAsync()` is the same as calling
`client.StartActivityAsync()` to durably enqueue the Standalone Activity, and then calling
`await handle.GetResultAsync()` to wait for the Activity to be executed and return the result:

```csharp
var result = await handle.GetResultAsync();
```

Or use the Temporal CLI to wait for a result by Activity ID:

```bash
temporal activity result --activity-id my-standalone-activity-id
```

## List Standalone Activities 

Use
[`client.ListActivitiesAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html)
to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is
an `IAsyncEnumerable` that yields `ActivityExecution` entries.

These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included.

[src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs)

```csharp
using Temporalio.Client;
using Temporalio.Common.EnvConfig;

var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.TargetHost ??= "localhost:7233";
var client = await TemporalClient.ConnectAsync(connectOptions);

await foreach (var info in client.ListActivitiesAsync(
    "TaskQueue = 'standalone-activity-sample'"))
{
    Console.WriteLine(
        $"ActivityID: {info.ActivityId}, Type: {info.ActivityType}, Status: {info.Status}");
}
```

Run it:

```
dotnet run --project src/StandaloneActivity list-activities
```

Or use the Temporal CLI:

```bash
temporal activity list
```

The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow
Visibility](/visibility). For example, `"ActivityType = 'ComposeGreeting' AND ExecutionStatus = 'Running'"`.

## Count Standalone Activities 

Use
[`client.CountActivitiesAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html)
to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns
the total count of executions (running, completed, failed, etc.) - not the number of queued tasks.
It works the same way as counting Workflow Executions.

[src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs)

```csharp
using Temporalio.Client;
using Temporalio.Common.EnvConfig;

var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.TargetHost ??= "localhost:7233";
var client = await TemporalClient.ConnectAsync(connectOptions);

var resp = await client.CountActivitiesAsync(
    "TaskQueue = 'standalone-activity-sample'");
Console.WriteLine($"Total activities: {resp.Count}");
```

Run it:

```
dotnet run --project src/StandaloneActivity count-activities
```

Or use the Temporal CLI:

```bash
temporal activity count
```

## Run Standalone Activities with Temporal Cloud 

The code samples on this page use `ClientEnvConfig.LoadClientConnectOptions()`, so the same code
works against Temporal Cloud - just configure the connection via environment variables or a TOML
profile. No code changes are needed.

For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate
generation, and authentication setup in the Cloud UI, see
[Connect to Temporal Cloud](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud).

### Connect with mTLS

Set these environment variables with values from your Temporal Cloud Namespace settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem'
export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key'
```

### Connect with an API key

Set these environment variables with values from your Temporal Cloud API key settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_API_KEY=<your-api-key>
```

Then run the Worker and starter code as shown in the [Standalone Activities quickstart](/develop/dotnet/activities/standalone-activities-quickstart).

---

# Standalone Activities .NET quickstart - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/standalone-activities-quickstart

> Execute a Standalone Activity with the Temporal .NET SDK without writing a Workflow.

# Quickstart

Standalone Activities are Activities that run independently, without being orchestrated by a
Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/dotnet/activities/basics#develop-activity). The only difference is that you execute a
Standalone Activity directly from your Temporal Client.

> **📝 Note:**
>
> This documentation uses source code from the [StandaloneActivity](https://github.com/temporalio/samples-dotnet/tree/main/src/StandaloneActivity) sample project.
>

## Get started with Standalone Activities 

Prerequisites:

- **[.NET](https://dotnet.microsoft.com/download)** 10.0+ for the sample project
  (the Temporal .NET SDK requires .NET Core 3.1+, Framework 4.6.2+, or Standard 2.0+)

- **Temporal .NET SDK** (v1.19.0 or higher). See the [.NET Quickstart](/develop/dotnet/set-up-your-local-dotnet) for install instructions.

- **Temporal CLI** v1.9.1 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`.

Start the Temporal development server with `temporal server start-dev`.

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Server will now be available for client connections on `localhost:7233`, and the
Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233).

```bash
brew install temporal
```

```bash
temporal --version
```

```bash
temporal server start-dev
```

## Clone the sample

Clone the [samples-dotnet](https://github.com/temporalio/samples-dotnet) repository to follow along:

```
git clone https://github.com/temporalio/samples-dotnet.git
cd samples-dotnet
```

The sample project is structured as follows:

```
src/StandaloneActivity/
├── MyActivities.cs
├── Program.cs
├── README.md
└── TemporalioSamples.StandaloneActivity.csproj
```

## Define your Activity 

An Activity in the Temporal .NET SDK is a method decorated with the `[Activity]` attribute. The way
you write a Standalone Activity is identical to how you write an Activity orchestrated by a Workflow.
In fact, the same Activity can be executed both as a Standalone Activity and as a Workflow Activity.

[src/StandaloneActivity/MyActivities.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/MyActivities.cs)

```csharp
namespace TemporalioSamples.StandaloneActivity;

using Temporalio.Activities;

public static class MyActivities
{
    [Activity]
    public static Task<string> ComposeGreetingAsync(ComposeGreetingInput input) =>
        Task.FromResult($"{input.Greeting}, {input.Name}!");
}

public record ComposeGreetingInput(string Greeting, string Name);
```

## Run a Worker with the Activity registered 

Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities —
you create a Worker, register the Activity, and run the Worker. The Worker doesn't need to know
whether the Activity will be invoked from a Workflow or as a Standalone Activity. See
[Develop a Worker](/develop/dotnet/workers/run-worker-process) for more details on Worker setup and configuration
options.

[src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs)

Open a new terminal, navigate to the `samples-dotnet` directory, and run the Worker.
Leave this terminal running - the Worker needs to stay up to process activities.

```csharp
using Microsoft.Extensions.Logging;
using Temporalio.Client;
using Temporalio.Common.EnvConfig;
using Temporalio.Worker;
using TemporalioSamples.StandaloneActivity;

var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.TargetHost ??= "localhost:7233";
connectOptions.LoggerFactory = LoggerFactory.Create(builder =>
    builder.
        AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] ").
        SetMinimumLevel(LogLevel.Information));
var client = await TemporalClient.ConnectAsync(connectOptions);

const string taskQueue = "standalone-activity-sample";

using var tokenSource = new CancellationTokenSource();
Console.CancelKeyPress += (_, eventArgs) =>
{
    tokenSource.Cancel();
    eventArgs.Cancel = true;
};

using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions(taskQueue).
        AddActivity(MyActivities.ComposeGreetingAsync));

await worker.ExecuteAsync(tokenSource.Token);
```

```bash
dotnet run --project src/StandaloneActivity worker
```

## Execute a Standalone Activity 

Use
[`client.ExecuteActivityAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClientExtensions.html)
to execute a Standalone Activity and wait for the result. Call this from your application code, not
from inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal
Server, waits for it to be executed on your Worker, and then returns the result.

[src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs)

You can pass the Activity as either a lambda expression or a string Activity type name.

`StartActivityOptions` requires `Id`, `TaskQueue`, and at least one of `ScheduleToCloseTimeout` or
`StartToCloseTimeout`. See
[`StartActivityOptions`](https://dotnet.temporal.io/api/Temporalio.Client.StartActivityOptions.html)
in the API reference for the full set of options.

To run it:

1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above).
2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above).
3. Open a new terminal, navigate to the `samples-dotnet` directory, and run:
    ```
    dotnet run --project src/StandaloneActivity execute-activity
    ```

Or use the Temporal CLI.

```csharp
using Temporalio.Client;
using Temporalio.Common.EnvConfig;
using TemporalioSamples.StandaloneActivity;

var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.TargetHost ??= "localhost:7233";
var client = await TemporalClient.ConnectAsync(connectOptions);

var result = await client.ExecuteActivityAsync(
    () => MyActivities.ComposeGreetingAsync(new ComposeGreetingInput("Hello", "World")),
    new("standalone-activity-id", "standalone-activity-sample")
    {
        ScheduleToCloseTimeout = TimeSpan.FromSeconds(10),
    });
Console.WriteLine($"Activity result: {result}");
```

```csharp
// Using a lambda expression (type-safe)
var result = await client.ExecuteActivityAsync(
    () => MyActivities.ComposeGreetingAsync(new ComposeGreetingInput("Hello", "World")),
    new("standalone-activity-id", "standalone-activity-sample")
    {
        ScheduleToCloseTimeout = TimeSpan.FromSeconds(10),
    });

// Using a string type name
var result = await client.ExecuteActivityAsync<string>(
    "ComposeGreeting",
    new object?[] { new ComposeGreetingInput("Hello", "World") },
    new("standalone-activity-id", "standalone-activity-sample")
    {
        ScheduleToCloseTimeout = TimeSpan.FromSeconds(10),
    });
```

```bash
dotnet run --project src/StandaloneActivity execute-activity
```

```bash
temporal activity execute \\
  --type ComposeGreeting \\
  --activity-id standalone-activity-id \\
  --task-queue standalone-activity-sample \\
  --schedule-to-close-timeout 10s \\
  --input '{"Greeting": "Hello", "Name": "World"}'
```

## Run with Temporal Cloud

All code samples on this page use
[`ClientEnvConfig.LoadClientConnectOptions()`](https://dotnet.temporal.io/api/Temporalio.Common.EnvConfig.ClientEnvConfig.html)
to configure the Temporal Client connection. It responds to [environment
variables](/references/client-environment-configuration) and [TOML configuration
files](/references/client-environment-configuration), so the same code works against a local dev
server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal
Cloud](/develop/dotnet/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the feature guide
for mTLS and API key setup.

## Next steps

- **[Standalone Activities feature guide](/develop/dotnet/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud.
- **[Activity basics](/develop/dotnet/activities/basics)**: Write and register Activities with the .NET SDK.

---

# Activity Timeouts - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/activities/timeouts

> Set Activity Timeouts and a Retry Policy with the Temporal .NET SDK, override the next retry delay, and Heartbeat long-running Activities.

## Activity Timeouts 

Each Activity Timeout controls the maximum duration of a different aspect of an Activity Execution.

The following Timeouts are available in the Activity Options.

- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution).
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution).
- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

These values can be set in the `ActivityOptions` when calling `ExecuteActivityAsync`.

Available timeouts are:

- ScheduleToCloseTimeout
- StartToCloseTimeout
- ScheduleToStartTimeout

```csharp
return await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.MyActivity(param),
    new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
```

### Set an Activity Retry Policy 

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided.

To create an Activity Retry Policy in .NET, set the `RetryPolicy` on the `ActivityOptions` when calling `ExecuteActivityAsync`.

```csharp
return await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.MyActivity(param),
    new()
    {
        StartToCloseTimeout = TimeSpan.FromMinutes(5),
        RetryPolicy = new() { MaximumInterval = TimeSpan.FromSeconds(10) },
    });
```

### Override the Retry interval with `nextRetryDelay` 

When you throw an [Application Failure](/references/failures#application-failure) and assign the `nextRetryDelay` field, its value replaces and overrides the Retry interval defined in the active Retry Policy.

For example, you might scale the next Retry delay interval based on the current number of attempts.
Here's how you'd do that in an Activity.
In the following sample, the `attempt` count is retrieved from the Activity Execution context and used to set the number of seconds for the next Retry delay:

```csharp
var attempt = ActivityExecutionContext.Current.Info.Attempt;

throw new ApplicationFailureException(
    $"Something bad happened on attempt {attempt}",
    errorType: "my_failure_type",
    nextRetryDelay: TimeSpan.FromSeconds(3 * attempt));
```

## Heartbeat an Activity 

An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service).
Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed.
If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy.

Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation.
Heartbeat throttling may lead to Cancellation getting delivered later than expected.

Heartbeats can contain a `Details` field describing the Activity's current progress.
If an Activity gets retried, the Activity can access the `Details` from the last Heartbeat that was sent to the Temporal Service.

To Heartbeat an Activity Execution in .NET, use the [`Heartbeat()`](https://dotnet.temporal.io/api/Temporalio.Activities.ActivityExecutionContext.html#Temporalio_Activities_ActivityExecutionContext_Heartbeat_System_Object___) method on the `ActivityExecutionContext`.

```csharp
[Activity]
public async Task MyActivityAsync()
{
    while (true)
    {
        // Send heartbeat
        ActivityExecutionContext.Current.Heartbeat();

        // Do some work, passing the cancellation token
        await Task.Delay(1000, ActivityExecutionContext.Current.CancellationToken);
    }
}
```

In addition to obtaining cancellation information, Heartbeats also support detail data that persists on the server for retrieval during Activity retry.
If an Activity calls `Heartbeat(123, 456)` and then fails and is retried, `HeartbeatDetails` on the `ActivityInfo` returns a collection containing `123` and `456` on the next Run.

### Set a Heartbeat Timeout 

A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat).

`HeartbeatTimeout` is a property on `ActivityOptions` for `ExecuteActivityAsync` used to set the maximum time between Activity Heartbeats.

```csharp
await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.MyActivity(param),
    new()
    {
        StartToCloseTimeout = TimeSpan.FromMinutes(5),
        HeartbeatTimeout = TimeSpan.FromSeconds(30),
    });
```

---

# Best practices - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/best-practices

> This section explains how to implement best practices with the .NET SDK

![.NET SDK Banner](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Best practices

- [Error handling](/develop/dotnet/best-practices/error-handling)
- [Testing](/develop/dotnet/best-practices/testing-suite)
- [Debugging](/develop/dotnet/best-practices/debugging)
- [Converters and encryption](/develop/dotnet/best-practices/data-handling)

---

# Data handling - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/best-practices/data-handling

All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three
layers that handle different concerns:

![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)

Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON
serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when
your application requires non-JSON types, encryption, or payload offloading.

|                           | [PayloadConverter](/develop/dotnet/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/dotnet/best-practices/data-handling/data-encryption) |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- |
| **Purpose**               | Serialize application data to bytes                               | Transform encoded payloads (encrypt, compress)                |
| **Default**               | JSON serialization                                                | None (passthrough)                                            |

For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage).

---

# Payload conversion - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/best-practices/data-handling/data-conversion

> Customize how Temporal serializes application objects using Payload Converters in the .NET SDK.

## Payload conversion

Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back.

### Conversion sequence 

The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter.
You can set multiple encoding Payload Converters to run your conversions.
When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

Payload Converters can be customized independently of a Payload Codec.
Temporal's Converter architecture looks like this:

![Temporal converter architecture](https://docs.temporal.io/img/info/converter-architecture.png)

## Custom Payload Converter 

Data converters are used to convert raw Temporal payloads to/from actual .NET types.
A custom data converter can be set via the `DataConverter` option when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters.
Payload converters convert .NET values to/from serialized bytes. Payload codecs convert bytes to bytes (for example, for compression or encryption). Failure converters convert exceptions to/from serialized failures.

Data converters are in the `Temporalio.Converters` namespace.
The default data converter uses a default payload converter, which supports the following types:

- `null`
- `byte[]`
- `Google.Protobuf.IMessage` instances
- Anything that `System.Text.Json` supports
- `IRawValue` as unconverted raw payloads

Custom converters can be created for all uses. For example, to create client with a data converter that converts all C#
property names to camel case, you would:

```csharp
using System.Text.Json;
using Temporalio.Client;
using Temporalio.Converters;

public class CamelCasePayloadConverter : DefaultPayloadConverter
{
    public CamelCasePayloadConverter()
      : base(new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase })
    {
    }
}

var client = await TemporalClient.ConnectAsync(new()
{
    TargetHost = "localhost:7233",
    Namespace = "my-namespace",
    DataConverter = DataConverter.Default with { PayloadConverter = new CamelCasePayloadConverter() },
});
```

---

# Payload encryption - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/best-practices/data-handling/data-encryption

> Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the .NET SDK.

Temporal's security model is designed around client-side encryption of Payloads.
A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server.
This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data.
It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures.

A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client.
Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these.

The server itself never adds encryption over Payloads.
Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows.
When working with sensitive data, you should always implement Payload encryption.

## Custom Payload Codec 

Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps.
If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`.

The `IPayloadCodec` needs to implement `EncodeAsync()` and `DecodeAsync()` methods.
These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field.
Do not mutate the existing payloads.
Here is an example of an encryption codec that just uses base64 in each direction:

```csharp
public class EncryptionCodec : IPayloadCodec
{
    public Task<IReadOnlyCollection<Payload>> EncodeAsync(IReadOnlyCollection<Payload> payloads) =>
        Task.FromResult<IReadOnlyCollection<Payload>>(payloads.Select(p =>
        {
            return new Payload()
            {
                // Set our specific encoding. We may also want to add a key ID in here for use by
                // the decode side
                Metadata = { ["encoding"] = "binary/my-payload-encoding" },
                Data = ByteString.CopyFrom(Encrypt(p.ToByteArray())),
            };
        }).ToList());

    public Task<IReadOnlyCollection<Payload>> DecodeAsync(IReadOnlyCollection<Payload> payloads) =>
        Task.FromResult<IReadOnlyCollection<Payload>>(payloads.Select(p =>
        {
            // Ignore if it doesn't have our expected encoding
            if (p.Metadata.GetValueOrDefault("encoding") != "binary/my-payload-encoding")
            {
                return p;
            }
            // Decrypt
            return Payload.Parser.ParseFrom(Decrypt(p.Data.ToByteArray()));
        }).ToList());

    private byte[] Encrypt(byte[] data) => Encoding.ASCII.GetBytes(Convert.ToBase64String(data));

    private byte[] Decrypt(byte[] data) => Convert.FromBase64String(Encoding.ASCII.GetString(data));
}
```

### Set Data Converter to use custom Payload Codec

When creating a client, the default `DataConverter` can be updated with the payload codec like so:

```csharp
var myClient = await TemporalClient.ConnectAsync(new("localhost:7233")
{
    DataConverter = DataConverter.Default with { PayloadCodec = new EncryptionCodec() },
});
```

- Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted.
- Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output.

For reference, see the [Encryption](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption) sample.

## Using a Codec Server

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints.
You create, operate, and manage access to your Codec Server in your own environment.
The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.
Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server.

---

# Debugging - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/best-practices/debugging

> Debug Workflows in development and production environments using Temporal .NET SDK. Use logging, debugger, Web UI, CLI, replay, tracing, and more for efficient troubleshooting.

## Debugging 

This page shows how to do the following:

- [Debug in a development environment](#debug-in-a-development-environment)
- [Debug in a production environment](#debug-in-a-development-production)

### Debug in a development environment 

In developing Workflows, you can use the normal development tools of logging and a debugger to see what’s happening in your Workflow.
In addition to the normal development tools of logging and a debugger, you can also see what’s happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli).
The Web UI provides insight into your Workflows, making it easier to identify issues and monitor the state of your Workflows in real time.

### Debug in a production environment 

You can debug production Workflows using:

- [Web UI](/web-ui)
- [Temporal CLI](/cli)
- [Replay](/develop/dotnet/best-practices/testing-suite#replay)
- [Tracing](/develop/dotnet/platform/observability#tracing)
- [Logging](/develop/dotnet/platform/observability#logging)

You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance).
For more information, see [Observability ▶️ Metrics](/develop/dotnet/platform/observability#metrics) for setting up SDK metrics.

Debug Server performance with [Cloud metrics](/cloud/metrics/openmetrics) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics).

---

# Error handling - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/best-practices/error-handling

> Handle errors with Temporal .NET SDK

## Raise and handle exceptions 

In each Temporal SDK, error handling is implemented idiomatically, following the conventions of the language.
Temporal uses several different error classes internally — for example, [`CancelledFailureException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.CanceledFailureException.html) in the .NET SDK, to handle a Workflow cancellation. 
You should not raise or otherwise implement these manually, as they are tied to Temporal platform logic.

The one Temporal error class that you will typically raise deliberately is [`ApplicationFailureException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.ApplicationFailureException.html).
In fact, *any* other exceptions that are raised from your C# code in a Temporal Activity will be converted to an `ApplicationFailureException` internally.
This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries.

In other words, these two code samples do the same thing:

```csharp
[Serializable]
public class InvalidDepartmentException : Exception
{
    public InvalidDepartmentException() : base() { }
    public InvalidDepartmentException(string message) : base(message) { }
    public InvalidDepartmentException(string message, Exception inner) : base(message, inner) { }
}

[Activity]
public Task<OrderConfirmation> SendBillAsync(Bill bill)
{
    throw new InvalidDepartmentException("Invalid department");
}
```

```csharp
[Activity]
public Task<OrderConfirmation> SendBillAsync(Bill bill)
{
    throw new ApplicationFailureException("Invalid department", errorType: "InvalidDepartmentException");
}
```

Depending on your implementation, you may decide to use either method.
One reason to use the Temporal `ApplicationFailureException` class is because it allows you to set an additional `non_retryable` parameter.
This way, you can decide whether an error should not be retried automatically by Temporal.
This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse:

```csharp
[Activity]
public Task<OrderConfirmation> SendBillAsync(Bill bill)
{
    throw new ApplicationFailureException("Invalid department", nonRetryable: true);
}
```

You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](/develop/dotnet/activities/timeouts#activity-retries).

## Failing Workflows 

One of the core design principles of Temporal is that an Activity Failure will never directly cause a Workflow Failure — a Workflow should never return as Failed unless deliberately.
The default retry policy associated with Temporal Activities is to retry them until reaching a certain timeout threshold.
Activities will not actually *return* a failure to your Workflow until this condition or another non-retryable condition is met.
At this point, you can decide how to handle an error returned by your Activity the way you would in any other program.
For example, you could implement a [Saga Pattern](/design-patterns/saga-pattern) — see this [.NET sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Saga) — that uses `try`/`catch` blocks to "unwind" some of the steps your Workflow has performed up to the point of Activity Failure.

**You will only fail a Workflow by manually raising an `ApplicationFailureException` from the Workflow code.**
You could do this in response to an Activity Failure, if the failure of that Activity means that your Workflow should not continue:

```csharp
try
{
    await Workflow.ExecuteActivityAsync(
        (Activities act) => act.ValidateCreditCardAsync(order.Customer.CreditCardNumber),
        options);
}
catch (ActivityFailureException err)
{
    logger.LogError("Unable to process credit card: {Message}", err.Message);
    throw new ApplicationFailureException(message: "Invalid credit card number error");
}
```

This works differently in a Workflow than raising exceptions from Activities.
In an Activity, any C# exceptions or custom exceptions are converted to a Temporal `ApplicationError`.
By default, in a Workflow, any exceptions that are raised other than an explicit Temporal `ApplicationError` will only fail that particular [Workflow Task](/tasks#workflow-task-execution) and be retried.
This includes any typical C# `RuntimeError`s that are raised automatically.
These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly.
To have other exception types fail the Workflow Execution, list them in `FailureExceptionTypes` on the `[Workflow]` attribute, or in `TemporalWorkerOptions.WorkflowFailureExceptionTypes`.
Both options are experimental.

---

# Testing - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/best-practices/testing-suite

> The .NET test-suite guide covers Workflow and integration testing for Temporal. It includes end-to-end, integration, and unit testing, emphasizing the use of the test server to optimize test execution.

The .NET test-suite feature guide describes the frameworks that facilitate Workflow and integration testing.

In the context of Temporal, you can create these types of automated tests:

- **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client.
- **Integration:** Anything between end-to-end and unit testing.
  - Running Activities with mocked Context and other SDK imports (and usually network requests).
  - Running Workers with mock Activities, and using a Client to start Workflows.
  - Running Workflows with mocked SDK imports.
- **Unit:** Running a piece of Workflow or Activity code and mocking any code it calls.

We generally recommend writing the majority of your tests as integration tests.

Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers.

## Test frameworks 

**Compatible testing frameworks**

The .NET SDK is compatible with any testing framework and does not have a specific recommendation.
Most .NET SDK samples use [xUnit](https://xunit.net/).

## Testing Workflows 

Workflow testing can be done in an integration-test fashion against a real server, however it is hard to simulate timeouts and other long time-based code.
Using the time-skipping Workflow test environment can help there.

### Testing Workflows with standard server

A non-time-skipping `Temporalio.Testing.WorkflowEnvironment` can be started via `StartLocalAsync` which supports all standard Temporal features.
It is actually the real Temporal dev server packaged in the Temporal CLI, lazily downloaded on first use, and run as a sub-process in the background.
Assuming tests properly use separate Task Queues, the same server can and should be reused across tests.

Here's a simple example of a Workflow:

```csharp
[Workflow]
public class SayHelloWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        return $"Hello, {name}!";
    }
}
```

Here's how a test of that Workflow may appear in xUnit:

```csharp
using Temporalio.Testing;
using Temporalio.Worker;

[Fact]
public async Task SayHelloWorkflow_SimpleRun_Succeeds()
{
    // Start local dev server
    await using var env = await WorkflowEnvironment.StartLocalAsync();

    // Create a worker
    using var worker = new TemporalWorker(
      env.Client,
      new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}").
          AddWorkflow<SayHelloWorkflow>());

    // Run the worker only for the life of the code within
    await worker.ExecuteAsync(async () =>
    {
        // Execute the workflow and confirm the result
        var result = await env.Client.ExecuteWorkflowAsync(
            (SayHelloWorkflow wf) => wf.RunAsync("Temporal"),
            new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!));
        Assert.Equal("Hello, Temporal!", result);
    });
}
```

While this is just a demonstration, a local server is often used as a fixture across many tests.

### Testing Workflows with time skipping

Sometimes there is a need to test Workflows that run a long time or to test that timeouts occur.
A time-skipping `Temporalio.Testing.WorkflowEnvironment` can be started via `StartTimeSkippingAsync` which is a reimplementation of the Temporal server with special time skipping capabilities.

Like `StartLocalAsync`, this also lazily downloads the process to run when first called.
Note, unlike `StartLocalAsync`, this class is not thread safe nor safe for use with independent tests.
It can be technically be reused, but only for one test at a time because time skipping is locked/unlocked at the environment level.
Developers are encouraged to run it per test needed.

#### Automatic time skipping

Here's a simple example of a Workflow that waits a day:

```csharp
[Workflow]
public class WaitADayWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        await Workflow.DelayAsync(TimeSpan.FromDays(1));
        return "all done";
    }
}
```

A regular integration test of this Workflow on a normal server would be way too slow.
However, the time-skipping server automatically skips to the next event when we wait on the result.
Here's a test for that Workflow in xUnit:

```csharp
using Temporalio.Testing;
using Temporalio.Worker;

[Fact]
public async Task WaitADayWorkflow_SimpleRun_Succeeds()
{
    // Start time-skipping test server
    await using var env = await WorkflowEnvironment.StartTimeSkippingAsync();

    // Create a worker
    using var worker = new TemporalWorker(
      env.Client,
      new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}").
          AddWorkflow<WaitADayWorkflow>());

    // Run the worker only for the life of the code within
    await worker.ExecuteAsync(async () =>
    {
        // Execute the workflow and confirm the result
        var result = await env.Client.ExecuteWorkflowAsync(
            (WaitADayWorkflow wf) => wf.RunAsync(),
            new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!));
        Assert.Equal("all done", result);
    });
}
```

This test will run almost instantly.
This is because by calling `ExecuteWorkflowAsync` on our client, we are actually calling `StartWorkflowAsync` + `GetResultAsync`, and `GetResultAsync` automatically skips time as much as it can (basically until the end of the Workflow or until an Activity is run).

To disable automatic time-skipping while waiting for a workflow result, run code as a lambda passed to `env.WithAutoTimeSkippingDisabled` or `env.WithAutoTimeSkippingDisabledAsync`.

#### Manual time skipping

Until a Workflow is waited on, all time skipping in the time-skipping environment is done manually via `WorkflowEnvironment.DelayAsync`.

Here's a Workflow that waits for a Signal or times out:

```csharp
[Workflow]
public class SignalWorkflow
{
    private bool signalReceived = false;

    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        // Wait for signal or timeout in 45 seconds
        if (Workflow.WaitConditionAsync(() => signalReceived, TimeSpan.FromSeconds(45)))
        {
            return "got signal";
        }
        return "got timeout";
    }

    [WorkflowSignal]
    public async Task SomeSignalAsync() => signalReceived = true;
}
```

To test a normal Signal in xUnit, you might:

```csharp
using Temporalio.Testing;
using Temporalio.Worker;

[Fact]
public async Task SignalWorkflow_SendSignal_HasExpectedResult()
{
    await using var env = await WorkflowEnvironment.StartTimeSkippingAsync();
    using var worker = new TemporalWorker(
        env.Client,
        new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}").
            AddWorkflow<SignalWorkflow>());
    await worker.ExecuteAsync(async () =>
    {
        var handle = await env.Client.StartWorkflowAsync(
            (SignalWorkflow wf) => wf.RunAsync(),
            new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!));
        await handle.SignalAsync(wf => wf.SomeSignalAsync());
        Assert.Equal("got signal", await handle.GetResultAsync());
    });
}
```

But how would you test the timeout part? Like so:

```csharp
using Temporalio.Testing;
using Temporalio.Worker;

[Fact]
public async Task SignalWorkflow_SignalTimeout_HasExpectedResult()
{
    await using var env = await WorkflowEnvironment.StartTimeSkippingAsync();
    using var worker = new TemporalWorker(
        env.Client,
        new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}").
            AddWorkflow<SignalWorkflow>());
    await worker.ExecuteAsync(async () =>
    {
        var handle = await env.Client.StartWorkflowAsync(
            (SignalWorkflow wf) => wf.RunAsync(),
            new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!));
        await env.DelayAsync(TimeSpan.FromSeconds(50));
        Assert.Equal("got timeout", await handle.GetResultAsync());
    });
}
```

### Mocking Activities

When testing Workflows, often you don't want to actually run the Activities.
Activities are just methods with the `[Activity]` attribute.
Simply write different/empty/fake/asserting ones and pass those to the Worker to have different activities called during the test.

## Testing Activities 

Unit testing an Activity or any code that could run in an Activity is done via the `Temporalio.Testing.ActivityEnvironment` class.
Simply instantiate the class, and any code inside `RunAsync` will be invoked inside the activity context.
The following important members are available on the environment to affect the activity context:

- `Info` - Activity info, defaulted to a basic set of values.
- `Logger` - Activity logger, defaulted to a null logger.
- `Cancel(CancelReason)` - Helper to set the reason and cancel the source.
- `CancelReason` - Cancel reason.
- `CancellationTokenSource` - Token source for issuing cancellation.
- `Heartbeater` - Callback invoked each heartbeat.
- `WorkerShutdownTokenSource` - Token source for issuing Worker shutdown.
- `PayloadConverter` - Defaulted to default payload converter.

## Replay test 

Given a Workflow's history, it can be replayed locally to check for things like non-determinism errors.
For example, assuming the `history` parameter below is given a JSON string of history exported from the CLI or web UI, the following method will replay it:

```csharp
using Temporalio;
using Temporalio.Worker;

public static async Task ReplayFromJsonAsync(string historyJson)
{
    var replayer = new WorkflowReplayer(
        new WorkflowReplayerOptions().AddWorkflow<MyWorkflow>());
    await replayer.ReplayWorkflowAsync(WorkflowHistory.FromJson("my-workflow-id", historyJson));
}
```

If there is a non-determinism, this will throw an exception.

Event history can be loaded from more than just JSON.
It can be fetched individually from a Workflow handle, or even in a list.
For example, the following code will check that all Workflow histories for a certain Workflow type (that is, workflow class) are safe with the current Workflow code.

```csharp
using Temporalio;
using Temporalio.Client;
using Temporalio.Worker;

public static async Task CheckPastHistoriesAsync(ITemporalClient client)
{
    var replayer = new WorkflowReplayer(
        new WorkflowReplayerOptions().AddWorkflow<MyWorkflow>());
    var listIter = client.ListWorkflowHistoriesAsync("WorkflowType = 'SayHello'");
    await foreach (var result in replayer.ReplayWorkflowsAsync(listIter))
    {
        if (result.ReplayFailure != null)
        {
            ExceptionDispatchInfo.Throw(result.ReplayFailure);
        }
    }
}
```

---

# Client - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/client

![.NET SDK Banner](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Temporal Client

- [Temporal Client](/develop/dotnet/client/temporal-client)

---

# Temporal Client - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service.
Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and
Queries to Workflow Executions, getting Workflow results, and more.

For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow.

This page shows you how to do the following using the .NET SDK with the Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-development-service)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Start a Workflow Execution](#start-workflow)
- [Get Workflow results](#get-workflow-results)

A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a
Temporal Client inside an Activity to communicate with a Temporal Service.

## Connect to development Temporal Service 

Use
[`TemporalClient.ConnectAsync`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_ConnectAsync_Temporalio_Client_TemporalClientConnectOptions_)
to create a client. Connection options include the Temporal Server address, Namespace, and (optionally) TLS
configuration. You can provide these options directly in code, or load them from **environment variables** and/or a
**TOML configuration file** using the `Temporalio.Client.EnvConfig` helpers. We recommend environment variables or a
configuration file for secure, repeatable configuration.

When you're running a Temporal Service locally (such as with the
[Temporal CLI dev server](/cli/command-reference/server#start-dev)), the required options are minimal. If you
don't specify a host/port, most connections default to `127.0.0.1:7233` and the `default` Namespace.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the
TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK
looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to
[Environment Configuration](/references/client-environment-configuration) for more details about configuration
files and profiles.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own
set of connection options.

```toml title="config.toml"
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Optional: Add custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
# TLS auto-enables when TLS config or an API key is present
# disabled = false
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path  = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment     = "production"
service-version = "v1.2.3"
```

You can create a Temporal Client using a profile from the configuration file as follows. In this example, you load the
`default` profile for local development:

```csharp title="LoadFromFile.cs" {27-30}
using Temporalio.Client;
using Temporalio.Client.EnvConfig;

namespace TemporalioSamples.EnvConfig;

/// <summary>
/// Sample demonstrating loading the default environment configuration profile
/// from a TOML file.
/// </summary>
public static class LoadFromFile
{
    public static async Task RunAsync()
    {
        Console.WriteLine("--- Loading default profile from config.toml ---");

        try
        {
            // For this sample to be self-contained, we explicitly provide the path to
            // the config.toml file included in this directory.
            // By default though, the config.toml file will be loaded from
            // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
            var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml");

            // LoadClientConnectOptions is a helper that loads a profile and prepares
            // the config for TemporalClient.ConnectAsync. By default, it loads the
            // "default" profile.
            var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions
            {
                ConfigSource = DataSource.FromPath(configFile),
            });

            Console.WriteLine($"Loaded 'default' profile from {configFile}.");
            Console.WriteLine($"  Address: {connectOptions.TargetHost}");
            Console.WriteLine($"  Namespace: {connectOptions.Namespace}");
            if (connectOptions.RpcMetadata?.Count > 0)
            {
                Console.WriteLine($"  gRPC Metadata: {string.Join(", ", connectOptions.RpcMetadata.Select(kv => $"{kv.Key}={kv.Value}"))}");
            }

            Console.WriteLine("\nAttempting to connect to client...");

            var client = await TemporalClient.ConnectAsync(connectOptions);
            Console.WriteLine("✅ Client connected successfully!");

            // Test the connection by checking the service
            var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new());
            Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            Console.WriteLine($"❌ Failed to connect: {ex.Message}");
        }
    }
}
```

**Environment Variables**

Use the `EnvConfig` package to set connection options for the Temporal Client using environment variables. For a list of
all available environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

Set the following environment variables before running your .NET application. Replace the placeholder values with your
actual configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and
the `default` Namespace. You may omit these variables entirely since they're the defaults.

```bash
export TEMPORAL_NAMESPACE="default"
export TEMPORAL_ADDRESS="localhost:7233"
```

After setting the environment variables, use the following code to create the Temporal Client:

```csharp
using Temporalio.Client;
using Temporalio.Client.EnvConfig;

namespace TemporalioSamples.EnvConfig;

/// <summary>
/// Sample demonstrating loading the default environment configuration profile
/// from a TOML file.
/// </summary>
public static class LoadFromFile
{
    public static async Task RunAsync()
    {
        try
        {
            var connectOptions = ClientEnvConfig.LoadClientConnectOptions();

            Console.WriteLine("\nAttempting to connect to client...");

            var client = await TemporalClient.ConnectAsync(connectOptions);
            Console.WriteLine("✅ Client connected successfully!");
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            Console.WriteLine($"❌ Failed to connect: {ex.Message}");
        }
    }
}
```

**Code**

If you don't want to use environment variables or a configuration file, you can specify connection options directly in
code. This is convenient for local development and testing. You can also load a base configuration from environment
variables or a configuration file, and then override specific options in code.

```csharp
using System;
using System.Threading.Tasks;
using Temporalio.Client;

namespace TemporalioSamples.Manual
{
    public static class ManualConnect
    {
        public static async Task RunAsync()
        {
            Console.WriteLine("--- Connecting manually to Temporal ---");

            var client = await TemporalClient.ConnectAsync(new TemporalClientConnectOptions
            {
                TargetHost = "localhost:7233",
                Namespace  = "default",
            });

            Console.WriteLine("✅ Connected to local Temporal service!");
        }
    }
}

```

## Connect to Temporal Cloud 

You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal
Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured
local development instance:

- Your credentials for authentication.
  - If you are using an API key, provide the API key value.
  - If you are using mTLS, provide the mTLS CA certificate and mTLS private key.
- Your _Namespace and Account ID_ combination, which follows the format `<namespace_id>.<account_id>`.
- The recommended _endpoint_ is the gRPC Namespace endpoint: `<namespace>.<account>.tmprl.cloud:7233`.
  This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability).
  See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options.

You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab.

For more information about managing and generating client certificates for Temporal Cloud, see
[How to manage certificates in Temporal Cloud](/cloud/certificates).

You can provide these connection options using environment variables, a configuration file, or directly in code.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to
[Environment Configuration](/references/client-environment-configuration).

You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path
to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the
default path `~/.config/temporalio/temporal.toml`.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines a `cloud` profile with the necessary connection options to
connect to Temporal Cloud via an API key:

```toml
# Cloud profile for Temporal Cloud
[profile.cloud]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```

If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and
private key:

```toml
# Cloud profile for Temporal Cloud
[profile.cloud]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
tls_client_cert_data = "your-tls-client-cert-data"
tls_client_key_path = "your-tls-client-key-path"
```

With the connections options defined in the configuration file, use the `ClientEnvConfig.LoadClientConnectOptions`
method to create a Temporal Client from a named profile as follows. After loading the profile, you can also
programmatically override specific connection options before creating the client.

```csharp title="LoadProfile.cs" {25,41}
using Temporalio.Client;
using Temporalio.Client.EnvConfig;

namespace TemporalioSamples.EnvConfig;

/// <summary>
/// Sample demonstrating loading a named environment configuration profile and
/// programmatically overriding its values.
/// </summary>
public static class LoadProfile
{
    public static async Task RunAsync()
    {
        Console.WriteLine("--- Loading 'staging' profile with programmatic overrides ---");

        try
        {
            var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml");
            var profileName = "staging";

            Console.WriteLine("The 'staging' profile in config.toml has an incorrect address (localhost:9999).");
            Console.WriteLine("We'll programmatically override it to the correct address.");

            // Load the 'staging' profile
            var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions
            {
                Profile = profileName,
                ConfigSource = DataSource.FromPath(configFile),
            });

            // Override the target host to the correct address.
            // This is the recommended way to override configuration values.
            connectOptions.TargetHost = "localhost:7233";

            Console.WriteLine($"\nLoaded '{profileName}' profile from {configFile} with overrides.");
            Console.WriteLine($"  Address: {connectOptions.TargetHost} (overridden from localhost:9999)");
            Console.WriteLine($"  Namespace: {connectOptions.Namespace}");

            Console.WriteLine("\nAttempting to connect to client...");

            var client = await TemporalClient.ConnectAsync(connectOptions);
            Console.WriteLine("✅ Client connected successfully!");

            // Test the connection by checking the service
            var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new());
            Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            Console.WriteLine($"❌ Failed to connect: {ex.Message}");
        }
    }
}
```

**Environment Variables**

The following environment variables are required to connect to Temporal Cloud:

- `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `<namespace_id>.<account_id>`.
- `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace.
- `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication.
- `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path.
  Required if you are using mTLS authentication.
- `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path.
  Required if you are using mTLS authentication.

Ensure these environment variables exist in your environment before running your .NET application.

Import the `Temporalio.Client.EnvConfig` namespace to set connection options for the Temporal Client using environment variables.
The `ClientEnvConfig.LoadClientConnectOptions` method will automatically load all environment variables. For a list of all available
environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

```csharp {16,20}
using Temporalio.Client;
using Temporalio.Client.EnvConfig;

namespace TemporalioSamples.EnvConfig;

/// <summary>
/// Sample demonstrating loading the default environment configuration profile
/// from a TOML file.
/// </summary>
public static class LoadFromFile
{
    public static async Task RunAsync()
    {
        try
        {
            var connectOptions = ClientEnvConfig.LoadClientConnectOptions();

            Console.WriteLine("\nAttempting to connect to client...");

            var client = await TemporalClient.ConnectAsync(connectOptions);
            Console.WriteLine("✅ Client connected successfully!");
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            Console.WriteLine($"❌ Failed to connect: {ex.Message}");
        }
    }
}
```

**Code**

You can also provide connections options in your .NET code directly. To create an initial connection, provide the
Namespace and API key values to the ` TemporalClient.ConnectAsync` method.

```csharp
var myClient = TemporalClient.ConnectAsync(new(<endpoint>)
{
    Namespace = "<namespace_id>.<account_id>",
    ApiKey = "<APIKey>",
    Tls = new(),
});
```

To update an API key, update the value of `ApiKey` on the existing client connection:

```csharp
myClient.Connection.ApiKey = myKeyUpdated;
```

To connect using mTLS instead of an API key, provide the client certificate and private key on `Tls`:

```csharp
var myClient = TemporalClient.ConnectAsync(new(<endpoint>)
{
    Namespace = "<namespace_id>.<account_id>",
    Tls = new()
    {
        ClientCert = File.ReadAllBytes("client-cert.pem"),
        ClientPrivateKey = File.ReadAllBytes("client-private-key.pem"),
    },
});
```

Unlike `ApiKey`, `TlsOptions` is not mutable on an existing connection. The certificate bytes are fixed once the
connection is established. To rotate an mTLS client certificate without restarting your Worker, connect a new client
with the new certificate and assign it to the running `TemporalWorker`'s `Client` property:

```csharp
var newClient = await TemporalClient.ConnectAsync(new(<endpoint>)
{
    Namespace = "<namespace_id>.<account_id>",
    Tls = new()
    {
        ClientCert = File.ReadAllBytes("client-cert-new.pem"),
        ClientPrivateKey = File.ReadAllBytes("client-private-key-new.pem"),
    },
});

// worker is the TemporalWorker instance already running against the old client
worker.Client = newClient;
```

Setting `Client` replaces the connection the Worker uses for subsequent calls to the Temporal Service (Workflow Task
completion, Activity Heartbeats, and so on); calls already in flight on the old client are not interrupted.

## Start a Workflow Execution 

[Workflow Execution](/workflow-execution) semantics rely on several parameters. To start a Workflow Execution, you must
supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific
contextual data, and Workflow Function parameters.

A request to spawn a Workflow Execution causes the Temporal Service to create the first Event
([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The
Temporal Service then creates the first Workflow Task, resulting in the first
[WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event.

To start a Workflow Execution in .NET, use either the `StartWorkflowAsync()` or `ExecuteWorkflowAsync()` methods in the
Client. You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) and [Task Queue](/task-queue) in
the `WorkflowOptions` given to the method.

```csharp
var result = await client.ExecuteWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue");
Console.WriteLine("Result: {0}", result);
```

### Set a Workflow's Task Queue 

The only Workflow Option that you must set is the name of the [Task Queue](/task-queue).

For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task
Queue name.

To set a Task Queue in .NET, specify the `taskQueue` argument in the `WorkflowOptions` given to `StartWorkflowAsync()`
or `ExecuteWorkflowAsync()`.

```csharp
var result = await client.ExecuteWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue"));
```

### Set a Workflow Id 

You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id).

When setting a Workflow Id, we recommend mapping it to a business process or business entity identifier, such as an
order identifier or customer identifier.

To set a Workflow Id in .NET, specify the `id` argument in the `WorkflowOptions` given to `StartWorkflowAsync()` or
`ExecuteWorkflowAsync()`.

```csharp
var result = await client.ExecuteWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue"));
```

## Get Workflow results 

If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id.

The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its
result.

It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time
(asynchronous execution).

In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and
results of Workflow Executions.

Use `StartWorkflowAsync()` or `GetWorkflowHandle()` to return a Workflow handle. Then use the `GetResultAsync()` method
to await on the result of the Workflow.

To get a handle for an existing Workflow by its Id, you can use `GetWorkflowHandle()`.

Then use
[`DescribeAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_DescribeAsync_Temporalio_Client_WorkflowDescribeOptions_)
to get the current status of the Workflow. If the Workflow does not exist, this call fails.

```csharp
var handle = client.GetWorkflowHandle("my-workflow-id");
var result = await handle.GetResultAsync<string>();
Console.WriteLine("Result: {0}", result);
```

---

# Nexus - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/nexus

> This section explains how to use Temporal Nexus with the .NET SDK

![.NET SDK Banner](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Temporal Nexus

- [Quickstart](/develop/dotnet/nexus/quickstart)
- [Feature guide](/develop/dotnet/nexus/feature-guide)
- [Standalone Operations](/develop/dotnet/nexus/standalone-operations)

---

# Nexus feature guide - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/nexus/feature-guide

> Use Temporal Nexus within the .NET SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **⚠️ Currently in: Pre-release:**
> This feature guide includes the new Nexus developer experience: pre-release APIs for the [Temporal Operation Handler](/nexus/temporal-operation-handler) and [Nexus Standalone Activity](/nexus/standalone-activity). These APIs are experimental and subject to change.

Use [Temporal Nexus](/evaluate/features/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **💡 Tip:**
>
> New to Nexus? Start with the [Nexus .NET quickstart](/develop/dotnet/nexus/quickstart).
>

This page shows how to do the following:

- [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server)
- [Create caller and handler Namespaces](#create-caller-handler-namespaces)
- [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint)
- [Define the Nexus Service contract](#define-nexus-service-contract)
- [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers)
- [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service)
- [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server)
- [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code derived from the [.NET Nexus sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusSimple) and the [NexusMessagingTemporalOperation sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessagingTemporalOperation).
>

## Run the Temporal Development Server with Nexus enabled 

Prerequisites:

- [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/dotnet/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli)
  (v1.9.1 or higher recommended)
- [Install the latest Temporal .NET SDK](https://learn.temporal.io/getting_started/dotnet/dev_environment/#install-the-temporal-net-sdk)
  (v1.20.0 or higher recommended)

The first step in working with Temporal Nexus involves starting a Temporal server with Nexus enabled.

```
temporal server start-dev \
  --dynamic-config-value history.enableUpdateCallbacks=true \
  --dynamic-config-value history.enableCHASMSignalBacklinks=true \
  --dynamic-config-value history.enableSignalWithStartFromWorkflow=true
```

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases.

The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`.

## Create caller and handler Namespaces 

Before setting up Nexus endpoints, create separate Namespaces for the caller and handler.

```
temporal operator namespace create --namespace nexus-simple-handler-namespace
temporal operator namespace create --namespace nexus-simple-caller-namespace
```

`nexus-simple-handler-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `nexus-simple-caller-namespace` to call that Operation handler.
We use different namespaces to demonstrate cross-Namespace Nexus calls.

## Create a Nexus Endpoint to route requests from caller to handler 

After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests.

```
temporal operator nexus endpoint create \
  --name nexus-simple-endpoint \
  --target-namespace nexus-simple-handler-namespace \
  --target-task-queue nexus-simple-handler-sample
```

You can also use the Web UI to create the Namespaces and Nexus endpoint.

## Define the Nexus Service contract 

Defining a clear contract for the Nexus Service is crucial for smooth communication.

In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint.

You can hand-write that package, but the preferred way is to generate it with [NexGen](/nexus/nexgen).
You write the contract once as a JSON or YAML definition file and run `nexgen` against it, and it emits the typed models,
runtime validators, and the Service definition itself.

This is what makes a Nexus Service polyglot. Both sides generate from the same definition file: the handler implements
the Service, the caller invokes its Operations, and neither side hand-writes request or response types. A C# handler
and a Go caller share no code, but they both run off that same service contract, so they interoperate with no
coordination between the teams beyond the contract itself.

The generated validators check every payload against the contract when a value is parsed off the wire and again when
it is serialized onto it, so bad data is rejected at the boundary rather than reaching your Workflow or Activity. A value validates
identically in every language, which is what lets a caller and a handler written in different languages trust the same
contract. See the [`chat.nexusrpc.yaml`](https://github.com/temporalio/nexgen/blob/main/samples/schemas/chat.nexusrpc.yaml)
sample contract and the [Definition files](https://github.com/temporalio/nexgen#definition-files) section of the
`nexgen` README for the file format.

## Develop a Nexus Service and Operation handlers 

Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract.
Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous.
They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code.
Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database.
Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors, blocking all Operations from the caller to that Endpoint.

Every Operation is written with a [Temporal Operation Handler](/nexus/temporal-operation-handler). Mark a method with
`[TemporalOperation]`; that method is the Operation handler and receives three things: a
`TemporalOperationStartContext`, an `ITemporalNexusClient`, and the Operation input. The Operation the method handles is
matched by method name to the corresponding `[NexusOperation]` method on the Service interface. What you do with the
Client determines what backs the Operation:

- **Synchronous.** Return `TemporalOperationResult<T>.SyncResult(...)` and the Operation completes during the handler
  call. The caller has its result as soon as the call returns.
- **Asynchronous.** Call `StartWorkflowAsync`, `StartActivityAsync`, or `StartWorkflowUpdateAsync` on the Client. The
  handler returns as soon as that Execution has started, and the Operation stays open until the Execution finishes,
  which may be days later. Its result is delivered to the caller through the Nexus completion callback. This is what
  lets an Operation outlive the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

A handler can perform any number of synchronous side effects, such as sending a Signal, but at most one asynchronous
backing per invocation.

### Develop a Synchronous Nexus Operation handler

Return a synchronous result when the Operation can answer immediately. The handler computes the answer and returns it,
and the Operation completes during the call.

Handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking), and the whole
call has to finish inside the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

```csharp
using NexusRpc.Handlers;
using Temporalio.Nexus;

[NexusServiceHandler(typeof(IHelloService))]
public class HelloService
{
    [TemporalOperation]
    public Task<TemporalOperationResult<IHelloService.EchoOutput>> Echo(
        TemporalOperationStartContext ctx,
        ITemporalNexusClient client,
        IHelloService.EchoInput input) =>
        Task.FromResult(TemporalOperationResult<IHelloService.EchoOutput>.SyncResult(
            new(input.Message)));
}
```

### Use the Temporal Client for Signals, Queries, and Updates

A common pattern is to reach a Workflow that is already running. Query it or Signal it from a synchronous Operation, or
use Signal-With-Start to make sure the Workflow exists before the Signal arrives. Those calls complete during the
handler call, so they have to finish inside the
[Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

Updates are the exception. Do not wait for one inside the handler. Start it with `StartWorkflowUpdateAsync` and it backs
the Operation. The handler returns immediately, and the Operation completes when the Update does, however long it
takes.

The [NexusMessaging](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessaging)
sample shows a Nexus Service that Queries and Signals a running Workflow from synchronous Operations, and backs an
Operation with a Workflow Update.

The Client your handler receives is not an ordinary Temporal Client. It propagates
[bidirectional links](/nexus/execution-debugging#bi-directional-linking) and request IDs on every call, so the
caller-side and handler-side Executions are connected in the UI without wiring anything. Reach the Workflow Client
through `client.TemporalClient` rather than constructing your own.

In this example the Workflow ID is derived from an identifier carried in the Operation input, so the caller only needs
the identifier it cares about:

```csharp
private static string WorkflowIdForUser(string userId) => $"GreetingWorkflow_for_{userId}";

[TemporalOperation]
public async Task<TemporalOperationResult<INexusGreetingService.ApproveOutput>> Approve(
    TemporalOperationStartContext ctx,
    ITemporalNexusClient client,
    INexusGreetingService.ApproveInput input)
{
    var handle = client.TemporalClient.GetWorkflowHandle<GreetingWorkflow>(
        WorkflowIdForUser(input.UserId));
    await handle.SignalAsync(wf => wf.ApproveAsync(input));
    return TemporalOperationResult<INexusGreetingService.ApproveOutput>.SyncResult(new());
}
```

There are two examples of messaging through Nexus in the sample code: the [caller pattern](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessaging/CallerPattern) and the [on-demand pattern](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessaging/OnDemandPattern).
The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it.

### Develop an Asynchronous Nexus Operation handler to start a Workflow

Call `StartWorkflowAsync` on the Client. The Operation completes when the Workflow returns, and the Workflow's return
value is delivered to the caller as the Operation's result.

```csharp
[NexusServiceHandler(typeof(IHelloService))]
public class HelloService
{
    [TemporalOperation]
    public Task<TemporalOperationResult<IHelloService.HelloOutput>> SayHello(
        TemporalOperationStartContext ctx,
        ITemporalNexusClient client,
        IHelloService.HelloInput input) =>
        client.StartWorkflowAsync(
            (HelloHandlerWorkflow wf) => wf.RunAsync(input),
            // Workflow IDs should typically be business meaningful IDs and are used to dedupe
            // workflow starts. Task queue defaults to the operation's task queue when omitted.
            new() { Id = $"hello-{input.Name}-{input.Language}" });
}
```

Workflow IDs should typically be business-meaningful identifiers and are used to dedupe Workflow starts. In general, the Workflow ID should be passed in the Operation input as part of the Nexus Service contract.

> **💡 Tip:**
> RESOURCES
>
> [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing.
>

#### Map a Nexus Operation input to multiple Workflow arguments

A Nexus Operation can only take one input parameter. To start a Workflow that takes several, pass them as separate
arguments in the `RunAsync` lambda:

```csharp
client.StartWorkflowAsync(
    (HelloHandlerWorkflow wf) => wf.RunAsync(input.Language, input.Name),
    new() { Id = $"hello-{input.Name}-{input.Language}" });
```

### Register a Nexus Service in a Worker

After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register a Nexus Service in a Worker.

[NexusSimple/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Program.cs)

```csharp
async Task RunHandlerWorkerAsync()
{
    // Run worker until cancelled
    logger.LogInformation("Running handler worker");
    using var worker = new TemporalWorker(
        await ConnectClientAsync("nexus-simple-handler-namespace"),
        new TemporalWorkerOptions(taskQueue: "nexus-simple-handler-sample").
            AddNexusService(new HelloService()).
            AddWorkflow<HelloHandlerWorkflow>());
    try
    {
        await worker.ExecuteAsync(tokenSource.Token);
    }
    catch (OperationCanceledException)
    {
        logger.LogInformation("Handler worker cancelled");
    }
}
```

Nexus Service handlers also support dependency injection through the generic-host Worker. See [NexusDependencyInjection sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusDependencyInjection).

## Develop a caller Workflow that uses the Nexus Service 

Import the Service interface that has the necessary Operation names and input/output types to execute a Nexus Operation
from the caller Workflow:

[NexusSimple/Caller/EchoCallerWorkflow.workflow.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Caller/EchoCallerWorkflow.workflow.cs)

```csharp
using Temporalio.Workflows;

[Workflow]
public class EchoCallerWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string message)
    {
        var output = await Workflow.CreateNexusWorkflowClient<IHelloService>(IHelloService.EndpointName).
            ExecuteNexusOperationAsync(svc => svc.Echo(new(message)));
        return output.Message;
    }
}
```

[NexusSimple/Caller/HelloCallerWorkflow.workflow.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Caller/HelloCallerWorkflow.workflow.cs)

```csharp
using Temporalio.Workflows;

[Workflow]
public class HelloCallerWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name, IHelloService.HelloLanguage language)
    {
        var output = await Workflow.CreateNexusWorkflowClient<IHelloService>(IHelloService.EndpointName).
            ExecuteNexusOperationAsync(svc => svc.SayHello(new(name, language)));
        return output.Message;
    }
}
```

### Register the caller Workflow in a Worker

After developing the caller Workflow, the next step is to register it with a Worker.

[NexusSimple/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Program.cs)

```csharp
async Task RunCallerWorkerAsync()
{
    // Run worker until cancelled
    logger.LogInformation("Running caller worker");
    using var worker = new TemporalWorker(
        await ConnectClientAsync("nexus-simple-caller-namespace"),
        new TemporalWorkerOptions(taskQueue: "nexus-simple-caller-sample").
            AddWorkflow<EchoCallerWorkflow>().
            AddWorkflow<HelloCallerWorkflow>());
    try
    {
        await worker.ExecuteAsync(tokenSource.Token);
    }
    catch (OperationCanceledException)
    {
        logger.LogInformation("Caller worker cancelled");
    }
}
```

### Develop a starter to start the caller Workflow

To initiate the caller Workflow, a starter program is used.

[NexusSimple/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Program.cs)

```csharp
async Task ExecuteCallerWorkflowAsync()
{
    logger.LogInformation("Executing caller echo workflow");
    var client = await ConnectClientAsync("nexus-simple-caller-namespace");
    var result1 = await client.ExecuteWorkflowAsync(
        (EchoCallerWorkflow wf) => wf.RunAsync("Nexus Echo 👋"),
        new(id: "nexus-simple-echo-id", taskQueue: "nexus-simple-caller-sample"));
    logger.LogInformation("Workflow result: {Result}", result1);

    logger.LogInformation("Executing caller hello workflow");
    var result2 = await client.ExecuteWorkflowAsync(
        (HelloCallerWorkflow wf) => wf.RunAsync("Temporal", IHelloService.HelloLanguage.Es),
        new(id: "nexus-simple-hello-id", taskQueue: "nexus-simple-caller-sample"));
    logger.LogInformation("Workflow result: {Result}", result2);
}
```

## Make Nexus calls across Namespaces with a development Server 

Follow the steps below to run the Nexus handler Worker, the Nexus caller Worker, and the starter app.

### Run Workers connected to a local development server

Run the Nexus handler Worker:

```bash
dotnet run handler-worker
```

In another terminal window, run the Nexus caller Worker:

```bash
dotnet run caller-worker
```

### Start a caller Workflow

With the Workers running, the final step in the local development process is to start a caller Workflow.

Run the starter:

```bash
dotnet run caller-workflow
```

This will show the two workflows started and their results.

### Canceling a Nexus Operation 

To cancel a Nexus Operation from within a Workflow, cancel the cancellation token passed to the operation call. Only asynchronous operations can be canceled in Nexus, since cancellation is sent using an operation token.
The Workflow or other resources backing the operation may choose to ignore the cancellation request.
If ignored, the operation may enter a terminal state.

When a Nexus operation is started, the caller can specify different cancellation types that control how the caller reacts to cancellation:

- `Abandon` - Do not request cancellation of the operation.
- `TryCancel` - Initiate a cancellation request and immediately report cancellation to the caller. Note that this type doesn't guarantee that cancellation is delivered to the operation handler if the caller exits before the delivery is done.
- `WaitCancellationRequested` - Request cancellation of the operation and wait for confirmation that the request was received. Doesn't wait for actual cancellation.
- `WaitCancellationCompleted` - Wait for operation completion. Operation may or may not complete as cancelled.

The default is `WaitCancellationCompleted`. Users can set a different option for `CancellationType` in `NexusWorkflowOperationOptions` when starting an operation.

Once the caller Workflow completes, the caller's Nexus Machinery stops attempting to cancel operations that have not yet
been canceled, letting them run to completion.

It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending
operations to finish before exiting the Workflow.

See the [Nexus cancellation sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusCancellation) for reference.

## Make Nexus calls across Namespaces in Temporal Cloud 

This section assumes you are already familiar with how to connect a Worker to Temporal Cloud.
The Temporal Cloud CLI is used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces.

### Install `tcld` and generate certificates

Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command
(on macOS):

```
brew install temporalio/brew/tcld
```

If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below:

```
tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key
```

These certificates will be valid for one year.

### Create caller and handler Namespaces

Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler.
If you already have these Namespaces, you don't need to do this.

**Temporal CLI**

```
temporal cloud login

temporal cloud namespace create \
  --name <your-caller-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

temporal cloud namespace create \
  --name <your-target-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

**tcld**

```
tcld login

tcld namespace create \
  --namespace <your-caller-namespace> \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

tcld namespace create \
  --namespace <your-target-namespace> \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/namespaces](https://cloud.temporal.io/namespaces).

### Create a Nexus Endpoint to route requests from caller to handler

To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`.

**Temporal CLI**

```
temporal cloud nexus endpoint create \
  --name nexus-simple-endpoint \
  --target-task-queue nexus-simple-handler-sample \
  --target-namespace <your-handler-namespace.account> \
  --allow-namespace <your-caller-namespace.account> \
  --description-file endpoint_description.md
```

**tcld**

```
tcld nexus endpoint create \
  --name nexus-simple-endpoint \
  --target-task-queue nexus-simple-handler-sample \
  --target-namespace <your-handler-namespace.account> \
  --allow-namespace <your-caller-namespace.account> \
  --description-file endpoint_description.md
```

The `--allow-namespace` flag builds an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in [Runtime Access Controls](/nexus/security#runtime-access-controls).

Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus).

## Observability

### Web UI

A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history:

![Observability Sync](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-sync.png)

An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history:

![Observability Async](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-async.png)

### Temporal CLI

Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow:

```
temporal workflow describe -w <ID>
```

Nexus events are included in the caller's Event history:

```
temporal workflow show -w <ID>
```

For **asynchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationStarted`
- `NexusOperationCompleted`

For **synchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationCompleted`

> **📝 Note:**
>
> `NexusOperationStarted` isn't reported in the caller's history for synchronous operations.
>

## Learn more

- Read the high-level description of the [Temporal Nexus feature](/evaluate/features/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082).
- Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus).
- Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus).

---

# Nexus .NET quickstart - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/nexus/quickstart

> Build a Nexus Service that wraps an existing Temporal Workflow using the .NET SDK

[Temporal Nexus](/evaluate/features/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow.

> **ℹ️ Info:**
> To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/features/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus).

**Prerequisites:** Complete the [.NET SDK Quickstart](/develop/dotnet/set-up-your-local-dotnet) first.
You should have `SayHelloWorkflow`, `MyActivities`, and a `Worker` project from that guide.

## What you'll build

You have `SayHelloWorkflow` running in the `default` Namespace.

By the end of this guide:

1. A Nexus Service will expose `SayHelloWorkflow` as an Operation.
2. A second Namespace will contain a Workflow that calls that Operation.
3. The caller Workflow will get back `"Hello, Temporal!"` — the same result, but across Namespaces.

## 1. Define the Nexus Service

Create a file called `ISayHelloNexusService.cs` in the `Workflow` project.

The `[NexusService]` attribute on an interface defines the Nexus Service contract. `[NexusOperation]` marks each method that callers can invoke. The `EndpointName` static field is shared between the handler and caller to keep the endpoint name in one place.

`SayHelloWorkflow` returns `string`, so the operation output type is `string`. The input is a `record` carrying the workflow argument.

```csharp
namespace MyNamespace;

using NexusRpc;

[NexusService]
public interface ISayHelloNexusService
{
    public static readonly string EndpointName = "my-nexus-endpoint-name";

    [NexusOperation]
    string SayHello(MyInput input);

    public record MyInput(string Name);
}
```

## 2. Define the Nexus Operation handler

Create a file called `SayHelloNexusServiceHandler.cs` in the `Workflow` project.

`[NexusServiceHandler]` links this class to the `ISayHelloNexusService` contract. Each `[NexusOperationHandler]` method returns an `IOperationHandler` that describes how the operation runs.

`WorkflowRunOperationHandler.FromHandleFactory` creates an asynchronous operation backed by a Workflow run. The `input.Name` bridges the Nexus `MyInput` record to `SayHelloWorkflow`'s `string` parameter.

Using `context.HandlerContext.RequestId` as the Workflow ID ensures that retried Nexus operation requests are deduplicated.

```csharp
namespace MyNamespace;

using NexusRpc.Handlers;
using Temporalio.Nexus;

[NexusServiceHandler(typeof(ISayHelloNexusService))]
public class SayHelloNexusServiceHandler
{
    [NexusOperationHandler]
    public IOperationHandler<ISayHelloNexusService.MyInput, string> SayHello() =>
        WorkflowRunOperationHandler.FromHandleFactory(
            (WorkflowRunOperationContext context, ISayHelloNexusService.MyInput input) =>
                context.StartWorkflowAsync(
                    (SayHelloWorkflow wf) => wf.RunAsync(input.Name),
                    new() { Id = context.HandlerContext.RequestId }));
}
```

## 3. Register the Nexus Service handler in a Worker

Update `Worker/Program.cs` to register the Nexus Service handler alongside the existing Workflow and Activity registrations.

A Worker will only handle incoming Nexus requests if the Nexus Service handlers are registered. Like `.AddActivity()`, `.AddNexusService()` takes an instance — both register concrete objects that the Worker dispatches work to.

```csharp
// Worker/Program.cs
var activities = new MyActivities();
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
        .AddActivity(activities.SayHello)
        .AddWorkflow<SayHelloWorkflow>()
        .AddNexusService(new SayHelloNexusServiceHandler()));
```

## 4. Develop the caller Workflow

Create a file called `CallerWorkflow.cs` in the `Workflow` project.

The caller Workflow uses `Workflow.CreateNexusWorkflowClient<T>()` to get a typed client bound to the Nexus Endpoint. `ExecuteNexusOperationAsync` starts the operation and waits for the result.

The caller only depends on the Service contract (`ISayHelloNexusService`), not the handler implementation. This decoupling is what allows the caller and handler to live in separate Namespaces or even separate codebases.

```csharp
namespace MyNamespace;

using Temporalio.Workflows;

[Workflow]
public class CallerWorkflow
{
    public static readonly string CallerTaskQueue = "my-caller-task-queue";

    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        return await Workflow
            .CreateNexusWorkflowClient<ISayHelloNexusService>(
                ISayHelloNexusService.EndpointName)
            .ExecuteNexusOperationAsync(svc => svc.SayHello(new(name)));
    }
}
```

## 5. Create the caller Namespace and Nexus Endpoint

Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler. The handler uses the `default` Namespace that was created when you started the dev server.

Namespaces provide isolation between the caller and handler sides. The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue. The endpoint name must match the `EndpointName` constant defined in Step 1.

Make sure your local Temporal dev server is running (`temporal server start-dev`).

```bash
temporal operator namespace create --namespace my-caller-namespace
```

```bash
temporal operator nexus endpoint create \\
  --name my-nexus-endpoint-name \\
  --target-namespace default \\
  --target-task-queue my-task-queue
```

## 6. Add the caller project

Create a `CallerStarter` console project that starts a caller Worker and executes the Workflow.

The commands on the right create a new console project, add it to the solution, reference the Workflow project, and add the Temporalio package.

```bash
dotnet new console -o CallerStarter
dotnet sln TemporalioHelloWorld.sln \\
  add CallerStarter/CallerStarter.csproj
dotnet add CallerStarter/CallerStarter.csproj \\
  reference Workflow/Workflow.csproj
dotnet add CallerStarter/CallerStarter.csproj package Temporalio
```

## 7. Create the caller starter

Create the `CallerStarter/Program.cs` file with the code on the right.

This brings everything together: the caller Worker hosts `CallerWorkflow`, which uses the Nexus client to invoke `SayHello` on the handler side. The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `SayHelloWorkflow`, and back to the caller.

```csharp
using MyNamespace;
using Temporalio.Client;
using Temporalio.Worker;

var client = await TemporalClient.ConnectAsync(
    new("localhost:7233") { Namespace = "my-caller-namespace" });

using var tokenSource = new CancellationTokenSource();
Console.CancelKeyPress += (_, eventArgs) =>
{
    tokenSource.Cancel();
    eventArgs.Cancel = true;
};

using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions(CallerWorkflow.CallerTaskQueue)
        .AddWorkflow<CallerWorkflow>());

Console.WriteLine("Running caller worker");
var workerTask = worker.ExecuteAsync(tokenSource.Token);

var result = await client.ExecuteWorkflowAsync(
    (CallerWorkflow wf) => wf.RunAsync("Temporal"),
    new(id: $"caller-workflow-{Guid.NewGuid()}",
        taskQueue: CallerWorkflow.CallerTaskQueue));
Console.WriteLine("Workflow result: {0}", result);

tokenSource.Cancel();
try { await workerTask; } catch (OperationCanceledException) { }
```

## 8. Run and verify

Run the application using the commands on the right.

You should see:

```
Workflow result: Hello, Temporal!
```

Open the [Temporal Web UI](http://localhost:8233) and find the `CallerWorkflow` execution in the `my-caller-namespace` Namespace. You should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in the Event history.

```bash
dotnet run --project Worker/Worker.csproj
```

```bash
dotnet run --project CallerStarter/CallerStarter.csproj
```

## Next steps

Now that you have a working Nexus Service, here are some resources to deepen your understanding:

- **[.NET Nexus feature guide](/develop/dotnet/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls.
- **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics.
- **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker.
- **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns.
- **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries.
- **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls.
- **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests.
- **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity.

---

# Standalone Nexus Operations - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/nexus/standalone-operations

> Execute Nexus Operations independently without a Workflow using the Temporal .NET SDK.

> **Public Preview**
> Requires .NET SDK `1.16.0` or above. API breaking changes are kept to a minimum before General Availability.

[Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without
being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using
`Workflow.CreateNexusWorkflowClient<TService>()`, you execute a Standalone Nexus Operation directly from a Nexus Client
created using `ITemporalClient.CreateNexusClient<TService>()`.

Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as
Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/dotnet/nexus/feature-guide)
for details on
[defining a Service contract](/develop/dotnet/nexus/feature-guide#define-nexus-service-contract),
[developing Operation handlers](/develop/dotnet/nexus/feature-guide#develop-nexus-service-operation-handlers), and
[registering a Service in a Worker](/develop/dotnet/nexus/feature-guide#register-a-nexus-service-in-a-worker).

This page focuses on the client-side APIs that are unique to Standalone Nexus Operations:

- [Execute a Standalone Nexus Operation](#execute-operation)
- [Get the result of a Standalone Nexus Operation](#get-operation-result)
- [List Standalone Nexus Operations](#list-operations)
- [Count Standalone Nexus Operations](#count-operations)
- [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud)

> **📝 Note:**
> This documentation uses source code from the
> [.NET Nexus Standalone sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusStandaloneOperations).
>

## Prerequisites 

Standalone Nexus Operations are a Public Preview feature and require Temporal CLI v1.9.0 or higher.

### 1. Install and verify the Temporal CLI 

The `temporal nexus operation` commands require Temporal CLI v1.9.0 or higher. See
[Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for install options,
then verify:

```bash
temporal --version
```

Which should output v1.9.0 or higher.

### 2. Start a local dev server 

The dev server enables Standalone Nexus Operations by default — no dynamic config is
required. Start it with the caller and handler Namespaces pre-created:

```bash
temporal server start-dev \
  --namespace my-caller-namespace \
  --namespace my-handler-namespace
```

The starter and Worker connect to two different Namespaces (a caller Namespace and a handler
Namespace), mirroring how Nexus crosses Namespace boundaries.

To run the examples on this page against the [.NET sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusStandaloneOperations),
create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue:

```bash
temporal operator nexus endpoint create \
  --name my-nexus-endpoint \
  --target-namespace my-handler-namespace \
  --target-task-queue nexus-handler-queue
```

Start the sample Worker in the handler Namespace:

```bash
TEMPORAL_NAMESPACE=my-handler-namespace dotnet run worker
```

Run the starter in the caller Namespace (from a separate terminal):

```bash
TEMPORAL_NAMESPACE=my-caller-namespace dotnet run starter
```

## Execute a Standalone Nexus Operation 

To execute a Standalone Nexus Operation, first create a
[`NexusClient`](https://dotnet.temporal.io/api/Temporalio.Client.NexusClient.html) using
`ITemporalClient.CreateNexusClient<TService>()`, bound to a specific Nexus Endpoint and Service. The endpoint must be
pre-created on the server. Then call `ExecuteNexusOperationAsync()` from application code (for example, a starter
program), not from inside a Workflow Definition.

`ExecuteNexusOperationAsync` is a shortcut that starts the Operation and waits for the result. If you need a handle to
the Operation while it runs, call `StartNexusOperationAsync` instead — it returns a
[`NexusOperationHandle`](https://dotnet.temporal.io/api/Temporalio.Client.NexusOperationHandle.html) that you can use to
get the result, describe, cancel, or terminate the Operation. `Id` is required on
[`NexusOperationOptions`](https://dotnet.temporal.io/api/Temporalio.Client.NexusOperationOptions.html);
`ScheduleToCloseTimeout` is optional and defaults to the maximum allowed by the Temporal server.

```csharp
var nexusClient = client.CreateNexusClient<IHelloService>("my-nexus-endpoint");

var result = await nexusClient.ExecuteNexusOperationAsync(
    svc => svc.Echo(new("Nexus Echo 👋")),
    new("unique-operation-id")
    {
        ScheduleToCloseTimeout = TimeSpan.FromSeconds(10),
    });
```

You can also use the untyped overload that takes the Operation name as a string:

```csharp
var nexusClient = client.CreateNexusClient("my-nexus-endpoint", "HelloService");

var handle = await nexusClient.StartNexusOperationAsync<IHelloService.EchoOutput>(
    "Echo",
    new IHelloService.EchoInput("Nexus Echo 👋"),
    new("unique-operation-id")
    {
        ScheduleToCloseTimeout = TimeSpan.FromSeconds(10),
    });
```

Or use the Temporal CLI to execute a Standalone Nexus Operation:

```bash
temporal nexus operation execute \
  --namespace my-caller-namespace \
  --endpoint my-nexus-endpoint \
  --service HelloService \
  --operation Echo \
  --operation-id my-echo-op \
  --input '{"Message":"hello"}'
```

## Get the result of a Standalone Nexus Operation 

Use `NexusOperationHandle<TResult>.GetResultAsync()` to await the Operation's completion and retrieve its result. This
works for both synchronous and asynchronous (Workflow-backed) Operations.

```csharp
var output = await handle.GetResultAsync();
logger.LogInformation("Operation result: {Message}", output.Message);
```

If the Operation completed successfully, the result is deserialized into the handle's result type. If the Operation
failed, a `NexusOperationFailedException` is thrown.

You can also recover a handle for an already-started Operation using `GetNexusOperationHandle<TResult>()` on the
Temporal Client:

```csharp
var handle = client.GetNexusOperationHandle<IHelloService.EchoOutput>("unique-operation-id");
var output = await handle.GetResultAsync();
```

Or use the Temporal CLI to wait for a result by Operation ID:

```bash
temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op
```

## List Standalone Nexus Operations 

Use [`ITemporalClient.ListNexusOperationsAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_ListNexusOperationsAsync_System_String_Temporalio_Client_NexusOperationListOptions_)
to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. The call returns an
`IAsyncEnumerable<NexusOperationExecution>` that you can iterate with `await foreach`.

Note that `ListNexusOperationsAsync` is called on the base `ITemporalClient`, not on the `NexusClient`.

```csharp
await foreach (var execution in client.ListNexusOperationsAsync(
    "Endpoint = 'my-nexus-endpoint'"))
{
    logger.LogInformation(
        "OperationID: {Id}, Operation: {Operation}, Status: {Status}",
        execution.OperationId, execution.Operation, execution.Status);
}
```

The query string accepts [List Filter](/list-filter) syntax. For example,
`"Endpoint = 'my-endpoint' AND ExecutionStatus = 'Running'"`.

Or use the Temporal CLI:

```bash
temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Count Standalone Nexus Operations 

Use [`ITemporalClient.CountNexusOperationsAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_CountNexusOperationsAsync_System_String_Temporalio_Client_NexusOperationCountOptions_)
to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query.

Note that `CountNexusOperationsAsync` is called on the base `ITemporalClient`, not on the `NexusClient`.

```csharp
var count = await client.CountNexusOperationsAsync(
    "Endpoint = 'my-nexus-endpoint'");
logger.LogInformation("Total Nexus operations: {Count}", count.Count);
```

Or use the Temporal CLI:

```bash
temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Run Standalone Nexus Operations with Temporal Cloud 

Standalone Nexus Operations work against Temporal Cloud with the same code — only the client connection options change.
For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint setup, certificate
generation, and authentication options, see
[Make Nexus calls across Namespaces in Temporal Cloud](/develop/dotnet/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud)
and [Connect to Temporal Cloud](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud).

---

# Platform - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/platform

> This section explains how to implement platform with the .NET SDK

![.NET SDK Banner](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Platform

- [Observability](/develop/dotnet/platform/observability)
- [Enriching the UI](/develop/dotnet/platform/enriching-ui)

---

# Enriching the user interface - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/platform/enriching-ui

> Add contextual information to Workflows and events in the Temporal UI using the .NET SDK.

Temporal supports adding context to Workflows and events with metadata. 
This helps users identify and understand Workflows and their operations.

## Adding summary and details to Workflows

### Starting a Workflow

When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI:

```csharp
using Temporalio.Client;

// Create client
var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

// Start a Workflow with static summary and details
var handle = await client.StartWorkflowAsync(
    (YourWorkflow wf) => wf.RunAsync("Workflow input"),
    new WorkflowOptions
    {
        Id = "your-workflow-id",
        TaskQueue = "your-task-queue",
        StaticSummary = "Order processing for customer #12345",
        StaticDetails = "Processing premium order with expedited shipping"
    });
```

`StaticSummary` is a single-line description that appears in the Workflow list view, limited to 200 bytes.
`StaticDetails` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes.

The input format is standard Markdown excluding images, HTML, and scripts.

You can also use the `ExecuteWorkflowAsync` method with the same parameters:

```csharp
var result = await client.ExecuteWorkflowAsync(
    (YourWorkflow wf) => wf.RunAsync("Workflow input"),
    new WorkflowOptions
    {
        Id = "your-workflow-id",
        TaskQueue = "your-task-queue",
        StaticSummary = "Order processing for customer #12345",
        StaticDetails = "Processing premium order with expedited shipping"
    });
```

### Inside the Workflow

Within a Workflow, you can get and set the _current Workflow details_. 
Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. 
Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines.

```csharp
using Temporalio.Workflows;

[Workflow]
public class YourWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string input)
    {
        // Get the current details
        var currentDetails = Workflow.CurrentDetails;
        Workflow.Logger.LogInformation($"Current details: {currentDetails}");
        
        // Set/update the current details
        Workflow.CurrentDetails = "Updated Workflow details with new status";
        
        return "Workflow completed";
    }
}
```

### Adding Summary to Activities and Timers

You can attach a metadata parameter `Summary` to Activities when starting them from within a Workflow:

```csharp
using Temporalio.Activities;
using Temporalio.Workflows;

[Workflow]
public class YourWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string input)
    {
        // Execute an activity with a summary
        var result = await Workflow.ExecuteActivityAsync(
            (YourActivities act) => act.YourActivityAsync(input),
            new ActivityOptions
            {
                StartToCloseTimeout = TimeSpan.FromSeconds(10),
                Summary = "Processing user data"
            });
        
        return result;
    }
}
```

Similarly, you can attach a `Summary` to timers within a Workflow:

```csharp
using Temporalio.Workflows;

[Workflow]
public class YourWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string input)
    {
        // Create a timer with a summary
        await Workflow.DelayWithOptionsAsync(new DelayOptions(TimeSpan.FromMinutes(5))
        {
            Summary = "Waiting for payment confirmation"
        });
        
        return "Timer completed";
    }
}
```

The input format for `Summary` is a string, and limited to 200 bytes.

## Viewing summary and details in the UI

Once you've added summaries and details to your Workflows, Activities, and timers, you can view this enriched information in the Temporal Web UI. 
Navigate to your Workflow's details page to see the metadata displayed in three key locations:

### User Metadata

On the Workflow Details page, under the **User Metadata** tab, you'll find the workflow-level metadata:

- **Summary & Details** - Displays the static summary and static details set when starting the Workflow
- **Current Details** - Displays the dynamic details that can be updated during Workflow execution

All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays.

### Timeline

The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance.

Labels longer than 120 characters are truncated with an ellipsis.

Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. 

Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later.

### Event History

Individual events in the Workflow's Event History display their associated summaries when available.

Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the Event details. 
When you do expand an Event, the summary is also prominently displayed in the detailed view.

---

# Observability - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/platform/observability

> Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility. Track Workflow Executions, set up Prometheus endpoints, customize metrics, configure tracing, and more.

This page covers features related to viewing the state of the application, including:

- [Metrics](#metrics)
- [Tracing](#tracing)
- [Logging](#logging)
- [Visibility](#visibility)

The observability feature guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application).
This includes the ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

## Emit metrics 

Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process.
For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics).

- For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide.
- For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics).
- For an end-to-end example that exposes metrics with the .NET SDK, refer to the [samples-dotnet](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry) repo.

Metrics in .NET are configured on the `Metrics` property of the `Telemetry` property on the `TemporalRuntime`. That object should be created globally and should be used for all clients; therefore, you should configure this before any other Temporal code.

### Set a Prometheus endpoint

The following example exposes a Prometheus endpoint on port `9000`.

```csharp
using Temporalio.Client;
using Temporalio.Runtime;

var runtime = new TemporalRuntime(new()
{
    Telemetry = new() { Metrics = new() { Prometheus = new("0.0.0.0:9000") } },
});
var client = await TemporalClient.ConnectAsync(new("localhost:7233") { Runtime = runtime });
```

### Set a custom metric meter

A custom metric meter can be set on the telemetry options to handle metrics programmatically.
The [Temporalio.Extensions.DiagnosticSource](https://github.com/temporalio/sdk-dotnet/tree/main/src/Temporalio.Extensions.DiagnosticSource) extension provides a custom metric meter implementation that sends all metrics to a [System.Diagnostics.Metrics.Meter](https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.metrics.meter) instance.

```csharp
using System.Diagnostics.Metrics;
using Temporalio.Client;
using Temporalio.Extensions.DiagnosticSource;
using Temporalio.Runtime;

// Create .NET meter
using var meter = new Meter("My.Meter");
// Can create MeterListener or OTel meter provider here...

// Create Temporal runtime with a custom metric meter for that meter
var runtime = new TemporalRuntime(new()
{
    Telemetry = new()
    {
        Metrics = new() { CustomMetricMeter = new CustomMetricMeter(meter) },
    },
});
var client = await TemporalClient.ConnectAsync(new("localhost:7233") { Runtime = runtime });
```

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Set [`GlobalTags`](https://dotnet.temporal.io/api/Temporalio.Runtime.MetricsOptions.html#Temporalio_Runtime_MetricsOptions_GlobalTags) on the [`Metrics` telemetry options](https://dotnet.temporal.io/api/Temporalio.Runtime.MetricsOptions.html) to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker.

```csharp
using Temporalio.Client;
using Temporalio.Runtime;

var runtime = new TemporalRuntime(new()
{
    Telemetry = new()
    {
        Metrics = new()
        {
            Prometheus = new("0.0.0.0:9000"),
            GlobalTags = new Dictionary<string, string>
            {
                ["team"] = "content-platform",
                ["service"] = "checkout",
                ["cost_center"] = "cc-1042",
                ["environment"] = "production",
            },
        },
    },
});
var client = await TemporalClient.ConnectAsync(new("localhost:7233") { Runtime = runtime });
```

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag           | Example            | Question it answers                                       |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team`        | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service`     | `checkout`         | Which application emits these metrics?                    |
| `cost_center` | `cc-1042`          | Which budget does this Worker fleet belong to?            |
| `environment` | `production`       | Is this production traffic, or staging or test?           |
| `region`      | `us-east-2`        | Where does the Worker fleet run?                          |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Set up tracing 

Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows.

To configure OpenTelemetry tracing in .NET, use the [Temporalio.Extensions.OpenTelemetry](https://github.com/temporalio/sdk-dotnet/tree/main/src/Temporalio.Extensions.OpenTelemetry) extension.

The [`Temporalio.Extensions.OpenTelemetry.TracingInterceptor`](https://dotnet.temporal.io/api/Temporalio.Extensions.OpenTelemetry.TracingInterceptor.html) class can be set as an interceptor in the client options, or provided through a [Plugin](/develop/plugins-guide#interceptors) if you're building a reusable library.

When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker.
Spans are created and serialized through the server to give one trace for a Workflow Execution.

## Log from a Workflow 

Logging enables you to record critical information during code execution.
Loggers create an audit trail and capture information about your Workflow's operation.
An appropriate logging level depends on your specific needs.
During development or troubleshooting, you might use debug or even trace.
In production, you might use info or warn to avoid excessive log volume.

Logging uses the .NET standard logging APIs. You can find the log levels supported in [their official documentation](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.loglevel?view=net-11.0-pp). The Temporal SDK core normally uses `WARN` as its default logging level.

The `LoggerFactory` can be set in the client.
The following example shows logging on the console and sets the level to `Information`.

```csharp
var client = await TemporalClient.ConnectAsync(new("localhost:7233")
{
    LoggerFactory = LoggerFactory.Create(builder =>
        builder.
            AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] ").
            SetMinimumLevel(LogLevel.Information)),
});
```

You can log from a Workflow using `Workflow.Logger` which is an instance of .NET's `ILogger`.

```csharp
Workflow.Logger.LogInformation("Given name: {Name}", name);
```

## Use Visibility APIs 

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### Use Search Attributes 

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service in the CLI or Web UI.
  - For example: `temporal operator search-attribute create --name CustomKeywordField --type Text`
    - Replace `CustomKeywordField` with the name of your Search Attribute.
    - Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList`
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `UpsertTypedSearchAttributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `Describe` on a `WorkflowHandle`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [In the Temporal CLI](/cli/command-reference/operator#list-2)
  - In code by calling `ListWorkflowsAsync`.

### List Workflow Executions 

Use the [ListWorkflowsAsync()](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_ListWorkflowsAsync_System_String_Temporalio_Client_WorkflowListOptions_) method on the Client and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows.
The result is an async enumerable.

```csharp
await foreach (var wf in client.ListWorkflowsAsync("WorkflowType='GreetingWorkflow'"))
{
    Console.WriteLine("Workflow: {0}", wf.Id);
}
```

### Set Custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

To set custom Search Attributes, use the `TypedSearchAttributes` property on `WorkflowOptions` for `StartWorkflowAsync` or `ExecuteWorkflowAsync`.
Typed search attributes are a `SearchAttributeCollection` created with a builder.

```csharp
// This only needs to be created once, so it is common to make it a static readonly even though we
// create inline here for demonstration
var myKeywordAttributeKey = SearchAttributeKey.CreateKeyword("MyKeywordAttribute");

// Start workflow with the search attribute collection
var handle = await client.StartWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue")
    {
        TypedSearchAttributes = new SearchAttributeCollection.Builder().
            Set(myKeywordAttributeKey, "SomeKeywordValue").
            ToSearchAttributeCollection(),
    });
```

### Upsert Search Attributes 

You can upsert Search Attributes to add, update, or remove Search Attributes from within Workflow code.

To upsert custom Search Attributes, use the [`UpsertTypedSearchAttributes()`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_UpsertTypedSearchAttributes_Temporalio_Workflows_SearchAttributeUpdate___) method with a set of updates.
Keys can be predefined for reuse.

```csharp
// These only need to be created once, so it is common to make them static readonly even though we
// create inline here for demonstration
var myKeywordAttributeKey = SearchAttributeKey.CreateKeyword("MyKeywordAttribute");
var myTextAttributeKey = SearchAttributeKey.CreateText("MyTextAttribute");

// Add/Update the keyword one and remove the text one
Workflow.UpsertTypedSearchAttributes(
    myKeywordAttributeKey.ValueSet("SomeKeywordValue"),
    myTextAttributeKey.ValueUnset());
```

---

# Set up a local development environment with the .NET SDK

Source: https://docs.temporal.io/develop/dotnet/set-up-your-local-dotnet

> Configure your local development environment to get started developing with Temporal

# Quickstart

Configure your local development environment to get started developing with Temporal.

## Install .NET

    The .NET SDK requires .NET 6.0 or later.
    Install the latest version of .NET by following the [official .NET instructions](https://dotnet.microsoft.com/download).

## Install the Temporal .NET SDK

    Create a solution and the three projects used in this guide: `Workflow` (class library), `Worker` (console), and `Client` (console). Add them to the solution.

    Tip: You can also centralize the `Temporalio` package for all projects using `Directory.Packages.props` and `Directory.Build.props` at the solution root.

```bash
# Create solution and projects
mkdir TemporalioHelloWorld
cd TemporalioHelloWorld

dotnet new sln -n TemporalioHelloWorld

dotnet new classlib -o Workflow
dotnet new console -o Worker
dotnet new console -o Client

# Add projects to the solution
dotnet sln TemporalioHelloWorld.sln add Workflow/Workflow.csproj Worker/Worker.csproj Client/Client.csproj

# Add project references
dotnet add Worker/Worker.csproj reference Workflow/Workflow.csproj
dotnet add Client/Client.csproj reference Workflow/Workflow.csproj

# Install Temporal SDK in each project
dotnet add Workflow/Workflow.csproj package Temporalio
dotnet add Worker/Worker.csproj package Temporalio
dotnet add Client/Client.csproj package Temporalio
```

Build the solution:

```bash
dotnet build
```

## Install Temporal CLI and start the development server

    The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli).

    Choose your operating system to install Temporal CLI:

**macOS**

Install the Temporal CLI using Homebrew:
```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH, for example:
```bash
sudo mv temporal /usr/local/bin
```

## Start the development server

    Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

    This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database.

    The Temporal Service will be available on localhost:7233.
    The Temporal Web UI will be available at http://localhost:8233.

    Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C.

    Once you have everything installed, you're ready to build apps with Temporal on your local machine.

After installing, open a new Terminal window and start the development server:
```bash
temporal server start-dev
```

#### Change the Web UI port

The Temporal Web UI may be on a different port in some examples or tutorials.
To change the `--ui-port` option when starting the server:

```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity.

This test will confirm that:

- Your .NET SDK installation is working
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

#### Tip: Example Directory Structure

```text
TemporalioHelloWorld/
├── Client/
│   ├── Client.csproj
│   └── Program.cs              # Starts a workflow
├── Worker/
│   ├── Worker.csproj
│   └── Program.cs              # Runs a worker
├── Workflow/
│   ├── Workflow.csproj
│   ├── MyActivities.cs         # Activity definition
│   └── SayHelloWorkflow.cs     # Workflow definition
└── TemporalioHelloWorld.sln
```

### 1. Create the Activity and Workflow 

#### Create an Activity file (MyActivities.cs) in the Workflow project:

```csharp
namespace MyNamespace;

using Temporalio.Activities;

public class MyActivities
{
    // Activities can be async and/or static too! We just demonstrate instance
    // methods since many will use them that way.
    [Activity]
    public string SayHello(string name) => $"Hello, {name}!";
}
```

An Activity is a normal function or method that executes a single, well-defined action (either short or long running), which often involve interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling an API, which are prone to failure.
If an Activity fails, Temporal automatically retries it based on your configuration.

#### Create a Workflow file (SayHelloWorkflow.cs) in the Workflow project: 

```csharp
namespace MyNamespace;

using Temporalio.Workflows;

[Workflow]
public class SayHelloWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        // This workflow just runs a simple activity to completion.
        // StartActivityAsync could be used to just start and there are many
        // other things that you can do inside a workflow.
        return await Workflow.ExecuteActivityAsync(
            // This is a lambda expression where the instance is typed. If this
            // were static, you wouldn't need a parameter.
            (MyActivities act) => act.SayHello(name),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }
        );
    }
}
```

Workflows orchestrate Activities and contain the application logic.
Temporal Workflows are resilient.
They can run and keep running for years, even if the underlying infrastructure fails.
If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off.

### 2. Create the Worker 

With your Activity and Workflow defined, you need a Worker to execute them.

#### Create a Worker file (Program.cs) in the Worker project:

```csharp
using MyNamespace;
using Temporalio.Client;
using Temporalio.Worker;

// Create a client to localhost on "default" namespace
var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

// Cancellation token to shutdown worker on ctrl+c
using var tokenSource = new CancellationTokenSource();
Console.CancelKeyPress += (_, eventArgs) =>
{
    tokenSource.Cancel();
    eventArgs.Cancel = true;
};

// Create an activity instance since we have instance activities. If we had
// all static activities, we could just reference those directly.
var activities = new MyActivities();

// Create worker with the activity and workflow registered
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
        .AddActivity(activities.SayHello)
        .AddWorkflow<SayHelloWorkflow>()
);

// Run worker until cancelled
Console.WriteLine("Running worker");
try
{
    await worker.ExecuteAsync(tokenSource.Token);
}
catch (OperationCanceledException)
{
    Console.WriteLine("Worker cancelled");
}
```

Run the Worker:

```bash
dotnet run --project Worker/Worker.csproj
```
Keep this terminal running - you should see `Running worker` displayed.

A Worker polls a Task Queue, that you configure it to poll, looking for work to do. 
Once the Worker dequeues the Workflow or Activity task from the Task Queue, it then executes that task.

Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities.
For more information on Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers).

### 3. Execute the Workflow 

Now that your Worker is running, it's time to start a Workflow Execution.
This final step will validate that everything is working correctly. 

#### Create a Client file (Program.cs) in the Client project: 

```csharp
using MyNamespace;
using Temporalio.Client;

// Create a client to localhost on "default" namespace
var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

// Run workflow
var result = await client.ExecuteWorkflowAsync(
    (SayHelloWorkflow wf) => wf.RunAsync("Temporal"),
    new(id: $"my-workflow-id-{Guid.NewGuid()}", taskQueue: "my-task-queue")
);

Console.WriteLine("Workflow result: {0}", result);
```

While the Worker is still running, run the Workflow:

```bash
dotnet run --project Client/Client.csproj
```

### Verify success

If everything is working correctly, you should see:

- Worker processing the workflow and activity
- Output: `Workflow result: Hello Temporal`
- Workflow Execution details in the [Temporal Web UI](http://localhost:8233)

- [Run your first Temporal Application](https://learn.temporal.io/getting_started/dotnet/first_program_in_dotnet/): Create a basic Workflow and run it with the Temporal .NET SDK

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Workers - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workers

> This section explains how to implement Workers with the .NET SDK

![.NET SDK Banner](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Workers

- [Run a Worker](/develop/dotnet/workers/run-worker-process)
- [Interceptors](/develop/dotnet/workers/interceptors)
- [Serverless Workers](/develop/dotnet/workers/serverless-workers)

---

# Interceptors - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workers/interceptors

Interceptors are SDK hooks that let you intercept inbound and outbound Temporal calls. You use them to apply shared
behavior across many calls, such as tracing and authorization, before calls reach the application code and after they return.
This is similar to middleware in other frameworks, like [ASP.NET Core](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/middleware).

There are two main types of interceptors: inbound and outbound.

* Outbound interceptors wrap network calls, running before they reach the network and after they return.
* Inbound interceptors run after the network hop, wrapping application code and running before it starts and after it returns.

Concretely, there are five categories of inbound and outbound calls that you can modify in this way:

| | [Outbound Client](https://dotnet.temporal.io/api/Temporalio.Client.Interceptors.ClientOutboundInterceptor.html) | [Inbound Workflow](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.WorkflowInboundInterceptor.html) | [Outbound Workflow](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.WorkflowOutboundInterceptor.html) | [Inbound Activity](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.ActivityInboundInterceptor.html) | [Outbound Activity](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.ActivityOutboundInterceptor.html) |
| --- | --- | --- | --- | --- | --- |
| **Description** | Wraps calls from your application to the Temporal Client to start a Workflow or send [Messages](/encyclopedia/workflow-message-passing) to it | Wraps calls arriving into a [Workflow Execution](/workflow-execution), such as executing the Workflow, handling [Messages](/encyclopedia/workflow-message-passing) | Wraps calls a [Workflow](/workflow-definition) makes to the SDK, such as scheduling [Activities](/activities), starting [Child Workflows](/child-workflows), and invoking [Nexus Operations](/nexus) | Wraps calls arriving into an [Activity Execution](/activity-execution) | Wraps calls an [Activity](/activities) makes to the SDK, such as sending [Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) and reading Activity info |
| **Runs on** | Client | Worker (Workflow sandbox) | Worker (Workflow sandbox) | Worker (Activity context) | Worker (Activity context) |
| **Example methods** | `StartWorkflowAsync()`, `WorkflowHandle.SignalAsync()`, `ListWorkflowsAsync()` | `ExecuteWorkflowAsync()`, `WorkflowHandle.QueryAsync()`, `WorkflowHandle.SignalAsync ()`, `WorkflowHandle.ExecuteUpdateAsync()` | `StartActivityAsync()`, `StartChildWorkflowAsync()`, `ChildWorkflowHandle.SignalAsync()`, `StartNexusOperationAsync()` | `ExecuteActivityAsync()` | `Info()`, `Heartbeat()` |

> **⚠️ Warning:**
> Workflow interceptors and replay
>
> Workflow inbound and outbound interceptor methods also execute during [replay](/develop/dotnet/best-practices/testing-suite#replay). Use replay-safe APIs for logging, randomness, and time in these interceptors.
> See [Develop Workflow logic](/develop/dotnet/workflows/basics#workflow-logic-requirements) for details.
>
> If you want to write generic code shared by all inbound Workflow call handlers but want to skip read-only operations, check [`Workflow.Unsafe.IsReplaying`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.Unsafe.html#Temporalio_Workflows_Workflow_Unsafe_IsReplaying).
>
> Activity and Client interceptors are not affected by replay.
>

## Register an Interceptor 

Registering an interceptor means supplying an interceptor instance to the SDK so Temporal can invoke it when matching
Client or Worker calls occur. Once registered, the interceptor runs as part of the call path and can observe or modify
request and response data.

### Register on the Client 

Pass interceptors in the `Interceptors` property of [`TemporalClientConnectOptions`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClientConnectOptions.html). Client interceptors modify outbound calls such
as starting and signaling Workflows. One example is [setting up tracing](/develop/dotnet/platform/observability) to see your call graph of a Workflow.

```csharp
using Temporalio.Extensions.OpenTelemetry;

var interceptor = new TracingInterceptor();

var client = await TemporalClient.ConnectAsync(new()
{
    TargetHost = "localhost:7233",
    Interceptors = [interceptor],
});
```

The `Interceptors` list can contain multiple interceptors.
The default behavior for interceptors is to form a chain. A method implemented on an interceptor instance in the list can perform side effects, and modify the data, before passing it on to the corresponding method on the next interceptor in the list.

### Register via a Plugin

If you're building a reusable library or want to bundle interceptors with other primitives, you can register them through a [Plugin](/develop/plugins-guide#interceptors).

### Register on the Worker only 

If your interceptor doesn't affect the Client, you can pass interceptors in the `Interceptors` argument of `TemporalWorkerOptions`.
Worker interceptors modify inbound and outbound Workflow and Activity calls.

```csharp
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
    {
        Interceptors = [interceptor]
    }
    .AddActivity(activities.SayHello)
    .AddWorkflow<SayHelloWorkflow>()
);
```

## How to implement Interceptors

Interceptors run as a chain.  Each interceptor wraps the entire inner call: your code runs before the call, invokes `next` to execute the rest of the chain, and then runs after the call completes. This means you can inspect or modify both the `input` and the result, handle errors, and perform side effects at either stage.

### Implementing Client call Interceptors

To modify outbound Client calls, define a class implementing [`IClientInterceptor`](https://dotnet.temporal.io/api/Temporalio.Client.Interceptors.IClientInterceptor.html). Implement `InterceptClient()` to return a [`ClientOutboundInterceptor`](https://dotnet.temporal.io/api/Temporalio.Client.Interceptors.ClientOutboundInterceptor.html), overriding the outbound Client calls you want to modify. `IClientInterceptor.InterceptClient` receives the next `ClientOutboundInterceptor` in the chain and returns the created interceptor.

This example implements an Interceptor on outbound Client calls that sets a certain key in the outbound `headers` field.
A User ID is context-propagated by being sent in a header field with outbound requests:

```csharp
using Google.Protobuf;
using Temporalio.Api.Common.V1;
using Temporalio.Client;
using Temporalio.Client.Interceptors;

public static class UserContext
{
    private static readonly AsyncLocal<string?> CurrentUser = new();

    public static string? UserId
    {
        get => CurrentUser.Value;
        set => CurrentUser.Value = value;
    }
}

public class ContextPropagationInterceptor : IClientInterceptor
{
    public ClientOutboundInterceptor InterceptClient(
        ClientOutboundInterceptor nextInterceptor) =>
        new ContextPropagationClientOutboundInterceptor(nextInterceptor);
}

public class ContextPropagationClientOutboundInterceptor(
    ClientOutboundInterceptor next)
    : ClientOutboundInterceptor(next)
{
    public override Task<WorkflowHandle<TWorkflow, TResult>>
        StartWorkflowAsync<TWorkflow, TResult>(StartWorkflowInput input)
    {
        var headers = input.Headers ?? new Dictionary<string, Payload>();
        headers["user-id"] = new Payload
        {
            Metadata = { ["encoding"] = ByteString.CopyFromUtf8("plain/text") },
            Data = ByteString.CopyFromUtf8(UserContext.UserId),
        };

        return base.StartWorkflowAsync<TWorkflow, TResult>(input with { Headers = headers });
    }
}
```

You can then [register](#register) this interceptor in your client/starter code.

Your interceptor classes don't need to implement every method. The default implementation is always to pass the data on to the next method in the interceptor chain.
During execution, when the SDK encounters an Inbound Activity call, it will look to the first Interceptor instance, get hold of the appropriate intercepted method, and call it.
The intercepted method will perform its function then call the same method on the next Interceptor in the chain.
At the end of the chain the SDK will call the "real" SDK method.

### Implementing Worker call Interceptors

To modify inbound Workflow and Activity calls, define a class implementing [`IWorkerInterceptor`](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.IWorkerInterceptor.html). It provides `InterceptActivity()`, `InterceptWorkflow()`, and `InterceptNexusOperation()` methods for Activity, Workflow, and Nexus interception.

This example demonstrates using an interceptor to measure [Schedule-To-Start](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) and Schedule-To-Close latency.
Notice how the interceptor wraps the call. It records Schedule-To-Start before `ExecuteActivityAsync`, then records Schedule-To-Close after it completes:

```csharp
using Temporalio.Activities;
using Temporalio.Worker;
using Temporalio.Worker.Interceptors;

public class SimpleWorkerInterceptor : IWorkerInterceptor
{
    public ActivityInboundInterceptor InterceptActivity(
        ActivityInboundInterceptor nextInterceptor) =>
        new ActivityMetricsInterceptor(nextInterceptor);

    public WorkflowInboundInterceptor InterceptWorkflow(
        WorkflowInboundInterceptor nextInterceptor) =>
        nextInterceptor;

    public NexusOperationInboundInterceptor InterceptNexusOperation(
        NexusOperationInboundInterceptor nextInterceptor) =>
        nextInterceptor;
}

public class ActivityMetricsInterceptor(ActivityInboundInterceptor next)
    : ActivityInboundInterceptor(next)
{
    public override async Task<object?> ExecuteActivityAsync(
        ExecuteActivityInput input)
    {
        var info = ActivityExecutionContext.Current.Info;
        var started = DateTimeOffset.UtcNow;

        // Before the activity executes
        var scheduleToStart =
            started - info.CurrentAttemptScheduledTime;

        Console.WriteLine(
            $"Schedule-To-Start latency: {scheduleToStart}");

        // Execute the activity
        var result = await base.ExecuteActivityAsync(input);

        // After the activity completes
        var scheduleToClose =
            DateTimeOffset.UtcNow - info.CurrentAttemptScheduledTime;

        Console.WriteLine(
            $"Schedule-To-Close latency: {scheduleToClose}");

        return result;
    }
}
```

Register it on the Worker:

```csharp
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
    {
        Interceptors = new IWorkerInterceptor[]
        {
            new SimpleWorkerInterceptor(),
        },
    }
    .AddActivity(activities.SayHello)
    .AddWorkflow<SayHelloWorkflow>());

await worker.ExecuteAsync();
```

---

# Run a Worker - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workers/run-worker-process

> Create and run a Temporal Worker using the .NET SDK.

This page covers long-lived Workers that you host and run as persistent processes.
For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/dotnet/workers/serverless-workers).

## Create and run a Worker 

Create a `TemporalWorker` with a [Temporal Client](/develop/dotnet/client/temporal-client) and a `TemporalWorkerOptions` that names the Task Queue to poll.
Add the Workflows and Activities the Worker can execute, then call `ExecuteAsync()` to start polling.

<!--SNIPSTART dotnet-create-worker-->
[features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs)
```cs
var options = new TemporalWorkerOptions("my-task-queue");
options.AddWorkflow<GreetingWorkflow>();
options.AddAllActivities(typeof(GreetingActivities), null);

using var worker = new TemporalWorker(client, options);
await worker.ExecuteAsync(CancellationToken.None);
```
<!--SNIPEND-->

`ExecuteAsync()` takes a `CancellationToken` and polls until that token is cancelled.
The snippet passes `CancellationToken.None`, which never cancels, so the Worker runs until the process exits.
To stop a Worker on demand, pass a token you control instead. See [Shut down a Worker](#shut-down-a-worker).

`TemporalWorker` implements `IDisposable`, so declare it with `using` to release its resources when the process exits.

## Register Workflows and Activities 

All Workers polling the same Task Queue must register the same Workflow Types and Activity Types.
A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue.
A Worker that receives a Task for a type it did not register fails that Task.

Add Workflows with `AddWorkflow<T>()` and Activities with `AddActivity()` or `AddAllActivities()`:

```csharp
var options = new TemporalWorkerOptions("my-task-queue");
options.AddWorkflow<GreetingWorkflow>();
options.AddWorkflow<OrderWorkflow>();
options.AddAllActivities(new MyActivities(databaseClient));
```

`AddAllActivities()` registers every method marked with `[Activity]`. Pass an instance to register instance methods, which lets Activities share state such as a database client. For a class of static Activity methods, pass the type and `null` instead.

## Connect to Temporal Cloud 

To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials.
See [Connect to Temporal Cloud](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud) for setup instructions.

## Configure Worker options 

`TemporalWorkerOptions` controls concurrency limits, pollers, timeouts, and caching, including `MaxConcurrentActivities`, `MaxConcurrentWorkflowTasks`, and `MaxCachedWorkflows`.
The defaults work for most cases.

To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference).

## Run a versioned Worker 

Set a Worker Deployment Version and enable versioning in `DeploymentOptions`, then set a versioning behavior on each Workflow.

<!--SNIPSTART dotnet-versioned-worker-->
[features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs)
```cs
var options = new TemporalWorkerOptions("my-task-queue")
{
    DeploymentOptions = new WorkerDeploymentOptions(
        new WorkerDeploymentVersion("my-app", "1.0"),
        useWorkerVersioning: true),
};
options.AddWorkflow<VersionedGreetingWorkflow>();
options.AddAllActivities(typeof(GreetingActivities), null);

using var worker = new TemporalWorker(client, options);
```
<!--SNIPEND-->

Set the behavior per Workflow with `[Workflow(VersioningBehavior = VersioningBehavior.Pinned)]`, or set a default for the whole Worker with `DefaultVersioningBehavior` on `WorkerDeploymentOptions`.
`VersioningBehavior` comes from the `Temporalio.Common` namespace.

A versioning behavior applies only to a Worker that has versioning enabled.
If a Workflow declares one and its Worker does not enable versioning, the server rejects the Workflow Task and the Task retries instead of failing outright.

See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out.

## Shut down a Worker 

To stop a Worker on demand, start it with a token you can cancel yourself instead of `CancellationToken.None`.
Create a `CancellationTokenSource`, pass its `Token` to `ExecuteAsync()`, then cancel the source when the Worker should stop, such as from a `Console.CancelKeyPress` handler.
The Worker stops polling for new Tasks and waits for in-flight Tasks to finish, up to `GracefulShutdownTimeout`.

<!--SNIPSTART dotnet-worker-graceful-shutdown-->
[features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs)
```cs
using var tokenSource = new CancellationTokenSource();
Console.CancelKeyPress += (_, eventArgs) =>
{
    tokenSource.Cancel();
    eventArgs.Cancel = true;
};

var options = new TemporalWorkerOptions("my-task-queue")
{
    GracefulShutdownTimeout = TimeSpan.FromSeconds(30),
};
options.AddWorkflow<GreetingWorkflow>();

using var worker = new TemporalWorker(client, options);
await worker.ExecuteAsync(tokenSource.Token);
```
<!--SNIPEND-->

`ExecuteAsync()` throws `OperationCanceledException` once the Worker stops, so catch it where you want the process to exit cleanly.
See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities.

---

# Serverless Workers - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workers/serverless-workers

> Write Temporal Workers that run on serverless compute using the .NET SDK.

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview.

Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes.
Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done.

For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers).
For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Supported providers

- [**AWS Lambda**](/develop/dotnet/workers/serverless-workers/aws-lambda) - Use the `Temporalio.Extensions.Aws.Lambda` NuGet package to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, observability, and the invocation lifecycle.
- [**GCP Cloud Run**](/develop/dotnet/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, container packaging, and handling scale-in.

---

# Serverless Workers on AWS Lambda - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workers/serverless-workers/aws-lambda

> Write a Temporal Worker that runs on AWS Lambda using the .NET SDK Temporalio.Extensions.Aws.Lambda package.

> **Public Preview**

The `Temporalio.Extensions.Aws.Lambda` NuGet package lets you run a Temporal Serverless Worker on AWS Lambda.
Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive.
Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline.
You register Workflows and Activities the same way you would with a standard Worker.

For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda).

## Create and run a Worker in Lambda 

Add the `Temporalio.Extensions.Aws.Lambda` NuGet package:

```bash
dotnet add package Temporalio.Extensions.Aws.Lambda
```

Use `TemporalLambdaWorker.CreateHandler` to create a Lambda handler that runs a Temporal Worker.
Pass a `WorkerDeploymentVersion` and a configure callback that registers your Workflows and Activities.
Assign the result to a static field so the handler is created once during Lambda cold start and reused across invocations.

```csharp {10-17}
namespace MyApp;

using Amazon.Lambda.Core;
using Temporalio.Common;
using Temporalio.Extensions.Aws.Lambda;

public class LambdaFunction
{
    private static readonly Func<object?, ILambdaContext, Task> WorkerHandler =
        TemporalLambdaWorker.CreateHandler(
            new WorkerDeploymentVersion("my-app", "build-1"),
            config =>
            {
                config.WorkerOptions.TaskQueue = "my-task-queue";
                config.WorkerOptions.AddWorkflow<SampleWorkflow>();
                config.WorkerOptions.AddActivity(Activities.HelloActivity);
            });

    public Task HandlerAsync(Stream input, ILambdaContext context) =>
        WorkerHandler(input, context);
}
```

For a complete project, see the [Lambda Worker sample](https://github.com/temporalio/samples-dotnet/tree/main/src/LambdaWorker).

The `WorkerDeploymentVersion` is required.
Worker Deployment Versioning is always enabled for Serverless Workers.
Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or `Pinned`.
Set it per-Workflow with the `[Workflow]` attribute, or set a worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`.
The default versioning behavior is `AutoUpgrade`.

[src/LambdaWorker/Worker/SampleWorkflow.workflow.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/LambdaWorker/Worker/SampleWorkflow.workflow.cs)
```csharp {7}
namespace TemporalioSamples.LambdaWorker.Worker;

using Microsoft.Extensions.Logging;
using Temporalio.Common;
using Temporalio.Workflows;

[Workflow(VersioningBehavior = VersioningBehavior.Pinned)]
public class SampleWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        Workflow.Logger.LogInformation("SampleWorkflow started with name: {Name}", name);
        var result = await Workflow.ExecuteActivityAsync(
            () => Activities.HelloActivity(name),
            new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) });
        Workflow.Logger.LogInformation("SampleWorkflow completed with result: {Result}", result);
        return result;
    }
}
```

## Configure the Temporal connection 

The `Temporalio.Extensions.Aws.Lambda` package automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details.

Compared with long-lived Workers, the location of the config file is resolved differently, in the following order:

1. `TEMPORAL_CONFIG_FILE` environment variable, if set.
2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`).
3. `temporal.toml` in the current working directory.

The file is optional. If absent, only environment variables are used.

Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options.

### TLS/CA loading on Lambda 

Some AWS Lambda .NET images override the `SSL_CERT_FILE` environment variable in a way that prevents the SDK's Rust-based runtime from loading system root CAs.
If you encounter TLS certificate errors on Lambda, see the [AWS Lambda .NET CA loading workaround](https://github.com/temporalio/sdk-dotnet/blob/main/src/Temporalio/README.md#aws-lambda-net-8-ca-loading-issues) in the SDK README.

## Adjust Worker defaults for Lambda 

The `Temporalio.Extensions.Aws.Lambda` package applies conservative defaults suited to short-lived Lambda invocations.
These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment.

| Setting | Lambda default |
|---|---|
| `MaxConcurrentActivities` | 2 |
| `MaxConcurrentWorkflowTasks` | 10 |
| `MaxConcurrentLocalActivities` | 2 |
| `MaxConcurrentNexusTasks` | 5 |
| `MaxConcurrentWorkflowTaskPolls` | 2 |
| `MaxConcurrentActivityTaskPolls` | 1 |
| `MaxConcurrentNexusTaskPolls` | 1 |
| `MaxCachedWorkflows` | 30 |
| `GracefulShutdownTimeout` | 5 seconds |
| `DisableEagerActivityExecution` | Always `true` |
| `ShutdownDeadlineBuffer` | 7 seconds |

`DisableEagerActivityExecution` is always `true` and cannot be overridden.
Eager Activities require a persistent connection, which Lambda invocations don't maintain.

`ShutdownDeadlineBuffer` is specific to the `Temporalio.Extensions.Aws.Lambda` package.
It controls the time reserved after the worker run budget for worker shutdown and hooks.
The default is 7 seconds.

If your Worker handles long-running Activities, increase `GracefulShutdownTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together.
For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities).

## Add observability with OpenTelemetry 

The `Temporalio.Extensions.Aws.Lambda.OpenTelemetry` NuGet package provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer.
With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions.
The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch.

The underlying metrics and traces are the same ones the .NET SDK emits in any environment.
For general observability concepts and the full list of available metrics, see the [SDK metrics reference](/references/sdk-metrics).

Add the OpenTelemetry extension package:

```bash
dotnet add package Temporalio.Extensions.Aws.Lambda.OpenTelemetry
```

Call `ApplyOpenTelemetryDefaults` on the options in the configure callback:

[src/LambdaWorker/Worker/Function.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/LambdaWorker/Worker/Function.cs)
```csharp {17}
namespace TemporalioSamples.LambdaWorker.Worker;

using Amazon.Lambda.Core;
using Temporalio.Common;
using Temporalio.Extensions.Aws.Lambda;
using Temporalio.Extensions.Aws.Lambda.OpenTelemetry;

public class LambdaFunction
{
    private static readonly Func<object?, ILambdaContext, Task> WorkerHandler =
        TemporalLambdaWorker.CreateHandler(
            new WorkerDeploymentVersion(
                LambdaWorkerSample.DeploymentName,
                LambdaWorkerSample.BuildId),
            config =>
            {
                config.ApplyOpenTelemetryDefaults();
                LambdaWorkerSample.ConfigureWorkerOptions(config.WorkerOptions);
            });

    public Task HandlerAsync(Stream input, ILambdaContext context) =>
        WorkerHandler(input, context);
}
```

`ApplyOpenTelemetryDefaults` configures Temporal tracing with `TracingInterceptor`, creates an OTLP trace exporter and tracer provider, configures Core SDK OTLP metrics, uses AWS X-Ray-compatible trace IDs, and registers a per-invocation shutdown hook that force-flushes traces.
By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint.
The endpoint can be overridden with the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable.

You can customize the defaults by passing a `LambdaWorkerOpenTelemetryOptions` object:

```csharp
config.ApplyOpenTelemetryDefaults(
    new LambdaWorkerOpenTelemetryOptions
    {
        CollectorEndpoint = "http://localhost:4317",
        ServiceName = "my-worker",
        MetricsExportInterval = TimeSpan.FromSeconds(5),
    });
```

Core SDK metrics export every 10 seconds by default. Set `MetricsExportInterval` shorter than your Lambda timeout to increase the chance that at least one metrics export happens during each invocation.

To collect this telemetry, attach the [ADOT Collector layer](https://aws-otel.github.io/docs/getting-started/lambda) to your Lambda function.
.NET does not need a language-specific ADOT layer because the OTel SDK is included as a dependency of the package.

The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline.
You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines.
Bundle the following `otel-collector-config.yaml` in your Lambda deployment package:

[src/LambdaWorker/otel-collector-config.template.yaml](https://github.com/temporalio/samples-dotnet/blob/main/src/LambdaWorker/otel-collector-config.template.yaml)
```yaml
receivers:
    otlp:
        protocols:
            grpc:
                endpoint: "localhost:4317"
            http:
                endpoint: "localhost:4318"

exporters:
    awsxray:
        region: ${env:AWS_REGION}
    awsemf:
        namespace: TemporalWorkerMetrics
        log_group_name: /aws/lambda/${env:AWS_LAMBDA_FUNCTION_NAME}
        region: ${env:AWS_REGION}
        dimension_rollup_option: NoDimensionRollup
        resource_to_telemetry_conversion:
            enabled: true

service:
    pipelines:
        traces:
            receivers: [otlp]
            exporters: [awsxray]
        metrics:
            receivers: [otlp]
            exporters: [awsemf]
    telemetry:
        logs:
            level: info
        metrics:
            address: localhost:8888
```

Set the following environment variable on the Lambda function to point the Collector at the bundled config:

- `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml`

Enable X-Ray active tracing on the Lambda function:

```bash
aws lambda update-function-configuration \
  --function-name <your-function-name> \
  --tracing-config Mode=Active
```

The Lambda execution role must have permissions to write to X-Ray and CloudWatch.
Add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions to the execution role.
Without these permissions, the Collector fails silently and no telemetry appears.

You can also configure tracing and metrics manually using `TracingInterceptor` and `TemporalRuntime`:

```csharp
using Temporalio.Extensions.OpenTelemetry;

config.ClientOptions.Interceptors = new[] { new TracingInterceptor() };
config.ClientOptions.Runtime = new TemporalRuntime(new TemporalRuntimeOptions
{
    Telemetry = new TelemetryOptions
    {
        Metrics = new MetricsOptions(new OpenTelemetryOptions("http://collector:4317")),
    },
});
```

---

# Serverless Workers on GCP Cloud Run - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workers/serverless-workers/cloud-run

> Run a Temporal Worker on a GCP Cloud Run worker pool using the .NET SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other .NET Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).

## Create a versioned Worker 

Build the Worker as you would any long-running .NET Worker, then set `DeploymentOptions` on `TemporalWorkerOptions` to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

```csharp
using Temporalio.Client;
using Temporalio.Common;
using Temporalio.Worker;

var client = await TemporalClient.ConnectAsync(
    new(Environment.GetEnvironmentVariable("TEMPORAL_ADDRESS")!)
    {
        Namespace = Environment.GetEnvironmentVariable("TEMPORAL_NAMESPACE")!,
        ApiKey = Environment.GetEnvironmentVariable("TEMPORAL_API_KEY"),
        Tls = new(),
    });

var options = new TemporalWorkerOptions(
    Environment.GetEnvironmentVariable("TEMPORAL_TASK_QUEUE")!)
{
    DeploymentOptions = new(new("my-app", "build-1"), useWorkerVersioning: true)
    {
        DefaultVersioningBehavior = VersioningBehavior.Pinned,
    },
};
options.AddWorkflow<GreetingWorkflow>();
options.AddAllActivities(typeof(GreetingActivities), null);

using var worker = new TemporalWorker(client, options);
await worker.ExecuteAsync(CancellationToken.None);
```

The two arguments to `WorkerDeploymentVersion` are the deployment name and the build ID, and together they identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage.

Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `Pinned` or `AutoUpgrade`.
Setting `DefaultVersioningBehavior` as shown above covers every Workflow on the Worker.
To set the behavior per Workflow instead, set `VersioningBehavior` on the `Workflow` attribute:

```csharp
using Temporalio.Common;
using Temporalio.Workflows;

[Workflow(VersioningBehavior = VersioningBehavior.Pinned)]
public class GreetingWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name) => // ...
}
```

For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/dotnet/workers/run-worker-process).

## Configure the Temporal connection 

Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext.
The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace.

To load those values through the shared configuration format instead of reading them yourself, use `ClientEnvConfig.LoadClientConnectOptions()` from the `Temporalio.Common.EnvConfig` namespace.
For the full list of supported variables, the config file format, and profiles, see [Environment configuration](/develop/environment-configuration).

## Package the Worker image 

Publish the Worker and run it on a .NET runtime image:

```dockerfile
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build

WORKDIR /src
COPY *.csproj ./
RUN dotnet restore
COPY . .
RUN dotnet publish -c Release -o /out

FROM mcr.microsoft.com/dotnet/runtime:10.0

WORKDIR /app
COPY --from=build /out ./
CMD ["dotnet", "MyWorker.dll"]
```

The Worker runs on a Rust core that reads TLS roots from the operating system's certificate store, so the runtime image must include one.
The Debian-based `mcr.microsoft.com/dotnet/runtime` images do.

## Keep Activities safe across scale-in 

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing.
An instance running a long Activity can be stopped mid-execution.

Use [Activity Heartbeats](/develop/dotnet/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over:

```csharp
[Activity]
public static string Process(IReadOnlyList<string> items)
{
    for (var i = 0; i < items.Count; i++)
    {
        ActivityExecutionContext.Current.Heartbeat(i);
        // ... process items[i]
    }
    return "done";
}
```

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Configure OpenTelemetry 

Run an OpenTelemetry Collector as a sidecar in the Worker Pool. Call `ApplyGoogleCloudRunOpenTelemetryDefaults()`
to configure tracing and Temporal Core metrics for OTLP export to the Collector at `localhost:4317`. Load the Client
connection settings with `ClientEnvConfig.LoadClientConnectOptions()`, then configure both plugins before connecting:

<!--SNIPSTART dotnet-cloud-run-->
[src/Gcp/CloudRun/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/Gcp/CloudRun/Program.cs)
```cs
// The Cloud Run Id plugin sets the client identity to "{instanceId}@{revision}" from Cloud Run
// metadata at connect time; every Worker created from the client inherits it.
connectOptions.Plugins = new ITemporalClientPlugin[] { new CloudRunIdPlugin() };

// ApplyGoogleCloudRunOpenTelemetryDefaults adds the tracing interceptor and a runtime exporting Core
// metrics + traces over OTLP to the collector sidecar; the returned handle owns the tracer provider.
// Both plugins configure the same connect options.
using var telemetry = connectOptions.ApplyGoogleCloudRunOpenTelemetryDefaults();

var client = await TemporalClient.ConnectAsync(connectOptions);
```
<!--SNIPEND-->

After the Worker stops, call `telemetry.FlushAsync(TimeSpan.FromSeconds(2))` to flush traces within the Cloud Run
termination window. For the Collector sidecar configuration and deployment steps, see the
[.NET Cloud Run sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Gcp/CloudRun).

## Set a Worker identity 

Use `CloudRunIdPlugin` to identify each Worker instance as `<instance-id>@<revision>`. The plugin reads Cloud Run environment variables and instance metadata when the Client connects, then Workers created from that Client inherit the identity.

Register the plugin on the Client options before connecting:

```csharp
using Temporalio.Client;
using Temporalio.Common.EnvConfig;
using Temporalio.Extensions.Gcp.CloudRun.Id;

// ...
    var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
    connectOptions.Plugins = new ITemporalClientPlugin[] { new CloudRunIdPlugin() };
    var client = await TemporalClient.ConnectAsync(connectOptions);
    // ...
```

The [preceding example](#opentelemetry) registers `CloudRunIdPlugin` alongside the OpenTelemetry defaults.

The plugin requires the Cloud Run metadata server, so it fails when the Worker runs outside Cloud Run. For a complete example, see the [.NET Cloud Run sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Gcp/CloudRun).

---

# Workflows - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows

> This section explains how to implement Workflows with the .NET SDK

![.NET SDK Banner](https://docs.temporal.io/img/assets/banner-dotnet-temporal.png)

## Workflows

- [Workflow basics](/develop/dotnet/workflows/basics)
- [Child Workflows](/develop/dotnet/workflows/child-workflows)
- [Continue-As-New](/develop/dotnet/workflows/continue-as-new)
- [Cancellation](/develop/dotnet/workflows/cancellation)
- [Timeouts](/develop/dotnet/workflows/timeouts)
- [Message passing](/develop/dotnet/workflows/message-passing)
- [Schedules](/develop/dotnet/workflows/schedules)
- [Timers](/develop/dotnet/workflows/timers)
- [Dynamic Workflow](/develop/dotnet/workflows/dynamic-workflow)
- [Versioning](/develop/dotnet/workflows/versioning)
- [Workflow Streams](/develop/dotnet/workflows/workflow-streams)

---

# Workflow basics - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/basics

> This section explains Workflow basics with the .NET SDK

## Develop a Workflow 

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal .NET SDK programming model, Workflows are defined as classes.

Specify the `[Workflow]` attribute from the `Temporalio.Workflows` namespace on the Workflow class to identify a Workflow.

Use the `[WorkflowRun]` attribute to mark the entry point method to be invoked. This must be set on one asynchronous method defined on the same class as `[Workflow]`.

```csharp
using Temporalio.Workflows;

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        var param = MyActivityParams("Hello", name);
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivity(param),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
}
```

### Define Workflow parameters 

Temporal Workflows may have any number of custom parameters.
However, we strongly recommend that you pass a single class parameter that contains all the input fields, so that you can add fields without breaking the signature of the Workflow.
All Workflow Definition parameters must be serializable.
For more reasons, see [Workflow parameters](/workflow-definition#workflow-parameters).

### Use Workflow constructors

Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. The `[WorkflowInit]` attribute gives message handlers access to Workflow input. When you use the `[WorkflowInit]` attribute on your constructor, you give the constructor the same Workflow parameters as your `[WorkflowRun]` method.

The SDK will then ensure that your constructor receives the Workflow input arguments that the Client sent. The Workflow input arguments are also passed to your `[WorkflowRun]` method. That always happens, whether or not you use the `[WorkflowInit]` attribute.

Here's an example. The constructor and `RunAsync` must have the same parameters with the same types:

```csharp
[Workflow]
public class WorkflowInitWorkflow
{
    public record Input(string Name);

    private readonly string nameWithTitle;
    private bool titleHasBeenChecked;

    [WorkflowInit]
    public WorkflowInitWorkflow(Input input) =>
        nameWithTitle = $"Knight {input.Name}";

    [WorkflowRun]
    public async Task<string> RunAsync(Input ignored)
    {
        await Workflow.WaitConditionAsync(() => titleHasBeenChecked);
        return $"Hello, {nameWithTitle}";
    }
}
```

## Workflow logic requirements 

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with application code outside the Workflow.

This means there are several things Workflows shouldn't do such as:

- Perform IO (network, disk, stdio, etc)
- Access/alter external mutable state
- Do any threading
- Do anything using the system clock (for example, `DateTime.Now`)
  - This includes .NET timers (for example, `Task.Delay` or `Thread.Sleep`)
- Make any random calls
- Make any not-guaranteed-deterministic calls (for example, iterating over a dictionary)

The SDK provides replay-safe alternatives for common needs.

### Logging

Use [`Workflow.Logger`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Logger)
instead of `Console.WriteLine` or a logger you resolve yourself.
The SDK logger appends Workflow details to every log and skips logging during replay:

```csharp
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        Workflow.Logger.LogInformation("Starting workflow for {Name}", name);
        // ...
    }
}
```

For logger configuration, see [Observability: Log from a Workflow](/develop/dotnet/platform/observability#logging).

### Random numbers and GUIDs

Use [`Workflow.Random`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Random)
to get a deterministic random instance seeded per Workflow Execution, and
[`Workflow.NewGuid()`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_NewGuid)
instead of `Guid.NewGuid()`:

```csharp
var value = Workflow.Random.Next(1, 100);
var uniqueId = Workflow.NewGuid();
```

### Current time

Use [`Workflow.UtcNow`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_UtcNow)
instead of `DateTime.Now` or `DateTime.UtcNow`. It returns the time of the last Workflow Task, which is consistent
across replays:

```csharp
var currentTime = Workflow.UtcNow;
```

To wait, use `Workflow.DelayAsync` instead of `Task.Delay` or `Thread.Sleep`.

### Detecting replay (advanced)

Use [`Workflow.Unsafe.IsReplaying`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.Unsafe.html#Temporalio_Workflows_Workflow_Unsafe_IsReplaying)
to guard code that should only run on the first execution, such as emitting metrics or sending external notifications
from an [Interceptor](/develop/dotnet/workers/interceptors).

> **⚠️ Caution:**
>
> Never use this to affect Workflow business logic. Branching on replay status breaks determinism.
>

```csharp
if (!Workflow.Unsafe.IsReplaying)
{
    EmitMetric("workflow_started", 1);
}
```

If your goal is to always take action when something new is happening, check that
[`Workflow.Unsafe.IsReplayingHistoryEvents`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.Unsafe.html#Temporalio_Workflows_Workflow_Unsafe_IsReplayingHistoryEvents)
is false instead. That is false during read-only operations like Queries and Update validators. This is what the SDK's
built-in logger and tracing interceptors use internally.

### .NET Task determinism

Some calls in .NET do unsuspecting non-deterministic things and are easy to accidentally use.
This is especially true with `Task`s.
Temporal requires that the deterministic `TaskScheduler.Current` is used, but many .NET async calls will use `TaskScheduler.Default` implicitly (and some analyzers even encourage this).
Here are some known gotchas to avoid with .NET tasks inside of Workflows:

-  Use `Workflow.RunTaskAsync` instead of `Task.Run`. `Task.Run` uses the default scheduler and puts work on the thread pool.
    - You can also use `Task.Factory.StartNew` with current scheduler or instantiate the `Task` and run `Task.Start` on it.
- If you need to use `Task.ConfigureAwait`, use `Task.ConfigureAwait(true)`. `Task.ConfigureAwait(false)` won't use the current context.
  - There is no significant performance benefit to `Task.ConfigureAwait` in workflows because of how the scheduler works.
- Avoid anything that defaults to the default task scheduler.
- Use `Workflow.DelayAsync`, `Workflow.WaitConditionAsync`, or non-timeout-based cancellation token sources instead of `Task.Delay`, `Task.Wait`, timeout-based `CancellationTokenSource`, or anything that uses .NET built-in timers.
- Use `Workflow.WhenAnyAsync` instead of `Task.WhenAny`.
  - Technically this only applies to an enumerable set of tasks with results or more than 2 tasks with results. Other
    uses are safe. See [this issue](https://github.com/dotnet/runtime/issues/87481).
- Use `Workflow.WhenAllAsync` instead of `Task.WhenAll`.
  - Technically `Task.WhenAll` is currently deterministic in .NET and safe, but it is better to use the wrapper to be
    sure.
- Use `CancellationTokenSource.Cancel` instead of `CancellationTokenSource.CancelAsync`.
- Use `Temporalio.Workflows.Semaphore` or `Temporalio.Workflows.Mutex` instead of `System.Threading.Semaphore`, `System.Threading.SemaphoreSlim`, or `System.Threading.Mutex`.
  - _Technically_ `SemaphoreSlim` does work if only the async form of `WaitAsync` is used without no timeouts and
    `Release` is used. But anything else can deadlock the workflow and its use is cumbersome since it must be disposed.
- Be wary of additional libraries' implicit use of the default scheduler.
  - For example, while there are articles for `Dataflow` about [using a specific scheduler](https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-specify-a-task-scheduler-in-a-dataflow-block), there are hidden implicit uses of `TaskScheduler.Default`. For example, see [this bug](https://github.com/dotnet/runtime/issues/83159).

To help catch wrong scheduler use, the Temporal .NET SDK adds an event source listener for info-level task events by default.
While this technically receives events from all uses of tasks in the process, we make sure to ignore anything that is not running in a Workflow in a high performant way (basically one thread local check).

For code that does run in a Workflow and accidentally starts a task in another scheduler, an `InvalidWorkflowOperationException` will be thrown which "pauses" the Workflow (fails the Workflow Task which continually retries until the code is fixed).
This is unfortunately a runtime-only check, but can help catch mistakes early. If this needs to be turned off for any reason, set `DisableWorkflowTracingEventListener` to `true` in Worker options.

In the near future for modern .NET versions we hope to use the
[new `TimeProvider` API](https://github.com/dotnet/runtime/issues/36617) which will allow us to control current time and
timers.

### Workflow .editorconfig

Since Workflow code follows some different logic rules than regular C# code, there are some common analyzer rules that developers may want to disable.
To ensure these are only disabled for Workflows, current recommendation is to use the `.workflow.cs` extension for files containing Workflows.

Here are the rules to disable:

- [CA1024](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1024) - This encourages properties instead of methods that look like getters. However for reflection reasons we cannot use property getters for queries, so it is very normal to have

  ```csharp
  [WorkflowQuery]
  public string GetSomeThing() => someThing;
  ```

- [CA1822](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1822) - This encourages static methods when methods don't access instance state. Workflows however use instance methods for run, Signals, Queries, or Updates even if they could be static.
- [CA2007](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2007) - This encourages users to use `ConfigureAwait` instead of directly waiting on a task. But in Workflows, there is no benefit to this and it just adds noise (and if used, needs to be `ConfigureAwait(true)` not `ConfigureAwait(false)`).
- [CA2008](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2008) - This encourages users to always apply an explicit task scheduler because the default of `TaskScheduler.Current` is bad. But for Workflows, the default of `TaskScheduler.Current` is good/required.
- [CA5394](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca5394) - This discourages use of non-crypto random. But deterministic Workflows, via `Workflow.Random` intentionally provide a deterministic non-crypto random instance.
- `CS1998` - This discourages use of `async` on async methods that don't `await`. But Workflows handlers like Signals are often easier to write in one-line form this way, for example `public async Task SignalSomethingAsync(string value) => this.value = value;`.
- [VSTHRD105](https://github.com/microsoft/vs-threading/blob/main/doc/analyzers/VSTHRD105.md) - This is similar to `CA2008` above in that use of implicit current scheduler is discouraged. That does not apply to Workflows where it is encouraged/required.

Here is the `.editorconfig` snippet for the above which may frequently change as more analyzers need to be adjusted:

```ini
##### Configuration specific for Temporal workflows #####
[*.workflow.cs]

# We use getters for queries, they cannot be properties
dotnet_diagnostic.CA1024.severity = none

# Don't force workflows to have static methods
dotnet_diagnostic.CA1822.severity = none

# Do not need ConfigureAwait for workflows
dotnet_diagnostic.CA2007.severity = none

# Do not need task scheduler for workflows
dotnet_diagnostic.CA2008.severity = none

# Workflow randomness is intentionally deterministic
dotnet_diagnostic.CA5394.severity = none

# Allow async methods to not have await in them
dotnet_diagnostic.CS1998.severity = none

# Don't avoid, but rather encourage things using TaskScheduler.Current in workflows
dotnet_diagnostic.VSTHRD105.severity = none
```

### Customize Workflow Type 

Workflows have a Type that are referred to as the Workflow name.

The following examples demonstrate how to set a custom name for your Workflow Type.

You can customize the Workflow name with a custom name in the attribute. For example, `[Workflow("my-workflow-name")]`. If the name parameter is not specified, the Workflow name defaults to the unqualified class name.

```csharp
using Temporalio.Workflows;

[Workflow("MyDifferentWorkflowName")]
public class MyWorkflow
{
    public async Task<string> RunAsync(string name)
    {
        var param = MyActivityParams("Hello", name);
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivity(param),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
}
```

---

# Cancellation - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/cancellation

> Handle Cancellation in .NET Workflows and Activities, request Cancellation from a Client, and choose between Cancellation, termination, and reset.

This page shows how to interrupt a Workflow Execution.

You can interrupt a Workflow Execution in one of the following ways:

- [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution.
- [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution.

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the
Workflow is stuck and cannot be canceled normally.

## Cancellation 

To give a Workflow and its Activities the ability to be cancelled, do the following:

- Handle a Cancellation request within a Workflow.
- Set Activity Heartbeat Timeouts.
- Listen for and handle a Cancellation request within an Activity.
- Send a Cancellation request from a Temporal Client.

### Handle Cancellation in Workflow 

Workflow Definitions can be written to respond to cancellation requests. It is common for an Activity to be run on
Cancellation to perform cleanup.

Cancellation Requests on Workflows cancel the `Workflow.CancellationToken`. This is the token that is implicitly used
for all calls within the workflow as well (for example, Timers, Activities, etc) and therefore cancellation is propagated to
them to be handled and bubble out.

```csharp
[WorkflowRun]
public async Task RunAsync()
{
    try
    {
        // Whether this workflow waits on the activity to handle the cancellation or not is
        // dependent upon the CancellationType option. We leave the default here which sends the
        // cancellation but does not wait on it to be handled.
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyNormalActivity(),
            new() { ScheduleToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
    catch (Exception e) when (TemporalException.IsCanceledException(e))
    {
        // The "when" clause above is because we only want to apply the logic to cancellation, but
        // this kind of cleanup could be done on any/all exceptions too.
        Workflow.Logger.LogError(e, "Cancellation occurred, performing cleanup");

        // Call cleanup activity. If this throws, it will swallow the original exception which we
        // are ok with here. This could be changed to just log a failure and let the original
        // cancellation continue. 
        // The default token on Workflow.CancellationToken is now marked
        // cancelled, so we pass a different one. We use CancellationToken.None here because the
        // cleanup activity itself doesn't need to be cancellable; if it did (e.g. you want to
        // cancel cleanup from a timeout or another signal), create a new detached
        // CancellationTokenSource and pass its Token instead.
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyCancellationCleanupActivity(),
            new()
            {
                ScheduleToCloseTimeout = TimeSpan.FromMinutes(5),
                CancellationToken = CancellationToken.None,
            });

        // Rethrow the cancellation
        throw;
    }
}
```

### Handle Cancellation in an Activity 

Ensure that the Activity is [Heartbeating](/develop/dotnet/activities/timeouts#activity-heartbeats) to receive the
Cancellation request and stop execution. Also make sure that the
[Heartbeat Timeout](/develop/dotnet/activities/timeouts#heartbeat-timeout) is set on the Activity Options when calling
from the Workflow. An Activity Cancellation Request cancels the `CancellationToken` on the `ActivityExecutionContext`.

```csharp
[Activity]
public async Task MyActivityAsync()
{
    // This is a naive loop simulating work, but similar heartbeat/cancellation logic applies to
    // other scenarios as well
    while (true)
    {
        // Send heartbeat
        ActivityExecutionContext.Current.Heartbeat();

        // Do some work, passing the cancellation token
        await Task.Delay(1000, ActivityExecutionContext.Current.CancellationToken);
    }
}
```

### Request Cancellation 

Use `CancelAsync` on the `WorkflowHandle` to cancel a Workflow Execution.

```csharp
// Get a workflow handle by its workflow ID. This could be made specific to a run by passing run ID.
// This could also just be a handle that is returned from StartWorkflowAsync instead.
var handle = myClient.GetWorkflowHandle("my-workflow-id");

// Send cancellation. This returns when cancellation is received by the server. Wait on the handle's
// result to wait for cancellation to be applied.
await handle.CancelAsync();
```

#### Request Cancellation of an Activity 

By default, Activities are automatically cancelled when the Workflow is cancelled since the workflow cancellation token
is used by activities by default. To issue a cancellation explicitly, a new cancellation token can be created.

```csharp
[WorkflowRun]
public async Task RunAsync()
{
    // Create a source linked to workflow cancellation. A new source could be created instead if we
    // didn't want it associated with workflow cancellation.
    using var cancelActivitySource = CancellationTokenSource.CreateLinkedTokenSource(
        Workflow.CancellationToken);

    // Start the activity. Whether this workflow waits on the activity to handle the cancellation
    // or not is dependent upon the CancellationType option. We leave the default here which sends
    // the cancellation but does not wait on it to be handled.
    var activityTask = Workflow.ExecuteActivityAsync(
        (MyActivities a) => a.MyNormalActivity(),
        new()
        {
            ScheduleToCloseTimeout = TimeSpan.FromMinutes(5),
            CancellationToken = cancelActivitySource.Token;
        });
    activityTask.Start();

    // Wait 5 minutes, then cancel it
    await Workflow.DelayAsync(TimeSpan.FromMinutes(5));
    cancelActivitySource.Cancel();

    // Wait on the activity which will throw cancellation which will fail the workflow
    await activityTask;
}
```

## Termination 

To Terminate a Workflow Execution in .NET, use the
[TerminateAsync()](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_TerminateAsync_System_String_Temporalio_Client_WorkflowTerminateOptions_)
method on the Workflow handle.

```csharp
// Get a workflow handle by its workflow ID. This could be made specific to a run by passing run ID.
// This could also just be a handle that is returned from StartWorkflowAsync instead.
var handle = myClient.GetWorkflowHandle("my-workflow-id");

// Terminate
await handle.TerminateAsync();
```

Workflow Executions can also be Terminated directly from the WebUI. In this case, a custom note can be logged from the
UI when that happens.

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Child Workflows - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/child-workflows

> Start a Child Workflow Execution and set a Parent Close Policy using Temporal .NET SDK. Discover methods like ExecuteChildWorkflowAsync and manage Workflow behaviors.

This page shows how to do the following:

- [Start a Child Workflow Execution](#child-workflows)
- [Set a Parent Close Policy](#parent-close-policy)

## Start a Child Workflow Execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In .NET, awaiting `StartChildWorkflowAsync()` or `ExecuteChildWorkflowAsync()` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call resolves.
If you start a Child Workflow from a non-main context (for example, a Signal or Update handler), make sure the Parent Workflow doesn't complete before that call resolves.

To spawn a Child Workflow Execution in .NET, use the `ExecuteChildWorkflowAsync()` method which starts the Child Workflow and waits for completion or
use the `StartChildWorkflowAsync()` method to start a Child Workflow and return its handle.
This is useful if you want to do something after it has only started, or to get the Workflow/Run ID, or to be able to signal it while running.

> **📝 Note:**
>
> `ExecuteChildWorkflowAsync()` is a helper method for `StartChildWorkflowAsync()` plus `await handle.GetResultAsync()`.
>

```csharp
await Workflow.ExecuteChildWorkflowAsync((MyChildWorkflow wf) => wf.RunAsync());
```

## Set a Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

Set the `ParentClosePolicy` property inside the [`ChildWorkflowOptions`](https://dotnet.temporal.io/api/Temporalio.Workflows.ChildWorkflowOptions.html) for `ExecuteChildWorkflowAsync` or `StartChildWorkflowAsync` to specify the behavior of the Child Workflow when the Parent Workflow closes.

```csharp
await Workflow.ExecuteChildWorkflowAsync(
  (MyChildWorkflow wf) => wf.RunAsync(),
  new() { ParentClosePolicy = ParentClosePolicy.Abandon });
```

---

# Continue-As-New - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/continue-as-new

> Use Temporal's Continue-As-New in .NET to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters.

This page covers the following for .NET developers:

- [What is Continue-As-New?](#what)
- [Use Continue-As-New](#how)
- [When is it right to Continue-As-New?](#when)
- [Test Continue-As-New](#how-to-test)

## What is Continue-As-New? 

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution.
You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History.
It also receives your Workflow's usual parameters.

## Use Continue-As-New with the .NET SDK 

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically set to `None` for the original caller of the Workflow.

[View the source code](https://github.com/temporalio/samples-dotnet/blob/main/src/SafeMessageHandlers/ClusterManagerWorkflow.workflow.cs) in the context of the rest of the application code.

```csharp
public record Input
    {
        public State State { get; init; } = new();

        public bool TestContinueAsNew { get; init; }
    }

[WorkflowInit]
public ClusterManagerWorkflow(Input input)

````
The test hook in the above snippet is covered [below](#how-to-test).

Inside your Workflow, throw a [`CreateContinueAsNewException`](https://dotnet.temporal.io/api/Temporalio.Workflows.ContinueAsNewException.html) exception.
This stops the Workflow right away and starts a new one.

[View the source code](https://github.com/temporalio/samples-dotnet/blob/main/src/SafeMessageHandlers/ClusterManagerWorkflow.workflow.cs) in the context of the rest of the application code.

```csharp
throw Workflow.CreateContinueAsNewException((ClusterManagerWorkflow wf) => wf.RunAsync(new()
{
    State = CurrentState,
    TestContinueAsNew = input.TestContinueAsNew,
}));
````

### Considerations for Workflows with message handlers 

If you use Updates or Signals, don't call Continue-as-New from the handlers.
Instead, wait for your handlers to finish in your main Workflow before you throw `CreateContinueAsNewException`.
See the [`AllHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance.

## When is it right to Continue-As-New with the .NET SDK? 

Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New.
Call `Workflow.ContinueAsNewSuggested` to check if it's time.

## Test Continue-As-New with the .NET SDK 

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive.
Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, when `TestContinueAsNew == true`, this sample creates a test-only variable called `maxHistoryLength` and sets it to a small value.
A helper variable in the Workflow checks it each time it considers using Continue-as-New:

[View the source code](https://github.com/temporalio/samples-dotnet/blob/main/src/SafeMessageHandlers/ClusterManagerWorkflow.workflow.cs) in the context of the rest of the application code.

```csharp
private bool ShouldContinueAsNew =>
    // Don't continue as new while update running
    Workflow.AllHandlersFinished &&
    // Continue if suggested or, for ease of testing, max history reached
    (Workflow.ContinueAsNewSuggested || Workflow.CurrentHistoryLength > maxHistoryLength);
```

---

# Dynamic Workflow - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/dynamic-workflow

> This section explains Dynamic Workflows with the .NET SDK

## Set a Dynamic Workflow 

A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered.
A Workflow can be made dynamic by setting `Dynamic` as `true` on the `[Workflow]` attribute.
You must register the Workflow with the Worker before it can be invoked.
Only one Dynamic Workflow can be present on a Worker.

The Workflow Definition must then accept a single argument of type `Temporalio.Converters.IRawValue[]`.
The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` namespace.

```csharp
[Workflow(Dynamic = true)]
public class DynamicWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(IRawValue[] args)
    {
        var name = Workflow.PayloadConverter.ToValue<string>(args.Single());
        var param = MyActivityParams("Hello", name);
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivity(param),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
}
```

---

# Workflow message passing - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/message-passing

> Develop with Queries, Signals, and Updates with the Temporal .NET SDK.

A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates.
The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values.
Temporal Clients use messages to read Workflow state and control execution.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic.
This page introduces these features for the Temporal .NET SDK.

## Write message handlers 

> **ℹ️ Info:**
> The code that follows is part of a [working solution](https://github.com/temporalio/samples-dotnet/tree/main/src/SafeMessageHandlers).

Follow these guidelines when writing your message handlers:

- Message handlers are defined as methods on the Workflow class, using one of the three attributes: [`WorkflowQueryAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowQueryAttribute.html), [`WorkflowSignalAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowSignalAttribute.html), and [`WorkflowUpdateAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateAttribute.html).
- The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion).
- Prefer data classes to multiple input parameters. Data class parameters allow you to add fields without changing the calling signature. Keep in mind that serialization and deserialization can fail with the default data converter if the new field does not have a default value.

### Query handlers 

A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution.
Define as a method:

```csharp
[Workflow]
public class GreetingWorkflow
{
    public enum Language
    {
        Chinese,
        English,
        French,
        Spanish,
        Portuguese,
    }

    public record GetLanguagesInput(bool IncludeUnsupported);

    // ...

    [WorkflowQuery]
    public IList<Language> GetLanguages(GetLanguagesInput input) =>
        Enum.GetValues<Language>().
            Where(language => input.IncludeUnsupported || Greetings.ContainsKey(language)).
            ToList();

    // ...
```

Or as a property getter:

```csharp
[Workflow]
public class GreetingWorkflow
{
    public enum Language
    {
        Chinese,
        English,
        French,
        Spanish,
        Portuguese,
    }

    // ...

    [WorkflowQuery]
    public Language CurrentLanguage { get; private set; } = Language.English;

    // ...
```

- The Query attribute can accept arguments.
  See the API reference docs: [`WorkflowQueryAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowQueryAttribute.html).
- A Query handler must not modify Workflow state.
- You can't perform async blocking operations such as executing an Activity in a Query handler.

### Signal handlers 

A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow:

```csharp
[Workflow]
public class GreetingWorkflow
{
    public record ApproveInput(string Name);

    // ...

    [WorkflowSignal]
    public async Task ApproveAsync(ApproveInput input)
    {
        approvedForRelease = true;
        approverName = input.Name;
    }

    // ...
```

- The Signal attribute can accept arguments.
  Refer to the API docs: [`WorkflowSignalAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowSignalAttribute.html).

- The handler should not return a value.
  The response is sent immediately from the server, without waiting for the Workflow to process the Signal.

- Signal (and Update) handlers can be asynchronous and blocking.
  This allows you to use Activities, Child Workflows, durable [`Workflow.DelayAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_DelayAsync_System_Int32_System_Nullable_System_Threading_CancellationToken__) Timers, [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Signal and Update handlers.

### Update handlers and validators 

An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution.
It can change the Workflow state, control its flow, and return a result.
The sender must wait until the Worker accepts or rejects the Update.
The sender may wait further to receive a returned value or an exception if something goes wrong:

```csharp
[Workflow]
public class GreetingWorkflow
{
    public enum Language
    {
        Chinese,
        English,
        French,
        Spanish,
        Portuguese,
    }

    // ...

    [WorkflowUpdateValidator(nameof(SetCurrentLanguageAsync))]
    public void ValidateLanguage(Language language)
    {
        if (!Greetings.ContainsKey(language))
        {
            throw new ApplicationFailureException($"{language} is not supported");
        }
    }

    [WorkflowUpdate]
    public async Task<Language> SetCurrentLanguageAsync(Language language)
    {
        var previousLanguage = CurrentLanguage;
        CurrentLanguage = language;
        return previousLanguage;
    }

    // ...
```

- The Update attribute can take arguments (like, `Name`, `Dynamic` and `UnfinishedPolicy`) as described in the API reference docs for [`WorkflowUpdateAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateAttribute.html).

- About validators:
  - Use validators to reject an Update before it is written to History.
    Validators are always optional.
    If you don't need to reject Updates, you can skip them.
  - Define an Update validator with the [`WorkflowUpdateValidatorAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateValidatorAttribute.html) attribute.
    Use the Name argument when declaring the validator to connect it to its Update.
    The validator must be a `void` type and accept the same argument types as the handler.

- Accepting and rejecting Updates with validators:
  - To reject an Update, raise an exception of any type in the validator.
  - Without a validator, Updates are always accepted.
- Validators and Event History:
  - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic.
  - When a Validator raises an error, the Update is rejected, the Update is not run, and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History.
    The caller receives an "Update failed" error.

- Use [`Workflow.CurrentUpdateInfo`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_CurrentUpdateInfo) to obtain information about the current Update.
  This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
- Update (and Signal) handlers can be asynchronous and blocking.
  This allows you to use Activities, Child Workflows, durable [`Workflow.DelayAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_DelayAsync_System_Int32_System_Nullable_System_Threading_CancellationToken__) Timers, [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Update and Signal handlers.

## Send messages 

To send Queries, Signals, or Updates you call methods on a [`WorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html) object.
To obtain the WorkflowStub, you can:

- Use [`TemporalClient.StartWorkflowAsync`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_StartWorkflowAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowOptions_) to start a Workflow and return its handle.
- Use the [`TemporalClient.GetWorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_GetWorkflowHandle_System_String_System_String_System_String_) method to retrieve a Workflow handle by its Workflow Id.

For example:

```csharp
var client = await TemporalClient.ConnectAsync(new("localhost:7233"));
var workflowHandle = await client.StartWorkflowAsync(
    (GreetingWorkflow wf) => wf.RunAsync(),
    new(id: "message-passing-workflow-id", taskQueue: "message-passing-sample"));
```

To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition.

> **⚠️ Warning:**
> Using Continue-as-New and Updates
>
> - Temporal _does not_ support Continue-as-New functionality within Update handlers.
> - Complete all handlers _before_ using Continue-as-New.
> - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution.
>

### Send a Query 

Call a Query method with [`WorkflowHandle.QueryAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_QueryAsync__1_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowQueryOptions_):

```csharp
var supportedLanguages = await workflowHandle.QueryAsync(wf => wf.GetLanguages(new(false)));
```

- Sending a Query doesn’t add events to a Workflow's Event History.

- You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period.
  This includes Workflows that have completed, failed, or timed out.
  Querying terminated Workflows is not safe and, therefore, not supported.

- A Worker must be online and polling the Task Queue to process a Query.

### Send a Signal 

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution.
However, you can only send Signals to Workflow Executions that haven’t closed.

#### Send a Signal from a Client 

Use [`WorkflowHandle.SignalAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_SignalAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowSignalOptions_) from Client code to send a Signal:

```csharp
await workflowHandle.SignalAsync(wf => wf.ApproveAsync(new("MyUser")));
```

- The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution.

- The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History.

#### Send a Signal from a Workflow 

A Workflow can send a Signal to another Workflow, known as an _External Signal_.
In this case you need to obtain a Workflow handle for the external Workflow.
Use `Workflow.GetExternalWorkflowHandle`, passing a running Workflow Id, to retrieve a typed Workflow handle:

```csharp
// ...

[Workflow]
public class WorkflowB
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        var handle = Workflow.GetExternalWorkflowHandle<WorkflowA>("workflow-a");
        await handle.SignalAsync(wf => wf.YourSignalAsync("signal argument"));
    }

    // ...
```

When an External Signal is sent:

- A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History.
- A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History.

#### Signal-With-Start 

Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running.
If there's a Workflow running with the given Workflow Id, it will be signaled.
If there isn't, a new Workflow will be started and immediately signaled.
To use Signal-With-Start, call `SignalWithStart` with a lambda expression invoking it:

```csharp
var client = await TemporalClient.ConnectAsync(new("localhost:7233"));
var options = new WorkflowOptions(id: "your-signal-with-start-workflow", taskQueue: "signal-tq");
options.SignalWithStart((GreetingWorkflow wf) => wf.SubmitGreetingAsync("User Signal with Start"));
await client.StartWorkflowAsync((GreetingWorkflow wf) => wf.RunAsync(), options);
```

### Send an Update 

An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result.

A Client sending an Update must wait until the Server delivers the Update to a Worker.
Workers must be available and responsive.
If you need a response as soon as the Server receives the request, use a Signal instead.
You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity.

- `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation.
- `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished.

To send an Update to a Workflow Execution, you can:

- Call the Update method with `ExecuteUpdateAsync` from the Client and wait for the Update to complete.
  This code fetches an Update result:

  ```csharp
  var previousLanguage = await workflowHandle.ExecuteUpdateAsync(
    wf => wf.SetCurrentLanguageAsync(GreetingWorkflow.Language.Chinese));
  ```

2. Use `StartUpdateAsync` to receive a handle as soon as the Update is accepted.
     It returns an `UpdateHandle`

  - Use this `UpdateHandle` later to fetch your results.
  - Asynchronous Update handlers normally perform long-running async Activities.
  - `StartUpdateAsync` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete.

  For example:

  ```csharp
  // Wait until the update is accepted
  var updateHandle = await workflowHandle.StartUpdateAsync(
      wf => wf.SetGreetingAsync(new HelloWorldInput("World")),
      new(waitForStage: WorkflowUpdateStage.Accepted));
  // Wait until the update is completed
  var updateResult = await updateHandle.GetResultAsync();
  ```

  For more details, see the "Async handlers" section.

#### Update-with-Start 

> **💡 Tip:**
>
> For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended.
>

[Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](/develop/dotnet/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists:

- If the Workflow exists, the Update is processed.
- If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute.

Use `ExecuteUpdateWithStartAsync` to start the Update and wait for the result in one go.

Alternatively, use `StartUpdateWithStartAsync` to start the Update and receive a `WorkflowUpdateHandle`, and then use `await updateHandle.GetResultAsync()` to retrieve the result from the Update.

These calls return once the requested Update wait stage has been reached, or when the request times out.

- You will need to provide a `WithStartWorkflowOperation` to define the Workflow that will be started if necessary, and its arguments.
- You must specify an [IdConflictPolicy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) when creating the `WithStartWorkflowOperation`.
  Note that a `WithStartWorkflowOperation` can only be used once.

Here's an example taken from the [UpdateWithStartLazyInit](https://github.com/temporalio/samples-dotnet/blob/main/src/UpdateWithStartLazyInit/Program.cs) sample:

```csharp
async Task<AddCartItemResult> AddCartItemAsync(string sessionId, ShoppingCartItem item)
{
    // Issue an update-with-start that will create the workflow if it does not
    // exist before attempting the update

    // Create the start operation
    var startOperation = WithStartWorkflowOperation.Create(
        (ShoppingCartWorkflow wf) => wf.RunAsync(),
        new(id: $"cart-{sessionId}", taskQueue: TaskQueue)
        {
            IdConflictPolicy = Temporalio.Api.Enums.V1.WorkflowIdConflictPolicy.UseExisting,
        });

    // Issue the update-with-start, swallowing item-unavailable failure
    decimal? subtotal;
    try
    {
        subtotal = await client.ExecuteUpdateWithStartWorkflowAsync(
            (ShoppingCartWorkflow wf) => wf.AddItemAsync(item),
            new(startOperation));
    }
    catch (WorkflowUpdateFailedException e) when (
        e.InnerException is ApplicationFailureException appErr && appErr.ErrorType == "ItemUnavailable")
    {
        // Set subtotal to null if item was not found
        subtotal = null;
    }

    return new(await startOperation.GetHandleAsync(), subtotal);
}
```

> **ℹ️ Info:**
> NON-TYPE SAFE API CALLS
>
> In real-world development, sometimes you may be unable to import Workflow Definition method signatures.
> When you don't have access to the Workflow Definition or it isn't written in .NET, you can still use non-type safe APIs and dynamic method invocation.
> Pass method names instead of method objects to:
>
> - [`TemporalClient.StartWorkflowAsync`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_StartWorkflowAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowOptions_)
> - [`WorkflowHandle.QueryAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_QueryAsync__1_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowQueryOptions_)
> - [`WorkflowHandle.SignalAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_SignalAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowSignalOptions_)
> - [`WorkflowHandle.ExecuteUpdateAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_ExecuteUpdateAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowUpdateOptions_)
> - [`WorkflowHandle.StartUpdateAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_StartUpdateAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowUpdateStartOptions_)
>
> Use non-type safe overloads of these APIs:
>
> - [`TemporalClient.GetWorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_GetWorkflowHandle_System_String_System_String_System_String_)
> - [`Workflow.GetExternalWorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_GetExternalWorkflowHandle_System_String_System_String_)
>

## Message handler patterns 

This section covers common write operations, such as Signal and Update handlers.
It doesn't apply to pure read operations, like Queries or Update Validators.

> **💡 Tip:**
>
> For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow) and [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
>

### Add async handlers to use `await` 

Signal and Update handlers can be asynchronous as well as blocking.
Using asynchronous calls allows you to `await` Activities, Child Workflows, [`Workflow.DelayAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_DelayAsync_System_Int32_System_Nullable_System_Threading_CancellationToken__) Timers, [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) wait conditions, and so on.
This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls.

To use asynchronous handlers safely, understand what can go wrong.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below.

The following code executes an Activity that simulates a network call to a remote service:

```csharp
public class MyActivities
{
    private static readonly Dictionary<Language, string> Greetings = new()
    {
        [Language.Arabic] = "مرحبا بالعالم",
        [Language.Chinese] = "你好，世界",
        [Language.English] = "Hello, world",
        [Language.French] = "Bonjour, monde",
        [Language.Hindi] = "नमस्ते दुनिया",
        [Language.Spanish] = "Hola mundo",
    };

    [Activity]
    public async Task<string?> CallGreetingServiceAsync(Language language)
    {
        // Pretend that we are calling a remove service
        await Task.Delay(200);
        return Greetings.TryGetValue(language, out var value) ? value : null;
    }
}
```

The following code modifies a `WorkflowUpdate` for asynchronous use of the preceding Activity:

```csharp
[Workflow]
public class GreetingWorkflow
{
    private readonly Mutex mutex = new();

    // ...

    [WorkflowUpdate]
    public async Task<Language> SetLanguageAsync(Language language)
    {
        // 👉 Use a mutex here to ensure that multiple calls to SetLanguageAsync are processed in order.
        await mutex.WaitOneAsync();
        try
        {
            if (!greetings.ContainsKey(language))
            {
                var greeting = Workflow.ExecuteActivityAsync(
                    (MyActivities acts) => acts.CallGreetingServiceAsync(language),
                    new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) });
                if (greeting == null)
                {
                    // 👉 An update validator cannot be async, so cannot be used to check that the remote
                    // CallGreetingServiceAsync supports the requested language. Throwing ApplicationFailureException
                    // will fail the Update, but the WorkflowExecutionUpdateAccepted event will still be
                    // added to history.
                    throw new ApplicationFailureException(
                        $"Greeting service does not support {language}");
                }
                greetings[language] = greeting;
            }
            var previousLanguage = CurrentLanguage;
            CurrentLanguage = language;
            return previousLanguage;
        }
        finally
        {
            mutex.ReleaseMutex();
        }
    }
}
```

After updating the code for asynchronous calls, your Update handler can schedule an Activity and await the result.
Although an async Signal handler can initiate similar network tasks, using an Update handler allows the Client to receive a result or error once the Activity completes.
This lets your Client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, and so on.

### Add wait conditions to block 

Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue.
Using a wait condition with [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) sets a function that prevents the code from proceeding until the condition returns `true`.
This is an important feature that helps you control your handler logic.

Here are two important use cases for `Workflow.WaitConditionAsync`:

- Waiting in a handler until it is appropriate to continue.
- Waiting in the main Workflow until all active handlers have finished.

The condition state you're waiting for can be updated by and reflect any part of the Workflow code.
This includes the main Workflow method, other handlers, or child coroutines spawned by the main Workflow method, and so forth.

#### Use wait conditions in handlers 

Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue.
Using a wait condition with [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) sets a function that prevents the code from proceeding until the condition returns `true`.
This is an important feature that helps you control your handler logic.

Consider a `ReadyForUpdateToExecute` method that runs before your Update handler executes.
The [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) call waits until your condition is met:

```csharp
[WorkflowUpdate]
public async Task<string> MyUpdateAsync(UpdateInput updateInput)
{
    await Workflow.WaitConditionAsync(() => ReadyForUpdateToExecute(updateInput));
    // ...
}
```

Remember: Handlers can execute before the main Workflow method starts.

#### Ensure your handlers finish before the Workflow completes 

Workflow wait conditions can ensure your handler completes before a Workflow finishes.
When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result.
The Workflow completing may interrupt the handler before it finishes crucial work and cause Client errors when trying retrieve Update results.
Use `Workflow.AllHandlersFinished` to address this problem and allow your Workflow to end smoothly:

```csharp
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        // ...
        await Workflow.WaitConditionAsync(() => Workflow.AllHandlersFinished);
        return "workflow-result";
    }

    // ...
```

By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions.
You can silence these warnings on a per-handler basis by passing the `UnfinishedPolicy` argument to the [`WorkflowSignalAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowSignalAttribute.html) / [`WorkflowUpdateAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateAttribute.html) decorator:

```csharp
[WorkflowUpdate(UnfinishedPolicy = HandlerUnfinishedPolicy.Abandon)]
public async Task MyUpdateAsync()
{
    // ...
```

See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information.

### Use `[WorkflowInit]` to operate on Workflow input before any handler executes

The `[WorkflowInit]` attribute gives message handlers access to [Workflow input](/handling-messages#workflow-initializers).
When you use the `[WorkflowInit]` attribute on your constructor, you give the constructor the same Workflow parameters as your `[WorkflowRun]` method.
The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/dotnet/client/temporal-client#start-workflow).
The Workflow input arguments are also passed to your `[WorkflowRun]` method -- that always happens, whether or not you use the `[WorkflowInit]` attribute.

Here's an example.
The constructor and `RunAsync` must have the same parameters with the same types:

```csharp
[Workflow]
public class WorkflowInitWorkflow
{
    public record Input(string Name);

    private readonly string nameWithTitle;
    private bool titleHasBeenChecked;

    [WorkflowInit]
    public WorkflowInitWorkflow(Input input) =>
        nameWithTitle = $"Sir {input.Name}";

    [WorkflowRun]
    public async Task<string> RunAsync(Input ignored)
    {
        await Workflow.WaitConditionAsync(() => titleHasBeenChecked);
        return $"Hello, {nameWithTitle}";
    }

    [WorkflowUpdate]
    public async Task<bool> CheckTitleValidityAsync()
    {
        // The handler is now guaranteed to see the workflow input after it has
        // been processed by the constructor.
        var valid = await Workflow.ExecuteActivityAsync(
            (MyActivities acts) -> acts.CheckTitleValidityAsync(nameWithTitle),
            new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) });
        titleHasBeenChecked = true;
        return valid;
    }
}
```

### Use locks to prevent concurrent handler execution 

Concurrent processes can interact in unpredictable ways.
Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously.
Here's an example of a pathological case:

```csharp
[Workflow]
public class MyWorkflow
{
    // ...

    [WorkflowSignal]
    public async Task BadHandlerAsync()
    {
        var data = await Workflow.ExecuteActivityAsync(
            (MyActivities acts) => acts.FetchDataAsync(),
            new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) });
        this.x = data.X;
        // 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then
        // there may be times when the Workflow has this.x from one Activity execution and this.y from another.
        await Workflow.DelayAsync(1000);
        this.y = data.Y;
    }
}
```

Coordinating access with [`Workflows.Mutex`](https://dotnet.temporal.io/api/Temporalio.Workflows.Mutex.html), a mutual exclusion lock, corrects this code.
Locking makes sure that only one handler instance can execute a specific section of code at any given time:

```csharp
[Workflow]
public class MyWorkflow
{
    private readonly Mutex mutex = new();

    // ...

    [WorkflowSignal]
    public async Task SafeHandlerAsync()
    {
        await mutex.WaitOneAsync();
        try
        {
            var data = await Workflow.ExecuteActivityAsync(
                (MyActivities acts) => acts.FetchDataAsync(),
                new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) });
            this.x = data.X;
            // ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow
            // method, but no other execution of this handler can run until this execution finishes.
            await Workflow.DelayAsync(1000);
            this.y = data.Y;
        }
        finally
        {
            mutex.ReleaseMutex();
        }
    }
}
```

For additional concurrency options, you can use [`Workflows.Semaphore`](https://dotnet.temporal.io/api/Temporalio.Workflows.Semaphore.html).
Semaphores manage access to shared resources and coordinate the order in which threads or processes execute.

## Message handler troubleshooting 

When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors:

- **The Client can't contact the server**:
  You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) exception whose `Code` property is [`RpcException.StatusCode`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.StatusCode.html) with a status of `Unavailable` (after some retries).

- **The Workflow does not exist**:
  You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) exception whose `Code` property is [`RpcException.StatusCode`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.StatusCode.html) with a status of `NotFound`.

See [Exceptions in message handlers](/handling-messages#exceptions) for a non–.NET-specific discussion of this topic.

### Problems when sending a Signal 

When using Signal, the only exception that will result from your requests during its execution is `RpcException`.
All handlers may experience additional exceptions during the initial (pre-Worker) part of a handler request lifecycle.

For Queries and Updates, the Client waits for a response from the Worker.
If an issue occurs during the handler Execution by the Worker, the Client may receive an exception.

### Problems when sending an Update 

When working with Updates, you may encounter these errors:

- **No Workflow Workers are polling the Task Queue**:
  Your request will be retried by the SDK Client indefinitely.
  Use a `CancellationToken` in your [RPC options](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowUpdateOptions.html#Temporalio_Client_WorkflowUpdateOptions_Rpc) to cancel the Update.
  This raises a [Temporalio.Exceptions.WorkflowUpdateRpcTimeoutOrCanceledException](https://dotnet.temporal.io/api/Temporalio.Exceptions.WorkflowUpdateRpcTimeoutOrCanceledException.html) exception.

- **Update failed**: You'll receive a [`Temporalio.Exceptions.WorkflowUpdateFailedException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.WorkflowUpdateFailedException.html) exception.
  There are two ways this can happen:

  - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler.

  - The Update failed after having been accepted.

  Update failures are like [Workflow failures](/references/failures).
  Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.
  These might include:

      - A failed Child Workflow
      - A failed Activity (if the Activity retries have been set to a finite number)
      - The Workflow author raising `ApplicationFailure`
      - Any error listed in [`TemporalWorkerOptions.WorkflowFailureExceptionTypes`](https://dotnet.temporal.io/api/Temporalio.Worker.TemporalWorkerOptions.html#Temporalio_Worker_TemporalWorkerOptions_WorkflowFailureExceptionTypes) on the Worker or [`WorkflowAttribute.FailureExceptionTypes`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowAttribute.html#Temporalio_Workflows_WorkflowAttribute_FailureExceptionTypes) on the Workflow (empty by default)

- **The handler caused the Workflow Task to fail**:
  A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage:
  - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) exception.
  - If the request has been accepted, it is durable.
    Once the Workflow is healthy again after a code deploy, use an [`UpdateHandle`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowUpdateHandle.html) to fetch the Update result.

- **The Workflow finished while the Update handler execution was in progress**:
  You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) "workflow execution already completed".

  This will happen if the Workflow finished while the Update handler execution was in progress, for example because

  - The Workflow was canceled or failed.

  - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers).

### Problems when sending a Query 

When working with Queries, you may encounter these errors:

- **There is no Workflow Worker polling the Task Queue**:
  You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) on which the `Code` is a [`RpcException.StatusCode`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.StatusCode.html) with a status of `FailedPrecondition`.

- **Query failed**:
  You'll receive a [`Temporalio.Exceptions.WorkflowQueryFailedException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.WorkflowQueryFailedException.html) exception if something goes wrong during a Query.
  Any exception in a Query handler will trigger this error.
  This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead.

- **The handler caused the Workflow Task to fail.**
  This would happen, for example, if the Query handler blocks the thread for too long without yielding.

## Dynamic Handler 

Temporal supports Dynamic Queries, Signals, Updates, Workflows, and Activities.
These are unnamed handlers that are invoked if no other statically defined handler with the given name exists.

Dynamic Handlers provide flexibility to handle cases where the names of Queries, Signals, Updates, Workflows, or Activities, aren't known at run time.

> **⚠️ Caution:**
>
> Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach.
> Overusing them can lead to maintainability and debugging issues down the line.
>
> Instead, Signals, Queries, Workflows, or Activities should be defined statically whenever possible, with clear names that indicate their purpose.
> Use static definitions as the primary way of structuring your Workflows.
>
> Reserve Dynamic Handlers for cases where the handler names are not known at compile time and need to be looked up dynamically at runtime.
> They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic.
>

### Set a Dynamic Query 

A Dynamic Query in Temporal is a Query method that is invoked dynamically at runtime if no other Query with the same name is registered.
A Query can be made dynamic by setting `Dynamic` to `true` on the `[WorkflowQuery]` attribute.
Only one Dynamic Query can be present on a Workflow.

The Query Handler parameters must accept a `string` name and `Temporalio.Converters.IRawValue[]` for the arguments.
The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` Namespace.

```csharp
[WorkflowQuery(Dynamic = true)]
public string DynamicQueryAsync(string queryName, IRawValue[] args)
{
    var input = Workflow.PayloadConverter.ToValue<MyStatusParam>(args.Single());
    return statuses[input.Type];
}
```

### Set a Dynamic Signal 

A Dynamic Signal in Temporal is a Signal that is invoked dynamically at runtime if no other Signal with the same input is registered.
A Signal can be made dynamic by setting `Dynamic` to `true` on the `[WorkflowSignal]` attribute.
Only one Dynamic Signal can be present on a Workflow.

The Signal Handler parameters must accept a `string` name and `Temporalio.Converters.IRawValue[]` for the arguments.
The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` Namespace.

```csharp
[WorkflowSignal(Dynamic = true)]
public async Task DynamicSignalAsync(string signalName, IRawValue[] args)
{
    var input = Workflow.PayloadConverter.ToValue<DoSomethingParam>(args.Single());
    pendingThings.Add(input);
}
```

### Set a Dynamic Update 

A Dynamic Update in Temporal is an Update that is invoked dynamically at runtime if no other Update with the same input is registered.
An Update can be made dynamic by setting `Dynamic` to `true` on the `[WorkflowUpdate]` attribute.
Only one Dynamic Update can be present on a Workflow.

The Update Handler parameters must accept a `string` name and `Temporalio.Converters.IRawValue[]` for the arguments.
The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` Namespace.

```csharp
[WorkflowUpdate(Dynamic = true)]
public async Task<string> DynamicUpdateAsync(string updateName, IRawValue[] args)
{
    var input = Workflow.PayloadConverter.ToValue<DoSomethingParam>(args.Single());
    pendingThings.Add(input);
    return statuses[input.Type];
}
```

---

# Schedules - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/schedules

> Manage and optimize Scheduled Workflows using the Temporal .NET SDK; Schedule, Create, Backfill, Update, Delete, Describe, List, Pause, Trigger, and use Start Delay options.

This page shows how to do the following:

- [Schedule a Workflow](#schedule-a-workflow)
- [Create a Scheduled Workflow](#create-a-workflow)
- [Backfill a Scheduled Workflow](#backfill-a-scheduled-workflow)
- [Delete a Scheduled Workflow](#delete-a-scheduled-workflow)
- [Describe a Scheduled Workflow](#describe-a-scheduled-workflow)
- [List a Scheduled Workflow](#list-a-scheduled-workflow)
- [Pause a Scheduled Workflow](#pause-a-scheduled-workflow)
- [Trigger a Scheduled Workflow](#trigger-a-scheduled-workflow)
- [Update a Scheduled Workflow](#update-a-scheduled-workflow)
- [Use Start Delay](#start-delay)

## Schedule a Workflow 

Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes.

Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process.

Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy).
If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions.

### Create a Scheduled Workflow 

The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands.

To create a Scheduled Workflow Execution in .NET, use the [CreateScheduleAsync](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_CreateScheduleAsync_System_String_Temporalio_Client_Schedules_Schedule_Temporalio_Client_Schedules_ScheduleOptions_)
method on the Client.
Then pass the Schedule ID and the Schedule object to the method to create a Scheduled Workflow Execution.
Set the Schedule's `Action` property to an instance of `ScheduleActionStartWorkflow` to schedule a Workflow Execution.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var handle = await client.CreateScheduleAsync(
    "my-schedule-id",
    new(
        Action: ScheduleActionStartWorkflow.Create(
            (MyWorkflow wf) => wf.RunAsync(),
            new(id: "my-workflow-id", taskQueue: "my-task-queue")),
        Spec: new()
        {
            Intervals = new List<ScheduleIntervalSpec> { new(Every: TimeSpan.FromDays(5)) },
        }));
```

> **💡 Tip:**
> Schedule Auto-Deletion
>
> Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again.
> The Temporal Service doesn't guarantee when this removal will happen.
>

### Backfill a Scheduled Workflow 

The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time.

To backfill a Scheduled Workflow Execution in .NET, use the [BackfillAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_BackfillAsync_System_Collections_Generic_IReadOnlyCollection_Temporalio_Client_Schedules_ScheduleBackfill__Temporalio_Client_RpcOptions_)
method on the Schedule Handle.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var handle = client.GetScheduleHandle("my-schedule-id");
var now = DateTime.Now;
await handle.BackfillAsync(new List<ScheduleBackfill>
{
    new(
        StartAt: now - TimeSpan.FromDays(30),
        EndAt: now - TimeSpan.FromDays(20),
        Overlap: ScheduleOverlapPolicy.AllowAll),
});
```

### Delete a Scheduled Workflow 

The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule.

To delete a Scheduled Workflow Execution in .NET, use the [DeleteAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_DeleteAsync_Temporalio_Client_RpcOptions_) method on the Schedule Handle.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var handle = client.GetScheduleHandle("my-schedule-id");
await handle.DeleteAsync();
```

### Describe a Scheduled Workflow 

The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs.

To describe a Scheduled Workflow Execution in .NET, use the [DescribeAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_DescribeAsync_Temporalio_Client_RpcOptions_) method on the Schedule Handle.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var handle = client.GetScheduleHandle("my-schedule-id");
var desc = await handle.DescribeAsync();
Console.WriteLine("Schedule info: {0}", desc.Info);
```

### List a Scheduled Workflow 

The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs.

To list all schedules, use the [ListSchedulesAsync()](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_ListSchedulesAsync_Temporalio_Client_Schedules_ScheduleListOptions_) asynchronous method on the Client.
This returns an async enumerable.
If a schedule is added or deleted, it may not be available in the list immediately.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));
await foreach (var desc in client.ListSchedulesAsync())
{
    Console.WriteLine("Schedule info: {0}", desc.Info);
}
```

### Pause a Scheduled Workflow 

The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason.

To pause a Scheduled Workflow Execution in .NET, use the [PauseAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_PauseAsync_System_String_Temporalio_Client_RpcOptions_) method on the Schedule Handle.
You can pass a note to the `PauseAsync()` method to provide a reason for pausing the schedule.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var handle = client.GetScheduleHandle("my-schedule-id");
await handle.PauseAsync("Pausing the schedule for now");
```

### Trigger a Scheduled Workflow 

The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time.

To trigger a Scheduled Workflow Execution in .NET, use the [TriggerAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_TriggerAsync_Temporalio_Client_Schedules_ScheduleTriggerOptions_) method on the Schedule Handle.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var handle = client.GetScheduleHandle("my-schedule-id");
await handle.TriggerAsync();
```

### Update a Scheduled Workflow 

The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval.

To update a Scheduled Workflow Execution in .NET, use the [UpdateAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_UpdateAsync_System_Func_Temporalio_Client_Schedules_ScheduleUpdateInput_Temporalio_Client_Schedules_ScheduleUpdate__Temporalio_Client_RpcOptions_) method on the Schedule Handle.
This method accepts a callback that provides input with the current schedule.
A new schedule can be created and returned from that callback to perform the update.

```csharp
using Temporalio.Client;
using Temporalio.Client.Schedules;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var handle = client.GetScheduleHandle("my-schedule-id");
await handle.UpdateAsync(input =>
{
    var newAction =  ScheduleActionStartWorkflow.Create(
        (MyWorkflow wf) => wf.RunAsync(),
        new(id: "my-workflow-id", taskQueue: "my-task-queue"));
    return new(input.Description.Schedule with { Action = newAction });
});
```

## Use Start Delay 

Use the `StartDelay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule.

Use the `StartDelay` option on `WorkflowOptions` in either the `StartWorkflowAsync()` or `ExecuteWorkflowAsync()` methods in the Client.

```csharp
var handle = await client.StartWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue")
    {
        StartDelay = TimeSpan.FromHours(3),
    });
```

---

# Workflow Timeouts - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/timeouts

> Optimize Workflow Execution with Temporal .NET SDK - Set Workflow Timeouts and Retry Policies efficiently.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts).

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed.
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last.
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task.

These values can be set in the `WorkflowOptions` when calling `StartWorkflowAsync` or `ExecuteWorkflowAsync`.

Available timeouts are:

- ExecutionTimeout
- RunTimeout
- TaskTimeout

```csharp
var result = await client.ExecuteWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue")
    {
        WorkflowExecutionTimeout = TimeSpan.FromMinutes(5),
    });
```

## Workflow retries 

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

The `RetryPolicy` can be set in the `WorkflowOptions` when calling `StartWorkflowAsync` or `ExecuteWorkflowAsync`.

```csharp
var result = await client.ExecuteWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue")
    {
        RetryPolicy = new() { MaximumInterval = TimeSpan.FromSeconds(10) },
    });
```

---

# Timers - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/timers

> Set a Durable Timer using the Temporal .NET SDK. Pause Workflow execution for days or months. Timers are persisted and highly resource-efficient using Workflow.DelayAsync.

This page describes how to set a Durable Timer using the Temporal .NET SDK.

A [Durable Timer](/workflow-execution/timers-delays) is used to pause the execution of a Workflow for a specified duration.
A Workflow can sleep for days or even months.
Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the Durable Timer call will resolve and your code will continue executing.

Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker.

To add a Timer in a Workflow, use `Workflow.DelayAsync`. This is like a deterministic form of `Task.Delay`.

```csharp
// Sleep for 3 days
await Workflow.DelayAsync(TimeSpan.FromDays(3));
```

<!--  ## Skip time in tests 

**How to skip time while testing Workflows using the .NET SDK**

TODO(cretz): This should not be under a durable timers section -->

---

# Versioning - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/versioning

> Use the .NET SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks.

Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress.

The Temporal Platform requires that Workflow Definitions are [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably.

With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version.
There are two primary Versioning methods that you can use:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths.
- [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.

## Worker Versioning

Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching.

## Versioning with Patching 

### Adding a patch

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow Executions, create a patch.

Suppose you have an initial Workflow version called `PrePatchActivity`:

```csharp
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        this.result = await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.PrePatchActivity(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });

        // ...
    }
}
```

Now, you want to update your code to run `PostPatchActivity` instead. This represents your desired end state.

```csharp
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        this.result = await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.PostPatchActivity(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });

        // ...
    }
}
```

The problem is that you cannot deploy `PostPatchActivity` directly until you're certain there are no more running Workflows created using the `PrePatchActivity` code, otherwise you are likely to cause a nondeterminism error.
Instead, you'll need to deploy `PostPatchActivity` and use the [Patched](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Patched_System_String_) method to determine which version of the code to execute.

Patching is a three step process:

1. Use [Patched](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Patched_System_String_) to patch in new code and run it alongside the old code.
2. Remove the old code and apply [DeprecatePatch](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_DeprecatePatch_System_String_).
3. Once all old Workflows have left retention, remove `DeprecatePatch`.

### Patching in new code 

Using `Patched` inserts a marker into the Event History.

During replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker (in this case, `my-patch`). This ensures you can safely deploy code from `PostPatchActivity` as a "feature flag" alongside the original version (`PrePatchActivity`).

```csharp
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        if (Workflow.Patched("my-patch"))
        {
            this.result = await Workflow.ExecuteActivityAsync(
                (MyActivities a) => a.PostPatchActivity(),
                new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
        }
        else
        {
            this.result = await Workflow.ExecuteActivityAsync(
                (MyActivities a) => a.PrePatchActivity(),
                new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
        }

        // ...
    }
}
```

### Deprecating patches 

After all Workflows started with `PrePatchActivity` code have left retention, you can [deprecate the patch](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_DeprecatePatch_System_String_).

Deprecated patches serve as a bridge between the final stage of the patching process and the final state that no longer has patches. They function similarly to regular patches by adding a marker to the Event History. However, this marker won't cause a replay failure when the Workflow code doesn't produce it.

If, during the deployment of `PostPatchActivity`, there are still live Workers running `PrePatchActivity` code and these Workers pick up Workflow histories generated by `PostPatchActivity`, they will safely use the patched branch.

```csharp
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        Workflow.DeprecatePatch("my-patch")
        this.result = await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.PostPatchActivity(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });

        // ...
    }
}
```

### Removing a patch 

You can safely deploy `PostPatchActivity` once all Workflows labeled my-patch or earlier have left retention, based on the previously mentioned assertion.

```csharp
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        this.result = await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.PostPatchActivity(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });

        // ...
    }
}
```

Patching allows you to make changes to currently running Workflows.
It introduces compatible changes without causing non-determinism errors.

### Detailed overview of the `Patched` method 

This video provides an overview of how the `Workflow.Patched()` method works:

<br/>
For a more in-depth explanation, refer to the [Patching](/patching) Encyclopedia entry.

### Workflow cutovers

To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow.

Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes. To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers.

For example, you would duplicate `SayHelloWorkflow` as `SayHelloWorkflowV2`:

```csharp
[Workflow]
public class SayHelloWorkflow
{
    [WorkflowRun]
    # this function contains the original code
}
        
[Workflow]
public class SayHelloWorkflowV2
{
    [WorkflowRun]
    # this function contains the updated code
}
```

You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types:

```csharp
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("greeting-tasks")
        .AddWorkflow<SayHelloWorkflow>()
        .AddWorkflow<SayHelloWorkflowV2>());
```

The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow. This can become impractical over time. This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching.

### Testing a Workflow for replay safety

To determine whether your Workflow your needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/dotnet/best-practices/testing-suite#replay).

---

# Workflow Streams - .NET SDK

Source: https://docs.temporal.io/develop/dotnet/workflows/workflow-streams

> Workflow Streams in .NET supports typed events, Activity publishers, and subscriptions that retain offsets across Continue-As-New.

> **Public Preview**

[Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its
progress in real time.

Workflow Streams is available through the experimental `Temporalio.Extensions.WorkflowStreams` package in
[.NET SDK v1.20.0](https://github.com/temporalio/sdk-dotnet/releases/tag/1.20.0) and later. Install the extension package:

```bash
dotnet add package Temporalio.Extensions.WorkflowStreams --version 1.20.0
```

See the [basic publish/subscribe sample](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/BasicPublishSubscribe)
for a complete Worker and client setup using the released package.

## Enable streaming on a Workflow

Enable streaming by constructing one `WorkflowStream` in a [`[WorkflowInit]`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowInitAttribute.html)
constructor. The constructor runs before message handlers can be dispatched, so the stream can register its publish
Signal, poll Update, and offset Query handlers before the first request arrives.

```csharp
using Temporalio.Extensions.WorkflowStreams;
using Temporalio.Workflows;

public record OrderInput(
    string OrderId,
    WorkflowStreamState? StreamState = null);

[Workflow]
public class OrderWorkflow
{
    private readonly WorkflowStream stream;

    [WorkflowInit]
    public OrderWorkflow(OrderInput input) => stream = new(input.StreamState);

    [WorkflowRun]
    public async Task RunAsync(OrderInput input)
    {
        // ... rest of the Workflow
    }
}
```

Pass `null` on the first run. After Continue-As-New, pass the `WorkflowStreamState` captured from the previous run. See
[Stream from long-running Workflows](#continue-as-new) for the complete rollover sequence.

Construct exactly one `WorkflowStream` per Workflow. Each construction registers the same protocol handler names, so a
second instance replaces the first instance's handlers.

## Publish from a Workflow

Bind a [topic name](/workflow-streams#topics) to an event type with `GetTopic<T>()`, then call `Publish()` on the returned
handle. The generic type gives publish call sites compile-time type checking and tells typed subscribers how to decode
the topic.

```csharp
public record StatusEvent(string State, int Progress = 0, string? Detail = null);

[Workflow]
public class OrderWorkflow
{
    private readonly WorkflowStream stream;
    private readonly WorkflowStreamTopicHandle<StatusEvent> status;

    [WorkflowInit]
    public OrderWorkflow(OrderInput input)
    {
        stream = new(input.StreamState);
        status = stream.GetTopic<StatusEvent>("status");
    }

    [WorkflowRun]
    public async Task RunAsync(OrderInput input)
    {
        status.Publish(new("validating", Detail: "checking inventory"));
        await ValidateOrderAsync(input.OrderId);

        status.Publish(new("charging", 33, "authorizing payment"));
        await ChargePaymentAsync(input.OrderId);

        status.Publish(new("shipping", 66, "dispatching to warehouse"));
        await DispatchOrderAsync(input.OrderId);

        status.Publish(new("completed", 100));
        await Workflow.DelayAsync(TimeSpan.FromSeconds(30));
    }
}
```

`Publish()` runs the Workflow's payload converter and appends the resulting `Payload` directly to the in-memory log. A
Workflow-side publication isn't buffered and doesn't need to be flushed.
The delay keeps poll Updates available for subscribers to receive the final status. See
[Close the stream](#closing-the-stream) for an acknowledgment-based alternative.

The type binding belongs to the handle, not to the stream protocol. If separate publishers use different types for the
same topic name, publication succeeds and a typed subscriber can fail while decoding the mismatched value.

## Publish from a client

Any process with a Temporal Client and the target Workflow ID can publish by constructing a `WorkflowStreamClient`.
This includes HTTP services, Workflow starters, scripts, Activities from other Workflows, and standalone Activities.

Use `await using` so the client flushes buffered events and releases its publishing resources when the scope exits:

```csharp
using Temporalio.Client;
using Temporalio.Extensions.WorkflowStreams;

public static async Task PublishStatusAsync(
    ITemporalClient temporalClient,
    string workflowId,
    CancellationToken cancellationToken)
{
    await using var streamClient = new WorkflowStreamClient(
        temporalClient,
        workflowId,
        new()
        {
            BatchInterval = TimeSpan.FromMilliseconds(200),
        });

    var status = streamClient.GetTopic<StatusEvent>("status");
    status.Publish(new("started"));
    status.Publish(new("completed", 100), forceFlush: true);
    await streamClient.FlushAsync(cancellationToken);
}
```

By default, the client sends buffered publications every two seconds. Set `MaxBatchSize` to flush after the buffer
reaches a specific item count. A value of `0`, the default, disables size-based flushing.

Pass `forceFlush: true` when an event should wake the background flusher without waiting for the next interval. The
`Publish()` call still returns after buffering the event; it doesn't wait for the Signal to reach the Temporal Service.
Use `FlushAsync()` when subsequent work must wait until all earlier publications have been acknowledged.

When an event originates in an Activity, publish it from the Activity instead of returning it for the Workflow to
forward. `FromActivity()` gets the Temporal Client, parent Workflow ID, and payload converter from the current Activity
context:

```csharp
using Temporalio.Activities;
using Temporalio.Extensions.WorkflowStreams;

public record ProgressEvent(string Message);

public static class OrderActivities
{
    [Activity]
    public static async Task ProcessOrderAsync(IEnumerable<string> steps)
    {
        await using var streamClient = WorkflowStreamClient.FromActivity();
        var progress = streamClient.GetTopic<ProgressEvent>("progress");

        foreach (var step in steps)
        {
            progress.Publish(new(step));
            ActivityExecutionContext.Current.Heartbeat(step);
        }
    }
}
```

`FromActivity()` requires an Activity started by a Workflow. For a standalone Activity, pass the target Workflow ID in
the Activity input and construct `WorkflowStreamClient` with `ActivityExecutionContext.Current.TemporalClient`.

Publishing from the Activity lets subscribers see partial output from failed and retried attempts without adding that
output to the Workflow's durable application state. Each Activity attempt creates a new publisher, so subscribers can
receive partial output from an unsuccessful attempt before receiving the retried attempt's output. See
[How events are delivered](/workflow-streams#how-events-are-delivered).

`Publish()` applies no backpressure. Client-side publication appends to an in-memory buffer, while subscribers read the
Workflow log on their own schedule. If a publisher produces events faster than batches reach the Temporal Service, its
buffer grows. Apply an application-specific policy before `Publish()` if you need to block, drop, sample, or reject
events when the publisher outruns the network.

## Subscribe

Subscribe from any process that has a Temporal Client and the Workflow ID. A typed topic handle returns a reusable
`IAsyncEnumerable<WorkflowStreamItem<T>>`. Each enumeration keeps its own offset and polling state.

```csharp
using Temporalio.Client;
using Temporalio.Extensions.WorkflowStreams;

public static async Task WatchOrderAsync(
    ITemporalClient temporalClient,
    string workflowId,
    CancellationToken cancellationToken)
{
    await using var streamClient = new WorkflowStreamClient(temporalClient, workflowId);
    var status = streamClient.GetTopic<StatusEvent>("status");

    await foreach (var item in status.SubscribeAsync().WithCancellation(cancellationToken))
    {
        Console.WriteLine(
            $"[{item.Value.Progress,3}%] {item.Value.State}: {item.Value.Detail}");
        if (item.Value.State == "completed")
        {
            break;
        }
    }
}
```

Pass the next global offset to `SubscribeAsync(fromOffset)` when reconnecting. The iterator follows Continue-As-New
chains, repolls immediately when a response reaches the approximate 1 MB page limit, and advances to the retained
beginning of the log if its requested offset was truncated.

Canceling the enumeration cancels its in-flight RPC and throws `OperationCanceledException`. Disposing the owning
`WorkflowStreamClient` ends its active enumerations without an exception. An enumeration also ends cleanly when the
Workflow reaches a terminal state.

### Subscribe to heterogeneous topics

A typed subscription decodes every item as one type. To consume topics with different payload types, call the untyped
`SubscribeAsync()` overload with a topic filter. It yields `WorkflowStreamItem` values containing raw Temporal
`Payload`s. Dispatch on `Topic` and decode each payload with the matching type:

```csharp
using Temporalio.Converters;

var options = new WorkflowStreamSubscribeOptions
{
    Topics = new[] { "status", "progress" },
};
var converter = temporalClient.Options.DataConverter.WithSerializationContext(
    new ISerializationContext.Workflow(temporalClient.Options.Namespace, workflowId)).PayloadConverter;

await foreach (var item in streamClient.SubscribeAsync(options))
{
    if (item.Topic == "status")
    {
        var evt = converter.ToValue<StatusEvent>(item.Payload);
        Console.WriteLine($"[status] {evt.State}: {evt.Detail}");
    }
    else if (item.Topic == "progress")
    {
        var evt = converter.ToValue<ProgressEvent>(item.Payload);
        Console.WriteLine($"[progress] {evt.Message}");
    }
}
```

An empty topic collection subscribes to every topic. The empty string represents the cross-SDK no-topic value.
Use the target Workflow's Namespace and Workflow ID for the converter's serialization context, as typed subscriptions
do automatically. Payload codecs apply to the enclosing Signal or Update payload; don't apply them again to each item.

### Close the stream 

A subscription doesn't know which event your application considers final. Publish an in-band terminal event and have
the subscriber break when it receives that event. In the preceding example, the `completed` status is the terminal
event.

Keep the Workflow open briefly after publishing the terminal event. If the Workflow completes immediately, an in-flight
poll might close before the subscriber receives the event. You can provide this overlap in either of these ways:

- **Fixed delay.** Call `Workflow.DelayAsync()` after publishing the terminal event and before returning.

  ```csharp
  status.Publish(new("completed", 100));
  await Workflow.DelayAsync(TimeSpan.FromSeconds(30));
  return result;
  ```

- **Acknowledgment handshake.** Have the subscriber Signal the Workflow after it receives the terminal event. Wait for
  that Signal with a timeout so the Workflow still completes when no subscriber is attached.

  ```csharp
  private bool subscriberAcknowledged;

  [WorkflowSignal]
  public Task AcknowledgeStreamAsync()
  {
      subscriberAcknowledged = true;
      return Task.CompletedTask;
  }

  // After publishing the terminal event:
  await Workflow.WaitConditionAsync(
      () => subscriberAcknowledged,
      TimeSpan.FromSeconds(30));
  return result;
  ```

The subscription ends cleanly for every terminal Workflow status. If the application must distinguish completion from
failure, cancellation, termination, or timeout, describe the Workflow after the enumeration ends and inspect its
status.

## Stream from long-running Workflows 

Use [Continue-As-New](/workflow-execution/continue-as-new) to keep Event History bounded for a Workflow that runs for
hours or accumulates many events. Subscribers automatically follow the new run, but you must carry the stream state
through the Workflow input so they don't see a gap.

```csharp
using Temporalio.Extensions.WorkflowStreams;
using Temporalio.Workflows;

public record PipelineInput(
    int ItemsProcessed = 0,
    WorkflowStreamState? StreamState = null);

[Workflow]
public class PipelineWorkflow
{
    private readonly WorkflowStream stream;

    [WorkflowInit]
    public PipelineWorkflow(PipelineInput input) => stream = new(input.StreamState);

    [WorkflowRun]
    public async Task RunAsync(PipelineInput input)
    {
        var itemsProcessed = input.ItemsProcessed;
        var progress = stream.GetTopic<ProgressEvent>("progress");

        while (true)
        {
            await ProcessNextItemAsync();
            itemsProcessed++;
            progress.Publish(new($"processed {itemsProcessed} items"));

            if (Workflow.ContinueAsNewSuggested)
            {
                var streamState = await stream.CaptureStateForContinueAsNewAsync();
                throw Workflow.CreateContinueAsNewException(
                    (PipelineWorkflow workflow) => workflow.RunAsync(
                        new(itemsProcessed, streamState)));
            }
        }
    }
}
```

`CaptureStateForContinueAsNewAsync()` detaches waiting pollers, waits for message handlers to finish, and captures the
retained log and publisher deduplication state. It also disables Workflow-side publication. Create and throw the
Continue-As-New exception immediately after the snapshot.

The snapshot contains the entire retained log. Use `stream.Truncate(offset)` after all required consumers have advanced
to keep it small. If individual events contain large data, store that data in [External Storage](/external-storage) and
publish a reference instead.

## Configure the deduplication window

See [How events are delivered](/workflow-streams#how-events-are-delivered) for publisher and subscriber guarantees. Two
settings bound the deduplication window:

- `WorkflowStreamOptions.PublisherTtl` controls how long publisher sequence state remains in a Continue-As-New
  snapshot. Its default is 15 minutes. A publisher that returns after its entry expires can produce a duplicate.
- `WorkflowStreamClientOptions.MaxRetryDuration` controls how long a client retries an ambiguously delivered batch. Its
  default is 10 minutes. Keep this value lower than the publisher TTL.

```csharp
// Workflow initialization
stream = new(
    input.StreamState,
    new() { PublisherTtl = TimeSpan.FromMinutes(30) });

// External client or Activity
await using var streamClient = new WorkflowStreamClient(
    temporalClient,
    workflowId,
    new() { MaxRetryDuration = TimeSpan.FromMinutes(20) });
```

When the retry duration expires, the client drops the ambiguous batch and reports `FlushTimeoutException` from
`FlushAsync()` or asynchronous disposal. The batch might already be in the Workflow log, or it might be lost. Later
batches continue with a new sequence.

## Follow best practices

- **Construct one stream during Workflow initialization.** This registers protocol handlers before the first request
  and prevents another stream from replacing them.
- **Dispose publishing clients asynchronously.** Use `await using` or call `DisposeAsync()` so the last buffer is
  drained. Publishing after disposal throws `ObjectDisposedException`.
- **Use consistent topic types.** Generic topic handles provide local compile-time checking, but publishers don't share
  their bindings. A subscriber reports a conversion error if a publisher sends a different type.
- **Keep payload conversion consistent.** Stream items use the payload converter. Payload codecs run once on the Signal
  or Update envelope, not once per item. `FromActivity()` subscriptions aren't compatible with custom converters that
  require the serialization context used for deserialization to match publication's Activity context.
- **Subscribe outside the hosting Workflow.** The Workflow owns the durable log but doesn't consume it. This keeps
  partial output from retried Activities separate from the Workflow's successful Activity result.

## Example: Stream LLM output 

An Activity can publish model output as it arrives while the Workflow waits for the Activity result. The subscriber
renders deltas, clears output when an Activity retry begins, and stops on a terminal event. This pattern works for a
terminal client, desktop UI, or Server-Sent Events endpoint.

The following example uses the OpenAI .NET client, as in the
[LLM token streaming sample](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/LlmTokenStreaming).
Install the client package and set `OPENAI_API_KEY` before running it:

```bash
dotnet add package OpenAI --version 2.13.0
```

**LlmActivities.cs**

```csharp
using System.ClientModel;
using System.ClientModel.Primitives;
using System.Text;
using OpenAI;
using OpenAI.Chat;
using Temporalio.Activities;
using Temporalio.Extensions.WorkflowStreams;

public record TextDelta(string Text);
public record RetryEvent(int Attempt);
public record TextComplete(string FullText);

public static class LlmActivities
{
    [Activity]
    public static async Task<string> StreamCompletionAsync(LlmInput input)
    {
        var apiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY");
        if (string.IsNullOrWhiteSpace(apiKey))
        {
            throw new InvalidOperationException("OPENAI_API_KEY must be set");
        }

        var context = ActivityExecutionContext.Current;
        await using var streamClient = WorkflowStreamClient.FromActivity(
            new() { BatchInterval = TimeSpan.FromMilliseconds(200) });

        var deltas = streamClient.GetTopic<TextDelta>("delta");
        var retries = streamClient.GetTopic<RetryEvent>("retry");
        var complete = streamClient.GetTopic<TextComplete>("complete");

        if (context.Info.Attempt > 1)
        {
            retries.Publish(new(context.Info.Attempt), forceFlush: true);
        }

        // Disable provider retries and let the Activity Retry Policy own retries.
        var clientOptions = new OpenAIClientOptions
        {
            RetryPolicy = new ClientRetryPolicy(0),
        };
        var chatClient = new ChatClient(input.Model, new ApiKeyCredential(apiKey), clientOptions);
        var fullText = new StringBuilder();
        var updates = chatClient.CompleteChatStreamingAsync(
            [new UserChatMessage(input.Prompt)],
            cancellationToken: context.CancellationToken);
        await foreach (var update in updates)
        {
            foreach (var contentPart in update.ContentUpdate)
            {
                if (string.IsNullOrEmpty(contentPart.Text))
                {
                    continue;
                }
                deltas.Publish(new(contentPart.Text));
                fullText.Append(contentPart.Text);
            }
        }

        var result = fullText.ToString();
        complete.Publish(new(result), forceFlush: true);
        await streamClient.FlushAsync();
        return result;
    }
}
```

**LlmWorkflow.workflow.cs**

```csharp
using Temporalio.Extensions.WorkflowStreams;
using Temporalio.Workflows;

public record LlmInput(
    string Prompt,
    string Model = "gpt-4o-mini",
    WorkflowStreamState? StreamState = null);

[Workflow]
public class LlmWorkflow
{
    [WorkflowInit]
    public LlmWorkflow(LlmInput input) => _ = new WorkflowStream(input.StreamState);

    [WorkflowRun]
    public async Task<string> RunAsync(LlmInput input)
    {
        var result = await Workflow.ExecuteActivityAsync(
            () => LlmActivities.StreamCompletionAsync(input),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(2) });

        // Give the subscriber's in-flight poll time to receive the terminal event.
        await Workflow.DelayAsync(TimeSpan.FromSeconds(30));
        return result;
    }
}
```

**LlmSubscriber.cs**

```csharp
using System.Text;
using Temporalio.Client;
using Temporalio.Converters;
using Temporalio.Extensions.WorkflowStreams;

public static async Task<string> StreamResponseAsync(
    ITemporalClient temporalClient,
    string workflowId,
    CancellationToken cancellationToken)
{
    await using var streamClient = new WorkflowStreamClient(temporalClient, workflowId);
    var converter = temporalClient.Options.DataConverter.WithSerializationContext(
        new ISerializationContext.Workflow(temporalClient.Options.Namespace, workflowId)).PayloadConverter;
    var output = new StringBuilder();
    var options = new WorkflowStreamSubscribeOptions
    {
        Topics = new[] { "delta", "retry", "complete" },
    };

    await foreach (var item in streamClient.SubscribeAsync(options).
        WithCancellation(cancellationToken))
    {
        if (item.Topic == "retry")
        {
            output.Clear();
            Render(output.ToString());
        }
        else if (item.Topic == "delta")
        {
            output.Append(converter.ToValue<TextDelta>(item.Payload).Text);
            Render(output.ToString());
        }
        else if (item.Topic == "complete")
        {
            _ = converter.ToValue<TextComplete>(item.Payload);
            break;
        }
    }

    return output.ToString();
}
```

The Activity publishes a `RetryEvent` when its attempt number is greater than one. This tells the subscriber to discard
partial output from the previous attempt. Deltas use the 200 millisecond batch interval to reduce Signal volume;
retries and the terminal event force a flush. Supply your own `Render()` function to update your UI, or use the sample's
terminal renderer. The Workflow uses the fixed-delay closing pattern; use an
[acknowledgment handshake](#closing-the-stream) when it should return as soon as a subscriber confirms receipt.

## See also

The [Workflow Streams samples](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams) contain six
runnable scenarios:

| Sample | What it demonstrates |
| --- | --- |
| [Basic publish/subscribe](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/BasicPublishSubscribe) | Publish from a Workflow and an Activity, and consume heterogeneous topics. |
| [Concurrent subscriptions](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/ConcurrentSubscriptions) | Consume streams from two Workflows concurrently. |
| [Reconnecting subscriber](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/ReconnectingSubscriber) | Resume from a saved global offset after disconnecting. |
| [External publisher](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/ExternalPublisher) | Publish from an external client across Continue-As-New. |
| [Bounded log](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/BoundedLog) | Truncate retained events and wait for subscriber acknowledgments before closing. |
| [LLM token streaming](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkflowStreams/LlmTokenStreaming) | Stream OpenAI output and reset the subscriber's display on Activity retries. |

- [`Temporalio.Extensions.WorkflowStreams` API reference](https://dotnet.temporal.io/api/Temporalio.Extensions.WorkflowStreams.html).
- [Workflow message passing](/develop/dotnet/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on.

---

# Environment configuration

Source: https://docs.temporal.io/develop/environment-configuration

> Configure Temporal Clients using environment variables and TOML configuration files

Temporal CLI and SDKs support configuring a Temporal Client using environment variables and TOML configuration files,
rather than setting connection options programmatically in your code. This decouples connection settings from
application logic, making it easier to manage different environments such as development, staging, and production
without code changes.

For a list of all available configuration settings, their corresponding environment variables, and TOML file paths,
refer to [Temporal Client Environment Configuration Reference](../references/client-environment-configuration).

## Configuration methods

You can configure your client using a TOML file, environment variables, or a combination of both. The configuration is
loaded with a specific order of precedence:

1. Environment variables: These have the highest precedence. If an environment variable defines a setting, it will
   always override any value set in a configuration file. This makes it easy to provide secrets in dynamic environments.
2. TOML configuration file: A TOML file can be used to define one or more configuration profiles. This file is located
   by checking the following sources in order:
   1. The path specified by the `TEMPORAL_CONFIG_FILE` environment variable.
   2. The default configuration path for your operating system:
      - Linux: `~/.config/temporalio/temporal.toml`
      - macOS: `$HOME/Library/Application Support/temporalio/temporal.toml`
      - Windows: `%AppData%\temporalio\temporal.toml`

## TOML file configuration

You can use configuration profiles to maintain separate configurations within a single file for different environments.
The Temporal client uses the `default` profile unless you specify another via the `TEMPORAL_PROFILE` environment
variable or in the SDK's load options. If a requested profile doesn't exist, the application will return an error.

Here is an example `temporal.toml` file that defines two profiles: `default` for local development and `prod` for
production.

```toml
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Optional: Add custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
# TLS is auto-enabled when this TLS config or API key is present, but you can configure it explicitly
# disabled = false
# Use certificate files for mTLS
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment = "production"
service-version = "v1.2.3"

# Staging profile with inline certificate data
[profile.staging]
address = "staging.temporal.example.com:7233"
namespace = "staging"

[profile.staging.tls]
# Example of providing certificate data directly (base64 or PEM format)
client_cert_data = """-----BEGIN CERTIFICATE-----
MIICertificateDataHere...
-----END CERTIFICATE-----"""
client_key_data = """-----BEGIN PRIVATE KEY-----
MIIPrivateKeyDataHere...
-----END PRIVATE KEY-----"""
```

Select a concept to highlight the matching connection settings in a Cloud profile:

  annotations={[
    {
      label: 'Address',
      description:
        'Temporal Service host and port. For Temporal Cloud, use your Namespace endpoint (for example, your-namespace.account.tmprl.cloud:7233).',
      lines: [2],
    },
    {
      label: 'Namespace',
      description:
        'Namespace this Client connects to. On Temporal Cloud, include the account suffix when your tooling expects it.',
      lines: [3],
    },
    {
      label: 'API key',
      description:
        'Authenticates the Client to Temporal Cloud. Prefer environment variables for secrets in real deployments; TOML is fine for local profiles.',
      lines: [4],
    },
    {
      label: 'TLS',
      description:
        'Optional mTLS settings. TLS is often auto-enabled when an API key or TLS block is present; set certificate paths for mutual TLS.',
      lines: [6, 7, 8],
    },
  ]}
>
```toml
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

[profile.prod.tls]
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path = "/etc/temporal/certs/client.key"
```

The [`temporal cloud login`](/cli/cloud#interactive-login) command also writes to this file. When you run `temporal cloud login --profile prod`, the OAuth token is stored in the specified profile automatically. Subsequent commands that use that profile read the token from the TOML file to authenticate with Temporal Cloud.

## CLI integration

The Temporal CLI tool includes `temporal config` commands that allow you to read and write to the TOML configuration
file. This provides a convenient way to manage your connection profiles without manually editing the file. Refer to
[Temporal CLI Reference - `temporal config`](../cli/command-reference/config.mdx) for more details.

- `temporal config get <property>`: Reads a specific value from the current profile.
- `temporal config set <property> <value>`: Sets a property in the current profile.
- `temporal config delete <property>`: Deletes a property from the current profile.
- `temporal config list`: Lists all available profiles in the config file.

These CLI commands directly manipulate the `temporal.toml` file. This differs from the SDKs, which only _read_ from the
file and environment at runtime to establish a client connection. You can select a profile for the CLI to use with the
`--profile` flag. For example, `temporal --profile prod ...`.

Connection flags on other `temporal` commands (for example, `--address`, `--namespace`, `--api-key`) sit above both
environment variables and the TOML file in precedence. The full order for the CLI is:

1. Command-line flags
2. Environment variables
3. TOML configuration file (active profile)

So `temporal workflow list --address localhost:7233` uses `localhost:7233` even if `TEMPORAL_ADDRESS` or the active
profile's `address` sets something else.

The following code blocks provide copy-paste-friendly examples for setting up CLI profiles for both local development
and Temporal Cloud.

**Local + Prod with Cloud API key**

This example shows how to set up a default profile for local development and a `prod` profile for Temporal Cloud using
an API key.

```bash
# (Optional) initialize the default profile for local development
temporal config set --prop address --value "localhost:7233"
temporal config set --prop namespace --value "default"

# Configure a Temporal Cloud profile that authenticates with an API key
temporal --profile prod config set --prop address --value "<namespace_id>.<account_id>.tmprl.cloud:7233"
temporal --profile prod config set --prop namespace --value "<namespace_id>.<account_id>"
temporal --profile prod config set --prop api_key --value "<your-api-key>"
```

**API key + advanced options**

This example shows how to set up a more advanced Temporal Cloud profile with TLS overrides and custom gRPC metadata.

```bash
# Base API key properties (replace the placeholders)
temporal --profile prod config set --prop address --value "<namespace_id>.<account_id>.tmprl.cloud:7233"
temporal --profile prod config set --prop namespace --value "<namespace_id>.<account_id>"
temporal --profile prod config set --prop api_key --value "<your-api-key>"

# Optional TLS overrides (only needed when you must pin certs or tweak SNI)
temporal --profile prod config set --prop tls.server_name --value "<namespace_id>.<account_id>"
temporal --profile prod config set --prop tls.ca_cert_path --value "/path/to/ca.pem"

# Optional gRPC metadata for observability or routing
temporal --profile prod config set --prop grpc_meta.environment --value "production"
temporal --profile prod config set --prop grpc_meta.service-version --value "v1.2.3"
```

## Load configuration profile and environment variables

If you don't specify a profile, the SDKs load the `default` profile and the environment variables. If you haven't set
`TEMPORAL_CONFIG_FILE`, the SDKs will look for the configuration file in the default location. Refer to
[Configuration methods](#configuration-methods) for the default locations for your operating system.

No matter what profile you choose to load, environment variables are always loaded when you use the APIs in the
environment configuration package to load Temporal Client connection options. They always take precedence over TOML file
settings in the profiles.

**Python**

To load the `default` profile along with any environment variables in Python, use the `ClientConfigProfile.load()`
method from the `temporalio.envconfig` package.

```python {7-8}
import asyncio
from temporalio.client import Client
from temporalio.envconfig import ClientConfigProfile

async def main():
    # Load the "default" profile from default locations and environment variables.
    default_profile = ClientConfigProfile.load()
    connect_config = default_profile.to_client_connect_config()

    # Connect to the client using the loaded configuration.
    client = await Client.connect(**connect_config)
    print(f"✅ Client connected to {client.service_client.config.target_host} in namespace '{client.namespace}'")

if __name__ == "__main__":
    asyncio.run(main())
```

**Go**

To load the `default` profile along with any environment variables in Go, use the
`envconfig.MustLoadDefaultClientOptions()` function from the `temporalio.envconfig` package.

```go {13}
package main

import (
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
)

func main() {
	// Loads the "default" profile from the standard location and environment variables.
	c, err := client.Dial(envconfig.MustLoadDefaultClientOptions())
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}
	defer c.Close()

	fmt.Printf("✅ Connected to Temporal Service")
}
```

**Ruby**

To load the `default` profile along with any environment variables in Ruby, use the
`EnvConfig::ClientConfig.load_client_connect_options()` method from the `temporalio.env_config` package.

```Ruby {16-18}
require 'temporalio/client'
require 'temporalio/env_config'

def main
  puts '--- Loading default profile from config.toml ---'

  # For this sample to be self-contained, we explicitly provide the path to
  # the config.toml file included in this directory.
  # By default though, the config.toml file will be loaded from
  # ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
  config_file = File.join(__dir__, 'config.toml')

  # load_client_connect_options is a helper that loads a profile and prepares
  # the configuration for Client.connect. By default, it loads the
  # "default" profile.
  args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options(
    config_source: Pathname.new(config_file)
  )

  puts "Loaded 'default' profile from #{config_file}."
  puts "  Address: #{args[0]}"
  puts "  Namespace: #{args[1]}"
  puts "  gRPC Metadata: #{kwargs[:rpc_metadata]}"

  puts "\nAttempting to connect to client..."
  begin
    client = Temporalio::Client.connect(*args, **kwargs)
    puts '✅ Client connected successfully!'
    sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new)
    puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}"
  rescue StandardError => e
    puts "❌ Failed to connect: #{e}"
  end
end
```

**.NET**

To load the `default` profile along with any environment variables in .NET C#, use the
`ClientEnvConfig.LoadClientConnectOptions()` method from the `Temporalio.Client.EnvConfig` package.

```csharp {22,27-30}
using Temporalio.Client;
using Temporalio.Client.EnvConfig;

namespace TemporalioSamples.EnvConfig;

/// <summary>
/// Sample demonstrating loading the default environment configuration profile
/// from a TOML file.
/// </summary>
public static class LoadFromFile
{
    public static async Task RunAsync()
    {
        Console.WriteLine("--- Loading default profile from config.toml ---");

        try
        {
            // For this sample to be self-contained, we explicitly provide the path to
            // the config.toml file included in this directory.
            // By default though, the config.toml file will be loaded from
            // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
            var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml");

            // LoadClientConnectOptions is a helper that loads a profile and prepares
            // the config for TemporalClient.ConnectAsync. By default, it loads the
            // "default" profile.
            var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions
            {
                ConfigSource = DataSource.FromPath(configFile),
            });

            Console.WriteLine($"Loaded 'default' profile from {configFile}.");
            Console.WriteLine($"  Address: {connectOptions.TargetHost}");
            Console.WriteLine($"  Namespace: {connectOptions.Namespace}");
            if (connectOptions.RpcMetadata?.Count > 0)
            {
                Console.WriteLine($"  gRPC Metadata: {string.Join(", ", connectOptions.RpcMetadata.Select(kv => $"{kv.Key}={kv.Value}"))}");
            }

            Console.WriteLine("\nAttempting to connect to client...");

            var client = await TemporalClient.ConnectAsync(connectOptions);
            Console.WriteLine("✅ Client connected successfully!");

            // Test the connection by checking the service
            var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new());
            Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            Console.WriteLine($"❌ Failed to connect: {ex.Message}");
        }
    }
}
```

**TypeScript**

To load the `default` profile along with any environment variables in TypeScript, use the `loadClientConnectConfig` helper from `@temporalio/envconfig` package.

[env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts)
```ts {17-19,28-29}
import { Connection, Client } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { resolve } from 'path';

async function main() {
  console.log('--- Loading default profile from config.toml ---');

  // For this sample to be self-contained, we explicitly provide the path to
  // the config.toml file included in this directory.
  // By default though, the config.toml file will be loaded from
  // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
  const configFile = resolve(__dirname, '../config.toml');

  // loadClientConnectConfig is a helper that loads a profile and prepares
  // the configuration for Connection.connect and Client. By default, it loads the
  // "default" profile.
  const config = loadClientConnectConfig({
    configSource: { path: configFile },
  });

  console.log(`Loaded 'default' profile from ${configFile}.`);
  console.log(`  Address: ${config.connectionOptions.address}`);
  console.log(`  Namespace: ${config.namespace}`);
  console.log(`  gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`);

  console.log('\nAttempting to connect to client...');
  try {
    const connection = await Connection.connect(config.connectionOptions);
    const client = new Client({ connection, namespace: config.namespace });
    console.log('✅ Client connected successfully!');
    await connection.close();
  } catch (err) {
    console.log(`❌ Failed to connect: ${err}`);
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

**Java**

To load the `default` profile along with any environment variables in Java, use the `ClientConfigProfile.load` method
from the `envconfig` package. This method will load the `default` profile from the default location and any environment
variables. Environment variables take precedence over the configuration file settings.

Then use `profile.toWorkflowServiceStubsOptions` and `profile.toWorkflowClientOptions` to convert the profile to
`WorkflowServiceStubsOptions` and `WorkflowClientOptions` respectively. Then use `WorkflowClient.newInstance` to create
a Temporal Client.

```java
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.envconfig.LoadClientConfigProfileOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import java.nio.file.Paths;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoadFromFile {

  private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class);

  public static void main(String[] args) {
    try {

      ClientConfigProfile profile = ClientConfigProfile.load(LoadClientConfigProfileOptions.newBuilder().build());

      WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions();
      WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions();

      try {
        // Create the workflow client using the loaded configuration
        WorkflowClient client =
            WorkflowClient.newInstance(
                WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions);

        // Test the connection by getting system info
        var systemInfo =
            client
                .getWorkflowServiceStubs()
                .blockingStub()
                .getSystemInfo(
                    io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance());

        logger.info("✅ Client connected successfully!");
        logger.info("   Server version: {}", systemInfo.getServerVersion());

      } catch (Exception e) {
        logger.error("❌ Failed to connect: {}", e.getMessage());
      }

    } catch (Exception e) {
      logger.error("Failed to load configuration: {}", e.getMessage(), e);
      System.exit(1);
    }
  }
}
```

## Load configuration from a custom path

To load configuration from a non-standard file location without relying on the `TEMPORAL_CONFIG_FILE` environment
variable, you can use a function from the `temporalio.envconfig` package. The specific method you need to call depends
on the SDK you are using.

This is useful if you store application-specific configurations separately. Loading connection options using this method
will still respect environment variables, which take precedence over the file settings.

**Python**

To load a specific profile from a custom path in Python, use the `ClientConfig.load_client_connect_config()` method with
the `config_file` parameter. In this example, we construct the path to a `config.toml` file located in the same
directory as the script.

After loading the connection options, you can override specific settings programmatically before passing them to
`Client.connect()`.

```py {12-13,21-23}
import asyncio
from pathlib import Path
from temporalio.client import Client
from temporalio.envconfig import ClientConfig

async def main():
    """
    Demonstrates loading a named profile and overriding values programmatically.
    """
    print("--- Loading 'staging' profile with programmatic overrides ---")

    config_file = Path(__file__).parent / "config.toml"
    profile_name = "staging"

    print(
        "The 'staging' profile in config.toml has an incorrect address (localhost:9999)."
    )
    print("We'll programmatically override it to the correct address.")

    # Load the 'staging' profile.
    connect_config = ClientConfig.load_client_connect_config(
        profile=profile_name,
        config_file=str(config_file),
    )

    # Override the target host to the correct address.
    # This is the recommended way to override configuration values.
    connect_config["target_host"] = "localhost:7233"

    print(f"\nLoaded '{profile_name}' profile from {config_file} with overrides.")
    print(
        f"  Address: {connect_config.get('target_host')} (overridden from localhost:9999)"
    )
    print(f"  Namespace: {connect_config.get('namespace')}")

    print("\nAttempting to connect to client...")
    try:
        await Client.connect(**connect_config)  # type: ignore
        print("✅ Client connected successfully!")
    except Exception as e:
        print(f"❌ Failed to connect: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```

**Go**

To load a specific profile from a custom filepath in Go, use the `envconfig.LoadClientOptions()` function with the
`ConfigFilePath` field set in the `LoadClientOptionsRequest` struct. Use the `ConfigFileProfile` field to specify the
profile name.

After loading the connection options, you can override specific settings programmatically before passing them to
`client.Dial()`. Refer to the [GO SDK API documentation](https://pkg.go.dev/go.temporal.io/sdk/contrib/envconfig) for
all available options.

```go {14-16}
package main

import (
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
)

func main() {
  // Load a specific profile from the TOML config file.
  // This requires a [profile.prod] section in your config.
  opts, err := envconfig.LoadClientOptions(envconfig.LoadClientOptionsRequest{
    ConfigFileProfile: "prod",
    ConfigFilePath:    "/Users/yourname/.config/my-app/temporal.toml",
  })
  if err != nil {
    log.Fatalf("Failed to load 'prod' profile: %v", err)
  }

  // Programmatically override the Namespace value.
  opts.Namespace = "new-namespace"

  c, err := client.Dial(opts)
  if err != nil {
    log.Fatalf("Failed to connect using 'prod' profile: %v", err)
  }
  defer c.Close()

  fmt.Printf("✅ Connected to Temporal namespace %q on %s using 'prod' profile\n", c.Options().Namespace, c.Options().HostPort)
}
```

**Ruby**

To load a specific profile from a custom path in Ruby, use the `EnvConfig::ClientConfig.load_client_connect_options()`
method with the `config_source` parameter. In this example, we construct the path to a `config.toml` file located in the
same directory as the script. Use the `profile` parameter to specify the profile name.

After loading the connection options, you can override specific settings programmatically before passing them to
`Client.connect()`. Refer to the [Ruby SDK API documentation](https://ruby.temporal.io/Temporalio/EnvConfig.html) for
all available options.

```Ruby {7-8,14-16}
require 'temporalio/client'
require 'temporalio/env_config'

def main
  puts "--- Loading 'staging' profile with programmatic overrides ---"

  config_file = File.join(__dir__, 'config.toml')
  profile_name = 'staging'

  puts "The 'staging' profile in config.toml has an incorrect address (localhost:9999)."
  puts "We'll programmatically override it to the correct address."

  # Load the 'staging' profile.
  args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options(
    profile: profile_name,
    config_source: Pathname.new(config_file)
  )

  # Override the target host to the correct address.
  # This is the recommended way to override configuration values.
  args[0] = 'localhost:7233'

  puts "\nLoaded '#{profile_name}' profile from #{config_file} with overrides."
  puts "  Address: #{args[0]} (overridden from localhost:9999)"
  puts "  Namespace: #{args[1]}"

  puts "\nAttempting to connect to client..."
  begin
    client = Temporalio::Client.connect(*args, **kwargs)
    puts '✅ Client connected successfully!'
    sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new)
    puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}"
  rescue StandardError => e
    puts "❌ Failed to connect: #{e}"
  end
end

main if $PROGRAM_NAME == __FILE__
```

**.NET**

To load a specific profile from a custom path in .NET C#, use the `ClientEnvConfig.LoadClientConnectOptions()` method
with the `ProfileLoadOptions` parameter. Use the `Profile` property to specify the profile name and the `ConfigSource`
property to specify the file path.

After loading the connection options, you can override specific settings programmatically before passing them to
`TemporalClient.ConnectAsync()`. Refer to the
[C# SDK API documentation](https://dotnet.temporal.io/api/Temporalio.Common.EnvConfig.html) for all available options.

```csharp {18-19,25-28}
using Temporalio.Client;
using Temporalio.Client.EnvConfig;

namespace TemporalioSamples.EnvConfig;

/// <summary>
/// Sample demonstrating loading a named environment configuration profile and
/// programmatically overriding its values.
/// </summary>
public static class LoadProfile
{
    public static async Task RunAsync()
    {
        Console.WriteLine("--- Loading 'staging' profile with programmatic overrides ---");

        try
        {
            var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml");
            var profileName = "staging";

            Console.WriteLine("The 'staging' profile in config.toml has an incorrect address (localhost:9999).");
            Console.WriteLine("We'll programmatically override it to the correct address.");

            // Load the 'staging' profile
            var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions
            {
                Profile = profileName,
                ConfigSource = DataSource.FromPath(configFile),
            });

            // Override the target host to the correct address.
            // This is the recommended way to override configuration values.
            connectOptions.TargetHost = "localhost:7233";

            Console.WriteLine($"\nLoaded '{profileName}' profile from {configFile} with overrides.");
            Console.WriteLine($"  Address: {connectOptions.TargetHost} (overridden from localhost:9999)");
            Console.WriteLine($"  Namespace: {connectOptions.Namespace}");

            Console.WriteLine("\nAttempting to connect to client...");

            var client = await TemporalClient.ConnectAsync(connectOptions);
            Console.WriteLine("✅ Client connected successfully!");

            // Test the connection by checking the service
            var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new());
            Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            Console.WriteLine($"❌ Failed to connect: {ex.Message}");
        }
    }
}
```

**TypeScript**

To load a specific profile from a custom path in TypeScript, use the `loadClientConnectConfig` helper from
`@temporalio/envconfig` package with the `profile` and `configFile` options.

[env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts)
```ts {17-19,28-29}
import { Connection, Client } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { resolve } from 'path';

async function main() {
  console.log('--- Loading default profile from config.toml ---');

  // For this sample to be self-contained, we explicitly provide the path to
  // the config.toml file included in this directory.
  // By default though, the config.toml file will be loaded from
  // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
  const configFile = resolve(__dirname, '../config.toml');

  // loadClientConnectConfig is a helper that loads a profile and prepares
  // the configuration for Connection.connect and Client. By default, it loads the
  // "default" profile.
  const config = loadClientConnectConfig({
    configSource: { path: configFile },
  });

  console.log(`Loaded 'default' profile from ${configFile}.`);
  console.log(`  Address: ${config.connectionOptions.address}`);
  console.log(`  Namespace: ${config.namespace}`);
  console.log(`  gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`);

  console.log('\nAttempting to connect to client...');
  try {
    const connection = await Connection.connect(config.connectionOptions);
    const client = new Client({ connection, namespace: config.namespace });
    console.log('✅ Client connected successfully!');
    await connection.close();
  } catch (err) {
    console.log(`❌ Failed to connect: ${err}`);
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

**Java**

To load a profile configuration file from a custom path in Java, use the `ClientConfigProfile.load` method from the
`envconfig` package with the `ConfigFilePath` parameter. This method will load the profile from the custom path and any
environment variables. Environment variables take precedence over the configuration file settings.

```java {21-25}
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.envconfig.LoadClientConfigProfileOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import java.nio.file.Paths;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoadFromFile {

  private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class);

  public static void main(String[] args) {
    try {

      String configFilePath =
          Paths.get(LoadFromFile.class.getResource("/config.toml").toURI()).toString();

      ClientConfigProfile profile =
          ClientConfigProfile.load(
              LoadClientConfigProfileOptions.newBuilder()
                  .setConfigFilePath(configFilePath)
                  .build());

      WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions();
      WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions();

      try {
        // Create the workflow client using the loaded configuration
        WorkflowClient client =
            WorkflowClient.newInstance(
                WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions);

        // Test the connection by getting system info
        var systemInfo =
            client
                .getWorkflowServiceStubs()
                .blockingStub()
                .getSystemInfo(
                    io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance());

        logger.info("✅ Client connected successfully!");
        logger.info("   Server version: {}", systemInfo.getServerVersion());

      } catch (Exception e) {
        logger.error("❌ Failed to connect: {}", e.getMessage());
      }

    } catch (Exception e) {
      logger.error("Failed to load configuration: {}", e.getMessage(), e);
      System.exit(1);
    }
  }
}
```

---

# Go SDK developer guide

Source: https://docs.temporal.io/develop/go

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Install and get started

You can find detailed installation instructions for the Go SDK in the [Quickstart](/develop/go/set-up-your-local-go).

There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running.

Once your local Temporal Service is set up, continue building with the following resources:

- [Workflow basics](/develop/go/workflows/basics)
- [Activity basics](/develop/go/activities/basics)
- [Start an Activity Execution](/develop/go/activities/execution)
- [Run a Worker](/develop/go/workers/run-worker-process)

From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases.

## [Workflows](/develop/go/workflows)

- [Workflow basics](/develop/go/workflows/basics)
- [Child Workflows](/develop/go/workflows/child-workflows)
- [Continue-As-New](/develop/go/workflows/continue-as-new)
- [Cancellation](/develop/go/workflows/cancellation)
- [Timeouts](/develop/go/workflows/timeouts)
- [Message passing](/develop/go/workflows/message-passing)
- [Selectors](/develop/go/workflows/selectors)
- [Side effects](/develop/go/workflows/side-effects)
- [Schedules](/develop/go/workflows/schedules)
- [Timers](/develop/go/workflows/timers)
- [Dynamic Workflow](/develop/go/workflows/dynamic-workflow)
- [Versioning](/develop/go/workflows/versioning)
- [Workflow Streams](/develop/go/workflows/workflow-streams)

## [Activities](/develop/go/activities)

- [Activity basics](/develop/go/activities/basics)
- [Activity Execution](/develop/go/activities/execution)
- [Standalone Activities](/develop/go/activities/standalone-activities-quickstart)
- [Timeouts](/develop/go/activities/timeouts)
- [Asynchronous Activity Completion](/develop/go/activities/asynchronous-activity)
- [Dynamic Activity](/develop/go/activities/dynamic-activity)
- [Benign exceptions](/develop/go/activities/benign-exceptions)

## [Workers](/develop/go/workers)

- [Run a Worker](/develop/go/workers/run-worker-process)
- [Sessions](/develop/go/workers/sessions)
- [Serverless Workers](/develop/go/workers/serverless-workers)

## [Temporal Client](/develop/go/client)

- [Temporal Client](/develop/go/client/temporal-client)
- [Namespaces](/develop/go/client/namespaces)

## [Temporal Nexus](/develop/go/nexus)

- [Quickstart](/develop/go/nexus/quickstart)
- [Feature guide](/develop/go/nexus/feature-guide)
- [Standalone Operations](/develop/go/nexus/standalone-operations)

## [Platform](/develop/go/platform)

- [Observability](/develop/go/platform/observability)
- [Enriching the UI](/develop/go/platform/enriching-ui)

## [Best practices](/develop/go/best-practices)

- [Multithreading](/develop/go/best-practices/multithreading)
- [Context propagation](/develop/go/best-practices/context-propagation)
- [Error handling](/develop/go/best-practices/error-handling)
- [Debugging](/develop/go/best-practices/debugging)
- [Testing](/develop/go/best-practices/testing-suite)
- [Data handling](/develop/go/data-handling)

## [Integrations](/develop/go/integrations)

- [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#go)
- [Google ADK integration](/develop/go/integrations/google-adk)
- [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2)

## Temporal Go technical resources

- [Go SDK Quickstart - Setup Guide](/develop/go/set-up-your-local-go)
- [Go API Documentation](https://pkg.go.dev/go.temporal.io/sdk)
- [Go SDK Code Samples](https://github.com/temporalio/samples-go)
- [Go SDK GitHub](https://github.com/temporalio/sdk-go)
- [Temporal 101 in Go Free Course](https://learn.temporal.io/courses/temporal_101/go/)
- [Build a Job Queue with Standalone Activities Tutorial](https://learn.temporal.io/tutorials/go/standalone-activities/)

### Where are SDK-specific code examples? 

- [Background Check application](https://github.com/temporalio/background-checks): Provides a non-trivial Temporal
  Application implementation in conjunction with
  [application documentation](https://learn.temporal.io/examples/go/background-checks/).
- [Hello world application template in Go](https://github.com/temporalio/hello-world-project-template-go): Provides a
  quick-start development app for users. This sample works in conjunction with the
  ["Hello World!" from scratch tutorial in Go](https://learn.temporal.io/getting_started/go/hello_world_in_go/).
- [Money transfer application template in Go](https://github.com/temporalio/money-transfer-project-template-go):
  Provides a quick-start development app for users. It demonstrates a basic "money transfer" Workflow Definition and
  works in conjunction with the
  [Run your first app tutorial in Go](https://learn.temporal.io/getting_started/go/first_program_in_go/).
- [Subscription-style Workflow Definition in Go](https://github.com/temporalio/subscription-workflow-project-template-go):
  Demonstrates some of the patterns that could be implemented for a subscription-style business process.
- [eCommerce application example in Go](https://github.com/temporalio/temporal-ecommerce): Showcases a per-user shopping
  cart–style Workflow Definition that includes an API for adding and removing items from the cart as well as a web UI.
  This application sample works in conjunction with the
  [eCommerce in Go tutorial](https://learn.temporal.io/tutorials/go/build-an-ecommerce-app).

## Get connected with the Temporal Go community

- [Temporal Go Community Slack](https://temporalio.slack.com/archives/CTDTU3J4T)
- [Go SDK Forum](https://community.temporal.io/tag/go-sdk)

---

# Activities - Go SDK

Source: https://docs.temporal.io/develop/go/activities

> Activity Definitions, Activity Execution, Timeouts, Standalone Activities, Dynamic Activities, and benign exceptions in the Temporal Go SDK.

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Activities

- [Activity basics](/develop/go/activities/basics)
- [Activity Execution](/develop/go/activities/execution)
- [Standalone Activities quickstart](/develop/go/activities/standalone-activities-quickstart)
- [Standalone Activities feature guide](/develop/go/activities/standalone-activities)
- [Timeouts](/develop/go/activities/timeouts)
- [Asynchronous Activity Completion](/develop/go/activities/asynchronous-activity)
- [Dynamic Activity](/develop/go/activities/dynamic-activity)
- [Benign exceptions](/develop/go/activities/benign-exceptions)

---

# Asynchronous Activity Completion - Go SDK

Source: https://docs.temporal.io/develop/go/activities/asynchronous-activity

> Asynchronous Activity Completion lets the Activity Function return without finishing Activity Execution. Use Task Tokens and Temporal Client to complete the Activity externally.

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow ID, and Activity ID.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The external system uses the Temporal Client to Heartbeat and complete the Activity.

**Step 1: Provide the external system with a Task Token to complete the Activity Execution.**
   To do this, use the `GetInfo()` API from the `go.temporal.io/sdk/activity` package.

<!--SNIPSTART go-async-activity-get-task-token-->
[go-sandbox/activities/asynccompletion/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/asynccompletion/activity.go)
```go
// Retrieve the Activity information needed to asynchronously complete the Activity.
activityInfo := activity.GetInfo(ctx)
taskToken := activityInfo.TaskToken
// Send the taskToken to the external service that will complete the Activity.
if err := a.External.Submit(ctx, taskToken, request); err != nil {
	return "", err
}
```
<!--SNIPEND-->

**Step 2: Return an `activity.ErrResultPending` error to indicate that the Activity is completing asynchronously.**

<!--SNIPSTART go-async-activity-return-pending-->
[go-sandbox/activities/asynccompletion/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/asynccompletion/activity.go)
```go
return "", activity.ErrResultPending
```
<!--SNIPEND-->

**Step 3: Use the Temporal Client to complete the Activity using the Task Token.**

<!--SNIPSTART go-async-activity-complete-->
[go-sandbox/activities/asynccompletion/completion.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/asynccompletion/completion.go)
```go
// Instantiate a Temporal service client.
// The same client can be used to complete or fail any number of Activities.
// The client is a heavyweight object that should be created once per process.
temporalClient, err := client.Dial(client.Options{})
if err != nil {
	return err
}
defer temporalClient.Close()

// Complete the Activity.
return temporalClient.CompleteActivity(context.Background(), taskToken, result, nil)
```
<!--SNIPEND-->

The following are the parameters of the `CompleteActivity` function:

- `taskToken`: The value of the binary `TaskToken` field of the `ActivityInfo` struct retrieved inside
  the Activity.
- `result`: The return value to record for the Activity. The type of this value must match the type
  of the return value declared by the Activity function.
- `err`: The error to record if the Activity fails.

If `err` is not `nil`, the value of `result` is ignored.

While the external system works, it can call `RecordActivityHeartbeat` with the same Task Token so the Activity doesn't exceed its Heartbeat Timeout.

To fail the Activity, pass an error to `CompleteActivity`.
A retryable error starts a new Activity attempt under the Activity's Retry Policy, which gives the external system a new Task Token.
To fail the Activity without another attempt, use a non-retryable error:

<!--SNIPSTART go-async-activity-fail-->
[go-sandbox/activities/asynccompletion/completion.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/asynccompletion/completion.go)
```go
// Fail the Activity. A non-retryable error stops the Retry Policy from
// starting another Activity attempt.
reason := temporal.NewNonRetryableApplicationError("order rejected", "Rejected", nil)
return temporalClient.CompleteActivity(context.Background(), taskToken, nil, reason)
```
<!--SNIPEND-->

To complete the Activity without a Task Token, use `CompleteActivityByID`.
Set `ActivityID` in the Workflow's `ActivityOptions` so the external system knows which Activity ID to pass.

<!--SNIPSTART go-async-activity-complete-by-id-->
[go-sandbox/activities/asynccompletion/completion.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/asynccompletion/completion.go)
```go
// Identify the Activity by Namespace, Workflow ID, and Activity ID.
// An empty Run ID targets the latest Run of the Workflow.
runID := ""
return temporalClient.CompleteActivityByID(context.Background(), namespace, workflowID, runID, ApprovalActivityID, result, nil)
```
<!--SNIPEND-->

---

# Activity basics - Go SDK

Source: https://docs.temporal.io/develop/go/activities/basics

> An Activity Definition in Go is an exportable function or struct method. Set its parameters, return values, and a custom Activity Type name.

## Develop an Activity Definition 

In the Temporal Go SDK programming model, an Activity Definition is an exportable function or a `struct` method.

Standalone Activities are Activity Executions that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client.

The Activity definition and Worker registration are identical to regular Activities, and only the execution path differs. See [Standalone Activities](/develop/go/activities/standalone-activities-quickstart).

Below is an example of both a basic Activity Definition and of an Activity defined as a Struct method.
An _Activity struct_ can have more than one method, with each method acting as a separate Activity Type.
Activities written as struct methods can use shared struct variables, such as:

- an application level DB pool
- client connection to another service
- reusable utilities
- any other expensive resources that you only want to initialize once per process

Because this is such a common need, the rest of this guide shows Activities written as `struct` methods.

> **📝 Note:**
>
> While it is possible to register struct methods as Workflows, this is strongly discouraged.
> In some cases, struct methods as Workflows may cause non-deterministic errors. We recommend
> only using struct methods for Activities.
>

<!--SNIPSTART go-activities-basics {"selectedLines": ["1-11" ,"18-21", "28-42"]}-->
[go-sandbox/activities/basics/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/activity.go)
```go
package basics

import (
	"context"

	"go.temporal.io/sdk/activity"
)

func YourSimpleActivityDefinition(ctx context.Context) error {
	return nil
}
// ...
type YourActivityObject struct {
	Message *string
	Number  *int
}
// ...
func (a *YourActivityObject) YourActivityDefinition(ctx context.Context, param YourActivityParam) (*YourActivityResultObject, error) {
	// Use Activities for calling external APIs.
	// This is just an example of using the logger to print "Hello World!"
	logger := activity.GetLogger(ctx)
	logger.Info("The message is:", param.ActivityParamX)
	logger.Info("The number is:", param.ActivityParamY)
	// Return data using a Struct so that the function signature is backward compatible.
	result := &YourActivityResultObject{
		ResultFieldX: "Success",
		ResultFieldY: 1,
	}
	// Return the results back to the Workflow Execution.
	// The results persist within the Event History of the Workflow Execution.
	return result, nil
}
```
<!--SNIPEND-->

### Define Activity parameters 

There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support.
However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB.
And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB.

Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance.
This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task).

Some SDKs require that you pass context objects, others do not.
When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities.
This is so that you can change what data is passed to the Activity without breaking a function or method signature.

The first parameter of an Activity Definition is `context.Context`.
This parameter is optional for an Activity Definition, though it is recommended, especially if the Activity is expected to use other Go SDK APIs.

An Activity Definition can support as many other custom parameters as needed.
However, all parameters must be serializable (parameters can't be channels, functions, variadic, or unsafe pointers), and it is recommended to pass a single struct that can be updated later.

<!--SNIPSTART go-activities-basics {"highlightedLines": "7-10,13", "selectedLines": ["13-22", "28-42"]}-->
[go-sandbox/activities/basics/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/activity.go)
```go {7-10,13}
// ...
type YourActivityParam struct {
	ActivityParamX string
	ActivityParamY int
}

type YourActivityObject struct {
	Message *string
	Number  *int
}

// ...
func (a *YourActivityObject) YourActivityDefinition(ctx context.Context, param YourActivityParam) (*YourActivityResultObject, error) {
	// Use Activities for calling external APIs.
	// This is just an example of using the logger to print "Hello World!"
	logger := activity.GetLogger(ctx)
	logger.Info("The message is:", param.ActivityParamX)
	logger.Info("The number is:", param.ActivityParamY)
	// Return data using a Struct so that the function signature is backward compatible.
	result := &YourActivityResultObject{
		ResultFieldX: "Success",
		ResultFieldY: 1,
	}
	// Return the results back to the Workflow Execution.
	// The results persist within the Event History of the Workflow Execution.
	return result, nil
}
```
<!--SNIPEND-->

### Define Activity return values 

All data returned from an Activity must be serializable.

Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/evaluate/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history).

A Go-based Activity Definition can return either just an `error` or a `customValue, error` combination (same as a Workflow Definition).
You may wish to use a `struct` type to hold all custom values, just keep in mind they must all be serializable.

<!--SNIPSTART go-activities-basics {"highlightedLines": "2-5,14-17", "selectedLines": ["23-42"]}-->
[go-sandbox/activities/basics/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/activity.go)
```go {2-5,14-17}
// ...
type YourActivityResultObject struct {
	ResultFieldX string
	ResultFieldY int
}

func (a *YourActivityObject) YourActivityDefinition(ctx context.Context, param YourActivityParam) (*YourActivityResultObject, error) {
	// Use Activities for calling external APIs.
	// This is just an example of using the logger to print "Hello World!"
	logger := activity.GetLogger(ctx)
	logger.Info("The message is:", param.ActivityParamX)
	logger.Info("The number is:", param.ActivityParamY)
	// Return data using a Struct so that the function signature is backward compatible.
	result := &YourActivityResultObject{
		ResultFieldX: "Success",
		ResultFieldY: 1,
	}
	// Return the results back to the Workflow Execution.
	// The results persist within the Event History of the Workflow Execution.
	return result, nil
}
```
<!--SNIPEND-->

### Customize Activity Type 

To customize the Activity Type, set the `Name` parameter with `RegisterOptions` when registering your Activity with a Worker.

<!--SNIPSTART go-activities-worker {"highlightedLines": "9,24-27", "selectedLines": ["14-45"]}-->
[go-sandbox/activities/workers/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/workers/main.go)
```go {9,24-27}
// ...
func main() {
	temporalClient, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer temporalClient.Close()

	yourWorker := worker.New(temporalClient, "your-custom-task-queue-name", worker.Options{})
	yourWorker.RegisterWorkflow(basics.YourWorkflowDefinition)
	registerWFOptions := workflow.RegisterOptions{
		Name: "JustAnotherWorkflow",
	}
	yourWorker.RegisterWorkflowWithOptions(basics.YourSimpleWorkflowDefinition, registerWFOptions)

	message := "This could be a connection string or endpoint details"
	number := 100
	activities := &basics.YourActivityObject{
		Message: &message,
		Number:  &number,
	}
	yourWorker.RegisterActivity(activities)

	registerAOptions := activity.RegisterOptions{
		Name: "JustAnotherActivity",
	}
	yourWorker.RegisterActivityWithOptions(basics.YourSimpleActivityDefinition, registerAOptions)

	err = yourWorker.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start Worker", err)
	}
}
```
<!--SNIPEND-->

---

# Benign exceptions - Go SDK

Source: https://docs.temporal.io/develop/go/activities/benign-exceptions

> Mark expected or non-severe Activity errors as benign in Go to reduce noise in logs, metrics, and OpenTelemetry traces.

When Activities return errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues.
By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic.

To mark an error as benign, use [`temporal.NewApplicationErrorWithOptions`](https://pkg.go.dev/go.temporal.io/sdk/temporal#NewApplicationErrorWithOptions) and set the `Category` field to `temporal.ApplicationErrorCategoryBenign` in the `ApplicationErrorOptions`.

Benign errors:
- Have Activity failure logs downgraded to DEBUG level
- Do not emit Activity failure metrics
- Do not set the OpenTelemetry failure status to ERROR

<!--SNIPSTART go-benign-activity-error-->
[go-sandbox/activities/benign/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/benign/activity.go)
```go
import (
	"context"

	"go.temporal.io/sdk/temporal"
)

func MyActivity(ctx context.Context) (string, error) {
	result, err := callExternalService()
	if err != nil {
		// Mark this error as benign since it's expected
		return "", temporal.NewApplicationErrorWithOptions(
			err.Error(),
			"",
			temporal.ApplicationErrorOptions{
				Category: temporal.ApplicationErrorCategoryBenign,
			},
		)
	}
	return result, nil
}
```
<!--SNIPEND-->

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried.

---

# Dynamic Activity - Go SDK

Source: https://docs.temporal.io/develop/go/activities/dynamic-activity

> Register one Dynamic Activity per Worker with RegisterDynamicActivity to handle Activity Types that match no other registered Activity.

## Set a Dynamic Activity 

A Dynamic Activity in Temporal is an Activity that is invoked dynamically at runtime if no other Activity with the same name is registered.
An Activity can be registered as dynamic by using `worker.RegisterDynamicActivity()`.
You must register the Activity with the Worker before it can be invoked.
Only one Dynamic Activity can be present on a Worker.

The Activity Definition must then accept a single argument of type `converter.EncodedValues`.

<!--SNIPSTART go-dynamic-activity-definition-->
[go-sandbox/activities/dynamic/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/dynamic/activity.go)
```go
func DynamicActivity(ctx context.Context, args converter.EncodedValues) (string, error) {
	var arg1, arg2 string
	err := args.Get(&arg1, &arg2)
	if err != nil {
		return "", fmt.Errorf("failed to decode arguments: %w", err)
	}

	info := activity.GetInfo(ctx)
	result := fmt.Sprintf("%s - %s - %s", info.WorkflowType.Name, arg1, arg2)

	return result, nil
}
```
<!--SNIPEND-->

---

# Activity Execution - Go SDK

Source: https://docs.temporal.io/develop/go/activities/execution

> Call an Activity from a Go Workflow with workflow.ExecuteActivity, set the required Timeout, and look up every ActivityOptions field in one table.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition).
The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command.
This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History.

A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations.
Activity implementation code should be _idempotent_.

The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history.
The entire Execution history is transferred from the Temporal Service to Workflow Workers when a Workflow state needs to recover.
A large Execution history can thus adversely impact the performance of your Workflow.

Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values.
Otherwise, no additional limitations exist on Activity implementations.

To spawn an [Activity Execution](/activity-execution), call [`ExecuteActivity()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ExecuteActivity) inside your Workflow Definition.
The API is available from the [`go.temporal.io/sdk/workflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow) package.
The `ExecuteActivity()` API call requires an instance of `workflow.Context`, the Activity function name, and any variables to be passed to the Activity Execution.
The Activity function name can be provided as a variable object (no quotations) or as a string.
The benefit of passing the actual function object is that the framework can validate the parameters against the Activity Definition.
The `ExecuteActivity` call returns a Future, which can be used to get the result of the Activity Execution.

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["20", "28", "31", "37-48", "57-59", "66"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) {
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		StartToCloseTimeout: 10 * time.Second,
// ...
	}
	ctx = workflow.WithActivityOptions(ctx, activityOptions)
	activityParam := YourActivityParam{
		ActivityParamX: param.WorkflowParamX,
		ActivityParamY: param.WorkflowParamY,
	}

	var a *YourActivityObject

	var activityResult YourActivityResultObject

	future := workflow.ExecuteActivity(ctx, a.YourActivityDefinition, activityParam)
// ...
	if err := future.Get(ctx, &activityResult); err != nil {
		return nil, err
	}
// ...
}
```
<!--SNIPEND-->

### Set the required Activity Timeouts 

Activity Execution semantics rely on several parameters.
The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).
These values are set in the Activity Options.

To set an Activity Timeout in Go, create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the Activity Timeout field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

Available timeouts are:

- `StartToCloseTimeout`
- `ScheduleToCloseTimeout`
- `ScheduleToStartTimeout`

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28", "31", "35-38", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		StartToCloseTimeout: 10 * time.Second,
// ...
		// ScheduleToCloseTimeout: 10 * time.Second,
		// ScheduleToStartTimeout: 10 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, activityOptions)
// ...

```
<!--SNIPEND-->

### Go ActivityOptions reference 

Create an instance of [`ActivityOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ActivityOptions) from the `go.temporal.io/sdk/workflow` package and use [`WithActivityOptions()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#WithActivityOptions) to apply it to the instance of `workflow.Context`.

The instance of `workflow.Context` is then passed to the `ExecuteActivity()` call.

| Field                                               | Required                          | Type                                                                        |
| --------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------- |
| [`ActivityID`](#activityid)                         | No                                | `string`                                                                    |
| [`TaskQueue`](#taskqueue)                           | No                                | `string`                                                                    |
| [`ScheduleToCloseTimeout`](#scheduletoclosetimeout) | Yes (or `StartToCloseTimeout`)    | `time.Duration`                                                             |
| [`ScheduleToStartTimeout`](#scheduletostarttimeout) | No                                | `time.Duration`                                                             |
| [`StartToCloseTimeout`](#scheduletoclosetimeout)    | Yes (or `ScheduleToCloseTimeout`) | `time.Duration`                                                             |
| [`HeartbeatTimeout`](#heartbeattimeout)             | No                                | `time.Duration`                                                             |
| [`WaitForCancellation`](#waitforcancellation)       | No                                | `bool`                                                                      |
| [`RetryPolicy`](#retrypolicy)                       | No                                | [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) |

#### ActivityID

- Type: `string`
- Default: None

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28-29", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
		ActivityID:          "your-activity-id",
// ...
	}
// ...

```
<!--SNIPEND-->

- [What is an Activity ID](/activity-execution#activity-id)

#### TaskQueue

- Type: `string`
- Default: Inherits the TaskQueue name from the Workflow.

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28", "30", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		TaskQueue:           "your-custom-task-queue-name",
// ...
	}
// ...

```
<!--SNIPEND-->

- [What is a Task Queue](/task-queue)

#### ScheduleToCloseTimeout

To set a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `ScheduleToCloseTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

This or `StartToCloseTimeout` must be set.

- Type: `time.Duration`
- Default: ∞ (infinity - no limit)

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28", "35", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		// ScheduleToCloseTimeout: 10 * time.Second,
// ...
	}
// ...

```
<!--SNIPEND-->

#### ScheduleToStartTimeout

To set a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `ScheduleToStartTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

- Type: `time.Duration`
- Default: ∞ (infinity - no limit)

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28", "36", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		// ScheduleToStartTimeout: 10 * time.Second,
// ...
	}
// ...

```
<!--SNIPEND-->

#### StartToCloseTimeout

To set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `StartToCloseTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

This or `ScheduleToCloseTimeout` must be set.

- Type: `time.Duration`
- Default: Same as the `ScheduleToCloseTimeout`

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28", "31", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		StartToCloseTimeout: 10 * time.Second,
// ...
	}
// ...

```
<!--SNIPEND-->

#### HeartbeatTimeout

To set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `HeartbeatTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28", "32", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		HeartbeatTimeout:    5 * time.Second,
// ...
	}
// ...

```
<!--SNIPEND-->

#### WaitForCancellation

If `true` the Activity Execution will finish executing should there be a Cancellation request.

- Type: `bool`
- Default: `false`

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["28", "33", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	activityOptions := workflow.ActivityOptions{
// ...
		WaitForCancellation: false,
// ...
	}
// ...

```
<!--SNIPEND-->

#### RetryPolicy

To set a [RetryPolicy](/encyclopedia/retry-policies), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `RetryPolicy` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

- Type: [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy)
- Default:

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["21-27"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	retryPolicy := &temporal.RetryPolicy{
		InitialInterval:        time.Second,
		BackoffCoefficient:     2.0,
		MaximumInterval:        time.Second * 100, // 100 * InitialInterval
		MaximumAttempts:        0,                 // Unlimited
		NonRetryableErrorTypes: []string{},        // empty
	}
```
<!--SNIPEND-->

Providing a Retry Policy here is a customization that overwrites individual Field defaults.

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["21-28", "34", "37", "45"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	retryPolicy := &temporal.RetryPolicy{
		InitialInterval:        time.Second,
		BackoffCoefficient:     2.0,
		MaximumInterval:        time.Second * 100, // 100 * InitialInterval
		MaximumAttempts:        0,                 // Unlimited
		NonRetryableErrorTypes: []string{},        // empty
	}
	activityOptions := workflow.ActivityOptions{
// ...
		RetryPolicy:         retryPolicy,
// ...
	}
// ...

```
<!--SNIPEND-->

### Get the results of an Activity Execution 

The call to spawn an [Activity Execution](/activity-execution) generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable.
Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available.

The `ExecuteActivity` API call returns an instance of [`workflow.Future`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Futures) which has the following two methods:

- `Get()`: Takes an instance of the `workflow.Context`, that was passed to the Activity Execution, and a pointer as parameters.
  The variable associated with the pointer is populated with the Activity Execution result.
  This call blocks until the results are available.
- `IsReady()`: Returns `true` when the result of the Activity Execution is ready.

Call the `Get()` method on the instance of `workflow.Future` to get the result of the Activity Execution.
The type of the result parameter must match the type of the return value declared by the Activity function.

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["20", "44-48", "57-59", "66"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) {
// ...
	var a *YourActivityObject

	var activityResult YourActivityResultObject

	future := workflow.ExecuteActivity(ctx, a.YourActivityDefinition, activityParam)
// ...
	if err := future.Get(ctx, &activityResult); err != nil {
		return nil, err
	}
// ...
}
```
<!--SNIPEND-->

Use the `IsReady()` method first to make sure the `Get()` call doesn't cause the Workflow Execution to wait on the result.

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["20", "44-54", "66"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) {
// ...
	var a *YourActivityObject

	var activityResult YourActivityResultObject

	future := workflow.ExecuteActivity(ctx, a.YourActivityDefinition, activityParam)
	if future.IsReady() {
		var yourActivityResult YourActivityResultObject
		if err := future.Get(ctx, &yourActivityResult); err != nil {
			return nil, err
		}
	}
// ...
}
```
<!--SNIPEND-->

It is idiomatic to invoke multiple Activity Executions from within a Workflow.
Therefore, it is also idiomatic to either block on the results of the Activity Executions or continue on to execute additional logic, checking for the Activity Execution results at a later time.

---

# Standalone Activities feature guide - Go SDK

Source: https://docs.temporal.io/develop/go/activities/standalone-activities

> Execute Activities independently without a Workflow using the Temporal Go SDK.

Standalone Activities are Activity Executions that run independently, without being orchestrated by a Workflow. Instead
of starting an Activity from within a Workflow Definition using `workflow.ExecuteActivity()`, you start a Standalone
Activity directly from a Temporal Client using `client.ExecuteActivity()`.

The Activity definition and Worker registration are identical to regular Activities, and only the execution path
differs.

> **💡 Tip:**
>
> New to Standalone Activities? Start with the [Standalone Activities quickstart](/develop/go/activities/standalone-activities-quickstart).
>

This page covers the following:

- [Get the result of a Standalone Activity](#get-activity-result)
- [Get a handle to an existing Standalone Activity](#get-activity-handle)
- [List Standalone Activities](#list-activities)
- [Count Standalone Activities](#count-activities)
- [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud)

## Get the result of a Standalone Activity 

Use `ActivityHandle.Get()` to block until the Activity completes and retrieve its result. This is analogous to calling
`Get()` on a `WorkflowRun`.

<!--SNIPSTART go-standalone-activity-get-result-->
[go-sandbox/activities/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/standalone/starter/main.go)
```go
var result string
err = handle.Get(context.Background(), &result)
if err != nil {
	log.Fatalln("Activity failed", err)
}
log.Println("Activity result:", result)
```
<!--SNIPEND-->

If the Activity completed successfully, the result is deserialized into the provided pointer. If the Activity failed,
the failure is returned as an error.

Or use the Temporal CLI to wait for a result by Activity ID:

```bash
temporal activity result --activity-id standalone_activity_helloworld_ActivityID
```

## Get a handle to an existing Standalone Activity 

Use `client.GetActivityHandle()` to create a handle to a previously started Standalone Activity. This is analogous to
`client.GetWorkflow()` for Workflow Executions.

`ActivityID` is required. `RunID` is optional. Leave it empty to target the latest Activity Execution with
the given Activity ID.

<!--SNIPSTART go-standalone-activity-get-handle-->
[go-sandbox/activities/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/standalone/starter/main.go)
```go
handle := c.GetActivityHandle(client.GetActivityHandleOptions{
	ActivityID: "standalone_activity_helloworld_ActivityID",
	RunID:      runID, // Leave empty to target the latest Activity Execution.
})

// Use the handle to get the result, describe, cancel, or terminate
var result string
err := handle.Get(context.Background(), &result)
if err != nil {
	log.Fatalln("Unable to get activity result", err)
}
```
<!--SNIPEND-->

## List Standalone Activities 

Use [`client.ListActivities()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to list Standalone Activity
Executions that match a [List Filter](/list-filter) query. The result contains an iterator that yields
[`ActivityExecutionInfo`](https://pkg.go.dev/go.temporal.io/sdk/client#ActivityExecutionInfo) entries.

These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included.

<!--SNIPSTART go-standalone-activity-list-->
[go-sandbox/activities/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/standalone/starter/main.go)
```go
resp, err := c.ListActivities(context.Background(), client.ListActivitiesOptions{
	Query: "TaskQueue = 'standalone-activity-helloworld'",
})
if err != nil {
	log.Fatalln("Unable to list activities", err)
}

for info, err := range resp.Results {
	if err != nil {
		log.Fatalln("Error iterating activities", err)
	}
	log.Printf("ActivityID: %s, Type: %s, Status: %v\n",
		info.ActivityID, info.ActivityType, info.Status)
}
```
<!--SNIPEND-->

Or use the Temporal CLI:

```bash
temporal activity list
```

The `Query` field accepts the same [List Filter](/list-filter) syntax used for Workflow Visibility. For example,
`"ActivityType = 'Activity' AND ExecutionStatus = 'Running'"`.

## Count Standalone Activities 

Use [`client.CountActivities()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to count Standalone Activity
Executions that match a [List Filter](/list-filter) query. This returns the total count of executions (running,
completed, failed, etc.) - not the number of queued tasks. It works the same way as counting Workflow Executions.

<!--SNIPSTART go-standalone-activity-count-->
[go-sandbox/activities/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/standalone/starter/main.go)
```go
resp, err := c.CountActivities(context.Background(), client.CountActivitiesOptions{
	Query: "TaskQueue = 'standalone-activity-helloworld'",
})
if err != nil {
	log.Fatalln("Unable to count activities", err)
}

log.Println("Total activities:", resp.Count)
```
<!--SNIPEND-->

Or use the Temporal CLI:

```bash
temporal activity count
```

## Run Standalone Activities with Temporal Cloud 

The Worker and Client code in the [Standalone Activities quickstart](/develop/go/activities/standalone-activities-quickstart)
use `envconfig.MustLoadDefaultClientOptions()`, so the same code works against Temporal Cloud -
configure the connection via environment variables or a TOML profile. No code changes are needed.

For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate
generation, and authentication setup in the Cloud UI, see
[Connect to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud).

### Connect with mTLS

Set these environment variables with values from your Temporal Cloud Namespace settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem'
export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key'
```

### Connect with an API key

Set these environment variables with values from your Temporal Cloud API key settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_API_KEY=<your-api-key>
```

Then run the Worker and starter code as shown in the [Standalone Activities quickstart](/develop/go/activities/standalone-activities-quickstart).

---

# Standalone Activities Go quickstart - Go SDK

Source: https://docs.temporal.io/develop/go/activities/standalone-activities-quickstart

> Execute a Standalone Activity with the Temporal Go SDK without writing a Workflow.

# Quickstart

Standalone Activities are Activity Executions that run independently, without being orchestrated by a Workflow. Instead
of starting an Activity from within a Workflow Definition using `workflow.ExecuteActivity()`, you start a Standalone
Activity directly from a Temporal Client using `client.ExecuteActivity()`.

The Activity definition and Worker registration are identical to regular Activities, and only the execution path
differs.

> **📝 Note:**
>
> This documentation uses source code from the
> [standalone-activity/helloworld](https://github.com/temporalio/samples-go/tree/main/standalone-activity/helloworld).
>

## Get started with Standalone Activities 

Prerequisites:

- **[Go](https://go.dev/dl/)** 1.24+

- **[Temporal Go SDK](/develop/go/set-up-your-local-go#install-the-temporal-go-sdk)** (v1.49.0 or higher)

- **Temporal CLI** v1.9.1 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`.

Start the Temporal development server with `temporal server start-dev`.

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Server should now be available for client connections on `localhost:7233`, and the
Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233).

```bash
brew install temporal
```

```bash
temporal --version
```

```bash
temporal server start-dev
```

## Clone the sample

Clone the [samples-go](https://github.com/temporalio/samples-go) repository to follow along:

```
git clone https://github.com/temporalio/samples-go.git
cd samples-go
```

The sample project is structured as follows:

```
standalone-activity/helloworld/
├── activity.go
├── worker/
│   └── main.go
└── starter/
    └── main.go
```

## Define your Activity 

Define your Activity in a shared file so that both the Worker and starter can reference it.

[standalone-activity/helloworld/activity.go](https://github.com/temporalio/samples-go/blob/main/standalone-activity/helloworld/activity.go)

```go
package helloworld

import (
	"context"
	"go.temporal.io/sdk/activity"
)

func Activity(ctx context.Context, name string) (string, error) {
	logger := activity.GetLogger(ctx)
	logger.Info("Activity", "name", name)
	return "Hello " + name + "!", nil
}
```

## Run a Worker with the Activity registered 

Running a Worker for Standalone Activities is the same as running a Worker for Workflow-driven Activities — you create a
Worker, register the Activity, and call `Run()`. The Worker doesn't need to know whether the Activity will be invoked
from a Workflow or as a Standalone Activity.

See [Develop a Worker in Go](/develop/go/workers/run-worker-process#develop-worker) for more details on Worker setup and
configuration options.

[standalone-activity/helloworld/worker/main.go](https://github.com/temporalio/samples-go/blob/main/standalone-activity/helloworld/worker/main.go)

Open a new terminal, navigate to the `samples-go` directory, and run the Worker.
Leave this terminal running - the Worker needs to stay up to process activities.

```go
package main

import (
	"github.com/temporalio/samples-go/standalone-activity/helloworld"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
	"go.temporal.io/sdk/worker"
	"log"
)

func main() {
	c, err := client.Dial(envconfig.MustLoadDefaultClientOptions())
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, "standalone-activity-helloworld", worker.Options{})

	w.RegisterActivity(helloworld.Activity)

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```

```bash
go run standalone-activity/helloworld/worker/main.go
```

## Execute a Standalone Activity 

Use [`client.ExecuteActivity()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to start a Standalone Activity
Execution. This is called from application code (for example, a starter program), not from inside a Workflow Definition.

`ExecuteActivity` returns an [`ActivityHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#ActivityHandle) that you
can use to get the result, describe, cancel, or terminate the Activity.

The following starter program executes a Standalone Activity, gets its result, lists activities, and counts
activities:

Create [standalone-activity/helloworld/starter/main.go](https://github.com/temporalio/samples-go/blob/main/standalone-activity/helloworld/starter/main.go).

You can pass the Activity as either a function reference or a string Activity type name:

```go
handle, err := c.ExecuteActivity(ctx, options, helloworld.Activity, "arg1")

// Using a string type name
handle, err := c.ExecuteActivity(ctx, options, "Activity", "arg1")
```

`client.StartActivityOptions` requires `ID`, `TaskQueue`, and at least one of `ScheduleToCloseTimeout` or
`StartToCloseTimeout`. See [`StartActivityOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartActivityOptions)
in the API reference for the full set of options.

To run the starter:

1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above).
2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above).
3. Open a new terminal, navigate to the `samples-go` directory, and run:
    ```
    go run standalone-activity/helloworld/starter/main.go
    ```

Or use the Temporal CLI to execute a Standalone Activity.

```go
package main

import (
	"context"
	"github.com/temporalio/samples-go/standalone-activity/helloworld"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
	"log"
	"time"
)

func main() {
	c, err := client.Dial(envconfig.MustLoadDefaultClientOptions())
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	activityOptions := client.StartActivityOptions{
		ID:                     "standalone_activity_helloworld_ActivityID",
		TaskQueue:              "standalone-activity-helloworld",
		ScheduleToCloseTimeout: 10 * time.Second,
	}

	handle, err := c.ExecuteActivity(context.Background(), activityOptions, helloworld.Activity, "Temporal")
	if err != nil {
		log.Fatalln("Unable to execute activity", err)
	}

	log.Println("Started standalone activity", "ActivityID", handle.GetID(), "RunID", handle.GetRunID())

	var result string
	err = handle.Get(context.Background(), &result)
	if err != nil {
		log.Fatalln("Unable get standalone activity result", err)
	}
	log.Println("Activity result:", result)

	resp, err := c.ListActivities(context.Background(), client.ListActivitiesOptions{
		Query: "TaskQueue = 'standalone-activity-helloworld'",
	})
	if err != nil {
		log.Fatalln("Unable to list activities", err)
	}

	log.Println("ListActivity results")
	for info, err := range resp.Results {
		if err != nil {
			log.Fatalln("Error iterating activities", err)
		}
		log.Printf("\tActivityID: %s, Type: %s, Status: %v\n",
			info.ActivityID, info.ActivityType, info.Status)
	}

	resp1, err := c.CountActivities(context.Background(), client.CountActivitiesOptions{
		Query: "TaskQueue = 'standalone-activity-helloworld'",
	})
	if err != nil {
		log.Fatalln("Unable to count activities", err)
	}

	log.Println("Total activities:", resp1.Count)
}
```

```bash
go run standalone-activity/helloworld/starter/main.go
```

```bash
temporal activity execute \\
  --type Activity \\
  --activity-id standalone_activity_helloworld_ActivityID \\
  --task-queue standalone-activity-helloworld \\
  --schedule-to-close-timeout 10s \\
  --input '"Temporal"'
```

## Run with Temporal Cloud

All code samples on this page use
[`envconfig.MustLoadDefaultClientOptions()`](https://pkg.go.dev/go.temporal.io/sdk/contrib/envconfig)
to configure the Temporal Client connection. It responds to [environment
variables](/references/client-environment-configuration) and [TOML configuration
files](/references/client-environment-configuration), so the same code works against a local dev
server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal
Cloud](/develop/go/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the feature guide
for mTLS and API key setup.

## Next steps

- **[Standalone Activities feature guide](/develop/go/activities/standalone-activities)**: Get results, Activity handles, list and count Activities, and connect to Temporal Cloud.
- **[Build a Job Queue with Standalone Activities](https://learn.temporal.io/tutorials/go/standalone-activities/)**: Tutorial that builds a durable job queue, with idempotency keys, server-side deduplication, throughput limits, [Priority and Fairness](/develop/task-queue-priority-fairness), and heartbeat checkpoints.
- **[Activity basics](/develop/go/activities/basics)**: Write and register Activities with the Go SDK.

---

# Activity Timeouts - Go SDK

Source: https://docs.temporal.io/develop/go/activities/timeouts

> Set the Schedule-To-Close, Start-To-Close, and Heartbeat Timeouts on a Go Activity, tune its Retry Policy, and override the next retry delay.

## Set Activity timeouts 

Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution.

The following timeouts are available in the Activity Options.

- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution).
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution).
- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

To set an Activity Timeout in Go, create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the Activity Timeout field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

Available timeouts are:

- `StartToCloseTimeout`
- `ScheduleToCloseTimeout`
- `ScheduleToStartTimeout`

<!--SNIPSTART go-activity-timeouts-set-->
[go-sandbox/activities/timeouts/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/timeouts/workflow.go)
```go
activityOptions := workflow.ActivityOptions{
	// Set Activity Timeout duration
	ScheduleToCloseTimeout: 10 * time.Second,
	// StartToCloseTimeout: 10 * time.Second,
	// ScheduleToStartTimeout: 10 * time.Second,
}
ctx = workflow.WithActivityOptions(ctx, activityOptions)
var yourActivityResult YourActivityResult
err := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

### Set a custom Activity Retry Policy 

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided.

To set a [RetryPolicy](/encyclopedia/retry-policies), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `RetryPolicy` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

- Type: [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy)
- Default:

<!--SNIPSTART go-activities-execution-workflow {"selectedLines": ["21-27"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
// ...
	retryPolicy := &temporal.RetryPolicy{
		InitialInterval:        time.Second,
		BackoffCoefficient:     2.0,
		MaximumInterval:        time.Second * 100, // 100 * InitialInterval
		MaximumAttempts:        0,                 // Unlimited
		NonRetryableErrorTypes: []string{},        // empty
	}
```
<!--SNIPEND-->

Providing a Retry Policy here is a customization, and overwrites individual Field defaults.
The Activity still needs a Start-To-Close or Schedule-To-Close Timeout.

<!--SNIPSTART go-activity-retry-policy-custom-->
[go-sandbox/activities/timeouts/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/timeouts/workflow.go)
```go
retryPolicy := &temporal.RetryPolicy{
	InitialInterval:    time.Second,
	BackoffCoefficient: 2.0,
	MaximumInterval:    time.Second * 100,
}

activityOptions := workflow.ActivityOptions{
	StartToCloseTimeout: 10 * time.Second,
	RetryPolicy:         retryPolicy,
}
ctx = workflow.WithActivityOptions(ctx, activityOptions)
var yourActivityResult YourActivityResult
err := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

### Override the retry interval with `NextRetryDelay` 

You may return an [Application Failure](/references/failures#application-failure) with the `NextRetryDelay` field set.
This value will replace and override whatever the Retry interval would be on the Retry Policy.

For example, if in an Activity, you want to base the interval on the number of attempts:

<!--SNIPSTART go-activity-next-retry-delay-->
[go-sandbox/activities/timeouts/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/timeouts/activity.go)
```go
attempt := activity.GetInfo(ctx).Attempt

return temporal.NewApplicationErrorWithOptions(fmt.Sprintf("Something bad happened on attempt %d", attempt), "NextDelay", temporal.ApplicationErrorOptions{
	NextRetryDelay: 3 * time.Second * time.Duration(attempt),
})
```
<!--SNIPEND-->

## Activity Heartbeats 

An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service).
Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed.
If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy.

Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation.
Heartbeat throttling may lead to Cancellation getting delivered later than expected.

Heartbeats can contain a `details` field describing the Activity's current progress.
If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service.

To [Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) in an Activity in Go, use the `RecordHeartbeat` API.

<!--SNIPSTART go-activity-heartbeat-->
[go-sandbox/activities/timeouts/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/timeouts/activity.go)
```go
func YourActivityDefinition(ctx context.Context, param YourActivityParam) (YourActivityResult, error) {
	for i := range param.Items {
		// Process item i...
		details := fmt.Sprintf("processed item %d", i)
		activity.RecordHeartbeat(ctx, details)
	}
	return YourActivityResult{ProcessedCount: len(param.Items)}, nil
}
```
<!--SNIPEND-->

When an Activity Task Execution times out due to a missed Heartbeat, the last value of the `details` variable above is returned to the calling Workflow in the `details` field of `TimeoutError` with `TimeoutType` set to `Heartbeat`.

You can also Heartbeat an Activity from an external source:

<!--SNIPSTART go-activity-heartbeat-external-->
[go-sandbox/activities/timeouts/heartbeat.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/timeouts/heartbeat.go)
```go
// The client is a heavyweight object that should be created once per process.
temporalClient, err := client.Dial(client.Options{})
if err != nil {
	return err
}
defer temporalClient.Close()

// Record heartbeat.
return temporalClient.RecordActivityHeartbeat(ctx, taskToken, details)
```
<!--SNIPEND-->

The parameters of the `RecordActivityHeartbeat` function are:

- `taskToken`: The value of the binary `TaskToken` field of the `ActivityInfo` struct retrieved inside
  the Activity.
- `details`: The serializable payload containing progress information.

If an Activity Execution Heartbeats its progress before it failed, the retry attempt will have access to the progress information, so that the Activity Execution can resume from the failed state.
Here's an example of how this can be implemented:

<!--SNIPSTART go-activity-heartbeat-resume-->
[go-sandbox/activities/timeouts/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/timeouts/activity.go)
```go
func SampleActivity(ctx context.Context, inputArg InputParams) error {
	startIdx := inputArg.StartIndex
	if activity.HasHeartbeatDetails(ctx) {
		// Recover from finished progress.
		var finishedIndex int
		if err := activity.GetHeartbeatDetails(ctx, &finishedIndex); err == nil {
			startIdx = finishedIndex + 1 // Start from next one.
		}
	}

	// Normal Activity logic...
	for i := startIdx; i < inputArg.EndIndex; i++ {
		// Code for processing item i goes here...
		activity.RecordHeartbeat(ctx, i) // Report progress.
	}
	return nil
}
```
<!--SNIPEND-->

### Set a Heartbeat Timeout 

A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat).

To set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `HeartbeatTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`.

<!--SNIPSTART go-activity-heartbeat-timeout-->
[go-sandbox/activities/timeouts/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/timeouts/workflow.go)
```go
activityOptions := workflow.ActivityOptions{
	StartToCloseTimeout: 10 * time.Minute,
	HeartbeatTimeout:    10 * time.Second,
}
ctx = workflow.WithActivityOptions(ctx, activityOptions)
var yourActivityResult YourActivityResult
err := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

---

# Best practices - Go SDK

Source: https://docs.temporal.io/develop/go/best-practices

> This section explains how to implement best practices with the Go SDK

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Best practices

- [Multithreading](/develop/go/best-practices/multithreading)
- [Error handling](/develop/go/best-practices/error-handling)
- [Debugging](/develop/go/best-practices/debugging)
- [Testing](/develop/go/best-practices/testing-suite)
- [Data handling](/develop/go/data-handling)

---

# Context propagation - Go SDK

Source: https://docs.temporal.io/develop/go/best-practices/context-propagation

> How to propagate custom key-value data across Workflow, Activity, and Child Workflow boundaries using the Temporal Go SDK.

Context propagation lets you pass custom key-value data from a Client to Workflows, and from Workflows to Activities and Child Workflows, without threading it through every function signature. Common use cases include propagating tenant identifiers, auth tokens, or other request-scoped metadata.

> **💡 Tip:**
>
> If you want to propagate tracing context, the Go SDK provides tracing integrations that handle propagation for you.
> [Choose a tracing integration](/develop/go/platform/observability#tracing) before implementing a custom context
> propagator.
>

## How it works

1. **Register** a context propagator on the Client via `ContextPropagators` in [ClientOptions](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions)
2. **Inject** - On outbound calls, the SDK calls `Inject` (from `context.Context`) or `InjectFromWorkflow` (from `workflow.Context`) to serialize values into Temporal headers
3. **Extract** - On inbound calls, the SDK calls `Extract` (into `context.Context`) or `ExtractToWorkflow` (into `workflow.Context`) to deserialize headers back into the context
4. **Access** - Your Workflow and Activity code reads values from the context as usual

## Implement a context propagator

A context propagator implements the [`ContextPropagator`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ContextPropagator) interface:

```go
type ContextPropagator interface {
    // Inject writes values from a Go context.Context into headers (Client/Activity side)
    Inject(context.Context, HeaderWriter) error
    // Extract reads headers into a Go context.Context (Client/Activity side)
    Extract(context.Context, HeaderReader) (context.Context, error)
    // InjectFromWorkflow writes values from a workflow.Context into headers
    InjectFromWorkflow(Context, HeaderWriter) error
    // ExtractToWorkflow reads headers into a workflow.Context
    ExtractToWorkflow(Context, HeaderReader) (Context, error)
}
```

There are two pairs of methods because Go uses `context.Context` in non-Workflow code (Client, Activities) and `workflow.Context` inside Workflows. You must implement all four methods for values to propagate across every boundary (Client → Workflow → Activity/Child Workflow).

Here is a propagator that carries a custom key-value pair from the Client to Workflows and Activities (from the [context propagation sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/go-sandbox/bestpractices/ctxpropagation)):

<!--SNIPSTART go-ctx-propagation-propagator-->
[go-sandbox/bestpractices/ctxpropagation/propagator.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/ctxpropagation/propagator.go)
```go
type (
	// contextKey is an unexported type used as key for items stored in the
	// Context object
	contextKey struct{}

	// propagator implements the custom context propagator
	propagator struct{}

	// Values is a struct holding values
	Values struct {
		Key   string `json:"key"`
		Value string `json:"value"`
	}
)

// PropagateKey is the key used to store the value in the Context object
var PropagateKey = contextKey{}

// HeaderKey is the key used by the propagator to pass values through the
// Temporal server headers
const HeaderKey = "custom-header"

// NewContextPropagator returns a context propagator that propagates a set of
// string key-value pairs across a workflow
func NewContextPropagator() workflow.ContextPropagator {
	return &propagator{}
}

// Inject injects values from context into headers for propagation
func (s *propagator) Inject(ctx context.Context, writer workflow.HeaderWriter) error {
	value := ctx.Value(PropagateKey)
	payload, err := converter.GetDefaultDataConverter().ToPayload(value)
	if err != nil {
		return err
	}
	writer.Set(HeaderKey, payload)
	return nil
}

// InjectFromWorkflow injects values from context into headers for propagation
func (s *propagator) InjectFromWorkflow(ctx workflow.Context, writer workflow.HeaderWriter) error {
	value := ctx.Value(PropagateKey)
	payload, err := converter.GetDefaultDataConverter().ToPayload(value)
	if err != nil {
		return err
	}
	writer.Set(HeaderKey, payload)
	return nil
}

// Extract extracts values from headers and puts them into context
func (s *propagator) Extract(ctx context.Context, reader workflow.HeaderReader) (context.Context, error) {
	if value, ok := reader.Get(HeaderKey); ok {
		var values Values
		if err := converter.GetDefaultDataConverter().FromPayload(value, &values); err != nil {
			return ctx, nil
		}
		ctx = context.WithValue(ctx, PropagateKey, values)
	}

	return ctx, nil
}

// ExtractToWorkflow extracts values from headers and puts them into context
func (s *propagator) ExtractToWorkflow(ctx workflow.Context, reader workflow.HeaderReader) (workflow.Context, error) {
	if value, ok := reader.Get(HeaderKey); ok {
		var values Values
		if err := converter.GetDefaultDataConverter().FromPayload(value, &values); err != nil {
			return ctx, nil
		}
		ctx = workflow.WithValue(ctx, PropagateKey, values)
	}

	return ctx, nil
}
```
<!--SNIPEND-->

## Register the propagator and set context values

Register the propagator on the Client. Then set context values before starting a Workflow:

<!--SNIPSTART go-ctx-propagation-starter-->
[go-sandbox/bestpractices/ctxpropagation/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/ctxpropagation/starter/main.go)
```go
// The client is a heavyweight object that should be created once per process.
c, err := client.Dial(client.Options{
	HostPort:           client.DefaultHostPort,
	ContextPropagators: []workflow.ContextPropagator{ctxpropagation.NewContextPropagator()},
})
if err != nil {
	log.Fatalln("Unable to create client", err)
}
defer c.Close()

workflowID := "ctx-propagation_" + uuid.NewString()
workflowOptions := client.StartWorkflowOptions{
	ID:        workflowID,
	TaskQueue: "ctx-propagation",
}

ctx := context.Background()
ctx = context.WithValue(ctx, ctxpropagation.PropagateKey, &ctxpropagation.Values{Key: "test", Value: "tested"})

we, err := c.ExecuteWorkflow(ctx, workflowOptions, ctxpropagation.CtxPropWorkflow)
```
<!--SNIPEND-->

You can also register context propagators through a [Plugin](/develop/plugins-guide) if you are building a reusable library.

## Access propagated values

In your Workflow, the propagated values are available on the `workflow.Context`. When the Workflow starts an Activity, the SDK automatically propagates the same values:

<!--SNIPSTART go-ctx-propagation-workflow-->
[go-sandbox/bestpractices/ctxpropagation/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/ctxpropagation/workflow.go)
```go
// CtxPropWorkflow workflow definition
func CtxPropWorkflow(ctx workflow.Context) (err error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 2 * time.Second, // such a short timeout to make sample fail over very fast
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	if val := ctx.Value(PropagateKey); val != nil {
		vals := val.(Values)
		workflow.GetLogger(ctx).Info("custom context propagated to workflow", vals.Key, vals.Value)
	}

	var values Values
	if err = workflow.ExecuteActivity(ctx, SampleActivity).Get(ctx, &values); err != nil {
		workflow.GetLogger(ctx).Error("Workflow failed.", "Error", err)
		return err
	}
	workflow.GetLogger(ctx).Info("context propagated to activity", values.Key, values.Value)
	workflow.GetLogger(ctx).Info("Workflow completed.")
	return nil
}
```
<!--SNIPEND-->

<!--SNIPSTART go-ctx-propagation-activity-->
[go-sandbox/bestpractices/ctxpropagation/activities.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/ctxpropagation/activities.go)
```go
func SampleActivity(ctx context.Context) (*Values, error) {
	if val := ctx.Value(PropagateKey); val != nil {
		vals := val.(Values)
		return &vals, nil
	}
	return nil, nil
}
```
<!--SNIPEND-->

You can configure multiple context propagators on a single Client, each responsible for its own set of keys.

## Context propagation over Nexus

Nexus does not use the `ContextPropagator` interface. It relies on a Temporal-agnostic protocol with its own header format (`nexus.Header`, a wrapper around `map[string]string`).

To propagate context over Nexus Operation calls, use interceptors to explicitly serialize and deserialize context into the Nexus header. See the [Nexus Context Propagation sample](https://github.com/temporalio/samples-go/tree/main/nexus-context-propagation).

## Further reading

- [Passing Context with Temporal](https://spiralscout.com/blog/passing-context-with-temporal) - A conceptual guide to middleware and walkthrough of building a context propagator in Go

---

# Debugging - Go SDK

Source: https://docs.temporal.io/develop/go/best-practices/debugging

> Use debugger tools and set TEMPORAL_DEBUG to true for debugging Workflow Definitions with the Temporal Go SDK, and debug production Workflows via Web UI, CLI, or tracing.

You can use a debugger tool provided by your favorite IDE to debug your Workflow Definitions prior to testing or executing them.

The Temporal Go SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control.
Because of this you can often encounter a `PanicError: Potential deadlock detected` while stepping through Workflow Definitions during debugging.

To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to `true` before debugging your Workflow Definition.

> **📝 Note:**
>
> Make sure to set `TEMPORAL_DEBUG` to true only during debugging.
>

## How to debug in a development environment 

In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli).

## How to debug in a production environment 

You can debug production Workflows using:

- [Web UI](/web-ui)
- [Temporal CLI](/cli)
- [Replay](/develop/go/best-practices/testing-suite#replay)
- [Tracing](/develop/go/platform/observability#tracing)
- [Logging](/develop/go/platform/observability#logging)

You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance).
For more information, see [Metrics](/develop/go/platform/observability#metrics) for setting up SDK metrics.

Debug Server performance with [Cloud metrics](/cloud/metrics/openmetrics) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics).

## How to test Workflow Definitions in Go 

The Temporal Go SDK provides a test framework to facilitate testing Workflow implementations.

This framework is suited for implementing unit tests as well as functional tests of the Workflow logic.

The following code implements unit tests for the `SimpleWorkflow` sample:

<!--SNIPSTART go-debugging-unit-test-suite-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
package debugging

import (
	"context"
	"errors"
	"testing"

	"github.com/stretchr/testify/mock"
	"github.com/stretchr/testify/suite"

	"go.temporal.io/sdk/temporal"
	"go.temporal.io/sdk/testsuite"
)

type UnitTestSuite struct {
	suite.Suite
	testsuite.WorkflowTestSuite

	env *testsuite.TestWorkflowEnvironment
}

func (s *UnitTestSuite) SetupTest() {
	s.env = s.NewTestWorkflowEnvironment()
	// Register the real Activity for tests that don't mock it.
	s.env.RegisterActivity(SimpleActivity)
}

func (s *UnitTestSuite) AfterTest(suiteName, testName string) {
	s.env.AssertExpectations(s.T())
}

func (s *UnitTestSuite) Test_SimpleWorkflow_Success() {
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_success")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}

func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityParamCorrect() {
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		func(ctx context.Context, value string) (string, error) {
			s.Equal("test_success", value)
			return value, nil
		})
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_success")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}

func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityFails() {
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		"", errors.New("SimpleActivityFailure"))
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_failure")

	s.True(s.env.IsWorkflowCompleted())

	err := s.env.GetWorkflowError()
	s.Error(err)
	var applicationErr *temporal.ApplicationError
	s.True(errors.As(err, &applicationErr))
	s.Equal("SimpleActivityFailure", applicationErr.Error())
}

func TestUnitTestSuite(t *testing.T) {
	suite.Run(t, new(UnitTestSuite))
}
```
<!--SNIPEND-->

#### Setup

To run unit tests, we first define a test suite struct that absorbs both the
basic suite functionality from [testify](https://pkg.go.dev/github.com/stretchr/testify/suite)
via `suite.Suite` and the suite functionality from the Temporal test framework via
`testsuite.WorkflowTestSuite`. Because every test in this test suite will test our Workflow, we
add a property to our struct to hold an instance of the test environment. This allows us to initialize
the test environment in a setup method. For testing Workflows, we use a `testsuite.TestWorkflowEnvironment`.

Next, we implement a `SetupTest` method to set up a new test environment before each test. Doing so
ensures that each test runs in its own isolated sandbox. We also implement an `AfterTest` function
where we assert that all the mocks we set up were indeed called by invoking `s.env.AssertExpectations(s.T())`.

Timeout for the entire test can be set using `SetTestTimeout` in the Workflow or Activity environment.

Finally, we create a regular test function recognized by the `go test` command, and pass the struct to `suite.Run`.

#### A simple test

The simplest test case we can write is to have the test environment execute the Workflow and then
evaluate the results.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["32-37"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) Test_SimpleWorkflow_Success() {
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_success")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

Calling `s.env.ExecuteWorkflow(...)` executes the Workflow logic and any invoked Activities inside the
test process. The first parameter of `s.env.ExecuteWorkflow(...)` contains the Workflow functions,
and any subsequent parameters contain values for custom input parameters declared by the Workflow
function.

> Note that unless the Activity invocations are mocked or Activity implementation
> replaced (see [Activity mocking and overriding](#activity-mocking-and-overriding)), the test environment
> will execute the actual Activity code including any calls to outside services.

After executing the Workflow in the above example, we assert that the Workflow ran through completion
via the call to `s.env.IsWorkflowComplete()`. We also assert that no errors were returned by asserting
on the return value of `s.env.GetWorkflowError()`. If our Workflow returned a value, we could have
retrieved that value via a call to `s.env.GetWorkflowResult(&value)` and had additional asserts on that
value.

#### Activity mocking and overriding

When running unit tests on Workflows, we want to test the Workflow logic in isolation. Additionally,
we want to inject Activity errors during our test runs. The test framework provides two mechanisms
that support these scenarios: Activity mocking and Activity overriding. Both of these mechanisms allow
you to change the behavior of Activities invoked by your Workflow without the need to modify the actual
Workflow code.

Let's take a look at a test that simulates a test that fails via the "Activity mocking" mechanism.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["51-63"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityFails() {
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		"", errors.New("SimpleActivityFailure"))
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_failure")

	s.True(s.env.IsWorkflowCompleted())

	err := s.env.GetWorkflowError()
	s.Error(err)
	var applicationErr *temporal.ApplicationError
	s.True(errors.As(err, &applicationErr))
	s.Equal("SimpleActivityFailure", applicationErr.Error())
}
```
<!--SNIPEND-->

This test simulates the execution of the Activity `SimpleActivity` that is invoked by our Workflow
`SimpleWorkflow` returning an error. We accomplish this by setting up a mock on the test environment
for the `SimpleActivity` that returns an error.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["52-53"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		"", errors.New("SimpleActivityFailure"))
```
<!--SNIPEND-->

With the mock set up we can now execute the Workflow via the `s.env.ExecuteWorkflow(...)` method and
assert that the Workflow completed successfully and returned the expected error.

Mocking the execution to return a desired value or error isolates Workflow logic for testing. However, sometimes we want to replace the Activity with an alternate implementation
to support a more complex test scenario. Let's assume we want to validate that the Activity gets called
with the correct parameters.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["39-49"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityParamCorrect() {
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		func(ctx context.Context, value string) (string, error) {
			s.Equal("test_success", value)
			return value, nil
		})
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_success")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

In this example, we provide a function implementation as the parameter to `Return`. This allows us to
provide an alternate implementation for the Activity `SimpleActivity`. The framework will execute this
function whenever the Activity is invoked and pass on the return value from the function as the result
of the Activity invocation. Additionally, the framework will validate that the signature of the "mock"
function matches the signature of the original Activity function.

Since this can be an entire function, there is no limitation as to what we can do here. In this
example, we assert that the `value` param has the same content as the value param we passed to the Workflow.

#### Queries

`TestWorkflowEnvironment` instances have a [`QueryWorkflow()` method](https://pkg.go.dev/go.temporal.io/temporal/internal#TestWorkflowEnvironment.QueryWorkflow) that lets you query the state of the currently running Workflow.
For example, suppose you have a Workflow that lets you query the progress of a long running task as shown below.

<!--SNIPSTART go-debugging-progress-workflow-->
[go-sandbox/bestpractices/debugging/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow.go)
```go
func ProgressWorkflow(ctx workflow.Context, percent int) error {
	logger := workflow.GetLogger(ctx)

	err := workflow.SetQueryHandler(ctx, "getProgress", func(input []byte) (int, error) {
		return percent, nil
	})
	if err != nil {
		logger.Info("SetQueryHandler failed.", "Error", err)
		return err
	}

	for percent = 0; percent < 100; percent++ {
		// Important! Use `workflow.Sleep()`, not `time.Sleep()`, because Temporal's
		// test environment doesn't stub out `time.Sleep()`.
		if err := workflow.Sleep(ctx, time.Second*1); err != nil {
			return err
		}
	}

	return nil
}
```
<!--SNIPEND-->

This Workflow tracks the current progress of a task in percentage terms, and increments the percentage by 1 every second.
Below is how you would write a test case that queries this Workflow.
Note that you should always query the Workflow either after `ExecuteWorkflow()` is done or in a `RegisterDelayedCallback()` callback, otherwise you'll get a `runtime error` panic.

<!--SNIPSTART go-debugging-query-test-->
[go-sandbox/bestpractices/debugging/progress_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/progress_test.go)
```go
func (s *UnitTestSuite) Test_ProgressWorkflow() {
	value := 0

	// After 10 seconds plus padding, progress should be 10.
	// Note that `RegisterDelayedCallback()` doesn't actually make your test wait for 10 seconds!
	// Temporal's test framework advances time internally, so this test should take < 1 second.
	s.env.RegisterDelayedCallback(func() {
		res, err := s.env.QueryWorkflow("getProgress")
		s.NoError(err)
		err = res.Get(&value)
		s.NoError(err)
		s.Equal(10, value)
	}, time.Second*10+time.Millisecond*1)

	s.env.ExecuteWorkflow(ProgressWorkflow, 0)

	s.True(s.env.IsWorkflowCompleted())

	// Once the workflow is completed, progress should always be 100
	res, err := s.env.QueryWorkflow("getProgress")
	s.NoError(err)
	err = res.Get(&value)
	s.NoError(err)
	s.Equal(value, 100)
}
```
<!--SNIPEND-->

> **📝 Note:**
>
> `RegisterDelayedCallback` can also be used to send [Signals](/sending-messages#sending-signals).
> When using "Signal-With-Start", set the delay to `0`.

#### Debugging

You can use a debugger tool provided by your favorite IDE to debug your Workflow Definitions prior to testing or executing them.

The Temporal Go SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control.
Because of this you can often encounter a `PanicError: Potential deadlock detected` while stepping through Workflow Definitions during debugging.

To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to `true` before debugging your Workflow Definition.

> **📝 Note:**
>
> Make sure to set `TEMPORAL_DEBUG` to true only during debugging.
>

---

# Error handling - Go SDK

Source: https://docs.temporal.io/develop/go/best-practices/error-handling

> How to handle errors using the Temporal Go SDK.

## Error handling 

Within a Workflow, an Activity or Child Workflow execution might fail. You can
handle errors differently based on the error type.

If the Activity returns an error as `errors.New()` or `fmt.Errorf()`, that error is converted into `*temporal.ApplicationError`.

If the Activity returns an error as `temporal.NewNonRetryableApplicationError("error message", "ErrorType", cause, details)`, that error is returned as `*temporal.ApplicationError`.

There are other types of errors such as `*temporal.TimeoutError`, `*temporal.CanceledError` and
`*temporal.PanicError`.

Here's an example of handling Activity errors within Workflow code that differentiates between different error types.

<!--SNIPSTART go-error-handling-activity-errors-->
[go-sandbox/bestpractices/errorhandling/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/errorhandling/workflow.go)
```go
err := workflow.ExecuteActivity(ctx, YourActivity, failure).Get(ctx, nil)
if err != nil {
	var applicationErr *temporal.ApplicationError
	if errors.As(err, &applicationErr) {
		// retrieve error message
		workflow.GetLogger(ctx).Info("Application error", "error", applicationErr.Error())

		// handle Activity errors (created via NewApplicationError() API)
		var detailMsg string // assuming Activity return error by NewApplicationError("message", "ErrType", "string details")
		if err := applicationErr.Details(&detailMsg); err != nil { // extract strong typed details
			workflow.GetLogger(ctx).Warn("Unable to decode error details", "error", err)
		}

		// handle Activity errors (errors created other than using NewApplicationError() API)
		switch applicationErr.Type() {
		case "CustomErrTypeA":
			// handle CustomErrTypeA
			outcome = "CustomErrTypeA: " + detailMsg
		case CustomErrTypeB:
			// handle CustomErrTypeB
			outcome = "CustomErrTypeB: " + detailMsg
		default:
			// newer version of Activity could return new errors that Workflow was not aware of.
			outcome = "unknown application error"
		}
	}

	var canceledErr *temporal.CanceledError
	if errors.As(err, &canceledErr) {
		// handle cancellation
		outcome = "canceled"
	}

	var timeoutErr *temporal.TimeoutError
	if errors.As(err, &timeoutErr) {
		// handle timeout, could check timeout type by timeoutErr.TimeoutType()
		switch timeoutErr.TimeoutType() {
		case enumspb.TIMEOUT_TYPE_SCHEDULE_TO_START:
			// Handle ScheduleToStart timeout.
			outcome = "ScheduleToStart timeout"
		case enumspb.TIMEOUT_TYPE_START_TO_CLOSE:
			// Handle StartToClose timeout.
			outcome = "StartToClose timeout"
		case enumspb.TIMEOUT_TYPE_HEARTBEAT:
			// Handle heartbeat timeout.
			outcome = "Heartbeat timeout"
		default:
		}
	}

	var panicErr *temporal.PanicError
	if errors.As(err, &panicErr) {
		// handle panic, message and call stack are available by panicErr.Error() and panicErr.StackTrace()
		outcome = "panic"
	}
}
```
<!--SNIPEND-->

### Panics and deferred functions

In Go, [`defer` schedules cleanup functions and `recover()` catches panics](https://go.dev/blog/defer-panic-and-recover) to prevent them from crashing the program.
This doesn't work the same way in Temporal Workflow code — you cannot `recover()` from a panic inside a `defer`.
Deferred functions that try to interact with the Temporal SDK during panic unwinding will re-panic immediately.

Use `defer` only for local cleanup.
Handle Temporal API cleanup through explicit error checks instead.

---

# Temporal Go SDK multithreading

Source: https://docs.temporal.io/develop/go/best-practices/multithreading

> The Temporal Go SDK ensures deterministic multithreading in Workflows using workflow.Go(), avoiding race conditions and eliminating the need for mutexes.

The Temporal Go SDK allows you to create additional goroutines (threads) in your Workflows by calling `workflow.Go()`.
Native Go threading is never allowed in Workflow code, as it would create determinism errors.

You might sometimes need to execute multiple Activities or Child Workflows in parallel and then await the result of all of them.
Normally, this would require a lock or [mutex](https://en.wikipedia.org/wiki/Lock_(computer_science)) around some shared data structure to avoid race conditions that could occur when multiple asynchronous operations try to modify the data structure.

Although Temporal Workflows run Asynchronously in Go, there is a control in place that ensures only one thread can access at a time.

## How multithreading works

Temporal's Go SDK contains a deterministic runner to control the thread execution.
This deterministic runner will decide which Workflow thread to run in the right order, and one at a time.
Each task will execute in a loop until all threads are blocked.

`workflow.Go()` creates a new thread and adds it to this runner.
This significantly minimizes the likelihood of race conditions, and eliminates the need to use a mutex.

For a complex example, refer to the [Go Particle Swarm Operation Sample](https://github.com/temporalio/samples-go/tree/main/pso).

For an example using Signals, refer to the [Go Await Signal Sample](https://github.com/temporalio/samples-go/tree/main/await-signals)

## Static analysis with the workflowcheck tool

The Temporal Go SDK also provides a command line tool called [`workflowcheck`](https://github.com/temporalio/sdk-go/blob/main/contrib/tools/workflowcheck/README.md) to statically analyze Workflow Definitions. This can help eliminate potential instances of non-determinism.

---

# Testing - Go SDK

Source: https://docs.temporal.io/develop/go/best-practices/testing-suite

> The Testing section of the Temporal Application development guide details frameworks for Workflow and integration testing. Create end-to-end, integration, unit tests, and more for Workflows and Activities. Each test runs in an isolated environment, ensuring accurate and reliable testing. Discover how to mock and override Activities, test

The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing.

In the context of Temporal, you can create these types of automated tests:

- **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client.
- **Integration:** Anything between end-to-end and unit testing.
  - Running Activities with mocked Context and other SDK imports (and usually network requests).
  - Running Workers with mock Activities, and using a Client to start Workflows.
  - Running Workflows with mocked SDK imports.
- **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls.

We generally recommend writing the majority of your tests as integration tests.

Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers.

## Test frameworks 

The Temporal Go SDK provides a test framework to facilitate testing Workflow implementations. This framework is suited for implementing unit tests as well as functional tests of the Workflow logic.

## Test setup

To run unit tests, we first define a test suite struct that absorbs both the basic suite functionality from [testify](https://pkg.go.dev/github.com/stretchr/testify/suite) via `suite.Suite` and the suite functionality from the Temporal test framework via `testsuite.WorkflowTestSuite`.

Because every test in this test suite will test our Workflow, we
add a property to our struct to hold an instance of the test environment. This allows us to initialize the test environment in a setup method.

For testing Workflows, we use a `testsuite.TestWorkflowEnvironment`.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["15-20"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
type UnitTestSuite struct {
	suite.Suite
	testsuite.WorkflowTestSuite

	env *testsuite.TestWorkflowEnvironment
}
```
<!--SNIPEND-->

Next, we implement a `SetupTest` method to set up a new test environment before each test.
Doing so ensures that each test runs in its own isolated sandbox.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["22-26"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) SetupTest() {
	s.env = s.NewTestWorkflowEnvironment()
	// Register the real Activity for tests that don't mock it.
	s.env.RegisterActivity(SimpleActivity)
}
```
<!--SNIPEND-->

We also implement an `AfterTest` function where we assert that all the mocks we set up were indeed called by invoking `s.env.AssertExpectations(s.T())`.
Timeout for the entire test can be set using `SetTestTimeout` in the Workflow or Activity environment.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["28-30"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) AfterTest(suiteName, testName string) {
	s.env.AssertExpectations(s.T())
}
```
<!--SNIPEND-->

Finally, we create a regular test function recognized by the `go test` command, and pass the struct to `suite.Run`.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["65-67"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func TestUnitTestSuite(t *testing.T) {
	suite.Run(t, new(UnitTestSuite))
}
```
<!--SNIPEND-->

## Testing Activities 

An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity.
This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity.

Here's an example of a simple Activity that will be referenced in the sections below:

<!--SNIPSTART go-quickstart-activity-->
[go-sandbox/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activity.go)
```go
package quickstart

import (
	"context"
	"fmt"
)

func Greet(ctx context.Context, name string) (string, error) {
	return fmt.Sprintf("Hello %s", name), nil
}
```
<!--SNIPEND-->

### Run an Activity 

If an Activity references its context, you need to mock that context when testing in isolation. You can do that by using [`NewTestActivityEnvironment`](https://pkg.go.dev/go.temporal.io/sdk/testsuite#TestActivityEnvironment) from the [`WorkflowTestSuite`](https://pkg.go.dev/go.temporal.io/sdk/testsuite#WorkflowTestSuite).

<!--SNIPSTART go-testing-activity-suite-->
[go-sandbox/activity_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activity_test.go)
```go
package quickstart_test

import (
	"testing"

	"github.com/stretchr/testify/suite"
	"go.temporal.io/sdk/testsuite"

	"my-org/quickstart"
)

type ActivityTestSuite struct {
	suite.Suite
	testsuite.WorkflowTestSuite
}

func TestActivityTestSuite(t *testing.T) {
	suite.Run(t, new(ActivityTestSuite))
}

func (s *ActivityTestSuite) TestGreet() {
	env := s.NewTestActivityEnvironment()
	env.RegisterActivity(quickstart.Greet)

	result, err := env.ExecuteActivity(quickstart.Greet, "Temporal")

	s.NoError(err)

	var greeting string
	s.NoError(result.Get(&greeting))
	s.Equal("Hello Temporal", greeting)
}
```
<!--SNIPEND-->

### Listen to Heartbeats 

When an Activity sends a Heartbeat, be sure that you can see the Heartbeats in your test code so that you can verify them.

<!--SNIPSTART go-testing-activity-heartbeat-->
[go-sandbox/activity_heartbeat_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activity_heartbeat_test.go)
```go
func (s *ActivityTestSuite) Test_HeartbeatActivity() {
	testSuite := &testsuite.WorkflowTestSuite{}
	env := testSuite.NewTestActivityEnvironment()

	env.RegisterActivity(quickstart.GreetWithHeartbeat)

	var heartBeatCount int
	env.SetOnActivityHeartbeatListener(func(activityInfo *activity.Info, details converter.EncodedValues) {
		var val string
		err := details.Get(&val)
		s.NoError(err)
		s.Equal("status-report-to-workflow", val)
		heartBeatCount++
	})

	_, err := env.ExecuteActivity(quickstart.GreetWithHeartbeat, "Temporal")
	s.NoError(err)
	s.Equal(1, heartBeatCount)
}
```
<!--SNIPEND-->

### Cancel an Activity 

If an Activity is supposed to react to a Cancellation, you can test whether it reacts correctly by canceling it.

<!--SNIPSTART go-testing-activity-cancel-->
[go-sandbox/activity_heartbeat_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activity_heartbeat_test.go)
```go
func (s *ActivityTestSuite) Test_CancelActivity() {
	testSuite := &testsuite.WorkflowTestSuite{}
	env := testSuite.NewTestActivityEnvironment()

	ctx, cancel := context.WithCancel(context.Background())

	env.SetWorkerOptions(worker.Options{
		BackgroundActivityContext: ctx,
	})
	env.RegisterActivity(quickstart.LongRunningGreet)

	done := make(chan struct{})

	go func() {
		defer close(done)
		// Cancel the activity after 2s
		time.Sleep(2 * time.Second)
		cancel()
	}()

	_, err := env.ExecuteActivity(quickstart.LongRunningGreet, "Temporal")

	<-done

	// Expect the activity to return a canceled error
	s.Error(err)
}
```
<!--SNIPEND-->

## Testing Workflows 

When running unit tests on Workflows, we want to test the Workflow logic in isolation.
The simplest test case we can write is to have the test environment execute the Workflow and then evaluate the results.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["32-37"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) Test_SimpleWorkflow_Success() {
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_success")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

Calling `s.env.ExecuteWorkflow(...)` executes the Workflow logic and any invoked Activities inside the test process. The first parameter of `s.env.ExecuteWorkflow(...)` contains the Workflow functions, and any subsequent parameters contain values for custom input parameters declared by the Workflow.

> Note that unless the Activity invocations are mocked or Activity implementation replaced (see [Activity mocking and overriding](#mock-activities)), the test environment will execute the actual Activity code including any calls to outside services.

After executing the Workflow in the above example, we assert that the Workflow ran through completion via the call to `s.env.IsWorkflowCompleted()`. We also assert that no errors were returned by asserting on the return value of `s.env.GetWorkflowError()`.
If our Workflow returned a value, we could have retrieved that value via a call to `s.env.GetWorkflowResult(&value)` and had additional asserts on that value.

### Query tests

`TestWorkflowEnvironment` instances have a [`QueryWorkflow()` method](https://pkg.go.dev/go.temporal.io/temporal/internal#TestWorkflowEnvironment.QueryWorkflow) that lets you query the state of the currently running Workflow.
For example, suppose you have a Workflow that lets you query the progress of a long running task as shown below.

<!--SNIPSTART go-debugging-progress-workflow-->
[go-sandbox/bestpractices/debugging/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow.go)
```go
func ProgressWorkflow(ctx workflow.Context, percent int) error {
	logger := workflow.GetLogger(ctx)

	err := workflow.SetQueryHandler(ctx, "getProgress", func(input []byte) (int, error) {
		return percent, nil
	})
	if err != nil {
		logger.Info("SetQueryHandler failed.", "Error", err)
		return err
	}

	for percent = 0; percent < 100; percent++ {
		// Important! Use `workflow.Sleep()`, not `time.Sleep()`, because Temporal's
		// test environment doesn't stub out `time.Sleep()`.
		if err := workflow.Sleep(ctx, time.Second*1); err != nil {
			return err
		}
	}

	return nil
}
```
<!--SNIPEND-->

This Workflow tracks the current progress of a task in percentage terms, and increments the percentage by 1 every second.
Below is how you would write a test case that queries this Workflow.
Note that you should always query the Workflow either after `ExecuteWorkflow()` is done or in a `RegisterDelayedCallback()` callback, otherwise you'll get a `runtime error` panic.

<!--SNIPSTART go-debugging-query-test-->
[go-sandbox/bestpractices/debugging/progress_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/progress_test.go)
```go
func (s *UnitTestSuite) Test_ProgressWorkflow() {
	value := 0

	// After 10 seconds plus padding, progress should be 10.
	// Note that `RegisterDelayedCallback()` doesn't actually make your test wait for 10 seconds!
	// Temporal's test framework advances time internally, so this test should take < 1 second.
	s.env.RegisterDelayedCallback(func() {
		res, err := s.env.QueryWorkflow("getProgress")
		s.NoError(err)
		err = res.Get(&value)
		s.NoError(err)
		s.Equal(10, value)
	}, time.Second*10+time.Millisecond*1)

	s.env.ExecuteWorkflow(ProgressWorkflow, 0)

	s.True(s.env.IsWorkflowCompleted())

	// Once the workflow is completed, progress should always be 100
	res, err := s.env.QueryWorkflow("getProgress")
	s.NoError(err)
	err = res.Get(&value)
	s.NoError(err)
	s.Equal(value, 100)
}
```
<!--SNIPEND-->

> **📝 Note:**
>
> `RegisterDelayedCallback` can also be used to send [Signals](/sending-messages#sending-signals).
> When using "Signal-With-Start", set the delay to `0`.

### How to mock Activities 

When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker to test the Workflow logic in isolation.

Additionally, you can inject Activity errors during your test runs. The test framework provides two mechanisms that support these scenarios: Activity mocking and Activity overriding. Both of these mechanisms allow you to change the behavior of Activities invoked by your Workflow without the need to modify the actual Workflow code.

Let's take a look at a test that simulates a test that fails via the Activity mocking.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["51-63"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityFails() {
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		"", errors.New("SimpleActivityFailure"))
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_failure")

	s.True(s.env.IsWorkflowCompleted())

	err := s.env.GetWorkflowError()
	s.Error(err)
	var applicationErr *temporal.ApplicationError
	s.True(errors.As(err, &applicationErr))
	s.Equal("SimpleActivityFailure", applicationErr.Error())
}
```
<!--SNIPEND-->

This test simulates the execution of the Activity `SimpleActivity` that is invoked by our Workflow `SimpleWorkflow` returning an error. We accomplish this by setting up a mock on the test environment for the `SimpleActivity` that returns an error.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["52-53"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		"", errors.New("SimpleActivityFailure"))
```
<!--SNIPEND-->

With the mock set up we can now execute the Workflow via the `s.env.ExecuteWorkflow(...)` method and assert that the Workflow completed successfully and returned the expected error.

Mocking the execution to return a desired value or error isolates Workflow logic for testing.
However, sometimes we want to replace the Activity with an alternate implementation to support a more complex test scenario.
Let's assume we want to validate that the Activity gets called with the correct parameters.

<!--SNIPSTART go-debugging-unit-test-suite {"selectedLines": ["39-49"]}-->
[go-sandbox/bestpractices/debugging/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/debugging/workflow_test.go)
```go
// ...
func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityParamCorrect() {
	s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return(
		func(ctx context.Context, value string) (string, error) {
			s.Equal("test_success", value)
			return value, nil
		})
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_success")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

In this example, we provide a function implementation as the parameter to `Return`.
This allows us to provide an alternate implementation for the Activity `SimpleActivity`.
The framework will execute this function whenever the Activity is invoked and pass on the return value from the function as the result of the Activity invocation.

Additionally, the framework will validate that the signature of the "mock" function matches the signature of the original Activity function.

Since this can be an entire function, there is no limitation as to what we can do here. In this example, we assert that the `value` param has the same content as the value param we passed to the Workflow.

### How to mock Nexus operations

Mocking Nexus operations lets you test a Workflow that executes Nexus operations without needing a
Nexus handler to run the actual Nexus operation. You can mock the Nexus operation or override its
implementation with the test Workflow environment.

Consider a test that simulates a Nexus operation call. In this example, the Nexus operation is
called `sample-operation`, the input type is `SampleInput`, the output type is `SampleOutput`, and
it belongs to the Nexus service `sample-service`.

The example below mocks a call to a Nexus synchronous operation, indicated by the returned value
type `*nexus.HandlerStartOperationResultSync[T]`. Since `OnNexusOperation` needs to know the
operation's name, input type and output type, and you might not have access to the Nexus operation
on the handler side, you can use `nexus.NewOperationReference` to create a Nexus operation reference
that represents the operation without its implementation (basically, it represents the signature of
the Nexus operation). You may also use the operation itself instead of creating the operation
reference if you have it available.

<!--SNIPSTART go-testing-nexus-mock-sync-->
[go-sandbox/bestpractices/testing/nexusmock/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/testing/nexusmock/workflow_test.go)
```go
func (s *UnitTestSuite) Test_SimpleWorkflow_NexusSyncOperation() {
	s.env.OnNexusOperation(
		"sample-service",
		nexus.NewOperationReference[SampleInput, SampleOutput]("sample-operation"),
		SampleInput{},
		mock.Anything, // NexusOperationOptions
	).Return(
		&nexus.HandlerStartOperationResultSync[SampleOutput]{
			Value: SampleOutput{},
		},
		nil, // error if you want to simulate an error in the ExecuteOperation call
	)
	// You can also add a delay to return the mock values by calling After().
	// Eg: s.env.OnNexusOperation(...).Return(...).After(1*time.Second)

	s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

Besides the synchronous operations, you can also mock asynchronous operations. The following example
demonstrates how to test a Workflow executing a Nexus asynchronous operation. The returned value
type in this case must be `*nexus.HandlerStartOperationResultAsync` with an `OperationToken`, which can
be any string of your choice. Furthermore, you must call `RegisterNexusAsyncOperationCompletion` to
register the result of the asynchronous operation identified by the tuple service name, operation
name, and operation token.

<!--SNIPSTART go-testing-nexus-mock-async-->
[go-sandbox/bestpractices/testing/nexusmock/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/testing/nexusmock/workflow_test.go)
```go
func (s *UnitTestSuite) Test_SimpleWorkflow_NexusAsyncOperation() {
	s.env.OnNexusOperation(
		"sample-service",
		nexus.NewOperationReference[SampleInput, SampleOutput]("sample-operation"),
		SampleInput{},
		mock.Anything, // NexusOperationOptions
	).Return(
		&nexus.HandlerStartOperationResultAsync{
			OperationToken: "sample-operation-token",
		},
		nil, // error if you want to simulate an error in the ExecuteOperation call
	)
	err := s.env.RegisterNexusAsyncOperationCompletion(
		"sample-service",
		"sample-operation",
		"sample-operation-token", // must match the OperationToken above
		SampleOutput{},
		nil,           // error if you want to simulate an error in the operation
		2*time.Second, // delay to simulate how long the operation takes after it starts
	)
	s.NoError(err)
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

If your Workflow executes multiple Nexus asynchronous operations, you can mock each of them with
different operation tokens, and register the completion results using the corresponding operation token.

If mocking Nexus operations is not enough, and you need to run some custom logic when the Nexus
operation is executed, you can override it as follows.

<!--SNIPSTART go-testing-nexus-override-sync-->
[go-sandbox/bestpractices/testing/nexusmock/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/testing/nexusmock/workflow_test.go)
```go
func (s *UnitTestSuite) Test_SimpleWorkflow_NexusSyncOperationOverride() {
	var SampleOperation = nexus.NewSyncOperation(
		"sample-operation",
		func(ctx context.Context, input SampleInput, options nexus.StartOperationOptions) (SampleOutput, error) {
			// Custom logic here.
			return SampleOutput{}, nil
		},
	)

	service := nexus.NewService("sample-service")
	s.NoError(service.Register(SampleOperation))
	s.env.RegisterNexusService(service)
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

The following example shows how to override a Nexus asynchronous operation.

<!--SNIPSTART go-testing-nexus-override-async-->
[go-sandbox/bestpractices/testing/nexusmock/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/testing/nexusmock/workflow_test.go)
```go
func (s *UnitTestSuite) Test_SimpleWorkflow_NexusAsyncOperationOverride() {
	// The handler Workflow must be a named function that is registered with the test environment.
	s.env.RegisterWorkflow(SampleHandlerWorkflow)
	SampleOperation := temporalnexus.NewWorkflowRunOperation(
		"sample-operation",
		SampleHandlerWorkflow,
		func(ctx context.Context, input SampleInput, options nexus.StartOperationOptions) (client.StartWorkflowOptions, error) {
			// Custom logic to build client.StartWorkflowOptions.
			return client.StartWorkflowOptions{ID: "sample-handler-workflow"}, nil
		},
	)

	service := nexus.NewService("sample-service")
	s.NoError(service.Register(SampleOperation))
	s.env.RegisterNexusService(service)
	s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation")

	s.True(s.env.IsWorkflowCompleted())
	s.NoError(s.env.GetWorkflowError())
}
```
<!--SNIPEND-->

### How to skip time 

Some long-running Workflows can persist for months or even years.
Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount.

For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works.
Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner.

The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time.
Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests.
If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server.
For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel.

#### Skip time automatically 

The Go SDK test environment automatically skips time when possible. When using the `testsuite.TestWorkflowEnvironment`, time advances automatically whenever there are no Activities running. This means:

- Workflow timers (like `workflow.Sleep`) are fast-forwarded
- Time doesn't skip while Activities are executing

Example:

<!--SNIPSTART go-testing-auto-time-skipping-->
[go-sandbox/bestpractices/testing/timeskip/workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/testing/timeskip/workflow_test.go)
```go
import (
	"testing"

	"go.temporal.io/sdk/testsuite"
)

func TestWorkflow_AutoTimeSkipping(t *testing.T) {
	var ts testsuite.WorkflowTestSuite
	env := ts.NewTestWorkflowEnvironment()

	env.ExecuteWorkflow(MyWorkflow)

	if !env.IsWorkflowCompleted() {
		t.Fatal("workflow did not complete")
	}

	if err := env.GetWorkflowError(); err != nil {
		t.Fatal(err)
	}
}
```
<!--SNIPEND-->

#### Skip time manually 

By default, this test suite uses a mock Workflow clock which automatically moves forward to fire the next Timer when the Workflow is blocked. In addition to automatic skipping, you can manually control time progression in the Go test environment.

Use `env.RegisterDelayedCallback()` to create a new Timer with a specified `delayDuration` using the mock Workflow clock. When the Timer fires, the callback will be called. This is useful when you want fine-grained control over when Timers fire.

Example:

<!--SNIPSTART go-testing-sleep-for-days-->
[go-sandbox/sleep_for_days_workflow_test.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/sleep_for_days_workflow_test.go)
```go
package quickstart

import (
	"testing"
	"time"

	"github.com/stretchr/testify/mock"
	"github.com/stretchr/testify/require"

	"go.temporal.io/sdk/testsuite"
)

func TestSleepForDaysWorkflow(t *testing.T) {
	testSuite := &testsuite.WorkflowTestSuite{}
	env := testSuite.NewTestWorkflowEnvironment()

	numActivityCalls := 0
	env.RegisterActivity(SendEmailActivity)
	env.OnActivity(SendEmailActivity, mock.Anything, mock.Anything).Run(
		func(args mock.Arguments) { numActivityCalls++ },
	).Return(nil)

	startTime := env.Now()

	// Time-skip 90 days.
	env.RegisterDelayedCallback(func() {
		// Check that the activity has been called 3 times.
		require.Equal(t, 3, numActivityCalls)
		// Send the signal to complete the workflow.
		env.SignalWorkflow("complete", nil)
		// Expect no more activity calls to have been made - workflow is complete.
		require.Equal(t, 3, numActivityCalls)
		// Expect more than 90 days to have passed.
		require.Equal(t, env.Now().Sub(startTime), time.Hour*24*90)
	}, time.Hour*24*90)

	// Execute workflow.
	env.ExecuteWorkflow(SleepForDaysWorkflow)
}
```
<!--SNIPEND-->

## How to Replay a Workflow Execution 

Replay recreates the exact state of a Workflow Execution.
You can replay a Workflow from the beginning of its Event History.

Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view.

When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks:

1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test.
2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI.
3. Run the Event Histories through replay.
4. Fail CI if any error is encountered during replay.

The following are examples of fetching and replaying Event Histories:

Use the [worker.WorkflowReplayer](https://pkg.go.dev/go.temporal.io/sdk/worker#WorkflowReplayer) to replay an existing Workflow Execution from its Event History to replicate errors.

For example, the following code retrieves the Event History of a Workflow:

<!--SNIPSTART go-testing-get-workflow-history-->
[go-sandbox/bestpractices/testing/replay/replay.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/testing/replay/replay.go)
```go
func GetWorkflowHistory(ctx context.Context, client client.Client, id, runID string) (*history.History, error) {
	var hist history.History
	iter := client.GetWorkflowHistory(ctx, id, runID, false, enums.HISTORY_EVENT_FILTER_TYPE_ALL_EVENT)
	for iter.HasNext() {
		event, err := iter.Next()
		if err != nil {
			return nil, err
		}
		hist.Events = append(hist.Events, event)
	}
	return &hist, nil
}
```
<!--SNIPEND-->

This history can then be used to _replay_.
For example, the following code creates a `WorkflowReplayer` and register the `YourWorkflow` Workflow function.
Then it calls the `ReplayWorkflowHistory` to _replay_ the Event History and return an error code.

<!--SNIPSTART go-testing-replay-workflow-->
[go-sandbox/bestpractices/testing/replay/replay.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/testing/replay/replay.go)
```go
func ReplayWorkflow(ctx context.Context, client client.Client, id, runID string) error {
	hist, err := GetWorkflowHistory(ctx, client, id, runID)
	if err != nil {
		return err
	}
	replayer := worker.NewWorkflowReplayer()
	replayer.RegisterWorkflow(YourWorkflow)
	return replayer.ReplayWorkflowHistory(nil, hist)
}
```
<!--SNIPEND-->

The code above will cause the Worker to re-execute the Workflow's Workflow Function using the original Event History.
If a noticeably different code path was followed or some code caused a deadlock, it will be returned in the error code.
Replaying a Workflow Execution locally is a good way to see exactly what code path was taken for given input and events.

You can replay many Event Histories by registering all the needed Workflow implementation and then calling `ReplayWorkflowHistory` repeatedly.

---

# Client - Go SDK

Source: https://docs.temporal.io/develop/go/client

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Temporal Client

- [Temporal Client](/develop/go/client/temporal-client)
- [Namespaces](/develop/go/client/namespaces)

---

# Namespaces - Go SDK

Source: https://docs.temporal.io/develop/go/client/namespaces

> Register and manage Namespaces in Temporal using CLI or SDK APIs. Isolate Workflow Executions, match development lifecycles, and secure Namespace workflows.

This page shows how to do the following:

- [Register a Namespace](#register-namespace)
- [Manage Namespaces](#manage-namespaces)

You can create, update, deprecate, or delete your [Namespaces](/namespaces) using either the Temporal CLI or SDK APIs.

Use Namespaces to isolate your Workflow Executions according to your needs.
For example, you can use Namespaces to match the development lifecycle by having separate `dev` and `prod` Namespaces.
You could also use them to ensure Workflow Executions between different teams never communicate - such as ensuring that the `teamA` Namespace never impacts the `teamB` Namespace.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) to create and manage a Namespace from the UI, or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces from the command line.

On self-hosted Temporal Service, you can register and manage your Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

You must register a Namespace with the Temporal Service before setting it in the Temporal Client.

## Register a Namespace 

Registering a Namespace creates a Namespace on the Temporal Service or Temporal Cloud.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to create Namespaces.

On self-hosted Temporal Service, you can register your Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

Use [`Register` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) with the `NamespaceClient` interface to register a [Namespace](/namespaces) and set the [Retention Period](/temporal-service/temporal-server#retention-period) for the Workflow Execution Event History for the Namespace.

You can also [register Namespaces using the Temporal CLI command-line tool](/cli/command-reference/operator#create).

<!--SNIPSTART go-namespace-register-->
[go-sandbox/client/namespaces/namespaces.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/namespaces/namespaces.go)
```go
namespaceClient, err := client.NewNamespaceClient(client.Options{HostPort: client.DefaultHostPort})
if err != nil {
	return err
}
defer namespaceClient.Close()

retention := 3 * 24 * time.Hour
err = namespaceClient.Register(ctx, &workflowservice.RegisterNamespaceRequest{
	Namespace:                        namespaceName,
	WorkflowExecutionRetentionPeriod: durationpb.New(retention),
})
```
<!--SNIPEND-->

The Retention Period setting using `WorkflowExecutionRetentionPeriod` is mandatory.
The minimum value you can set for this period is 1 day.

Once registered, set Namespace using `Dial` in a Workflow Client to run your Workflow Executions within that Namespace.
See [how to set Namespace in a Client in Go](/develop/go/client/temporal-client#connect-to-temporal-cloud) for details.

Note that Namespace registration using this API takes up to 10 seconds to complete.
Ensure that you wait for this registration to complete before starting the Workflow Execution against the Namespace.

To update your Namespace, use the [`Update` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) with the `NamespaceClient`.

To update your Namespace using the Temporal CLI, use the [temporal operator namespace update](/cli/command-reference/operator#update) command.

## Manage Namespaces 

You can get details for your Namespaces, update Namespace configuration, and deprecate or delete your Namespaces.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces.

On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

You must register a Namespace with the Temporal Service before setting it in the Temporal Client.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces.

On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

- Update information and configuration for a registered Namespace on your Temporal Service:

  - With the Temporal CLI: [`temporal operator namespace update`](/cli/command-reference/operator#update)
    Example
  - Use the [`UpdateNamespace` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) to update configuration on a Namespace.
    Example

    <!--SNIPSTART go-namespace-update-->
    [go-sandbox/client/namespaces/namespaces.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/namespaces/namespaces.go)
    ```go
    err := namespaceClient.Update(ctx, &workflowservice.UpdateNamespaceRequest{
    	Namespace: "your-namespace-name",
    	UpdateInfo: &namespacepb.UpdateNamespaceInfo{ // updates info for the namespace "your-namespace-name"
    		Description: "updated namespace description",
    		OwnerEmail:  "newowner@mail.com",
    		// Data:  nil,
    		// State: 0,
    	},
    	/* other details that you can update:
    	Config: &namespacepb.NamespaceConfig{ // updates the configuration of the namespace with the following options
    		// WorkflowExecutionRetentionTtl: nil,
    		// BadBinaries:                   nil,
    		// HistoryArchivalState:          0,
    		// HistoryArchivalUri:            "",
    		// VisibilityArchivalState:       0,
    		// VisibilityArchivalUri:         "",
    	},
    	ReplicationConfig: &replication.NamespaceReplicationConfig{ // updates the replication configuration for the namespace
    		// ActiveClusterName: "",
    		// Clusters:          nil,
    		// State:             0,
    	},
    	SecurityToken:    "",
    	DeleteBadBinary:  "",
    	PromoteNamespace: false,
    	*/
    })
    ```
    <!--SNIPEND-->

- Get details for a registered Namespace on your Temporal Service:

  - With the Temporal CLI: [`temporal operator namespace describe`](/cli/command-reference/operator#describe)
  - Use the [`DescribeNamespace` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) to return information and configuration details for a registered Namespace.
    Example

    <!--SNIPSTART go-namespace-describe-->
    [go-sandbox/client/namespaces/namespaces.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/namespaces/namespaces.go)
    ```go
    namespaceClient, err := client.NewNamespaceClient(client.Options{})
    if err != nil {
    	return nil, err
    }
    defer namespaceClient.Close()

    description, err := namespaceClient.Describe(ctx, "default")
    ```
    <!--SNIPEND-->

- Get details for all registered Namespaces on your Temporal Service:

  - With the Temporal CLI: [`temporal operator namespace list`](/cli/command-reference/operator#list)
  - Use the [`ListNamespaces` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) on the Workflow Service to return information and configuration details for all registered Namespaces on your Temporal Service.
    Example

  <!--SNIPSTART go-namespace-list-->
  [go-sandbox/client/namespaces/namespaces.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/namespaces/namespaces.go)
  ```go
  var names []string
  var nextPageToken []byte
  for {
  	// Each call returns one page of Namespaces. Loop until NextPageToken is empty.
  	resp, err := temporalClient.WorkflowService().ListNamespaces(ctx, &workflowservice.ListNamespacesRequest{
  		PageSize:      100,
  		NextPageToken: nextPageToken,
  	})
  	if err != nil {
  		return nil, err
  	}
  	for _, ns := range resp.GetNamespaces() {
  		names = append(names, ns.GetNamespaceInfo().GetName())
  	}
  	nextPageToken = resp.GetNextPageToken()
  	if len(nextPageToken) == 0 {
  		break
  	}
  }
  ```
  <!--SNIPEND-->

- Delete a Namespace: The [`DeleteNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/operatorservice/v1/service.proto) deletes a Namespace. Deleting a Namespace deletes all running and completed Workflow Executions on the Namespace, and removes them from the persistence store and the visibility store.
  Example:

  <!--SNIPSTART go-namespace-delete-->
  [go-sandbox/client/namespaces/namespaces.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/namespaces/namespaces.go)
  ```go
  _, err := temporalClient.OperatorService().DeleteNamespace(ctx, &operatorservice.DeleteNamespaceRequest{
  	Namespace: namespaceName,
  })
  ```
  <!--SNIPEND-->

---

# Temporal Client - Go SDK

Source: https://docs.temporal.io/develop/go/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the
[Temporal Service](/temporal-service). Communication with a Temporal Service lets you perform actions such as starting
Workflow Executions, sending Signals to Workflow Executions, sending Queries to Workflow Executions, getting the results
of a Workflow Execution, and providing Activity Task Tokens.

For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow.

This page shows you how to do the following using the Go SDK with the Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-development-service)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Start a Workflow Execution](#start-workflow-execution)
- [Get Workflow results](#get-workflow-results)

> **⚠️ Caution:**
>
> A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a
> Temporal Client inside an Activity to communicate with a Temporal Service.
>

## Connect to development Temporal Service 

Use the [`Dial()`](https://pkg.go.dev/go.temporal.io/sdk/client#Dial) API available in the
[`go.temporal.io/sdk/client`](https://pkg.go.dev/go.temporal.io/sdk/client) package to create a
[`Client`](https://pkg.go.dev/go.temporal.io/sdk/client#Client). The `Dial()` API expects connection options such as the
Temporal Server address, the Namespace to connect to, and Transport Layer Security (TLS) configuration. You can specify
these options in the function call, or specify them using environment variables or a configuration file. We recommend
you use environment variables or a configuration file to manage these connection options securely.

> **ℹ️ Info:**
> Versioning Requirements
>
> Environment variable and configuration file support were added in Go SDK v1.28.0.
>

When you are running a Temporal Service locally, such as the
[Temporal CLI](/cli/command-reference/server#start-dev), the connection options you must provide are minimal.

If you don't provide [`HostPort`](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions), the Client defaults the
address and port number to `127.0.0.1:7233`, which is the port of the development Temporal Service. If you don't set a
custom Namespace name in the Namespace field, the client connects to the default Namespace.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the
TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK
looks for it at the path `~/.config/temporalio/temporal.toml`. For a list of all available configuration options, refer
to [Environment Configuration](/references/client-environment-configuration)

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own
set of connection options.

```toml
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
# TLS is auto-enabled when this TLS config or API key is present, but you can configure it explicitly
# disabled = false
# Use certificate files for mTLS
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment = "production"
service-version = "v1.2.3"
```

You can create a Temporal Client using a specific profile from the configuration file as follows:

<!--SNIPSTART go-client-connect-profile-->
[go-sandbox/client/devprofile/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/devprofile/main.go)
```go
package main

import (
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
)

func main() {
	// Load a specific profile from the TOML config file.
	// This requires a [profile.prod] section in your config.
	opts, err := envconfig.LoadClientOptions(envconfig.LoadClientOptionsRequest{
		ConfigFileProfile: "prod",
	})
	if err != nil {
		log.Fatalf("Failed to load 'prod' profile: %v", err)
	}

	c, err := client.Dial(opts)
	if err != nil {
		log.Fatalf("Failed to connect using 'prod' profile: %v", err)
	}
	defer c.Close()

	fmt.Printf("✅ Connected to Temporal namespace %q on %s using 'prod' profile\n", opts.Namespace, opts.HostPort)
}
```
<!--SNIPEND-->

**Environment Variables**

Use the `envconfig` package to set connection options for the Temporal Client using environment variables. For a list of
all available environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

<!--SNIPSTART go-client-connect-env-->
[go-sandbox/client/devenv/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/devenv/main.go)
```go
package main

import (
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
)

func main() {
	// Loads the "default" profile from the standard location and environment variables.
	c, err := client.Dial(envconfig.MustLoadDefaultClientOptions())
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}
	defer c.Close()

	fmt.Println("✅ Connected to Temporal Service")
}
```
<!--SNIPEND-->

**Code**

If you don't want to use environment variables or a configuration file, you can specify connection options directly in
code. This is convenient for local development and testing. You can also load a base configuration from environment
variables or a configuration file, and then override specific options in code.

<!--SNIPSTART go-client-connect-code-->
[go-sandbox/client/httpserver/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/httpserver/main.go)
```go
package main

import (
	"log"
	"net/http"

	"go.temporal.io/sdk/client"
)

func main() {
	// Create a Temporal Client to communicate with the Temporal Service.
	// A Temporal Client should be created once per process.
	temporalClient, err := client.Dial(client.Options{
		HostPort: client.DefaultHostPort,
	})
	if err != nil {
		log.Fatalln("Unable to create Temporal Client", err)
	}
	defer temporalClient.Close()

	// Start an HTTP server and listen on /start
	http.HandleFunc("/start", func(w http.ResponseWriter, r *http.Request) {
		startWorkflowHandler(w, r, temporalClient)
	})
	err = http.ListenAndServe(":8091", nil)
	if err != nil {
		log.Fatalln("Unable to run http server", err)
	}
}
```
<!--SNIPEND-->

## Connect to Temporal Cloud 

You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal
Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured
local development instance:

- Your credentials for authentication.
  - If you are using an API key, provide the API key value.
  - If you are using mTLS, provide the mTLS CA certificate and mTLS private key.
- Your _Namespace and Account ID_ combination, which follows the format `<namespace_id>.<account_id>`.
- The recommended _endpoint_ is the gRPC Namespace endpoint: `<namespace>.<account>.tmprl.cloud:7233`.
  This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability).
  See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options.

You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab.

For more information about managing and generating client certificates for Temporal Cloud, see
[How to manage certificates in Temporal Cloud](/cloud/certificates).

You can provide these connection options using environment variables, a configuration file, or directly in code.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to
[Environment Configuration](/references/client-environment-configuration).

You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path
to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the
default path `~/.config/temporalio/temporal.toml`.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines a `cloud` profile with the necessary connection options to
connect to Temporal Cloud via an API key:

```toml
# Cloud profile for Temporal Cloud
[profile.cloud]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```

If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and
private key:

```toml
# Cloud profile for Temporal Cloud
[profile.cloud]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
tls_client_cert_data = "your-tls-client-cert-data"
tls_client_key_path = "your-tls-client-key-path"
```

With the connections options defined in the configuration file, use the `LoadClientOptions` function in the `envconfig`
package to create a Temporal Client using the `cloud` profile as follows:

<!--SNIPSTART go-client-cloud-profile {"highlightedLines": "13-17"}-->
[go-sandbox/client/cloudprofile/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloudprofile/main.go)
```go {13-17}
package main

import (
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
)

func main() {
	// Replace with the actual path to your TOML file.
	configFilePath := "/Users/yourname/.config/my-app/temporal.toml"

	opts, err := envconfig.LoadClientOptions(envconfig.LoadClientOptionsRequest{
		ConfigFilePath:    configFilePath,
		ConfigFileProfile: "cloud",
	})
	if err != nil {
		log.Fatalf("Failed to load client config from custom file: %v", err)
	}

	c, err := client.Dial(opts)
	if err != nil {
		log.Fatalf("Failed to connect using custom config file: %v", err)
	}
	defer c.Close()

	fmt.Printf("✅ Connected using custom config at: %s\n", configFilePath)
}
```
<!--SNIPEND-->

**Environment Variables**

The following environment variables are required to connect to Temporal Cloud:

- `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `<namespace_id>.<account_id>`.
- `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace.
- `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication.
- `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path.
  Required if you are using mTLS authentication.
- `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path.
  Required if you are using mTLS authentication.

Ensure these environment variables exist in your environment before running your Go application.

Import the `envconfig` package to set connection options for the Temporal Client using environment variables. The
`MustLoadDefaultClientOptions` function will automatically load all environment variables. For a list of all available
environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

<!--SNIPSTART go-client-connect-env {"highlightedLines": "13"}-->
[go-sandbox/client/devenv/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/devenv/main.go)
```go {13}
package main

import (
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
)

func main() {
	// Loads the "default" profile from the standard location and environment variables.
	c, err := client.Dial(envconfig.MustLoadDefaultClientOptions())
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}
	defer c.Close()

	fmt.Println("✅ Connected to Temporal Service")
}
```
<!--SNIPEND-->

**Code**

You can also provide connections options in your Go code directly. When instantiating a Temporal `client` in your
Temporal Go SDK code, provide the following `clientOptions`:

<!--SNIPSTART go-client-cloud-api-key-->
[go-sandbox/client/cloud/cloud.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloud/cloud.go)
```go
clientOptions := client.Options{
	HostPort:          endpoint,  // <namespace>.<account>.tmprl.cloud:7233
	Namespace:         namespace, // <namespace_id>.<account_id>
	ConnectionOptions: client.ConnectionOptions{TLS: &tls.Config{}},
	Credentials:       client.NewAPIKeyStaticCredentials(apiKey),
}
c, err := client.Dial(clientOptions)
```
<!--SNIPEND-->

To update an API key, use the Go `context` object:

<!--SNIPSTART go-client-cloud-api-key-dynamic-->
[go-sandbox/client/cloud/cloud.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloud/cloud.go)
```go
// Assuming client Credentials created with
myKey := initialKey
creds := client.NewAPIKeyDynamicCredentials(
	func(context.Context) (string, error) { return myKey, nil })
// Update by replacing
myKey = myKeyUpdated
```
<!--SNIPEND-->

To rotate an mTLS client certificate without restarting your Worker, set `GetClientCertificate` on the `tls.Config`
instead of setting `Certificates` directly. Go's standard library `crypto/tls` package calls this function on every new
connection, so it always picks up the current certificate:

<!--SNIPSTART go-client-cloud-mtls-rotation-->
[go-sandbox/client/cloud/cloud.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloud/cloud.go)
```go
clientOptions := client.Options{
	HostPort:  endpoint,  // <namespace>.<account>.tmprl.cloud:7233
	Namespace: namespace, // <namespace_id>.<account_id>
	ConnectionOptions: client.ConnectionOptions{
		TLS: &tls.Config{
			GetClientCertificate: func(*tls.CertificateRequestInfo) (*tls.Certificate, error) {
				cert, err := tls.LoadX509KeyPair(clientCertPath, clientKeyPath)
				if err != nil {
					return nil, err
				}
				return &cert, nil
			},
		},
	},
}
c, err := client.Dial(clientOptions)
```
<!--SNIPEND-->

Rotate the certificate by overwriting the files at `clientCertPath` and `clientKeyPath`; the next time the connection
re-establishes (for example, after Temporal Cloud's periodic connection recycling), `GetClientCertificate` reads the new
files. This works because `ConnectionOptions.TLS` is a real `*tls.Config`, so any option `crypto/tls` supports is
available. See this [reference implementation](https://github.com/temporal-sa/temporal-worker-cert-rotation) for a full
walkthrough, including automating certificate issuance with cert-manager on Kubernetes.

You can use a combination of environment variables, configuration files, and code to set connection options. For
example, you can load a base configuration from environment variables or a configuration file, and then override
specific options in code.

For example, the following code snippet loads the base configuration from environment variables and the default profile
with `envconfig.MustLoadDefaultClientOptions()`. It then overrides the `HostPort` and `Namespace` options
programmatically. Refer to
[Client Options type in the Go SDK](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions) for a list of all
available connection options you can override.

<!--SNIPSTART go-client-override-->
[go-sandbox/client/override/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/override/main.go)
```go
package main

import (
	"fmt"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
)

func main() {
	// Load the base configuration (e.g., from the default profile).
	opts := envconfig.MustLoadDefaultClientOptions()

	// Apply overrides programmatically.
	opts.HostPort = "localhost:7233"
	opts.Namespace = "test-namespace"

	c, err := client.Dial(opts)
	if err != nil {
		log.Fatalf("Failed to connect with overridden options: %v", err)
	}
	defer c.Close()

	fmt.Printf("✅ Connected with overridden config to: %s in namespace: %s\n", opts.HostPort, opts.Namespace)
}
```
<!--SNIPEND-->

#### <strong>v1.26.0 to v1.32.x </strong>

Create an initial connection:

<!--SNIPSTART go-client-cloud-api-key-v126-->
[go-sandbox/client/cloud/legacy.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloud/legacy.go)
```go
clientOptions := client.Options{
	HostPort:  endpoint,  // <namespace>.<account>.tmprl.cloud:7233
	Namespace: namespace, // <namespace_id>.<account_id>
	ConnectionOptions: client.ConnectionOptions{
		TLS: &tls.Config{},
		DialOptions: []grpc.DialOption{
			grpc.WithUnaryInterceptor(
				func(ctx context.Context, method string, req any, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
					return invoker(
						metadata.AppendToOutgoingContext(ctx, "temporal-namespace", namespace),
						method,
						req,
						reply,
						cc,
						opts...,
					)
				},
			),
		},
	},
	Credentials: client.NewAPIKeyStaticCredentials(apiKey),
}
c, err := client.Dial(clientOptions)
if err != nil {
	log.Fatalf("error creating temporal client: %v", err)
}
```
<!--SNIPEND-->

Update an API key:

<!--SNIPSTART go-client-cloud-api-key-dynamic-->
[go-sandbox/client/cloud/cloud.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloud/cloud.go)
```go
// Assuming client Credentials created with
myKey := initialKey
creds := client.NewAPIKeyDynamicCredentials(
	func(context.Context) (string, error) { return myKey, nil })
// Update by replacing
myKey = myKeyUpdated
```
<!--SNIPEND-->

#### <strong>pre v1.26.0</strong>

Create an initial connection:

<!--SNIPSTART go-client-cloud-api-key-headers-provider-->
[go-sandbox/client/cloud/legacy.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloud/legacy.go)
```go
// Create headers provider
type APIKeyProvider struct {
	APIKey    string
	Namespace string
}

func (a *APIKeyProvider) GetHeaders(context.Context) (map[string]string, error) {
	return map[string]string{"Authorization": "Bearer " + a.APIKey, "temporal-namespace": a.Namespace}, nil
}

// DialWithHeadersProvider is the API key connection for Go SDK versions before v1.26.0.
func DialWithHeadersProvider(endpoint, namespace, apiKey string) (client.Client, *APIKeyProvider, error) {
	// Use headers provider
	apiKeyProvider := &APIKeyProvider{APIKey: apiKey, Namespace: namespace}
	c, err := client.Dial(client.Options{
		HostPort:          endpoint,  // <namespace>.<account>.tmprl.cloud:7233
		Namespace:         namespace, // <namespace_id>.<account_id>
		HeadersProvider:   apiKeyProvider,
		ConnectionOptions: client.ConnectionOptions{TLS: &tls.Config{}},
	})
	return c, apiKeyProvider, err
}
```
<!--SNIPEND-->

Update an API key:

<!--SNIPSTART go-client-cloud-api-key-headers-provider-update-->
[go-sandbox/client/cloud/legacy.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/cloud/legacy.go)
```go
apiKeyProvider.APIKey = myKeyUpdated
```
<!--SNIPEND-->

## Start a Workflow Execution 

[Workflow Execution](/workflow-execution) semantics rely on several parameters. To start a Workflow Execution, you must
supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific
contextual data, and Workflow Function parameters.

In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from
within another Workflow Execution, use either the [Child Workflow](/develop/go/workflows/child-workflows) or External Workflow
APIs.

See the [Customize Workflow Type](/develop/go/workflows/basics#customize-workflow-type) section to see how to customize
the name of the Workflow Type.

A request to spawn a Workflow Execution causes the Temporal Service to create the first Event
([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The
Temporal Service then creates the first Workflow Task, resulting in the first
[WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event.

To spawn a [Workflow Execution](/workflow-execution), use the `ExecuteWorkflow()` method on the Go SDK
[`Client`](https://pkg.go.dev/go.temporal.io/sdk/client#Client).

The `ExecuteWorkflow()` API call requires an instance of [`context.Context`](https://pkg.go.dev/context#Context), an
instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions), a Workflow Type
name, and all variables to be passed to the Workflow Execution. The `ExecuteWorkflow()` call returns a Future, which can
be used to get the result of the Workflow Execution.

<!--SNIPSTART go-client-start-workflow-->
[go-sandbox/client/start/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/start/main.go)
```go
package main

import (
	"context"
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/workflow"
)

func main() {
	temporalClient, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer temporalClient.Close()
	// ...
	workflowOptions := client.StartWorkflowOptions{
		TaskQueue: "your-task-queue",
		// ...
	}
	param := YourWorkflowParam{Name: "Temporal"}
	workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
	if err != nil {
		log.Fatalln("Unable to execute workflow", err)
	}
	log.Println("Started workflow", "WorkflowID", workflowRun.GetID(), "RunID", workflowRun.GetRunID())
	// ...
}

func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (YourWorkflowResponse, error) {
	// ...
	return YourWorkflowResponse{Greeting: "Hello " + param.Name}, nil
}
```
<!--SNIPEND-->

If the invocation process has access to the function directly, then the Workflow Type name parameter can be passed as if
the function name were a variable, without quotations.

<!--SNIPSTART go-client-start-by-function-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
```
<!--SNIPEND-->

If the invocation process does not have direct access to the statically defined Workflow Definition, for example, if the
Workflow Definition is in an un-importable package, or it is written in a completely different language, then the
Workflow Type can be provided as a `string`.

<!--SNIPSTART go-client-start-by-name-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, "YourWorkflowDefinition", param)
```
<!--SNIPEND-->

### Set a Workflow's Task Queue 

The only Workflow Option that you must set is the name of the [Task Queue](/task-queue).

For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task
Queue name.

Create an instance of
[`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the
`go.temporal.io/sdk/client` package, set the `TaskQueue` field, and pass the instance to the `ExecuteWorkflow` call.

- Type: `string`
- Default: None, this is a required field to be set by the developer

<!--SNIPSTART go-client-option-task-queue-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	// ...
	TaskQueue: "your-task-queue",
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

You can configure Task Queues that are host-specific, Worker-specific or Workflow-specific to distribute your
application load. For more information, refer to
[Task Queues Processing Tuning](/develop/worker-performance/task-queues#task-queues-processing-tuning) and
[Worker Versioning](/worker-versioning).

### Set a Workflow ID 

Although it is not required, we recommend providing your own
[Workflow ID](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity
identifier, such as an order identifier or customer identifier.

Create an instance of
[`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the
`go.temporal.io/sdk/client` package, set the `ID` field, and pass the instance to the `ExecuteWorkflow` call.

- Type: `string`
- Default: System generated UUID

<!--SNIPSTART go-client-option-id-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	// ...
	ID:        "Your-Custom-Workflow-Id",
	TaskQueue: taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

### Go StartWorkflowOptions reference 

Create an instance of
[`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the
`go.temporal.io/sdk/client` package, and pass the instance to the `ExecuteWorkflow` call.

The following fields are available:

| Field                                                                                   | Required | Type                                                                                            |
| --------------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------- |
| [`ID`](#id)                                                                             | No       | `string`                                                                                        |
| [`TaskQueue`](#taskqueue)                                                               | **Yes**  | `string`                                                                                        |
| [`WorkflowExecutionTimeout`](#workflowexecutiontimeout)                                 | No       | `time.Duration`                                                                                 |
| [`WorkflowRunTimeout`](#workflowruntimeout)                                             | No       | `time.Duration`                                                                                 |
| [`WorkflowTaskTimeout`](#workflowtasktimeout)                                           | No       | `time.Duration`                                                                                 |
| [`WorkflowIDReusePolicy`](#workflowidreusepolicy)                                       | No       | [`WorkflowIdReusePolicy`](https://pkg.go.dev/go.temporal.io/api/enums/v1#WorkflowIdReusePolicy) |
| [`WorkflowExecutionErrorWhenAlreadyStarted`](#workflowexecutionerrorwhenalreadystarted) | No       | `bool`                                                                                          |
| [`RetryPolicy`](#retrypolicy)                                                           | No       | [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy)                     |
| [`CronSchedule`](#cronschedule)                                                         | No       | `string`                                                                                        |
| [`Memo`](#memo)                                                                         | No       | `map[string]interface{}`                                                                        |
| [`SearchAttributes`](#searchattributes)                                                 | No       | `map[string]interface{}`                                                                        |

#### ID

Although it is not required, we recommend providing your own
[Workflow ID](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity
identifier, such as an order identifier or customer identifier.

Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions)
from the `go.temporal.io/sdk/client` package, set the `ID` field, and pass the instance to the `ExecuteWorkflow` call.

- Type: `string`
- Default: System generated UUID

<!--SNIPSTART go-client-option-id-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	// ...
	ID:        "Your-Custom-Workflow-Id",
	TaskQueue: taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### TaskQueue

Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions)
from the `go.temporal.io/sdk/client` package, set the `TaskQueue` field, and pass the instance to the `ExecuteWorkflow`
call.

- Type: `string`
- Default: None; this is a required field to be set by the developer

<!--SNIPSTART go-client-option-task-queue-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	// ...
	TaskQueue: "your-task-queue",
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### WorkflowExecutionTimeout

Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the
`go.temporal.io/sdk/client` package, set the `WorkflowExecutionTimeout` field, and pass the instance to the
`ExecuteWorkflow` call.

- Type: `time.Duration`
- Default: Unlimited

<!--SNIPSTART go-workflow-timeouts-set-->
[go-sandbox/workflows/timeouts/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/timeouts/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	ID:        "your-workflow-id",
	TaskQueue: TaskQueueName,
	// Set Workflow Timeout duration
	WorkflowExecutionTimeout: 24 * 365 * 10 * time.Hour,
	// WorkflowRunTimeout: 24 * 365 * 10 * time.Hour,
	// WorkflowTaskTimeout: 10 * time.Second,
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### WorkflowRunTimeout

Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the
`go.temporal.io/sdk/client` package, set the `WorkflowRunTimeout` field, and pass the instance to the `ExecuteWorkflow`
call.

- Type: `time.Duration`
- Default: Same as [`WorkflowExecutionTimeout`](#workflowexecutiontimeout)

<!--SNIPSTART go-client-option-run-timeout-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	WorkflowRunTimeout: 24 * 365 * 10 * time.Hour,
	TaskQueue:          taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### WorkflowTaskTimeout

Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the
`go.temporal.io/sdk/client` package, set the `WorkflowTaskTimeout` field, and pass the instance to the `ExecuteWorkflow`
call.

- Type: `time.Duration`
- Default: 10 seconds

<!--SNIPSTART go-client-option-task-timeout-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	WorkflowTaskTimeout: 10 * time.Second,
	TaskQueue:           taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### WorkflowIDReusePolicy

- Type: [WorkflowIdReusePolicy](https://pkg.go.dev/go.temporal.io/api/enums/v1#WorkflowIdReusePolicy)
- Default: `enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE`

Set a value from the `go.temporal.io/api/enums/v1` package.

<!--SNIPSTART go-client-option-id-reuse-policy-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	WorkflowIDReusePolicy: enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE,
	TaskQueue:             taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### WorkflowExecutionErrorWhenAlreadyStarted

- Type: `bool`
- Default: `false`

<!--SNIPSTART go-client-option-error-when-already-started-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	WorkflowExecutionErrorWhenAlreadyStarted: false,
	TaskQueue:                                taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### RetryPolicy

Create an instance of a [RetryPolicy](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) from the
`go.temporal.io/sdk/temporal` package and provide it as the value to the `RetryPolicy` field of the instance of
`StartWorkflowOptions`.

- Type: [RetryPolicy](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy)
- Default: None

<!--SNIPSTART go-workflow-retry-policy-->
[go-sandbox/workflows/timeouts/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/timeouts/start.go)
```go
retryPolicy := &temporal.RetryPolicy{
	InitialInterval:    time.Second,
	BackoffCoefficient: 2.0,
	MaximumInterval:    time.Second * 100,
}
workflowOptions := client.StartWorkflowOptions{
	ID:          "your-workflow-id",
	TaskQueue:   TaskQueueName,
	RetryPolicy: retryPolicy,
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### CronSchedule

- Type: `string`
- Default: None

<!--SNIPSTART go-schedule-cron-->
[go-sandbox/workflows/schedules/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	ID:           "your-cron-workflow-id",
	TaskQueue:    TaskQueue,
	CronSchedule: "15 8 * * *",
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

[Sample](https://github.com/temporalio/samples-go/tree/main/cron)

#### Memo

- Type: `map[string]interface{}`
- Default: Empty

<!--SNIPSTART go-client-option-memo-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	Memo: map[string]interface{}{
		"description": "Test search attributes workflow",
	},
	TaskQueue: taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

#### SearchAttributes

- Type: `map[string]interface{}`
- Default: Empty.

These are the corresponding [Search Attribute value types](/search-attribute#supported-types) in Go:

- Keyword = string
- Int = int64
- Double = float64
- Bool = bool
- Datetime = time.Time
- Text = string

<!--SNIPSTART go-client-option-search-attributes-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
searchAttributes := map[string]interface{}{
	"CustomIntField": 1,
	"MiscData":       "yellow",
}
workflowOptions := client.StartWorkflowOptions{
	SearchAttributes: searchAttributes,
	TaskQueue:        taskQueue,
	// ...
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

## Get Workflow results 

If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run ID.

The Workflow ID, Run ID, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its
result.

It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time
(asynchronous execution).

In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and
results of Workflow Executions.

The `ExecuteWorkflow` call returns an instance of
[`WorkflowRun`](https://pkg.go.dev/go.temporal.io/sdk/client#WorkflowRun), which is the `workflowRun` variable in the
following line.

<!--SNIPSTART go-client-start-by-function-->
[go-sandbox/client/options/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/start.go)
```go
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
```
<!--SNIPEND-->

The instance of `WorkflowRun` has the following three methods:

- `GetID()`: Returns the Workflow ID of the invoked Workflow Execution.
- `GetRunID()`: Always returns the Run ID of the initial Run (see
  [Continue-As-New](/workflow-execution/continue-as-new)) in the series of Runs that make up the full Workflow
  Execution.
- `Get`: Takes a pointer as a parameter and populates the associated variable with the Workflow Execution result.

To wait on the result of Workflow Execution in the same process that invoked it, call `Get()` on the instance of
`WorkflowRun` that is returned by the `ExecuteWorkflow()` call.

<!--SNIPSTART go-client-get-result-->
[go-sandbox/client/options/results.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/results.go)
```go
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param)
if err != nil {
	return YourWorkflowResponse{}, err
}
var result YourWorkflowResponse
err = workflowRun.Get(context.Background(), &result)
if err != nil {
	return YourWorkflowResponse{}, err
}
```
<!--SNIPEND-->

However, the result of a Workflow Execution can be obtained from a completely different process. All that is needed is
the [Workflow ID](/workflow-execution/workflowid-runid#workflow-id).
(A [Run ID](/workflow-execution/workflowid-runid#run-id) is optional if more than one closed Workflow Execution has
the same Workflow ID.) The result of the Workflow Execution is available for as long as the Workflow Execution Event
History remains in the system.

Call the `GetWorkflow()` method on an instance of the Go SDK Client and pass it the Workflow ID used to spawn the
Workflow Execution. Then call the `Get()` method on the instance of `WorkflowRun` that is returned, passing it a pointer
to populate the result.

<!--SNIPSTART go-client-get-workflow-->
[go-sandbox/client/options/results.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/results.go)
```go
workflowID := "Your-Custom-Workflow-Id"
// An empty Run ID gets the latest Run of the Workflow Execution.
workflowRun := temporalClient.GetWorkflow(context.Background(), workflowID, "")

var result YourWorkflowResponse
err := workflowRun.Get(context.Background(), &result)
if err != nil {
	return YourWorkflowResponse{}, err
}
```
<!--SNIPEND-->

### Get last completion result

In the case of a [Temporal Cron Job](/cron-job), you might need to get the result of the previous Workflow Run and use
it in the current Workflow Run.

To do this, use the [`HasLastCompletionResult`](https://pkg.go.dev/go.temporal.io/sdk/workflow#HasLastCompletionResult)
and [`GetLastCompletionResult`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetLastCompletionResult) APIs, available
from the [`go.temporal.io/sdk/workflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow) package, directly in your
Workflow code.

<!--SNIPSTART go-client-last-completion-result-->
[go-sandbox/client/options/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/client/options/workflow.go)
```go
type CronResult struct {
	Count int
}

func YourCronWorkflowDefinition(ctx workflow.Context) (CronResult, error) {
	count := 1

	if workflow.HasLastCompletionResult(ctx) {
		var lastResult CronResult
		if err := workflow.GetLastCompletionResult(ctx, &lastResult); err == nil {
			count = count + lastResult.Count
		}
	}

	newResult := CronResult{
		Count: count,
	}
	return newResult, nil
}
```
<!--SNIPEND-->

This will work even if one of the cron Workflow Runs fails. The next Workflow Run gets the result of the last
successfully Completed Workflow Run.

---

# Data handling - Go SDK

Source: https://docs.temporal.io/develop/go/data-handling

All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three
layers that handle different concerns:

![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)

Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON
serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when
your application requires non-JSON types, encryption, or payload offloading.

|                           | [PayloadConverter](/develop/go/data-handling/data-conversion) | [PayloadCodec](/develop/go/data-handling/data-encryption) | [ExternalStorage](/develop/go/data-handling/external-storage) |
| ------------------------- | ------------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------ |
| **Purpose**               | Serialize application data to bytes                           | Transform encoded payloads (encrypt, compress)            | Offload large payloads to external store                           |
| **Default**               | JSON serialization                                            | None (passthrough)                                        | None (all payloads are stored in Event History)                  |

For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage).

---

# Payload conversion - Go SDK

Source: https://docs.temporal.io/develop/go/data-handling/data-conversion

> Customize how Temporal serializes application objects using Payload Converters in the Go SDK, including composite data converters and custom type examples.

Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back.

The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter.
You can set multiple encoding Payload Converters to run your conversions.
When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

Payload Converters can be customized independently of a Payload Codec.
Temporal's Converter architecture looks like this:

![Temporal converter architecture](https://docs.temporal.io/img/info/converter-architecture.png)

## Use a custom Payload Converter 

Use a [Composite Data Converter](https://pkg.go.dev/go.temporal.io/sdk/converter#CompositeDataConverter) to apply custom, type-specific Payload Converters in a specified order.
Defining a new Composite Data Converter is not always necessary to implement custom data handling.
You can override the default Converter with a custom Codec, but a Composite Data Converter may be necessary for complex Workflow logic.

`NewCompositeDataConverter` creates a new instance of `CompositeDataConverter` from an ordered list of type-specific Payload Converters.
The following type-specific Payload Converters are available in the Go SDK, listed in the order that they are applied by the default Data Converter:

- [NewNilPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#NilPayloadConverter.ToString)
- [NewByteSlicePayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#ByteSlicePayloadConverter)
- [NewProtoJSONPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#ProtoJSONPayloadConverter)
- [NewProtoPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#ProtoPayloadConverter)
- [NewJSONPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#JSONPayloadConverter)

The order in which the Payload Converters are applied is important because during serialization the Data Converter tries the Payload Converters in that specific order until a Payload Converter returns a non-nil Payload.

To set your custom Payload Converter, use [`NewCompositeDataConverter`](https://pkg.go.dev/go.temporal.io/sdk/converter#NewCompositeDataConverter) and set it as the Data Converter in the Client options.

- To replace the default Data Converter with a custom `NewCompositeDataConverter`, use the following.

  <!--SNIPSTART go-data-conversion-replace-default-->
  [go-sandbox/bestpractices/dataconversion/data_converter.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/dataconversion/data_converter.go)
  ```go
  dataConverter := converter.NewCompositeDataConverter(YourCustomPayloadConverter())
  ```
  <!--SNIPEND-->

- To add your custom type conversion to the default Data Converter, use the following to keep the defaults but set yours just before the default JSON fall through.

  <!--SNIPSTART go-data-conversion-extend-default-->
  [go-sandbox/bestpractices/dataconversion/data_converter.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/dataconversion/data_converter.go)
  ```go
  dataConverter := converter.NewCompositeDataConverter(
  	converter.NewNilPayloadConverter(),
  	converter.NewByteSlicePayloadConverter(),
  	converter.NewProtoJSONPayloadConverter(),
  	converter.NewProtoPayloadConverter(),
  	YourCustomPayloadConverter(),
  	converter.NewJSONPayloadConverter(),
  )
  ```
  <!--SNIPEND-->

`YourCustomPayloadConverter()` returns your implementation of the
[`PayloadConverter`](https://pkg.go.dev/go.temporal.io/sdk/converter#PayloadConverter) interface.
For a complete example that encodes a custom `Money` type, see
[money_converter.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/dataconversion/money_converter.go).

---

# Payload encryption - Go SDK

Source: https://docs.temporal.io/develop/go/data-handling/data-encryption

> Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Go SDK.

Temporal's security model is designed around client-side encryption of Payloads.
A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server.
This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data.
It also gives implementers more power and more freedom regarding which client is able to read which data. Implementers can control access with keys, algorithms, or other security measures.

A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client.
Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these.

The server itself never adds encryption over Payloads.
Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows.
When working with sensitive data, you should always implement Payload encryption.

## Use a custom Payload Codec 

**Step 1: Create a custom Payload Codec**

Create a custom [PayloadCodec](https://pkg.go.dev/go.temporal.io/sdk/converter#PayloadCodec) implementation and define your encryption/compression and decryption/decompression logic in the `Encode` and `Decode` functions.

The Payload Codec converts bytes to bytes.
It must be used in an instance of [CodecDataConverter](https://pkg.go.dev/go.temporal.io/sdk/converter#CodecDataConverter) that wraps a Data Converter to do the [Payload](/dataconversion#payload) conversions, and applies the custom encoding and decoding in `PayloadCodec` to the converted Payloads.

The following example from the [Data Converter sample](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/mycodecpackage/data_converter.go) shows how to create a custom `NewCodecDataConverter` that wraps an instance of a Data Converter with a custom `PayloadCodec`.

<!--SNIPSTART go-codec-data-converter-->
[go-sandbox/bestpractices/mycodecpackage/data_converter.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/mycodecpackage/data_converter.go)
```go
// Create an instance of Data Converter with your codec.
var DataConverter = converter.NewCodecDataConverter(
	converter.GetDefaultDataConverter(),
	NewPayloadCodec(),
)

// Create an instance of PayloadCodec.
func NewPayloadCodec() converter.PayloadCodec {
	return &Codec{}
}
```
<!--SNIPEND-->

Implement your encryption/compression logic in the `Encode` function and the decryption/decompression logic in the `Decode` function in your custom `PayloadCodec`, as shown in the following example.

<!--SNIPSTART go-codec-payload-codec-->
[go-sandbox/bestpractices/mycodecpackage/codec.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/mycodecpackage/codec.go)
```go
// Codec implements converter.PayloadEncoder for snappy compression.
type Codec struct{}

// Encode implements converter.PayloadCodec.Encode.
func (Codec) Encode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		// Marshal proto
		origBytes, err := p.Marshal()
		if err != nil {
			return payloads, err
		}
		// Compress
		b := snappy.Encode(nil, origBytes)
		result[i] = &commonpb.Payload{
			Metadata: map[string][]byte{converter.MetadataEncoding: []byte("binary/snappy")},
			Data:     b,
		}
	}

	return result, nil
}

// Decode implements converter.PayloadCodec.Decode.
func (Codec) Decode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		// Decode only if it's our encoding
		if string(p.Metadata[converter.MetadataEncoding]) != "binary/snappy" {
			result[i] = p
			continue
		}
		// Uncompress
		b, err := snappy.Decode(nil, p.Data)
		if err != nil {
			return payloads, err
		}
		// Unmarshal proto
		result[i] = &commonpb.Payload{}
		err = result[i].Unmarshal(b)
		if err != nil {
			return payloads, err
		}
	}

	return result, nil
}
```
<!--SNIPEND-->

### Handle Payload Codec failures

Payload Codec methods can run during Workflow Task processing or in a Client or Activity context.
Handle transient failures within the Payload Codec when possible, such as with bounded retries.

During Workflow Task processing, the SDK handles an error returned by `Encode` or `Decode` according to the operation. Some operations convert the error to a panic and apply the Worker's configured [`WorkflowPanicPolicy`](https://pkg.go.dev/go.temporal.io/sdk/worker#WorkflowPanicPolicy); other returned errors can fail the Workflow Execution. To apply the policy consistently, panic from the Payload Codec instead of returning an error.
With the default [`BlockWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/worker#BlockWorkflow) policy, the panic fails the current Workflow Task so that the Temporal Service can retry it.
With the [`FailWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/worker#FailWorkflow) policy, the panic instead fails the Workflow Execution.

Panicking outside Workflow Task processing does not request a Workflow Task retry.

**Step 2: Set Data Converter to use custom Payload Codec.**

Set your custom `PayloadCodec` with an instance of `DataConverter` in your `Dial` client options that you use to create the client.

The following example shows how to set your custom Data Converter from a package called `mycodecpackage`.

<!--SNIPSTART go-codec-client-->
[go-sandbox/bestpractices/mycodecpackage/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/mycodecpackage/starter/main.go)
```go
c, err := client.Dial(client.Options{
	// Set DataConverter here to ensure that Workflow inputs and results are
	// encoded as required.
	DataConverter: mycodecpackage.DataConverter,
})
```
<!--SNIPEND-->

- Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Service. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted.
- Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Service. Instead, they are stored encoded on the Temporal Service, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output.

For reference, see the [Encryption](https://github.com/temporalio/samples-go/tree/main/encryption) sample.

### Using a Codec Server

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
The Codec Server is independent of the Temporal Service and decodes your encrypted payloads through predefined endpoints.
You create, operate, and manage access to your Codec Server in your own environment.
The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.
Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server.

For reference, see the [Codec server](https://github.com/temporalio/samples-go/tree/main/codec-server) sample.

---

# External Storage - Go SDK

Source: https://docs.temporal.io/develop/go/data-handling/external-storage

> Offload large payloads to external storage using the claim check pattern in the Go SDK.

> **Public Preview**
> APIs and configuration may change before General Availability. Join the [#large-payloads Slack
> channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for help.

The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted
deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external
storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how
to set up External Storage with Amazon S3 or Google Cloud Storage, and how to implement a custom storage driver.

For a conceptual overview of External Storage and its use cases, see [External Storage](/external-storage).

## Store and retrieve large payloads with Amazon S3 or Google Cloud Storage 

The Go SDK includes storage drivers for Amazon S3 and Google Cloud Storage. Select your storage backend in the tabs that
follow. Only the driver setup differs between the two. Everything after that is the same.

### Prerequisites

- A bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) to
  ensure that your payloads remain available for the entire lifetime of the Workflow. For multi-region durability, see
  [Durable External Storage](/external-storage#durable-external-storage).
- Bucket credentials on both your Temporal Client and your Workers, since each reaches the bucket directly. Refer to
  [Amazon S3 access control](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-access-control.html) or
  [Cloud Storage IAM permissions](https://cloud.google.com/storage/docs/access-control/iam-permissions) for how to grant
  access.
- Permission to write objects on processes that store payloads, and to read objects on processes that retrieve them.
  Storing also requires read permission, because the drivers check whether an object already exists before uploading.
- Install the driver module, the client adapter for your cloud provider's SDK, and that SDK:

**Amazon S3**

  ```sh
  go get go.temporal.io/sdk/contrib/aws/s3driver \
    go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 \
    github.com/aws/aws-sdk-go-v2/config \
    github.com/aws/aws-sdk-go-v2/service/s3
  ```

**Google Cloud Storage**

  ```sh
  go get go.temporal.io/sdk/contrib/gcp/gcsdriver \
    go.temporal.io/sdk/contrib/gcp/gcsdriver/gcssdk \
    cloud.google.com/go/storage
  ```

### Procedure

1.  Create a storage client, wrap it in the matching driver client, and pass the result to the driver. Each cloud SDK
    picks up your standard credentials from the environment:

**Amazon S3**

    The AWS SDK reads
    [environment variables, an IAM role, or your AWS config file](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html).

         <!--SNIPSTART go-external-storage-s3-driver-->
         [go-sandbox/bestpractices/externalstorage/drivers.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/drivers.go)
         ```go
         cfg, err := config.LoadDefaultConfig(context.Background(),
         	config.WithRegion("us-east-2"),
         )
         if err != nil {
         	log.Fatalf("load AWS config: %v", err)
         }

         driver, err := s3driver.NewDriver(s3driver.Options{
         	Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)),
         	Bucket: s3driver.StaticBucket("my-temporal-payloads"),
         })
         if err != nil {
         	log.Fatalf("create S3 driver: %v", err)
         }
         ```
         <!--SNIPEND-->

**Google Cloud Storage**

    The Google Cloud SDK reads
    [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials).

    <!--SNIPSTART go-external-storage-gcs-driver-->
    [go-sandbox/bestpractices/externalstorage/drivers.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/drivers.go)
    ```go
    gcsClient, err := storage.NewClient(context.Background())
    if err != nil {
    	log.Fatalf("create GCS client: %v", err)
    }

    driver, err := gcsdriver.NewDriver(gcsdriver.Options{
    	Client: gcssdk.NewClient(gcsClient),
    	Bucket: gcsdriver.StaticBucket("my-temporal-payloads"),
    })
    if err != nil {
    	log.Fatalf("create GCS driver: %v", err)
    }
    ```
    <!--SNIPEND-->

    To route payloads to different buckets at runtime, pass a `BucketFunc` as `Bucket` instead of using `StaticBucket`.
    The function receives the store context and the payload, and returns a bucket name.

2.  Configure the driver on `ExternalStorage` and pass it in your Client options. This step is the same for every
    driver:

    <!--SNIPSTART go-external-storage-setup-->
    [go-sandbox/bestpractices/externalstorage/setup.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/setup.go)
    ```go
    c, err := client.Dial(client.Options{
    	HostPort: "localhost:7233",
    	ExternalStorage: converter.ExternalStorage{
    		Drivers: []converter.StorageDriver{driver},
    	},
    })
    if err != nil {
    	log.Fatalf("connect to Temporal: %v", err)
    }
    defer c.Close()

    w := worker.New(c, "my-task-queue", worker.Options{})
    ```
    <!--SNIPEND-->

    A Worker inherits this configuration from the Client it is created with. When your Workers run in their own process,
    repeat this setup there.

    By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the
    `PayloadSizeThreshold` option, even setting it to 1 to externalize all payloads regardless of size. Refer to
    [Configure payload size threshold](#configure-payload-size-threshold) for more information.

    All Workflows and Activities running on the Worker use the storage driver automatically without changes to your
    business logic. Both drivers upload and download payloads concurrently, and verify a SHA-256 hash of the contents on
    retrieve. Each driver rejects any single payload larger than `MaxPayloadSize`, which defaults to 50 MiB.

    The hash forms part of the object key, along with the Namespace, Workflow ID, and Run ID. When one Workflow Run
    passes the same payload to several Activities, the key is identical each time, so the driver uploads the payload
    once and the later writes reuse that object. A different Run, Workflow, or Namespace produces a different key, so it
    stores its own copy even when the bytes are identical. Storage therefore scales with the number of Runs, not with
    how many times a Run passes a payload around.

    The S3 driver includes diagnostic metadata, such as the AWS region, in error messages to help troubleshoot storage
    failures.

## Implement a custom storage driver

If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver.
Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and
[Lifecycle management](/external-storage#lifecycle) for retention requirements.

The following example shows a custom driver that uses local disk as the backing store. This example is for local
development and testing only. In production, use a durable storage system that is accessible to all Workers:

<!--SNIPSTART go-external-storage-custom-driver-->
[go-sandbox/bestpractices/externalstorage/custom_storage_driver.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/custom_storage_driver.go)
```go
type LocalDiskStorageDriver struct {
	storeDir string
}

func NewLocalDiskStorageDriver(storeDir string) converter.StorageDriver {
	return &LocalDiskStorageDriver{storeDir: storeDir}
}

func (d *LocalDiskStorageDriver) Name() string {
	return "my-local-disk"
}

func (d *LocalDiskStorageDriver) Type() string {
	return "local-disk"
}

func (d *LocalDiskStorageDriver) Store(
	ctx converter.StorageDriverStoreContext,
	payloads []*commonpb.Payload,
) ([]converter.StorageDriverClaim, error) {
	dir := d.storeDir
	switch info := ctx.Target.(type) {
	case converter.StorageDriverWorkflowInfo:
		if info.WorkflowID != "" {
			dir = filepath.Join(d.storeDir, info.Namespace, info.WorkflowID)
		}
	case converter.StorageDriverActivityInfo:
		// StorageDriverActivityInfo is only used for standalone (non-workflow-bound)
		// activities. Activities started by a workflow use StorageDriverWorkflowInfo.
		if info.ActivityID != "" {
			dir = filepath.Join(d.storeDir, info.Namespace, info.ActivityID)
		}
	}
	if err := os.MkdirAll(dir, 0o755); err != nil {
		return nil, fmt.Errorf("create store directory: %w", err)
	}

	claims := make([]converter.StorageDriverClaim, len(payloads))
	for i, payload := range payloads {
		key := uuid.NewString() + ".bin"
		filePath := filepath.Join(dir, key)

		data, err := proto.Marshal(payload)
		if err != nil {
			return nil, fmt.Errorf("marshal payload: %w", err)
		}
		if err := os.WriteFile(filePath, data, 0o644); err != nil {
			return nil, fmt.Errorf("write payload: %w", err)
		}

		claims[i] = converter.StorageDriverClaim{
			ClaimData: map[string]string{"path": filePath},
		}
	}
	return claims, nil
}

func (d *LocalDiskStorageDriver) Retrieve(
	ctx converter.StorageDriverRetrieveContext,
	claims []converter.StorageDriverClaim,
) ([]*commonpb.Payload, error) {
	payloads := make([]*commonpb.Payload, len(claims))
	for i, claim := range claims {
		filePath := claim.ClaimData["path"]
		data, err := os.ReadFile(filePath)
		if err != nil {
			return nil, fmt.Errorf("read payload: %w", err)
		}
		payload := &commonpb.Payload{}
		if err := proto.Unmarshal(data, payload); err != nil {
			return nil, fmt.Errorf("unmarshal payload: %w", err)
		}
		payloads[i] = payload
	}
	return payloads, nil
}
```
<!--SNIPEND-->

The following sections walk through the key parts of the driver implementation.

### 1. Implement the StorageDriver interface

A custom driver implements the `converter.StorageDriver` interface with four methods:

- `Name()` returns a unique string that identifies the driver instance. The SDK stores this name in the claim check
  reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored
  breaks retrieval. For example, two S3 drivers could be named `"s3-primary"` and `"s3-archive"`.
- `Type()` returns a string that identifies the driver implementation. Unlike `Name()`, this must be the same across all
  instances of the same driver type regardless of configuration. Two S3 drivers named `"s3-primary"` and `"s3-archive"`
  would both return `"aws.s3driver"` as their type, while the local disk driver in the custom driver code sample returns
  `"local-disk"`.
- `Store()` receives a slice of payloads and returns one `StorageDriverClaim` per payload. A claim is a set of string
  key-value pairs that the driver uses to locate the payload later.
- `Retrieve()` receives the claims that `Store()` produced and returns the original payloads.

### 2. Store payloads

In `Store()`, marshal each Payload protobuf message to bytes with `proto.Marshal(payload)` and write the bytes to your
storage system. The application data has already been serialized by the
[Payload Converter](/develop/go/data-handling/data-conversion) and
[Payload Codec](/develop/go/data-handling/data-encryption) before it reaches the driver. See the
[data conversion pipeline](/external-storage#data-pipeline) for more details.

Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `ctx.Target` provides
identity information (namespace, Workflow ID) depending on the operation. Use a type switch on
`StorageDriverWorkflowInfo` and `StorageDriverActivityInfo` to access the concrete values. Consider structuring your
storage keys to include this information so that you can identify which Workflow owns each payload.

### 3. Retrieve payloads

In `Retrieve()`, download the bytes using the claim data, then reconstruct the Payload protobuf message with
`proto.Unmarshal(data, payload)`. The Payload Converter handles deserializing the application data after the driver
returns the payload.

### 4. Configure the Client

Pass an `ExternalStorage` struct with your driver in the Client options:

<!--SNIPSTART go-external-storage-custom-driver-client-->
[go-sandbox/bestpractices/externalstorage/setup.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/setup.go)
```go
c, err := client.Dial(client.Options{
	ExternalStorage: converter.ExternalStorage{
		Drivers: []converter.StorageDriver{NewLocalDiskStorageDriver("/tmp/temporal-payload-store")},
	},
})
```
<!--SNIPEND-->

You can also package your driver as a [plugin](/develop/plugins-guide) for easier reuse across services.

## Configure payload size threshold

You can configure the payload size threshold that triggers external storage. By default, payloads of 256 KiB or larger
are offloaded to external storage. You can adjust this with the `PayloadSizeThreshold` option, or set it to 1 to
externalize all payloads regardless of size. A value of 0 is interpreted as the default (256 KiB). Payloads smaller than
the threshold stay inline in Event History. The size compared against the threshold is that of the serialized Payload,
which includes its metadata, not just your data.

<!--SNIPSTART go-external-storage-size-threshold-->
[go-sandbox/bestpractices/externalstorage/setup.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/setup.go)
```go
c, err := client.Dial(client.Options{
	ExternalStorage: converter.ExternalStorage{
		Drivers:              []converter.StorageDriver{driver},
		PayloadSizeThreshold: 1,
	},
})
```
<!--SNIPEND-->

## Use multiple storage drivers

When you register multiple drivers, you must provide a `DriverSelector` that implements the `StorageDriverSelector`
interface. The selector chooses which driver stores each payload. Any driver in the list that is not selected for
storing is still available for retrieval, which is useful when migrating between storage backends. Return `nil` from the
selector to keep a specific payload inline in Event History.

Multiple drivers are useful in scenarios such as:

- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one
  you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old
  driver remains available for retrieving existing claims.
- Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3
  for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver
  based on the runtime environment.

Every registered driver needs a distinct name. The S3 and GCS drivers default to `"aws.s3driver"` and `"gcp.gcsdriver"`,
so you can register one of each without extra configuration. Registering two drivers of the same type requires setting
the `DriverName` option on at least one of them.

The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver` is
only registered so the Worker can retrieve payloads that were previously stored with it:

<!--SNIPSTART go-external-storage-multi-driver-selector-->
[go-sandbox/bestpractices/externalstorage/multiple_drivers.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/multiple_drivers.go)
```go
type PreferredSelector struct {
	preferred converter.StorageDriver
}

func (s *PreferredSelector) SelectDriver(
	ctx converter.StorageDriverSelectContext,
	payload *commonpb.Payload,
) (converter.StorageDriver, error) {
	return s.preferred, nil
}

func MultipleDriversSetup(preferredDriver, legacyDriver converter.StorageDriver) converter.ExternalStorage {
	return converter.ExternalStorage{
		Drivers:        []converter.StorageDriver{preferredDriver, legacyDriver},
		DriverSelector: &PreferredSelector{preferred: preferredDriver},
	}
}
```
<!--SNIPEND-->

## Multi-region durability with Amazon S3

To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with
[Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) and an
[S3 Multi-Region Access Point (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/), then point the
driver at the MRAP ARN instead of a bucket name. See
[Durable External Storage](/external-storage#durable-external-storage) for the full pattern and trade-offs.

In code, the only change is the value you pass to `s3driver.StaticBucket`:

<!--SNIPSTART go-external-storage-s3-mrap-->
[go-sandbox/bestpractices/externalstorage/drivers.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/bestpractices/externalstorage/drivers.go)
```go
driver, err := s3driver.NewDriver(s3driver.Options{
	Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)),
	Bucket: s3driver.StaticBucket("arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap"),
})
```
<!--SNIPEND-->

The AWS SDK for Go v2 automatically uses
[SigV4A](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html) signing when the
bucket value is an MRAP ARN, so no additional client configuration is required.

---

# Integrations - Go SDK

Source: https://docs.temporal.io/develop/go/integrations

> This section covers integrations with the Go SDK

The following integrations are available for the Temporal Go SDK.
You can also use the Temporal Go SDK's [Plugin system](/develop/plugins-guide) to build your own integrations.

- [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#go) — Monitor and evaluate AI application performance with Braintrust observability. _(Go · Agent observability)_
- [Google ADK](/develop/go/integrations/google-adk) — Run Google ADK agents as durable Temporal Workflows. _(Go · Agent framework)_
- [OpenTelemetry v2](/develop/go/integrations/opentelemetry-v2) — Export tracing and metrics from Temporal Go SDK applications with OpenTelemetry. _(Go · Observability)_

---

# Google ADK integration - Go SDK

Source: https://docs.temporal.io/develop/go/integrations/google-adk

> Run Google ADK agents with durable execution using the Temporal Go SDK and the standalone googleadk module for model calls and tools.

Temporal's integration with [Google ADK](https://google.github.io/adk-docs/) (`adk-go`) gives your agents
[Durable Execution](/temporal#durable-execution): the agent's orchestration loop runs inside a Temporal Workflow, each
LLM call becomes a durable Temporal Activity, and any tool that does I/O runs as an Activity too — so every step is
retried, timed out, recorded in Workflow history, and replayable after crashes or restarts.

You keep building agents the native ADK way — `llmagent.New(...)` with a `model.LLM`, `tool.Tool`s / `tool.Toolset`s and
`SubAgents`, wrapped in `runner.New(...)` and driven by `r.Run(...)`. You change two things:

1. Use `googleadk.NewModel("<model-name>")` as your agent's `Model`. It is a `model.LLM` whose calls dispatch to the
   `InvokeModel` Activity; the real model is reconstructed **worker-side**, never in the Workflow.
2. Pass `googleadk.NewContext(workflowCtx)` to `r.Run`, which installs Temporal-deterministic time, UUID, and
   task-fan-out providers so the agent loop replays deterministically.

Tools run **in-workflow by default** (the idiomatic Temporal model: the Workflow Definition is deterministic, and anything touching
the network, clock, or disk goes through an Activity). Opt a tool into an Activity with `googleadk.ActivityAsTool`, or
use `googleadk.NewMCPToolset` for MCP.

Code snippets in this guide are taken from the
[Google ADK plugin samples](https://github.com/temporalio/samples-go/tree/main/googleadk). Refer to the samples for the
complete, runnable code.

## Prerequisites

- This guide assumes you are already familiar with Google ADK. If you aren't, refer to the
  [Google ADK documentation](https://google.github.io/adk-docs/) for more details.
- If you are new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the
  [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following the
  [Set up your local development environment](/develop/go/set-up-your-local-go) guide. Leave the Temporal development
  server running if you want to test your code locally.
- Provide model credentials **worker-side** — for Gemini, set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) in the worker's
  environment. Credentials are captured in the worker's `ModelFactory` and never cross the Activity boundary into the
  Workflow.

## Install the plugin

Install the standalone [`go.temporal.io/googleadk`](https://pkg.go.dev/go.temporal.io/googleadk) module:

```bash
go get go.temporal.io/googleadk@latest
```

```go
import "go.temporal.io/googleadk"
```

The module is versioned separately from the Temporal Go SDK. Its source and releases are maintained in
[`temporalio/ai-integrations`](https://github.com/temporalio/ai-integrations/tree/main/go/googleadk).

## Run an agent with Durable Execution

An agent has two halves: the **worker** registers the real model behind the `InvokeModel` Activity, and the **workflow**
builds a vanilla ADK agent and drives it.

### Configure the Worker

Configure the Worker with the integration's plugin, which registers the model and MCP Activities at Worker start (and
closes any cached MCP toolsets at Worker stop). The real Gemini model lives in the plugin's config, behind the Activity
boundary; the API key is read worker-side. Disable the model SDK's own retries so Temporal's `RetryPolicy` is the
single source of truth.

<!--SNIPSTART googleadk-hello-worker-->
[googleadk/worker/main.go](https://github.com/temporalio/samples-go/blob/main/googleadk/worker/main.go)
```go
// The plugin registers the integration's Activities on the worker (and closes
// any cached MCP toolsets at worker stop). The real Gemini model lives in the
// factory, behind the Activity boundary; the API key is read worker-side from
// the env and never crosses into the workflow. Disable the model SDK's own
// retries so Temporal's RetryPolicy is the single source of truth.
adkPlugin, err := googleadk.NewPlugin(googleadk.Config{
	Models: map[string]googleadk.ModelFactory{
		adk.ModelName: func(ctx context.Context, name string) (model.LLM, error) {
			// nil config reads GEMINI_API_KEY / GOOGLE_API_KEY from the env.
			return gemini.NewModel(ctx, name, nil)
		},
	},
})
if err != nil {
	log.Fatalln("Unable to build googleadk plugin", err)
}

w := worker.New(c, adk.TaskQueue, worker.Options{
	Plugins: []worker.Plugin{adkPlugin},
})

w.RegisterWorkflow(adk.AgentWorkflow)
// Register GetWeather under the tool name the ActivityAsTool dispatches, so the
// agent's get_weather call resolves to this activity.
w.RegisterActivityWithOptions(adk.GetWeather, activity.RegisterOptions{Name: adk.WeatherToolName})

if err := w.Run(worker.InterruptCh()); err != nil {
	log.Fatalln("Unable to start worker", err)
}
```
<!--SNIPEND-->

`Config.Models` is optional for providers ADK's registry already knows (for example, `gemini-*`): when a model name is
absent, `InvokeModel` falls back to `model.NewLLM`. Supply a factory to inject credentials, disable the model SDK's own
retries, or override the default.

### Define the Workflow

Build the agent the ordinary ADK way, using `googleadk.NewModel` for the model and passing `googleadk.NewContext(ctx)`
to `r.Run`. The `get_weather` tool is an ordinary Temporal Activity exposed to the agent with `googleadk.ActivityAsTool`.

<!--SNIPSTART googleadk-hello-workflow-->
[googleadk/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/workflow.go)
```go
func AgentWorkflow(ctx workflow.Context, question string) (string, error) {
	weatherTool, err := googleadk.ActivityAsTool(GetWeather, googleadk.ActivityToolOptions{
		Name:        WeatherToolName,
		Description: "Get the current weather for a city.",
	})
	if err != nil {
		return "", err
	}

	// Build the agent the ordinary ADK way. NewModel is a model.LLM that carries
	// only the model name in-workflow; the real Gemini client lives worker-side.
	root, err := llmagent.New(llmagent.Config{
		Name:        "assistant",
		Description: "a helpful weather assistant",
		Model:       googleadk.NewModel(ModelName),
		Instruction: "Answer the user's question. Use the get_weather tool when asked about the weather.",
		Tools:       []tool.Tool{weatherTool},
	})
	if err != nil {
		return "", err
	}

	r, err := runner.New(runner.Config{
		AppName:           "weather",
		Agent:             root,
		SessionService:    session.InMemoryService(),
		AutoCreateSession: true,
	})
	if err != nil {
		return "", err
	}

	// NewContext bridges the workflow.Context into the context ADK reads its
	// determinism/executor seams from. Pass it straight to Run.
	adkCtx := googleadk.NewContext(ctx)
	msg := genai.NewContentFromText(question, genai.RoleUser)

	var answer string
	for ev, err := range r.Run(adkCtx, "user-1", "session-1", msg, agent.RunConfig{}) {
		if err != nil {
			return "", err
		}
		if ev != nil && ev.Content != nil {
			for _, p := range ev.Content.Parts {
				if p != nil && p.Text != "" {
					answer = p.Text
				}
			}
		}
	}
	return answer, nil
}

```
<!--SNIPEND-->

The tool itself is an ordinary Temporal Activity — register it on the worker as usual, and expose it to the agent with
`ActivityAsTool` (its parameter schema is inferred from the argument type):

<!--SNIPSTART googleadk-hello-tool-->
[googleadk/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/workflow.go)
```go
func GetWeather(ctx context.Context, in GetWeatherInput) (GetWeatherOutput, error) {
	return GetWeatherOutput{City: in.City, Conditions: "sunny, 72°F"}, nil
}

```
<!--SNIPEND-->

### Start the Workflow

Start the Workflow like any other and read its result:

<!--SNIPSTART googleadk-hello-starter-->
[googleadk/starter/main.go](https://github.com/temporalio/samples-go/blob/main/googleadk/starter/main.go)
```go
question := "What's the weather in San Francisco?"
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, adk.AgentWorkflow, question)
if err != nil {
	log.Fatalln("Unable to execute workflow", err)
}

log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID())

// Synchronously wait for the workflow completion.
var answer string
if err := we.Get(context.Background(), &answer); err != nil {
	log.Fatalln("Unable to get workflow result", err)
}
log.Println("Agent answer:", answer)
```
<!--SNIPEND-->

## Tools

- **Function tools run in-workflow by default.** Ordinary `functiontool.New(...)` tools run on Temporal's deterministic
  dispatcher inside the Workflow — no Activity overhead — and their session-state mutations propagate normally. Their
  code must be deterministic and replay-safe: no direct network, clock, randomness, or goroutines.
- **Opt a tool into an Activity when it does I/O.** `googleadk.ActivityAsTool(myActivity, ...)` exposes an existing
  `func(context.Context, TArgs) (TResults, error)` Temporal Activity to the agent as a tool (shown in the Hello World
  Workflow above); its call dispatches the Activity, so it is retried, timed out, and visible in the UI.
- **MCP, statelessly.** `googleadk.NewMCPToolset(...)` is a workflow-side proxy that lists remote tools via the
  `ListMcpTools` Activity and executes calls via `CallMcpTool`. The live, stateful `mcptoolset.New(...)` runs worker-side
  (registered in `Config.MCPToolsets`), never in the Workflow.

## Multi-agent systems

Build a coordinator agent with specialist `SubAgents`; ADK wires the parent/child relationship and exposes the built-in
`transfer_to_agent` tool automatically. The entire tree — including the transfer hop — runs in the Workflow; only the
model calls and any Activity-backed tools leave it.

<!--SNIPSTART googleadk-multiagent-workflow-->
[googleadk/multiagent/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/multiagent/workflow.go)
```go
func MultiAgentWorkflow(ctx workflow.Context, question string) (string, error) {
	weatherTool, err := googleadk.ActivityAsTool(GetWeather, googleadk.ActivityToolOptions{
		Name:        WeatherToolName,
		Description: "Get the current weather for a city.",
	})
	if err != nil {
		return "", err
	}

	// The weather specialist owns the get_weather tool.
	weather, err := llmagent.New(llmagent.Config{
		Name:        "weather",
		Description: "answers questions about the current weather in a city",
		Model:       googleadk.NewModel(WeatherModelName),
		Instruction: "You are a weather specialist. Use the get_weather tool to answer weather questions.",
		Tools:       []tool.Tool{weatherTool},
	})
	if err != nil {
		return "", err
	}

	// The jokes specialist just tells jokes.
	jokes, err := llmagent.New(llmagent.Config{
		Name:        "jokes",
		Description: "tells a light-hearted joke",
		Model:       googleadk.NewModel(JokesModelName),
		Instruction: "You are a comedian. Respond with a short, friendly joke.",
	})
	if err != nil {
		return "", err
	}

	// The coordinator delegates to whichever specialist fits the question. ADK
	// wires the parent/child relationship from SubAgents and exposes the built-in
	// transfer_to_agent tool automatically.
	coordinator, err := llmagent.New(llmagent.Config{
		Name:        "coordinator",
		Description: "routes the user's request to the right specialist",
		Model:       googleadk.NewModel(CoordinatorModelName),
		Instruction: "You are a router. Delegate weather questions to the weather agent " +
			"and requests for a joke to the jokes agent. Do not answer directly.",
		SubAgents: []agent.Agent{weather, jokes},
	})
	if err != nil {
		return "", err
	}

	r, err := runner.New(runner.Config{
		AppName:           "multiagent",
		Agent:             coordinator,
		SessionService:    session.InMemoryService(),
		AutoCreateSession: true,
	})
	if err != nil {
		return "", err
	}

	adkCtx := googleadk.NewContext(ctx)
	msg := genai.NewContentFromText(question, genai.RoleUser)

	var answer string
	for ev, err := range r.Run(adkCtx, "user-1", "session-1", msg, agent.RunConfig{}) {
		if err != nil {
			return "", err
		}
		if ev == nil || ev.Content == nil {
			continue
		}
		// Keep the last non-empty text produced by any agent in the tree; after a
		// transfer_to_agent hop this is the specialist's answer.
		for _, p := range ev.Content.Parts {
			if p != nil && p.Text != "" {
				answer = p.Text
			}
		}
	}
	return answer, nil
}

```
<!--SNIPEND-->

## Human-in-the-loop tool confirmation

A sensitive tool calls ADK's `ctx.RequestConfirmation(hint, payload)`, which ends the turn with an
`adk_request_confirmation` function call. The Workflow detects pending confirmations with
`googleadk.PendingConfirmations`, durably waits for the human's decision (delivered as a Temporal signal), and resumes
the agent with `googleadk.ConfirmationResponse`. Because the wait is durable, the Workflow can sit idle for days and
survive worker restarts — when the approval signal arrives, the agent resumes exactly where it paused.

<!--SNIPSTART googleadk-hitl-workflow-->
[googleadk/humanintheloop/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/humanintheloop/workflow.go)
```go
func ApprovalWorkflow(ctx workflow.Context, request string) (Result, error) {
	delTool, err := functiontool.New[DeleteArgs, map[string]any](
		functiontool.Config{
			Name:        DeleteToolName,
			Description: "Delete a named resource. Requires human confirmation before it runs.",
		},
		deleteResource,
	)
	if err != nil {
		return Result{}, err
	}

	root, err := llmagent.New(llmagent.Config{
		Name:        "assistant",
		Description: "an assistant that can delete resources with human approval",
		Model:       googleadk.NewModel(ModelName),
		Instruction: "Use the delete_resource tool when the user asks to delete something.",
		Tools:       []tool.Tool{delTool},
	})
	if err != nil {
		return Result{}, err
	}

	r, err := runner.New(runner.Config{
		AppName:           "hitl",
		Agent:             root,
		SessionService:    session.InMemoryService(),
		AutoCreateSession: true,
	})
	if err != nil {
		return Result{}, err
	}

	adkCtx := googleadk.NewContext(ctx)
	msg := genai.NewContentFromText(request, genai.RoleUser)

	var res Result
	// Drive the run in passes: each Run call is one pass over the same session. A
	// pass either completes (no pending confirmation) or pauses awaiting a human.
	for {
		var events []*session.Event
		for ev, err := range r.Run(adkCtx, "user-1", "session-1", msg, agent.RunConfig{}) {
			if err != nil {
				return Result{}, err
			}
			if ev == nil {
				continue
			}
			events = append(events, ev)
			if ev.Content != nil {
				for _, p := range ev.Content.Parts {
					if p != nil && p.Text != "" {
						res.Answer = p.Text
					}
				}
			}
		}

		pending := googleadk.PendingConfirmations(events)
		if len(pending) == 0 {
			// The agent finished without (further) confirmations needed.
			return res, nil
		}

		// The agent paused. Durably wait for the human's decision to arrive as a
		// Temporal signal. This is the whole point: the workflow can sit here for
		// as long as it takes — across worker restarts — without losing state.
		//
		// This handles one pending confirmation per pass — the recommended
		// pattern (see the googleadk.ConfirmationResponse docs): resuming
		// several decisions at once can re-dispatch the approved tool calls in
		// an order that is not replay-stable. Any other pending confirmations
		// simply surface again on the next pass.
		var decision googleadk.ConfirmationDecision
		workflow.GetSignalChannel(ctx, googleadk.ConfirmationSignalName).Receive(ctx, &decision)
		res.Approved = decision.Confirmed

		// Match the decision to the pending confirmation and resume the run with
		// it as the next message. ADK re-dispatches (or blocks) the original tool
		// call based on Confirmed.
		if decision.FunctionCallID == "" {
			decision.FunctionCallID = pending[0].FunctionCallID
		}
		msg = googleadk.ConfirmationResponse(decision)
	}
}

```
<!--SNIPEND-->

## Continue-as-new for long conversations

A conversation's history lives in the ADK session. To keep a Workflow's history bounded, snapshot the session with
`googleadk.ExportSession` and [continue-as-new](/develop/go/workflows/continue-as-new); rebuild it on the next run with
`googleadk.ImportSession`. `SessionSnapshot` is JSON-serializable (session-scoped state plus the full event history), so
every value in session state and every tool result must be JSON-encodable.

<!--SNIPSTART googleadk-chat-workflow-->
[googleadk/chat/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/chat/workflow.go)
```go
func ChatWorkflow(ctx workflow.Context, in ChatInput) error {
	// A fresh in-memory session service, kept in a local so we can Export it later.
	svc := session.InMemoryService()

	adkCtx := googleadk.NewContext(ctx)

	// Resume a prior conversation if this run was continued-as-new.
	if in.Snapshot != nil {
		if _, err := googleadk.ImportSession(adkCtx, svc, in.Snapshot); err != nil {
			return err
		}
	}

	root, err := llmagent.New(llmagent.Config{
		Name:        "assistant",
		Description: "a friendly conversational assistant",
		Model:       googleadk.NewModel(ModelName),
		Instruction: "You are a helpful assistant. Answer the user, using the conversation history for context.",
	})
	if err != nil {
		return err
	}

	r, err := runner.New(runner.Config{
		AppName:           AppName,
		Agent:             root,
		SessionService:    svc,
		AutoCreateSession: true,
	})
	if err != nil {
		return err
	}

	turns := 0
	// One agent turn runs at a time: serialize concurrent Updates so they can't
	// interleave on the shared ADK session.
	busy := false

	err = workflow.SetUpdateHandlerWithOptions(
		ctx,
		SendMessageUpdateName,
		func(ctx workflow.Context, text string) (string, error) {
			if err := workflow.Await(ctx, func() bool { return !busy }); err != nil {
				return "", err
			}
			busy = true
			defer func() { busy = false }()

			// Build the ADK context from this Update handler's own workflow.Context so
			// the model Activity is scheduled on the handler's coroutine.
			turnCtx := googleadk.NewContext(ctx)
			var answer string
			msg := genai.NewContentFromText(text, genai.RoleUser)
			for ev, err := range r.Run(turnCtx, UserID, SessionID, msg, agent.RunConfig{}) {
				if err != nil {
					return "", err
				}
				if ev == nil || ev.Content == nil {
					continue
				}
				for _, p := range ev.Content.Parts {
					if p != nil && p.Text != "" {
						answer = p.Text
					}
				}
			}
			turns++
			return answer, nil
		},
		workflow.UpdateHandlerOptions{
			Validator: func(ctx workflow.Context, text string) error {
				if text == "" {
					return fmt.Errorf("message must not be empty")
				}
				return nil
			},
		},
	)
	if err != nil {
		return err
	}

	// Serve messages until Temporal suggests continue-as-new (history getting large)
	// or the demo turn cap is reached.
	if err := workflow.Await(ctx, func() bool {
		return workflow.GetInfo(ctx).GetContinueAsNewSuggested() || (in.MaxTurns > 0 && turns >= in.MaxTurns)
	}); err != nil {
		return err
	}

	// Let any in-flight Update finish so its turn is captured in the snapshot.
	if err := workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }); err != nil {
		return err
	}

	snap, err := googleadk.ExportSession(adkCtx, svc, AppName, UserID, SessionID)
	if err != nil {
		return err
	}
	return workflow.NewContinueAsNewError(ctx, ChatWorkflow, ChatInput{
		Snapshot: snap,
		MaxTurns: in.MaxTurns,
	})
}

```
<!--SNIPEND-->

## Streaming

`googleadk.NewModel(name, googleadk.WithStreaming(topic, 0))` drives the model in streaming mode: the `InvokeModel`
Activity calls the model with `stream=true`, heartbeats, and publishes each chunk to a per-run
[`workflowstreams`](https://pkg.go.dev/go.temporal.io/sdk/contrib/workflowstreams) topic for external (UI) consumers,
then returns the aggregated final response into the Workflow so replay stays deterministic.

Call `googleadk.StreamServer(ctx)` once near the top of the Workflow that drives `r.Run`, and set
`agent.RunConfig{StreamingMode: agent.StreamingModeSSE}`:

```go
func StreamingAgentWorkflow(ctx workflow.Context, q string) (string, error) {
	if err := googleadk.StreamServer(ctx); err != nil { // required when streaming
		return "", err
	}
	topic := "run-" + workflow.GetInfo(ctx).WorkflowExecution.ID
	root, _ := llmagent.New(llmagent.Config{
		Model: googleadk.NewModel("gemini-2.0-flash", googleadk.WithStreaming(topic, 0)),
		// ...
	})
	// ... build the runner, set agent.RunConfig{StreamingMode: agent.StreamingModeSSE}, and drive r.Run
}
```

External consumers read chunks with `workflowstreams.NewClient(c, workflowID, ...).Subscribe(...)`. The bidirectional
`RunLive` path (hard-coded goroutines/channels) is **not** supported.

## Error handling

Model, tool, and MCP failures surface as Temporal `ApplicationError`s tagged `googleadk.ModelError`, `.ToolError`, and
`.McpError`. Classify them with `googleadk.IsNonRetryable(err)` rather than string-matching. For model calls, the
upstream HTTP status drives retryability (`408`/`409`/`429`/`5xx` are retryable; other `4xx` are not).

> **💡 Tip:**
>
> Disable your model client's own retries in the `ModelFactory`. `InvokeModel` already runs under Temporal's
> `RetryPolicy`; leaving the model SDK's retries on retries a transient failure twice over. Let Temporal own retries.
>

## Composing with other plugins

The integration's plugin only registers its Activities and closes cached MCP toolsets — it uses the default JSON data
converter and ships no client/worker interceptor — so it composes with interceptor- or converter-based plugins (for example
[`sdk-go/contrib/opentelemetry`](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry)) without conflict. ADK
emits its own OpenTelemetry spans; register your tracing interceptor on the worker as usual.

## Testing without a live LLM

The plugin ships test helpers so you can unit-test agent Workflows with no network: `FakeModel` (with `TextResponse` /
`FunctionCallResponse` builders) and `FakeMCPServer`. Register them through the same `googleadk.Config` your production
worker uses — via `googleadk.NewActivities` and `Register` rather than the plugin, since the test environments construct
no real Worker and therefore run no plugins.

## Supported and not-yet-supported

- **Supported:** single- and multi-agent (`SubAgents`) trees, in-workflow function tools, `ActivityAsTool`, stateless
  MCP, Gemini built-in tools (executed server-side inside `InvokeModel`), human-in-the-loop tool confirmation,
  continue-as-new session-state carry, the in-memory session service, and SSE streaming.
- **Not yet:** `RunLive` (bidirectional streaming), sub-agent-as-child-workflow, live memory/artifact tools that require
  in-workflow network I/O, and database/Vertex session services. These raise or are documented rather than silently
  degrading.

## Samples

The [Google ADK plugin samples](https://github.com/temporalio/samples-go/tree/main/googleadk) demonstrate a basic agent
with a tool, a [multi-agent](https://github.com/temporalio/samples-go/tree/main/googleadk/multiagent) system, durable
[human-in-the-loop](https://github.com/temporalio/samples-go/tree/main/googleadk/humanintheloop) tool approval, and a
[continue-as-new chat](https://github.com/temporalio/samples-go/tree/main/googleadk/chat).

---

# OpenTelemetry v2 integration - Go SDK

Source: https://docs.temporal.io/develop/go/integrations/opentelemetry-v2

> Configure trace propagation, automatic tracing, custom tracing, and metrics with the Go SDK OpenTelemetry v2 plugin.

Temporal's OpenTelemetry integration lets you understand the internal state
of Temporal applications across Clients, Workflows, Activities, and Nexus
Operations by instrumenting them with
[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/).

OpenTelemetry instruments your applications to give you insight into your
deployed environments. Temporal Workflows complicate that picture because a
trace can span across different Workers over long stretches of time, which
can scatter a trace into disconnected fragments. The OpenTelemetry plugin
solves this by propagating OpenTelemetry context across those Temporal
boundaries, keeping a trace intact end to end. It can also generate spans and
emit metrics for Temporal SDK operations automatically.

> **Pre-release**

All code snippets in this guide are taken from the
[OpenTelemetry v2 sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2).
Refer to the sample for complete code.

## Prerequisites

- This guide assumes you are already familiar with OpenTelemetry. If you aren't, refer to the
  [OpenTelemetry documentation](https://opentelemetry.io/docs/) for more details.
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the
  [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Ensure you have set up your local development environment by following the
  [Set up your local development environment](/develop/go/set-up-your-local-go) guide. When you're done, leave the
  Temporal Development Server running if you want to test your code locally.

## Install

Add the OpenTelemetry v2 integration to your Go module:

```bash
go get go.temporal.io/sdk/contrib/opentelemetry-v2@latest
```

Also add the OpenTelemetry SDK packages and the exporter or metric reader your
backend requires.

## Set up the tracer provider

A [Tracer Provider](https://opentelemetry.io/docs/concepts/signals/traces/#tracer-provider)
is a factory for Tracers, and it configures the Tracers it creates, including
how they generate span IDs. A standard Tracer Provider assigns a new random
span ID each time a span is created, but Temporal Workflows replay,
re-executing the same code and recreating what should be the same span with a
different random ID each time. Temporal's replay-safe Tracer Provider avoids
this by generating span IDs from a deterministic source tied to the
Workflow, so the same span gets the same ID on every replay. Create it and
install it as the OpenTelemetry global before you create the plugin or call
`Tracer`.

<!--SNIPSTART samples-go-opentelemetry-v2-tracer-provider {"selectedLines": ["14-22"]}-->
[opentelemetry-v2/setup.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/setup.go)
```go
// ...
	provider := temporalotel.NewReplaySafeTracerProvider(
		// WithBatcher performs exporter I/O outside the Workflow goroutine.
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetTracerProvider(provider)
```
<!--SNIPEND-->

Your application owns the Tracer Provider for the life of the process. Shut it
down before exit so remaining spans can flush through the
[trace exporter](https://opentelemetry.io/docs/concepts/signals/traces/#trace-exporters).

## Set up the meter provider

A [Meter Provider](https://opentelemetry.io/docs/concepts/signals/metrics/#meter-provider)
is a factory for Meters. OpenTelemetry's default global Meter Provider is a
no-op, so if you enable `MetricsHandlerOptions`, you need to supply a
configured one yourself, either by installing it with `otel.SetMeterProvider`
before you create the plugin, or by passing a Meter directly through
`MetricsHandlerOptions.Meter`.

## Add the plugin

Pass the plugin to your Temporal Client when you create it. Workers made from
that Client get the plugin automatically.

<!--SNIPSTART samples-go-opentelemetry-v2-plugin-client-->
[opentelemetry-v2/workflow-activity-propagation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/worker/main.go)
```go
plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

c, err := client.Dial(client.Options{Plugins: []client.Plugin{plugin}})
if err != nil {
	return fmt.Errorf("unable to create client: %w", err)
}
defer c.Close()
```
<!--SNIPEND-->

By default the plugin only performs
[context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
so [Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context)
can cross Temporal boundaries.

## Add custom spans

### In Workflows

A [Tracer](https://opentelemetry.io/docs/concepts/signals/traces/#tracer)
creates spans that capture information about a given operation. A standard
Tracer stamps a span with the current time and emits it as soon as it
completes, but Temporal Workflows replay, re-executing the same code and
stamping what should be the same span with a new time and emitting a
duplicate span. Temporal's replay-safe `Tracer` avoids this by stamping a
span with `workflow.Now`, Temporal's replay-safe clock, and skipping a span
that already completed on a previous successful execution. Use it instead
of `otel.Tracer` in Workflows.

<!--SNIPSTART samples-go-opentelemetry-v2-application-spans {"selectedLines": ["3-18"]}-->
[opentelemetry-v2/workflow-activity-propagation/opentelemetry.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/opentelemetry.go)
```go
// ...
func Workflow(ctx workflow.Context, name string) (string, error) {
	tracer := temporalotel.Tracer(instrumentationName)
	ctx, span := tracer.Start(ctx, "workflow-operation")
	defer span.End()

	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	})

	var result string
	if err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result); err != nil {
		return "", err
	}

	return result, nil
}
```
<!--SNIPEND-->

As in
[OpenTelemetry Go](https://opentelemetry.io/docs/languages/go/instrumentation/),
`Start` returns a context that contains the active span. Pass that
`workflow.Context` to downstream Temporal calls so later spans nest under it as
children.

### Outside Workflows

In Clients, Activities, and other non-Workflow code, use an ordinary OpenTelemetry
[Tracer](https://opentelemetry.io/docs/concepts/signals/traces/#tracer):

<!--SNIPSTART samples-go-opentelemetry-v2-application-spans {"selectedLines": ["20-25"]}-->
[opentelemetry-v2/workflow-activity-propagation/opentelemetry.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/opentelemetry.go)
```go
// ...
func Activity(ctx context.Context, name string) (string, error) {
	_, span := otel.Tracer(instrumentationName).Start(ctx, "activity-operation")
	defer span.End()

	return fmt.Sprintf("Hello, %s!", name), nil
}
```
<!--SNIPEND-->

## Enable automatic instrumentation

<!--SNIPSTART samples-go-opentelemetry-v2-metrics-plugin-->
[opentelemetry-v2/automatic-instrumentation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/automatic-instrumentation/worker/main.go)
```go
plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{
	TracerOptions: tracing.TracerOptions{
		AddTemporalSpans: true,
	},
	MetricsHandlerOptions: &temporalotel.MetricsHandlerOptions{
		UseMonotonicCounters: true,
	},
})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}
```
<!--SNIPEND-->

### `AddTemporalSpans`

Set `AddTemporalSpans` to `true` to create spans for Temporal SDK operations
across Clients, Workflows, Activities, and Nexus Operations.

### `MetricsHandlerOptions`

Set `MetricsHandlerOptions` to a non-`nil` value to emit
[Temporal SDK metrics](/references/sdk-metrics) through OpenTelemetry.

## Configure context propagation

[Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
is how OpenTelemetry moves context across process boundaries, injecting it
on the way out and extracting it on the way in. The plugin performs this
propagation for you across Temporal boundaries, carrying
[Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context),
which keeps spans linked into one trace, and
[baggage](https://opentelemetry.io/docs/concepts/signals/baggage/), optional
key-value data that travels with the context.

Do not put credentials, tokens, or personal data in baggage since the
plugin serializes it into Temporal headers that can be persisted in
Workflow Event History.

### `TextMapPropagator`

The plugin injects and extracts both with a
[TextMapPropagator](https://opentelemetry.io/docs/specs/otel/context/api-propagators/#textmap-propagator).
By default that propagator supports
[W3C Trace Context](https://www.w3.org/TR/trace-context/) and
[W3C Baggage](https://www.w3.org/TR/baggage/). Set
`PluginOptions.TextMapPropagator` to override it.

### `HeaderKey`

Propagated values are stored in the Temporal header under `_tracer-data`. Set
`TracerOptions.HeaderKey` to use a different key.

### `DisableBaggage`

Set `DisableBaggage` to `true` to stop propagating baggage.

### `AllowInvalidParentSpans`

Set `AllowInvalidParentSpans` to `true` to ignore errors when extracting
[Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context)
from Temporal headers. Use this when migrating between tracing libraries
while Workflows or Activities are still in progress.

## Resources

- [OpenTelemetry v2 sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2)
- [OpenTelemetry v2 Go package](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry-v2)
- [Traces](https://opentelemetry.io/docs/concepts/signals/traces/)
- [Metrics](https://opentelemetry.io/docs/concepts/signals/metrics/)
- [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/)
- [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
- [Go SDK observability guide](/develop/go/platform/observability)

---

# Nexus - Go SDK

Source: https://docs.temporal.io/develop/go/nexus

> This section explains how to use Temporal Nexus with the Go SDK

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Temporal Nexus

- [Quickstart](/develop/go/nexus/quickstart)
- [Feature guide](/develop/go/nexus/feature-guide)
- [Standalone Operations](/develop/go/nexus/standalone-operations)

---

# Nexus feature guide - Go SDK

Source: https://docs.temporal.io/develop/go/nexus/feature-guide

> Use Temporal Nexus within the Go SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **⚠️ Currently in: Pre-release:**
> This feature guide includes the new Nexus developer experience: pre-release APIs for the [Temporal Operation Handler](/nexus/temporal-operation-handler) and [Nexus Standalone Activity](/nexus/standalone-activity). These APIs are experimental and subject to change.

Use [Temporal Nexus](/evaluate/features/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **💡 Tip:**
>
> New to Nexus? Start with the [Nexus Go quickstart](/develop/go/nexus/quickstart).
>

This page shows how to do the following:

- [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server)
- [Create caller and handler Namespaces](#create-caller-handler-namespaces)
- [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint)
- [Define the Nexus Service contract](#define-nexus-service-contract)
- [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers)
- [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service)
- [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server)
- [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from this [Go Nexus sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/go-sandbox/nexus).
> Run the `go run` commands on this page from that `nexus` directory.
>

## Run the Temporal Development Server with Nexus enabled 

Prerequisites:

- [Install the latest Temporal CLI](/develop/run-a-development-server) (v1.9.1 or higher recommended)
- [Install the latest Temporal Go SDK](/develop/go/set-up-your-local-go) (v1.49.0 or higher recommended)

The first step in working with Temporal Nexus involves starting a Temporal server with Nexus enabled.

```
temporal server start-dev \
  --dynamic-config-value history.enableUpdateCallbacks=true \
  --dynamic-config-value history.enableCHASMSignalBacklinks=true \
  --dynamic-config-value history.enableSignalWithStartFromWorkflow=true
```

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases.

The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`.

## Create caller and handler Namespaces 

Before setting up Nexus endpoints, create separate Namespaces for the caller and handler.

```
temporal operator namespace create --namespace my-target-namespace
temporal operator namespace create --namespace my-caller-namespace
```

`my-target-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `my-caller-namespace` to call that Operation handler.
We use different namespaces to demonstrate cross-Namespace Nexus calls.

## Create a Nexus Endpoint to route requests from caller to handler 

After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests.

```
temporal operator nexus endpoint create \
  --name my-nexus-endpoint-name \
  --target-namespace my-target-namespace \
  --target-task-queue my-handler-task-queue
```

You can also use the Web UI to create the Namespaces and Nexus endpoint.

## Define the Nexus Service contract 

Defining a clear contract for the Nexus Service is crucial for smooth communication.

In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint.

You can hand-write that package, but the preferred way is to generate it with [NexGen](/nexus/nexgen).
You write the contract once as a JSON or YAML definition file and run `nexgen` against it, and it emits the typed models,
runtime validators, and the Service definition itself.

This is what makes a Nexus Service polyglot. Both sides generate from the same definition file: the handler implements
the Service, the caller invokes its Operations, and neither side hand-writes request or response types. A Python handler
and a Go caller share no code, but they both run off that same service contract, so they interoperate with no
coordination between the teams beyond the contract itself.

The generated validators check every payload against the contract when a value is parsed off the wire and again when
it is serialized onto it, so bad data is rejected at the boundary rather than reaching your Workflow or Activity. A value validates
identically in every language, which is what lets a caller and a handler written in different languages trust the same
contract. See the [`chat.nexusrpc.yaml`](https://github.com/temporalio/nexgen/blob/main/samples/schemas/chat.nexusrpc.yaml)
sample contract and the [Definition files](https://github.com/temporalio/nexgen#definition-files) section of the
`nexgen` README for the file format.

## Develop a Nexus Service and Operation handlers 

Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract.
Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous.
They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code.
Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database.
Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors, blocking all Operations from the caller to that Endpoint.

Every Operation is written with a [Temporal Operation Handler](/nexus/temporal-operation-handler). `temporalnexus.MustNewTemporalOperation(...)` takes a `Start` callback that receives three things: a context, a `NexusClient`, and the
Operation input. What you do with the Client determines what backs the Operation:

- **Synchronous.** Return `temporalnexus.NewSyncResult(...)` and the Operation completes during the handler call. The
  caller has its result as soon as the call returns.
- **Asynchronous.** Call `temporalnexus.StartWorkflow`, `temporalnexus.StartActivity`, or
  `temporalnexus.StartUpdateWorkflow` with the Client. The handler returns as soon as that Execution has started, and
  the Operation stays open until the Execution finishes, which may be days later. Its result is delivered to the caller
  through the Nexus completion callback. This is what lets an Operation outlive the
  [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

A handler can perform any number of synchronous side effects, such as sending a Signal, but at most one asynchronous
backing per invocation.

### Develop a Synchronous Nexus Operation handler

Return a synchronous result when the Operation can answer immediately. The handler computes the answer and returns it,
and the Operation completes during the call.

Handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking), and the whole
call has to finish inside the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

<!--SNIPSTART go-nexus-sync-operation-->
[go-sandbox/nexus/handler/app.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/handler/app.go)
```go
var EchoOperation = temporalnexus.MustNewTemporalOperation(
	temporalnexus.TemporalOperationOptions[service.EchoInput, service.EchoOutput]{
		Name: service.EchoOperationName,
		Start: func(
			ctx context.Context,
			nc temporalnexus.NexusClient,
			input service.EchoInput,
			options temporalnexus.StartTemporalOperationOptions,
		) (temporalnexus.TemporalOperationResult[service.EchoOutput], error) {
			return temporalnexus.NewSyncResult(service.EchoOutput(input)), nil
		},
	})
```
<!--SNIPEND-->

### Use the Temporal Client for Signals, Queries, and Updates

A common pattern is to reach a Workflow that is already running. Query it or Signal it from a synchronous Operation, or
use Signal-With-Start to make sure the Workflow exists before the Signal arrives. Those calls complete during the
handler call, so they have to finish inside the
[Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).
The context provided to the handler is automatically set with this deadline, so passing it directly to Temporal Client
calls propagates the timeout.

Updates are the exception. Do not wait for one inside the handler. Start it with `temporalnexus.StartUpdateWorkflow` and
it backs the Operation. The handler returns immediately, and the Operation completes when the Update does, however
long it takes.

The [nexus-messaging](https://github.com/temporalio/samples-go/tree/main/nexus-messaging)
sample shows a Nexus Service that Queries and Signals a running Workflow from synchronous Operations, and backs an
Operation with a Workflow Update.

The Client your handler receives is not an ordinary Temporal Client. It propagates
[bidirectional links](/nexus/execution-debugging#bi-directional-linking) and request IDs on every call, so the
caller-side and handler-side Executions are connected in the UI without wiring anything. Reach the Workflow Client
through `nc.GetWorkflowClient()` rather than constructing your own.

In this example the Workflow ID is derived from an identifier carried in the Operation input, so the caller only needs
the identifier it cares about:

<!--SNIPSTART go-nexus-signal-operation-->
[go-sandbox/nexus/handler/app.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/handler/app.go)
```go
var ApproveOperation = temporalnexus.MustNewTemporalOperation(
	temporalnexus.TemporalOperationOptions[service.ApproveInput, service.ApproveOutput]{
		Name: service.ApproveOperationName,
		Start: func(
			ctx context.Context,
			nc temporalnexus.NexusClient,
			input service.ApproveInput,
			options temporalnexus.StartTemporalOperationOptions,
		) (temporalnexus.TemporalOperationResult[service.ApproveOutput], error) {
			err := nc.GetWorkflowClient().SignalWorkflow(
				ctx, GetWorkflowID(input.UserID), "", service.ApproveSignalName, input)
			if err != nil {
				return temporalnexus.TemporalOperationResult[service.ApproveOutput]{}, err
			}
			return temporalnexus.NewSyncResult(service.ApproveOutput{}), nil
		},
	})
```
<!--SNIPEND-->

There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-go/tree/main/nexus-messaging/callerpattern/) and [on-demand pattern](https://github.com/temporalio/samples-go/tree/main/nexus-messaging/ondemandpattern/).
The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it.

### Develop an Asynchronous Nexus Operation handler to start a Workflow

Call `temporalnexus.StartWorkflow` with the Client. The Operation completes when the Workflow returns, and the
Workflow's return value is delivered to the caller as the Operation's result.

<!--SNIPSTART go-nexus-async-operation-->
[go-sandbox/nexus/handler/app.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/handler/app.go)
```go
var HelloOperation = temporalnexus.MustNewTemporalOperation(
	temporalnexus.TemporalOperationOptions[service.HelloInput, service.HelloOutput]{
		Name: service.HelloOperationName,
		Start: func(
			ctx context.Context,
			nc temporalnexus.NexusClient,
			input service.HelloInput,
			options temporalnexus.StartTemporalOperationOptions,
		) (temporalnexus.TemporalOperationResult[service.HelloOutput], error) {
			return temporalnexus.StartWorkflow(ctx, nc, client.StartWorkflowOptions{
				ID: service.HelloWorkflowID(input),
				// Task queue defaults to the task queue this operation is handled on.
			}, HelloHandlerWorkflow, input)
		},
	})
```
<!--SNIPEND-->

Workflow IDs should typically be business-meaningful identifiers and are used to dedupe Workflow starts.
In general, the Workflow ID should be passed in the Operation input as part of the Nexus Service contract.

> **💡 Tip:**
> RESOURCES
>
> [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing.
>

#### Map a Nexus Operation input to multiple Workflow arguments

A Nexus Operation can only take one input parameter. `temporalnexus.StartWorkflow` is typed for a Workflow that takes a
single argument, so to start a Workflow that takes several, use `temporalnexus.StartUntypedWorkflow` and pass the
arguments after the Workflow function:

<!--SNIPSTART go-nexus-untyped-workflow-->
[go-sandbox/nexus/handler/app.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/handler/app.go)
```go
return temporalnexus.StartUntypedWorkflow[service.HelloOutput](ctx, nc, client.StartWorkflowOptions{
	ID: service.HelloWorkflowID(input),
}, HelloHandlerWorkflowWithArgs, input.Name, input.Language)
```
<!--SNIPEND-->

### Register a Nexus Service in a Worker

After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register a Nexus Service in a Worker.

<!--SNIPSTART go-nexus-handler-worker-->
[go-sandbox/nexus/handler/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/handler/worker/main.go)
```go
package main

import (
	"log"
	"os"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"

	"github.com/nexus-rpc/sdk-go/nexus"

	"my-org/quickstart/nexus/handler"
	"my-org/quickstart/nexus/options"
	"my-org/quickstart/nexus/service"
)

const (
	taskQueue = "my-handler-task-queue"
)

func main() {
	// The client and worker are heavyweight objects that should be created once per process.
	clientOptions, err := options.ParseClientOptionFlags(os.Args[1:])
	if err != nil {
		log.Fatalf("Invalid arguments: %v", err)
	}
	c, err := client.Dial(clientOptions)
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, taskQueue, worker.Options{})
	service := nexus.NewService(service.HelloServiceName)
	err = service.Register(handler.EchoOperation, handler.HelloOperation, handler.HelloWithArgsOperation, handler.ApproveOperation)
	if err != nil {
		log.Fatalln("Unable to register operations", err)
	}
	w.RegisterNexusService(service)
	w.RegisterWorkflow(handler.HelloHandlerWorkflow)
	w.RegisterWorkflow(handler.HelloHandlerWorkflowWithArgs)
	w.RegisterWorkflow(handler.ApprovalWorkflow)

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```
<!--SNIPEND-->

## Develop a caller Workflow that uses the Nexus Service 

Import the Service API package that has the necessary service and operation names and input/output types to execute a Nexus Operation from the caller Workflow:

<!--SNIPSTART go-nexus-caller-workflow-->
[go-sandbox/nexus/caller/workflows.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/caller/workflows.go)
```go
package caller

import (
	"go.temporal.io/sdk/workflow"

	"my-org/quickstart/nexus/service"
)

const (
	TaskQueue    = "my-caller-workflow-task-queue"
	endpointName = "my-nexus-endpoint-name"
)

func EchoCallerWorkflow(ctx workflow.Context, message string) (string, error) {
	c := workflow.NewNexusClient(endpointName, service.HelloServiceName)

	fut := c.ExecuteOperation(ctx, service.EchoOperationName, service.EchoInput{Message: message}, workflow.NexusOperationOptions{})

	var res service.EchoOutput
	if err := fut.Get(ctx, &res); err != nil {
		return "", err
	}

	return res.Message, nil
}

func HelloCallerWorkflow(ctx workflow.Context, name string, language service.Language) (string, error) {
	c := workflow.NewNexusClient(endpointName, service.HelloServiceName)

	fut := c.ExecuteOperation(ctx, service.HelloOperationName, service.HelloInput{Name: name, Language: language}, workflow.NexusOperationOptions{})
	var res service.HelloOutput

	// Optionally wait for the operation to be started. NexusOperationExecution will contain the operation token in
	// case this operation is asynchronous, which is a handle that can be used to perform additional actions like
	// cancelling an operation.
	var exec workflow.NexusOperationExecution
	if err := fut.GetNexusOperationExecution().Get(ctx, &exec); err != nil {
		return "", err
	}
	if err := fut.Get(ctx, &res); err != nil {
		return "", err
	}

	return res.Message, nil
}
```
<!--SNIPEND-->

### Register the caller Workflow in a Worker

After developing the caller Workflow, the next step is to register it with a Worker.

<!--SNIPSTART go-nexus-caller-worker-->
[go-sandbox/nexus/caller/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/caller/worker/main.go)
```go
package main

import (
	"log"
	"os"

	"my-org/quickstart/nexus/caller"
	"my-org/quickstart/nexus/options"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	// The client and worker are heavyweight objects that should be created once per process.
	clientOptions, err := options.ParseClientOptionFlags(os.Args[1:])
	if err != nil {
		log.Fatalf("Invalid arguments: %v", err)
	}
	c, err := client.Dial(clientOptions)
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, caller.TaskQueue, worker.Options{})

	w.RegisterWorkflow(caller.EchoCallerWorkflow)
	w.RegisterWorkflow(caller.HelloCallerWorkflow)

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```
<!--SNIPEND-->

### Develop a starter to start the caller Workflow

To initiate the caller Workflow, a starter program is used.

<!--SNIPSTART go-nexus-caller-starter-->
[go-sandbox/nexus/caller/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/caller/starter/main.go)
```go
package main

import (
	"context"
	"log"
	"os"
	"time"

	"go.temporal.io/sdk/client"

	"my-org/quickstart/nexus/caller"
	"my-org/quickstart/nexus/options"
	"my-org/quickstart/nexus/service"
)

func main() {
	clientOptions, err := options.ParseClientOptionFlags(os.Args[1:])
	if err != nil {
		log.Fatalf("Invalid arguments: %v", err)
	}
	c, err := client.Dial(clientOptions)
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()
	runWorkflow(c, caller.EchoCallerWorkflow, "Nexus Echo 👋")
	runWorkflow(c, caller.HelloCallerWorkflow, "Nexus", service.ES)
}

func runWorkflow(c client.Client, workflow interface{}, args ...interface{}) {
	ctx := context.Background()
	workflowOptions := client.StartWorkflowOptions{
		ID:        "nexus_hello_caller_workflow_" + time.Now().Format("20060102150405"),
		TaskQueue: caller.TaskQueue,
	}

	wr, err := c.ExecuteWorkflow(ctx, workflowOptions, workflow, args...)
	if err != nil {
		log.Fatalln("Unable to execute workflow", err)
	}
	log.Println("Started workflow", "WorkflowID", wr.GetID(), "RunID", wr.GetRunID())

	// Synchronously wait for the workflow completion.
	var result string
	err = wr.Get(context.Background(), &result)
	if err != nil {
		log.Fatalln("Unable get workflow result", err)
	}
	log.Println("Workflow result:", result)
}
```
<!--SNIPEND-->

## Make Nexus calls across Namespaces with a development Server 

Follow the steps below to run the Nexus handler Worker, the Nexus caller Worker, and the starter app.

### Run Workers connected to a local development server

Run the Nexus handler Worker:

```bash
cd handler
go run ./worker \
  -target-host localhost:7233 \
  -namespace my-target-namespace
```

In another terminal window, run the Nexus caller Worker:

```bash
cd caller
go run ./worker \
  -target-host localhost:7233 \
  -namespace my-caller-namespace
```

### Start a caller Workflow

With the Workers running, the final step in the local development process is to start a caller Workflow.

Run the starter:

```bash
cd caller
go run ./starter \
  -target-host localhost:7233 \
  -namespace my-caller-namespace
```

This will result in:

```
2024/10/04 19:57:40 Workflow result: Nexus Echo 👋
2024/10/04 19:57:40 Started workflow WorkflowID nexus_hello_caller_workflow_20240723195740 RunID c9789128-2fcd-4083-829d-95e43279f6d7
2024/10/04 19:57:40 Workflow result: ¡Hola! Nexus 👋
```

### Canceling a Nexus Operation 

To cancel a Nexus Operation from within a Workflow, create a Go context using the `workflow.WithCancel` API.
This returns a new context and a function that, when called, cancels the context and any SDK method that was passed this context.
The future returned by `NexusClient.ExecuteOperation` is resolved when the operation finishes, whether it succeeds, fails, times out, or is canceled.

Only asynchronous operations can be canceled in Nexus, as cancelation is sent using an operation token.
The Workflow or other resources backing the operation may choose to ignore the cancelation request.
If ignored, the operation may enter a terminal state.

Once the caller Workflow completes, the caller's Nexus Machinery stops attempting to cancel operations that have not yet
been canceled, letting them run to completion.

It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending
operations to finish before exiting the Workflow.

See the [Nexus cancelation sample](https://github.com/temporalio/samples-go/tree/main/nexus-cancelation) for reference.

## Make Nexus calls across Namespaces in Temporal Cloud 

This section assumes you are already familiar with [how to connect a Worker to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud).
The same [source code](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/go-sandbox/nexus) is used in this section, but the Temporal Cloud CLI will be used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces.

### Install `tcld` and generate certificates

Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command
(on macOS):

```
brew install temporalio/brew/tcld
```

If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below:

```
tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key
```

These certificates will be valid for one year.

### Create caller and handler Namespaces

Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler.
If you already have these Namespaces, you don't need to do this.

**Temporal CLI**

```
temporal cloud login

temporal cloud namespace create \
  --name <your-caller-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

temporal cloud namespace create \
  --name <your-target-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

**tcld**

```
tcld login

tcld namespace create \
  --namespace <your-caller-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

tcld namespace create \
  --namespace <your-target-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/namespaces](https://cloud.temporal.io/namespaces).

### Create a Nexus Endpoint to route requests from caller to handler

To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`.

**Temporal CLI**

```
temporal cloud nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file description.md
```

**tcld**

```
tcld nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file description.md
```

The `--allow-namespace` flag builds an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as
described in [Runtime Access Controls](/nexus/security#runtime-access-controls).

Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus).

### Run Workers connected to Temporal Cloud

Run the handler Worker:

```
cd handler

go run ./worker \
  -target-host <your-target-namespace.account>.tmprl.cloud:7233 \
  -namespace <your-target-namespace.account> \
  -client-cert 'path/to/your/ca.pem' \
  -client-key 'path/to/your/ca.key'
```

Run the caller Worker:

```
cd caller

go run ./worker \
  -target-host <your-caller-namespace.account>.tmprl.cloud:7233 \
  -namespace <your-caller-namespace.account> \
  -client-cert 'path/to/your/ca.pem' \
  -client-key 'path/to/your/ca.key'
```

To connect with an API key instead of mTLS certificates, replace `-client-cert` and `-client-key` with
`-api-key <your-api-key>`.

### Start a caller Workflow in Temporal Cloud

```
cd caller

go run ./starter \
  -target-host <your-caller-namespace.account>.tmprl.cloud:7233 \
  -namespace <your-caller-namespace.account> \
  -client-cert 'path/to/your/ca.pem' \
  -client-key 'path/to/your/ca.key'
```

This will result in:

```
2024/10/04 19:57:40 Workflow result: Nexus Echo 👋
2024/10/04 19:57:40 Workflow result: ¡Hola! Nexus 👋
```

## Observability

### Web UI

A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history:

![Observability Sync](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-sync.png)

An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history:

![Observability Async](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-async.png)

### Temporal CLI

Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow:

```
temporal workflow describe -w <ID>
```

Nexus events are included in the caller's Event history:

```
temporal workflow show -w <ID>
```

For **asynchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationStarted`
- `NexusOperationCompleted`

For **synchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationCompleted`

> **📝 Note:**
>
> `NexusOperationStarted` isn't reported in the caller's history for synchronous operations.
>

## Learn more

- Read the high-level description of the [Temporal Nexus feature](/evaluate/features/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082).
- Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus).
- Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus).

---

# Nexus Go quickstart - Go SDK

Source: https://docs.temporal.io/develop/go/nexus/quickstart

> Build a Nexus Service that wraps an existing Temporal Workflow using the Go SDK

[Temporal Nexus](/evaluate/features/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow.

> **ℹ️ Info:**
> To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/features/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus).

**Prerequisites:** Complete the [Go SDK Quickstart](/develop/go/set-up-your-local-go) first.
You should have `activity.go`, `workflow.go`, `worker/main.go`, and `start/main.go` from that guide.

## What you'll build

You have `SayHelloWorkflow` running in the `default` Namespace.
By the end of this guide:

1. A Nexus Service will expose `SayHelloWorkflow` as an Operation.
2. A second Namespace will contain a Workflow that calls that Operation.
3. The caller Workflow will get back `"Hello Temporal"` — the same result, but across Namespaces.

<!--SNIPSTART go-nexus-quickstart-service-->
[go-sandbox/service.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/service.go)
```go
package quickstart

const (
	HelloServiceName   = "my-hello-service"
	HelloOperationName = "say-hello"
)

type HelloInput struct {
	Name string
}
```
<!--SNIPEND-->

## 1. Define the Nexus Service

Create a file called `service.go` that defines the Nexus Service contract.

Creating a Nexus Service establishes the contract between your implementation and any callers.
It provides type safety when invoking Nexus Operations and ensures that Operation Handlers fulfill the contract.

`HelloServiceName` and `HelloOperationName` are string constants that uniquely identify the Service and Operation.
`HelloInput` defines the input type for the Operation.
`SayHelloWorkflow` returns `string`, so the Operation output type is also `string`.

<!--SNIPSTART go-nexus-quickstart-handler-->
[go-sandbox/nexus_handler.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus_handler.go)
```go
package quickstart

import (
	"context"

	"github.com/nexus-rpc/sdk-go/nexus"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/temporalnexus"
	"go.temporal.io/sdk/workflow"
)

// HelloNexusWorkflow is the handler Workflow for the Nexus SayHello Operation.
// It bridges the Nexus HelloInput to SayHelloWorkflow's string parameter.
func HelloNexusWorkflow(ctx workflow.Context, input HelloInput) (string, error) {
	return SayHelloWorkflow(ctx, input.Name)
}

var HelloOperation = temporalnexus.NewWorkflowRunOperation(
	HelloOperationName,
	HelloNexusWorkflow,
	func(ctx context.Context, input HelloInput, options nexus.StartOperationOptions) (client.StartWorkflowOptions, error) {
		return client.StartWorkflowOptions{
			// RequestID is stable across retries, making it safe to use as a Workflow ID.
			ID: options.RequestID,
		}, nil
	},
)
```
<!--SNIPEND-->

## 2. Define the Nexus Operation handlers

Create a file called `nexus_handler.go` that implements the Nexus Operation handler.

Operation handlers contain the logic that runs when a caller invokes a Nexus Operation.

`HelloNexusWorkflow` acts as the handler Workflow.
It bridges the Nexus `HelloInput` to `SayHelloWorkflow`'s `string` parameter by extracting `input.Name`.

`temporalnexus.NewWorkflowRunOperation` creates an asynchronous Nexus Operation that starts `HelloNexusWorkflow` when invoked.
The Options function returns `client.StartWorkflowOptions`, including a stable Workflow ID derived from `options.RequestID`.

<!--SNIPSTART go-nexus-quickstart-worker-->
[go-sandbox/nexusquickstart/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexusquickstart/worker/main.go)
```go
package main

import (
	"log"

	"my-org/quickstart"

	"github.com/nexus-rpc/sdk-go/nexus"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	// Empty Options defaults to the "default" Namespace.
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, "my-task-queue", worker.Options{})
	w.RegisterWorkflow(quickstart.SayHelloWorkflow)
	w.RegisterWorkflow(quickstart.HelloNexusWorkflow)
	w.RegisterActivity(quickstart.Greet)

	service := nexus.NewService(quickstart.HelloServiceName)
	err = service.Register(quickstart.HelloOperation)
	if err != nil {
		log.Fatalln("Unable to register operations", err)
	}
	w.RegisterNexusService(service)

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```
<!--SNIPEND-->

## 3. Register the Nexus Service handler in a Worker

Update your existing `worker/main.go` to register the Nexus Service Handler.

This Worker runs in the `default` Namespace — the same Namespace where `SayHelloWorkflow` is already registered.

A Worker will only poll for and process incoming Nexus requests if the Nexus Service Handlers are registered.
This is the same Worker concept used for Workflows and Activities.

`nexus.NewService` creates a named Nexus Service.
`service.Register` adds the `HelloOperation` to the Service.
`w.RegisterNexusService` registers the Service with the Worker so it can receive Nexus Operation requests.
`HelloNexusWorkflow` must also be registered so the Worker can execute it as the handler Workflow.

<!--SNIPSTART go-nexus-quickstart-caller-workflow-->
[go-sandbox/caller_workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/caller_workflow.go)
```go
package quickstart

import (
	"time"

	"go.temporal.io/sdk/workflow"
)

const (
	CallerTaskQueue = "my-caller-task-queue"
	NexusEndpoint   = "my-nexus-endpoint-name"
)

func CallerWorkflow(ctx workflow.Context, name string) (string, error) {
	c := workflow.NewNexusClient(NexusEndpoint, HelloServiceName)

	fut := c.ExecuteOperation(ctx, HelloOperationName, HelloInput{Name: name}, workflow.NexusOperationOptions{
		ScheduleToCloseTimeout: 10 * time.Second,
	})

	var result string
	if err := fut.Get(ctx, &result); err != nil {
		return "", err
	}
	return result, nil
}
```
<!--SNIPEND-->

## 4. Develop the caller Workflow

Create a file called `caller_workflow.go` that defines a Workflow which invokes the Nexus Operation.

The caller Workflow demonstrates the consumer side of Nexus.
Instead of importing handler code directly, the caller only depends on the Service contract.
This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams.

`workflow.NewNexusClient` creates a client bound to your Nexus Service and Endpoint.
`ExecuteOperation` starts the Operation and returns a future.
`fut.Get` blocks until the Operation completes and writes the result into `result`.

## 5. Create the caller Namespace and Nexus Endpoint

Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler.
The handler uses the `default` Namespace that was created when you started the dev server.

Namespaces provide isolation between the caller and handler sides.
The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue.
The endpoint name must match the `NexusEndpoint` constant defined in `caller_workflow.go` from step 4.

Make sure your local Temporal dev server is running (`temporal server start-dev`).

```bash
temporal operator namespace create --namespace my-caller-namespace
```

```bash
temporal operator nexus endpoint create \\
  --name my-nexus-endpoint-name \\
  --target-namespace default \\
  --target-task-queue my-task-queue
```

<!--SNIPSTART go-nexus-quickstart-caller-main-->
[go-sandbox/nexusquickstart/caller/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexusquickstart/caller/main.go)
```go
package main

import (
	"context"
	"log"

	"my-org/quickstart"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	c, err := client.Dial(client.Options{
		Namespace: "my-caller-namespace",
	})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, quickstart.CallerTaskQueue, worker.Options{})
	w.RegisterWorkflow(quickstart.CallerWorkflow)

	if err := w.Start(); err != nil {
		log.Fatalln("Unable to start worker", err)
	}
	defer w.Stop()

	wr, err := c.ExecuteWorkflow(context.Background(), client.StartWorkflowOptions{
		ID:        "nexus-caller-workflow",
		TaskQueue: quickstart.CallerTaskQueue,
	}, quickstart.CallerWorkflow, "Temporal")
	if err != nil {
		log.Fatalln("Unable to execute workflow", err)
	}

	var result string
	if err := wr.Get(context.Background(), &result); err != nil {
		log.Fatalln("Unable to get workflow result", err)
	}
	log.Println("Workflow result:", result)
}
```
<!--SNIPEND-->

## 6. Run and verify

Create `caller/main.go` to start the caller Worker and execute the Workflow.

This brings everything together: the caller Worker hosts `CallerWorkflow`, which uses the Nexus client to invoke `say-hello` on the handler side.
The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `HelloNexusWorkflow` (which calls `SayHelloWorkflow`), and back to the caller.

**Ensure the `nexus-rpc` dependency is synchronized:**

In a terminal, from within your project:

```bash
go mod tidy
```

**Run the application:**

1. Start the handler Worker in one terminal:

```bash
go run worker/main.go
```

2. Run the caller in another terminal:

```bash
go run caller/main.go
```

You should see:

```
Workflow result: Hello Temporal
```

Open the [Temporal Web UI](http://localhost:8233) and switch between Namespaces to see both Workflow Executions.
In `my-caller-namespace`, find the `CallerWorkflow` execution — you should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in its history.
In `default`, find the `HelloNexusWorkflow` execution that was started by the Nexus Operation.

## Next steps

Now that you have a working Nexus Service, here are some resources to deepen your understanding:

- **[Go Nexus feature guide](/develop/go/nexus/feature-guide)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls.
- **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics.
- **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker.
- **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns.
- **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries.
- **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls.
- **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests.
- **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity.

---

# Standalone Nexus Operations - Go SDK

Source: https://docs.temporal.io/develop/go/nexus/standalone-operations

> Execute Nexus Operations independently without a Workflow using the Temporal Go SDK.

> **Public Preview**
> Requires Go SDK `v1.46.0` or above. API breaking changes are kept to a minimum before General Availability.

[Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without
being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using
`workflow.NewNexusClient()`, you execute a Standalone Nexus Operation directly from a Nexus Client created using
`client.NewNexusClient()`.

Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as
Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/go/nexus/feature-guide) for details on
[defining a Service contract](/develop/go/nexus/feature-guide#define-nexus-service-contract),
[developing Operation handlers](/develop/go/nexus/feature-guide#develop-nexus-service-operation-handlers), and
[registering a Service in a Worker](/develop/go/nexus/feature-guide#register-a-nexus-service-in-a-worker).

This page focuses on the client-side APIs that are unique to Standalone Nexus Operations:

- [Execute a Standalone Nexus Operation](#execute-operation)
- [Get the result of a Standalone Nexus Operation](#get-operation-result)
- [List Standalone Nexus Operations](#list-operations)
- [Count Standalone Nexus Operations](#count-operations)
- [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud)

> **📝 Note:**
> This documentation uses source code from the
> [Go Nexus Standalone sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/go-sandbox/nexus/standalone).
> It reuses the Nexus Service and Operation handlers from the [Nexus feature guide](/develop/go/nexus/feature-guide).
>

## Prerequisites 

Standalone Nexus Operations are a Public Preview feature and require Temporal CLI v1.9.0 or higher.

### 1. Install and verify the Temporal CLI 

The `temporal nexus operation` commands require Temporal CLI v1.9.0 or higher. See
[Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for install options,
then verify:

```bash
temporal --version
```

Which should output v1.9.0 or higher.

### 2. Start a local dev server 

The dev server enables Standalone Nexus Operations by default — no dynamic config is
required. Start it with the caller and handler Namespaces pre-created:

```bash
temporal server start-dev \
  --namespace my-caller-namespace \
  --namespace my-handler-namespace
```

The starter and Worker connect to two different Namespaces (a caller Namespace and a handler
Namespace), mirroring how Nexus crosses Namespace boundaries.

To run the examples on this page against the [Go sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/go-sandbox/nexus/standalone),
create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue:

```bash
temporal operator nexus endpoint create \
  --name my-nexus-endpoint \
  --target-namespace my-handler-namespace \
  --target-task-queue nexus-handler-queue
```

Then start the sample Worker in the handler Namespace from the `go-sandbox` directory:

```bash
TEMPORAL_NAMESPACE=my-handler-namespace go run ./nexus/standalone/worker
```

## Execute a Standalone Nexus Operation 

To execute a Standalone Nexus Operation, first create a
[`NexusClient`](https://pkg.go.dev/go.temporal.io/sdk/client#NexusClient) using `client.NewNexusClient()`, bound to a
specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call `ExecuteOperation()` from
application code (for example, a starter program), not from inside a Workflow Definition.

`ExecuteOperation` returns a [`NexusOperationHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#NexusOperationHandle)
that you can use to get the result of the Operation.
[`StartNexusOperationOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartNexusOperationOptions) requires `ID`.
`ScheduleToCloseTimeout` is optional and defaults to the maximum allowed by the Temporal server.

<!--SNIPSTART go-nexus-standalone-execute-->
[go-sandbox/nexus/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/standalone/starter/main.go)
```go
nexusClient, err := c.NewNexusClient(client.NexusClientOptions{
	Endpoint: "my-nexus-endpoint",
	Service:  "my-hello-service",
})
if err != nil {
	return nil, err
}

handle, err := nexusClient.ExecuteOperation(ctx, operationName, input, client.StartNexusOperationOptions{
	ID:                     operationID, // A unique Operation ID, such as "my-echo-op"
	ScheduleToCloseTimeout: 10 * time.Second,
})
```
<!--SNIPEND-->

See the full
[starter sample](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/standalone/starter/main.go)
for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and
counts Operations.

To run the starter in the caller Namespace (in a separate terminal from the Worker):

```
TEMPORAL_NAMESPACE=my-caller-namespace go run ./nexus/standalone/starter
```

Or use the Temporal CLI to execute a Standalone Nexus Operation:

```bash
temporal nexus operation execute \
  --namespace my-caller-namespace \
  --endpoint my-nexus-endpoint \
  --service my-hello-service \
  --operation echo \
  --operation-id my-echo-op \
  --input '{"Message":"hello"}'
```

## Get the result of a Standalone Nexus Operation 

Use `NexusOperationHandle.Get()` to block until the Operation completes and retrieve its result. This works for both
synchronous and asynchronous (Workflow-backed) Operations.

<!--SNIPSTART go-nexus-standalone-get-result-->
[go-sandbox/nexus/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/standalone/starter/main.go)
```go
var result service.EchoOutput
err := handle.Get(context.Background(), &result)
if err != nil {
	log.Fatalln("Operation failed", err)
}
log.Println("Operation result:", result.Message)
```
<!--SNIPEND-->

If the Operation completed successfully, the result is deserialized into the provided pointer. If the Operation failed,
the failure is returned as an error.

Or use the Temporal CLI to wait for a result by Operation ID:

```bash
temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op
```

## List Standalone Nexus Operations 

Use [`client.ListNexusOperations()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to list Standalone Nexus
Operation Executions that match a [List Filter](/list-filter) query. The result contains an iterator that yields
operation metadata entries.

Note that `ListNexusOperations` is called on the base `client.Client`, not on the `NexusClient`.

<!--SNIPSTART go-nexus-standalone-list-->
[go-sandbox/nexus/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/standalone/starter/main.go)
```go
resp, err := c.ListNexusOperations(context.Background(), client.ListNexusOperationsOptions{
	Query: "Endpoint = 'my-nexus-endpoint'",
})
if err != nil {
	log.Fatalln("Unable to list Nexus operations", err)
}

for metadata, err := range resp.Results {
	if err != nil {
		log.Fatalln("Error iterating operations", err)
	}
	log.Printf("OperationID: %s, Operation: %s, Status: %v\n",
		metadata.OperationID, metadata.Operation, metadata.Status)
}
```
<!--SNIPEND-->

The `Query` field accepts [List Filter](/list-filter) syntax. For example,
`"Endpoint = 'my-endpoint' AND ExecutionStatus = 'Running'"`.

Or use the Temporal CLI:

```bash
temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Count Standalone Nexus Operations 

Use [`client.CountNexusOperations()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to count Standalone Nexus
Operation Executions that match a [List Filter](/list-filter) query.

Note that `CountNexusOperations` is called on the base `client.Client`, not on the `NexusClient`.

<!--SNIPSTART go-nexus-standalone-count-->
[go-sandbox/nexus/standalone/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/nexus/standalone/starter/main.go)
```go
resp, err := c.CountNexusOperations(context.Background(), client.CountNexusOperationsOptions{
	Query: "Endpoint = 'my-nexus-endpoint'",
})
if err != nil {
	log.Fatalln("Unable to count Nexus operations", err)
}

log.Println("Total Nexus operations:", resp.Count)
```
<!--SNIPEND-->

Or use the Temporal CLI:

```bash
temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Run Standalone Nexus Operations with Temporal Cloud 

The code samples on this page use `envconfig.MustLoadDefaultClientOptions()`, so the same code
works against Temporal Cloud — just configure the connection via environment variables or a TOML
profile. No code changes are needed.

For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint
setup, certificate generation, and authentication options, see
[Make Nexus calls across Namespaces in Temporal Cloud](/develop/go/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud)
and [Connect to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud).

---

# Platform - Go SDK

Source: https://docs.temporal.io/develop/go/platform

> This section explains how to implement platform with the Go SDK

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Platform

- [Observability](/develop/go/platform/observability)
- [Enriching the UI](/develop/go/platform/enriching-ui)

---

# Enriching the user interface - Go SDK

Source: https://docs.temporal.io/develop/go/platform/enriching-ui

> Add contextual information to workflows and events in the Temporal UI using the Go SDK.

Temporal supports adding context to Workflows and Events with metadata. 
This helps users identify and understand Workflows and their operations.

## Adding summary and details to Workflows

### Starting a Workflow

When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI:

<!--SNIPSTART go-enrich-ui-static-summary-->
[go-sandbox/platform/enrichui/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/enrichui/starter/main.go)
```go
package main

import (
	"context"
	"log"

	"go.temporal.io/sdk/client"

	"my-org/quickstart/platform/enrichui"
)

func main() {
	// Create the client
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	// Start workflow options with static summary and details
	workflowOptions := client.StartWorkflowOptions{
		ID:            "your-workflow-id",
		TaskQueue:     "your-task-queue",
		StaticSummary: "Order processing for customer #12345",
		StaticDetails: "Processing premium order with expedited shipping",
	}

	// Start the workflow
	we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, enrichui.YourWorkflow, "workflow input")
	if err != nil {
		log.Fatalln("Unable to execute workflow", err)
	}
	log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID())
}
```
<!--SNIPEND-->

`StaticSummary` is a single-line description that appears in the Workflow list view, limited to 200 bytes.
`StaticDetails` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes.

The input format is standard Markdown excluding images, HTML, and scripts.

### Inside the Workflow

Within a Workflow, you can get and set the _current workflow details_. 
Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. 
Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines.

<!--SNIPSTART go-enrich-ui-current-details-->
[go-sandbox/platform/enrichui/details.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/enrichui/details.go)
```go
import (
	"go.temporal.io/sdk/workflow"
)

func YourWorkflow(ctx workflow.Context, input string) (string, error) {
	// Get the current details
	currentDetails := workflow.GetCurrentDetails(ctx)
	workflow.GetLogger(ctx).Info("Current details", "details", currentDetails)

	// Set/update the current details
	workflow.SetCurrentDetails(ctx, "Updated workflow details with new status")

	return "Workflow completed", nil
}
```
<!--SNIPEND-->

### Adding Summary to Activities and Timers

You can attach a metadata parameter `Summary` to Activities when starting them from within a Workflow:

<!--SNIPSTART go-enrich-ui-activity-summary-->
[go-sandbox/platform/enrichui/activity_summary.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/enrichui/activity_summary.go)
```go
import (
	"time"

	"go.temporal.io/sdk/workflow"
)

func ActivitySummaryWorkflow(ctx workflow.Context, input string) (string, error) {
	// Activity options with summary
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
		Summary:             "Processing user data",
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	// Execute the activity
	var result string
	err := workflow.ExecuteActivity(ctx, YourActivity, input).Get(ctx, &result)
	if err != nil {
		return "", err
	}

	return result, nil
}
```
<!--SNIPEND-->

Similarly, you can attach a `Summary` to timers within a Workflow:

<!--SNIPSTART go-enrich-ui-timer-summary-->
[go-sandbox/platform/enrichui/timer_summary.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/enrichui/timer_summary.go)
```go
import (
	"time"

	"go.temporal.io/sdk/workflow"
)

func TimerSummaryWorkflow(ctx workflow.Context, input string) (string, error) {
	// Create a timer with options including summary
	timerFuture := workflow.NewTimerWithOptions(ctx, 5*time.Minute, workflow.TimerOptions{
		Summary: "Waiting for payment confirmation",
	})

	// Wait for the timer
	err := timerFuture.Get(ctx, nil)
	if err != nil {
		return "", err
	}

	return "Timer completed", nil
}
```
<!--SNIPEND-->

The input format for `Summary` is a string, and limited to 200 bytes. 

## Viewing summary and details in the UI

Once you've added summaries and details to your workflows, activities, and timers, you can view this enriched information in the Temporal Web UI. 
Navigate to your Workflow's details page to see the metadata displayed in three key locations:

### User Metadata

On the Workflow Details page, under the **User Metadata** tab, you'll find the workflow-level metadata:

- **Summary & Details** - Displays the static summary and static details set when starting the workflow
- **Current Details** - Displays the dynamic details that can be updated during workflow execution

All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays.

### Timeline

The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance.

Labels longer than 120 characters are truncated with an ellipsis.

Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. 

Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later.

### Event History

Individual events in the Workflow's Event History display their associated summaries when available, both in the timeline graph at the top of the tab and in the events table below it.

Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the event details. 
When you do expand an event, the summary is also prominently displayed in the detailed view.

---

# Observability - Go SDK

Source: https://docs.temporal.io/develop/go/platform/observability

> Monitor your Temporal Application state using Metrics, Tracing, Logging, and Visibility features. Emit metrics, configure tracing, customize logging, and use Search Attributes with the Temporal Go SDK for enhanced Workflow Execution insights.

This page covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

This section covers features related to viewing the state of the application, including:

- [Metrics](#metrics)
- [Tracing](#tracing)
- [Logging](#logging)
- [Visibility](#visibility)

## How to emit metrics 

Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process.
For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics).

> **💡 Use the OpenTelemetry v2 integration:**
>
> To instrument Temporal applications with OpenTelemetry, use the
> [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2)
> ([Pre-release](/evaluate/product-release-stages#pre-release)).
> It propagates OpenTelemetry context across Temporal boundaries and
> is replay-safe when instrumenting Workflows.
>

- For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics).
- For an end-to-end example that exposes metrics with the Go SDK, refer to the [samples-go](https://github.com/temporalio/samples-go/tree/main/metrics) repo.

To emit metrics from the Temporal Client in Go, create a [metrics handler](https://pkg.go.dev/go.temporal.io/sdk/internal/common/metrics#Handler) from the [Client Options](https://pkg.go.dev/go.temporal.io/sdk@v1.15.0/internal#ClientOptions) and specify a listener address to be used by Prometheus.

<!--SNIPSTART go-observability-prometheus-metrics-->
[go-sandbox/platform/observability/metrics.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/metrics.go)
```go
clientOptions := client.Options{
	MetricsHandler: sdktally.NewMetricsHandler(newPrometheusScope(prometheus.Configuration{
		ListenAddress: "0.0.0.0:9090",
		TimerType:     "histogram",
	})),
}
```
<!--SNIPEND-->

The Go SDK provides metrics handlers for [Tally](https://pkg.go.dev/go.temporal.io/sdk/contrib/tally) and [OpenTelemetry](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry). Tally offers [extensible custom metrics reporting](https://github.com/uber-go/tally#report-your-metrics), which is exposed through the [`WithCustomMetricsHandler`](/references/server-options#withcustommetricshandler) API.

For more information, see the [Go sample for metrics](https://github.com/temporalio/samples-go/tree/main/metrics).

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Call [`WithTags`](https://pkg.go.dev/go.temporal.io/sdk/client#MetricsHandler) on the metrics handler before you set it on the Client Options.
Every metric created from that handler carries the tags, from both the Client and the Worker.

<!--SNIPSTART go-observability-global-tags-->
[go-sandbox/telemetry-worker/main-prometheus.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/telemetry-worker/main-prometheus.go)
```go
// Create the base OTel metrics handler
metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{})

// Add global/static tags to all emitted metrics
globalTagsHandler := metricsHandler.WithTags(map[string]string{
	"team":        "content-platform",
	"service":     "checkout",
	"cost_center": "cc-1042",
	"environment": "production",
})

// Attach the tagged handler to client options
clientOptions := client.Options{
	MetricsHandler: globalTagsHandler,
}

temporalClient, err := client.Dial(clientOptions)
```
<!--SNIPEND-->

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag           | Example            | Question it answers                                       |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team`        | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service`     | `checkout`         | Which application emits these metrics?                    |
| `cost_center` | `cc-1042`          | Which budget does this Worker fleet belong to?            |
| `environment` | `production`       | Is this production traffic, or staging or test?           |
| `region`      | `us-east-2`        | Where does the Worker fleet run?                          |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow ID or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

### Configure OpenTelemetry counters as monotonic 

> **📝 Note:**
>
> `UseMonotonicCounters` is available in `go.temporal.io/sdk/contrib/opentelemetry` version 0.8.0 and later.
>

By default, the OpenTelemetry metrics handler represents counters as `Int64UpDownCounter` instruments to preserve compatibility with earlier releases.
To represent Temporal SDK counters as monotonic `Int64Counter` instruments, set `UseMonotonicCounters` to `true` when you create the handler:

<!--SNIPSTART go-observability-monotonic-counters-->
[go-sandbox/platform/observability/monotonic.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/monotonic.go)
```go
metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{
	Meter:                otel.GetMeterProvider().Meter("temporal-sdk-go"),
	UseMonotonicCounters: true,
})

temporalClient, err := client.Dial(client.Options{
	MetricsHandler: metricsHandler,
})
```
<!--SNIPEND-->

Monotonic counters let exporters and metrics backends classify Temporal SDK counters correctly.
The [`MetricsCounter`](https://pkg.go.dev/go.temporal.io/sdk/client#MetricsCounter) contract defines counters as ever-increasing.
If you create custom counters through the same metrics handler, pass only non-negative values to [`client.MetricsCounter.Inc`](https://pkg.go.dev/go.temporal.io/sdk/internal/common/metrics#Counter).
Negative values can produce invalid or backend-dependent metric data when `UseMonotonicCounters` is enabled.

## Tracing 

Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and Child Workflows.

> **💡 Use the OpenTelemetry v2 integration:**
>
> To instrument Temporal applications with OpenTelemetry, use the
> [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2)
> ([Pre-release](/evaluate/product-release-stages#pre-release)).
> It propagates OpenTelemetry context across Temporal boundaries and
> is replay-safe when instrumenting Workflows.
>

The Go SDK provides tracing interceptors for [OpenTelemetry](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry), [OpenTracing](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentracing), and [Datadog](https://pkg.go.dev/go.temporal.io/sdk/contrib/datadog/tracing). 

First, create a tracing interceptor for Client instantiation.

<!--SNIPSTART go-observability-tracing-interceptors-->
[go-sandbox/platform/observability/tracing.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/tracing.go)
```go
// OpenTelemetry
func newOpenTelemetryInterceptor() (interceptor.Interceptor, error) {
	return opentelemetry.NewTracingInterceptor(opentelemetry.TracerOptions{})
}

// OpenTracing
func newOpenTracingInterceptor() (interceptor.Interceptor, error) {
	return opentracing.NewInterceptor(opentracing.TracerOptions{})
}

// Datadog
func newDatadogInterceptor() interceptor.Interceptor {
	return tracing.NewTracingInterceptor(tracing.TracerOptions{})
}
```
<!--SNIPEND-->

 and register it by passing it to [ClientOptions](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions):

<!--SNIPSTART go-observability-register-tracing-->
[go-sandbox/platform/observability/tracing.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/tracing.go)
```go
c, err := client.Dial(client.Options{
	Interceptors: []interceptor.ClientInterceptor{tracingInterceptor},
})
```
<!--SNIPEND-->

You can also register interceptors through a [Plugin](/develop/plugins-guide#interceptors) if you’re building a reusable library.

Each tracing interceptor uses its library's native propagation mechanism to serialize trace spans into Temporal headers. For example, OpenTelemetry uses its `TextMapPropagator` with the W3C TraceContext format. The SDK carries these headers across Workflow, Activity, and Child Workflow boundaries, so the tracing library can reconstruct the call graph.
For more information, see the documentation for [OpenTelemetry](https://opentelemetry.io/), [OpenTracing](https://opentracing.io), and [Datadog](https://docs.datadoghq.com/tracing/).

To build custom context propagation (for example, tenant IDs, auth tokens), see [Context propagation](/develop/go/best-practices/context-propagation).

## Log from a Workflow 

Send logs and errors to a logging service, so that when things go wrong, you can see what happened.

Loggers create an audit trail and capture information about your Workflow's operation.
An appropriate logging level depends on your specific needs.
During development or troubleshooting, you might use debug or even trace.
In production, you might use info or warn to avoid excessive log volume.

You can find the log levels supported by `slog` in [their official documentation](https://pkg.go.dev/log/slog#Level). The Temporal SDK core normally uses `WARN` as its default logging level.

In Workflow Definitions you can use [`workflow.GetLogger(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetLogger) to write logs.

<!--SNIPSTART go-observability-workflow-logger-->
[go-sandbox/platform/observability/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/workflow.go)
```go
import (
	"time"

	"go.temporal.io/sdk/workflow"
)

// Workflow is a standard workflow definition.
func Workflow(ctx workflow.Context, name string) (string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	// Getting the logger from the context.
	logger := workflow.GetLogger(ctx)
	// Logging a message with the key value pair `name` and `name`
	logger.Info("Workflow started", "name", name)

	var result string
	err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result)
	if err != nil {
		logger.Error("Activity failed.", "Error", err)
		return "", err
	}

	logger.Info("Workflow completed.", "result", result)

	return result, nil
}
```
<!--SNIPEND-->

### Provide a custom logger 

This field sets a custom Logger that is used for all logging actions of the instance of the Temporal Client.

The Go SDK supports custom loggers via `log.NewStructuredLogger()`, which wraps Go's standard [`slog.Logger`](https://pkg.go.dev/log/slog) (Go 1.21+).
Because most modern logging libraries (zap, zerolog, logrus, etc.) can back a `slog.Handler`, `slog` serves as the universal bridge to third-party loggers.

**Using slog directly:**

<!--SNIPSTART go-observability-slog-logger-->
[go-sandbox/platform/observability/slog_logger.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/slog_logger.go)
```go
import (
	"log/slog"
	"os"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/log"
)

func DialWithSlog() (client.Client, error) {
	// ...
	slogHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})
	logger := log.NewStructuredLogger(slog.New(slogHandler))
	clientOptions := client.Options{
		Logger: logger,
	}
	temporalClient, err := client.Dial(clientOptions)
	// ...
	return temporalClient, err
}
```
<!--SNIPEND-->

**Bridging a third-party logger through slog (example with zap):**

<!--SNIPSTART go-observability-zap-logger-->
[go-sandbox/platform/observability/zap_logger.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/zap_logger.go)
```go
import (
	"log/slog"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/log"
	"go.uber.org/zap"
	"go.uber.org/zap/exp/zapslog"
)

func DialWithZap() (client.Client, error) {
	// ...
	zapLogger, err := zap.NewProduction()
	if err != nil {
		return nil, err
	}
	handler := zapslog.NewHandler(zapLogger.Core())
	logger := log.NewStructuredLogger(slog.New(handler))
	clientOptions := client.Options{
		Logger: logger,
	}
	temporalClient, err := client.Dial(clientOptions)
	// ...
	return temporalClient, err
}
```
<!--SNIPEND-->

As an alternative, you can implement the `log.Logger` interface directly.
The Temporal samples repo has a [zap adapter](https://github.com/temporalio/samples-go/blob/main/zapadapter/zap_adapter.go) that can be used as a reference.

## Visibility APIs 

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### Search Attributes 

The typical method of retrieving a Workflow Execution is by its Workflow ID.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI.
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `UpsertSearchAttributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `DescribeWorkflow`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [In the Temporal CLI](/cli/command-reference/workflow#list).
  - In code by calling `ListWorkflowExecutions`.

Here is how to query Workflow Executions:

The [ListWorkflow()](https://pkg.go.dev/go.temporal.io/sdk/client#Client.ListWorkflow) function retrieves a list of [Workflow Executions](/workflow-execution) that match the [Search Attributes](/search-attribute) of a given [List Filter](/list-filter).
The metadata returned from the [Visibility](/temporal-service/visibility) store can be used to get a Workflow Execution's history and details from the [Persistence](/temporal-service/persistence) store.

Use a List Filter to define a `request` to pass into `ListWorkflow()`.

<!--SNIPSTART go-observability-list-request-->
[go-sandbox/platform/observability/visibility.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/visibility.go)
```go
request := &workflowservice.ListWorkflowExecutionsRequest{Query: "ExecutionStatus = 'Running'"}
```
<!--SNIPEND-->

This `request` value returns only open Workflows.
For more List Filter examples, see the [examples provided for List Filters in the Temporal Visibility guide.](/list-filter#list-filter-examples)

<!--SNIPSTART go-observability-list-workflows-->
[go-sandbox/platform/observability/visibility.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/visibility.go)
```go
resp, err := temporalClient.ListWorkflow(context.Background(), request)
if err != nil {
	return err
}

fmt.Println("First page of results:")
for _, exec := range resp.Executions {
	fmt.Printf("Workflow ID %v\n", exec.Execution.WorkflowId)
}
```
<!--SNIPEND-->

### Set custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using the `temporal operator search-attribute create` command or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

Provide key-value pairs in [`StartWorkflowOptions.SearchAttributes`](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions).

Search Attributes are represented as `map[string]interface{}`.
The values in the map must correspond to the [Search Attribute's value type](/search-attribute#supported-types):

- Bool = `bool`
- Datetime = `time.Time`
- Double = `float64`
- Int = `int64`
- Keyword = `string`
- Text = `string`

If you had custom Search Attributes `CustomerId` of type Keyword and `MiscData` of type Text, you would provide `string` values:

<!--SNIPSTART go-observability-set-search-attributes-->
[go-sandbox/platform/observability/visibility.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/visibility.go)
```go
func (c *Client) CallYourWorkflow(ctx context.Context, workflowID string, payload map[string]interface{}) error {
	// ...
	searchAttributes := map[string]interface{}{
		"CustomerId": payload["customer"],
		"MiscData":   payload["miscData"],
	}
	options := client.StartWorkflowOptions{
		ID:               workflowID,
		TaskQueue:        "your-task-queue",
		SearchAttributes: searchAttributes,
		// ...
	}
	_, err := c.Client.ExecuteWorkflow(ctx, options, YourWorkflow, payload)
	// ...
	return err
}
```
<!--SNIPEND-->

### Upsert Search Attributes 

You can upsert Search Attributes to add or update Search Attributes from within Workflow code.

In advanced cases, you may want to dynamically update these attributes as the Workflow progresses.
[UpsertSearchAttributes](https://pkg.go.dev/go.temporal.io/sdk/workflow#UpsertSearchAttributes) is used to add or update Search Attributes from within Workflow code.

`UpsertSearchAttributes` will merge attributes to the existing map in the Workflow.
Consider this example Workflow code:

<!--SNIPSTART go-observability-upsert-search-attributes-->
[go-sandbox/platform/observability/visibility.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/platform/observability/visibility.go)
```go
func UpsertWorkflow(ctx workflow.Context, input string) error {

	attr1 := map[string]interface{}{
		"CustomIntField":  1,
		"CustomBoolField": true,
	}
	if err := workflow.UpsertSearchAttributes(ctx, attr1); err != nil {
		return err
	}

	attr2 := map[string]interface{}{
		"CustomIntField":     2,
		"CustomKeywordField": "seattle",
	}
	if err := workflow.UpsertSearchAttributes(ctx, attr2); err != nil {
		return err
	}
	return nil
}
```
<!--SNIPEND-->

After the second call to `UpsertSearchAttributes`, the map will contain:

```go
map[string]interface{}{
  "CustomIntField": 2, // last update wins
  "CustomBoolField": true,
  "CustomKeywordField": "seattle",
}
```

### Remove a Search Attribute from a Workflow 

To remove a Search Attribute that was previously set, set it to an empty array: `[]`.

**There is no support for removing a field.**

However, to achieve a similar effect, set the field to some placeholder value.
For example, you could set `CustomKeywordField` to `impossibleVal`.
Then searching `CustomKeywordField != 'impossibleVal'` will match Workflows with `CustomKeywordField` not equal to `impossibleVal`, which includes Workflows without the `CustomKeywordField` set.

---

# Set up a local development environment with the Go SDK

Source: https://docs.temporal.io/develop/go/set-up-your-local-go

> Configure your local development environment to get started developing with Temporal

# Quickstart

Configure your local development environment to get started developing with Temporal.

## Install Go

Make sure you have Go installed. These tutorials were produced using Go 1.18. Check your version of Go with the
following command:

This will return your installed Go version.

```bash
go version
```

```bash
go version go1.18.1 darwin/amd64
```

## Install the Temporal Go SDK

If you are creating a new project using the Temporal Go SDK, you can start by creating a new directory.

Next, switch to the new directory.

Then, initialize a Go project in that directory.

Finally, install the Temporal SDK with `go get`.

```bash
mkdir goproject
```

```bash
cd goproject
```

```bash
go mod init my-org/quickstart
```

```bash
go get go.temporal.io/sdk
```

```bash
go get go.temporal.io/sdk/client
```

## Install Temporal CLI and start the development server

The fastest way to get a development version of the Temporal Service running on your local machine is to use
[Temporal CLI](/cli).

Choose your operating system to install Temporal CLI:

**macOS**

Install the Temporal CLI using Homebrew:
```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH, for example:
```bash
sudo mv temporal /usr/local/bin
```

## Start the development server

Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory
database.

The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at
http://localhost:8233.

Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal
Service at any time by pressing CTRL+C.

Once you have everything installed, you're ready to build apps with Temporal on your local machine.

After installing, open a new Terminal window and start the development server:
```bash
temporal server start-dev
```

#### Change the Web UI port
The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server:
```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and
Activity.

This test will confirm that:

- The Temporal Go SDK is properly installed
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

### 1. Create the Activity

An Activity is a normal function or method that executes a single, well-defined action (either short- or long-running)
that is typically prone to failure. Examples include any action that interacts with the outside world, such as sending
emails, making network requests, writing to a database, or calling an API. If an Activity fails, Temporal automatically
retries it based on your configuration.

Create an Activity file (activity.go):

<!--SNIPSTART go-quickstart-activity-->
[go-sandbox/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activity.go)
```go
package quickstart

import (
	"context"
	"fmt"
)

func Greet(ctx context.Context, name string) (string, error) {
	return fmt.Sprintf("Hello %s", name), nil
}
```
<!--SNIPEND-->

### 2. Create the Workflow

Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run—and
keep running—for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will
automatically recreate its pre-failure state so it can continue right where it left off.

Create a Workflow file (workflow.go):

<!--SNIPSTART go-quickstart-workflow-->
[go-sandbox/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflow.go)
```go
package quickstart

import (
	"time"

	"go.temporal.io/sdk/workflow"
)

func SayHelloWorkflow(ctx workflow.Context, name string) (string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: time.Second * 10,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)

	var result string
	err := workflow.ExecuteActivity(ctx, Greet, name).Get(ctx, &result)
	if err != nil {
		return "", err
	}

	return result, nil
}
```
<!--SNIPEND-->

### 3. Create and Run the Worker

With your Activity and Workflow defined, you need a Worker to execute them. A Worker polls a Task Queue, that you
configure it to poll, looking for work to do. Once the Worker dequeues a Workflow or Activity task from the Task Queue,
it then executes that task.

Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your
Workflows and Activities. For more information on Workers, see
[Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers).

Create a Worker file (worker/main.go):

<!--SNIPSTART go-worker-main-->
[go-sandbox/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/worker/main.go)
```go
package main

import (
	"log"

	"my-org/quickstart"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, "my-task-queue", worker.Options{})

	w.RegisterWorkflow(quickstart.SayHelloWorkflow)
	w.RegisterActivity(quickstart.Greet)

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```
<!--SNIPEND-->

Run the Worker:

```bash
go run worker/main.go
```

### 4. Execute the Workflow

Now that your Worker is running, it's time to start a Workflow Execution.

Create a separate file called start/main.go:

<!--SNIPSTART go-quickstart-start-workflow-->
[go-sandbox/start/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/start/main.go)
```go
package main

import (
	"context"
	"log"
	"os"

	"my-org/quickstart"

	"go.temporal.io/sdk/client"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	options := client.StartWorkflowOptions{
		ID:        "greeting-workflow",
		TaskQueue: "my-task-queue",
	}

	we, err := c.ExecuteWorkflow(context.Background(), options, quickstart.SayHelloWorkflow, os.Args[1])
	if err != nil {
		log.Fatalln("Unable to execute workflow", err)
	}
	log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID())

	var result string
	err = we.Get(context.Background(), &result)
	if err != nil {
		log.Fatalln("Unable get workflow result", err)
	}
	log.Println("Workflow result:", result)
}
```
<!--SNIPEND-->

Then run:

```bash
go run start/main.go Temporal
```

### Verify success

If everything is working correctly, you should see:

- Worker processing the workflow and activity
- Output: `Workflow result: Hello Temporal`
- Workflow Execution details in the [Temporal Web UI](http://localhost:8233)

- [Run your first Temporal Application](https://learn.temporal.io/getting_started/go/first_program_in_go/): Create a basic Workflow and run it with the Temporal Go SDK

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Worker Versioning (legacy) - Go SDK

Source: https://docs.temporal.io/develop/go/worker-versioning-legacy

> Learn the Go SDK's outdated Worker Versioning APIs.

## How to use Worker Versioning in Go (Deprecated) 

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>
> See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information.
>

A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp.
To use Worker Versioning, you need to pass a Build ID to your Go Worker and opt in to Worker Versioning.

### Assign a Build ID to your Worker and opt in to Worker Versioning

You should understand assignment rules before completing this step.
See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information.

To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on.

```go
// ...
workerOptions := worker.Options{
   BuildID: buildID,
   UseBuildIDForVersioning: true,
// ...
}
w := worker.New(c, "your_task_queue_name", workerOptions)
// ...
```

> **⚠️ Warning:**
>
> Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules.
>

### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows

By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the Workflow that created them if they are also configured to run on the same Task Queue.
When configured to run on a separate Task Queue, they will default to using the current assignment rules.

If you want to override this behavior, you can specify your intent via the `VersioningIntent` field on the appropriate options struct.

For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent:

```go
// ...
ao := workflow.ActivityOptions{
    VersioningIntent: VersioningIntentUseAssignmentRules,
    // ...other options
}
activityCtx := workflow.WithActivityOptions(ctx, ao)
var yourActivityResult YourActivityResultType
err := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult)
// ...
```

#### Specifying versions for Continue-As-New

When using the Continue-As-New feature, use the `WithWorkflowVersioningIntent` context modifier:

```go
ctx = workflow.WithWorkflowVersioningIntent(ctx, temporal.VersioningIntentUseAssignmentRules)
err := workflow.NewContinueAsNewError(ctx, "WorkflowName")
```

### Tell the Task Queue about your Worker's Build ID (Deprecated)

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID.
You might want to do this as part of your CI deployment process.

```go
// ...
err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{
   TaskQueue: "your_task_queue_name",
   Operation: &client.BuildIDOpAddNewIDInNewDefaultSet{
      BuildID: "deadbeef",
   },
})
```

This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue.
New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers.

If, instead, you want to add the Build ID to an existing compatible set, you can do this:

```go
// ...
err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{
   TaskQueue: "your_task_queue_name",
   Operation: &client.BuildIDOpAddNewCompatibleVersion{
      BuildID:                   "deadbeef",
      ExistingCompatibleBuildID: "some-existing-build-id",
   },
})
```

This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set.

You can also promote an existing Build ID in a set to be the default for that set:

```go
// ...
err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{
   TaskQueue: "your_task_queue_name",
   Operation: &client.BuildIDPromoteIDWithinSet{
      BuildID: "some-existing-build-id",
   },
})
```

---

# Workers - Go SDK

Source: https://docs.temporal.io/develop/go/workers

> This section explains how to implement Workers with the Go SDK

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Workers

- [Run a Worker](/develop/go/workers/run-worker-process)
- [Sessions](/develop/go/workers/sessions)
- [Interceptors](/develop/go/workers/interceptors)
- [Serverless Workers](/develop/go/workers/serverless-workers)

---

# Interceptors - Go SDK

Source: https://docs.temporal.io/develop/go/workers/interceptors

Interceptors wrap inbound and outbound Temporal calls so you can apply shared behavior such as tracing, logging, or
authorization. See [Interceptors](/encyclopedia/interceptors) for the inbound vs outbound model.

Register them on the Client (`client.Options.Interceptors`), the Worker (`worker.Options.Interceptors`), or both. Full
API: [`go.temporal.io/sdk/interceptor`](https://pkg.go.dev/go.temporal.io/sdk/interceptor).

## Implement a Client interceptor

Embed `ClientInterceptorBase` on a type that implements `InterceptClient`, and embed
`ClientOutboundInterceptorBase` on a type that overrides the calls you want to intercept. Forward every call you don't
override to `Next` so the rest of the chain still runs.

<!--SNIPSTART go-interceptor-client-->
[go-sandbox/workers/interceptors/interceptor.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/interceptors/interceptor.go)
```go
type loggingClientInterceptor struct {
	interceptor.ClientInterceptorBase
}

func (loggingClientInterceptor) InterceptClient(
	next interceptor.ClientOutboundInterceptor,
) interceptor.ClientOutboundInterceptor {
	return &loggingClientOutboundInterceptor{
		ClientOutboundInterceptorBase: interceptor.ClientOutboundInterceptorBase{Next: next},
	}
}

type loggingClientOutboundInterceptor struct {
	interceptor.ClientOutboundInterceptorBase
}

func (l *loggingClientOutboundInterceptor) ExecuteWorkflow(
	ctx context.Context,
	in *interceptor.ClientExecuteWorkflowInput,
) (client.WorkflowRun, error) {
	log.Printf("starting workflow %q (%s)", in.Options.ID, in.WorkflowType)
	return l.Next.ExecuteWorkflow(ctx, in)
}
```
<!--SNIPEND-->

Register the interceptor when you create the Client:

<!--SNIPSTART go-interceptor-client-register-->
[go-sandbox/workers/interceptors/interceptor.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/interceptors/interceptor.go)
```go
c, err := client.Dial(client.Options{
	Interceptors: []interceptor.ClientInterceptor{&loggingClientInterceptor{}},
})
```
<!--SNIPEND-->

Worker interceptors follow the same pattern for `WorkerInterceptorBase`, `WorkflowInboundInterceptorBase` /
`WorkflowOutboundInterceptorBase`, and `ActivityInboundInterceptorBase` / `ActivityOutboundInterceptorBase`. See the
[Logging Interceptor sample](https://github.com/temporalio/samples-go/tree/main/logger-interceptor) for a complete
Worker interceptor and its registration.

> **⚠️ Warning:**
> Workflow interceptors and replay
>
> Workflow interceptor methods also run during [replay](/develop/go/best-practices/testing-suite#replay). Use replay-safe
> APIs for logging, randomness, and time.
>

## Related

- [Observability - Tracing](/develop/go/platform/observability#tracing)
- [Context propagation](/develop/go/best-practices/context-propagation)
- [Logging Interceptor sample](https://github.com/temporalio/samples-go/tree/main/logger-interceptor)
- [Plugins - Interceptors](/develop/plugins-guide#interceptors)

---

# Run a Worker - Go SDK

Source: https://docs.temporal.io/develop/go/workers/run-worker-process

> Create and run a Temporal Worker using the Go SDK.

This page covers long-lived Workers that you host and run as persistent processes.
For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/go/workers/serverless-workers).

## Create and run a Worker 

Create a [`Worker`](https://pkg.go.dev/go.temporal.io/sdk/worker#Worker) by calling [`worker.New()`](https://pkg.go.dev/go.temporal.io/sdk/worker#New) and passing:

1. A [Temporal Client](/develop/go/client/temporal-client).
2. The name of the Task Queue to poll.
3. A [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) struct (can be empty for defaults).

Register your Workflow and Activity types, then call `Run()` to start polling.
The Worker blocks while it polls, so run it in a separate terminal from your starter code.

<!--SNIPSTART go-worker-main-->
[go-sandbox/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/worker/main.go)
```go
package main

import (
	"log"

	"my-org/quickstart"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, "my-task-queue", worker.Options{})

	w.RegisterWorkflow(quickstart.SayHelloWorkflow)
	w.RegisterActivity(quickstart.Greet)

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```
<!--SNIPEND-->

`Run()` accepts an interrupt channel so the Worker shuts down on `SIGINT` or `SIGTERM`.
You can also call `Start()` and `Stop()` separately for more control over the lifecycle.

> **💡 Tip:**
>
> If you have [`gow`](https://github.com/mitranim/gow) installed, the Worker automatically reloads when you update the file:
>
> ```bash
> go install github.com/mitranim/gow@latest
> gow run worker/main.go
> ```
>

## Register Workflows and Activities 

All Workers polling the same Task Queue must register the same Workflow Types and Activity Types.
A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue.
A Worker that receives a Task for a type it did not register fails that Task.

Use `RegisterWorkflow()` and `RegisterActivity()` to register types.
To register an Activity struct with multiple methods, pass the struct. The Worker gets access to all exported methods.

<!--SNIPSTART go-worker-register-types-->
[go-sandbox/workers/registration/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/registration/main.go)
```go
w.RegisterWorkflow(WorkflowA)
w.RegisterWorkflow(WorkflowB)
w.RegisterActivity(&MyActivities{})
```
<!--SNIPEND-->

To customize the registered name or other options, use `RegisterWorkflowWithOptions()` or `RegisterActivityWithOptions()`.
See [`workflow.RegisterOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#RegisterOptions) and [`activity.RegisterOptions`](https://pkg.go.dev/go.temporal.io/sdk/activity#RegisterOptions).

## Connect to Temporal Cloud 

To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials.
See [Connect to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud) for setup instructions.

## Configure Worker options 

Pass a [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) struct to `worker.New()` to configure concurrency limits, pollers, timeouts, and other Worker behavior.
An empty struct uses defaults that work for most cases.

To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference).

## Run a versioned Worker 

Set a Worker Deployment Version and enable versioning in `worker.Options`, then set a versioning behavior on each Workflow.

<!--SNIPSTART go-worker-versioned-->
[go-sandbox/workers/versioned/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/versioned/main.go)
```go
w := worker.New(c, "my-task-queue", worker.Options{
	DeploymentOptions: worker.DeploymentOptions{
		UseVersioning: true,
		Version: worker.WorkerDeploymentVersion{
			DeploymentName: "my-app",
			BuildID:        "1.0",
		},
	},
})

w.RegisterWorkflowWithOptions(HelloWorkflow, workflow.RegisterOptions{
	VersioningBehavior: workflow.VersioningBehaviorPinned,
})
```
<!--SNIPEND-->

Set the behavior per Workflow with `RegisterWorkflowWithOptions()`, or set a default for every Workflow on the Worker with `DefaultVersioningBehavior` in `worker.DeploymentOptions`.

A versioning behavior applies only to a Worker that has versioning enabled, and setting `DefaultVersioningBehavior` without `UseVersioning` is an error.
With versioning enabled and no default set, a Workflow that does not set its own behavior fails at registration time.

See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out.

## Shut down a Worker 

A Worker started with `Run(worker.InterruptCh())` shuts down when the process receives `SIGINT` or `SIGTERM`.
The Worker stops polling for new Tasks and waits for in-flight Tasks to finish, up to the `WorkerStopTimeout` set in `worker.Options`.

See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities.

---

# Serverless Workers - Go SDK

Source: https://docs.temporal.io/develop/go/workers/serverless-workers

> Write Temporal Workers that run on serverless compute using the Go SDK.

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview.

Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes.
Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done.

For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers).
For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Supported providers

- [**AWS Lambda**](/develop/go/workers/serverless-workers/aws-lambda) - Use the `lambdaworker` package to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, observability, and the invocation lifecycle.
- [**GCP Cloud Run**](/develop/go/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in.

---

# Serverless Workers on AWS Lambda - Go SDK

Source: https://docs.temporal.io/develop/go/workers/serverless-workers/aws-lambda

> Write a Temporal Worker that runs on AWS Lambda using the Go SDK lambdaworker package.

> **Public Preview**

The `lambdaworker` package lets you run a Temporal Serverless Worker on AWS Lambda.
Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive.
Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline.
You register Workflows and Activities the same way you would with a standard Worker.

For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda).

## Create and run a Worker in Lambda 

Use the `RunWorker` function to start a Lambda-based Worker.
Pass a `WorkerDeploymentVersion` and a callback that registers your Workflows and Activities.

<!--SNIPSTART go-serverless-lambda-worker {"selectedLines": ["1-6", "8-18", "22-30"], "highlightedLines": "13-17"}-->
[go-sandbox/workers/lambda/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/lambda/worker/main.go)
```go {13-17}
package main

import (
	greeting "my-org/quickstart/workers/lambda/greeting"

	lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"
// ...
	"go.temporal.io/sdk/worker"
	"go.temporal.io/sdk/workflow"
)

func main() {
	lambdaworker.RunWorker(worker.WorkerDeploymentVersion{
		DeploymentName: "my-app",
		BuildID:        "build-1",
	}, func(opts *lambdaworker.Options) error {
		opts.TaskQueue = "serverless-task-queue-1"

// ...

		opts.RegisterWorkflowWithOptions(greeting.SampleWorkflow, workflow.RegisterOptions{
			VersioningBehavior: workflow.VersioningBehaviorPinned,
		})
		opts.RegisterActivity(greeting.HelloActivity)

		return nil
	})
}
```
<!--SNIPEND-->

The `WorkerDeploymentVersion` is required.
Worker Deployment Versioning is always enabled for Serverless Workers.
Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or `Pinned`.
Set it per-Workflow at registration time, or set a worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`.

The `Options` callback gives you access to the same registration methods you use with a traditional Worker: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, and `RegisterNexusService`.

## Configure the Temporal connection 

The `lambdaworker` package automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details.

Compared with long-lived Workers, the location of the config file is resolved differently, in the following order:

1. `TEMPORAL_CONFIG_FILE` environment variable, if set.
2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`).
3. `temporal.toml` in the current working directory.

The file is optional. If absent, only environment variables are used.

Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options.

## Adjust Worker defaults for Lambda 

The `lambdaworker` package applies conservative defaults suited to short-lived Lambda invocations.
These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment.
Except for `ShutdownDeadlineBuffer`, these are the same [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk@v1.42.0/internal#WorkerOptions) available to any Temporal Worker, just with lower values for Lambda's constrained environment.

| Setting | Lambda default |
|---|---|
| `MaxConcurrentActivityExecutionSize` | 2 |
| `MaxConcurrentWorkflowTaskExecutionSize` | 10 |
| `MaxConcurrentLocalActivityExecutionSize` | 2 |
| `MaxConcurrentNexusTaskExecutionSize` | 5 |
| `MaxConcurrentActivityTaskPollers` | 1 |
| `MaxConcurrentWorkflowTaskPollers` | 2 |
| `MaxConcurrentNexusTaskPollers` | 1 |
| `WorkerStopTimeout` | 5 seconds |
| `DisableEagerActivities` | Always true |
| Sticky cache size | 100 |
| `ShutdownDeadlineBuffer` | 7 seconds |

`DisableEagerActivities` is always true and cannot be overridden.
Eager Activities require a persistent connection, which Lambda invocations don't maintain.

`ShutdownDeadlineBuffer` is specific to the `lambdaworker` package.
It controls how much time before the Lambda deadline the Worker begins its graceful shutdown.
The default is `WorkerStopTimeout` + 2 seconds.

If your Worker handles long-running Activities, increase `WorkerStopTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together.
For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities).

## Add observability with OpenTelemetry 

The `lambdaworker/otel` sub-package provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer.
With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions.
The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch.

The underlying metrics and traces are the same ones the Go SDK emits in any environment.
For general observability concepts and the full list of available metrics, see [Observability - Go SDK](/develop/go/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).

<!--SNIPSTART go-serverless-lambda-worker {"highlightedLines": "7, 19-21"}-->
[go-sandbox/workers/lambda/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/lambda/worker/main.go)
```go {7, 19-21}
package main

import (
	greeting "my-org/quickstart/workers/lambda/greeting"

	lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"
	otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel"
	"go.temporal.io/sdk/worker"
	"go.temporal.io/sdk/workflow"
)

func main() {
	lambdaworker.RunWorker(worker.WorkerDeploymentVersion{
		DeploymentName: "my-app",
		BuildID:        "build-1",
	}, func(opts *lambdaworker.Options) error {
		opts.TaskQueue = "serverless-task-queue-1"

		if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil {
			return err
		}

		opts.RegisterWorkflowWithOptions(greeting.SampleWorkflow, workflow.RegisterOptions{
			VersioningBehavior: workflow.VersioningBehaviorPinned,
		})
		opts.RegisterActivity(greeting.HelloActivity)

		return nil
	})
}
```
<!--SNIPEND-->

`ApplyDefaults` configures both metrics and tracing.
By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint.

To collect this telemetry, attach the [ADOT Collector layer](https://aws-otel.github.io/docs/getting-started/lambda) to your Lambda function.
The layer runs a collector sidecar that receives telemetry on `localhost:4317` and forwards traces to X-Ray and metrics to CloudWatch.
Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary.

The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline.
You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines.
Bundle the following `otel-collector-config.yaml` in your Lambda deployment package:

<!--SNIPSTART go-serverless-lambda-otel-config-->
[go-sandbox/workers/lambda/otel-collector-config.yaml](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/lambda/otel-collector-config.yaml)
```yaml
receivers:
    otlp:
        protocols:
            grpc:
                endpoint: "localhost:4317"
            http:
                endpoint: "localhost:4318"

exporters:
    debug:
    awsxray:
        region: us-west-2
    awsemf:
        # AWS EMF exporter for metrics
        # These are example configurations
        namespace: TemporalWorkerMetrics
        log_group_name: /aws/lambda/<your-function-name>
        region: us-west-2
        dimension_rollup_option: NoDimensionRollup
        resource_to_telemetry_conversion:
            enabled: true

service:
    pipelines:
        traces:
            receivers: [otlp]
            exporters: [awsxray, debug]
        metrics:
            receivers: [otlp]
            exporters: [awsemf]
    telemetry:
        logs:
            level: debug
        metrics:
            address: localhost:8888
```
<!--SNIPEND-->

Set the following environment variable on the Lambda function to point the Collector at the bundled config:

- `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml`

Enable X-Ray active tracing on the Lambda function:

```bash
aws lambda update-function-configuration \
  --function-name <your-function-name> \
  --tracing-config Mode=Active
```

The Lambda execution role must have permissions to write to X-Ray and CloudWatch.
Add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions to the execution role.
Without these permissions, the Collector fails silently and no telemetry appears.

If you only need metrics or tracing, use `otel.ApplyMetrics` or `otel.ApplyTracing` individually.

---

# Serverless Workers on GCP Cloud Run - Go SDK

Source: https://docs.temporal.io/develop/go/workers/serverless-workers/cloud-run

> Run a Temporal Worker on a GCP Cloud Run worker pool using the Go SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Go Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).

## Create a versioned Worker 

Build the Worker as you would any long-running Go Worker, then set `DeploymentOptions` in [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

<!--SNIPSTART go-serverless-cloud-run-versioned-worker-->
[go-sandbox/workers/cloudrun/versioned/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/cloudrun/versioned/main.go)
```go
package main

import (
	"log"
	"os"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
	"go.temporal.io/sdk/worker"
	"go.temporal.io/sdk/workflow"

	"my-org/quickstart/workers/cloudrun/myapp"
)

func main() {
	c, err := client.Dial(envconfig.MustLoadDefaultClientOptions())
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, os.Getenv("TEMPORAL_TASK_QUEUE"), worker.Options{
		DeploymentOptions: worker.DeploymentOptions{
			UseVersioning: true,
			Version: worker.WorkerDeploymentVersion{
				DeploymentName: "my-app",
				BuildID:        "build-1",
			},
		},
	})

	w.RegisterWorkflowWithOptions(myapp.MyWorkflow, workflow.RegisterOptions{
		VersioningBehavior: workflow.VersioningBehaviorPinned,
	})
	w.RegisterActivity(myapp.MyActivity)

	if err := w.Run(worker.InterruptCh()); err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```
<!--SNIPEND-->

`DeploymentName` and `BuildID` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage.

Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `VersioningBehaviorPinned` or `VersioningBehaviorAutoUpgrade`.
Set it per Workflow at registration as shown above, or set `DefaultVersioningBehavior` in `DeploymentOptions` to cover every Workflow on the Worker.
If a Version is set and neither is specified, registration panics with `workflow type does not have a versioning behavior`.

For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/go/workers/run-worker-process).

## Configure the Temporal connection 

The `envconfig` package loads [Temporal Client](/develop/go/client/temporal-client) configuration from environment variables and an optional TOML config file, so the Worker code carries no Namespace or credentials.
Set the non-secret values as environment variables on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager.
For the full list of supported variables, the config file format, and profiles, see [Environment configuration](/develop/environment-configuration).

`MustLoadDefaultClientOptions` panics if the configuration is invalid. To handle a bad configuration yourself, use `envconfig.LoadDefaultClientOptions` and check the returned error.

## Keep Activities safe across scale-in 

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing.
An instance running a long Activity can be stopped mid-execution.

Use [Activity Heartbeats](/develop/go/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over:

<!--SNIPSTART go-serverless-cloud-run-heartbeat-->
[go-sandbox/workers/cloudrun/myapp/myapp.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/cloudrun/myapp/myapp.go)
```go
func MyActivity(ctx context.Context, input MyInput) (string, error) {
	// Resume after the last item a previous attempt recorded.
	start := 0
	if activity.HasHeartbeatDetails(ctx) {
		var lastIndex int
		if err := activity.GetHeartbeatDetails(ctx, &lastIndex); err == nil {
			start = lastIndex + 1
		}
	}
	for i := start; i < len(input.Items); i++ {
		// ... process input.Items[i]
		activity.RecordHeartbeat(ctx, i)
	}
	return "done", nil
}
```
<!--SNIPEND-->

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Configure OpenTelemetry 

Run an OpenTelemetry Collector as a sidecar in the Worker Pool. The Cloud Run OpenTelemetry plugin exports metrics and traces by OTLP gRPC to the Collector at `localhost:4317`. By default, it derives the service name from `OTEL_SERVICE_NAME`, `CLOUD_RUN_WORKER_POOL`, or `K_SERVICE`.

Create both Cloud Run plugins before connecting, then add them to the Client options. Client plugins that implement
`worker.Plugin` also apply to Workers created from that Client:

<!--SNIPSTART go-serverless-cloud-run-plugins {"selectedLines": ["24-43"]}-->
[go-sandbox/workers/cloudrun/plugins/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/cloudrun/plugins/main.go)
```go
// ...
	// Exports OTLP metrics and traces to the collector sidecar (defaults to localhost:4317).
	otelPlugin, err := otel.NewPlugin(ctx, otel.PluginOptions{})
	if err != nil {
		log.Fatalln("Unable to create OpenTelemetry plugin", err)
	}
	// Sets the client identity to "<instanceID>@<revision>" from Cloud Run instance metadata.
	idPlugin := id.NewCloudRunIDPlugin()

	// A client plugin that also implements worker.Plugin is applied to workers automatically.
	clientOptions, err := envconfig.LoadDefaultClientOptions()
	if err != nil {
		log.Fatalln("Unable to load Temporal client options", err)
	}
	clientOptions.Plugins = append(clientOptions.Plugins, otelPlugin, idPlugin)

	c, err := client.Dial(clientOptions)
	if err != nil {
		log.Fatalln("Unable to create Temporal client", err)
	}
	log.Println("Client identity:", idPlugin.Metadata().Identity())
```
<!--SNIPEND-->

Configure the Collector sidecar to receive OTLP gRPC on `localhost:4317`, export traces to Google Cloud, and export metrics to Google Managed Service for Prometheus:

<!--SNIPSTART go-serverless-cloud-run-otel-config-->
[go-sandbox/workers/cloudrun/otel-collector-config.yaml](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/cloudrun/otel-collector-config.yaml)
```yaml
# Google-Built OpenTelemetry Collector sidecar: metrics -> Managed Service for Prometheus, traces -> Cloud Trace.
# Auth uses the worker-pool service account (ADC).
receivers:
    otlp:
        protocols:
            grpc:
                endpoint: localhost:4317

processors:
    memory_limiter:
        check_interval: 1s
        limit_percentage: 65
        spike_limit_percentage: 20
    resource_detection:
        detectors: [gcp]
        timeout: 10s
    # Batch traces for throughput. Do NOT batch the cumulative-metrics pipeline: a shutdown flush
    # could be merged with a recent periodic export of the same series and rejected as a duplicate.
    batch/traces:
        send_batch_size: 200
        timeout: 5s
    # Rename Temporal datapoint labels that collide with the target labels Managed Service for Prometheus injects (e.g. namespace).
    transform/collision:
        metric_statements:
            - context: datapoint
              statements:
                  - set(attributes["exported_location"], attributes["location"])
                  - delete_key(attributes, "location")
                  - set(attributes["exported_cluster"], attributes["cluster"])
                  - delete_key(attributes, "cluster")
                  - set(attributes["exported_namespace"], attributes["namespace"])
                  - delete_key(attributes, "namespace")
                  - set(attributes["exported_job"], attributes["job"])
                  - delete_key(attributes, "job")
                  - set(attributes["exported_instance"], attributes["instance"])
                  - delete_key(attributes, "instance")
                  - set(attributes["exported_project_id"], attributes["project_id"])
                  - delete_key(attributes, "project_id")
    # The Telemetry API expects the Google Cloud project in gcp.project_id.
    transform/set_project_id:
        error_mode: ignore
        trace_statements:
            - set(resource.attributes["gcp.project_id"], resource.attributes["gcp.project.id"]) where resource.attributes["gcp.project.id"] != nil
            - set(resource.attributes["gcp.project_id"], resource.attributes["cloud.account.id"]) where resource.attributes["gcp.project_id"] == nil and resource.attributes["cloud.account.id"] != nil

exporters:
    googlemanagedprometheus:
    otlp_grpc:
        endpoint: telemetry.googleapis.com:443
        compression: none
        balancer_name: pick_first
        auth:
            authenticator: googleclientauth

extensions:
    googleclientauth:
    health_check:
        endpoint: 0.0.0.0:13133

service:
    extensions: [googleclientauth, health_check]
    pipelines:
        metrics:
            receivers: [otlp]
            processors: [memory_limiter, resource_detection, transform/collision]
            exporters: [googlemanagedprometheus]
        traces:
            receivers: [otlp]
            processors: [memory_limiter, resource_detection, transform/set_project_id, batch/traces]
            exporters: [otlp_grpc]
    telemetry:
        logs:
            level: info
```
<!--SNIPEND-->

On shutdown, stop the Worker and call `otelPlugin.Shutdown` with a deadline shorter than Cloud Run's termination window so telemetry can flush. For a Collector configuration that exports traces to Google Cloud and metrics to Google Managed Service for Prometheus, see the [Go Cloud Run sample](https://github.com/temporalio/samples-go/tree/main/gcp/cloudrun).

## Set a Worker identity 

Use the Cloud Run ID plugin to identify each Worker instance as `<instance-id>@<revision>`. The plugin reads Cloud Run environment variables and instance metadata once when the Client connects, then Workers created from that Client inherit the identity.

Register the plugin on the Client options before connecting:

<!--SNIPSTART go-serverless-cloud-run-identity-->
[go-sandbox/workers/cloudrun/identity/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/cloudrun/identity/main.go)
```go
package main

import (
	"log"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
	"go.temporal.io/sdk/contrib/gcp/cloudrun/id"
)

func main() {
	clientOptions, err := envconfig.LoadDefaultClientOptions()
	if err != nil {
		log.Fatalln("Unable to load Temporal client options", err)
	}
	clientOptions.Plugins = append(clientOptions.Plugins, id.NewCloudRunIDPlugin())
	c, err := client.Dial(clientOptions)
	if err != nil {
		log.Fatalln("Unable to create Temporal client", err)
	}
	defer c.Close()

	// Workers created from this Client use the Cloud Run identity.
	runWorker(c)
}
```
<!--SNIPEND-->

The [preceding example](#opentelemetry) registers `id.NewCloudRunIDPlugin()` alongside the OpenTelemetry plugin.

The plugin requires the Cloud Run metadata server, so it fails when the Worker runs outside Cloud Run. For a complete example, see the [Go Cloud Run sample](https://github.com/temporalio/samples-go/tree/main/gcp/cloudrun).

---

# Worker Sessions - Go SDK

Source: https://docs.temporal.io/develop/go/workers/sessions

> Enable Worker Sessions, change concurrent Sessions, and create Session Workers using the Go SDK for precise Task Routing, ensuring efficient Activity Tasks execution.

This page shows how to do the following:

- [Enable Worker Sessions](#enable-sessions)
- [Change the maximum concurrent Sessions of a Worker](#max-concurrent-sessions)
- [Create a Worker Session](#create-a-session)

> **💡 Tip:**
>
> This feature is currently available only in the Go SDK.
>

A Worker Session is a feature that provides a straightforward API for [Task Routing](/task-routing) to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names.

## Enable Worker Sessions 

Set `EnableSessionWorker` to `true` in the Worker options.

<!--SNIPSTART go-session-worker {"highlightedLines": "12,15-19"}-->
[go-sandbox/workers/sessions/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/sessions/worker/main.go)
```go {12,15-19}
func main() {
	// The client and worker are heavyweight objects that should be created once per process.
	temporalClient, err := client.Dial(client.Options{
		HostPort: client.DefaultHostPort,
	})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer temporalClient.Close()

	workerOptions := worker.Options{
		EnableSessionWorker:               true,
		MaxConcurrentSessionExecutionSize: 1000,
	}
	w := worker.New(temporalClient, "fileprocessing", workerOptions)
	w.RegisterWorkflow(sessions.SomeFileProcessingWorkflow)
	w.RegisterActivity(&sessions.FileActivities{})

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```
<!--SNIPEND-->

### Change the maximum concurrent Sessions of a Worker 

You can adjust the maximum concurrent Sessions of a Worker.

To limit the number of concurrent Sessions running on a Worker, set the `MaxConcurrentSessionExecutionSize` field of `worker.Options` to the desired value.
By default, this field is set to a very large value, so there's no need to manually set it if no limitation is needed.

If a Worker hits this limitation, it won't accept any new `CreateSession()` requests until one of the existing sessions is completed.
If the session can't be created within `CreationTimeout`, `CreateSession()` returns an error.

<!--SNIPSTART go-session-worker {"highlightedLines": "5", "selectedLines": ["1", "11-14", "23"]}-->
[go-sandbox/workers/sessions/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/sessions/worker/main.go)
```go {5}
func main() {
// ...
	workerOptions := worker.Options{
		EnableSessionWorker:               true,
		MaxConcurrentSessionExecutionSize: 1000,
	}
// ...
}
```
<!--SNIPEND-->

## Create a Worker Session 

Within the Workflow code use the Workflow APIs to create a Session with whichever Worker picks up the first Activity Task.

Use the [`CreateSession`](https://pkg.go.dev/go.temporal.io/sdk/workflow#CreateSession) API to create a Context object that can be passed to calls to spawn Activity Executions.

Pass an instance of `workflow.Context` and [`SessionOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SessionOptions) to the `CreateSession` API call and get a Session Context that contains metadata information of the Session.

Use the Session Context to spawn all Activity Executions that should belong to the Session.
All associated Activity Tasks are then processed by the same Worker Entity.
When the `CreateSession` API is called, the Task Queue name that is specified in `ActivityOptions` (or in `StartWorkflowOptions` if the Task Queue name is not specified in `ActivityOptions`) is used, and a Session is created with one of the Workers polling that Task Queue.

The Session Context is cancelled if the Worker executing this Session dies or `CompleteSession()` is called.
When using the returned Session Context to spawn Activity Executions, a `workflow.ErrSessionFailed` error is returned if the Session framework detects that the Worker executing this Session has died.
The failure of Activity Executions won't affect the state of the Session, so you still need to handle the errors returned from your Activities and call `CompleteSession()` if necessary.

If the context passed in already contains an open Session, `CreateSession()` returns an error.
If all the Workers are currently busy and unable to handle a new Session, the framework keeps retrying until the `CreationTimeout` period you specified in `SessionOptions` has passed before returning an error.
(For more details, see [Change the maximum concurrent Sessions of a Worker](#max-concurrent-sessions).)

`CompleteSession()` releases the resources reserved on the Worker, so it's important to call it as soon as you no longer need the Session.
It cancels the session context and therefore all the Activity Executions using that Session Context.
It is safe to call `CompleteSession()` on a failed Session, meaning that you can call it from a `defer` function after the Session is successfully created.

If the Worker goes down between Activities, any scheduled Activities meant for the Session Worker are canceled.
If not, you get a `workflow.ErrSessionFailed` error when the next call of `workflow.ExecuteActivity()` is made from that Workflow.

<!--SNIPSTART go-session-workflow {"highlightedLines": "13-17,23-28,37,44,50"}-->
[go-sandbox/workers/sessions/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/sessions/workflow.go)
```go {13-17,23-28,37,44,50}
package sessions

import (
	"time"

	"go.temporal.io/sdk/workflow"
)

type FileProcessingWFParam struct {
	CloudFileLocation string
}

func SomeFileProcessingWorkflow(ctx workflow.Context, param FileProcessingWFParam) error {
	activityOptions := workflow.ActivityOptions{
		StartToCloseTimeout: time.Minute,
	}
	ctx = workflow.WithActivityOptions(ctx, activityOptions)
	err := processFile(ctx, param)
	return err
}

func processFile(ctx workflow.Context, param FileProcessingWFParam) error {
	sessionOptions := &workflow.SessionOptions{
		CreationTimeout:  time.Minute,
		ExecutionTimeout: time.Minute,
	}

	sessionCtx, err := workflow.CreateSession(ctx, sessionOptions)
	if err != nil {
		return err
	}
	defer workflow.CompleteSession(sessionCtx)

	var a *FileActivities
	var downloadResult FileActivityResult

	err = workflow.ExecuteActivity(sessionCtx, a.DownloadFile, param).Get(sessionCtx, &downloadResult)
	if err != nil {
		return err
	}

	processParam := FileActivityParam(downloadResult)
	var processResult FileActivityResult
	err = workflow.ExecuteActivity(sessionCtx, a.ProcessFile, processParam).Get(sessionCtx, &processResult)
	if err != nil {
		return err
	}

	uploadParam := FileActivityParam(processResult)
	err = workflow.ExecuteActivity(sessionCtx, a.UploadFile, uploadParam).Get(sessionCtx, nil)
	return err
}
```
<!--SNIPEND-->

## Additional Session usage information 

```go
type SessionInfo struct {
 // A unique ID for the session
 SessionID         string
 // The hostname of the worker that is executing the session
 HostName          string
 // ... other unexported fields
}

func GetSessionInfo(ctx Context) *SessionInfo
```

The Session Context also stores some Session metadata, which can be retrieved by the `GetSessionInfo()` API.
If the Context passed in doesn't contain any Session metadata, this API will return a `nil` pointer.

### Recreate Session

For long-running Sessions, you may want to use the `ContinueAsNew` feature to split the Workflow into multiple runs when all Activities need to be executed by the same Worker.
The `RecreateSession()` API is designed for such a use case.

```go
func RecreateSession(ctx Context, recreateToken []byte, sessionOptions *SessionOptions) (Context, error)
```

Its usage is the same as `CreateSession()` except that it also takes in a `recreateToken`, which is needed to create a new Session on the same Worker as the previous one.
You can get the token by calling the `GetRecreateToken()` method of the `SessionInfo` object.

<!--SNIPSTART go-session-recreate-token-->
[go-sandbox/workers/sessions/recreate.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workers/sessions/recreate.go)
```go
token := workflow.GetSessionInfo(sessionCtx).GetRecreateToken()
```
<!--SNIPEND-->

**Is there a complete example?**

Yes, the [file processing example](https://github.com/temporalio/samples-go/tree/main/fileprocessing) in the [temporalio/samples-go](https://github.com/temporalio/samples-go) repo has been updated to use the session framework.

**What happens to my Activity if the Worker dies?**

If your Activity has already been scheduled, it will be canceled.
If not, you will get a `workflow.ErrSessionFailed` error when you call `workflow.ExecuteActivity()`.

**Is the concurrent session limitation per process or per host?**

It's per Worker Process, so make sure there's only one Worker Process running on the host if you plan to use this feature.

**Future Work**

- Right now, a Session is considered failed if the Worker Process dies.
  However, for some use cases, you may only care whether the Worker host is alive or not.
  For these use cases, the Session should be automatically re-established if the Worker Process is restarted.

- The current implementation assumes that all Sessions are consuming the same type of resource and there's only one global limitation.
  Our plan is to allow you to specify what type of resource your Session will consume and enforce different limitations on different types of resources.

---

# Workflows - Go SDK

Source: https://docs.temporal.io/develop/go/workflows

> This section explains how to implement Workflows with the Go SDK

![Go SDK Banner](https://docs.temporal.io/img/assets/banner-go-temporal.png)

## Workflows

- [Workflow basics](/develop/go/workflows/basics)
- [Child Workflows](/develop/go/workflows/child-workflows)
- [Continue-As-New](/develop/go/workflows/continue-as-new)
- [Cancellation](/develop/go/workflows/cancellation)
- [Timeouts](/develop/go/workflows/timeouts)
- [Message passing](/develop/go/workflows/message-passing)
- [Selectors](/develop/go/workflows/selectors)
- [Side effects](/develop/go/workflows/side-effects)
- [Schedules](/develop/go/workflows/schedules)
- [Timers](/develop/go/workflows/timers)
- [Dynamic Workflow](/develop/go/workflows/dynamic-workflow)
- [Versioning](/develop/go/workflows/versioning)
- [Workflow Streams](/develop/go/workflows/workflow-streams)

---

# Workflow basics - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/basics

> This section explains Workflow basics with the Go SDK

## Develop a basic Workflow 

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal Go SDK programming model, a [Workflow Definition](/workflow-definition) is an exportable function.
Below is an example of a basic Workflow Definition.

<!--SNIPSTART go-workflows-basics-simple-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go
func YourSimpleWorkflowDefinition(ctx workflow.Context) error {
	// ...
	return nil
}
```
<!--SNIPEND-->

### Define Workflow parameters 

Temporal Workflows may have any number of custom parameters.
However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow.
All Workflow Definition parameters must be serializable.
For more reasons, see [Workflow parameters](/workflow-definition#workflow-parameters).

The first parameter of a Go-based Workflow Definition must be of the [`workflow.Context`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Context) type.
It is used by the Temporal Go SDK to pass around Workflow Execution context, and virtually all the Go SDK APIs that are callable from the Workflow require it.
It is acquired from the [`go.temporal.io/sdk/workflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow) package.

The `workflow.Context` entity operates similarly to the standard `context.Context` entity provided by Go.
The only difference between `workflow.Context` and `context.Context` is that the `Done()` function, provided by `workflow.Context`, returns `workflow.Channel` instead of the standard Go `chan`.

Additional parameters can be passed to the Workflow when it is invoked.
A Workflow Definition may support multiple custom parameters, or none.
These parameters can be regular type variables or safe pointers.
However, the best practice is to pass a single parameter that is of a `struct` type, so there can be some backward compatibility if new parameters are added.

All Workflow Definition parameters must be serializable and can't be channels, functions, variadic, or unsafe pointers.

<!--SNIPSTART go-activities-execution-workflow {"highlightedLines": "2-5,7", "selectedLines": ["10-13", "20", "38-42", "66"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go {2-5,7}
// ...
type YourWorkflowParam struct {
	WorkflowParamX string
	WorkflowParamY int
}
// ...
func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) {
// ...
	ctx = workflow.WithActivityOptions(ctx, activityOptions)
	activityParam := YourActivityParam{
		ActivityParamX: param.WorkflowParamX,
		ActivityParamY: param.WorkflowParamY,
	}
// ...
}
```
<!--SNIPEND-->

### Define Workflow return parameters 

Workflow return values must also be serializable.
Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported.
However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error.

A Go-based Workflow Definition can return either just an `error` or a `customValue, error` combination.
Again, the best practice here is to use a `struct` type to hold all custom values.
A Workflow Definition written in Go can return both a custom value and an error.
However, it's not possible to receive both a custom value and an error in the calling process, as is normal in Go.
The caller will receive either one or the other.
Returning a non-nil `error` from a Workflow indicates that an error was encountered during its execution and the Workflow Execution should be terminated, and any custom return values will be ignored by the system.

<!--SNIPSTART go-activities-execution-workflow {"highlightedLines": "2-5,9-13", "selectedLines": ["15-18", "20", "61-66"]}-->
[go-sandbox/activities/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/basics/workflow.go)
```go {2-5,9-13}
// ...
type YourWorkflowResultObject struct {
	WFResultFieldX string
	WFResultFieldY int
}
// ...
func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) {
// ...
	workflowResult := &YourWorkflowResultObject{
		WFResultFieldX: activityResult.ResultFieldX,
		WFResultFieldY: activityResult.ResultFieldY,
	}
	return workflowResult, nil
}
```
<!--SNIPEND-->

### Customize Workflow Type 

In Go, by default, the Workflow Type name is the same as the function name.

To customize the Workflow Type, set the `Name` parameter with `RegisterOptions` when registering your Workflow with a Worker.

<!--SNIPSTART go-activities-worker {"highlightedLines": "22-26"}-->
[go-sandbox/activities/workers/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/activities/workers/main.go)
```go {22-26}
package main

import (
	"log"

	"go.temporal.io/sdk/activity"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
	"go.temporal.io/sdk/workflow"

	"my-org/quickstart/activities/basics"
)

func main() {
	temporalClient, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer temporalClient.Close()

	yourWorker := worker.New(temporalClient, "your-custom-task-queue-name", worker.Options{})
	yourWorker.RegisterWorkflow(basics.YourWorkflowDefinition)
	registerWFOptions := workflow.RegisterOptions{
		Name: "JustAnotherWorkflow",
	}
	yourWorker.RegisterWorkflowWithOptions(basics.YourSimpleWorkflowDefinition, registerWFOptions)

	message := "This could be a connection string or endpoint details"
	number := 100
	activities := &basics.YourActivityObject{
		Message: &message,
		Number:  &number,
	}
	yourWorker.RegisterActivity(activities)

	registerAOptions := activity.RegisterOptions{
		Name: "JustAnotherActivity",
	}
	yourWorker.RegisterActivityWithOptions(basics.YourSimpleActivityDefinition, registerAOptions)

	err = yourWorker.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start Worker", err)
	}
}
```
<!--SNIPEND-->

### Develop Workflow logic 

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with application code outside the Workflow.

In Go, Workflow Definition code cannot directly do the following:

- Iterate over maps using `range`, because with `range` the order of the map's iteration is randomized.
  Instead you can collect the keys of the map, sort them, and then iterate over the sorted keys to access the map.
  This technique provides deterministic results.
  You can also use a Side Effect or an Activity to process the map instead.
- Call an external API, conduct a file I/O operation, talk to another service, and so on. (Use an Activity for these.)

The Temporal Go SDK has APIs to handle equivalent Go constructs:

- `workflow.Now()` This is a replacement for `time.Now()`.
- `workflow.Sleep()` This is a replacement for `time.Sleep()`.
- `workflow.GetLogger()` This ensures that the provided logger does not duplicate logs during a replay.
- `workflow.Go()` This is a replacement for the `go` statement.
- `workflow.Channel` This is a replacement for the native `chan` type.
  Temporal provides support for both buffered and unbuffered channels.
- `workflow.Selector` This is a replacement for the `select` statement.
  Learn more on the [Go SDK Selectors](https://legacy-documentation-sdks.temporal.io/go/selectors) page.
- `workflow.Context` This is a replacement for `context.Context`.
  See [Tracing](/develop/go/platform/observability#tracing) for more information about context propagation.

#### Logging

Use [`workflow.GetLogger(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetLogger) instead of the standard
`log` package or `fmt.Println`. The SDK logger skips log messages during replay to avoid duplicates:

<!--SNIPSTART go-workflows-basics-logging-->
[go-sandbox/workflows/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/basics/workflow.go)
```go
func MyWorkflow(ctx workflow.Context, name string) (string, error) {
	logger := workflow.GetLogger(ctx)
	logger.Info("Starting workflow", "name", name)
	// ...
	return "Hello " + name, nil
}
```
<!--SNIPEND-->

For logger configuration, see [Observability: Log from a Workflow](/develop/go/platform/observability#logging).

#### Random numbers and UUIDs

The Go SDK does not provide a seeded random source or a UUID helper. Generate these inside a
[Side Effect](/develop/go/workflows/side-effects), which records the result in the Event History and returns the
recorded value on replay:

<!--SNIPSTART go-workflows-basics-side-effect-uuid-->
[go-sandbox/workflows/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/basics/workflow.go)
```go
var id string
encodedID := workflow.SideEffect(ctx, func(ctx workflow.Context) interface{} {
	return uuid.New().String()
})
if err := encodedID.Get(&id); err != nil {
	return nil, err
}
```
<!--SNIPEND-->

An Activity works for this too, and is the better choice when the value comes from an external system. A Side Effect is
cheaper for purely local generation.

#### Current time

Use [`workflow.Now(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Now) instead of `time.Now()`. It returns the
time of the last Workflow Task, which is consistent across replays:

<!--SNIPSTART go-workflows-basics-now-->
[go-sandbox/workflows/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/basics/workflow.go)
```go
currentTime := workflow.Now(ctx)
```
<!--SNIPEND-->

To wait, use [`workflow.Sleep(ctx, d)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) instead of `time.Sleep`.

#### Detecting replay (advanced)

Use [`workflow.IsReplaying(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#IsReplaying) to guard code that should
only run on the first execution, such as emitting metrics or sending external notifications from an Interceptor.

> **⚠️ Caution:**
>
> Never use this to affect Workflow business logic. Branching on replay status breaks determinism.
>

<!--SNIPSTART go-workflows-basics-is-replaying-->
[go-sandbox/workflows/basics/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/basics/workflow.go)
```go
if !workflow.IsReplaying(ctx) {
	emitMetric("workflow_started", 1)
}
```
<!--SNIPEND-->

---

# Cancel a Workflow - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/cancellation

This page shows the following:

- How to handle a Cancellation request within a Workflow.
- How to set an Activity Heartbeat Timeout.
- How to listen for and handle a Cancellation request within an Activity.
- How to send a Cancellation request from a Temporal Client.
- Heartbeating after a Cancellation.

## Handle Cancellation in Workflow 

Workflow Definitions can be written to handle execution cancellation requests with Go's `defer` and the
`workflow.NewDisconnectedContext` API. In the Workflow Definition, there is a special Activity that handles clean up
should the execution be cancelled.

If the Workflow receives a Cancellation Request, but all Activities gracefully handle the Cancellation, and/or no
Activities are skipped then the Workflow status will be Complete. It is completely up to the needs of the business
process and your use case which determines whether you want to return the Cancellation error to show a Canceled status
or Complete status regardless of whether a Cancellation has propagated to and/or skipped Activities.

<!--SNIPSTART go-cancellation-workflow {"highlightedLines": "7-24,29,32,35"}-->
[go-sandbox/workflows/cancellation/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/cancellation/workflow.go)
```go {7-24,29,32,35}
const WorkflowID = "example-cancellation-workflow"
const TaskQueueName = "cancellation"

func YourWorkflow(ctx workflow.Context) error {
	logger := workflow.GetLogger(ctx)
	var a *Activities
	activityOptions := workflow.ActivityOptions{
		StartToCloseTimeout: 30 * time.Minute,
		HeartbeatTimeout:    5 * time.Second,
		WaitForCancellation: true,
	}
	defer func() {
		if !errors.Is(ctx.Err(), workflow.ErrCanceled) {
			return
		}

		// A disconnected Context isn't canceled, so the cleanup Activity can still run.
		newCtx, _ := workflow.NewDisconnectedContext(ctx)

		err := workflow.ExecuteActivity(newCtx, a.CleanupActivity).Get(newCtx, nil)
		if err != nil {
			logger.Error("CleanupActivity failed", "Error", err)
		}
	}()

	ctx = workflow.WithActivityOptions(ctx, activityOptions)
	var result string

	err := workflow.ExecuteActivity(ctx, a.ActivityToBeCanceled).Get(ctx, &result)
	logger.Info(fmt.Sprintf("ActivityToBeCanceled returns %v, %v", result, err))

	err = workflow.ExecuteActivity(ctx, a.ActivityToBeSkipped).Get(ctx, nil)
	logger.Error("Error from ActivityToBeSkipped", "Error", err)

	return err
}
```
<!--SNIPEND-->

## Handle Cancellation in an Activity 

Ensure that the Activity is Heartbeating to receive the Cancellation request and stop execution.

<!--SNIPSTART go-cancellation-activity-->
[go-sandbox/workflows/cancellation/activities.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/cancellation/activities.go)
```go
func (a *Activities) ActivityToBeCanceled(ctx context.Context) (string, error) {
	logger := activity.GetLogger(ctx)
	logger.Info("Activity started, to cancel the Workflow Execution and this Activity, use 'go run ./workflows/cancellation/cancel' " +
		"or use the CLI: 'temporal workflow cancel --workflow-id <WorkflowID>'")
	// A for select statement is a common approach to listening for a Cancellation in an Activity
	for {
		select {
		case <-time.After(1 * time.Second):
			logger.Info("Heartbeating...")
			activity.RecordHeartbeat(ctx, "")
		// Listen for ctx.Done() to know if a Cancellation Request has propagated to the Activity.
		case <-ctx.Done():
			logger.Info("This Activity is canceled!")
			return "I am canceled by Done", nil
		}
	}
}
```
<!--SNIPEND-->

## Request Cancellation 

Use the `CancelWorkflow` API to cancel a Workflow Execution using its Workflow ID.

<!--SNIPSTART go-cancellation-request-->
[go-sandbox/workflows/cancellation/cancel/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/cancellation/cancel/main.go)
```go
func main() {
	temporalClient, err := client.Dial(client.Options{
		HostPort: client.DefaultHostPort,
	})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer temporalClient.Close()
	// Call the CancelWorkflow API to cancel a Workflow
	// In this call we are relying on the Workflow ID only.
	// But a Run ID can also be supplied to ensure the correct Workflow is Canceled.
	err = temporalClient.CancelWorkflow(context.Background(), cancellation.WorkflowID, "")
	if err != nil {
		log.Fatalln("Unable to cancel Workflow Execution", err)
	}
	log.Println("Workflow Execution cancelled", "WorkflowID", cancellation.WorkflowID)
}
```
<!--SNIPEND-->

## Heartbeating after a Cancellation

Sometimes you may want to continue running your Activity, even after a Cancellation has been issued. You may want to
completely ignore the cancellation and continue Activity execution, including Heartbeating, or you may want to send one
final Heartbeat after Cancellation. Even though the context is cancelled when the Workflow is Cancelled, you are still
able to send Activity Heartbeats.

When you call `activity.RecordHeartbeat` after Cancellation has occurred, a
`WARN RecordActivityHeartbeat with error Error context canceled` message will be logged, and a `context canceled` error
will be returned from the call. However, the Heartbeat **has** still been sent.

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Child Workflows - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/child-workflows

> Use the Go SDK to start a Child Workflow Execution and set a Parent Close Policy, including details on Workflow Options and future management.

This page shows how to do the following:

- [Start a Child Workflow Execution](#child-workflows)
- [Set a Parent Close Policy](#parent-close-policy)

## Start a Child Workflow Execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In Go, you must explicitly call `GetChildWorkflowExecution()` on the `ChildWorkflowFuture` and then call `Get()` on the returned Future to wait for this Event.
See the [Async Child Workflows](#async-child-workflows) section below for a complete example.

To spawn a [Child Workflow Execution](/child-workflows) in Go, use the [`ExecuteChildWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ExecuteChildWorkflow) API, which is available from the `go.temporal.io/sdk/workflow` package.

The `ExecuteChildWorkflow` call requires an instance of [`workflow.Context`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Context), with an instance of [`workflow.ChildWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ChildWorkflowOptions) applied to it, the Workflow Type, and any parameters that should be passed to the Child Workflow Execution.

`workflow.ChildWorkflowOptions` contain the same fields as `client.StartWorkflowOptions`.
Workflow Option fields automatically inherit their values from the Parent Workflow Options if they are not explicitly set.
If a custom `WorkflowID` is not set, one is generated when the Child Workflow Execution is spawned.
Use the [`WithChildOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#WithChildOptions) API to apply Child Workflow Options to the instance of `workflow.Context`.

The `ExecuteChildWorkflow` call returns an instance of a [`ChildWorkflowFuture`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ChildWorkflowFuture).

Call the `.Get()` method on the instance of `ChildWorkflowFuture` to wait for the result.

<!--SNIPSTART go-child-workflow-execute-->
[go-sandbox/workflows/childworkflows/parent.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/childworkflows/parent.go)
```go
func YourWorkflowDefinition(ctx workflow.Context, params ParentParams) (ParentResp, error) {
	childWorkflowOptions := workflow.ChildWorkflowOptions{}
	ctx = workflow.WithChildOptions(ctx, childWorkflowOptions)

	var result ChildResp
	err := workflow.ExecuteChildWorkflow(ctx, YourOtherWorkflowDefinition, ChildParams{Name: params.Name}).Get(ctx, &result)
	if err != nil {
		return ParentResp{}, err
	}
	// ...
	return ParentResp{Greeting: result.Greeting}, nil
}
```
<!--SNIPEND-->

The Child Workflow Definition is a regular Workflow Definition:

<!--SNIPSTART go-child-workflow-definition-->
[go-sandbox/workflows/childworkflows/child.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/childworkflows/child.go)
```go
func YourOtherWorkflowDefinition(ctx workflow.Context, params ChildParams) (ChildResp, error) {
	// ...
	return ChildResp{Greeting: "Hello " + params.Name}, nil
}
```
<!--SNIPEND-->

### Async Child Workflows

To asynchronously spawn a Child Workflow Execution, the Child Workflow must have an "Abandon" Parent Close Policy set in the Child Workflow Options.
Additionally, the Parent Workflow Execution must wait for the `ChildWorkflowExecutionStarted` Event to appear in its Event History before it completes.

If the Parent makes the `ExecuteChildWorkflow` call and then immediately completes, the Child Workflow Execution does not spawn.

To be sure that the Child Workflow Execution has started, first call the `GetChildWorkflowExecution` method on the instance of the `ChildWorkflowFuture`, which will return a different Future.
Then call the `Get()` method on that Future, which is what will wait until the Child Workflow Execution has spawned.

<!--SNIPSTART go-child-workflow-async-->
[go-sandbox/workflows/childworkflows/async.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/childworkflows/async.go)
```go
import (
	"go.temporal.io/api/enums/v1"
	"go.temporal.io/sdk/workflow"
)

func YourAsyncWorkflowDefinition(ctx workflow.Context, params ParentParams) (ParentResp, error) {
	childWorkflowOptions := workflow.ChildWorkflowOptions{
		ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON,
	}
	ctx = workflow.WithChildOptions(ctx, childWorkflowOptions)

	childWorkflowFuture := workflow.ExecuteChildWorkflow(ctx, YourOtherWorkflowDefinition, ChildParams{Name: params.Name})
	// Wait for the Child Workflow Execution to spawn
	var childWE workflow.Execution
	if err := childWorkflowFuture.GetChildWorkflowExecution().Get(ctx, &childWE); err != nil {
		return ParentResp{}, err
	}
	// ...
	return ParentResp{ChildWorkflowID: childWE.ID}, nil
}
```
<!--SNIPEND-->

#### Set a Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

In Go, a Parent Close Policy is set on the `ParentClosePolicy` field of an instance of [`workflow.ChildWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ChildWorkflowOptions).
The possible values can be obtained from the [`go.temporal.io/api/enums/v1`](https://pkg.go.dev/go.temporal.io/api/enums/v1#ParentClosePolicy) package.

- `PARENT_CLOSE_POLICY_ABANDON`
- `PARENT_CLOSE_POLICY_TERMINATE`
- `PARENT_CLOSE_POLICY_REQUEST_CANCEL`

The Child Workflow Options are then applied to the instance of `workflow.Context` by using the `WithChildOptions` API, which is then passed to the `ExecuteChildWorkflow()` call.

- Type: [`ParentClosePolicy`](https://pkg.go.dev/go.temporal.io/api/enums/v1#ParentClosePolicy)
- Default: `PARENT_CLOSE_POLICY_TERMINATE`

<!--SNIPSTART go-child-workflow-parent-close-policy-->
[go-sandbox/workflows/childworkflows/parentclose.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/childworkflows/parentclose.go)
```go
import (
	"go.temporal.io/api/enums/v1"
	"go.temporal.io/sdk/workflow"
)

func YourParentClosePolicyWorkflowDefinition(ctx workflow.Context, params ParentParams) (ParentResp, error) {
	// ...
	childWorkflowOptions := workflow.ChildWorkflowOptions{
		// ...
		ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON,
	}
	ctx = workflow.WithChildOptions(ctx, childWorkflowOptions)
	childWorkflowFuture := workflow.ExecuteChildWorkflow(ctx, YourOtherWorkflowDefinition, ChildParams{Name: params.Name})
	// ...
	var result ChildResp
	if err := childWorkflowFuture.Get(ctx, &result); err != nil {
		return ParentResp{}, err
	}
	return ParentResp{Greeting: result.Greeting}, nil
}
```
<!--SNIPEND-->

---

# Continue-As-New - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/continue-as-new

> Use Temporal's Continue-As-New in Go to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow ID and fresh parameters.

This page covers the following for Go developers:

- [What is Continue-As-New?](#what)
- [Use Continue-As-New](#how)
- [When is it right to Continue-As-New?](#when)
- [Test Continue-As-New](#how-to-test)

## What is Continue-As-New? 

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution.
You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow ID but gets a new Run ID and a fresh Event History.
It also receives your Workflow's usual parameters.

## Use Continue-As-New with the Go SDK 

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically set to `nil` for the original caller of the Workflow.

<!--SNIPSTART go-continue-as-new-input-->
[go-sandbox/workflows/continueasnew/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/continueasnew/workflow.go)
```go
type ClusterManagerInput struct {
	State             *ClusterManagerState
	TestContinueAsNew bool
}

func newClusterManager(ctx workflow.Context, wfInput ClusterManagerInput) (*ClusterManager, error) {
	cm := &ClusterManager{testContinueAsNew: wfInput.TestContinueAsNew}
	// The original caller passes a nil State. Later runs receive the state from the previous run.
	if wfInput.State != nil {
		cm.state = *wfInput.State
	}
	// Test hook: use a small history limit so tests reach Continue-As-New quickly.
	if wfInput.TestContinueAsNew {
		cm.maxHistoryLength = 120
	}
	return cm, nil
}
```
<!--SNIPEND-->

The test hook in the above snippet is covered [below](#how-to-test).

Inside your Workflow, return the [`NewContinueAsNewError`](https://pkg.go.dev/go.temporal.io/sdk/workflow#NewContinueAsNewError) error.
This stops the Workflow right away and starts a new one.

<!--SNIPSTART go-continue-as-new-return-->
[go-sandbox/workflows/continueasnew/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/continueasnew/workflow.go)
```go
return ClusterManagerResult{}, workflow.NewContinueAsNewError(
	ctx,
	ClusterManagerWorkflow,
	ClusterManagerInput{
		State:             &cm.state,
		TestContinueAsNew: cm.testContinueAsNew,
	},
)
```
<!--SNIPEND-->

### Considerations for Workflows with message handlers 

If you use Updates or Signals, don't call Continue-as-New from the handlers.
Instead, wait for your handlers to finish in your main Workflow before you return `NewContinueAsNewError`.
See the [`AllHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance.

## When is it right to Continue-As-New with the Go SDK? 

Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New.
Call `GetInfo(ctx).GetContinueAsNewSuggested()` to check if it's time.

## Test Continue-As-New with the Go SDK 

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive.
Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, when `TestContinueAsNew` is `true`, this sample creates a test-only variable called `maxHistoryLength` and sets it to a small value.
A helper method in the Workflow checks it each time it considers using Continue-as-New:

<!--SNIPSTART go-continue-as-new-should-continue-->
[go-sandbox/workflows/continueasnew/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/continueasnew/workflow.go)
```go
func (cm *ClusterManager) shouldContinueAsNew(ctx workflow.Context) bool {
	if workflow.GetInfo(ctx).GetContinueAsNewSuggested() {
		return true
	}
	if cm.maxHistoryLength > 0 && workflow.GetInfo(ctx).GetCurrentHistoryLength() > cm.maxHistoryLength {
		return true
	}
	return false
}
```
<!--SNIPEND-->

---

# Dynamic Workflow - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/dynamic-workflow

> This section explains Dynamic Workflows with the Go SDK

## Set a Dynamic Workflow 

A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered.
A Workflow can be registered as dynamic by using `worker.RegisterDynamicWorkflow()`.
You must register the Workflow with the Worker before it can be invoked.
Only one Dynamic Workflow can be present on a Worker.

The Workflow Definition must then accept a single argument of type `converter.EncodedValues`.

<!--SNIPSTART go-dynamic-workflow-definition-->
[go-sandbox/workflows/dynamic/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/dynamic/workflow.go)
```go
func DynamicWorkflow(ctx workflow.Context, args converter.EncodedValues) (string, error) {
	var result string
	info := workflow.GetInfo(ctx)

	var arg1, arg2 string
	err := args.Get(&arg1, &arg2)
	if err != nil {
		return "", fmt.Errorf("failed to decode arguments: %w", err)
	}

	if info.WorkflowType.Name == "dynamic-activity" {
		ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{StartToCloseTimeout: 10 * time.Second})
		err := workflow.ExecuteActivity(ctx, "random-activity-name", arg1, arg2).Get(ctx, &result)
		if err != nil {
			return "", err
		}
	} else {
		result = fmt.Sprintf("%s - %s - %s", info.WorkflowType.Name, arg1, arg2)
	}

	return result, nil
}
```
<!--SNIPEND-->

---

# Workflow message passing - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/message-passing

> Develop with Queries, Signals, and Updates with the Temporal Go SDK.

A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates.
The Workflow implementation defines these endpoints via handler methods that can react to incoming Queries and Updates, and via Signal channels.
Temporal Clients use messages to read Workflow state and control its execution.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic.
This page introduces these features for the Temporal Go SDK.

## Handle messages 

> **ℹ️ Info:**
> The code that follows is part of a working message passing [sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/go-sandbox/workflows/messagepassing).

Follow these guidelines when writing message handlers:

- Values sent in messages, and the return values of message handlers and the main Workflow function, must be [serializable](/dataconversion).
- Prefer using a single struct over multiple input parameters.
  This allows you to add fields without changing the calling signature.

### Query handlers 

A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution:

<!--SNIPSTART go-message-greeting-workflow {"selectedLines": ["11-19", "23-25", "31", "35-47", "74"]}-->
[go-sandbox/workflows/messagepassing/message/greeting.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/greeting.go)
```go
// ...
type Language string

const Chinese Language = "chinese"
const English Language = "english"
const French Language = "french"
const Spanish Language = "spanish"
const Portuguese Language = "portuguese"

const GetLanguagesQuery = "GetLanguages"
// ...
type GetLanguagesInput struct {
	IncludeUnsupported bool
}
// ...
func GreetingWorkflow(ctx workflow.Context) (string, error) {
// ...
	greeting := map[Language]string{English: "Hello", Chinese: "你好，世界"}

	err := workflow.SetQueryHandler(ctx, GetLanguagesQuery, func(input GetLanguagesInput) ([]Language, error) {
		// 👉 A Query handler returns a value: it can inspect but must not mutate the Workflow state.
		if input.IncludeUnsupported {
			return []Language{Chinese, English, French, Spanish, Portuguese}, nil
		}
		// Sort the map keys so the result is the same every time.
		return slices.Sorted(maps.Keys(greeting)), nil
	})
	if err != nil {
		return "", err
	}
// ...
}
```
<!--SNIPEND-->

- Use [`SetQueryHandler`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetQueryHandler) to set a Query Handler that listens for a Query by name.
- The handler must be a function that returns two values, a serializable result and an error.
- You can't perform async operations such as executing an Activity in a Query handler.

### Signal Channels 

A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow.
Handle Signal messages by receiving them from their channel:

<!--SNIPSTART go-message-greeting-workflow {"selectedLines": ["20", "27-29", "31-33", "67-71", "74"]}-->
[go-sandbox/workflows/messagepassing/message/greeting.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/greeting.go)
```go
// ...
const ApproveSignal = "approve"
// ...
type ApproveInput struct {
	Name string
}
// ...
func GreetingWorkflow(ctx workflow.Context) (string, error) {
	logger := workflow.GetLogger(ctx)
	approverName := ""
// ...
	// Block until the language is approved
	var approveInput ApproveInput
	workflow.GetSignalChannel(ctx, ApproveSignal).Receive(ctx, &approveInput)
	approverName = approveInput.Name
	logger.Info("Received approval", "Approver", approverName)
// ...
}
```
<!--SNIPEND-->

- Pass the Signal's name to [`GetSignalChannel`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetSignalChannel) to get the Signal Channel that listens for Signals of that type.

Alternatively, you might want the Workflow to proceed and still be capable of handling external Signals.

<!--SNIPSTART go-message-signal-async-->
[go-sandbox/workflows/messagepassing/message/signals.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/signals.go)
```go
func SignalAsyncWorkflow(ctx workflow.Context, param YourWorkflowParam) error {
	var signal MySignal
	signalChan := workflow.GetSignalChannel(ctx, "your-signal-name")
	workflow.Go(ctx, func(ctx workflow.Context) {
		for {
			selector := workflow.NewSelector(ctx)
			selector.AddReceive(signalChan, func(c workflow.ReceiveChannel, more bool) {
				c.Receive(ctx, &signal)
				workflow.GetLogger(ctx).Info("Received Signal", "Message", signal.Message)
			})
			selector.Select(ctx)
		}
	})
	// You could now submit an activity; any signals will still be received while the activity is pending.
	return workflow.Sleep(ctx, time.Minute)
}
```
<!--SNIPEND-->

In the example above, the Workflow code uses `workflow.GetSignalChannel` to open a `workflow.Channel` for the Signal type (identified by the Signal name).

- Before completing the Workflow or using [Continue-As-New](/develop/go/workflows/continue-as-new), make sure to do an asynchronous drain on the Signal channel.
  Otherwise, the Signals will be lost.
  The [batch sliding window](https://github.com/temporalio/samples-go/tree/main/batch-sliding-window) sample contains an example:
- Delay calling `workflow.GetSignalChannel` until the Workflow initialization needed to process the Signal channel has finished.
  This is safe because the SDK buffers signals when there are no channels created for them.

<!--SNIPSTART go-message-signal-drain-->
[go-sandbox/workflows/messagepassing/message/signals.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/signals.go)
```go
reportCompletionChannel := workflow.GetSignalChannel(ctx, "ReportCompletion")
// Drain signals async
for {
	var recordID int
	ok := reportCompletionChannel.ReceiveAsync(&recordID)
	if !ok {
		break
	}
	s.recordCompletion(ctx, recordID)
}
```
<!--SNIPEND-->

### Update handlers and validators 

An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution.
It can change the Workflow state, control its flow, and return a result.
The sender must wait until the Worker accepts or rejects the Update.
The sender may wait further to receive a returned value or an exception if something goes wrong:

<!--SNIPSTART go-message-greeting-workflow {"selectedLines": ["11", "21", "31", "34-35", "49-65", "74"]}-->
[go-sandbox/workflows/messagepassing/message/greeting.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/greeting.go)
```go
// ...
type Language string
// ...
const SetLanguageUpdate = "set-language"
// ...
func GreetingWorkflow(ctx workflow.Context) (string, error) {
// ...
	language := English
	greeting := map[Language]string{English: "Hello", Chinese: "你好，世界"}
// ...
	err = workflow.SetUpdateHandlerWithOptions(ctx, SetLanguageUpdate, func(ctx workflow.Context, newLanguage Language) (Language, error) {
		// 👉 An Update handler can mutate the Workflow state and return a value.
		var previousLanguage Language
		previousLanguage, language = language, newLanguage
		return previousLanguage, nil
	}, workflow.UpdateHandlerOptions{
		Validator: func(ctx workflow.Context, newLanguage Language) error {
			if _, ok := greeting[newLanguage]; !ok {
				// 👉 In an Update validator you return any error to reject the Update.
				return fmt.Errorf("%s unsupported language", newLanguage)
			}
			return nil
		},
	})
	if err != nil {
		return "", err
	}
// ...
}
```
<!--SNIPEND-->

- Register an Update handler for a given name using either [workflow.SetUpdateHandler](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandler) or [workflow.SetUpdateHandlerWithOptions](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandlerWithOptions).
- The handler must be a function that accepts a `workflow.Context` as its first parameter.
- The function can return either a serializable value with an error or just an error.

- About validators:
  - Use validators to reject an Update before it is written to History.
    Validators are always optional.
    If you don't need to reject Updates, you don't need a validator.
  - To set a validator, pass the validator function in the [workflow.UpdateHandlerOptions](https://pkg.go.dev/go.temporal.io/sdk@v1.29.1/internal#UpdateHandlerOptions) when calling [workflow.SetUpdateHandlerWithOptions](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandlerWithOptions).
    The validator must be a function that accepts the same argument types as the handler and returns a single value of type error.

- Accepting and rejecting Updates with validators:
  - To reject an Update you must return an error or panic in the validator.
    The Workflow's `WorkflowPanicPolicy` determines how panics are handled inside the Handler function.
  - Without a validator, Updates are always accepted.
- Validators and Event History:
  - The `WorkflowExecutionUpdateAccepted` event is written into History whether the acceptance was automatic or due to a validator function not throwing an error or panicking.
  - When a validator throws an error, the Update is rejected and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History.
    The caller receives an "Update failed" error.

- Use [`workflow.GetCurrentUpdateInfo`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetCurrentUpdateInfo) to obtain information about the current Update.
  This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
- Update handlers can use Activities, Child Workflows, durable [workflow.Sleep](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) Timers, [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) conditions, and more.
  See [Blocking handlers](#blocking-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for safe usage guidelines.
- Delay calling [`workflow.SetUpdateHandler`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandler) until the Workflow initialization needed by Update handlers is finished.
  This is safe because the SDK buffers messages when there are no registered handlers for them.
  Note that [`workflow.SetUpdateHandler`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandler) will immediately invoke the handler of buffered Updates with matching types.
  This could lead to out-of-order processing of messages with different types.

## Send messages 

To send Queries, Signals, or Updates, you call methods on a Temporal [Client](https://pkg.go.dev/go.temporal.io/sdk/client#Client).
To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition.

> **⚠️ Warning:**
> Using Continue-as-New and Updates
>
> - Temporal _does not_ support Continue-as-New functionality within Update handlers.
> - Complete all handlers _before_ using Continue-as-New.
> - Use Continue-as-New from your main Workflow function, just as you would complete or fail a Workflow Execution.
>

### Send a Query 

Queries are sent from a Temporal Client.

Use [`Client.QueryWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.QueryWorkflow) or [`Client.QueryWorkflowWithOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.QueryWorkflowWithOptions).

<!--SNIPSTART go-message-send-query-->
[go-sandbox/workflows/messagepassing/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/starter/main.go)
```go
supportedLangResult, err := temporalClient.QueryWorkflow(context.Background(), we.GetID(), we.GetRunID(), message.GetLanguagesQuery, message.GetLanguagesInput{IncludeUnsupported: false})
if err != nil {
	log.Fatalf("Unable to query workflow: %v", err)
}
var supportedLang []message.Language
err = supportedLangResult.Get(&supportedLang)
if err != nil {
	log.Fatalf("Unable to get query result: %v", err)
}
log.Println("Supported languages:", supportedLang)
```
<!--SNIPEND-->

- Sending a Query doesn’t add events to a Workflow's Event History.

- You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period.
  This includes Workflows that have completed, failed, or timed out.
  Querying terminated Workflows is not supported.

- A Worker must be online and polling the Task Queue to process a Query.

### Send a Signal 

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution.
However, you can only send Signals to Workflow Executions that haven’t closed.

#### Send a Signal from a Client 

Use [`Client.SignalWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.SignalWorkflow).

Pass in both the [Workflow ID](/workflow-execution/workflowid-runid#workflow-id) and [Run ID](/workflow-execution/workflowid-runid#run-id) to uniquely identify the Workflow Execution.
If only the Workflow ID is supplied (provide an empty string as the Run ID param), the Workflow Execution that is running receives the Signal.

<!--SNIPSTART go-message-send-signal-->
[go-sandbox/workflows/messagepassing/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/starter/main.go)
```go
err := temporalClient.SignalWorkflow(context.Background(), we.GetID(), we.GetRunID(), message.ApproveSignal, message.ApproveInput{Name: ""})
if err != nil {
	log.Fatalf("Unable to signal workflow: %v", err)
}
```
<!--SNIPEND-->

- The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution.

- The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History.

#### Sending a Signal from a Workflow 

A Workflow can send a Signal to another Workflow, in which case it's called an External Signal.

<!--SNIPSTART go-message-signal-external-->
[go-sandbox/workflows/messagepassing/message/signals.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/signals.go)
```go
func SendExternalSignalWorkflow(ctx workflow.Context, param YourWorkflowParam) error {
	// ...
	signal := MySignal{
		Message: "Some important data",
	}
	err := workflow.SignalExternalWorkflow(ctx, "some-workflow-id", "", "your-signal-name", signal).Get(ctx, nil)
	if err != nil {
		return err
	}
	// ...
	return nil
}
```
<!--SNIPEND-->

When an External Signal is sent:

- A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History.
- A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History.

#### Signal-With-Start 

Signal-With-Start is used from the Client.
It takes a Workflow ID, Workflow arguments, a Signal name, and Signal arguments.

If there's a Workflow running with the given Workflow ID, it will be signaled. If there isn't, a new Workflow will be started and immediately signaled.

Use the [`Client.SignalWithStartWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.SignalWithStartWorkflow) API to start a Workflow Execution (if not already running) and pass it the Signal at the same time.

Because the Workflow Execution might not exist, this API does not take a Run ID as a parameter.
It also takes the `StartWorkflowOptions`, Workflow, and Workflow arguments to use if it starts a new Workflow Execution.

<!--SNIPSTART go-message-signal-with-start-->
[go-sandbox/workflows/messagepassing/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/starter/main.go)
```go
signal := message.MySignal{
	Message: "Some important data",
}
workflowOptions := client.StartWorkflowOptions{
	TaskQueue: message.TaskQueue,
}
we, err := temporalClient.SignalWithStartWorkflow(context.Background(), "your-workflow-id", "your-signal-name", signal,
	workflowOptions, message.SignalAsyncWorkflow, message.YourWorkflowParam{})
if err != nil {
	log.Fatalln("Error sending the Signal", err)
}
```
<!--SNIPEND-->

### Send an Update 

An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result.

A Client sending an Update must wait until the Server delivers the Update to a Worker.
Workers must be available and responsive.
Setting a timeout with a context provides a hard limit on how long a client will wait for a response.
If you need a response as soon as the Server receives the request, use a Signal instead.

- `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation.
- `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished.

Use the [`Client.UpdateWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.UpdateWorkflow) API to send an Update to a Workflow Execution.

You must provide the Workflow ID, but specifying a Run ID is optional.
If you supply only the Workflow ID (and provide an empty string as the Run ID param), the running Workflow Execution receives the Update.

You must provide a `WaitForStage` when calling `UpdateWorkflow()`.
This parameter controls the stage the update must reach before returning a handle to the caller:

- If `WaitForStage` is set to `WorkflowUpdateStageCompleted`, the handle is returned after the Update completes.
- If `WaitForStage` is set to `WorkflowUpdateStageAccepted`, the handle is returned after the Update is accepted (that is, after the validator has run, if there is a validator).

You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity.

<!--SNIPSTART go-message-send-update-->
[go-sandbox/workflows/messagepassing/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/starter/main.go)
```go
ctxWithTimeout, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()

updateHandle, err := temporalClient.UpdateWorkflow(ctxWithTimeout, client.UpdateWorkflowOptions{
	WorkflowID:   we.GetID(),
	RunID:        we.GetRunID(),
	UpdateName:   message.SetLanguageUpdate,
	WaitForStage: client.WorkflowUpdateStageAccepted,
	Args:         []interface{}{message.Chinese},
})
if err != nil {
	log.Fatalf("Unable to update workflow: %v", err)
}

var previousLang message.Language
err = updateHandle.Get(ctxWithTimeout, &previousLang)
if err != nil {
	log.Fatalf("Unable to get update result: %v", err)
}
```
<!--SNIPEND-->

#### Update-With-Start 

> **💡 Tip:**
>
> For open source server users, [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended.
>

[Update-with-Start](/sending-messages#update-with-start) lets you
[send an Update](/develop/go/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists:

- If the Workflow exists, the Update is processed.
- If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute.

Use the [`Client.UpdateWithStartWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.UpdateWithStartWorkflow) API call.
It returns once the requested Update wait stage has been reached; or when a provided context times out.
Use [`WorkflowUpdateHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#WorkflowUpdateHandle) to retrieve a result from the Update.

You will need to provide:

- [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions).
  The [Workflow ID Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) is required.
  Choose "Use Existing" and use an idempotent Update handler to ensure your code can be executed again in case of a Client failure.
  Not all `StartWorkflowOptions` are allowed.
  For example, specifying a Cron Schedule will result in an error.
  Refer to the [API documentation](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions) for further details.

- [`UpdateWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/internal#UpdateWorkflowOptions).
  Same as for [Update Workflow](/develop/go/workflows/message-passing#send-update-from-client), the update name and an update wait stage must be specified.
  For Update-with-Start, the Workflow ID is optional.
  When specified, the ID must match the one used in `StartWorkflowOptions`.
  Since a running Workflow Execution may not already exist, you can't set a Run ID.

- [`Client.NewWithStartWorkflowOperation`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.NewWithStartWorkflowOperation).
  Specify the workflow options, method and arguments.
  Note that a `WithStartWorkflowOperation` can only be used once.
  Re-using a previously used operation returns an error from `UpdateWithStartWorkflow`.

The following example shows the creation, configuration, and use of UpdateWithStart:

<!--SNIPSTART go-message-update-with-start-->
[go-sandbox/workflows/messagepassing/starter/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/starter/main.go)
```go
ctxWithTimeout, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()

workflowOptions := client.StartWorkflowOptions{
	ID:                       "some-workflow-id",
	TaskQueue:                message.TaskQueue,
	WorkflowIDConflictPolicy: enumspb.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING,
}

updateOptions := client.UpdateWorkflowOptions{
	UpdateName:   message.SetLanguageUpdate,
	WaitForStage: client.WorkflowUpdateStageCompleted,
	Args:         []interface{}{message.Chinese},
}

startWorkflowOp := temporalClient.NewWithStartWorkflowOperation(workflowOptions, message.GreetingWorkflow)
updateHandle, err := temporalClient.UpdateWithStartWorkflow(
	ctxWithTimeout,
	client.UpdateWithStartWorkflowOptions{
		StartWorkflowOperation: startWorkflowOp,
		UpdateOptions:          updateOptions,
	})
if err != nil {
	log.Fatalf("Unable to execute update-with-start: %v", err)
}

var previousLang message.Language
err = updateHandle.Get(ctxWithTimeout, &previousLang)
if err != nil {
	log.Fatalf("Unable to obtain update result: %v", err)
}

workflowRun, err := startWorkflowOp.Get(ctxWithTimeout)
if err != nil {
	log.Fatalf("Unable to obtain workflow run: %v", err)
}
```
<!--SNIPEND-->

For more examples, see the [Go sample for early-return pattern](https://github.com/temporalio/samples-go/tree/main/early-return).

## Message handler patterns 

This section covers common write operations, such as Signal and Update handlers.
It doesn't apply to pure read operations, like Queries or Update Validators.

> **💡 Tip:**
>
> For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing), and [this sample](https://github.com/temporalio/samples-go/blob/main/safe_message_handler/README.md) demonstrating safe blocking message handling.
>

### Blocking handlers 

Signal and Update handlers can block.
This allows you to use Activities, Child Workflows, durable [workflow.Sleep](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) Timers, [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) conditions, and so on.
This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls.

To use blocking handlers safely, understand what can go wrong.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of blocking Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below.

The following code modifies the Update handler from earlier on in this page.
The Update handler now makes a blocking call to execute an Activity:

<!--SNIPSTART go-message-blocking-update-->
[go-sandbox/workflows/messagepassing/message/patterns.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/patterns.go)
```go
func BlockingGreetingWorkflow(ctx workflow.Context) (string, error) {
	language := English
	greeting := map[Language]string{English: "Hello", Chinese: "你好，世界"}

	err := workflow.SetUpdateHandler(ctx, SetLanguageUpdate, func(ctx workflow.Context, newLanguage Language) (Language, error) {
		if _, ok := greeting[newLanguage]; !ok {
			ao := workflow.ActivityOptions{
				StartToCloseTimeout: 10 * time.Second,
			}
			ctx = workflow.WithActivityOptions(ctx, ao)

			var newGreeting string
			err := workflow.ExecuteActivity(ctx, CallGreetingService, newLanguage).Get(ctx, &newGreeting)
			if err != nil {
				return "", err
			}
			greeting[newLanguage] = newGreeting
		}
		var previousLanguage Language
		previousLanguage, language = language, newLanguage
		return previousLanguage, nil
	})
	if err != nil {
		return "", err
	}
	// ...
	workflow.GetSignalChannel(ctx, ApproveSignal).Receive(ctx, nil)
	if err := workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }); err != nil {
		return "", err
	}
	return greeting[language], nil
}
```
<!--SNIPEND-->

### Add blocking wait conditions 

Sometimes, blocking Signal or Update handlers need to meet certain conditions before they should continue.
You can use [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) to prevent the code from proceeding until a condition is true.
You specify the condition by passing a function that returns `true` or `false`.
This is an important feature that helps you control your handler logic.

Here are three important use cases for `workflow.Await`:

- Waiting until a specific Update has arrived.
- Waiting in a handler until it is appropriate to continue.
- Waiting in the main Workflow until all active handlers have finished.

<!--SNIPSTART go-message-await-in-handler-->
[go-sandbox/workflows/messagepassing/message/patterns.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/patterns.go)
```go
err := workflow.SetUpdateHandler(ctx, "UpdateHandler", func(ctx workflow.Context, input UpdateInput) error {
	if err := workflow.Await(ctx, updateUnblockedFunc); err != nil {
		return err
	}
	// ...
	handled++
	return nil
})
```
<!--SNIPEND-->

This is necessary if your Update handlers require something in the main Workflow function to be done first, since an Update handler can execute concurrently with the main Workflow function.

You can also use `workflow.Await` anywhere else in the handler to wait for a specific condition to become true.
This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become true.

#### Ensure your handlers finish before the Workflow completes 

`workflow.Await` can ensure your handler completes before a Workflow finishes.
When your Workflow uses blocking Update handlers, your main Workflow method can return or Continue-as-New while a handler is still waiting on an async task, such as an Activity.
The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying to retrieve Update results.
Use [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) to wait for [`AllHandlersFinished`](https://pkg.go.dev/go.temporal.io/sdk/workflow#AllHandlersFinished) to return `true` to address this problem and allow your Workflow to end smoothly:

<!--SNIPSTART go-message-wait-all-handlers-->
[go-sandbox/workflows/messagepassing/message/patterns.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/patterns.go)
```go
func WaitForHandlersWorkflow(ctx workflow.Context, param YourWorkflowParam) error {
	// ...
	err := workflow.Await(ctx, func() bool {
		return workflow.AllHandlersFinished(ctx)
	})
	if err != nil {
		return err
	}
	return nil
}
```
<!--SNIPEND-->

By default, your Worker will log a warning if you allow your Workflow Execution to finish with unfinished Update handler executions.
You can silence these warnings on a per-handler basis by setting `UnfinishedPolicy` field on [`workflow.UpdateHandlerOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#UpdateHandlerOptions) struct:

<!--SNIPSTART go-message-unfinished-policy-->
[go-sandbox/workflows/messagepassing/message/patterns.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/patterns.go)
```go
err := workflow.SetUpdateHandlerWithOptions(ctx, UpdateHandlerName, UpdateFunc, workflow.UpdateHandlerOptions{
	UnfinishedPolicy: workflow.HandlerUnfinishedPolicyAbandon,
})
```
<!--SNIPEND-->

See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information.

#### Use `workflow.Mutex` to prevent concurrent handler execution 

See [Message handler concurrency](/handling-messages#message-handler-concurrency).

Concurrent processes can interact in unpredictable ways.
Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously.
Here's an example of a pathological case:

<!--SNIPSTART go-message-handler-concurrency-bad-->
[go-sandbox/workflows/messagepassing/message/concurrency.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/concurrency.go)
```go
func BadHandlerWorkflow(ctx workflow.Context, param YourWorkflowParam) (Data, error) {
	var x, y int
	// ...
	err := workflow.SetUpdateHandler(ctx, "BadUpdateHandler", func(ctx workflow.Context) error {
		ao := workflow.ActivityOptions{
			StartToCloseTimeout: 10 * time.Second,
		}
		ctx = workflow.WithActivityOptions(ctx, ao)

		var result Data
		err := workflow.ExecuteActivity(ctx, FetchData, param.Name).Get(ctx, &result)
		if err != nil {
			return err
		}
		x = result.X
		// 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then
		// there may be times when the Workflow has x from one Activity execution and y from another.
		err = workflow.Sleep(ctx, time.Second)
		if err != nil {
			return err
		}
		y = result.Y
		return nil
	})
	if err != nil {
		return Data{}, err
	}
	// ...
	workflow.GetSignalChannel(ctx, ApproveSignal).Receive(ctx, nil)
	if err := workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }); err != nil {
		return Data{}, err
	}
	return Data{X: x, Y: y}, nil
}
```
<!--SNIPEND-->

Coordinating access with `workflow.Mutex` corrects this code.
Locking makes sure that only one handler instance can execute a specific section of code at any given time:

<!--SNIPSTART go-message-handler-concurrency-mutex-->
[go-sandbox/workflows/messagepassing/message/concurrency.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/messagepassing/message/concurrency.go)
```go
func SafeHandlerWorkflow(ctx workflow.Context, param YourWorkflowParam) (Data, error) {
	var x, y int
	mutex := workflow.NewMutex(ctx)
	// ...
	err := workflow.SetUpdateHandler(ctx, "SafeUpdateHandler", func(ctx workflow.Context) error {
		err := mutex.Lock(ctx)
		if err != nil {
			return err
		}
		defer mutex.Unlock()
		ao := workflow.ActivityOptions{
			StartToCloseTimeout: 10 * time.Second,
		}
		ctx = workflow.WithActivityOptions(ctx, ao)

		var result Data
		err = workflow.ExecuteActivity(ctx, FetchData, param.Name).Get(ctx, &result)
		if err != nil {
			return err
		}
		x = result.X
		// ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow
		// method, but no other execution of this handler can run until this execution finishes.
		err = workflow.Sleep(ctx, time.Second)
		if err != nil {
			return err
		}
		y = result.Y
		return nil
	})
	if err != nil {
		return Data{}, err
	}
	// ...
	workflow.GetSignalChannel(ctx, ApproveSignal).Receive(ctx, nil)
	if err := workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }); err != nil {
		return Data{}, err
	}
	return Data{X: x, Y: y}, nil
}
```
<!--SNIPEND-->

## Troubleshooting

See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Go-specific discussion of this topic.

When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors:

- **The Client can't contact the server**

- **The Workflow does not exist**

Unlike Signals, for Queries and Updates, the Client waits for a response from the Worker.
If an issue occurs during the handler execution by the Worker, the Client may receive an exception.

### Problems when sending an Update

- **There is no Workflow Worker polling the Task Queue**

  Your request will be retried by the SDK Client until the calling context is cancelled.

- **Update failed.**

  Update failures are like [Workflow failures](/references/failures).
  Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.
  These might include:

  - A failed Child Workflow
  - A failed Activity if the activity retries have been set to a finite number
  - The Workflow author returning an `error`
  - A panic in the handler, depending on the `WorkflowPanicPolicy`

- **The handler caused the Workflow Task to fail**
  A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage:
  - If the request hasn't been accepted by the server, you receive a [`FAILED_PRECONDITION`](https://pkg.go.dev/go.temporal.io/api/serviceerror#FailedPrecondition) error.
  - If the request has been accepted, it is durable.
    Once the Workflow is healthy again after a code deploy, use a [`WorkflowUpdateHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#WorkflowUpdateHandle) to fetch the Update result.

- **The Workflow finished while the Update handler execution was in progress**:
  You'll receive a [`ServiceError`](https://pkg.go.dev/go.temporal.io/api/serviceerror#ServiceError) "workflow execution already completed"`.

  This will happen if the Workflow finished while the Update handler execution was in progress, for example because

  - The Workflow was canceled or failed.

  - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers).

### Problems when sending a Query

- **There is no Workflow Worker polling the Task Queue**

  You'll receive a [`ServiceError`](https://pkg.go.dev/go.temporal.io/api/serviceerror#ServiceError) on which the `status` is `FAILED_PRECONDITION`.

- **Query failed.**
  You'll receive a [`QueryFailed`](https://pkg.go.dev/go.temporal.io/api/serviceerror#QueryFailed) error.
  Any panic in a Query handler will trigger this error.
  This differs from Signal and Update, where panics can lead to Workflow Task Failure instead.

- **The handler caused the Workflow Task to fail.**
  This would happen, for example, if the Query handler blocks the thread for too long without yielding.

---

# Schedules - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/schedules

> Schedule Workflows, start them with delays or as Temporal Cron Jobs using the Go SDK. Master scheduling, backfilling, pausing, deleting, and updating Workflows.

This page shows how to do the following:

- [Scheduled Workflows](#schedule-a-workflow)
  - [Create a Schedule](#create-schedule)
  - [Backfill a Schedule](#backfill-schedule)
  - [Delete a Schedule](#delete-schedule)
  - [Describe a Schedule](#describe-schedule)
  - [List Schedules](#list-schedules)
  - [Pause a Schedule](#pause-schedule)
  - [Trigger a Schedule](#trigger-schedule)
  - [Update a Schedule](#update-schedule)
- [Start delay](#start-delay)
- [Temporal Cron Jobs](#temporal-cron-jobs)

## Scheduled Workflows 

Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes.

Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process.

Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy).
If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions.

### Create a Schedule 

Schedules are initiated with the `create` call.
The user generates a unique Schedule ID for each new Schedule.

To create a Schedule in Go, use `Create()` on the [Client](/encyclopedia/temporal-client).
Schedules must be initialized with a Schedule ID, [Spec](/schedule), and [Action](/schedule) in `client.ScheduleOptions{}`.

<!--SNIPSTART go-schedule-create {"highlightedLines": "2-3,5-13"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {2-3,5-13}
func CreateSchedule(ctx context.Context, temporalClient client.Client) (client.ScheduleHandle, error) {
	scheduleID := "schedule_id"
	workflowID := "schedule_workflow_id"

	scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{
		ID:   scheduleID,
		Spec: client.ScheduleSpec{},
		Action: &client.ScheduleWorkflowAction{
			ID:        workflowID,
			Workflow:  ScheduleWorkflow,
			TaskQueue: TaskQueue,
		},
	})
	if err != nil {
		return nil, fmt.Errorf("unable to create schedule: %w", err)
	}
	log.Println("Schedule created", "ScheduleID", scheduleID)
	return scheduleHandle, nil
}
```
<!--SNIPEND-->

> **💡 Tip:**
> Schedule Auto-Deletion
>
> Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again.
> The Temporal Service doesn't guarantee when this removal will happen.
>

### Backfill a Schedule 

Backfilling a Schedule executes [Workflow Tasks](/tasks#workflow-task) ahead of the Schedule's specified time range.
This is useful for executing a missed or delayed Action, or for testing the Workflow ahead of time.

To backfill a Schedule in Go, use `Backfill()` on `ScheduleHandle`.
Specify the start and end times to execute the Workflow, along with the overlap policy.

<!--SNIPSTART go-schedule-backfill {"highlightedLines": "25-38"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {25-38}
func BackfillSchedule(ctx context.Context, temporalClient client.Client) error {
	workflowID := "schedule_workflow_"
	// Create a paused Schedule to backfill
	now := time.Now()
	scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{
		ID: "backfill-schedule",
		Spec: client.ScheduleSpec{
			Intervals: []client.ScheduleIntervalSpec{
				{
					Every: time.Minute,
				},
			},
		},
		Action: &client.ScheduleWorkflowAction{
			ID:        workflowID,
			Workflow:  ScheduleWorkflow,
			TaskQueue: TaskQueue,
		},
		Paused: true,
	})
	if err != nil {
		return fmt.Errorf("unable to create schedule: %w", err)
	}

	err = scheduleHandle.Backfill(ctx, client.ScheduleBackfillOptions{
		Backfill: []client.ScheduleBackfill{
			{
				Start:   now.Add(-4 * time.Minute),
				End:     now.Add(-2 * time.Minute),
				Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL,
			},
			{
				Start:   now.Add(-2 * time.Minute),
				End:     now,
				Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL,
			},
		},
	})
	if err != nil {
		return fmt.Errorf("unable to backfill schedule: %w", err)
	}
	return nil
}
```
<!--SNIPEND-->

### Delete a Schedule 

Deleting a Schedule erases a Schedule.
Deletion does not affect any Workflows started by the Schedule.

To delete a Schedule, use `Delete()` on the `ScheduleHandle`.

<!--SNIPSTART go-schedule-delete {"highlightedLines": "6-10"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {6-10}
func DeleteSchedule(ctx context.Context, temporalClient client.Client) error {
	scheduleID := "schedule_id"
	// Retrieve the schedule handle by its ID
	scheduleHandle := temporalClient.ScheduleClient().GetHandle(ctx, scheduleID)

	log.Println("Deleting schedule", "ScheduleID", scheduleHandle.GetID())
	err := scheduleHandle.Delete(ctx)
	if err != nil {
		return fmt.Errorf("unable to delete schedule: %w", err)
	}
	return nil
}
```
<!--SNIPEND-->

### Describe a Schedule 

`Describe` retrieves information about the current Schedule configuration.
This can include details about the Schedule Spec (such as Intervals), CronExpressions, and Schedule State.

To describe a Schedule, use `Describe()` on the ScheduleHandle.

<!--SNIPSTART go-schedule-describe {"highlightedLines": "20"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {20}
func DescribeSchedule(ctx context.Context, temporalClient client.Client) (*client.ScheduleDescription, error) {
	// create Schedule
	scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{
		ID: "test-schedule-describe-spec-cron-schedule",
		Spec: client.ScheduleSpec{
			CronExpressions: []string{
				"0 12 * * MON",
			},
		},
		Action: &client.ScheduleWorkflowAction{
			ID:        "schedule_workflow_describe",
			Workflow:  ScheduleWorkflow,
			TaskQueue: TaskQueue,
		},
	})
	if err != nil {
		return nil, fmt.Errorf("unable to create schedule: %w", err)
	}
	// describe schedule
	description, err := scheduleHandle.Describe(ctx)
	if err != nil {
		return nil, fmt.Errorf("unable to describe schedule: %w", err)
	}
	log.Println("Schedule next action times", description.Info.NextActionTimes)
	return description, nil
}
```
<!--SNIPEND-->

### List Schedules 

The `List` action returns all available Schedules and their respective Schedule IDs.

To return information on all Schedules, use `ScheduleClient.List()`.

<!--SNIPSTART go-schedule-list {"highlightedLines": "20-22,27-33"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {20-22,27-33}
func ListSchedules(ctx context.Context, temporalClient client.Client) error {
	// Create Schedule and Workflow IDs
	scheduleID := "schedule_" + uuid.NewString()
	workflowID := "schedule_workflow_" + uuid.NewString()
	// Create the schedule.
	scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{
		ID:   scheduleID,
		Spec: client.ScheduleSpec{},
		Action: &client.ScheduleWorkflowAction{
			ID:        workflowID,
			Workflow:  ScheduleWorkflow,
			TaskQueue: TaskQueue,
		},
	})
	if err != nil {
		return fmt.Errorf("unable to create schedule: %w", err)
	}
	log.Println("Schedule created", "ScheduleID", scheduleHandle.GetID())
	// list schedules
	listView, err := temporalClient.ScheduleClient().List(ctx, client.ScheduleListOptions{
		PageSize: 1,
	})
	if err != nil {
		return fmt.Errorf("unable to list schedules: %w", err)
	}

	for listView.HasNext() {
		entry, err := listView.Next()
		if err != nil {
			return fmt.Errorf("unable to get next schedule: %w", err)
		}
		log.Println("Schedule", "ScheduleID", entry.ID)
	}
	return nil
}
```
<!--SNIPEND-->

### Pause a Schedule 

`Pause` and `Unpause` enable the start or stop of all future Workflow Runs on a given Schedule.

Pausing a Schedule halts all future Workflow Runs.
Pausing can be enabled by setting `State.Paused` to `true`, or by using `Pause()` on the ScheduleHandle.

Unpausing a Schedule allows the Workflow to execute as planned.
To unpause a Schedule, use `Unpause()` on `ScheduleHandle`.

<!--SNIPSTART go-schedule-pause {"highlightedLines": "6-8,18-20"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {6-8,18-20}
func PauseSchedule(ctx context.Context, temporalClient client.Client) error {
	scheduleID := "schedule_id"
	scheduleHandle := temporalClient.ScheduleClient().GetHandle(ctx, scheduleID)

	// Pause the schedule and print the status
	err := scheduleHandle.Pause(ctx, client.SchedulePauseOptions{
		Note: "The Schedule has been paused.",
	})
	if err != nil {
		return fmt.Errorf("unable to pause schedule: %w", err)
	}
	log.Println("The Schedule has been paused.")

	// Wait for 5 seconds
	time.Sleep(5 * time.Second)

	// Unpause the schedule
	err = scheduleHandle.Unpause(ctx, client.ScheduleUnpauseOptions{
		Note: "The Schedule has been unpaused.",
	})
	if err != nil {
		return fmt.Errorf("unable to unpause schedule: %w", err)
	}
	log.Println("The Schedule has been unpaused.")
	return nil
}
```
<!--SNIPEND-->

### Trigger a Schedule 

Triggering a Schedule immediately executes an Action defined in that Schedule.
By default, `trigger` is subject to the Overlap Policy.

To trigger a Scheduled Workflow Execution, use `Trigger()` on `ScheduleHandle`.

<!--SNIPSTART go-schedule-trigger {"highlightedLines": "18-26"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {18-26}
func TriggerSchedule(ctx context.Context, temporalClient client.Client) error {
	scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{
		ID:   "trigger-schedule",
		Spec: client.ScheduleSpec{},
		Action: &client.ScheduleWorkflowAction{
			ID:        "trigger-schedule-workflow",
			Workflow:  ScheduleWorkflow,
			TaskQueue: TaskQueue,
		},
		Paused:  true,
		Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL,
	})
	if err != nil {
		return fmt.Errorf("unable to create schedule: %w", err)
	}

	// Trigger Schedule
	for i := 0; i < 5; i++ {
		err = scheduleHandle.Trigger(ctx, client.ScheduleTriggerOptions{
			Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL,
		})
		if err != nil {
			return fmt.Errorf("unable to trigger schedule: %w", err)
		}
		time.Sleep(2 * time.Second)
	}
	return nil
}
```
<!--SNIPEND-->

### Update a Schedule 

Updating a Schedule changes the configuration of an existing Schedule.
These changes can be made to Workflow Actions, Action parameters, Memos, and the Workflow's Cancellation Policy.

Use `Update()` on the ScheduleHandle to modify a Schedule.
The `DoUpdate` function receives the current Schedule description and returns the Schedule to save.

<!--SNIPSTART go-schedule-update {"highlightedLines": "16-28"}-->
[go-sandbox/workflows/schedules/schedules.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/schedules.go)
```go {16-28}
func UpdateSchedule(ctx context.Context, temporalClient client.Client) error {
	scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{
		ID:   "update-schedule",
		Spec: client.ScheduleSpec{},
		Action: &client.ScheduleWorkflowAction{
			ID:        "update-schedule-workflow",
			Workflow:  ScheduleWorkflow,
			TaskQueue: TaskQueue,
		},
		Paused: true,
	})
	if err != nil {
		return fmt.Errorf("unable to create schedule: %w", err)
	}

	updateSchedule := func(input client.ScheduleUpdateInput) (*client.ScheduleUpdate, error) {
		// Change the Schedule to run every hour.
		input.Description.Schedule.Spec = &client.ScheduleSpec{
			Intervals: []client.ScheduleIntervalSpec{{Every: time.Hour}},
		}
		return &client.ScheduleUpdate{
			Schedule: &input.Description.Schedule,
		}, nil
	}

	err = scheduleHandle.Update(ctx, client.ScheduleUpdateOptions{
		DoUpdate: updateSchedule,
	})
	if err != nil {
		return fmt.Errorf("unable to update schedule: %w", err)
	}
	return nil
}
```
<!--SNIPEND-->

## Start Delay 

Use `StartDelay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule.

Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `StartDelay` field, and pass the instance to the `ExecuteWorkflow` call.

<!--SNIPSTART go-schedule-start-delay-->
[go-sandbox/workflows/schedules/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	ID:        "your-workflow-id",
	TaskQueue: TaskQueue,
	// Start the workflow in 12 hours
	StartDelay: 12 * time.Hour,
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

## Temporal Cron Jobs 

> **⚠️ Caution:**
> Cron support is not recommended
>
> We recommend using [Schedules](/schedule) instead of Cron Jobs.
> Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules.
>

A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution.

A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made.

Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `CronSchedule` field, and pass the instance to the `ExecuteWorkflow` call.

- Type: `string`
- Default: None

<!--SNIPSTART go-schedule-cron-->
[go-sandbox/workflows/schedules/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/schedules/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	ID:           "your-cron-workflow-id",
	TaskQueue:    TaskQueue,
	CronSchedule: "15 8 * * *",
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

Temporal Workflow Schedule Cron strings follow this format:

```
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
* * * * *
```

---

# Selectors - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/selectors

> Use Temporal’s Go SDK Selectors with Futures, Timers, and Channels. Ensure deterministic Workflow execution and handle multiple parallel tasks efficiently.

This page shows how to do the following:

- [Use Selectors with Futures](#use-selectors-with-futures)
- [Use Selectors with Timers](#use-selectors-with-timers)
- [Use Selectors with Channels](#use-selectors-with-channels)

In Go, the `select` statement lets a goroutine wait on multiple communication operations.
A `select` **blocks until one of its cases can run**, then it executes that case.
It chooses one at random if multiple are ready.

However, a normal Go select statement can not be used inside of Workflows directly because of the random nature.
Temporal's Go SDK `Selector`s are similar and act as a replacement.
They can block on sending and receiving from Channels but as a bonus can listen on Future deferred work.
Use Selectors to defer and process work in place of Go's `select`. Selectors keep the Workflow Definition deterministic. Using `select` in Activity code is fine.

## Full API example 

The API is sufficiently different from `select` that it bears documenting:

<!--SNIPSTART go-selector-api-example-->
[go-sandbox/workflows/selectors/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/selectors/workflow.go)
```go
func SampleWorkflow(ctx workflow.Context) error {
	// standard Workflow setup code
	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	})

	// API Example: declare a new selector
	selector := workflow.NewSelector(ctx)

	// API Example: defer code execution until the Future that represents Activity result is ready
	work := workflow.ExecuteActivity(ctx, ExampleActivity)
	selector.AddFuture(work, func(f workflow.Future) {
		// deferred code omitted...
	})

	// more parallel timers and activities initiated...

	// API Example: receive information from a Channel
	var signalVal string
	channel := workflow.GetSignalChannel(ctx, channelName)
	selector.AddReceive(channel, func(c workflow.ReceiveChannel, more bool) {
		// matching on the channel doesn't consume the message.
		// So it has to be explicitly consumed here
		c.Receive(ctx, &signalVal)
		// do something with received information
	})

	// API Example: block until the next Future is ready to run
	// important! none of the deferred code runs until you call selector.Select
	selector.Select(ctx)

	return nil
}
```
<!--SNIPEND-->

## Use Selectors with Futures

You usually add `Futures` after `Activities`:

<!--SNIPSTART go-selector-api-example {"selectedLines": ["10-14"]}-->
[go-sandbox/workflows/selectors/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/selectors/workflow.go)
```go
// ...
	// API Example: defer code execution until the Future that represents Activity result is ready
	work := workflow.ExecuteActivity(ctx, ExampleActivity)
	selector.AddFuture(work, func(f workflow.Future) {
		// deferred code omitted...
	})
```
<!--SNIPEND-->

Calling `ExecuteActivity` by itself doesn't result in actual Activity execution. It results in a command that gets batched with others when there's a yield point in the Workflow definition. Your `Futures` won't run until you call `selector.Select(ctx)`. So no calls to `ExecuteActivity` should result in commands being sent to the server until `selector.Select` is called.

A call to `selector.Select(ctx)` doesn't actually execute a `Future`, but it _requests_ execution for a `Future`. So this is a point where the SDK will communicate the command to execute the Activity to the server.

<!--SNIPSTART go-selector-select-loop-->
[go-sandbox/workflows/selectors/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/selectors/workflow.go)
```go
// API Example: blocking conditionally
if len(items) > 0 {
	selector.Select(ctx) // wait for the first Future
}

// API Example: popping off all remaining Futures
for i := 1; i < len(items); i++ {
	selector.Select(ctx) // this will wait for one branch
	// you can interrupt execution here
	if processErr != nil {
		return nil, processErr
	}
}
```
<!--SNIPEND-->

A Future matches only once per Selector instance even if Select is called multiple times.
If multiple items are available, the order of matching is not defined.

## Use Selectors with Timers

An important use case of futures is setting up a race between a timer and a pending activity, effectively adding a "soft" timeout that doesn't result in any errors or retries of that activity.

For example, [the Timer sample](https://github.com/temporalio/samples-go/tree/main/timer) shows how you can write a long running order processing operation where:

- if processing takes too long, we send out a notification email to user about the delay, but we won't cancel the operation
- if the operation finishes before the timer fires, then we want to cancel the timer.

<!--SNIPSTART go-selector-timer-race-->
[go-sandbox/workflows/selectors/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/selectors/workflow.go)
```go
// Create a cancelable Context for the Timer.
childCtx, cancelHandler := workflow.WithCancel(ctx)

var processingDone bool
f := workflow.ExecuteActivity(ctx, OrderProcessingActivity, processingTime)
selector.AddFuture(f, func(f workflow.Future) {
	processingDone = true
	// cancel timerFuture
	cancelHandler()
})

// use timer future to send notification email if processing takes too long
timerFuture := workflow.NewTimer(childCtx, processingTimeThreshold)
selector.AddFuture(timerFuture, func(f workflow.Future) {
	if !processingDone {
		// processing is not done yet when timer fires, send notification email
		_ = workflow.ExecuteActivity(ctx, SendEmailActivity).Get(ctx, nil)
		emailSent = true
	}
})

// wait the timer or the order processing to finish
selector.Select(ctx)
```
<!--SNIPEND-->

We create timers with the `workflow.NewTimer` API.

## Use Selectors with Channels

`selector.AddReceive(channel, func(c workflow.ReceiveChannel, more bool) {})` is the primary mechanism which receives messages from `Channels`.

<!--SNIPSTART go-selector-api-example {"selectedLines": ["18-26"]}-->
[go-sandbox/workflows/selectors/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/selectors/workflow.go)
```go
// ...
	// API Example: receive information from a Channel
	var signalVal string
	channel := workflow.GetSignalChannel(ctx, channelName)
	selector.AddReceive(channel, func(c workflow.ReceiveChannel, more bool) {
		// matching on the channel doesn't consume the message.
		// So it has to be explicitly consumed here
		c.Receive(ctx, &signalVal)
		// do something with received information
	})
```
<!--SNIPEND-->

Merely matching on the channel doesn't consume the message; it has to be explicitly consumed with a `c.Receive(ctx, &signalVal)` call.

## Query Selector state

You can use the `selector.HasPending` API to ensure that signals are not lost when a Workflow is closed (for example, by `ContinueAsNew`).

## Learn more

Usage of Selectors is best learned by example:

- Setting up a race condition between an Activity and a Timer, and conditionally execute ([Timer example](https://github.com/temporalio/samples-go/blob/main/timer/workflow.go))
- Receiving information in a Channel ([Mutex example](https://github.com/temporalio/samples-go/blob/main/mutex/mutex_workflow.go))
- Looping through a list of work and scheduling them all in parallel ([DSL example](https://github.com/temporalio/samples-go/blob/main/dsl/workflow.go))
- Executing activities in parallel, pick the first result, cancel remainder ([Pick First example](https://github.com/temporalio/samples-go/blob/main/pickfirst/pickfirst_workflow.go))

---

# Side Effects - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/side-effects

> Side Effects in Workflows execute non-deterministic code, storing results in the Workflow Event History to maintain determinism. Use Go's SideEffect function for integration.

Side Effects are used to execute non-deterministic code, such as generating a UUID or a random number, without compromising determinism in the Workflow.
This is done by storing the non-deterministic results of the Side Effect into the Workflow [Event History](/workflow-execution/event#event-history).

A Side Effect does not re-execute during a Replay. Instead, it returns the recorded result from the Workflow Execution Event History.

Side Effects should not fail. An exception that is thrown from the Side Effect causes failure and retry of the current Workflow Task.

An Activity or a Local Activity may also be used instead of a Side effect, as its result is also persisted in Workflow Execution History.

> **📝 Note:**
>
> You shouldn't modify the Workflow state inside a Side Effect function, because it is not reexecuted during Replay. Side Effect function should be used to return a value.
>

Use the [`SideEffect`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SideEffect) function from the `go.temporal.io/sdk/workflow` package to execute a [Side Effect](/workflow-execution/event#side-effect) directly in your Workflow.

Pass it an instance of `workflow.Context` and the function to execute.

The `SideEffect` API returns a Future, an instance of [`converter.EncodedValue`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SideEffect).

Use the `Get` method on the Future to retrieve the result of the Side Effect.

**Correct implementation**

The following example demonstrates the correct way to use `SideEffect`:

<!--SNIPSTART go-side-effect-correct-->
[go-sandbox/workflows/sideeffects/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/sideeffects/workflow.go)
```go
encodedRandom := workflow.SideEffect(ctx, func(ctx workflow.Context) interface{} {
	return rand.Intn(100)
})

var random int
if err := encodedRandom.Get(&random); err != nil {
	return 0, err
}
// ...
```
<!--SNIPEND-->

**Incorrect implementation**

The following example demonstrates how NOT to use `SideEffect`:

<!--SNIPSTART go-side-effect-incorrect-->
[go-sandbox/workflows/sideeffects/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/sideeffects/workflow.go)
```go
// Warning: This is an incorrect example.
// This code is non-deterministic.
var random int
workflow.SideEffect(ctx, func(ctx workflow.Context) interface{} {
	random = rand.Intn(100)
	return nil
})
// random will always be 0 in replay, so this code is non-deterministic.
```
<!--SNIPEND-->

On replay the provided function is not executed, the random number will always be 0, and the Workflow Execution could take a different path, breaking determinism.

## Mutable Side Effects 

Mutable Side Effects execute the provided function once, and then it looks up the History of the value with the given ID.

- If there is no existing value, then it records the function result as a value with the given ID on the History.
- If there is an existing value, then it compares whether the existing value from the History has changed from the new function results, by calling the equals function.
  - If the values are equal, then it returns the value without recording a new Marker Event
  - If the values aren't equal, then it records the new value with the same ID on the History.

> **📝 Note:**
>
> During a Workflow Execution, every new Side Effect call results in a new Marker recorded on the Event History; whereas Mutable Side Effects only records a new Marker on the Event History if the value for the Side Effect ID changes or is set the first time.
>
> During a Replay, Mutable Side Effects will not execute the function again. Instead, it returns the exact same value that was returned during the Workflow Execution.
>

To use [`MutableSideEffect()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#MutableSideEffect) in Go, provide a unique ID within the scope of the Workflow, a function that returns the value, and a function that compares two values.

<!--SNIPSTART go-mutable-side-effect-->
[go-sandbox/workflows/sideeffects/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/sideeffects/workflow.go)
```go
get := func(ctx workflow.Context) interface{} {
	return configuredNumber()
}
eq := func(a, b interface{}) bool {
	return a.(int) == b.(int)
}

var number int
if err := workflow.MutableSideEffect(ctx, "configureNumber", get, eq).Get(&number); err != nil {
	return 0, fmt.Errorf("can't decode number: %w", err)
}
```
<!--SNIPEND-->

---

# Workflow Timeouts - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/timeouts

> Optimize Workflow Execution with Temporal Go SDK - Set Workflow Timeouts and Retry Policies efficiently.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts).

Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient.
Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes.
If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer.

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed.
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last.
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task.

Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set a timeout, and pass the instance to the `ExecuteWorkflow` call.

Available timeouts are:

- `WorkflowExecutionTimeout`
- `WorkflowRunTimeout`
- `WorkflowTaskTimeout`

<!--SNIPSTART go-workflow-timeouts-set-->
[go-sandbox/workflows/timeouts/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/timeouts/start.go)
```go
workflowOptions := client.StartWorkflowOptions{
	ID:        "your-workflow-id",
	TaskQueue: TaskQueueName,
	// Set Workflow Timeout duration
	WorkflowExecutionTimeout: 24 * 365 * 10 * time.Hour,
	// WorkflowRunTimeout: 24 * 365 * 10 * time.Hour,
	// WorkflowTaskTimeout: 10 * time.Second,
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

## Workflow retries 

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

Create an instance of a [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) from the `go.temporal.io/sdk/temporal` package and provide it as the value to the `RetryPolicy` field of the instance of `StartWorkflowOptions`.

- Type: [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy)
- Default: None

<!--SNIPSTART go-workflow-retry-policy-->
[go-sandbox/workflows/timeouts/start.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/timeouts/start.go)
```go
retryPolicy := &temporal.RetryPolicy{
	InitialInterval:    time.Second,
	BackoffCoefficient: 2.0,
	MaximumInterval:    time.Second * 100,
}
workflowOptions := client.StartWorkflowOptions{
	ID:          "your-workflow-id",
	TaskQueue:   TaskQueueName,
	RetryPolicy: retryPolicy,
}
workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition)
if err != nil {
	return nil, err
}
```
<!--SNIPEND-->

---

# Timers - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/timers

> Set Durable Timers in a Workflow using the sleep() or NewTimer() functions in Go with Temporal. Timers persist through Worker and Temporal Service downtime.

A Workflow can set a Durable Timer for a fixed time period.
In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`.

A Workflow can sleep for days, months, or even years.
Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing.

Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker.

To set a Timer in Go, use the [`NewTimer()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#NewTimer) function and pass the duration you want to wait before continuing.

<!--SNIPSTART go-timer-new-timer-->
[go-sandbox/workflows/timers/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/timers/workflow.go)
```go
// Use a cancelable Context so the Workflow can cancel the Timer early.
timerCtx, cancelTimer := workflow.WithCancel(ctx)
defer cancelTimer()
timer := workflow.NewTimer(timerCtx, duration)
```
<!--SNIPEND-->

To set a sleep duration in Go, use the [`Sleep()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) function and pass the duration you want to wait before continuing.
A zero or negative sleep duration causes the function to return immediately.

<!--SNIPSTART go-timer-sleep-->
[go-sandbox/workflows/timers/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/timers/workflow.go)
```go
err := workflow.Sleep(ctx, 10*time.Second)
if err != nil {
	return err
}
```
<!--SNIPEND-->

For more information, see the [Timer](https://github.com/temporalio/samples-go/tree/main/timer) example in the [Go Samples repository](https://github.com/temporalio/samples-go).

---

# Versioning - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/versioning

> Patch Go Workflow code with GetVersion, deprecate old Workflow versions, and plan cutovers without breaking running Executions.

Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress.

The Temporal Platform requires that Workflow Definitions are [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably.

With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version.
There are two primary Versioning methods that you can use:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths.
- [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.

> **🚨 Danger:**
> Support for the experimental Worker Versioning method before 2025 was scheduled for removal from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance. You can still refer to the [Worker Versioning Legacy](/develop/go/worker-versioning-legacy) docs if needed.

## Worker Versioning

Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching.

## Versioning with Patching 

### Patching with `GetVersion`

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow Executions, create a patch.

Consider the following Workflow Definition:

<!--SNIPSTART go-versioning-original-->
[go-sandbox/workflows/versioning/original.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/original.go)
```go
func YourWorkflow(ctx workflow.Context, data string) (string, error) {
	ao := workflow.ActivityOptions{
		ScheduleToStartTimeout: time.Minute,
		StartToCloseTimeout:    time.Minute,
	}
	ctx = workflow.WithActivityOptions(ctx, ao)
	var result1 string
	err := workflow.ExecuteActivity(ctx, ActivityA, data).Get(ctx, &result1)
	if err != nil {
		return "", err
	}
	var result2 string
	err = workflow.ExecuteActivity(ctx, ActivityB, result1).Get(ctx, &result2)
	return result2, err
}
```
<!--SNIPEND-->

Suppose you replaced `ActivityA` with `ActivityC` and deployed the updated code.

If an existing Workflow Execution was started by the original version of the Workflow code, where `ActivityA` was run, and then resumed running on a new Worker where it was replaced with `ActivityC`, the server side Event History would be out of sync.
This would cause the Workflow to fail with a nondeterminism error.

To resolve this, you can use `workflow.GetVersion()` to patch to your Workflow:

<!--SNIPSTART go-versioning-get-version-->
[go-sandbox/workflows/versioning/patched.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/patched.go)
```go
var err error
v := workflow.GetVersion(ctx, "Step1", workflow.DefaultVersion, 1)
if v == workflow.DefaultVersion {
	err = workflow.ExecuteActivity(ctx, ActivityA, data).Get(ctx, &result1)
} else {
	err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1)
}
if err != nil {
	return "", err
}

var result2 string
err = workflow.ExecuteActivity(ctx, ActivityB, result1).Get(ctx, &result2)
return result2, err
```
<!--SNIPEND-->

When `workflow.GetVersion()` is run for the new Workflow Execution, it records a marker in the Event History so that all future calls to `GetVersion` for this change ID — `Step1` in the example — on this Workflow Execution will always return the given version number, which is `1` in the example.

If you make an additional change, such as replacing ActivityC with ActivityD, you need to
add some additional code:

<!--SNIPSTART go-versioning-second-change-->
[go-sandbox/workflows/versioning/patched.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/patched.go)
```go
v := workflow.GetVersion(ctx, "Step1", workflow.DefaultVersion, 2)
if v == workflow.DefaultVersion {
	err = workflow.ExecuteActivity(ctx, ActivityA, data).Get(ctx, &result1)
} else if v == 1 {
	err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1)
} else {
	err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1)
}
```
<!--SNIPEND-->

Note that we changed `maxSupported` from 1 to 2.
A Workflow that has already passed this `GetVersion()` call before it was introduced returns `DefaultVersion`.
A Workflow that was run with `maxSupported` set to 1 returns 1.
New Workflows return 2.

After all the Workflow Executions prior to version 1 have left retention, you can remove the code for that version:

<!--SNIPSTART go-versioning-remove-default-->
[go-sandbox/workflows/versioning/patched.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/patched.go)
```go
v := workflow.GetVersion(ctx, "Step1", 1, 2)
if v == 1 {
	err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1)
} else {
	err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1)
}
```
<!--SNIPEND-->

You'll note that `minSupported` has changed from `DefaultVersion` to `1`.
If an older version of the Workflow Execution history is replayed on this code, it fails because the minimum expected version is 1.
After all the Workflow Executions for version 1 have left retention, you can remove version 1 so that your code looks like the following:

<!--SNIPSTART go-versioning-keep-get-version-->
[go-sandbox/workflows/versioning/patched.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/patched.go)
```go
workflow.GetVersion(ctx, "Step1", 2, 2)
err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1)
```
<!--SNIPEND-->

Note that we have preserved the call to `GetVersion()`. There are two reasons to preserve this call:

1. This ensures that if there is a Workflow Execution still running for an older version, it will
   fail here and not proceed.
2. If you need to make additional changes for `Step1`, such as changing ActivityD to ActivityE, you
   only need to update `maxSupported` from 2 to 3 and branch from there.

You need to preserve only the first call to `GetVersion()` for each `changeID`.
All subsequent calls to `GetVersion()` with the same change ID are safe to remove.
If necessary, you can remove the first `GetVersion()` call, but you need to ensure the following:

- All executions with an older version have left retention.
- You can no longer use `Step1` for the `changeID`. If you need to make changes to that same part in
  the future, such as change from ActivityD to ActivityE, you would need to use a different `changeID`
  like `Step1-fix2`, and start `minSupported` from `DefaultVersion` again. The code would look like the
  following:

<!--SNIPSTART go-versioning-new-change-id-->
[go-sandbox/workflows/versioning/patched.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/patched.go)
```go
v := workflow.GetVersion(ctx, "Step1-fix2", workflow.DefaultVersion, 1)
if v == workflow.DefaultVersion {
	err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1)
} else {
	err = workflow.ExecuteActivity(ctx, ActivityE, data).Get(ctx, &result1)
}
```
<!--SNIPEND-->

You can add multiple calls to `GetVersion` in a single Workflow.
This can become challenging to manage if you have many long-running Workflows, as you will wind up with many code branches over time.
To clean these up, you can gradually deprecate older Workflow versions.

### Deprecating old Workflow versions

You can safely remove support for older Workflow versions once you are certain that there are no longer any open Workflow Executions based on that version. You can use the following [List Filter](/list-filter) syntax for this (the 1 near the end of the last line represents the version number):

```
WorkflowType = "PizzaWorkflow"
    AND ExecutionStatus = "Running"
    AND TemporalChangeVersion="ChangedNotificationActivityType-1"
```

Since Workflow Executions that were started before `GetVersion` was added to the code won't have the associated Marker in their Event History, you'll need to use a different query to determine if any of those are still running:

```
WorkflowType = "PizzaWorkflow"
    AND ExecutionStatus = "Running"
    AND TemporalChangeVersion IS NULL
```

If you have found that there are no longer any open executions for the first two versions of the Workflow, for example, then you could remove support for them by changing the code as shown below:

<!--SNIPSTART go-versioning-deprecate-->
[go-sandbox/workflows/versioning/deprecate.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/deprecate.go)
```go
version := workflow.GetVersion(ctx, "ChangedNotificationActivityType", 2, 3)
if version == 2 {
	err = workflow.ExecuteActivity(ctx, SendTextMessage).Get(ctx, nil)
} else {
	err = workflow.ExecuteActivity(ctx, SendTweet).Get(ctx, nil)
}
```
<!--SNIPEND-->

Patching allows you to make changes to currently running Workflows.
It introduces compatible changes without causing non-determinism errors.

### Workflow cutovers

To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow.

Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes.
To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers.

For example, you would duplicate `PizzaWorkflow` as `PizzaWorkflowV2`:

<!--SNIPSTART go-versioning-cutover-->
[go-sandbox/workflows/versioning/pizza/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/pizza/workflow.go)
```go
func PizzaWorkflow(ctx workflow.Context, order PizzaOrder) (OrderConfirmation, error) {
	// this function contains the original code
	return OrderConfirmation{OrderNumber: order.OrderNumber, Status: "SUCCESS"}, nil
}

func PizzaWorkflowV2(ctx workflow.Context, order PizzaOrder) (OrderConfirmation, error) {
	// this function contains the updated code
	return OrderConfirmation{OrderNumber: order.OrderNumber, Status: "CONFIRMED"}, nil
}
```
<!--SNIPEND-->

You can use any name you like for the new function, so long as the first character remains uppercase (this is a requirement for any Workflow Definition, since it must use an exported function).
Using some type of version identifier, such as V2 in this example, will make it easier to identify the change.

You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types:

<!--SNIPSTART go-versioning-cutover-register-->
[go-sandbox/workflows/versioning/worker/main.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/versioning/worker/main.go)
```go
w.RegisterWorkflow(pizza.PizzaWorkflow)
w.RegisterWorkflow(pizza.PizzaWorkflowV2)
```
<!--SNIPEND-->

The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow.
This can become impractical over time.
This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching.

## Runtime checking 

The Temporal Go SDK performs a runtime check to help prevent obvious incompatible changes.
Adding, removing, or reordering any of these methods without Versioning triggers the runtime check and results in a nondeterminism error:

- `workflow.ExecuteActivity()`
- `workflow.ExecuteChildWorkflow()`
- `workflow.NewTimer()`
- `workflow.RequestCancelWorkflow()`
- `workflow.SideEffect()`
- `workflow.SignalExternalWorkflow()`
- `workflow.Sleep()`

The runtime check does not perform a thorough check.
For example, it does not check on the Activity's input arguments or the Timer duration.
Each Temporal SDK implements these sanity checks differently, and they are not a complete check for non-deterministic changes.
Instead, you should incorporate [Replay Testing](/develop/go/best-practices/testing-suite#replay) when making revisions.

---

# Workflow Streams - Go SDK

Source: https://docs.temporal.io/develop/go/workflows/workflow-streams

> Stream events from a Workflow to subscribers using the Temporal Go SDK Workflow Streams contrib module.

> **Public Preview**

[Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its progress in real time. This page walks through enabling a stream, publishing events from Workflows and Activities, subscribing to a stream, and keeping a stream running across long-lived Workflows.

## Enable streaming on a Workflow

The library ships as `go.temporal.io/sdk/contrib/workflowstreams`. Enable streaming by constructing a `WorkflowStream` once at the top of your Workflow function, before any blocking call. Construction must happen there because the stream's handlers have to be registered before the first publish Signal arrives. Doing it after a blocking call would miss any publishes that arrived before the run body resumed.

<!--SNIPSTART go-streams-order-workflow {"selectedLines": ["3-14", "21-25", "58"]}-->
[go-sandbox/workflows/streams/order.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/order.go)
```go
// ...
import (
	"time"

	"go.temporal.io/sdk/contrib/workflowstreams"
	"go.temporal.io/sdk/workflow"
)

type OrderInput struct {
	OrderID     string
	StreamState *workflowstreams.WorkflowStreamState
}

// ...
func OrderWorkflow(ctx workflow.Context, input OrderInput) error {
	stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState)
	if err != nil {
		return err
	}
// ...
}
```
<!--SNIPEND-->

`NewWorkflowStream` creates the in-memory event log and registers the publish Signal, subscribe Update, and offset Query handlers on the current Workflow. The `priorState` argument is `nil` on a fresh start and a `*WorkflowStreamState` after a Continue-As-New rollover (see [Stream from long-running Workflows](#continue-as-new)).

Construct exactly one `WorkflowStream` per Workflow. The constructor re-registers the handlers unconditionally on every call, so constructing more than one on the same Workflow registers duplicate handlers. The Go SDK doesn't expose an inspection API for existing handlers, so the library can't return an error on a duplicate the way the Python SDK does.

## Publish from a Workflow

Bind a [topic name](/workflow-streams#topics) with `stream.Topic(name)`, then call `Publish()` on the returned `*WorkflowTopicHandle` to append events. Repeated calls with the same name return the same handle.

<!--SNIPSTART go-streams-order-workflow {"selectedLines": ["15-58"]}-->
[go-sandbox/workflows/streams/order.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/order.go)
```go
// ...
type StatusEvent struct {
	State    string
	Progress int
	Detail   string
}

func OrderWorkflow(ctx workflow.Context, input OrderInput) error {
	stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState)
	if err != nil {
		return err
	}
	status := stream.Topic("status")
	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{StartToCloseTimeout: time.Minute})

	if err := status.Publish(StatusEvent{State: "validating", Detail: "checking inventory"}); err != nil {
		return err
	}
	if err := workflow.ExecuteActivity(ctx, ValidateOrder, input.OrderID).Get(ctx, nil); err != nil {
		return err
	}

	if err := status.Publish(StatusEvent{State: "charging", Progress: 33, Detail: "authorizing payment"}); err != nil {
		return err
	}
	if err := workflow.ExecuteActivity(ctx, ChargePayment, input.OrderID).Get(ctx, nil); err != nil {
		return err
	}

	if err := status.Publish(StatusEvent{State: "shipping", Progress: 66, Detail: "dispatching to warehouse"}); err != nil {
		return err
	}
	if err := workflow.ExecuteActivity(ctx, DispatchOrder, input.OrderID).Get(ctx, nil); err != nil {
		return err
	}

	if err := status.Publish(StatusEvent{State: "completed", Progress: 100}); err != nil {
		return err
	}
	// Give in-flight subscriber polls time to fetch the terminator before the Workflow returns.
	if err := workflow.Sleep(ctx, 30*time.Second); err != nil {
		return err
	}
	return nil
}
```
<!--SNIPEND-->

`Publish()` runs the payload converter to encode each value. The codec chain (encryption, compression, etc.) runs once on the Signal or Update envelope that carries the batch, never per item, so encryption and compression are applied exactly once each direction.

Unlike the Python and TypeScript SDKs, Go topics carry no per-topic type binding. A topic handle is bound only to a name; published values are `any` and subscribers decode each item from its raw payload (see [Subscribe](#subscribe)). To customize per-item serialization, pass `workflowstreams.WithPayloadConverters(...)` to `NewWorkflowStream`, and use the matching converters on the subscriber side.

## Publish from a client

Any process that has a Temporal Client and the target Workflow ID can publish to that Workflow's stream by constructing a `Client`. This is the general pattern and covers HTTP backends, starters, one-off scripts, other Workflows' Activities, and standalone Activities.

Construct one with:

<!--SNIPSTART go-streams-new-client-->
[go-sandbox/workflows/streams/publish.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/publish.go)
```go
streamClient := workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{})
```
<!--SNIPEND-->

Then use it the same way you would the Workflow-side handle: bind a topic, publish through it, and call `Close(ctx)` when you're done publishing. `Close` flushes the buffer, so check its error to confirm the final batch was delivered.

When events originate in an Activity, publish from the Activity directly rather than returning them for the Workflow to forward. The Workflow hosts the stream but doesn't read its own stream; it processes the Activity's return value and emits its own lifecycle events. Keeping Workflow state independent of streamed output is what lets retried Activity attempts surface to subscribers without polluting the Workflow's durable state. See [How events are delivered](/workflow-streams#how-events-are-delivered).

<!--SNIPSTART go-streams-publish-from-client-->
[go-sandbox/workflows/streams/publish.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/publish.go)
```go
func PublishStatus(ctx context.Context, temporalClient client.Client, workflowID string) error {
	streamClient := workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{
		BatchInterval: 200 * time.Millisecond,
	})

	status := streamClient.Topic("status")
	status.Publish(StatusEvent{State: "started"}, false)
	// ...
	// Close flushes the buffer. Check its error to know the events were delivered.
	return streamClient.Close(ctx)
}
```
<!--SNIPEND-->

Inside an Activity scheduled by a Workflow, `workflowstreams.NewClientFromActivity()` infers the Temporal Client and the parent Workflow ID from the Activity context, so you don't have to thread them through the Activity's input:

<!--SNIPSTART go-streams-publish-from-activity-->
[go-sandbox/workflows/streams/stream_deltas.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/stream_deltas.go)
```go
import (
	"context"

	"go.temporal.io/sdk/activity"
	"go.temporal.io/sdk/contrib/workflowstreams"
)

func StreamDeltas(ctx context.Context, orderID string) error {
	streamClient, err := workflowstreams.NewClientFromActivity(ctx, workflowstreams.Options{})
	if err != nil {
		return err
	}

	deltas := streamClient.Topic("delta")
	for delta := range generateDeltas(orderID) {
		deltas.Publish(delta, false)
		activity.RecordHeartbeat(ctx)
	}
	// Close flushes the buffer. Return its error so a failed flush fails the Activity.
	return streamClient.Close(ctx)
}
```
<!--SNIPEND-->

For a [standalone Activity](/develop/go/activities) (one started directly via the Client rather than from a Workflow), there is no parent Workflow context to infer, so `NewClientFromActivity()` returns an error. Fall back to the general pattern with `activity.GetClient(ctx)` and the target Workflow ID threaded through the Activity's input.

Two operations give the application explicit control over when batches ship: the `forceFlush` argument on a publish for latency, and `client.Flush(ctx)` for confirmation that prior publications have landed.

Pass `true` as the `forceFlush` argument on a publish to wake the background flusher so the current buffer ships without waiting for the next interval. The flusher only runs while the client is open (between construction and `Close`). The call returns immediately after appending to the buffer and signaling the flusher. It doesn't wait for delivery to the Workflow or to subscribers:

<!--SNIPSTART go-streams-force-flush-->
[go-sandbox/workflows/streams/publish.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/publish.go)
```go
deltas.Publish(delta, true)
```
<!--SNIPEND-->

Use it for latency-sensitive events: the first delta of a response so the user sees something fast, or punctuated events like `RETRY` and `STATUS_CHANGE`. See [Tuning](/workflow-streams#tuning) for the trade-off against history pressure.

Use `client.Flush(ctx)` when you need a mid-stream barrier. Successful completion of the flush is proof that the Temporal server has received all prior publications, so subsequent work that depends on those events being durable can proceed. The client stays open for further publishing afterward. `Close` already flushes on its way out, so the explicit call is only for barriers in the middle:

<!--SNIPSTART go-streams-flush-barrier-->
[go-sandbox/workflows/streams/publish.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/publish.go)
```go
streamClient := workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{})
defer func() {
	if closeErr := streamClient.Close(ctx); err == nil {
		err = closeErr
	}
}()
deltas := streamClient.Topic("delta")

for _, delta := range firstPhase() {
	deltas.Publish(delta, false)
}

if err := streamClient.Flush(ctx); err != nil {
	return err
}
checkpointID, err := recordPhaseOneComplete(ctx) // only safe once phase-one events are durable
if err != nil {
	return err
}

for _, delta := range secondPhase(checkpointID) {
	deltas.Publish(delta, false)
}
```
<!--SNIPEND-->

`Publish()` is non-blocking and applies no backpressure. From an Activity or other client, it appends to the client's in-memory buffer and returns. From inside a Workflow, it appends synchronously to the in-memory log. [Subscribers](/workflow-streams#subscribing) pull from the Workflow's log on their own schedule, so a slow subscriber doesn't slow down [publishers](/workflow-streams#publishing). If a publisher emits faster than batches can ship to the server, the buffer grows: the process uses more memory, the stream falls further behind real time, and at the limit Signals can't keep up.

If your application needs to bound this (to cap memory, to keep the stream close to real time, or to apply a policy when the publisher overruns the network), apply that policy upstream of `Publish()`. The choice (block, drop, error, sample) is application-specific, and Workflow Streams doesn't pick one for you.

## Subscribe

[Subscribing](/workflow-streams#subscribing) uses the same client construction as publishing: `workflowstreams.NewClient(temporalClient, workflowID, opts)` from any process that has a Temporal Client, or `NewClientFromActivity()` inside an Activity. Subscribing from an Activity is less common in practice, so the general client case is the primary example below.

Once you have a client, range over `client.Subscribe()`, the counterpart to `Publish()`. It returns an [`iter.Seq2`](https://pkg.go.dev/iter#Seq2) iterator that yields a `WorkflowStreamItem` and an error on each step. Each item's `Data` is the raw payload; decode it at the call site with a payload converter.

<!--SNIPSTART go-streams-subscribe-->
[go-sandbox/workflows/streams/subscribe.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/subscribe.go)
```go
package streams

import (
	"context"
	"fmt"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/workflowstreams"
	"go.temporal.io/sdk/converter"
)

func WatchOrder(ctx context.Context, temporalClient client.Client, orderID string) error {
	stream := workflowstreams.NewClient(temporalClient, orderID, workflowstreams.Options{})

	for item, err := range stream.Subscribe(ctx, workflowstreams.SubscribeOptions{Topics: []string{"status"}}) {
		if err != nil {
			return err
		}
		var evt StatusEvent
		if err := converter.GetDefaultDataConverter().FromPayload(item.Data, &evt); err != nil {
			return err
		}
		fmt.Printf("[%3d%%] %s: %s\n", evt.Progress, evt.State, evt.Detail)
		if evt.State == "completed" {
			break
		}
	}
	return nil
}
```
<!--SNIPEND-->

`SubscribeOptions` controls the subscription: `Topics` filters by name (empty or nil means all topics), `FromOffset` resumes from a stored global offset (zero means the beginning), and `PollCooldown` sets the minimum interval between polls. The iterator handles re-polling, pagination when a poll response hits the ~1 MB cap, and Workflow-side log truncation transparently. A single-topic convenience method, `streamClient.Topic("status").Subscribe(ctx, fromOffset)`, is equivalent to passing one name in `Topics`.

A subscriber doesn't need `Close()` because the background flusher only runs for publishers.

### Heterogeneous topics

Every item arrives as a raw `*commonpb.Payload` in `item.Data`, so a single subscription naturally consumes multiple topics whose payload types differ. Pass the topic names in `SubscribeOptions.Topics` (or leave it empty for every topic on the stream), dispatch on `item.Topic`, and decode into the matching type:

<!--SNIPSTART go-streams-heterogeneous-topics-->
[go-sandbox/workflows/streams/subscribe.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/subscribe.go)
```go
for item, err := range stream.Subscribe(ctx, workflowstreams.SubscribeOptions{Topics: []string{"status", "progress"}}) {
	if err != nil {
		return err
	}
	switch item.Topic {
	case "status":
		var evt StatusEvent
		if err := converter.GetDefaultDataConverter().FromPayload(item.Data, &evt); err != nil {
			return err
		}
		fmt.Printf("[status] %s: %s\n", evt.State, evt.Detail)
		if evt.State == "completed" {
			return nil
		}
	case "progress":
		var evt ProgressEvent
		if err := converter.GetDefaultDataConverter().FromPayload(item.Data, &evt); err != nil {
			return err
		}
		fmt.Printf("[progress] %s\n", evt.Message)
	}
}
```
<!--SNIPEND-->

A single iterator over multiple topics also avoids the cancellation race that two concurrent subscribers would create. Because `item.Data` is the raw payload, it's also the right shape when you want to forward the bytes through to another system without decoding them.

### Closing the stream 

A subscriber's `for ... range` doesn't know when the publisher is done. How you [close a stream](/workflow-streams#closing-the-stream) depends on what the application needs. As one example, a common pattern combines two pieces:

1. **An in-band terminator.** The Workflow or its Activity publishes a sentinel event the subscriber recognizes and breaks on. In the `WatchOrder` example above, `StatusEvent{State: "completed"}` is the minimal form, and the consumer's `if evt.State == "completed" { break }` is the matching half. Each subscription decides what its own end-of-stream marker is.
2. **A brief overlap before the Workflow returns.** A poll Update that is still in flight when the Workflow returns is surfaced to the iterator and consumed silently, and no new polls can complete after that. If the Workflow returns immediately after publishing the terminator, subscribers may miss it.

There are two ways to provide that overlap.

- [Fixed sleep](/workflow-streams#fixed-sleep). Sleep between the terminator and the return so any in-flight poll has time to fetch the terminator before the Workflow exits:

  <!--SNIPSTART go-streams-order-workflow {"selectedLines": ["50-57"]}-->
  [go-sandbox/workflows/streams/order.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/order.go)
  ```go
  // ...
  	if err := status.Publish(StatusEvent{State: "completed", Progress: 100}); err != nil {
  		return err
  	}
  	// Give in-flight subscriber polls time to fetch the terminator before the Workflow returns.
  	if err := workflow.Sleep(ctx, 30*time.Second); err != nil {
  		return err
  	}
  	return nil
  ```
  <!--SNIPEND-->
- [Acknowledgment handshake](/workflow-streams#acknowledgment-handshake). The subscriber sends a Signal once it has the terminator; the Workflow waits up to a timeout, returning as soon as the ack arrives:

  <!--SNIPSTART go-streams-chat-workflow {"selectedLines": ["15-26", "35-40"]}-->
  [go-sandbox/workflows/streams/chat/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/chat/workflow.go)
  ```go
  // ...
  func ChatWorkflow(ctx workflow.Context, input ChatInput) (string, error) {
  	// The Workflow hosts the stream. The Activity publishes to it.
  	if _, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState); err != nil {
  		return "", err
  	}

  	subscriberDone := false
  	ackCh := workflow.GetSignalChannel(ctx, "subscriber-acknowledged-terminator")
  	workflow.Go(ctx, func(ctx workflow.Context) {
  		ackCh.Receive(ctx, nil)
  		subscriberDone = true
  	})
  // ...
  	// Wait for the subscriber to ack the terminal `close` event. The timeout
  	// is a fallback for when no subscriber is attached; with the ack, the
  	// typical case exits as soon as the subscriber confirms.
  	_, _ = workflow.AwaitWithTimeout(ctx, 30*time.Second, func() bool { return subscriberDone })
  	return result, nil
  }
  ```
  <!--SNIPEND-->

The full pattern is wired into the [Stream LLM output](#stream-llm-output) example below.

You can [inspect the terminal status](/workflow-streams#inspecting-terminal-status). `Subscribe()` ends cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELED`, `TERMINATED`, or `TIMED_OUT`, but doesn't distinguish among them. If your application needs to know which (to display success or failure to the user, log the outcome, or decide whether to retry), call `temporalClient.DescribeWorkflowExecution(ctx, workflowID, "")` after the loop returns to inspect the Workflow's status.

## Stream from long-running Workflows 

Workflows that run for hours or accumulate thousands of events need to periodically roll over via [Continue-As-New](/workflow-streams#stream-from-long-running-workflows) to keep history bounded. Subscribers automatically follow these rollovers. To keep a stream running across them without subscribers seeing a gap, carry both your application state and the stream state across the boundary. Add a `*WorkflowStreamState` field to your Workflow input, pass it to `NewWorkflowStream`, and return `stream.NewContinueAsNewError(ctx, wfn, buildArgs)` to invoke the rollover. The helper drains waiting subscribers, waits for in-flight handlers to finish, then returns a Continue-As-New error built from the args produced by `buildArgs(postDrainState)`:

<!--SNIPSTART go-streams-continue-as-new-->
[go-sandbox/workflows/streams/longrunning.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/longrunning.go)
```go
type WorkflowInput struct {
	ItemsProcessed int
	StreamState    *workflowstreams.WorkflowStreamState
}

func LongRunningWorkflow(ctx workflow.Context, input WorkflowInput) error {
	stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState)
	if err != nil {
		return err
	}
	itemsProcessed := input.ItemsProcessed

	for {
		if err := doOneIteration(ctx, stream); err != nil {
			return err
		}
		itemsProcessed++

		if workflow.GetInfo(ctx).GetContinueAsNewSuggested() {
			return stream.NewContinueAsNewError(ctx, LongRunningWorkflow, func(state *workflowstreams.WorkflowStreamState) []any {
				return []any{WorkflowInput{
					ItemsProcessed: itemsProcessed,
					StreamState:    state,
				}}
			})
		}
	}
}
```
<!--SNIPEND-->

The `*WorkflowStreamState` field is `nil` on a fresh start and a populated snapshot after a rollover. The `buildArgs` callback receives the post-detach `*WorkflowStreamState` as its only argument, so the snapshot is guaranteed to happen *after* pollers detach.

To pass other Continue-As-New parameters such as a different task queue, or to use a custom publisher TTL, use the explicit recipe instead. Drain the pollers, wait for handlers to finish, snapshot the state with your chosen TTL, then build the Continue-As-New error yourself (set options such as the task queue on the context first):

<!--SNIPSTART go-streams-continue-as-new-explicit-->
[go-sandbox/workflows/streams/longrunning.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/longrunning.go)
```go
stream.DetachPollers()
if err := workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }); err != nil {
	return err
}
state, err := stream.GetState(30 * time.Minute) // custom publisher TTL
if err != nil {
	return err
}
ctx = workflow.WithWorkflowTaskQueue(ctx, "other-tq")
return workflow.NewContinueAsNewError(ctx, LongRunningWorkflow, WorkflowInput{
	ItemsProcessed: itemsProcessed,
	StreamState:    state,
})
```
<!--SNIPEND-->

The carried `WorkflowStreamState` includes the entire in-memory log of the previous run, so streams that carry large items can hit Temporal's per-payload size limit at the rollover. Offload the bytes via [External Storage](/external-storage) so each item is a small reference rather than the full payload, and combine that with `stream.Truncate(upToOffset)` to keep the carried log itself small.

## Deduplication window

See [How events are delivered](/workflow-streams#how-events-are-delivered) for more details on subscriber and publisher behavior. See [Tuning](/workflow-streams#tuning) for more details on how to change your settings to meet the requirements for your Workflow Streams.

There are two limits on the [deduplication window](/workflow-streams#deduplication-window) worth highlighting:

- **Publisher TTL.** At each Continue-As-New, deduplicate entries whose last-seen time is older than this are dropped. The last-seen time is updated on each *successful* publish (not on each retry attempt), so a publisher that retries through a long partition without success can still age out. A publisher that returns after a longer pause may produce a duplicate. `stream.NewContinueAsNewError(...)` snapshots with a 15-minute default; to tune it, use the explicit recipe above and pass your value to `GetState(publisherTTL)`.
- **`MaxRetryDuration`.** A `Client` retries a failed batch for up to this long (default 10 minutes). If the duration elapses with the batch still pending, the client gives up, the pending batch is dropped, and a `FlushTimeoutError` is raised.

  <!--SNIPSTART go-streams-max-retry-duration-->
  [go-sandbox/workflows/streams/publish.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/publish.go)
  ```go
  streamClient := workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{
  	MaxRetryDuration: 10 * time.Minute,
  })
  ```
  <!--SNIPEND-->

  On timeout, the dropped batch is at-most-once: it may or may not have reached the Workflow. One operational caveat: the `FlushTimeoutError` is raised from inside the background flusher and terminates it. Until you call `client.Flush(ctx)` or `client.Close(ctx)`, subsequent publishes accumulate in the buffer with no flusher to ship them. `MaxRetryDuration` must be less than the workflow's publisher TTL to preserve exactly-once delivery.

## Best practices

There are a few details to note if you're writing custom message handlers or testing the library's capabilities:

- **Construct exactly one `WorkflowStream` per Workflow.** The constructor re-registers the publish Signal, poll Update, and offset Query handlers on every call, so a second construction registers duplicate handlers. Construct it once at the top of the Workflow function.
- **`item.Data` is always the raw payload.** Decode it with a converter built from the same `PayloadConverters` used by the publisher. When publishers and subscribers both rely on the defaults, `converter.GetDefaultDataConverter()` matches on both sides. If you pass `WithPayloadConverters` on the Workflow side, build a matching `converter.NewCompositeDataConverter(...)` on the subscriber side.
- **The codec chain runs once on the envelope.** Payload codecs (encryption, compression) configured on the Temporal client run on the Signal or Update envelope that carries each batch, never per item, so items are never double-encoded. `PayloadConverters` handle only per-item serialization.
- **The client publish path isn't goroutine-safe.** The client buffer is mutated on the publish path and read from the background flusher. Don't call `Publish()` on the same `Client` from multiple goroutines without coordinating; route events to a single owner.

## Example: Stream LLM output 

The headline use case fits the publish/subscribe shapes documented above. An Activity calls the model and publishes deltas as they arrive. The Workflow starts the Activity and waits for the consumer to acknowledge end-of-stream. The consumer subscribes, accumulates the deltas, and clears its accumulated state on `RETRY` before continuing. The shape works for a terminal client, a desktop UI, or a Server-Sent Events (SSE) endpoint forwarding to a browser. Anything that holds the displayed state calls `render()` to display it.

If your Activity can retry, the consumer side has to account for it. A retried attempt is a fresh publisher, so its output appears in the stream alongside the output from the previous attempt. In the LLM streaming pattern below, that means the failed attempt's partial deltas and the retried attempt's full output both reach a subscribed UI unless the UI resets on a `RETRY` event. The example wires up that pattern. See [How events are delivered](/workflow-streams#how-events-are-delivered) for the precise guarantees.

**activity.go**

<!--SNIPSTART go-streams-chat-activity-->
[go-sandbox/workflows/streams/chat/activity.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/chat/activity.go)
```go
package chat

import (
	"context"
	"strings"
	"time"

	"go.temporal.io/sdk/activity"
	"go.temporal.io/sdk/contrib/workflowstreams"
)

type TextDelta struct {
	Text string
}

type RetryEvent struct {
	Attempt int32
}

func StreamCompletion(ctx context.Context, prompt string) (string, error) {
	attempt := activity.GetInfo(ctx).Attempt
	streamClient, err := workflowstreams.NewClientFromActivity(ctx, workflowstreams.Options{
		BatchInterval: 200 * time.Millisecond,
	})
	if err != nil {
		return "", err
	}

	deltas := streamClient.Topic("delta")
	retry := streamClient.Topic("retry")
	closeTopic := streamClient.Topic("close")

	// Tell consumers an earlier attempt's deltas are stale.
	if attempt > 1 {
		retry.Publish(RetryEvent{Attempt: attempt}, true)
	}

	var full []string
	first := true
	// generateDeltas wraps the model call and yields tokens as they arrive.
	// Disable provider-side retries; let Temporal own retry policy at the Activity layer.
	for token := range generateDeltas(prompt) {
		// forceFlush only on the first delta so the user sees something
		// immediately; subsequent deltas batch at the 200 ms interval.
		deltas.Publish(TextDelta{Text: token}, first)
		first = false
		full = append(full, token)
	}
	closeTopic.Publish(struct{}{}, false)
	// Close flushes the buffer. Return its error so a failed flush fails the Activity.
	if err := streamClient.Close(ctx); err != nil {
		return "", err
	}
	return strings.Join(full, ""), nil
}
```
<!--SNIPEND-->

**workflow.go**

<!--SNIPSTART go-streams-chat-workflow-->
[go-sandbox/workflows/streams/chat/workflow.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/chat/workflow.go)
```go
package chat

import (
	"time"

	"go.temporal.io/sdk/contrib/workflowstreams"
	"go.temporal.io/sdk/workflow"
)

type ChatInput struct {
	Prompt      string
	StreamState *workflowstreams.WorkflowStreamState
}

func ChatWorkflow(ctx workflow.Context, input ChatInput) (string, error) {
	// The Workflow hosts the stream. The Activity publishes to it.
	if _, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState); err != nil {
		return "", err
	}

	subscriberDone := false
	ackCh := workflow.GetSignalChannel(ctx, "subscriber-acknowledged-terminator")
	workflow.Go(ctx, func(ctx workflow.Context) {
		ackCh.Receive(ctx, nil)
		subscriberDone = true
	})

	ao := workflow.ActivityOptions{StartToCloseTimeout: 5 * time.Minute}
	ctx = workflow.WithActivityOptions(ctx, ao)
	var result string
	if err := workflow.ExecuteActivity(ctx, StreamCompletion, input.Prompt).Get(ctx, &result); err != nil {
		return "", err
	}

	// Wait for the subscriber to ack the terminal `close` event. The timeout
	// is a fallback for when no subscriber is attached; with the ack, the
	// typical case exits as soon as the subscriber confirms.
	_, _ = workflow.AwaitWithTimeout(ctx, 30*time.Second, func() bool { return subscriberDone })
	return result, nil
}
```
<!--SNIPEND-->

**consumer.go**

<!--SNIPSTART go-streams-chat-consumer-->
[go-sandbox/workflows/streams/chat/consumer.go](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/go-sandbox/workflows/streams/chat/consumer.go)
```go
package chat

import (
	"context"
	"strings"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/workflowstreams"
	"go.temporal.io/sdk/converter"
)

func StreamChat(ctx context.Context, temporalClient client.Client, chatID string) (string, error) {
	// Subscribe-only; no Close needed because the flusher only runs for publishers.
	stream := workflowstreams.NewClient(temporalClient, chatID, workflowstreams.Options{})
	dc := converter.GetDefaultDataConverter()
	var output []string

	render := func() {
		// ... display the accumulated output (terminal redraw, UI update, etc.)
	}

	for item, err := range stream.Subscribe(ctx, workflowstreams.SubscribeOptions{Topics: []string{"delta", "retry", "close"}}) {
		if err != nil {
			return "", err
		}
		switch item.Topic {
		case "retry":
			// Earlier attempt's deltas are stale; drop what we've shown.
			output = output[:0]
			render()
		case "delta":
			var delta TextDelta
			if err := dc.FromPayload(item.Data, &delta); err != nil {
				return "", err
			}
			output = append(output, delta.Text)
			render()
		case "close":
			// Acknowledge so the Workflow can return without waiting on the fallback timeout.
			if err := temporalClient.SignalWorkflow(ctx, chatID, "", "subscriber-acknowledged-terminator", nil); err != nil {
				return "", err
			}
			return strings.Join(output, ""), nil
		}
	}

	return strings.Join(output, ""), nil
}
```
<!--SNIPEND-->

A few choices in this shape are deliberate:

- The Activity is the publisher because it owns the non-deterministic LLM call. The Workflow processes only the Activity's return value, never reading its own stream. See [Publish from a client](#publish-from-a-client) for why.
- The Activity publishes a `RETRY` event when `activity.GetInfo(ctx).Attempt > 1`. This lets the UI respond appropriately to the failure, typically by clearing accumulated deltas before the next attempt's deltas arrive (see [How events are delivered](/workflow-streams#how-events-are-delivered)).
- Termination uses an *ack handshake*: the consumer signals the Workflow once it has received the `close` event, so the Workflow can return as soon as the subscriber confirms. The `AwaitWithTimeout` timeout is the fallback when no subscriber is attached (see [Closing the stream](#closing-the-stream) for the simpler fixed-sleep alternative).
- `forceFlush` is `true` only on the first delta and on the `RETRY` sentinel, where latency matters. Subsequent deltas batch at the 200 ms `BatchInterval`; per-delta `forceFlush` would generate one Signal per token (see [Tuning](/workflow-streams#tuning) for the trade-off).

## See also

- [Workflow Streams samples (samples-go)](https://github.com/temporalio/samples-go/tree/main/workflowstreams): runnable scenarios covering basic publish/subscribe, reconnecting subscribers, external publishers, bounded logs, and LLM streaming.
- [`workflowstreams` API reference](https://pkg.go.dev/go.temporal.io/sdk/contrib/workflowstreams).
- [Workflow message passing](/develop/go/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on.

---

# Java SDK developer guide

Source: https://docs.temporal.io/develop/java

> Explore Temporal Java SDK feature guides to master developing Temporal Applications. Build Workflows, Activities, and Workers, connect to Temporal Services, set up a testing suite, handle failure detection, send messages, complete Activities asynchronously, implement Versioning, use Observability, debug applications, schedule Workflows

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Install and get started

You can find detailed installation instructions for the Java SDK in the [Quickstart](/develop/java/set-up-your-local-java).

There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running.

Once your local Temporal Service is set up, continue building with the following resources:

- [Develop a Workflow](/develop/java/workflows/basics)
- [Develop an Activity](/develop/java/activities/basics)
- [Start an Activity Execution](/develop/java/activities/execution)
- [Run a Worker](/develop/java/workers/run-worker-process)

From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases.

## [Workflows](/develop/java/workflows)

- [Workflow basics](/develop/java/workflows/basics)
- [Child Workflows](/develop/java/workflows/child-workflows)
- [Continue-As-New](/develop/java/workflows/continue-as-new)
- [Message passing](/develop/java/workflows/message-passing)
- [Cancellation](/develop/java/workflows/cancellation)
- [Timeouts](/develop/java/workflows/timeouts)
- [Schedules](/develop/java/workflows/schedules)
- [Timers](/develop/java/workflows/timers)
- [Side effects](/develop/java/workflows/side-effects)
- [Versioning](/develop/java/workflows/versioning)
- [Workflow Streams](/develop/java/workflows/workflow-streams)

## [Activities](/develop/java/activities)

- [Activity basics](/develop/java/activities/basics)
- [Activity Execution](/develop/java/activities/execution)
- [Standalone Activities](/develop/java/activities/standalone-activities-quickstart)
- [Timeouts](/develop/java/activities/timeouts)
- [Asynchronous Activity Completion](/develop/java/activities/asynchronous-activity)
- [Benign exceptions](/develop/java/activities/benign-exceptions)

## [Workers](/develop/java/workers)

- [Run a Worker](/develop/java/workers/run-worker-process)
- [Serverless Workers](/develop/java/workers/serverless-workers)
- [Observability](/develop/java/platform/observability)

## [Temporal Client](/develop/java/client)

- [Temporal Client](/develop/java/client/temporal-client)
- [Namespaces](/develop/java/client/namespaces)

## [Temporal Nexus](/develop/java/nexus)

- [Quickstart](/develop/java/nexus/quickstart)
- [Feature guide](/develop/java/nexus/feature-guide)
- [Standalone Operations](/develop/java/nexus/standalone-operations)
- [Microservice Development Walkthrough](/develop/java/nexus/development-walkthrough)

## [Platform](/develop/java/platform)

- [Observability](/develop/java/platform/observability)
- [Enriching the UI](/develop/java/platform/enriching-ui)

## [Best practices](/develop/java/best-practices)

- [Testing](/develop/java/best-practices/testing-suite)
- [Debugging](/develop/java/best-practices/debugging)
- [Data handling](/develop/java/best-practices/data-handling)

## [Integrations](/develop/java/integrations)

- [Parseable integration](https://github.com/parseablehq/temporal-plugin-java/blob/main/INTEGRATION.md)
- [Spring AI integration](/develop/java/integrations/spring-ai)
- [Spring Boot integration](/develop/java/integrations/spring-boot-integration)

## Temporal Java technical resources

- [Java SDK Quickstart - Setup Guide](/develop/java/set-up-your-local-java)
- [Java API Documentation](https://javadoc.io/doc/io.temporal/temporal-sdk)
- [Java SDK Code Samples](https://github.com/temporalio/samples-java)
- [Java SDK GitHub](https://github.com/temporalio/sdk-java)
- [Temporal 101 in Java Free Course](https://learn.temporal.io/courses/temporal_101/java/)
- [Build a Job Queue with Standalone Activities Tutorial](https://learn.temporal.io/tutorials/java/standalone-activities/)

## Get connected with the Temporal Java community

- [Temporal Java Community Slack](https://temporalio.slack.com/archives/CTT84KXK9)
- [Java SDK Forum](https://community.temporal.io/tag/java-sdk)

---

# Activities - Java SDK

Source: https://docs.temporal.io/develop/java/activities

> Activity Definitions, Activity Execution, Timeouts, Standalone Activities, and benign exceptions in the Temporal Java SDK.

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Activities

- [Activity basics](/develop/java/activities/basics)
- [Activity Execution](/develop/java/activities/execution)
- [Standalone Activities quickstart](/develop/java/activities/standalone-activities-quickstart)
- [Standalone Activities feature guide](/develop/java/activities/standalone-activities)
- [Timeouts](/develop/java/activities/timeouts)
- [Asynchronous Activity Completion](/develop/java/activities/asynchronous-activity)
- [Benign exceptions](/develop/java/activities/benign-exceptions)

---

# Asynchronous Activity Completion - Java SDK

Source: https://docs.temporal.io/develop/java/activities/asynchronous-activity

> Asynchronously complete an Activity in a Workflow with Temporal. Follow steps to provide identifying information, use Temporal Client, and set the complete() method.

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The Temporal Client is used to Heartbeat and complete the Activity.

To complete an Activity asynchronously, set the [`ActivityCompletionClient`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityCompletionClient.html) interface to the `complete()` method.

```java
    @Override
    public String composeGreeting(String greeting, String name) {

      // Get the activity execution context
      ActivityExecutionContext context = Activity.getExecutionContext();

      // Set a correlation token that can be used to complete the activity asynchronously
      byte[] taskToken = context.getTaskToken();

      /**
       * For the example we will use a {@link java.util.concurrent.ForkJoinPool} to execute our
       * activity. In real-life applications this could be any service. The composeGreetingAsync
       * method is the one that will actually complete workflow action execution.
       */
      ForkJoinPool.commonPool().execute(() -> composeGreetingAsync(taskToken, greeting, name));
      context.doNotCompleteOnReturn();

      // Since we have set doNotCompleteOnReturn(), the workflow action method return value is
      // ignored.
      return "ignored";
    }

    // Method that will complete action execution using the defined ActivityCompletionClient
    private void composeGreetingAsync(byte[] taskToken, String greeting, String name) {
      String result = greeting + " " + name + "!";

      // Complete our workflow activity using ActivityCompletionClient
      completionClient.complete(taskToken, result);
    }
  }
```

Alternatively, set the [`doNotCompleteOnReturn()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityExecutionContext.html#doNotCompleteOnReturn()) method during an Activity Execution.

```java
    @Override
    public String composeGreeting(String greeting, String name) {

      // Get the activity execution context
      ActivityExecutionContext context = Activity.getExecutionContext();

      // Set a correlation token that can be used to complete the activity asynchronously
      byte[] taskToken = context.getTaskToken();

      /**
       * For the example we will use a {@link java.util.concurrent.ForkJoinPool} to execute our
       * activity. In real-life applications this could be any service. The composeGreetingAsync
       * method is the one that will actually complete workflow action execution.
       */
      ForkJoinPool.commonPool().execute(() -> composeGreetingAsync(taskToken, greeting, name));
      context.doNotCompleteOnReturn();

      // Since we have set doNotCompleteOnReturn(), the workflow action method return value is
      // ignored.
      return "ignored";
    }
```

When this method is called during an Activity Execution, the Activity Execution does not complete when its method returns.

---

# Activity basics - Java SDK

Source: https://docs.temporal.io/develop/java/activities/basics

> Implement an Activity behind an annotated Java interface, define its parameters and return values, and control Activity Type names with prefixes.

## Develop an Activity

One of the primary things that Workflows do is orchestrate the execution of Activities.
An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file.
An Activity can interact with the world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service.
For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition).

[Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/java/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/java/activities/standalone-activities-quickstart).

An [Activity Definition](/activities) is a combination of the Temporal Java SDK [Activity](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/Activity.html) Class implementing a specially annotated interface.

An Activity interface is annotated with `@ActivityInterface` and an Activity implementation implements this Activity interface.
To handle Activity types that do not have an explicitly registered handler, you can directly implement a dynamic Activity.

```java
@ActivityInterface
public interface GreetingActivities {
    String composeGreeting(String greeting, String language);
}
```

Each method defined in the Activity interface defines a separate Activity method.
You can annotate each method in the Activity interface with the `@ActivityMethod` annotation, but this is completely optional.
The following example uses the `@ActivityMethod` annotation for the method defined in the previous example.

```java
@ActivityInterface
public interface GreetingActivities {
    @ActivityMethod
    String composeGreeting(String greeting, String language);
}
```

An Activity implementation is a Java class that implements an Activity annotated interface.

```java
// Implementation for the GreetingActivities interface example from in the previous section
 static class GreetingActivitiesImpl implements GreetingActivities {
    @Override
    public String composeGreeting(String greeting, String name) {
      return greeting + " " + name + "!";
    }
  }
```

## Define Activity parameters 

There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support.
However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB.
And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB.

Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance.
This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task).

Some SDKs require that you pass context objects, others do not.
When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities.
This is so that you can change what data is passed to the Activity without breaking a function or method signature.

An Activity interface can have any number of parameters.
All inputs should be serializable by the default Jackson JSON Payload Converter.

When implementing Activities, be mindful of the amount of data that you transfer using the Activity invocation parameters or return values as these are recorded in the Workflow Execution Events History.
Large Event Histories can adversely impact performance.

You can create a custom object, and pass it to the Activity interface, as shown in the following example.

```java
@ActivityInterface
public interface YourActivities {
    String getCustomObject(CustomObj customobj);
    void sendCustomObject(CustomObj customobj, String abc);
}
```

The `execute` method in the dynamic Activity interface implementation takes in `EncodedValues` that are inputs to the Activity Execution, as shown in the following example.

```java
// Dynamic Activity implementation
 public static class DynamicActivityImpl implements DynamicActivity {
   @Override
   public Object execute(EncodedValues args) {
     String activityType = Activity.getExecutionContext().getInfo().getActivityType();
     return activityType
         + ": "
         + args.get(0, String.class)
         + " "
         + args.get(1, String.class)
         + " from: "
         + args.get(2, String.class);
   }
 }
```

For more details, see [Dynamic Activity Reference](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/DynamicActivity.html).

## Define Activity return values 

All data returned from an Activity must be serializable.

Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/evaluate/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history).

Activity return values must be serializable and deserializable by the provided [`DataConverter`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html).

The `execute` method for `DynamicActivity` can return type Object.
Ensure that your Workflow or Client can handle an Object type return or is able to convert the Object type response.

- [Data Converter](/dataconversion)
- Java DataConverter reference: [https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html)

## Customize your Activity Type 

Each Activity has a Type, which may also be referred to as the Activity 'name'.
This name appears in the Workflow Execution Event History in the Summary tab for each Activity Task.
The name lets you identify Activity Types called during the Execution.

Custom Activity Type names prevent name collisions across interfaces and Workflows.
They offer descriptive Activity method names without concerns about re-using those names elsewhere in your project.
They also support code management, especially in larger projects with many Activities.
For example, you might use a prefix to group related activities together.
Custom names also distinguish keys for gathering metrics without name conflicts.

The following examples set custom names for your Activity Type.

### Default behavior

By default, an Activity Type is the method name with the first letter capitalized:

```java
@ActivityInterface
public interface GreetingActivities {
    String sendMessage(String input);

    @ActivityMethod
    String composeGreeting(String greeting, String language);
}
```

- Method Name: `sendMessage`
- Activity Type: `SendMessage`
- Method Name: `composeGreeting`
- Activity Type: `ComposeGreeting`

### Custom prefix

Using the `namePrefix` parameter in the `@ActivityInterface` annotation adds a prefix to each Activity Type name mentioned in the interface, unless the prefix is specifically overridden:

```java
@ActivityInterface(namePrefix = "Messaging_")
public interface GreetingActivities {
    String sendMessage(String input);

    @ActivityMethod
    String composeGreeting(String greeting, String language);
}
```

- Method Name: `sendMessage`
- Activity Type: `Messaging_SendMessage`
- Method Name: `composeGreeting`
- Activity Type: `Messaging_ComposeGreeting`

The Activity Type is capitalized, even when using a prefix.

### Custom name

To override the default name and any inherited prefixes, use the `name` parameter in the `@ActivityMethod` annotation:

```java
@ActivityInterface(namePrefix = "Messaging_")
public interface GreetingActivities {
    String sendMessage(String input);

    @ActivityMethod
    String composeGreeting(String greeting, String language);
    @ActivityMethod(name = "farewell")
    String composeFarewell(String farewell, String language);
}
```

Using the `name` parameter won't automatically capitalize the result:

- Method Name: `sendMessage`
- Activity Type: `Messaging_SendMessage`
- Method Name: `composeGreeting`
- Activity Type: `Messaging_ComposeGreeting`
- Method Name: `composeFarewell`
- Activity Type: `farewell`

Be cautious with names that contain special characters, as these can be used as metric tags.
Systems such as Prometheus may ignore metrics with tags using unsupported characters.

---

# Benign exceptions - Java SDK

Source: https://docs.temporal.io/develop/java/activities/benign-exceptions

> Mark expected or non-severe Activity errors as benign in Java to reduce noise in logs, metrics, and OpenTelemetry traces.

When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues.
By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic.

To mark an error as benign, set the category to `ApplicationErrorCategory.BENIGN` using the [`ApplicationFailure`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ApplicationFailure.html) builder.

Benign errors:
- Have Activity failure logs downgraded to DEBUG level
- Do not emit Activity failure metrics
- Do not set the OpenTelemetry failure status to ERROR

```java
import io.temporal.activity.ActivityInterface;
import io.temporal.activity.ActivityMethod;
import io.temporal.failure.ApplicationErrorCategory;
import io.temporal.failure.ApplicationFailure;

@ActivityInterface
public interface MyActivities {
  @ActivityMethod
  String myActivity();
}

public class MyActivitiesImpl implements MyActivities {
  @Override
  public String myActivity() {
    try {
      return callExternalService();
    } catch (Exception e) {
      // Mark this error as benign since it's expected
      throw ApplicationFailure.newBuilder()
          .setMessage(e.getMessage())
          .setType(e.getClass().getName())
          .setCause(e)
          .setCategory(ApplicationErrorCategory.BENIGN)
          .build();
    }
  }
}
```

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried.

---

# Activity Execution - Java SDK

Source: https://docs.temporal.io/develop/java/activities/execution

> Invoke Activities synchronously or asynchronously from a Java Workflow, set ActivityOptions and Timeouts, and read the ActivityExecutionContext.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition).
The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command.
This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History.

A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations.
Activity implementation code should be _idempotent_.

The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history.
The entire Execution history is transferred from the Temporal Service to Workflow Workers when a Workflow state needs to recover.
A large Execution history can thus adversely impact the performance of your Workflow.

Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values.
Otherwise, no additional limitations exist on Activity implementations.

Activities are remote procedure calls that must be invoked from within a Workflow using `ActivityStub`.
Activities are not executable on their own. You cannot start an Activity Execution by itself.

Note that before an Activity Execution is invoked:

- Activity options (either [`setStartToCloseTimeout`](/encyclopedia/detecting-activity-failures#start-to-close-timeout) or [`ScheduleToCloseTimeout`](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) are required) must be set for the Activity.
  For details, see [Activity Timeouts](/develop/java/activities/timeouts).
- The Activity must be registered with a Worker.
  See [Worker Program](/develop/java/workers/run-worker-process#run-a-dev-worker)
- Activity code must be thread-safe.

Activities should only be instantiated using stubs from within a Workflow.
An `ActivityStub` returns a client-side stub that implements an Activity interface.
You can invoke Activities using `Workflow.newActivityStub`(type-safe) or `Workflow.newUntypedActivityStub` (untyped).

Calling a method on the Activity interface schedules the Activity invocation with the Temporal Service, and generates an [`ActivityTaskScheduled` Event](/references/events#activitytaskscheduled).

Activities can be invoked synchronously or asynchronously.

### Invoke Activities synchronously 

In the following example, we use the type-safe `Workflow.newActivityStub` within the "FileProcessingWorkflow" Workflow implementation to create a client-side stub of the `FileProcessingActivities` class. We also define `ActivityOptions` and set `setStartToCloseTimeout` option to one hour.

```java
public class FileProcessingWorkflowImpl implements FileProcessingWorkflow {

    private final FileProcessingActivities activities;

    public FileProcessingWorkflowImpl() {
        this.activities = Workflow.newActivityStub(
                FileProcessingActivities.class,
                ActivityOptions.newBuilder()
                        .setStartToCloseTimeout(Duration.ofHours(1))
                        .build());
    }

    @Override
    public void processFile(Arguments args) {
        String localName = null;
        String processedName = null;
        try {
            localName = activities.download(args.getSourceBucketName(), args.getSourceFilename());
            processedName = activities.processFile(localName);
            activities.upload(args.getTargetBucketName(), args.getTargetFilename(), processedName);
        } finally {
            if (localName != null) {
                activities.deleteLocalFile(localName);
            }
            if (processedName != null) {
                activities.deleteLocalFile(processedName);
            }
        }
    }
    // ...
}
```

A Workflow can have multiple Activity stubs. Each Activity stub can have its own `ActivityOptions` defined.
The following example shows a Workflow implementation with two typed Activity stubs.

```java
public FileProcessingWorkflowImpl() {
    ActivityOptions options1 = ActivityOptions.newBuilder()
             .setTaskQueue("taskQueue1")
             .setStartToCloseTimeout(Duration.ofMinutes(10))
             .build();
    this.store1 = Workflow.newActivityStub(FileProcessingActivities.class, options1);

    ActivityOptions options2 = ActivityOptions.newBuilder()
             .setTaskQueue("taskQueue2")
             .setStartToCloseTimeout(Duration.ofMinutes(5))
             .build();
    this.store2 = Workflow.newActivityStub(FileProcessingActivities.class, options2);
}
```

To invoke Activities inside Workflows without referencing the interface it implements, use an untyped Activity stub `Workflow.newUntypedActivityStub`.
This is useful when the Activity type is not known at compile time, or to invoke Activities implemented in different programming languages.

```java
   // Workflow code
    ActivityOptions activityOptions =
        ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(3))
        .setTaskQueue("simple-queue-node")
        .build();

    ActivityStub activity = Workflow.newUntypedActivityStub(activityOptions);
    activity.execute("ComposeGreeting", String.class, "Hello World", "Spanish");
```

### Invoke Activities asynchronously 

Sometimes Workflows need to perform certain operations in parallel.
The Temporal Java SDK provides the `Async` class which includes static methods used to invoke any Activity asynchronously.
The calls return a result of type `Promise` which is similar to the Java `Future` and `CompletionStage`.
When invoking Activities, use `Async.function` for Activities that return a result, and `Async.procedure` for Activities that return void.

In the following asynchronous Activity invocation, the method reference is passed to `Async.function` followed by Activity arguments.

```java
Promise<String> localNamePromise = Async.function(activities::download, sourceBucket, sourceFile);
```

The following example calls two Activity methods, "download" and "upload", in parallel on multiple files.

```java
  public void processFile(Arguments args) {
    List<Promise<String>> localNamePromises = new ArrayList<>();
    List<String> processedNames = null;
    try {
      // Download all files in parallel.
      for (String sourceFilename : args.getSourceFilenames()) {
        Promise<String> localName =
            Async.function(activities::download, args.getSourceBucketName(), sourceFilename);
        localNamePromises.add(localName);
      }
      List<String> localNames = new ArrayList<>();
      for (Promise<String> localName : localNamePromises) {
        localNames.add(localName.get());
      }
      processedNames = activities.processFiles(localNames);

      // Upload all results in parallel.
      List<Promise<Void>> uploadedList = new ArrayList<>();
      for (String processedName : processedNames) {
        Promise<Void> uploaded =
            Async.procedure(
                activities::upload,
                args.getTargetBucketName(),
                args.getTargetFilename(),
                processedName);
        uploadedList.add(uploaded);
      }
      // Wait for all uploads to complete.
      Promise.allOf(uploadedList).get();
    } finally {
      for (Promise<String> localNamePromise : localNamePromises) {
        // Skip files that haven't completed downloading.
        if (localNamePromise.isCompleted()) {
          activities.deleteLocalFile(localNamePromise.get());
        }
      }
      if (processedNames != null) {
        for (String processedName : processedNames) {
          activities.deleteLocalFile(processedName);
        }
      }
    }
  }
```

### ActivityExecutionContext 

`ActivityExecutionContext` is a context object passed to each Activity implementation by default.
You can access it in your Activity implementations via `Activity.getExecutionContext()`.

It provides getters to access information about the Workflow that invoked the Activity.
Note that the Activity context information is stored in a thread-local variable.
Therefore, calls to `getExecutionContext()` succeed only within the thread that invoked the Activity function.

Following is an example of using the `ActivityExecutionContext`:

```java
public class FileProcessingActivitiesImpl implements FileProcessingActivities {

  @Override
  public String download(String bucketName, String remoteName, String localName) {

    ActivityExecutionContext ctx = Activity.getExecutionContext();
    ActivityInfo info = ctx.getInfo();

    log.info("namespace=" +  info.getActivityNamespace());
    log.info("workflowId=" + info.getWorkflowId());
    log.info("runId=" + info.getRunId());
    log.info("activityId=" + info.getActivityId());
    log.info("activityTimeout=" + info.getStartToCloseTimeout();

    return downloadFileFromS3(bucketName, remoteName, localDirectory + localName);
  }
    ...
}
```

For details on getting the results of an Activity Execution, see [Activity Execution Result](#activity-execution-result).

## Set the required Activity Timeouts 

Activity Execution semantics rely on several parameters.
The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).
These values are set in the Activity Options.

Set your Activity Timeout from the [`ActivityOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html) class.

Available timeouts are:

- ScheduleToCloseTimeout()
- ScheduleToStartTimeout()
- StartToCloseTimeout()

You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker.

The following uses `ActivityStub`.

```java
GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class,
                ActivityOptions.newBuilder()
                        .setScheduleToCloseTimeout(Duration.ofSeconds(5))
                        // .setStartToCloseTimeout(Duration.ofSeconds(2)
                        // .setScheduletoCloseTimeout(Duration.ofSeconds(20))
                        .build());
```

The following uses `WorkflowImplementationOptions`.

```java
WorkflowImplementationOptions options =
            WorkflowImplementationOptions.newBuilder()
                    .setActivityOptions(
                            ImmutableMap.of(
                                    "GetCustomerGreeting",
                                    // Set Activity Execution timeout
                                    ActivityOptions.newBuilder()
                                        .setScheduleToCloseTimeout(Duration.ofSeconds(5))
                                        // .setStartToCloseTimeout(Duration.ofSeconds(2))
                                        // .setScheduleToStartTimeout(Duration.ofSeconds(5))
                                        .build()))
                    .build();
```

> **📝 Note:**
>
> If you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting.
>

## Java ActivityOptions reference 

Use [`ActivityOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html) to configure an Activity Execution.

You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker.
Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting.

The following table lists all `ActivityOptions` that can be configured for an Activity invocation.

| Option                                                 | Required                                           | Type                     |
| ------------------------------------------------------ | -------------------------------------------------- | ------------------------ |
| [`setScheduleToCloseTimeout`](#scheduletoclosetimeout) | Yes (if `StartToCloseTimeout` is not specified)    | Duration                 |
| [`setScheduleToStartTimeout`](#scheduletostarttimeout) | No                                                 | Duration                 |
| [`setStartToCloseTimeout`](#starttoclosetimeout)       | Yes (if `ScheduleToCloseTimeout` is not specified) | Duration                 |
| [`setHeartbeatTimeout`](#heartbeattimeout)             | No                                                 | Duration                 |
| [`setTaskQueue`](#taskqueue)                           | No                                                 | String                   |
| [`setRetryOptions`](#retryoptions)                     | No                                                 | RetryOptions             |
| [`setCancellationType`](#setcancellationtype)          | No                                                 | ActivityCancellationType |

### ScheduleToCloseTimeout

To set a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout), use [`ActivityOptions.newBuilder.setScheduleToCloseTimeout​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html).

This or `StartToCloseTimeout` must be set.

- Type: `Duration`
- Default: Unlimited.
  Note that if `WorkflowRunTimeout` and/or `WorkflowExecutionTimeout` are defined in the Workflow, all Activity retries will stop when either or both of these timeouts are reached.

- With `ActivityStub`

  ```java
  GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class,
                  ActivityOptions.newBuilder()
                          .setScheduleToCloseTimeout(Duration.ofSeconds(5))
                          .build());
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
              WorkflowImplementationOptions.newBuilder()
                      .setActivityOptions(
                              ImmutableMap.of(
                                      "GetCustomerGreeting",
                                      ActivityOptions.newBuilder()
                                          .setScheduleToCloseTimeout(Duration.ofSeconds(5))
                                          .build()))
                      .build();
  ```

### ScheduleToStartTimeout

To set a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout), use [`ActivityOptions.newBuilder.setScheduleToStartTimeout​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html).

- Type: `Duration`
- Default: Unlimited. This timeout is non-retryable.

- With `ActivityStub`

  ```java
  GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class,
                  ActivityOptions.newBuilder()
                          .setScheduleToStartTimeout(Duration.ofSeconds(5))
                          // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
                          // required when setting Activity options.
                          .setScheduletoCloseTimeout(Duration.ofSeconds(20))
                          .build());
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
             WorkflowImplementationOptions.newBuilder()
                      .setActivityOptions(
                              ImmutableMap.of(
                                "GetCustomerGreeting",
                                ActivityOptions.newBuilder()
                                    .setScheduleToStartTimeout(Duration.ofSeconds(5))
                                    .build()))
                      .build();
  ```

### StartToCloseTimeout

To set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout), use [`ActivityOptions.newBuilder.setStartToCloseTimeout​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html).

This or `ScheduleToClose` must be set.

- Type: `Duration`
- Default: Defaults to [`ScheduleToCloseTimeout`](#scheduletoclosetimeout) value

- With `ActivityStub`

  ```java
  GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class,
              ActivityOptions.newBuilder()
                      .setStartToCloseTimeout(Duration.ofSeconds(2))
                      .build());
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
              WorkflowImplementationOptions.newBuilder()
                      .setActivityOptions(
                              ImmutableMap.of(
                                "EmailCustomerGreeting",
                                      ActivityOptions.newBuilder()
                                            // Set Activity Execution timeout (single run)
                                            .setStartToCloseTimeout(Duration.ofSeconds(2))
                                            .build()))
                      .build();
  ```

### HeartbeatTimeout

To set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), use [`ActivityOptions.newBuilder.setHeartbeatTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html).

- Type: `Duration`
- Default: None

- With `ActivityStub`

  ```java
  private final GreetingActivities activities =
      Workflow.newActivityStub(
          GreetingActivities.class,
          ActivityOptions.newBuilder()
              // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
              // required when setting Activity options.
              .setStartToCloseTimeout(Duration.ofSeconds(5))
              .setHeartbeatTimeout(Duration.ofSeconds(2))
              .build());
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
              WorkflowImplementationOptions.newBuilder()
                      .setActivityOptions(
                              ImmutableMap.of(
                                "EmailCustomerGreeting",
                                      ActivityOptions.newBuilder()
                                          // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
                                          // required when setting Activity options.
                                            .setStartToCloseTimeout(Duration.ofSeconds(5))
                                            .setHeartbeatTimeout(Duration.ofSeconds(2))
                                            .build()))
                      .build();
  ```

### TaskQueue

- Type: `String`
- Default: Defaults to the Task Queue that the Workflow was started with.

- With `ActivityStub`

  ```java
  GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class,
                  ActivityOptions.newBuilder()
                          // note that either StartToCloseTimeout or ScheduleToCloseTimeout are required when
                          // setting Activity options.
                          .setStartToCloseTimeout(Duration.ofSeconds(5))
                          .setTaskQueue("yourTaskQueue")
                          .build());
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
              WorkflowImplementationOptions.newBuilder()
                      .setActivityOptions(
                              ImmutableMap.of(
                                "EmailCustomerGreeting",
                                      ActivityOptions.newBuilder()
                                            // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
                                            // required when setting Activity options.
                                            .setStartToCloseTimeout(Duration.ofSeconds(5))
                                            .setTaskQueue("yourTaskQueue")
                                            .build()))
                      .build();
  ```

See [Task Queue](/task-queue)

### RetryOptions

To set a Retry Policy, known as the [Retry Options](/encyclopedia/retry-policies) in Java, use [`ActivityOptions.newBuilder.setRetryOptions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html).

- Type: `RetryOptions`
- Default: Server-defined Activity Retry policy.

- With `ActivityStub`

  ```java
  private final ActivityOptions options =
      ActivityOptions.newBuilder()
          // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
          // required when setting Activity options.
          .setStartToCloseTimeout(Duration.ofSeconds(5))
          .setRetryOptions(
              RetryOptions.newBuilder()
                  .setInitialInterval(Duration.ofSeconds(1))
                  .setMaximumInterval(Duration.ofSeconds(10))
                  .build())
          .build();
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
          WorkflowImplementationOptions.newBuilder()
                 .setActivityOptions(
                      ImmutableMap.of(
                          "EmailCustomerGreeting",
                          ActivityOptions.newBuilder()
                                // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
                                // required when setting Activity options.
                                .setStartToCloseTimeout(Duration.ofSeconds(5))
                                .setRetryOptions(
                                      RetryOptions.newBuilder()
                                          .setDoNotRetry(NullPointerException.class.getName())
                                          .build())
                                .build()))
                .build();
  ```

### setCancellationType

- Type: `ActivityCancellationType`
- Default: `ActivityCancellationType.TRY_CANCEL`

- With `ActivityStub`

  ```java
  private final GreetingActivities activities =
    Workflow.newActivityStub(
        GreetingActivities.class,
        ActivityOptions.newBuilder()
            .setCancellationType(ActivityCancellationType.WAIT_CANCELLATION_COMPLETED)
            .build());
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
          WorkflowImplementationOptions.newBuilder()
                 .setActivityOptions(
                      ImmutableMap.of(
                          "EmailCustomerGreeting",
                          ActivityOptions.newBuilder()
                                .setCancellationType(ActivityCancellationType.WAIT_CANCELLATION_COMPLETED)
                                .build()))
                .build();
  ```

## Get the results of an Activity Execution 

The call to spawn an [Activity Execution](/activity-execution) generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable.
Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available.

To get the results of an asynchronously invoked Activity method, use the `Promise` `get` method to block until the Activity method result is available.

Sometimes an Activity Execution lifecycle goes beyond a synchronous method invocation.
For example, a request can be put in a queue and later a reply comes and is picked up by a different Worker process.
The whole request-reply interaction can be modeled as a single Activity.

To indicate that an Activity should not be completed upon its method return, call `ActivityExecutionContext.doNotCompleteOnReturn()` from the original Activity thread.

Then later, when replies come, complete the Activity using the `ActivityCompletionClient`.
To correlate Activity invocation with completion, use either a `TaskToken` or Workflow and Activity Ids.

Following is an example of using `ActivityExecutionContext.doNotCompleteOnReturn()`:

```java
public class FileProcessingActivitiesImpl implements FileProcessingActivities {

  public String download(String bucketName, String remoteName, String localName) {

    ActivityExecutionContext ctx = Activity.getExecutionContext();

    // Used to correlate reply
    byte[] taskToken = ctx.getInfo().getTaskToken();

    asyncDownloadFileFromS3(taskToken, bucketName, remoteName, localDirectory + localName);
    ctx.doNotCompleteOnReturn();

    // Return value is ignored when doNotCompleteOnReturn was called.
    return "ignored";
  }
  ...
}
```

When the download is complete, the download service potentially can complete the Activity, or fail it from a different process, for example:

```java
  public <R> void completeActivity(byte[] taskToken, R result) {
    completionClient.complete(taskToken, result);
  }

  public void failActivity(byte[] taskToken, Exception failure) {
    completionClient.completeExceptionally(taskToken, failure);
  }
```

---

# Standalone Activities feature guide - Java SDK

Source: https://docs.temporal.io/develop/java/activities/standalone-activities

> Execute Activities independently without a Workflow using the Temporal Java SDK.

[Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated by a
Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a Temporal Client using `ActivityClient`.

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/java/activities/basics). The only difference is that you execute a Standalone
Activity directly from your Temporal Client.

> **💡 Tip:**
>
> New to Standalone Activities? Start with the [Standalone Activities quickstart](/develop/java/activities/standalone-activities-quickstart).
>

This page covers the following:

- [Start a Standalone Activity without waiting for the result](#start-activity)
- [Get a handle to an existing Standalone Activity](#get-activity-handle)
- [Wait for the result of a Standalone Activity](#get-activity-result)
- [List Standalone Activities](#list-activities)
- [Count Standalone Activities](#count-activities)
- [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from the
> [standaloneactivities](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities)
> sample.
>

## Start a Standalone Activity without waiting for the result 

Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue
your Activity job, without waiting for it to be executed by your Worker.

Use
[`ActivityClient.start()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html)
to start a Standalone Activity and get a handle without waiting for the result:

[StartActivity.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/StartActivity.java)

```java
ActivityHandle<String> handle =
    client.start(
        GreetingActivities.class,
        GreetingActivities::composeGreeting,
        options,
        "Hello",
        "World");
System.out.println("Started activity ID: " + ACTIVITY_ID);

// Wait for the result later
String result = handle.getResult();
System.out.println("Activity result: " + result);
```

With the Temporal Server and Worker running, open a new terminal in the `samples-java` directory and
run:

```bash
./gradlew -q :core:execute -PmainClass=io.temporal.samples.standaloneactivities.StartActivity
```

Or use the Temporal CLI:

```bash
./temporal activity start \
  --type ComposeGreeting \
  --activity-id standalone-activity-id \
  --task-queue standalone-activity-task-queue \
  --start-to-close-timeout 10s \
  --input '"Hello"' \
  --input '"World"'
```

## Get a handle to an existing Standalone Activity 

Use `client.getHandle()` to create a typed handle to a previously started Standalone Activity:

```java
ActivityHandle<String> handle =
    client.getHandle("standalone-activity-id", null, String.class);
```

Pass `null` as the run ID to target the latest run of the given activity ID. You can then use the
handle to wait for the result, describe, cancel, or terminate the Activity.

## Wait for the result of a Standalone Activity 

Under the hood, calling `client.execute()` is the same as calling `client.start()` to durably
enqueue the Standalone Activity, and then calling `handle.getResult()` to block until the Activity
completes and return the result:

```java
String result = handle.getResult();
```

To wait asynchronously without blocking the calling thread, use `handle.getResultAsync()`, which
returns a `CompletableFuture<R>`:

```java
CompletableFuture<String> future = handle.getResultAsync();
```

Or use the Temporal CLI to wait for a result by Activity ID:

```bash
./temporal activity result --activity-id standalone-activity-id
```

## List Standalone Activities 

Use
[`client.listExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html)
to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is
a `Stream<ActivityExecutionMetadata>` that fetches pages from the server on demand as the stream is
consumed.

These APIs return only Standalone Activity Executions. Activities running inside Workflows are not
included.

[ListActivities.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/ListActivities.java)

```java
client
    .listExecutions("TaskQueue = '" + TASK_QUEUE + "'")
    .forEach(
        info ->
            System.out.printf(
                "ActivityID: %s, Type: %s, Status: %s%n",
                info.getActivityId(), info.getActivityType(), info.getStatus()));
```

Run it:

```bash
./gradlew -q :core:execute -PmainClass=io.temporal.samples.standaloneactivities.ListActivities
```

Or use the Temporal CLI:

```bash
./temporal activity list
```

The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow
Visibility](/visibility). For example, `ActivityType = 'composeGreeting' AND ExecutionStatus = 'Running'`.

## Count Standalone Activities 

Use
[`client.countExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html)
to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns
the total count of executions (running, completed, failed, etc.) — not the number of queued tasks.
It works the same way as counting Workflow Executions.

[CountActivities.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/CountActivities.java)

```java
ActivityExecutionCount resp = client.countExecutions("TaskQueue = '" + TASK_QUEUE + "'");
System.out.println("Total activities: " + resp.getCount());
resp.getGroups()
    .forEach(
        group ->
            System.out.println("Group " + group.getGroupValues() + ": " + group.getCount()));
```

Run it:

```bash
./gradlew -q :core:execute -PmainClass=io.temporal.samples.standaloneactivities.CountActivities
```

Or use the Temporal CLI:

```bash
./temporal activity count
```

## Run Standalone Activities with Temporal Cloud 

The Worker and Client code in the [Standalone Activities quickstart](/develop/java/activities/standalone-activities-quickstart)
use `ClientConfigProfile.load()`, so the same code works against Temporal Cloud — configure the
connection via environment variables or a TOML profile. No code changes are needed.

For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate
generation, and authentication setup in the Cloud UI, see
[Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud).

### Connect with mTLS

Set these environment variables with values from your Temporal Cloud Namespace settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem'
export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key'
```

### Connect with an API key

Set these environment variables with values from your Temporal Cloud API key settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_API_KEY=<your-api-key>
```

Then run the Worker and starter code as shown in the [Standalone Activities quickstart](/develop/java/activities/standalone-activities-quickstart).

---

# Standalone Activities Java quickstart - Java SDK

Source: https://docs.temporal.io/develop/java/activities/standalone-activities-quickstart

> Execute a Standalone Activity with the Temporal Java SDK without writing a Workflow.

# Quickstart

[Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated by a
Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a Temporal Client using `ActivityClient`.

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/java/activities/basics). The only difference is that you execute a Standalone
Activity directly from your Temporal Client.

> **📝 Note:**
>
> This documentation uses source code from the
> [standaloneactivities](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/standaloneactivities)
> sample.
>

## Get started with Standalone Activities 

Prerequisites:

- **Java** 8+

- **Temporal Java SDK** (v1.39.0 or higher). See the [Java Quickstart](/develop/java/set-up-your-local-java) for
  install instructions.

- **Temporal CLI** v1.9.1 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`.

Start the Temporal development server with `temporal server start-dev`.

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Server will now be available for client connections on `localhost:7233`, and the
Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233).

```bash
brew install temporal
```

```bash
temporal --version
```

```bash
temporal server start-dev
```

## Clone the sample

Clone the [samples-java](https://github.com/temporalio/samples-java) repository to follow along:

```bash
git clone https://github.com/temporalio/samples-java.git
cd samples-java
```

The sample consists of separate programs in the `standaloneactivities` package:

```
core/src/main/java/io/temporal/samples/standaloneactivities/
├── GreetingActivities.java       # Activity interface
├── GreetingActivitiesImpl.java   # Activity implementation
├── StandaloneActivityWorker.java # Worker that processes activity tasks
├── ExecuteActivity.java          # Starts an activity and waits for the result
├── StartActivity.java            # Starts an activity without blocking
├── ListActivities.java           # Lists activity executions
└── CountActivities.java          # Counts activity executions
```

## Define your Activity 

An Activity in the Temporal Java SDK is an interface annotated with `@ActivityInterface`, with
methods annotated with `@ActivityMethod`. The way you define a Standalone Activity is identical to
how you define an Activity orchestrated by a Workflow. In fact, the same Activity can be executed
both as a Standalone Activity and as a Workflow Activity.

[GreetingActivities.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/GreetingActivities.java)

[GreetingActivitiesImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/GreetingActivitiesImpl.java)

```java
@ActivityInterface
public interface GreetingActivities {

  @ActivityMethod
  String composeGreeting(String greeting, String name);
}
```

```java
public class GreetingActivitiesImpl implements GreetingActivities {

  private static final Logger log = LoggerFactory.getLogger(GreetingActivitiesImpl.class);

  @Override
  public String composeGreeting(String greeting, String name) {
    log.info("Composing greeting...");
    return greeting + ", " + name + "!";
  }
}
```

## Run a Worker with the Activity registered 

Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities —
you create a `WorkerFactory`, register the Activity implementation, and call `factory.start()`. The
Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone
Activity. See [Run a Worker](/develop/java/workers/run-worker-process) for more details on
Worker setup and configuration options.

[StandaloneActivityWorker.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/StandaloneActivityWorker.java)

Open a new terminal, navigate to the `samples-java` directory, and run the Worker.
Leave this terminal running — the Worker needs to stay up to process activities.

```java
ClientConfigProfile profile = ClientConfigProfile.load();
WorkflowServiceStubs service =
    WorkflowServiceStubs.newServiceStubs(profile.toWorkflowServiceStubsOptions());

WorkflowClient client = WorkflowClient.newInstance(service, profile.toWorkflowClientOptions());
WorkerFactory factory = WorkerFactory.newInstance(client);
Worker worker = factory.newWorker(TASK_QUEUE);
worker.registerActivitiesImplementations(new GreetingActivitiesImpl());
factory.start();
System.out.println("Worker running on task queue: " + TASK_QUEUE);
```

```bash
./gradlew -q :core:execute -PmainClass=io.temporal.samples.standaloneactivities.StandaloneActivityWorker
```

## Execute a Standalone Activity 

Use
[`ActivityClient.execute()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html)
to execute a Standalone Activity and block until it completes. Call this from your application code,
not from inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal
Server, waits for it to be executed on your Worker, and then returns the typed result.

[ExecuteActivity.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/ExecuteActivity.java)

The typed `execute()` API takes the Activity interface class and an unbound method reference. The SDK
uses the method reference to infer the Activity type name and result type at runtime. You can also
call Activities by string type name.

`StartActivityOptions` requires `id`, `taskQueue`, and at least one of `startToCloseTimeout` or
`scheduleToCloseTimeout`.

To run it:

1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above).
2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above).
3. Open a new terminal, navigate to the `samples-java` directory, and run:
    ```
    ./gradlew -q :core:execute -PmainClass=io.temporal.samples.standaloneactivities.ExecuteActivity
    ```

Or use the Temporal CLI.

```java
ActivityClient client =
    ActivityClient.newInstance(
        service,
        ActivityClientOptions.newBuilder().setNamespace(profile.getNamespace()).build());

StartActivityOptions options =
    StartActivityOptions.newBuilder()
        .setId(ACTIVITY_ID)
        .setTaskQueue(TASK_QUEUE)
        .setStartToCloseTimeout(Duration.ofSeconds(10))
        .build();

String result =
    client.execute(
        GreetingActivities.class,
        GreetingActivities::composeGreeting,
        options,
        "Hello",
        "World");
System.out.println("Activity result: " + result);
```

```java
// Using a string type name
String result = client.execute("ComposeGreeting", String.class, options, "Hello", "World");
```

```bash
./gradlew -q :core:execute -PmainClass=io.temporal.samples.standaloneactivities.ExecuteActivity
```

```bash
temporal activity execute \\
  --type ComposeGreeting \\
  --activity-id standalone-activity-id \\
  --task-queue standalone-activity-task-queue \\
  --start-to-close-timeout 10s \\
  --input '"Hello"' \\
  --input '"World"'
```

## Run with Temporal Cloud

All code samples on this page use
[`ClientConfigProfile.load()`](https://www.javadoc.io/doc/io.temporal/temporal-envconfig/latest/io/temporal/envconfig/ClientConfigProfile.html)
to configure the Temporal Client connection. It responds to [environment
variables](/references/client-environment-configuration) and [TOML configuration
files](/references/client-environment-configuration), so the same code works against a local dev
server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal
Cloud](/develop/java/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the feature guide
for mTLS and API key setup.

## Next steps

- **[Standalone Activities feature guide](/develop/java/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud.
- **[Build a Job Queue with Standalone Activities](https://learn.temporal.io/tutorials/java/standalone-activities/)**: Tutorial that builds a durable job queue, with idempotency keys, server-side deduplication, throughput limits, [Priority and Fairness](/develop/task-queue-priority-fairness), and heartbeat checkpoints.
- **[Activity basics](/develop/java/activities/basics)**: Write and register Activities with the Java SDK.

---

# Activity Timeouts - Java SDK

Source: https://docs.temporal.io/develop/java/activities/timeouts

> Set the Schedule-To-Close, Start-To-Close, and Heartbeat Timeouts on a Java Activity, tune its Retry Policy, and override the next retry delay.

## Activity timeouts

Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution.

The following timeouts are available in the Activity Options.

- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution).
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution).
- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

Set your Activity Timeout from the [`ActivityOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html) class.

Available timeouts are:

- ScheduleToCloseTimeout
- StartToCloseTimeout
- ScheduleToStartTimeout

You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker.

The following uses `ActivityStub`.

```java
GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class,
    ActivityOptions.newBuilder()
        .setScheduleToCloseTimeout(Duration.ofSeconds(5))
        // .setStartToCloseTimeout(Duration.ofSeconds(2)
        // .setScheduletoCloseTimeout(Duration.ofSeconds(20))
        .build());
```

The following uses `WorkflowImplementationOptions`.

```java
WorkflowImplementationOptions options =
    WorkflowImplementationOptions.newBuilder()
        .setActivityOptions(
            ImmutableMap.of(
                "GetCustomerGreeting",
                // Set Activity Execution timeout
                ActivityOptions.newBuilder()
                    .setScheduleToCloseTimeout(Duration.ofSeconds(5))
                    // .setStartToCloseTimeout(Duration.ofSeconds(2))
                    // .setScheduleToStartTimeout(Duration.ofSeconds(5))
                    .build()))
        .build();
```

> **📝 Note:**
>
> If you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting.
>

### Custom Activity Retry Policy 

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided.

To set a Retry Policy, known as the [Retry Options](/encyclopedia/retry-policies) in Java, use [`ActivityOptions.newBuilder.setRetryOptions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html).

- Type: `RetryOptions`
- Default: Server-defined Activity Retry policy.

- With `ActivityStub`

  ```java
  private final ActivityOptions options =
      ActivityOptions.newBuilder()
          // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
          // required when setting Activity options.
          .setStartToCloseTimeout(Duration.ofSeconds(5))
          .setRetryOptions(
              RetryOptions.newBuilder()
                  .setInitialInterval(Duration.ofSeconds(1))
                  .setMaximumInterval(Duration.ofSeconds(10))
                  .build())
          .build();
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
        WorkflowImplementationOptions.newBuilder()
            .setActivityOptions(
                ImmutableMap.of(
                    "EmailCustomerGreeting",
                    ActivityOptions.newBuilder()
                        // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
                        // required when setting Activity options.
                        .setStartToCloseTimeout(Duration.ofSeconds(5))
                        .setRetryOptions(
                            RetryOptions.newBuilder()
                                .setDoNotRetry(NullPointerException.class.getName())
                                .build())
                        .build()))
            .build();
  ```

### Activity next retry delay 

You may throw an [`ApplicationFailure`](/references/failures#application-failure) with the `NextRetryDelay` field set. This value will replace and override whatever the retry interval would be on the retry policy.

For example, if in an activity, you want to base the interval on the number of attempts, you might do:

```java
int attempt = Activity.getExecutionContext().getInfo().getAttempt();

throw ApplicationFailure.newFailureWithCauseAndDelay(
    "Something bad happened on attempt " + attempt,
    "my_failure_type",
    null,
    3 * Duration.ofSeconds(attempt));
```

## Heartbeat an Activity 

An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service).
Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed.
If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy.

Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation.
Heartbeat throttling may lead to Cancellation getting delivered later than expected.

Heartbeats can contain a `details` field describing the Activity's current progress.
If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service.

To Heartbeat an Activity Execution in Java, use the `Activity.getExecutionContext().heartbeat()` Class method.

```java
public class YourActivityDefinitionImpl implements YourActivityDefinition {

  @Override
  public String yourActivityMethod(YourActivityMethodParam param) {
    // ...
    Activity.getExecutionContext().heartbeat(details);
    // ...
  }
  // ...
}
```

The method takes an optional argument, the `details` variable above that represents latest progress of the Activity Execution.
This method can take a variety of types such as an exception object, custom object, or string.

If the Activity Execution times out, the last Heartbeat `details` are included in the thrown `ActivityTimeoutException`, which can be caught by the calling Workflow.
The Workflow can then use the `details` information to pass to the next Activity invocation if needed.

In the case of Activity retries, the last Heartbeat's `details` are available and can be extracted from the last failed attempt by using `Activity.getExecutionContext().getHeartbeatDetails(Class<V> detailsClass)`

After a Worker starts a graceful shutdown, `heartbeat()` throws `ActivityWorkerShutdownException` by default.
To keep Heartbeating while the Worker drains, see [Heartbeat during shutdown](/develop/java/workers/run-worker-process#heartbeat-during-shutdown).

### Heartbeat Timeout 

A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat).

To set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), use [`ActivityOptions.newBuilder.setHeartbeatTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html).

- Type: `Duration`
- Default: None

You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker.
Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting.

- With `ActivityStub`

  ```java
  private final GreetingActivities activities =
      Workflow.newActivityStub(
          GreetingActivities.class,
          ActivityOptions.newBuilder()
              // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
              // required when setting Activity options.
              .setStartToCloseTimeout(Duration.ofSeconds(5))
              .setHeartbeatTimeout(Duration.ofSeconds(2))
              .build());
  ```

- With `WorkflowImplementationOptions`

  ```java
  WorkflowImplementationOptions options =
        WorkflowImplementationOptions.newBuilder()
            .setActivityOptions(
                ImmutableMap.of(
                "EmailCustomerGreeting",
                    ActivityOptions.newBuilder()
                        // note that either StartToCloseTimeout or ScheduleToCloseTimeout are
                        // required when setting Activity options.
                        .setStartToCloseTimeout(Duration.ofSeconds(5))
                        .setHeartbeatTimeout(Duration.ofSeconds(2))
                        .build()))
            .build();
  ```

---

# Best practices - Java SDK

Source: https://docs.temporal.io/develop/java/best-practices

> This section explains how to implement best practices with the Java SDK

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Best practices

- [Error handling](/develop/java/best-practices/error-handling)
- [Testing](/develop/java/best-practices/testing-suite)
- [Debugging](/develop/java/best-practices/debugging)
- [Data handling](/develop/java/best-practices/data-handling)

---

# Data handling - Java SDK

Source: https://docs.temporal.io/develop/java/best-practices/data-handling

All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three
layers that handle different concerns:

![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)

Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON
serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when
your application requires non-JSON types, encryption, or payload offloading.

|                           | [PayloadConverter](/develop/java/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/java/best-practices/data-handling/data-encryption) | [External Storage](/develop/java/data-handling/external-storage) |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------------------- |
| **Purpose**               | Serialize application data to bytes                               | Transform encoded payloads (encrypt, compress)                | Offload large Payloads to an external store                     |
| **Default**               | JSON serialization                                                | None (passthrough)                                            | Disabled                                                        |

For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage).

---

# Payload conversion - Java SDK

Source: https://docs.temporal.io/develop/java/best-practices/data-handling/data-conversion

> Customize how Temporal serializes application objects using Payload Converters in the Java SDK.

Payload Converters serialize your application objects into a `Payload` and deserialize them back.
A `Payload` is a binary form with metadata that Temporal uses to transport data.

By default, Temporal uses a Payload Converter that handles `null`, byte arrays, protobuf messages, and anything JSON-serializable.
You only need a custom Payload Converter when your application uses types that aren't natively supported.

## Default supported types

The default Data Converter supports converting multiple types including:

- `null`
- Byte arrays
- Protobuf JSON: if a value is an instance of a Protobuf message, it is encoded with proto3 JSON
- [Jackson JSON](https://docs.spring.io/spring-framework/reference/web/webmvc/mvc-controller/ann-methods/jackson.html)
- Anything that can be converted to JSON

## Using custom Payload conversion 

Temporal SDKs provide a [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back.

Implementing custom Payload conversion is optional.
It is needed only if the [default Data Converter](/default-custom-data-converters#default-data-converter) does not support your custom values.

To support custom Payload conversion, create a [custom Payload Converter](/payload-converter#composite-data-converters) and configure the Data Converter to use it in your Client options.

The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter.
You can set multiple encoding Payload Converters to run your conversions.
When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

Payload Converters can be customized independently of a Payload Codec.
Temporal's Converter architecture looks like this:

![Temporal converter architecture](https://docs.temporal.io/img/info/converter-architecture.png)

Create a custom implementation of a [PayloadConverter](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/PayloadConverter.html) interface and use the `withPayloadConverterOverrides` method to implement the custom object conversion with `DefaultDataConverter`.

`PayloadConverter` serializes and deserializes method parameters that need to be sent over the wire.
You can create a custom implementation of `PayloadConverter` for custom formats, as shown in the following example:

```java
/** Payload Converter specific to your custom object */
public class YourCustomPayloadConverter implements PayloadConverter {
 //...
  @Override
  public String getEncodingType() {
    return "json/plain"; // The encoding type determines which default conversion behavior to override.
  }

  @Override
  public Optional<Payload> toData(Object value) throws DataConverterException {
      // Add your convert-to logic here.
  }

  @Override
  public <T> T fromData(Payload content, Class<T> valueClass, Type valueType)
      throws DataConverterException {
    // Add your convert-from logic here.
  }
//...
}
```

You can also use [specific implementation classes](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/package-summary.html) provided in the Java SDK.

For example, to create a custom `JacksonJsonPayloadConverter`, use the following:

```java
//...
private static JacksonJsonPayloadConverter yourCustomJacksonJsonPayloadConverter() {
  ObjectMapper objectMapper = new ObjectMapper();
  // Add your custom logic here.
  return new JacksonJsonPayloadConverter(objectMapper);
}
//...
```

To set your custom Payload Converter, use it with [withPayloadConverterOverrides](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DefaultDataConverter.html#withPayloadConverterOverrides(io.temporal.common.converter.PayloadConverter...)) with a new instance of `DefaultDataConverter` in your `WorkflowClient` options that you use in your Worker process and to start your Workflow Executions.

The following example shows how to set a custom `YourCustomPayloadConverter` Payload Converter.

```java
//...
DefaultDataConverter ddc =
        DefaultDataConverter.newDefaultInstance()
            .withPayloadConverterOverrides(new YourCustomPayloadConverter());

    WorkflowClientOptions workflowClientOptions =
        WorkflowClientOptions.newBuilder().setDataConverter(ddc).build();
//...
```

---

# Payload encryption - Java SDK

Source: https://docs.temporal.io/develop/java/best-practices/data-handling/data-encryption

> Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Java SDK.

Temporal's security model is designed around client-side encryption of Payloads.
A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server.
This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data.
It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures.

A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client.
Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these.

The server itself never adds encryption over Payloads.
Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows.
When working with sensitive data, you should always implement Payload encryption.

## Custom Payload Codec in Java 

Create a custom implementation of [`PayloadCodec`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/payload/codec/PayloadCodec.html) and use it in [`CodecDataConverter`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/CodecDataConverter.html) to set a custom Data Converter.

The Payload Codec does byte-to-byte conversion and must be set with a Data Converter.

Define custom encryption/compression logic in your `encode` method and decryption/decompression logic in your `decode` method.

The following example from the [Java encryption sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads/CryptCodec.java) shows how to implement encryption and decryption logic on your payloads in your `encode` and `decode` methods.

```java
class YourCustomPayloadCodec implements PayloadCodec {

  static final ByteString METADATA_ENCODING =
      ByteString.copyFrom("binary/encrypted", StandardCharsets.UTF_8);

  private static final String CIPHER = "AES/GCM/NoPadding";

  // Define constants that you can add to your encoded Payload to create a new Payload.
  static final String METADATA_ENCRYPTION_CIPHER_KEY = "encryption-cipher";

  static final ByteString METADATA_ENCRYPTION_CIPHER =
      ByteString.copyFrom(CIPHER, StandardCharsets.UTF_8);

  static final String METADATA_ENCRYPTION_KEY_ID_KEY = "encryption-key-id";

  private static final Charset UTF_8 = StandardCharsets.UTF_8;

  // See the linked sample for details on the methods called here.
  @NotNull
  @Override
  public List<Payload> encode(@NotNull List<Payload> payloads) {
    return payloads.stream().map(this::encodePayload).collect(Collectors.toList());
  }

  @NotNull
  @Override
  public List<Payload> decode(@NotNull List<Payload> payloads) {
    return payloads.stream().map(this::decodePayload).collect(Collectors.toList());
  }

  private Payload encodePayload(Payload payload) {
    String keyId = getKeyId();
    SecretKey key = getKey(keyId);

    byte[] encryptedData;
    try {
      encryptedData = encrypt(payload.toByteArray(), key); // The encrypt method contains your custom encryption logic.
    } catch (Throwable e) {
      throw new DataConverterException(e);
    }
    // Apply metadata to the encoded Payload that you can verify in your decode method before decoding.
    // See the sample for details on the metadata values set.
    return Payload.newBuilder()
        .putMetadata(EncodingKeys.METADATA_ENCODING_KEY, METADATA_ENCODING)
        .putMetadata(METADATA_ENCRYPTION_CIPHER_KEY, METADATA_ENCRYPTION_CIPHER)
        .putMetadata(METADATA_ENCRYPTION_KEY_ID_KEY, ByteString.copyFromUtf8(keyId))
        .setData(ByteString.copyFrom(encryptedData))
        .build();
  }

  private Payload decodePayload(Payload payload) {
    // Verify the incoming encoded Payload metadata before applying decryption.
    if (METADATA_ENCODING.equals(
        payload.getMetadataOrDefault(EncodingKeys.METADATA_ENCODING_KEY, null))) {
      String keyId;
      try {
        keyId = payload.getMetadataOrThrow(METADATA_ENCRYPTION_KEY_ID_KEY).toString(UTF_8);
      } catch (Exception e) {
        throw new PayloadCodecException(e);
      }
      SecretKey key = getKey(keyId);
      byte[] plainData;
      Payload decryptedPayload;

      try {
        plainData = decrypt(payload.getData().toByteArray(), key); // The decrypt method contains your custom decryption logic.
        decryptedPayload = Payload.parseFrom(plainData);
        return decryptedPayload;
      } catch (Throwable e) {
        throw new PayloadCodecException(e);
      }
    } else {
      return payload;
    }
  }

  private String getKeyId() {
    // Currently there is no context available to vary which key is used.
    // Use a fixed key for all payloads.
    // This still supports key rotation as the key ID is recorded on payloads allowing
    // decryption to use a previous key.

    return "test-key-test-key-test-key-test!";
  }

  private SecretKey getKey(String keyId) {
    // Key must be fetched from KMS or other secure storage.
    // Hard coded here only for example purposes.
    return new SecretKeySpec(keyId.getBytes(UTF_8), "AES");
  }

  //...
}
```

**Set Data Converter to use custom Payload Codec**

Use `CodecDataConverter` with an instance of a Data Converter and the custom `PayloadCodec` in the `WorkflowClient` options that you use in your Worker process and to start your Workflow Executions.

For example, to set a custom `PayloadCodec` implementation with `DefaultDataConverter`, use the following code:

```java
WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
  // Client that can be used to start and signal Workflows
  WorkflowClient client =
      WorkflowClient.newInstance(
          service,
          WorkflowClientOptions.newBuilder()
              .setDataConverter(
                  new CodecDataConverter(
                      DefaultDataConverter.newDefaultInstance(),
                      Collections.singletonList(new YourCustomPayloadCodec()))) // Sets the custom Payload Codec created in the previous example with an instance of the default Data Converter.
              .build());
```

- Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted.
- Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output.

For reference, see the [Encryption](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads) sample.

### Using a Codec Server

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints.
You create, operate, and manage access to your Codec Server in your own environment.
The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.
Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server.

For reference, see the [Codec server](https://github.com/temporalio/sdk-java/tree/main/temporal-remote-data-encoder) sample.

---

# Debugging - Java SDK

Source: https://docs.temporal.io/develop/java/best-practices/debugging

> Debug your Temporal Java Workflows using your favorite Java IDE's debugger. Set the TEMPORAL_DEBUG environment variable to true during debugging to avoid deadlocks. Use Web UI, Temporal CLI, and logging for development and production. Optimize Worker performance with metrics and the Worker performance guide.

In addition to writing unit and integration tests, debugging your Workflows is also a very valuable testing tool.
You can debug your Workflow code using a debugger provided by your favorite Java IDE.

Note that when debugging your Workflow code, the Temporal Java SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control.
Because of this you can often encounter the `PotentialDeadlockException` Exception while stepping through Workflow code during debugging.

To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to true before debugging your Workflow code. Make sure to set `TEMPORAL_DEBUG` to true only during debugging.

## How to debug in a development environment 

In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli).

## How to debug in a production environment 

You can debug production Workflows using:

- [Web UI](/web-ui)
- [Temporal CLI](/cli)
- [Replay](/develop/java/best-practices/testing-suite#replay)
- [Tracing](/develop/java/platform/observability#tracing)
- [Logging](/develop/java/platform/observability#logging)

You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance).
For more information, see [Observability ▶️ Metrics](/develop/java/platform/observability#metrics) for setting up SDK metrics.

Debug Server performance with [Cloud metrics](/cloud/metrics/openmetrics) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics).

---

# Error handling - Java SDK

Source: https://docs.temporal.io/develop/java/best-practices/error-handling

> Catch the right exception types, wrap checked exceptions, and inspect failures correctly in Temporal Java Workflows and Activities.

Temporal represents failures with a small set of typed exceptions that all extend
[`TemporalFailure`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TemporalFailure.html).
For what each type means and when the Temporal Service raises it, see the
[Temporal Failures reference](/references/failures).
This page covers the two things that Java Workflow and Activity code get wrong most often: catching a wider exception
type than intended, and manually adding checked exceptions to method signatures instead of using the SDK's wrapping
helpers.

## Catch `Exception`, never `Throwable` or `Error` 

Workflow and Activity code should only ever catch `Exception` or a narrower type.
Never catch `Throwable` or `Error`.

The Java SDK uses subclasses of `Error` as internal control signals that must reach the SDK's own code uncaught:

- [`DestroyWorkflowThreadError`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/sync/DestroyWorkflowThreadError.html)
  interrupts a Workflow thread so the Worker can release it back to the pool, for example when the Workflow Execution
  is evicted from the [Worker's cache](/develop/java/workers/run-worker-process). If Workflow code catches it, the
  thread doesn't unwind and eviction can stall.
- [`UnsupportedVersion`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/statemachines/UnsupportedVersion.html)
  is thrown by [`Workflow.getVersion()`](/develop/java/workflows/versioning) when replayed history was produced by
  code outside the version range the current Workflow code declares. It extends `Error` specifically so that
  application code won't catch it by mistake.

Use this rule of thumb when deciding what to catch in a Workflow, Update, or Signal handler:

1. **`Error`** — never catch it. If you must run cleanup on any exit path, use a
   [detached Cancellation Scope](/develop/java/workflows/cancellation) rather than a broad catch (see [Cancellation](#cancellation) below).
2. **`CanceledFailure`** — rethrow it, optionally after cleanup in a detached Cancellation Scope. Don't swallow it: cancellation
   is cooperative, and swallowing it lets the Workflow Execution finish as "Completed" instead of "Canceled."
3. **`ActivityFailure`, `ChildWorkflowFailure`, or `ApplicationFailure`** that you recognize and can recover from — handle it.
4. **Everything else** — rethrow it. A plain `RuntimeException` that isn't recognized above fails the current
   [Workflow Task](/tasks#workflow-task-execution), which retries indefinitely rather than failing the Workflow Execution. To fail the
   Workflow Execution deliberately, throw an `ApplicationFailure` (see [Fail a Workflow deliberately](#fail-a-workflow) below).

### Don't swallow failures with a broad catch 

A `catch (Throwable t)` (or `catch (Exception e)` that only logs and returns) placed around Workflow logic "to be
safe" is the most common way this rule gets broken:

```java
// Anti-pattern: never do this in Workflow, Update, or Signal code.
try {
  return doWorkflowLogic();
} catch (Throwable t) {
  logger.error("workflow failed", t);
  return defaultResult();
}
```

This single catch block causes three separate problems at once:

- `DestroyWorkflowThreadError` and `UnsupportedVersion` are swallowed instead of reaching the SDK, which can stall
  Worker cache eviction and interfere with replay.
- `CanceledFailure` is swallowed, so a canceled Workflow Execution reports "Completed" instead of "Canceled."
- Every other exception — including real bugs — disappears with only a log line instead of failing the Workflow
  Task or Workflow Execution, so there's no signal in the Event History that anything went wrong.

Catch the narrowest type you can actually recover from, log and rethrow anything you don't recognize, and let `Error`
and `CanceledFailure` propagate untouched:

```java
try {
  return doWorkflowLogic();
} catch (ActivityFailure e) {
  if (e.getCause() instanceof ApplicationFailure appFailure
      && "ValidationError".equals(appFailure.getType())) {
    return defaultResult();
  }
  throw e; // don't recognize it — propagate
}
```

## Wrap checked exceptions instead of adding them to method signatures 

Activity and Workflow method signatures shouldn't declare `throws` for checked exceptions.
Instead, wrap a checked exception with
[`Activity.wrap()`](<https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/Activity.html#wrap(java.lang.Throwable)>)
inside an Activity, or
[`Workflow.wrap()`](<https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#wrap(java.lang.Exception)>)
inside a Workflow, before rethrowing it:

```java
static class GreetingActivitiesImpl implements GreetingActivities {
  @Override
  public String composeGreeting(String greeting, String name) {
    try {
      return callExternalService(greeting, name); // declares `throws IOException`
    } catch (IOException e) {
      throw Activity.wrap(e);
    }
  }
}
```

`wrap()` only does something to a checked exception. Given that, you don't need to reason about what to pass it:

- If `e` is a checked exception, `wrap()` returns a `CheckedExceptionWrapper` around it. The SDK unwraps it
  automatically while propagating the failure and attaches the original exception as the `cause` of the resulting
  `ApplicationFailure` — so the caller still sees your original exception type and message in the cause chain.
- If `e` already extends `RuntimeException`, `wrap()` returns it unchanged. Calling `wrap()` on an unchecked
  exception is a safe no-op, so you don't need to check the exception type before calling it.
- If `e` extends `Error`, `wrap()` rethrows it directly instead of wrapping it, consistent with [never catching
  `Throwable` or `Error`](#catch-exception-not-throwable).

This also answers the two questions that come up most when working with wrapped exceptions:

- **Do you need to wrap exceptions you throw yourself?** Only checked exceptions need `wrap()`. Any unhandled
  exception an Activity or Workflow throws — checked or not — is already converted to an `ApplicationFailure`
  automatically when it crosses the Activity or Workflow boundary. `wrap()` exists to satisfy the Java compiler when
  you want to throw a checked exception from a method that doesn't declare it, not to make the exception propagate.
- **Do you need to re-wrap an exception after unwrapping it?** No. Once you've unwrapped a cause to inspect it,
  either rethrow the failure you caught (`throw e;`) or throw a new `ApplicationFailure` with the original exception
  set as its cause. There's no wrapper left to reapply — `wrap()` only matters at the point where a checked exception
  would otherwise need a `throws` declaration.

## Read a failure's cause chain 

An exception thrown from an Activity or Child Workflow arrives at the caller wrapped with context about where it
failed. A failure from an Activity called from a Child Workflow called from a parent Workflow looks like this by the
time it reaches a synchronous client call:

```text
WorkflowFailedException           (thrown to the client)
└─ ChildWorkflowFailure           (the child Workflow Execution failed)
   └─ ActivityFailure             (the Activity Execution failed)
      └─ ApplicationFailure       (what your code actually threw)
```

Each wrapper adds context — `ActivityFailure` carries the Activity Type and Activity Id, `ChildWorkflowFailure`
carries the Workflow Type and Workflow Id — while `getCause()` on each layer moves toward what actually failed. See
[Temporal Failures reference](/references/failures) for what each of these types means for retry behavior.

Two details matter once you reach an `ApplicationFailure`:

- **Read `getOriginalMessage()`, not `getMessage()`.** `getMessage()` returns a decorated string such as
  `message='Invalid credit card number', type='ValidationError', nonRetryable=true` — meant for logs, not parsing.
  `getOriginalMessage()` returns the exact text you threw.
- **Match on `getType()`, a stable `String`, not `instanceof` your original exception class.** `ApplicationFailure`
  is `final` and the original exception object doesn't survive serialization: when an Activity in another process
  (or another SDK language) throws, the caller only ever gets an `ApplicationFailure` back, never your custom
  exception type. `type` defaults to the thrown exception's fully qualified class name unless you set it explicitly
  with `ApplicationFailure.newFailure(message, type, ...)`.

```java
try {
  return activities.processCreditCard(orderId);
} catch (ActivityFailure e) {
  if (e.getCause() instanceof ApplicationFailure appFailure) {
    if ("ValidationError".equals(appFailure.getType())) {
      return Result.rejected(appFailure.getOriginalMessage());
    }
  }
  throw e;
}
```

## Handle Activity and Child Workflow failures 

Catch `ActivityFailure` (or `ChildWorkflowFailure`) around a call, not `ApplicationFailure` directly — the Activity
or Child Workflow boundary always wraps the underlying failure. Always check for `CanceledFailure` as the cause
before handling anything else, and rethrow it unhandled:

```java
try {
  return activities.charge(order);
} catch (ActivityFailure e) {
  if (e.getCause() instanceof CanceledFailure) {
    throw e; // never swallow cancellation
  }
  if (e.getCause() instanceof ApplicationFailure appFailure
      && "PaymentDeclined".equals(appFailure.getType())) {
    return Result.declined(appFailure.getOriginalMessage());
  }
  throw e; // don't recognize it — propagate
}
```

For deciding which failures should skip retries, see [Non-Retryable Errors](/design-patterns/non-retryable-errors),
which has a Java example for both marking a failure non-retryable at the throw site and listing non-retryable types
in a `RetryPolicy`. For undoing the effects of Activities that already succeeded before a later step failed, see the
[Saga Pattern](/design-patterns/saga-pattern), which has a Java example built on the SDK's `Saga` helper. For the
full set of retry strategies, see [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns).

### Cancellation 

Cancellation is cooperative — the Worker never force-stops running Workflow code. A cancellation request cancels the
current [Cancellation Scope](/develop/java/workflows/cancellation), and the next cancelable call inside it (an
Activity, Timer, or Child Workflow) throws `CanceledFailure`. If you need cleanup to run after a cancellation — for
example, compensating an Activity that already applied its effect — run it in a
[detached Cancellation Scope](/develop/java/workflows/cancellation#cancel-activity), since a normal scope is a child
of the one that was just canceled and any call inside it would be canceled immediately:

```java
try {
  activities.longRunningWork();
} catch (CanceledFailure e) {
  Workflow.newDetachedCancellationScope(() -> activities.compensate()).run();
  throw e; // rethrow after cleanup so the Workflow Execution ends "Canceled"
}
```

## Centralize failure conversion with a Worker Interceptor 

Activity code that calls several external services often ends up repeating the same `catch` blocks to convert
domain exceptions into `ApplicationFailure` with a consistent `type` and non-retryable classification. A
[`WorkerInterceptor`](/develop/plugins-guide#interceptors) that overrides
[`ActivityInboundCallsInterceptor.execute()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/interceptors/ActivityInboundCallsInterceptor.html)
centralizes that mapping in one place instead of repeating it in every Activity implementation:

```java
public final class ErrorNormalizingWorkerInterceptor extends WorkerInterceptorBase {
  @Override
  public ActivityInboundCallsInterceptor interceptActivity(ActivityInboundCallsInterceptor next) {
    return new ActivityInboundCallsInterceptorBase(next) {
      @Override
      public ActivityOutput execute(ActivityInput input) {
        try {
          return super.execute(input);
        } catch (ApplicationFailure | TimeoutFailure | CanceledFailure f) {
          throw f; // already a well-formed Temporal failure — pass through
        } catch (PaymentDeclinedException e) {
          throw ApplicationFailure.newNonRetryableFailure(e.getMessage(), "PaymentDeclined", e.toDetail());
        } catch (Exception e) {
          throw Activity.wrap(e); // uniform fallback: type = class name, retryable
        }
      }
    };
  }
}
```

Register it on the Worker Factory:

```java
WorkerFactoryOptions.newBuilder()
    .setWorkerInterceptors(new ErrorNormalizingWorkerInterceptor())
    .build();
```

Because the interceptor sees the original exception before the SDK's default conversion runs, it's the layer to
attach `details` and a non-retryable classification consistently, rather than depending on every Activity
implementation to do it the same way. `WorkerInterceptor` and `ActivityInboundCallsInterceptor` are marked
`@Experimental`. For the general interceptor model — inbound versus outbound, and the other call categories you can
intercept — see [Interceptors](/develop/java/workers/interceptors).

## Fail a Workflow deliberately 

Throwing `ApplicationFailure` from Workflow code is the only way to fail a Workflow Execution deliberately. Any other
unhandled exception fails only the current Workflow Task, which the Worker retries indefinitely — this is intentional:
a plain bug should be fixable with a new deployment, not permanently fail every Workflow Execution that hit it.

```java
if (order.getTotal().compareTo(BigDecimal.ZERO) <= 0) {
  throw ApplicationFailure.newNonRetryableFailure(
      "Order total must be positive: " + order.getTotal(), "InvalidOrderTotal");
}
```

If you want specific plain exception types to fail the Workflow Execution instead of retrying the Workflow Task,
list them with
[`WorkflowImplementationOptions.setFailWorkflowExceptionTypes()`](<https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkflowImplementationOptions.html#setFailWorkflowExceptionTypes(java.lang.Class...)>)
when registering the Workflow implementation.

Never extend `TemporalFailure` or any of its subclasses in application code — throw `ApplicationFailure` instead.
The SDK reserves the other subclasses (`ActivityFailure`, `ChildWorkflowFailure`, `CanceledFailure`, `TimeoutFailure`,
`TerminatedFailure`, `ServerFailure`) for its own use.

---

# Testing - Java SDK

Source: https://docs.temporal.io/develop/java/best-practices/testing-suite

> The Testing section of the Temporal Application development guide covers frameworks for Workflow and integration testing, including end-to-end, integration, and unit tests. Unit tests can be set up and run using the Temporal Java SDK's TestWorkflowEnvironment and TestWorkflowExtension classes for automated testing, allowing developers to test Workflows

The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing.

In the context of Temporal, you can create these types of automated tests:

- **End-to-end:** Running a Temporal Server and Worker with all its Workflows, Activities, and Nexus Operations; starting and interacting with Workflows from a Client.
- **Integration:** Anything between end-to-end and unit testing.
  - Running Activities with mocked Context and other SDK imports (and usually network requests).
  - Running Workers with mock Activities and Nexus Operations, and using a Client to start Workflows.
  - Running Workflows with mocked SDK imports.
- **Unit:** Running a piece of Workflow, Activity, or Nexus Operation code (a function or method) and mocking any code it calls.

We generally recommend writing the majority of your tests as integration tests.

Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers.

## Test frameworks 

The Temporal Java SDK provides a test framework to facilitate Workflow unit and integration testing.
The test framework provides a `TestWorkflowEnvironment` class which includes an in-memory implementation
of the Temporal service that supports automatic time skipping. This allows you to
easily test long-running Workflows in seconds, without having to change your Workflow code.

You can use the provided `TestWorkflowEnvironment` with a Java unit testing framework of your choice,
such as JUnit.

### Setup testing dependency

To start using the Java SDK test framework, you need to add [`io.temporal:temporal-testing`](https://search.maven.org/artifact/io.temporal/temporal-testing)
as a dependency to your project:

**[Apache Maven](https://maven.apache.org/):**

```maven
<dependency>
    <groupId>io.temporal</groupId>
    <artifactId>temporal-testing</artifactId>
    <version>1.36.0</version>
    <scope>test</scope>
</dependency>
```

**[Gradle Groovy DSL](https://gradle.org/):**

```groovy
testImplementation ("io.temporal:temporal-testing:1.36.0")
```

If you need JUnit4 or JUnit5 extensions:

```
testImplementation("io.temporal:temporal-testing:1.36.0") {
    capabilities {
        requireCapability("io.temporal:temporal-testing-junit4")
        //requireCapability("io.temporal:temporal-testing-junit5")
    }
}
```

Make sure to set the version that matches your dependency version of the [Temporal Java SDK](https://github.com/temporalio/sdk-java).

## Test Activities 

An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity. This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity.

Temporal provides the `TestActivityEnvironment` and `TestActivityExtension` classes for testing Activities outside the scope of a Workflow. Testing Activities is similar to testing non-Temporal Java code.

For example, you can test an Activity for:

- Exceptions thrown when invoking the Activity Execution.
- Exceptions thrown when checking for the result of the Activity Execution.
- Activity's return values. Check that the return value matches the expected value.

Here's an example of an Activity that will be referenced in the sections below:

```java
package helloworkflow;

import java.util.concurrent.TimeUnit;

import io.temporal.activity.Activity;
import io.temporal.activity.ActivityExecutionContext;
import io.temporal.client.ActivityCompletionException;

public class GreetActivitiesImpl implements GreetActivities {

  @Override
  public String greet(String name) {
    ActivityExecutionContext context = Activity.getExecutionContext();

    for (int i = 0; i < 5; i++) {
      try {
        context.heartbeat(null);
      } catch (ActivityCompletionException e) {
        throw e;
      }
      
      sleep(1);
    }

    return "Hello " + name + "!";
  }

  private void sleep(int seconds) {
    try {
      Thread.sleep(TimeUnit.SECONDS.toMillis(seconds));
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new RuntimeException(e);
    }
  }
}
```

### Run an Activity 

The following code implements unit tests for the `greet` Activity above:

```java
package helloworkflow;

import static org.junit.jupiter.api.Assertions.assertEquals;

import org.junit.jupiter.api.Test;

import io.temporal.testing.TestActivityEnvironment;

public class GreetActivitiesTest {

  @Test
  public void testActivityImpl() {
    TestActivityEnvironment testEnv = TestActivityEnvironment.newInstance();

    testEnv.registerActivitiesImplementations(new GreetActivitiesImpl());

    GreetActivities activities = testEnv.newActivityStub(GreetActivities.class);

    String result = activities.greet("Temporal");

    assertEquals("Hello Temporal!", result);
  }
}
```

### Listen to Heartbeats 

Activities usually issue periodic Heartbeats, a ping that shows that an Activity is making progress and the Worker hasn't crashed. Heartbeats may include details that report task progress in the event an Activity Worker crashes.

When testing Activities that support Heartbeats, make sure you can see those Heartbeats in your test code.

```java
@Test
void testActivityHeartbeat() {
  TestActivityEnvironment env = TestActivityEnvironment.newInstance();

  AtomicInteger heartbeatCount = new AtomicInteger(0);

  env.setActivityHeartbeatListener(
    Void.class,
    heartbeat -> heartbeatCount.incrementAndGet());

  env.registerActivitiesImplementations(new GreetActivitiesImpl());

  GreetActivities activities = env.newActivityStub(GreetActivities.class);

  String result = activities.greet("Temporal");

  assertEquals("Hello Temporal!", result);
  assertEquals(5, heartbeatCount.get());
}
```

### Cancel an Activity 

Activity cancellation lets Activities know they don't need to continue work and gives time for the Activity to clean up any resources it's created. You can cancel Java-based Activities if they emit Heartbeats. To test an Activity that reacts to Cancellations, make sure that the Activity reacts correctly and cancels.

```java
@Test
void testCancelActivity() {
  try (TestWorkflowEnvironment env = TestWorkflowEnvironment.newInstance()) {
    Worker worker = env.newWorker(TASK_QUEUE);

    worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class);
    worker.registerActivitiesImplementations(new GreetActivitiesImpl());

    env.start();

    SayHelloWorkflow workflow =
      env.getWorkflowClient()
        .newWorkflowStub(
            SayHelloWorkflow.class,
            WorkflowOptions.newBuilder()
              .setTaskQueue(TASK_QUEUE)
              .build());

    WorkflowClient.start(workflow::sayHello, "Temporal");

    env.registerDelayedCallback(
        Duration.ofSeconds(1),
        () -> WorkflowStub.fromTyped(workflow).signal("cancelActivity"));

    try {
      WorkflowStub.fromTyped(workflow).getResult(String.class);
      fail("Workflow should have failed because the Activity was canceled");
    } catch (WorkflowFailedException e) {
      assertEquals(ActivityFailure.class, e.getCause().getClass());

      ActivityFailure activityFailure = (ActivityFailure) e.getCause();
      assertEquals(CanceledFailure.class, activityFailure.getCause().getClass());
    }
  }
}
```

## Testing Workflows 

### How to mock Activities 

When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker.

In cases where you do not wish to execute your actual Activity or Nexus Operation implementations during
unit testing, you can use a framework such as [Mockito](https://site.mockito.org/) to mock them.

The following code implements a unit test that shows how Activities can be mocked:

```java
@Test
public void testMockedActivity() {
  GreetActivities activities =
    mock(GreetActivities.class, withSettings().withoutAnnotations());

  when(activities.greet("Temporal")).thenReturn("Hello Temporal!");

  assertEquals("Hello Temporal!", activities.greet("Temporal"));
}
```

#### Testing with JUnit4

For JUnit4 tests, Temporal provides the `TestWorkflowRule` class which simplifies the Temporal test environment setup, as well as the creation and shutdown of Workflow Workers in your tests. You can now rewrite the above Activity test class as follows:

```java
public class GreetActivitiesJUnit4Test {
    @Rule
    public TestWorkflowRule testWorkflowRule =
        TestWorkflowRule.newBuilder()
            .setWorkflowTypes(SayHelloWorkflowImpl.class)
            .setActivityImplementations(new GreetActivitiesImpl())
            .build();

    @Test
    public void testActivityImpl() {
        // Get a workflow stub using the same task queue the worker uses.
        GreetingWorkflow workflow =
            testWorkflowRule
                .getWorkflowClient()
                .newWorkflowStub(
                    GreetingWorkflow.class,
                    WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build()
                );
        // Execute a workflow waiting for it to complete.
        String greeting = workflow.sayHello("Temporal");
        assertEquals("Hello Temporal!", greeting);

        testWorkflowRule.getTestEnvironment().shutdown();
    }
}
```

#### Testing with JUnit5

For JUnit5 tests, Temporal also provides the `TestWorkflowExtension` helper class.
This class can be used to simplify the Temporal test environment setup as well as Workflow Worker startup and shutdowns.

To start using JUnit5 `TestWorkflowExtension` in your tests with [Gradle](https://gradle.org/), you need to enable capability [`io.temporal:temporal-testing-junit5`]:

Now you can use JUnit5 and rewrite the above test class as follows:

```java
public class GreetActivitiesJUnit5Test {
  @RegisterExtension
  public static final TestWorkflowExtension testWorkflowExtension =
    TestWorkflowExtension.newBuilder()
      .setWorkflowTypes(SayHelloWorkflowImpl.class)
      .setActivityImplementations(new GreetActivitiesImpl())
      .build();

  @Test
  public void testActivityImpl(
    TestWorkflowEnvironment testEnv, Worker worker, SayHelloWorkflow workflow) {
    // Execute a workflow waiting for it to complete.
    String greeting = workflow.sayHello("Temporal");
    assertEquals("Hello Temporal!", greeting);
  }
}
```

You can find more unit test examples in the [Temporal Java samples](https://github.com/temporalio/samples-java) repository in [its test package](https://github.com/temporalio/samples-java/tree/main/core/src/test/java/io/temporal/samples).

### How to mock Nexus Operations 

When integration testing Workflows with a Worker, you can mock Nexus operations by providing mock Nexus Service handlers to the Worker.
Alternatively, you could just mock the Nexus service itself.

You can find more example unit tests for Nexus in the [Temporal Java samples](https://github.com/temporalio/samples-java) repository in [this test package](https://github.com/temporalio/samples-java/tree/main/core/src/test/java/io/temporal/samples/nexus/caller).
These samples show how to call Nexus services in tests using the Temporal testing package and also how to mock them, for both JUnit 4 and 5.
Detailed explanatory comments are included in the code in the repository.

To mock Nexus handlers, create a `Rule` (for JUnit4) or `Extension` (for JUnit5) from the Temporal testing package and add a call to `setNexusServiceImplementation` to the builder.
That sets up the Nexus endpoints needed for testing as well as the Nexus handler workflows defined by the Nexus Service implementation.

You will need to create Workers for each handler, using either `setWorkflowTypes` (for JUnit4) or `registerWorkflowImplementationTypes` (for JUnit5).
With that in place, you can then mock a Nexus endpoint exactly like any other Workflow like you do with Activity tests.

The following are samples derived from [the test package](https://github.com/temporalio/samples-java/tree/main/core/src/test/java/io/temporal/samples/nexus/caller) to demonstrate this.

#### Mocking Nexus handlers with JUnit4
<!--SNIPSTART java-nexus-sample-junit4-mock-->
[core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowMockTest.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowMockTest.java)
```java
public class CallerWorkflowMockTest {

  // Inject a mock EchoClient so sync Nexus operations can be stubbed per test.
  // JUnit 4 creates a new test class instance per test method, so this mock is fresh each time.
  private final EchoClient mockEchoClient = mock(EchoClient.class);

  @Rule
  public TestWorkflowRule testWorkflowRule =
      TestWorkflowRule.newBuilder()
          // If a Nexus service is registered as part of the test as in the following line of code,
          // the TestWorkflowRule will, by default, automatically create a Nexus service endpoint
          // and workflows registered as part of the TestWorkflowRule
          // will automatically inherit the endpoint if none is set.
          .setNexusServiceImplementation(new SampleNexusServiceImpl(mockEchoClient))
          // The Echo Nexus handler service just makes a call to a class, so no extra setup is
          // needed. But the Hello Nexus service needs a worker for both the caller and handler
          // in order to run.
          // setWorkflowTypes will take the classes given and create workers for them, enabling
          // workflows to run. This creates caller workflows, the handler workflows
          // will be mocked in the test methods.
          .setWorkflowTypes(HelloCallerWorkflowImpl.class, EchoCallerWorkflowImpl.class)
          // Disable automatic worker startup as we are going to register some workflows manually
          // per test
          .setDoNotStart(true)
          .build();

  @Test
  public void testHelloWorkflow() {
    testWorkflowRule
        .getWorker()
        // Workflows started by a Nexus service can be mocked just like any other workflow
        .registerWorkflowImplementationFactory(
            HelloHandlerWorkflow.class,
            () -> {
              HelloHandlerWorkflow wf = mock(HelloHandlerWorkflow.class);
              when(wf.hello(any()))
                  .thenReturn(new SampleNexusService.HelloOutput("Hello Mock World 👋"));
              return wf;
            });
    testWorkflowRule.getTestEnvironment().start();

    // Now create the caller workflow
    HelloCallerWorkflow workflow =
        testWorkflowRule
            .getWorkflowClient()
            .newWorkflowStub(
                HelloCallerWorkflow.class,
                WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build());
    String greeting = workflow.hello("World", SampleNexusService.Language.EN);
    assertEquals("Hello Mock World 👋", greeting);

    testWorkflowRule.getTestEnvironment().shutdown();
  }

  @Test
  public void testEchoWorkflow() {
    // Sync Nexus operations run inline in the handler thread — there is no backing workflow to
    // register a factory for. Instead, stub the injected EchoCient dependency directly.
    when(mockEchoClient.echo(any())).thenReturn(new SampleNexusService.EchoOutput("mocked echo"));
    testWorkflowRule.getTestEnvironment().start();

    EchoCallerWorkflow workflow =
        testWorkflowRule
            .getWorkflowClient()
            .newWorkflowStub(
                EchoCallerWorkflow.class,
                WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build());
    String greeting = workflow.echo("Hello");
    assertEquals("mocked echo", greeting);

    testWorkflowRule.getTestEnvironment().shutdown();
  }
}

```
<!--SNIPEND-->

#### Mocking Nexus handlers with JUnit5
<!--SNIPSTART java-nexus-sample-junit5-mock-->
[core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowJunit5MockTest.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowJunit5MockTest.java)
```java
public class CallerWorkflowJunit5MockTest {

  // Sync Nexus operations run inline in the handler thread — there is no backing workflow to
  // register a factory for. To mock one, inject a mock dependency into the service implementation.
  private static final EchoClient mockEchoClient = mock(EchoClient.class);

  @RegisterExtension
  public static final TestWorkflowExtension testWorkflowExtension =
      TestWorkflowExtension.newBuilder()
          // If a Nexus service is registered as part of the test as in the following line of code,
          // the TestWorkflowExtension will, by default, automatically create a Nexus service
          // endpoint and workflows registered as part of the TestWorkflowExtension will
          // automatically inherit the endpoint if none is set.
          .setNexusServiceImplementation(new SampleNexusServiceImpl(mockEchoClient))
          // The Echo Nexus handler service just makes a call to a class, so no extra setup is
          // needed. But the Hello Nexus service needs a worker for both the caller and handler
          // in order to run, and the Echo Nexus caller service needs a worker.
          //
          // registerWorkflowImplementationTypes will take the classes given and create workers for
          // them, enabling workflows to run.
          .registerWorkflowImplementationTypes(
              HelloCallerWorkflowImpl.class, EchoCallerWorkflowImpl.class)
          .setDoNotStart(true)
          .build();

  @Test
  public void testHelloWorkflow(
      TestWorkflowEnvironment testEnv, Worker worker, HelloCallerWorkflow workflow) {
    // Workflows started by a Nexus service can be mocked just like any other workflow
    worker.registerWorkflowImplementationFactory(
        HelloHandlerWorkflow.class,
        () -> {
          HelloHandlerWorkflow mockHandler = mock(HelloHandlerWorkflow.class);
          when(mockHandler.hello(any()))
              .thenReturn(new SampleNexusService.HelloOutput("Hello Mock World 👋"));
          return mockHandler;
        });
    testEnv.start();

    // Execute a workflow waiting for it to complete.
    String greeting = workflow.hello("World", SampleNexusService.Language.EN);
    assertEquals("Hello Mock World 👋", greeting);

    testEnv.shutdown();
  }

  @Test
  public void testEchoWorkflow(
      TestWorkflowEnvironment testEnv, Worker worker, EchoCallerWorkflow workflow) {
    // Sync Nexus operations run inline in the handler thread — there is no backing workflow to
    // register a factory for. Instead, stub the injected EchoClient dependency directly.
    when(mockEchoClient.echo(any())).thenReturn(new SampleNexusService.EchoOutput("mocked echo"));
    testEnv.start();

    // Execute a workflow waiting for it to complete.
    String greeting = workflow.echo("Hello");
    assertEquals("mocked echo", greeting);

    testEnv.shutdown();
  }
}

```
<!--SNIPEND-->

An alternative approach is to mock the Nexus service itself, instead of mocking the handlers.
This is useful if you want to test the calling logic, but can't easily mock the Nexus handlers.

The code will mock the implementation of the `SampleNexusService` class with the handler methods, but will need those methods stubbed in for the testing framework.
Those methods can be directly mocked with static return values or they can return an instance variable which each unit test can modify to return a desired value.

#### Mocking the Nexus Service with JUnit4
<!--SNIPSTART java-nexus-service-sample-junit4-mock-->
[core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceMockTest.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceMockTest.java)
```java
public class NexusServiceMockTest {

  private final SampleNexusService mockNexusService = mock(SampleNexusService.class);

  /**
   * A test-only Nexus service implementation that delegates to the Mockito mock defined above. Both
   * operations are implemented as synchronous handlers that forward calls to the mock, allowing
   * full control over return values and verification of inputs.
   */
  @ServiceImpl(service = SampleNexusService.class)
  public class TestNexusServiceImpl {
    @OperationImpl
    @SuppressWarnings("DirectInvocationOnMock")
    public OperationHandler<SampleNexusService.EchoInput, SampleNexusService.EchoOutput> echo() {
      return OperationHandler.sync((ctx, details, input) -> mockNexusService.echo(input));
    }

    @OperationImpl
    @SuppressWarnings("DirectInvocationOnMock")
    public OperationHandler<SampleNexusService.HelloInput, SampleNexusService.HelloOutput> hello() {
      return OperationHandler.sync((ctx, details, input) -> mockNexusService.hello(input));
    }
  }

  // Using OperationHandler.sync for both operations bypasses the need for a backing workflow,
  // returning results inline just like a synchronous call.
  //
  // Note that the Mocks need to be done before the rule
  // is defined, as creating the rule will fail if either call is still null.

  @Rule
  public TestWorkflowRule testWorkflowRule =
      TestWorkflowRule.newBuilder()
          .setNexusServiceImplementation(new TestNexusServiceImpl())
          .setWorkflowTypes(EchoCallerWorkflowImpl.class, HelloCallerWorkflowImpl.class)
          .build();

  @Test
  public void testHelloCallerWithMockedService() {
    when(mockNexusService.hello(any()))
        .thenReturn(new SampleNexusService.HelloOutput("Bonjour World"));

    HelloCallerWorkflow workflow =
        testWorkflowRule
            .getWorkflowClient()
            .newWorkflowStub(
                HelloCallerWorkflow.class,
                WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build());

    String result = workflow.hello("World", SampleNexusService.Language.FR);
    assertEquals("Bonjour World", result);

    // Verify the Nexus service was called with the correct name and language
    verify(mockNexusService)
        .hello(
            argThat(
                input ->
                    "World".equals(input.getName())
                        && SampleNexusService.Language.FR == input.getLanguage()));
  }

  @Test
  public void testEchoCallerWithMockedService() {
    when(mockNexusService.echo(any()))
        .thenReturn(new SampleNexusService.EchoOutput("echo response"));

    EchoCallerWorkflow workflow =
        testWorkflowRule
            .getWorkflowClient()
            .newWorkflowStub(
                EchoCallerWorkflow.class,
                WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build());

    String echoOutput = workflow.echo("echo input");

    assertEquals("echo response", echoOutput);

    // Verify the echo operation was called exactly once and no other operations were invoked
    verify(mockNexusService, times(1)).echo(any());
    // Verify the Nexus service was called with the correct input
    verify(mockNexusService).echo(argThat(input -> "echo input".equals(input.getMessage())));

    verifyNoMoreInteractions(mockNexusService);
  }
}

```
<!--SNIPEND-->

#### Mocking the Nexus Service with JUnit5
<!--SNIPSTART java-nexus-service-sample-junit5-mock-->
[core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceJunit5Test.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceJunit5Test.java)
```java
public class NexusServiceJunit5Test {

  private final SampleNexusService mockNexusService = mock(SampleNexusService.class);

  /**
   * A test-only Nexus service implementation that delegates to the Mockito mock defined above. Both
   * operations are implemented as synchronous handlers that forward calls to the mock, allowing
   * full control over return values and verification of inputs.
   */
  @ServiceImpl(service = SampleNexusService.class)
  public class TestNexusServiceImpl {
    @OperationImpl
    @SuppressWarnings("DirectInvocationOnMock")
    public OperationHandler<SampleNexusService.EchoInput, SampleNexusService.EchoOutput> echo() {
      return OperationHandler.sync((ctx, details, input) -> mockNexusService.echo(input));
    }

    @OperationImpl
    @SuppressWarnings("DirectInvocationOnMock")
    public OperationHandler<SampleNexusService.HelloInput, SampleNexusService.HelloOutput> hello() {
      return OperationHandler.sync((ctx, details, input) -> mockNexusService.hello(input));
    }
  }

  // Using OperationHandler.sync for both operations bypasses the need for a backing workflow,
  // returning results inline just like a synchronous call.
  //
  // Note that the Mocks need to be done before the extension
  // is defined, as creating the rule will fail if either call is still null.

  @RegisterExtension
  public final TestWorkflowExtension testWorkflowExtension =
      TestWorkflowExtension.newBuilder()
          // If a Nexus service is registered as part of the test as in the following line of code,
          // the TestWorkflowExtension will, by default, automatically create a Nexus service
          // endpoint and workflows registered as part of the TestWorkflowExtension will
          // automatically inherit the endpoint if none is set.
          .setNexusServiceImplementation(new TestNexusServiceImpl())
          // The Echo Nexus handler service just makes a call to a class, so no extra setup is
          // needed. But the Hello Nexus service needs a worker for both the caller and handler
          // in order to run, and the Echo Nexus caller service needs a worker.
          //
          // registerWorkflowImplementationTypes will take the classes given and create workers for
          // them, enabling workflows to run.
          // Since both operations are mocked with OperationHandler.sync, no backing workflow is
          // needed for hello — only the caller workflow types need to be registered.
          .registerWorkflowImplementationTypes(
              HelloCallerWorkflowImpl.class, EchoCallerWorkflowImpl.class)
          // The workflow will start before each test, and will shut down after each test.
          // See CallerWorkflowTest for an example of how to control this differently if needed.
          .build();

  // The TestWorkflowExtension extension in the Temporal testing library creates the
  // arguments to the test cases and initializes them from the extension setup call above.
  @Test
  public void testHelloWorkflow(
      TestWorkflowEnvironment testEnv, Worker worker, HelloCallerWorkflow workflow) {

    // Set the mock value to return
    when(mockNexusService.hello(any()))
        .thenReturn(new SampleNexusService.HelloOutput("Hello Mock World 👋"));

    // Execute a workflow waiting for it to complete.
    String greeting = workflow.hello("World", SampleNexusService.Language.EN);
    assertEquals("Hello Mock World 👋", greeting);
  }

  @Test
  public void testEchoWorkflow(
      TestWorkflowEnvironment testEnv, Worker worker, EchoCallerWorkflow workflow) {
    when(mockNexusService.echo(any()))
        .thenReturn(new SampleNexusService.EchoOutput("echo response"));

    // Execute a workflow waiting for it to complete.
    String greeting = workflow.echo("echo input");
    assertEquals("echo response", greeting);

    // Verify the echo operation was called exactly once and no other operations were invoked
    verify(mockNexusService, times(1)).echo(any());
    // Verify the Nexus service was called with the correct input
    verify(mockNexusService).echo(argThat(input -> "echo input".equals(input.getMessage())));

    verifyNoMoreInteractions(mockNexusService);
  }
}

```
<!--SNIPEND-->

### How to skip time 

Some long-running Workflows can persist for months or even years.
Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount.

For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works.
Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner.

The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time.
Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests.
If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server.
For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel.

#### Skip time automatically 

When you execute a Workflow and wait for the result, the test environment automatically skips Workflow timers such as `Workflow.sleep`. This means:

- Workflow timers (like `Workflow.sleep`) are fast-forwarded.
- Time doesn't skip while Activities and Nexus operations are executing. Nexus operation handlers timeout after 10 seconds and time skipping is allowed while waiting for retries.

Here's an example of a Workflow that implements `Workflow.sleep` to wait for a day before calling the Activity:

```java
public class SayHelloWorkflowImpl implements SayHelloWorkflow {
  private final GreetActivities activities =
    Workflow.newActivityStub(
      GreetActivities.class,
      ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(30))
        .build());

  @Override
  public String sayHello(String name) {
    // This will complete immediately in TestWorkflowEnvironment because
    // Workflow time is automatically advanced.
    Workflow.sleep(Duration.ofDays(1));

    return activities.greet(name);
  }
}
```

Here's what the test for that Workflow could look like:

```java
@Test
void testSleepCompletesWithoutWaitingOneDay() {
  try (TestWorkflowEnvironment testEnv = TestWorkflowEnvironment.newInstance()) {
    Worker worker = testEnv.newWorker(TASK_QUEUE);
    
    worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class);
    worker.registerActivitiesImplementations(new GreetActivitiesImpl());
    
    testEnv.start();

    SayHelloWorkflow workflow =
      testEnv
        .getWorkflowClient()
        .newWorkflowStub(
          SayHelloWorkflow.class,
          WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).build());

    String result = workflow.sayHello("Temporal");

    assertEquals("Hello Temporal!", result);
  }
}
```

Use `Workflow.sleep()` in Workflow code, not `Thread.sleep()`. `Workflow.sleep()` creates a Temporal timer that the test environment can skip. `Thread.sleep()` blocks a Java thread in real time and is not controlled by Temporal’s time-skipping test server.

#### Skip time manually 

Use `TestWorkflowEnvironment.sleep(Duration)` when you want to advance virtual time yourself and inspect intermediate Workflow state. Start the Workflow asynchronously with `WorkflowClient.start()`, then call `testEnv.sleep()` from the test.

Here's an example of a Workflow test that lets you manually advance time with `testEnv.sleep()`:

```java
package helloworkflow;

import java.time.Duration;

import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;

import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.testing.TestWorkflowEnvironment;
import io.temporal.worker.Worker;
import io.temporal.workflow.QueryMethod;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;

public class ManualTimeSkippingTest {
  private static final String TASK_QUEUE = "manual-time-skipping-test";

  @WorkflowInterface
  public interface ProgressWorkflow {
    @WorkflowMethod
    void run();

    @QueryMethod
    int daysElapsed();
  }

  public static class ProgressWorkflowImpl implements ProgressWorkflow {
    private int daysElapsed = 0;

    @Override
    public void run() {
      for (int i = 0; i < 100; i++) {
        Workflow.sleep(Duration.ofDays(1));
        daysElapsed++;
      }
    }

    @Override
    public int daysElapsed() {
      return daysElapsed;
    }
  }

  @Test
  void manuallyAdvanceWorkflowTime() {
    try (TestWorkflowEnvironment testEnv = TestWorkflowEnvironment.newInstance()) {
      Worker worker = testEnv.newWorker(TASK_QUEUE);
      worker.registerWorkflowImplementationTypes(ProgressWorkflowImpl.class);
      testEnv.start();

      ProgressWorkflow workflow =
          testEnv
              .getWorkflowClient()
              .newWorkflowStub(
                  ProgressWorkflow.class,
                  WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).build());

      WorkflowClient.start(workflow::run);

      assertEquals(0, workflow.daysElapsed());

      testEnv.sleep(Duration.ofHours(25));
      assertEquals(1, workflow.daysElapsed());

      testEnv.sleep(Duration.ofHours(25));
      assertEquals(2, workflow.daysElapsed());
    }
  }
}
```

## How to Replay a Workflow Execution 

Replay recreates the exact state of a Workflow Execution.
You can replay a Workflow from the beginning of its Event History.

Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view.

When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks:

1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test.
2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI.
3. Run the Event Histories through replay.
4. Fail CI if any error is encountered during replay.

The following are examples of fetching and replaying Event Histories:

To replay Workflow Executions, use the [WorkflowReplayer](https://www.javadoc.io/doc/io.temporal/temporal-testing/latest/io/temporal/testing/WorkflowReplayer.html) class in the `temporal-testing` package.

In the following example, Event Histories are downloaded from the server, and then replayed.
Note that this requires Advanced Visibility to be enabled.

```java
// Note we assume you already have a WorkflowServiceStubs (`service`) and WorkflowClient (`client`)
// in scope.
ListWorkflowExecutionsRequest listWorkflowExecutionRequest =
    ListWorkflowExecutionsRequest.newBuilder()
        .setNamespace(client.getOptions().getNamespace())
        .setQuery("TaskQueue = 'mytaskqueue'")
        .build();
ListWorkflowExecutionsResponse listWorkflowExecutionsResponse =
    service.blockingStub().listWorkflowExecutions(listWorkflowExecutionRequest);
List<WorkflowExecutionHistory> histories =
    listWorkflowExecutionsResponse.getExecutionsList().stream()
        .map(
            (info) -> {
              GetWorkflowExecutionHistoryResponse weh =
                  service.blockingStub().getWorkflowExecutionHistory(
                      GetWorkflowExecutionHistoryRequest.newBuilder()
                          .setNamespace(testEnvironment.getNamespace())
                          .setExecution(info.getExecution())
                          .build());
              return new WorkflowExecutionHistory(
                  weh.getHistory(), info.getExecution().getWorkflowId());
            })
        .collect(Collectors.toList());

WorkflowReplayer.replayWorkflowExecutions(
    histories, true, WorkflowA.class, WorkflowB.class, WorkflowC.class);
```

In the next example, a single history is loaded from a JSON file on disk:

```java
File file = new File("my_history.json");
WorkflowReplayer.replayWorkflowExecution(file, MyWorkflow.class);
```

In both examples, if Event History is non-deterministic, an error is thrown.
You can choose to wait until all histories have been replayed with `replayWorkflowExecutions` by setting the `failFast` argument to `false`.

---

# Client - Java SDK

Source: https://docs.temporal.io/develop/java/client

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Temporal Client

- [Temporal Client](/develop/java/client/temporal-client)
- [Namespaces](/develop/java/client/namespaces)

---

# Namespaces - Java SDK

Source: https://docs.temporal.io/develop/java/client/namespaces

> Register, update, deprecate, and delete Namespaces using Temporal CLI or SDK APIs. Manage Workflow Executions with isolated Namespaces to match your needs.

This page shows how to do the following:

- [Register a Namespace](#register-namespace)
- [Manage Namespaces](#manage-namespaces)

You can create, update, deprecate or delete your [Namespaces](/namespaces) using either the Temporal CLI or SDK APIs.

Use Namespaces to isolate your Workflow Executions according to your needs.
For example, you can use Namespaces to match the development lifecycle by having separate `dev` and `prod` Namespaces.
You could also use them to ensure Workflow Executions between different teams never communicate - such as ensuring that the `teamA` Namespace never impacts the `teamB` Namespace.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) to create and manage a Namespace from the UI, or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces from the command line.

On self-hosted Temporal Service, you can register and manage your Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

You must register a Namespace with the Temporal Service before setting it in the Temporal Client.

## Register a Namespace 

Registering a Namespace creates a Namespace on the Temporal Service or Temporal Cloud.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to create Namespaces.

On self-hosted Temporal Service, you can register your Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

Use the [`RegisterNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to register a [Namespace](/namespaces) and set the [Retention Period](/temporal-service/temporal-server#retention-period) for the Workflow Execution Event History for the Namespace.

```java
//...
import com.google.protobuf.util.Durations;
import io.temporal.api.workflowservice.v1.RegisterNamespaceRequest;
//...
public static void createNamespace(String name) {
    RegisterNamespaceRequest req = RegisterNamespaceRequest.newBuilder()
            .setNamespace("your-custom-namespace")
            .setWorkflowExecutionRetentionPeriod(Durations.fromDays(3)) // keeps the Workflow Execution
            //Event History for up to 3 days in the Persistence store. Not setting this value will throw an error.
            .build();
    service.blockingStub().registerNamespace(req);
}
//...
```

The Retention Period setting using `WorkflowExecutionRetentionPeriod` is mandatory.
The minimum value you can set for this period is 1 day.

Once registered, set Namespace using `WorkflowClientOptions` within a Workflow Client to run your Workflow Executions within that Namespace.
See [Connect to a Development Temporal Service](/develop/java/client/temporal-client#connect-to-development-service) for details.

Note that Namespace registration using this API takes up to 10 seconds to complete.
Ensure that you wait for this registration to complete before starting the Workflow Execution against the Namespace.

To update your Namespace use the [UpdateNamespace API](#manage-namespaces) with the NamespaceClient.

## Manage Namespaces 

You can get details for your Namespaces, update Namespace configuration, and deprecate or delete your Namespaces.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces.

On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

You must register a Namespace with the Temporal Service before setting it in the Temporal Client.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces.

On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

- Update information and configuration for a registered Namespace on your Temporal Service:

  - With the Temporal CLI: [`temporal operator namespace update`](/cli/command-reference/operator#update)
    Example
  - Use the [`UpdateNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to update configuration on a Namespace.
    Example

  ```java
  import io.temporal.api.workflowservice.v1.*;
  //...
  UpdateNamespaceRequest updateNamespaceRequest = UpdateNamespaceRequest.newBuilder()
              .setNamespace("your-namespace-name") //the namespace that you want to update
              .setUpdateInfo(UpdateNamespaceInfo.newBuilder() //has options to update namespace info
                      .setDescription("your updated namespace description") //updates description in the namespace info.
                      .build())
              .setConfig(NamespaceConfig.newBuilder() //has options to update namespace configuration
                      .setWorkflowExecutionRetentionTtl(Durations.fromHours(30)) //updates the retention period for the namespace "your-namespace--name" to 30 hrs.
                      .build())
              .build();
      UpdateNamespaceResponse updateNamespaceResponse = namespaceservice.blockingStub().updateNamespace(updateNamespaceRequest);
  //...
  ```

- Get details for a registered Namespace on your Temporal Service:

  - With the Temporal CLI: [`temporal operator namespace describe`](/cli/command-reference/operator#describe)
  - Use the [`DescribeNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to return information and configuration details for a registered Namespace.
    Example

  ```java
  import io.temporal.api.workflowservice.v1.*;
  //...
  DescribeNamespaceRequest descNamespace = DescribeNamespaceRequest.newBuilder()
              .setNamespace("your-namespace-name") //specify the namespace you want details for
              .build();
      DescribeNamespaceResponse describeNamespaceResponse = namespaceservice.blockingStub().describeNamespace(descNamespace);
      System.out.println("Namespace Description: " + describeNamespaceResponse);
  //...
  ```

- Get details for all registered Namespaces on your Temporal Service:

  - With the Temporal CLI: [`temporal operator namespace list`](/cli/command-reference/operator#list)
  - Use the [`ListNamespaces` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to return information and configuration details for all registered Namespaces on your Temporal Service.
    Example

  ```java
  import io.temporal.api.workflowservice.v1.*;
  //...
  ListNamespacesRequest listNamespaces = ListNamespacesRequest.newBuilder().build();
      ListNamespacesResponse listNamespacesResponse = namespaceservice.blockingStub().listNamespaces(listNamespaces); //lists 1-100 namespaces (1 page) in the active Temporal Service. To list all, set the page size or loop until NextPageToken is nil.
  //...
  ```

- Deprecate a Namespace: The [`DeprecateNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) updates the state of a registered Namespace to "DEPRECATED". Once a Namespace is deprecated, you cannot start new Workflow Executions on it. All existing and running Workflow Executions on a deprecated Namespace will continue to run.
  Example:

  ```java
  import io.temporal.api.workflowservice.v1.*;
  //...
  DeprecateNamespaceRequest deprecateNamespace = DeprecateNamespaceRequest.newBuilder()
              .setNamespace("your-namespace-name") //specify the namespace that you want to deprecate
              .build();
      DeprecateNamespaceResponse response = namespaceservice.blockingStub().deprecateNamespace(deprecateNamespace);
  //...
  ```

- Delete a Namespace: The [`DeleteNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) deletes a Namespace. Deleting a Namespace deletes all running and completed Workflow Executions on the Namespace, and removes them from the persistence store and the visibility store.

  Example:

  ```java
  //...
  DeleteNamespaceResponse res =
  OperatorServiceStubs.newServiceStubs(OperatorServiceStubsOptions.newBuilder()
          .setChannel(service.getRawChannel())
          .validateAndBuildWithDefaults())
      .blockingStub()
      .deleteNamespace(DeleteNamespaceRequest.newBuilder().setNamespace("your-namespace-name").build());
  //...
  ```

---

# Temporal Client - Java SDK

Source: https://docs.temporal.io/develop/java/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the
[Temporal Service](/temporal-service). Communication with a Temporal Service lets you perform actions such as starting
Workflow Executions, sending Signals to Workflow Executions, sending Queries to Workflow Executions, getting the results
of a Workflow Execution, and providing Activity Task Tokens.

For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow.

This page shows you how to do the following using the Java SDK with the Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-development-service)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Start a Workflow Execution](#start-workflow-execution)
- [Get Workflow results](#get-workflow-results)

> **⚠️ Caution:**
>
> A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a
> Temporal Client inside an Activity to communicate with a Temporal Service.
>

## Connect to development Temporal Service 

Use the `newLocalServiceStubs` method to create a stub that points to the Temporal development service, and then use the
[`WorkflowClient.newInstance` method](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#newInstance(io.temporal.serviceclient.WorkflowServiceStubs))
to create a Temporal Client.

```java
// Create an instance that connects to a Temporal Service running on the local machine, using the default port 7233
WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newLocalServiceStubs();

// Initialize the Client
WorkflowClient client = WorkflowClient.newInstance(serviceStub);
```

When you create a new Client with an instance of `newLocalServiceStubs`, the Client connects to the default local port
at port 7233. When you don't specify a custom Namespace, the Client connects to the `default` Namespace.

To connect to a custom Namespace, use the `WorkflowClientOptions.Builder.setNamespace` method to set the Namespace. Then
pass the `clientOptions` to the `WorkflowClient.newInstance` method.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the
TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK
looks for it at the path `~/.config/temporalio/temporal.toml`. For a list of all available configuration options, refer
to [Environment Configuration](/references/client-environment-configuration)

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own
set of connection options.

```toml
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
# TLS is auto-enabled when this TLS config or API key is present, but you can configure it explicitly
# disabled = false
# Use certificate files for mTLS
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment = "production"
service-version = "v1.2.3"
```

You can create a Temporal Client using a specific profile from the configuration file as follows. First use
`ClientConfigProfile.load` to load the profile from the configuration file. Then use
`profile.toWorkflowServiceStubsOptions` and `profile.toWorkflowClientOptions` to convert the profile to
`WorkflowServiceStubsOptions` and `WorkflowClientOptions` respectively. Then use `WorkflowClient.newInstance` to create
a Temporal Client.

```java {21-25,32-34}
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.envconfig.LoadClientConfigProfileOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import java.nio.file.Paths;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoadFromFile {

  private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class);

  public static void main(String[] args) {
    try {

      String configFilePath =
          Paths.get(LoadFromFile.class.getResource("/config.toml").toURI()).toString();

      ClientConfigProfile profile =
          ClientConfigProfile.load(
              LoadClientConfigProfileOptions.newBuilder()
                  .setConfigFilePath(configFilePath)
                  .build());

      WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions();
      WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions();

      try {
        // Create the workflow client using the loaded configuration
        WorkflowClient client =
            WorkflowClient.newInstance(
                WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions);

        // Test the connection by getting system info
        var systemInfo =
            client
                .getWorkflowServiceStubs()
                .blockingStub()
                .getSystemInfo(
                    io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance());

        logger.info("✅ Client connected successfully!");
        logger.info("   Server version: {}", systemInfo.getServerVersion());

      } catch (Exception e) {
        logger.error("❌ Failed to connect: {}", e.getMessage());
      }

    } catch (Exception e) {
      logger.error("Failed to load configuration: {}", e.getMessage(), e);
      System.exit(1);
    }
  }
}
```

**Environment Variables**

Use the `envconfig` package to set connection options for the Temporal Client using environment variables. For a list of
all available environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

```java {18-19,26-28}
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.envconfig.LoadClientConfigProfileOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import java.nio.file.Paths;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoadFromFile {

  private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class);

  public static void main(String[] args) {
    try {

      ClientConfigProfile profile =
          ClientConfigProfile.load(LoadClientConfigProfileOptions.newBuilder().build());

      WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions();
      WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions();

      try {
        // Create the workflow client using the loaded configuration
        WorkflowClient client =
            WorkflowClient.newInstance(
                WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions);

        // Test the connection by getting system info
        var systemInfo =
            client
                .getWorkflowServiceStubs()
                .blockingStub()
                .getSystemInfo(
                    io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance());

        logger.info("✅ Client connected successfully!");
        logger.info("   Server version: {}", systemInfo.getServerVersion());

      } catch (Exception e) {
        logger.error("❌ Failed to connect: {}", e.getMessage());
      }

    } catch (Exception e) {
      logger.error("Failed to load configuration: {}", e.getMessage(), e);
      System.exit(1);
    }
  }
}
```

**Code**

If you don't want to use environment variables or a configuration file, you can specify connection options directly in
code. This is convenient for local development and testing. You can also load a base configuration from environment
variables or a configuration file, and then override specific options in code.

```java
  // Add the Namespace as a Client Option
  WorkflowClientOptions clientOptions = WorkflowClientOptions
      .newBuilder()
      .setNamespace(namespace)
      .build();

  // Initialize the Client
  WorkflowClient client = WorkflowClient.newInstance(service, clientOptions);
```

## Connect to Temporal Cloud 

You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through [mTLS](/cloud/certificates).
Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to
connecting to an unsecured local development instance:

- Your credentials for authentication.
  - If you are using an API key, provide the API key value.
  - If you are using mTLS, provide the mTLS CA certificate and mTLS private key.
- Your _Namespace and Account ID_ combination, which follows the format `<namespace_id>.<account_id>`.
- The recommended _endpoint_ is the gRPC Namespace endpoint: `<namespace>.<account>.tmprl.cloud:7233`.
  This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability).
  See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options.

You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab.

You can provide these connection options using environment variables, a configuration file, or directly in code.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to
[Environment Configuration](/references/client-environment-configuration).

You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path
to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the
default path `~/.config/temporalio/temporal.toml`.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines a `cloud` profile with the necessary connection options to
connect to Temporal Cloud via an API key:

```toml
# Cloud profile for Temporal Cloud
[profile.cloud]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```

If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and
private key:

```toml
# Cloud profile for Temporal Cloud
[profile.cloud]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
tls_client_cert_data = "your-tls-client-cert-data"
tls_client_key_path = "your-tls-client-key-path"
```

With the connections options defined in the configuration file, use the `LoadClientOptions` function in the `envconfig`
package to create a Temporal Client using the `cloud` profile as follows. After loading the profile, you can also
programmatically override specific connection options before creating the client.

```java {25-30,33-35,42-44}
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.envconfig.LoadClientConfigProfileOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import java.nio.file.Paths;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoadProfile {

  private static final Logger logger = LoggerFactory.getLogger(LoadProfile.class);

  public static void main(String[] args) {
    String profileName = "cloud";

    try {
      String configFilePath =
          Paths.get(LoadProfile.class.getResource("/config.toml").toURI()).toString();

      logger.info("--- Loading '{}' profile from {} ---", profileName, configFilePath);

      // Load specific profile from file with environment variable overrides
      ClientConfigProfile profile =
          ClientConfigProfile.load(
              LoadClientConfigProfileOptions.newBuilder()
                  .setConfigFilePath(configFilePath)
                  .setConfigFileProfile(profileName)
                  .build());

      // Demonstrate programmatic override - fix the incorrect address from staging profile
      ClientConfigProfile.Builder profileBuilder = profile.toBuilder();
      profileBuilder.setAddress("localhost:7233"); // Override the incorrect address
      profile = profileBuilder.build();

      WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions();
      WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions();

      try {
        // Create the workflow client using the loaded configuration
        WorkflowClient client =
            WorkflowClient.newInstance(
                WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions);

        // Test the connection by getting system info
        var systemInfo =
            client
                .getWorkflowServiceStubs()
                .blockingStub()
                .getSystemInfo(
                    io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance());

        logger.info("✅ Client connected successfully!");
        logger.info("   Server version: {}", systemInfo.getServerVersion());

      } catch (Exception e) {
        logger.error("❌ Failed to connect: {}", e.getMessage());
      }

    } catch (Exception e) {
      logger.error("Failed to load configuration: {}", e.getMessage(), e);
      System.exit(1);
    }
  }
}
```

**Environment Variables**

The following environment variables are required to connect to Temporal Cloud:

- `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `<namespace_id>.<account_id>`.
- `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace.
- `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication.
- `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path.
  Required if you are using mTLS authentication.
- `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path.
  Required if you are using mTLS authentication.

Ensure these environment variables exist in your environment before running your Java application.

Import the `io.temporal.envconfig` package to set connection options for the Temporal Client using environment variables. The `ClientConfigProfile.load` method will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence.

```java {17-18,25-27}
import io.temporal.client.WorkflowClientOptions;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.envconfig.LoadClientConfigProfileOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import java.nio.file.Paths;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoadFromFile {

  private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class);

  public static void main(String[] args) {
    try {

      ClientConfigProfile profile =
          ClientConfigProfile.load(LoadClientConfigProfileOptions.newBuilder().build());

      WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions();
      WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions();

      try {
        // Create the workflow client using the loaded configuration
        WorkflowClient client =
            WorkflowClient.newInstance(
                WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions);

        // Test the connection by getting system info
        var systemInfo =
            client
                .getWorkflowServiceStubs()
                .blockingStub()
                .getSystemInfo(
                    io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance());

        logger.info("✅ Client connected successfully!");
        logger.info("   Server version: {}", systemInfo.getServerVersion());

      } catch (Exception e) {
        logger.error("❌ Failed to connect: {}", e.getMessage());
      }

    } catch (Exception e) {
      logger.error("Failed to load configuration: {}", e.getMessage(), e);
      System.exit(1);
    }
  }
}
```

**Code**

Now, when instantiating a Temporal `client` in your Temporal Java SDK code, provide the API key with
`WorkflowServiceStubsOptions` and the Namespace and Account ID in `WorkflowClient.newInstance`:

```java
    WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                .addApiKey(
                    () ->
                        <API key>)
                .setTarget(<endpoint>)
                .setEnableHttps(true)
                ...
                .build());

    WorkflowClient client =
        WorkflowClient.newInstance(
            service, WorkflowClientOptions.newBuilder().setNamespace(<namespace_id>.<account_id>).build());
```

To update the API key, update the `stubOptions`:

```java
String myKey = <APIKey>;
WorkflowServiceStubsOptions stubOptions =
    WorkflowServiceStubsOptions.newBuilder()
        .addApiKey(() -> myKey)
        .build();
// Update by replacing, this must be done in a thread safe way
myKey = "Bearer " + <new APIKey>;
```

To connect to Temporal Cloud using mTLS, you need to provide the mTLS CA certificate and mTLS private key. You can then
use the `SimpleSslContextBuilder` to build an SSL context. Then use the
`WorkflowServiceStubsOptions.Builder.setSslContext` method to set the SSL context.

When you use a remote service, you don't use the `newLocalServiceStubs` convenience method. Instead, set your
connection details as stub configuration options:

```java
  // Set the Service Stub options (SSL context and gRPC endpoint)
  WorkflowServiceStubsOptions stubsOptions = WorkflowServiceStubsOptions
      .newBuilder()
      .setSslContext(sslContext)
      .setTarget(gRPCEndpoint)
      .build();

  // Create a stub that accesses a Temporal Service
  WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions);
```

Each Temporal Cloud service Client has four prerequisites.

- The full Namespace Id from the [Cloud Namespace](https://cloud.temporal.io/namespaces) details page
- The gRPC endpoint from the [Cloud Namespace](https://cloud.temporal.io/namespaces) details page
- Your mTLS private key
- Your mTLS x509 Certificate

Retrieve these values before building your Client.

The following sample generates an SSL context from the mTLS .pem and .key files. Along with the gRPC endpoint, this
information configures a service stub for Temporal Cloud. Add the Namespace to your Client build options and initialize
the new Client:

```java
  // Generate an SSL context
  InputStream clientCertInputStream = new FileInputStream(clientCertPath);
  InputStream clientKeyInputStream = new FileInputStream(clientKeyPath);
  SslContext sslContext = SimpleSslContextBuilder.forPKCS8(clientCertInputStream, clientKeyInputStream).build();

  // Set the Service Stub options (SSL context and gRPC endpoint)
  WorkflowServiceStubsOptions stubsOptions = WorkflowServiceStubsOptions
      .newBuilder()
      .setSslContext(sslContext)
      .setTarget(gRPCEndpoint)
      .build();

  // Create a stub that accesses a Temporal Service
  WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions);

  // Set the Client options
  WorkflowClientOptions clientOptions = WorkflowClientOptions
      .newBuilder()
      .setNamespace(namespace)
      .build();

  // Initialize the Client
  WorkflowClient client = WorkflowClient.newInstance(serviceStub, clientOptions);
```

To rotate an mTLS client certificate without restarting your Worker, build the `SslContext` with gRPC's
[`AdvancedTlsX509KeyManager`](https://grpc.github.io/grpc-java/javadoc/io/grpc/util/AdvancedTlsX509KeyManager.html)
instead of `SimpleSslContextBuilder`. `updateIdentityCredentialsFromFile` schedules a periodic reread of the certificate
and key files, so the same `SslContext` keeps serving fresh credentials for the life of the process:

```java
String tlsCertPath = "/path/to/tls.crt";
String tlsKeyPath = "/path/to/tls.key"; // PKCS8 format

// Reread the certificate and key files every 5 minutes.
AdvancedTlsX509KeyManager keyManager = new AdvancedTlsX509KeyManager();
keyManager.updateIdentityCredentialsFromFile(
    new File(tlsKeyPath), new File(tlsCertPath),
    5, TimeUnit.MINUTES,
    Executors.newSingleThreadScheduledExecutor());

SslContextBuilder sslContextBuilder = SslContextBuilder.forClient();
GrpcSslContexts.configure(sslContextBuilder);
SslContext sslContext = sslContextBuilder.keyManager(keyManager).build();

WorkflowServiceStubsOptions stubsOptions = WorkflowServiceStubsOptions
    .newBuilder()
    .setSslContext(sslContext)
    .setTarget(gRPCEndpoint)
    .build();
WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions);
```

Rotate the certificate by overwriting the files at `tlsCertPath` and `tlsKeyPath`; `keyManager` picks up the new files on
its next scheduled check, and the Worker keeps running against the same `client`/`serviceStub`. See this
[reference implementation](https://github.com/temporal-sa/temporal-worker-cert-rotation) (written for the Go SDK, but the
"What if I am not using the Go SDK?" section covers this Java approach) for a full walkthrough.

## Start a Workflow Execution 

[Workflow Execution](/workflow-execution) semantics rely on several parameters. To start a Workflow Execution, you must
supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific
contextual data, and Workflow Function parameters.

In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from
within another Workflow Execution, use either the [Child Workflow](/develop/java/workflows/child-workflows) or External Workflow
APIs.

See the [Customize Workflow Type](/develop/java/workflows/basics#workflow-type) section to see how to customize the name
of the Workflow Type.

A request to spawn a Workflow Execution causes the Temporal Service to create the first Event
([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The
Temporal Service then creates the first Workflow Task, resulting in the first
[WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event.

Use `WorkflowStub` to start a Workflow Execution from within a Client, and `ExternalWorkflowStub` to start a different
Workflow Execution from within a Workflow.

See [`SignalwithStart`](/develop/java/workflows/message-passing#signal-with-start) to start a Workflow Execution to receive a
Signal from within another Workflow.

### Use WorkflowStub

`WorkflowStub` is a proxy generated by the `WorkflowClient`. Each time a new Workflow Execution is started, an instance
of the Workflow implementation object is created. Then, one of the methods (depending on the Workflow Type of the
instance) annotated with `@WorkflowMethod` can be invoked. As soon as this method returns, the Workflow Execution is
considered to be complete.

You can use a typed or untyped `WorkflowStub` in the client code.

- Typed `WorkflowStub` are useful because they are type safe and allow you to invoke your Workflow methods such as
  `@WorkflowMethod`, `@QueryMethod`, and `@SignalMethod` directly.
- An untyped `WorkflowStub` does not use the Workflow interface, and is not type safe. It is more flexible because it
  has methods from the `WorkflowStub` interface, such as `start`, `signalWithStart`, `getResults` (sync and async),
  `query`, `signal`, `cancel` and `terminate`. Note that the Temporal Java SDK also provides typed `WorkflowStub`
  versions for these methods. When using untyped `WorkflowStub`, we rely on the Workflow Type, Activity Type, Child
  Workflow Type, as well as Query and Signal names. For details, see [Temporal Client](/encyclopedia/temporal-client).

A Workflow Execution can be started either synchronously or asynchronously.

- Synchronous invocation starts a Workflow and then waits for its completion. If the process that started the Workflow
  crashes or stops waiting, the Workflow continues executing. Because Workflows are potentially long-running, and Client
  crashes happen, it is not very commonly found in production use. The following example is a type-safe approach for
  starting a Workflow Execution synchronously.

  ```java
    NotifyUserAccounts workflow = client.newWorkflowStub(
          NotifyUserAccounts.class,
          WorkflowOptions.newBuilder()
                  .setWorkflowId("notifyAccounts")
                  .setTaskQueue(taskQueue)
                  .build()
          );

  // start the Workflow and wait for a result.
    workflow.notify(new String[] { "Account1", "Account2", "Account3", "Account4", "Account5",
                  "Account6", "Account7", "Account8", "Account9", "Account10"});
      }
  // notify(String[] accountIds) is a Workflow method defined in the Workflow Definition.
  ```

- Asynchronous start initiates a Workflow Execution and immediately returns to the caller. This is the most common way
  to start Workflows in production code. The
  [`WorkflowClient`](https://github.com/temporalio/sdk-java/blob/main/temporal-sdk/src/main/java/io/temporal/client/WorkflowClient.java)
  provides some static methods, such as `start`, `execute`, and `signalWithStart`, that help with starting your
  Workflows asynchronously.

  The following examples show how to start Workflow Executions asynchronously, with either typed or untyped
  `WorkflowStub`.
  - **Typed WorkflowStub Example**

    ```java
    // create typed Workflow stub
    FileProcessingWorkflow workflow = client.newWorkflowStub(FileProcessingWorkflow.class,
          WorkflowOptions.newBuilder()
                  .setTaskQueue(taskQueue)
                  .setWorkflowId(workflowId)
                  .build());
    // use WorkflowClient.execute to return future that contains Workflow result or failure, or
    // use WorkflowClient.start to return WorkflowId and RunId of the started Workflow).
    WorkflowClient.start(workflow::greetCustomer);
    ```

  - **Untyped WorkflowStub Example**

    ```java
    WorkflowStub untyped = client.newUntypedWorkflowStub("FileProcessingWorkflow",
          WorkflowOptions.newBuilder()
                  .setWorkflowId(workflowId)
                  .setTaskQueue(taskQueue)
                  .build());

    // blocks until Workflow Execution has been started (not until it completes)
    untyped.start(argument);
    ```

You can call a Dynamic Workflow implementation using an untyped `WorkflowStub`. The following example shows how to call
the Dynamic Workflow implementation in the Client code.

```java
    WorkflowClient client = WorkflowClient.newInstance(service);
    /**
      * Note that for this part of the client code, the dynamic Workflow implementation must
      * be known to the Worker at runtime in order to dispatch Workflow tasks, and may be defined
      * in the Worker definition as:*/
    // worker.registerWorkflowImplementationTypes(DynamicGreetingWorkflowImpl.class);

    /* Create the Workflow stub to call the dynamic Workflow.
    * Note that the Workflow Type is not explicitly registered with the Worker.*/
    WorkflowOptions workflowOptions =
        WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).setWorkflowId(WORKFLOW_ID).build();
    WorkflowStub workflow = client.newUntypedWorkflowStub("DynamicWF", workflowOptions);
```

`DynamicWorkflow` can be used to invoke different Workflow Types. To check what type is running when your Dynamic
Workflow `execute` method runs, use `getWorkflowType()` in the implementation code.

```java
String type = Workflow.getInfo().getWorkflowType();
```

See [Workflow Execution Result](#get-workflow-results) for details on how to get the results of the Workflow Execution.

### Use ExternalWorkflowStub

Use `ExternalWorkflowStub` within a Workflow to invoke, and send Signals to, other Workflows by type.

This helps particularly for executing Workflows written in other language SDKs, as shown in the following example.

```java
@Override
  public String yourWFMethod(String name) {
      ExternalWorkflowStub callOtherWorkflow = Workflow.newUntypedExternalWorkflowStub("OtherWFId");
    }
```

See the [Temporal Polyglot](https://github.com/tsurdilo/temporal-polyglot) code for examples of executing Workflows
written in other language SDKs.

### Start a Workflow on a schedule

You can start a Workflow Execution on a regular schedule by using
[`setCronSchedule`](/develop/java/workflows/schedules#cron-schedule) Workflow option in the Client code.

### Set a Workflow's Task Queue 

The only Workflow Option that you must set is the name of the [Task Queue](/task-queue).

For your code to execute, a Worker Process must be running. This process needs a Worker Entity that is polling the same
Task Queue name.

Set the Workflow Task Queue with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setTaskQueue`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `String`
- Default: none

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWF")
                // Set the Task Queue
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                .build());
```

### Set a Workflow Id 

Although it is not required, we recommend providing your own
[Workflow Id](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity
identifier, such as an order identifier or customer identifier.

Set the Workflow Id with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setWorkflowId`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `String`
- Default: none

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                // Set the Workflow Id
                .setWorkflowId("YourWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                .build());
```

### Java WorkflowOptions reference 

Create a
[`newWorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) in
the Temporal Client code, call the instance of the Workflow, and set the Workflow options with the
[`WorkflowOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html)
class.

The following fields are available:

| Option                                                  | Required             | Type                                                                                                                 |
| ------------------------------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------- |
| [`WorkflowId`](#workflowid)                             | No (but recommended) | String                                                                                                               |
| [`TaskQueue`](#taskqueue)                               | **Yes**              | String                                                                                                               |
| [`WorkflowExecutionTimeout`](#workflowexecutiontimeout) | No                   | `Duration`                                                                                                           |
| [`WorkflowRunTimeout`](#workflowruntimeout)             | No                   | `Duration`                                                                                                           |
| [`WorkflowTaskTimeout`](#workflowtasktimeout)           | No                   | `Duration`                                                                                                           |
| [`WorkflowIdReusePolicy`](#workflowidreusepolicy)       | No                   | `WorkflowIdReusePolicy`                                                                                              |
| [`RetryOptions`](#retryoptions)                         | No                   | [`RetryOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/RetryOptions.html)    |
| [`CronSchedule`](#cronschedule)                         | No                   | String                                                                                                               |
| [`Memo`](#memo)                                         | No                   | `String`                                                                                                             |
| [`SearchAttributes`](#searchattributes)                 | No                   | `Map<String, Object>`                                                                                                |

#### WorkflowId

Set the Workflow Id with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setWorkflowId`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `String`
- Default: none

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                // Set the Workflow Id
                .setWorkflowId("YourWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                .build());
```

#### TaskQueue

Set the Workflow Task Queue with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setTaskQueue`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `String`
- Default: none

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWF")
                // Set the Task Queue
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                .build());
```

#### WorkflowExecutionTimeout

Set the [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setWorkflowExecutionTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `Duration`
- Default: Unlimited

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Workflow Execution Timeout duration
                .setWorkflowExecutionTimeout(Duration.ofSeconds(10))
                .build());
```

#### WorkflowRunTimeout

Set the Workflow Run Timeout with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setWorkflowRunTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `Duration`
- Default: Same as [WorkflowExecutionTimeout](#workflowexecutiontimeout).

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Workflow Run Timeout duration
                .setWorkflowRunTimeout(Duration.ofSeconds(10))
                .build());
```

#### WorkflowTaskTimeout

Set the Workflow Task Timeout with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setWorkflowTaskTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `Duration`
- Default: 10 seconds.
- Values: Maximum accepted value is 60 seconds.

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Workflow Task Timeout duration
                .setWorkflowTaskTimeout(Duration.ofSeconds(10))
                .build());
```

#### WorkflowIDReusePolicy

- Type: `WorkflowIdReusePolicy`
- Default: `AllowDuplicate`
- Values:
  - `enums.AllowDuplicateFailedOnly`: The Workflow can start if the earlier Workflow Execution failed, Canceled, or
    Terminated.
  - `AllowDuplicate`: The Workflow can start regardless of the earlier Execution's closure status.
  - `RejectDuplicate`: The Workflow can not start if there is an earlier Run.

```java
//create Workflow stub for GreetWorkflowInterface
GreetWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("GreetWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Workflow Id Reuse Policy
                .setWorkflowIdReusePolicy(
                        WorkflowIdReusePolicy.WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE)
                .build());
```

#### RetryOptions

To set a Workflow Retry Options in the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance use
[`WorkflowOptions.Builder.setWorkflowRetryOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `RetryOptions`
- Default: `Null` which means no retries will be attempted.

```java
//create Workflow stub for GreetWorkflowInterface
GreetWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("GreetWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Workflow Retry Options
                .setRetryOptions(RetryOptions.newBuilder()
                .build());
```

#### CronSchedule

A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the
call to spawn a Workflow Execution.

A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made.

Set the Cron Schedule with the
[`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html)
instance in the Client code using
[`WorkflowOptions.Builder.setCronSchedule`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

Setting `setCronSchedule` changes the Workflow Execution into a Temporal Cron Job. The default timezone for a Cron is
UTC.

- Type: `String`
- Default: None

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    YourWorker.yourclient.newWorkflowStub(
        YourWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWF")
                .setTaskQueue(YourWorker.TASK_QUEUE)
                // Set Cron Schedule
                .setCronSchedule("* * * * *")
                .build());
```

For more details, see the
[HelloCron Sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloCron.java).

#### Memo

- Type: `String`
- Default: None

```java
//create Workflow stub for GreetWorkflowInterface
GreetWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("GreetWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Memo. You can set additional non-indexed info via Memo
                        .setMemo(ImmutableMap.of(
                                "memoKey", "memoValue"
                        ))
                .build());
```

#### SearchAttributes

Search Attributes are additional indexed information attributed to Workflow and used for search and visibility. These
can be used in a query of List/Scan/Count Workflow APIs. The key and its value type must be registered on Temporal
server side.

- Type: `Map<String, Object>`
- Default: None

```java
private static void parentWorkflow() {
        ChildWorkflowOptions childworkflowOptions =
                ChildWorkflowOptions.newBuilder()
                        // Set Search Attributes
                        .setSearchAttributes(ImmutableMap.of("MySearchAttributeNAme", "value"))
                        .build();
```

The following Java types are supported:

- String
- Long, Integer, Short, Byte
- Boolean
- Double
- OffsetDateTime
- Collection of the types in this list.

## Get Workflow results 

If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id.

The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its
result.

It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time
(asynchronous execution).

In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and
results of Workflow Executions.

A synchronous Workflow Execution blocks your client thread until the Workflow Execution completes (or fails) and get the
results (or error in case of failure).

The following example is a type-safe approach for getting the results of a synchronous Workflow Execution.

```java
 FileProcessingWorkflow workflow = client.newWorkflowStub(
                FileProcessingWorkflow.class,
                WorkflowOptions.newBuilder()
                        .setWorkflowId(workflowId)
                        .setTaskQueue(taskQueue)
                        .build();

// start sync and wait for results (or failure)
String result = workflow.processfile(new Argument());
```

An asynchronous Workflow Execution immediately returns a value to the caller.

The following examples show how to get the results of a Workflow Execution through typed and untyped `WorkflowStub`.

- **Typed WorkflowStub Example**

  ```java
  // create typed Workflow stub
  FileProcessingWorkflow workflow = client.newWorkflowStub(FileProcessingWorkflow.class,
                WorkflowOptions.newBuilder()
                        .setTaskQueue(taskQueue)
                        .setWorkflowId(workflowId)
                        .build());
  // use WorkflowClient.execute (if your Workflow takes in arguments) or WorkflowClient.start (for zero arguments)
  WorkflowClient.start(workflow::greetCustomer);
  ```

- **Untyped WorkflowStub Example**

  ```java
  WorkflowStub untyped = client.newUntypedWorkflowStub("FileProcessingWorkflow",
                  WorkflowOptions.newBuilder()
                          .setWorkflowId(workflowId)
                          .setTaskQueue(taskQueue)
                          .build());

  // blocks until Workflow Execution has been started (not until it completes)
  untyped.start(argument);
  ```

If you need to wait for a Workflow Execution to complete after an asynchronous start, the most straightforward way is to
call the blocking Workflow instance again.

Note that if `WorkflowOptions.WorkflowIdReusePolicy` is not set to `AllowDuplicate`, then instead of throwing
`DuplicateWorkflowException`, it reconnects to an existing Workflow and waits for its completion.

The following example shows how to do this from a different process than the one that started the Workflow Execution.

```java
YourWorkflow workflow = client.newWorkflowStub(YourWorkflow.class, workflowId);

// Returns the result after waiting for the Workflow to complete.
String result = workflow.yourMethod();
```

Another way to connect to an existing Workflow and wait for its completion from another process, is to use
`UntypedWorkflowStub`. For example:

```java
WorkflowStub workflowStub = client.newUntypedWorkflowStub(workflowType, workflowOptions);

// Returns the result after waiting for the Workflow to complete.
String result = untyped.getResult(String.class);
```

### Get last completion result

For a Temporal Cron Job, get the result of previous successful runs using `GetLastCompletionResult()`. The method
returns `null` if there is no previous completion. The following example shows how to implement this in a Workflow.

```java
public String cronWorkflow() {
    String lastProcessedFileName = Workflow.getLastCompletionResult(String.class);

    // Process work starting from the lastProcessedFileName.
    // Business logic implementation goes here.
    // Updates lastProcessedFileName to the new value.

    return lastProcessedFileName;
}
```

Note that this works even if one of the Cron schedule runs failed. The next schedule will still get the last successful
result if it ever successfully completed at least once. For example, for a daily cron Workflow, if the run succeeds on
the first day and fails on the second day, then the third day run will get the result from first day's run using these
APIs.

---

# External Storage - Java SDK

Source: https://docs.temporal.io/develop/java/data-handling/external-storage

> Offload large payloads to Amazon S3 using the claim check pattern in the Java SDK.

> **Pre-release**
> APIs and configuration may change before General Availability. Join the
> [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for
> help.

When your Workflows or Activities handle data larger than the Temporal Service payload limit, offload the payloads to
an external store such as Amazon S3. Temporal stores a small reference in Event History, and the Java SDK retrieves the
payload before your Workflow or Activity receives it.

This page shows how to configure the Java SDK with Amazon S3. For the claim check pattern, retention requirements, and
storage design guidance, see [External Storage](/external-storage).

## Store and retrieve large payloads with Amazon S3

The Java SDK includes an experimental S3 storage driver. It requires Java SDK v1.39.0 or later.

### Prerequisites

- An S3 bucket that your Temporal Client and Workers can reach. Configure [lifecycle management](/external-storage#lifecycle)
  so objects remain available for the Workflow lifetime and Namespace retention period.
- AWS credentials that can read and write S3 objects. The AWS SDK for Java reads its standard credential provider chain,
  including environment variables, IAM roles, and AWS configuration files.
- The AWS SDK v2 S3 driver module. Use the same version as your Temporal Java SDK dependency. The module includes the
  generic S3 driver and AWS S3 client as transitive dependencies:

  ```gradle
  implementation "io.temporal:temporal-payload-storage-s3driver-awssdkv2:1.39.0"
  ```

### Procedure

1. Create an asynchronous S3 client, wrap it in an `S3AsyncClientAdapter`, and create an `S3StorageDriver`:

   <!--SNIPSTART java-s3-driver-create-->
   [java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/S3StorageDriverExamples.java](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/S3StorageDriverExamples.java)
   ```java
   S3AsyncClient s3Client = S3AsyncClient.builder().region(Region.US_EAST_2).build();

   S3StorageDriver driver =
       S3StorageDriver.newBuilder()
           .setClient(new S3AsyncClientAdapter(s3Client))
           .setBucket("my-temporal-payloads")
           .build();
   ```
   <!--SNIPEND-->

   To select an S3 bucket for each payload, use `setBucketResolver()` instead of `setBucket()`.

2. Add the driver to `ExternalStorage`, then set it on `WorkflowClientOptions`. A Worker created from that Client
   inherits the configuration:

   <!--SNIPSTART java-s3-external-storage-setup-->
   [java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/S3StorageDriverExamples.java](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/S3StorageDriverExamples.java)
   ```java
   ExternalStorage externalStorage = ExternalStorage.newBuilder().setDriver(driver).build();

   WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
   WorkflowClient client =
       WorkflowClient.newInstance(
           service,
           WorkflowClientOptions.newBuilder().setExternalStorage(externalStorage).build());
   WorkerFactory factory = WorkerFactory.newInstance(client);
   Worker worker = factory.newWorker("my-task-queue");
   ```
   <!--SNIPEND-->

   Configure External Storage on every Client and Worker process that can send or receive an offloaded payload. For
   example, a Client that starts a Workflow needs it to offload a large input, and a separate Worker process needs it to
   retrieve that input.

By default, the SDK offloads serialized Payloads that are 256 KiB or larger. For other thresholds, see
[Configure payload size threshold](#configure-payload-size-threshold).

The S3 driver stores serialized Payloads under content-addressed keys derived from their SHA-256 hash. It reuses an
existing object when a Workflow Run passes the same payload again, verifies the hash on retrieval, and rejects payloads
larger than 50 MiB by default. Use `setMaxPayloadSize()` to change that limit.

## Implement a custom storage driver

To use a storage system other than S3, implement `StorageDriver`. The following driver stores Payload protobuf messages
on local disk. Use it for local development and tests, not in production: Workers on other hosts cannot read its files.
A production driver must use a durable store that every Client and Worker can access.

<!--SNIPSTART java-custom-storage-driver-->
[java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/LocalDiskStorageDriver.java](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/LocalDiskStorageDriver.java)
```java
class LocalDiskStorageDriver implements StorageDriver {
  private static final String CLAIM_PATH = "path";

  private final Path storeDir;

  LocalDiskStorageDriver(Path storeDir) {
    this.storeDir = storeDir;
  }

  @Override
  public String getName() {
    return "local-disk";
  }

  @Override
  public String getType() {
    return "local-disk";
  }

  @Override
  public CompletableFuture<List<StorageDriverClaim>> store(
      StorageDriverStoreContext context, List<Payload> payloads) {
    try {
      context.getCancellationToken().throwIfCancellationRequested();
      Path directory = storeDirectory(context);
      Files.createDirectories(directory);

      List<StorageDriverClaim> claims = new ArrayList<>();
      for (Payload payload : payloads) {
        context.getCancellationToken().throwIfCancellationRequested();
        Path file = directory.resolve(UUID.randomUUID() + ".bin");
        Files.write(file, payload.toByteArray());
        claims.add(new StorageDriverClaim(Map.of(CLAIM_PATH, file.toString())));
      }
      return CompletableFuture.completedFuture(claims);
    } catch (IOException e) {
      return failedFuture(new IllegalStateException("Could not write Payload", e));
    }
  }

  @Override
  public CompletableFuture<List<Payload>> retrieve(
      StorageDriverRetrieveContext context, List<StorageDriverClaim> claims) {
    try {
      context.getCancellationToken().throwIfCancellationRequested();

      List<Payload> payloads = new ArrayList<>();
      for (StorageDriverClaim claim : claims) {
        context.getCancellationToken().throwIfCancellationRequested();
        Path file = Paths.get(claim.getClaimData().get(CLAIM_PATH));
        payloads.add(Payload.parseFrom(Files.readAllBytes(file)));
      }
      return CompletableFuture.completedFuture(payloads);
    } catch (IOException e) {
      return failedFuture(new IllegalStateException("Could not read Payload", e));
    }
  }

  private Path storeDirectory(StorageDriverStoreContext context) {
    StorageDriverTargetInfo target = context.getTarget();
    if (target instanceof StorageDriverWorkflowInfo) {
      StorageDriverWorkflowInfo workflow = (StorageDriverWorkflowInfo) target;
      if (workflow.getId() != null) {
        return storeDir.resolve(workflow.getNamespace()).resolve(workflow.getId());
      }
    } else if (target instanceof StorageDriverActivityInfo) {
      StorageDriverActivityInfo activity = (StorageDriverActivityInfo) target;
      if (activity.getId() != null) {
        return storeDir.resolve(activity.getNamespace()).resolve(activity.getId());
      }
    }
    return storeDir;
  }

  private static <T> CompletableFuture<T> failedFuture(Throwable error) {
    CompletableFuture<T> result = new CompletableFuture<>();
    result.completeExceptionally(error);
    return result;
  }
}
```
<!--SNIPEND-->

`store()` writes each serialized Payload and returns one `StorageDriverClaim` with its path, in the same order. It uses
the Workflow or Activity information from `StorageDriverStoreContext` to group the files. `retrieve()` reads each file
from its claim and returns the original Payloads in the same order. The Payload Converter and Payload Codec have already
encoded the application data before the driver receives it.

Give every driver instance a stable, unique `getName()` value. The SDK records that name in a reference and uses it to
choose the driver during retrieval. `getType()` identifies the driver implementation for Worker heartbeats and metrics;
keep it the same for every configuration of the same driver. For an asynchronous storage client, use each context's
cancellation token to cancel its in-flight request when the SDK abandons the operation.

Register the driver with `ExternalStorage` using the setup in [Store and retrieve large payloads with Amazon S3](#store-and-retrieve-large-payloads-with-amazon-s3).

## Configure payload size threshold

The size threshold applies to the serialized Payload, including its metadata. By default, serialized Payloads that are
256 KiB or larger are offloaded. Payloads smaller than the threshold stay inline in Event History. Set a higher value to
offload less data or set the value to `0` to offload every Payload.

<!--SNIPSTART java-external-storage-threshold-->
[java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/ExternalStorageConfigurationExamples.java](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/ExternalStorageConfigurationExamples.java)
```java
return ExternalStorage.newBuilder().setDriver(driver).setPayloadSizeThreshold(0).build();
```
<!--SNIPEND-->

## Use multiple storage drivers

When you register more than one driver, you must set a `StorageDriverSelector`. The selector chooses the registered
driver that stores each new Payload. It can return `null` to leave a specific Payload inline. Drivers that the selector
does not choose remain available for retrieval, which lets you migrate storage backends without making existing
references unreadable.

Every registered driver needs a distinct `getName()` value. For example, set a distinct name on each `S3StorageDriver`
when registering two S3 drivers. The following configuration stores new Payloads with `preferredDriver`, while keeping
`legacyDriver` available to retrieve references that it created:

<!--SNIPSTART java-external-storage-multiple-drivers-->
[java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/ExternalStorageConfigurationExamples.java](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/ExternalStorageConfigurationExamples.java)
```java
return ExternalStorage.newBuilder()
    .setDrivers(Arrays.asList(preferredDriver, legacyDriver))
    .setDriverSelector((context, payload) -> preferredDriver)
    .build();
```
<!--SNIPEND-->

## Configure multi-region durability with Amazon S3

To tolerate an AWS Region failure, configure [Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html)
and an [S3 Multi-Region Access Point (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/), then set
the driver bucket to the MRAP ARN. Enable ARN-region routing on the AWS SDK client so it sends a request to the Region
in the ARN:

<!--SNIPSTART java-s3-mrap-driver-create-->
[java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/S3StorageDriverExamples.java](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/java-sandbox/external-storage/src/main/java/io/temporal/docs/externalstorage/S3StorageDriverExamples.java)
```java
S3AsyncClient s3Client =
    S3AsyncClient.builder()
        .region(Region.US_EAST_2)
        .serviceConfiguration(S3Configuration.builder().useArnRegionEnabled(true).build())
        .build();

return S3StorageDriver.newBuilder()
    .setClient(new S3AsyncClientAdapter(s3Client))
    .setBucket("arn:aws:s3::123456789012:accesspoint/example.mrap")
    .build();
```
<!--SNIPEND-->

CRR is asynchronous. During replication lag, a Worker in another Region can temporarily fail to retrieve a new object.
Use appropriate Activity retry policies and prefer the same Region for an immediate read. See [Durable External Storage](/external-storage#durable-external-storage)
for the replication trade-offs and [Replication Time Control](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-time-control.html)
if you need a replication-time service-level agreement.

## Manage external objects

Temporal does not delete objects from your S3 bucket. Configure an S3 lifecycle rule with a TTL longer than the maximum
Workflow Run Timeout plus the Namespace retention period. For the formula and guidance for multi-region storage, see
[Lifecycle management](/external-storage#lifecycle) and [Durable External Storage](/external-storage#durable-external-storage).

---

# Integrations - Java SDK

Source: https://docs.temporal.io/develop/java/integrations

> This section covers integrations with the Java SDK

The following integrations are available for the Temporal Java SDK.
These integrations are built on the Temporal Java SDK's [Plugin system](/develop/plugins-guide), which you can also use to build your own integrations.

- [Parseable](https://github.com/parseablehq/temporal-plugin-java/blob/main/INTEGRATION.md) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(Java · Agent observability)_
- [Spring AI](/develop/java/integrations/spring-ai) — Build AI-powered Java applications with durable Spring AI tool calls. _(Java · Agent framework)_
- [Spring Boot](/develop/java/integrations/spring-boot-integration) — Use Temporal natively in Spring Boot with auto-configuration and dependency injection. _(Java · Framework)_

---

# Spring AI integration - Java SDK

Source: https://docs.temporal.io/develop/java/integrations/spring-ai

> Run Spring AI agents on Temporal with durable model calls, Activity tools, and replay-safe orchestration using Spring Boot.

[Spring AI](https://docs.spring.io/spring-ai/reference/) is an agent framework for Java applications — chat clients, tool calling, vector stores, embeddings, and MCP servers, all wired through Spring Boot. 

The [Temporal Spring AI integration](https://central.sonatype.com/artifact/io.temporal/spring-ai) makes Spring AI agents durable: model calls run through Temporal Activities recorded in Event history, and tools are dispatched per their type so each kind lands in the right place in Workflow execution — Activity stubs and Nexus stubs as durable operations, `@SideEffectTool` classes wrapped in `Workflow.sideEffect`, and plain tools running directly in Workflow code. Agents retry on failure and replay deterministically without changing how you write Spring AI code.

The integration is built on the Temporal Java SDK's [Plugin system](/develop/plugins-guide) and is distributed as the standalone `io.temporal:spring-ai` module alongside the existing [Spring Boot integration](/develop/java/integrations/spring-boot-integration).

> **Public Preview**

## Prerequisites

Your application needs a Java runtime, with Spring Boot, Spring AI, and the Temporal Java SDK on its classpath. Refer to the [integration's compatibility requirements](https://github.com/temporalio/ai-integrations/tree/main/java/spring-ai#requirements-and-installation) for supported versions.

You also need the [`temporal-spring-boot-starter`](/develop/java/integrations/spring-boot-integration) and a Spring AI model starter (for example, `spring-ai-starter-model-openai`). The integration does not pull in a model provider on its own.

## Add the dependency

Add `io.temporal:spring-ai` alongside `temporal-spring-boot-starter` and a Spring AI model starter (for example, `spring-ai-starter-model-openai`). The Spring AI integration is versioned separately from the Temporal Java SDK.

Choose a compatible [release from Maven Central](https://central.sonatype.com/artifact/io.temporal/spring-ai).
Set `temporal-spring-ai.version` in Maven or `temporalSpringAiVersion` in Gradle to that version.

**[Apache Maven](https://maven.apache.org/):**

```xml
<dependency>
    <groupId>io.temporal</groupId>
    <artifactId>spring-ai</artifactId>
    <version>${temporal-spring-ai.version}</version>
</dependency>
```

**[Gradle Groovy DSL](https://gradle.org/):**

```groovy
implementation "io.temporal:spring-ai:${temporalSpringAiVersion}"
```

When `io.temporal:spring-ai` is on the classpath, the `SpringAiPlugin` auto-registers `ChatModelActivity` with all Temporal Workers created by the Spring Boot integration. Optional Activities are auto-configured when their dependencies are present:

| Feature      | Dependency      | Registered Activity      |
| ------------ | --------------- | ------------------------ |
| Vector store | `spring-ai-rag` | `VectorStoreActivity`    |
| Embeddings   | `spring-ai-rag` | `EmbeddingModelActivity` |
| MCP          | `spring-ai-mcp` | `McpClientActivity`      |

### Migrate from the previous package

When moving from `io.temporal:temporal-spring-ai`, follow the
[migration guide](https://github.com/temporalio/ai-integrations/tree/main/java/spring-ai#moving-from-spring-ai-1)
to align your application's Spring AI and Spring Boot BOMs.

Pass option builders to `ChatClient`, such as `.defaultOptions(OpenAiChatOptions.builder().reasoningEffort("high"))`, or call `.mutate()` on existing options.

Complete existing Workflow Executions before switching, or keep them on separate Worker deployments. Workflow histories
created with `io.temporal:temporal-spring-ai` are incompatible with `io.temporal:spring-ai`.

## Call a chat model from a Workflow

Use `ActivityChatModel` as a Spring AI `ChatModel` inside a Workflow. Every call goes through a Temporal Activity, so model responses are durable and retried per your Activity options.

Wrap `ActivityChatModel` in a `TemporalChatClient` to build prompts and register tools:

<!--SNIPSTART samples-java-spring-ai-chat-workflow-init-->
[springai/basic/src/main/java/io/temporal/samples/springai/chat/ChatWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/ChatWorkflowImpl.java)
```java
@WorkflowInit
public ChatWorkflowImpl(String systemPrompt) {
  // Build an activity-backed chat model. The factory creates the activity stub
  // internally and registers per-call Summaries on the Temporal UI.
  ActivityChatModel activityChatModel = ActivityChatModel.forDefault();

  // Create an activity stub for weather tools - these execute as durable activities
  WeatherActivity weatherTool =
      Workflow.newActivityStub(
          WeatherActivity.class,
          ActivityOptions.newBuilder()
              .setStartToCloseTimeout(Duration.ofSeconds(30))
              .setRetryOptions(RetryOptions.newBuilder().setMaximumAttempts(3).build())
              .build());

  // Create deterministic tools - these execute directly in the workflow
  StringTools stringTools = new StringTools();

  // Create side-effect tools - these are wrapped in Workflow.sideEffect()
  // The result is recorded in history, making replay deterministic
  TimestampTools timestampTools = new TimestampTools();

  // Create chat memory - uses in-memory storage that gets rebuilt on replay
  ChatMemory chatMemory =
      MessageWindowChatMemory.builder()
          .chatMemoryRepository(new InMemoryChatMemoryRepository())
          .maxMessages(20)
          .build();

  // Build a TemporalChatClient with tools and memory
  // - Activity stubs (weatherTool) become durable AI tools
  // - plain workflow tool classes (stringTools) execute directly in workflow
  // - @SideEffectTool classes (timestampTools) are wrapped in sideEffect()
  // - MessageChatMemoryAdvisor maintains conversation history
  this.chatClient =
      TemporalChatClient.builder(activityChatModel)
          .defaultSystem(systemPrompt)
          .defaultTools(weatherTool, stringTools, timestampTools)
          .defaultAdvisors(
              advisors ->
                  advisors
                      .advisors(MessageChatMemoryAdvisor.builder(chatMemory).build())
                      .param(ChatMemory.CONVERSATION_ID, Workflow.getInfo().getWorkflowId()))
          .build();
}

```
<!--SNIPEND-->

`ActivityChatModel.forDefault()` resolves to the default Spring AI `ChatModel` bean. To target a specific model in a multi-model application, pass its bean name to `ActivityChatModel.forModel("openai")`.

> **📝 Note:**
>
> Streaming responses are not currently supported.

## Register tools

In Spring AI, [tools](https://docs.spring.io/spring-ai/reference/api/tools.html) are methods the model can choose to call to fetch data or take action — you make them available to a chat client by registering them, typically through `ChatClient.defaultTools(...)` or per-prompt `tools(...)`. The chat client advertises the methods to the model, the model decides which (if any) to call, and the framework runs the chosen method and feeds the result back into the conversation.

The Temporal integration extends this by inspecting the type of each tool you register and dispatching it to the appropriate Temporal primitive, so you can mix durable and in-Workflow tools in the same chat client. The integration handles Temporal determinism for you when the tool is durable, and gives you control when it isn't.

### Activity stubs

An interface annotated with both `@ActivityInterface` and Spring AI `@Tool` methods is auto-detected and executed as a Temporal Activity. Use this for external calls that need retries and timeouts.

<!--SNIPSTART samples-java-spring-ai-activity-tool-->
[springai/basic/src/main/java/io/temporal/samples/springai/chat/WeatherActivity.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/WeatherActivity.java)
```java
@ActivityInterface
public interface WeatherActivity {

  /**
   * Gets the current weather for a city.
   *
   * <p>The {@code @Tool} annotation makes this method available to the AI model, while the
   * {@code @ActivityInterface} ensures it executes as a Temporal activity.
   *
   * @param city the name of the city
   * @return a description of the current weather
   */
  @Tool(
      description =
          "Get the current weather for a city. Returns temperature, conditions, and humidity.")
  @ActivityMethod
  String getWeather(
      @ToolParam(description = "The name of the city (e.g., 'Seattle', 'New York')") String city);

  /**
   * Gets the weather forecast for a city.
   *
   * @param city the name of the city
   * @param days the number of days to forecast (1-7)
   * @return the weather forecast
   */
  @Tool(description = "Get the weather forecast for a city for the specified number of days.")
  @ActivityMethod
  String getForecast(
      @ToolParam(description = "The name of the city") String city,
      @ToolParam(description = "Number of days to forecast (1-7)") int days);
}
```
<!--SNIPEND-->

### Nexus service stubs

Nexus service stubs with `@Tool` methods are auto-detected and invoked as [Nexus operations](/develop/java/nexus), enabling cross-Namespace tool calls.

### `@SideEffectTool`

Classes annotated with `@SideEffectTool` have each `@Tool` method wrapped in `Workflow.sideEffect()`. The result is recorded in history on first execution and replayed from history afterward. Use this for cheap, non-deterministic operations such as timestamps or UUIDs.

<!--SNIPSTART samples-java-spring-ai-side-effect-tool-->
[springai/basic/src/main/java/io/temporal/samples/springai/chat/TimestampTools.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/TimestampTools.java)
```java
@SideEffectTool
public class TimestampTools {

  private static final DateTimeFormatter FORMATTER =
      DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss z").withZone(ZoneId.systemDefault());

  /**
   * Gets the current date and time.
   *
   * <p>This is non-deterministic (returns different values each time), but wrapped in sideEffect()
   * it becomes safe for workflow replay.
   *
   * @return the current date and time as a formatted string
   */
  @Tool(description = "Get the current date and time")
  public String getCurrentDateTime() {
    return FORMATTER.format(Instant.now());
  }

  /**
   * Gets the current Unix timestamp in milliseconds.
   *
   * @return the current time in milliseconds since epoch
   */
  @Tool(description = "Get the current Unix timestamp in milliseconds")
  public long getCurrentTimestamp() {
    return System.currentTimeMillis();
  }

  /**
   * Generates a random UUID.
   *
   * @return a new random UUID string
   */
  @Tool(description = "Generate a random UUID")
  public String generateUuid() {
    return UUID.randomUUID().toString();
  }

  /**
   * Gets the current date and time in a specific timezone.
   *
   * @param timezone the timezone ID (e.g., "America/New_York", "UTC", "Europe/London")
   * @return the current date and time in the specified timezone
   */
  @Tool(description = "Get the current date and time in a specific timezone")
  public String getDateTimeInTimezone(
      @ToolParam(description = "Timezone ID (e.g., 'America/New_York', 'UTC', 'Europe/London')")
          String timezone) {
    try {
      ZoneId zoneId = ZoneId.of(timezone);
      DateTimeFormatter formatter =
          DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss z").withZone(zoneId);
      return formatter.format(Instant.now());
    } catch (Exception e) {
      return "Invalid timezone: " + timezone + ". Use formats like 'America/New_York' or 'UTC'.";
    }
  }
}
```
<!--SNIPEND-->

### Plain tools

Any class with `@Tool` methods that isn't an Activity stub, Nexus stub, or `@SideEffectTool` runs directly on the Workflow thread. Use this for inherently deterministic tools (such as updating in-memory agent state), or for orchestration of durable primitives as you need, for example calling multiple Activities, child Workflows, wait conditions, or other Temporal durable primitives.

<!--SNIPSTART samples-java-spring-ai-plain-tool-->
[springai/basic/src/main/java/io/temporal/samples/springai/chat/StringTools.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/StringTools.java)
```java
public class StringTools {

  @Tool(description = "Reverse a string, returning the characters in opposite order")
  public String reverse(@ToolParam(description = "The string to reverse") String input) {
    if (input == null) {
      return null;
    }
    return new StringBuilder(input).reverse().toString();
  }

  @Tool(description = "Count the number of words in a text")
  public int countWords(@ToolParam(description = "The text to count words in") String text) {
    if (text == null || text.isBlank()) {
      return 0;
    }
    return text.trim().split("\\s+").length;
  }

  @Tool(description = "Convert text to all uppercase letters")
  public String toUpperCase(@ToolParam(description = "The text to convert") String text) {
    if (text == null) {
      return null;
    }
    return text.toUpperCase(java.util.Locale.ROOT);
  }

  @Tool(description = "Convert text to all lowercase letters")
  public String toLowerCase(@ToolParam(description = "The text to convert") String text) {
    if (text == null) {
      return null;
    }
    return text.toLowerCase(java.util.Locale.ROOT);
  }

  @Tool(description = "Check if a string is a palindrome (reads the same forwards and backwards)")
  public boolean isPalindrome(@ToolParam(description = "The text to check") String text) {
    if (text == null) {
      return false;
    }
    String normalized = text.toLowerCase(java.util.Locale.ROOT).replaceAll("\\s+", "");
    String reversed = new StringBuilder(normalized).reverse().toString();
    return normalized.equals(reversed);
  }
}
```
<!--SNIPEND-->

## Activity options and retry behavior

`ActivityChatModel.forDefault()` and `forModel(name)` build the chat Activity stub with sensible defaults: a 2-minute start-to-close timeout, 3 attempts, and `org.springframework.ai.retry.NonTransientAiException` and `java.lang.IllegalArgumentException` classified as non-retryable so a bad API key or invalid prompt fails fast.

Pass an `ActivityOptions` directly when you need finer control — a specific Task Queue, [heartbeats](/develop/java/activities/execution#heartbeattimeout), [priority](/develop/task-queue-priority-fairness), or a custom `RetryOptions`:

```java
ActivityChatModel chatModel = ActivityChatModel.forDefault(
        ActivityOptions.newBuilder(ActivityChatModel.defaultActivityOptions())
                .setTaskQueue("chat-heavy")
                .build());
```

For configuration-driven per-model overrides, declare a `ChatModelActivityOptions` bean. The plugin consults it whenever `forDefault()` or `forModel(name)` runs in a Workflow. Use the special key `ChatModelTypes.DEFAULT_MODEL_NAME` (the literal `"default"`) as a global catch-all that applies to any model not explicitly listed — including models contributed by third-party starters:

<!--SNIPSTART samples-java-spring-ai-per-model-options-->
[springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/ChatModelConfig.java](https://github.com/temporalio/samples-java/blob/main/springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/ChatModelConfig.java)
```java
@Bean
public ChatModelActivityOptions chatModelActivityOptions() {
  return new ChatModelActivityOptions(
      Map.of(
          "anthropicChatModel",
          ActivityOptions.newBuilder(ActivityChatModel.defaultActivityOptions())
              .setStartToCloseTimeout(Duration.ofMinutes(5))
              .setScheduleToCloseTimeout(Duration.ofMinutes(15))
              .build()));
}
```
<!--SNIPEND-->

Keys that neither match a registered `ChatModel` bean nor equal `"default"` cause plugin construction to fail, so a typo surfaces at startup rather than at first call.

`ActivityMcpClient.create()` and `create(ActivityOptions)` work the same way for MCP tool calls, with a 30-second default timeout.

### Disable provider retries

Disable the provider SDK's retries so each Temporal Activity attempt makes one provider request. For the Spring Boot OpenAI chat model, set:

```properties
spring.ai.openai.chat.max-retries=0
```

For manually constructed OpenAI or Anthropic models, set `.maxRetries(0)` on the provider options before building the Worker's model. Setting retries on a Workflow prompt does not reconfigure an existing provider client.

OpenAI HTTP failures retain their native exception types. The integration classifies retryable failures and honors provider retry delays; see the [provider retry reference](https://github.com/temporalio/ai-integrations/tree/main/java/spring-ai#provider-retries) for details.

## Provider-specific chat options

Provider-specific `ChatOptions` subclasses — for example, `AnthropicChatOptions` to enable extended thinking, or `OpenAiChatOptions` to set `reasoning_effort` — pass through the Activity boundary unchanged. Attach them via `ChatClient.defaultOptions(...)` and the plugin re-applies them on the Activity side before calling the underlying model:

<!--SNIPSTART samples-java-spring-ai-provider-options-->
[springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/MultiModelWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/MultiModelWorkflowImpl.java)
```java
AnthropicChatOptions.Builder thinkingOptions =
    AnthropicChatOptions.builder().thinkingEnabled(1024).temperature(1.0).maxTokens(4096);
chatClients.put(
    "think",
    TemporalChatClient.builder(anthropicModel)
        .defaultSystem(
            "You are a helpful assistant powered by Anthropic with extended thinking. "
                + "Use the thinking budget to reason carefully, then give a crisp answer "
                + "that reflects the reasoning you did.")
        .defaultOptions(thinkingOptions)
        .build());
```
<!--SNIPEND-->

Spring AI options are immutable. Pass their builder to `defaultOptions(...)`, or call `.mutate()` on an existing options object. The integration preserves provider-specific fields across the Activity boundary.

## Media in messages

Prefer URI-based media when attaching images, audio, or other binary content to chat messages. Raw `byte[]` media gets serialized into every chat Activity's input and result payload, which end up inside Temporal Event history events. Server-side history events have a fixed 2 MiB size limit; to leave headroom for messages, tool definitions, and options, the plugin enforces a **1 MiB default cap** on inline bytes and fails fast with a non-retryable `ApplicationFailure` pointing at the URI alternative.

```java
// Preferred — only the URL crosses the Activity boundary.
Media image = new Media(MimeTypeUtils.IMAGE_PNG, URI.create("https://cdn.example.com/pic.png"));
```

Override the cap by setting the system property `io.temporal.springai.maxMediaBytes` before your worker starts (positive integer; `0` disables the check). For anything larger than a small thumbnail, route the bytes to a binary store from an Activity and pass only the URL across the conversation.

## Use vector stores, embeddings, and MCP

When the corresponding Spring AI modules are on the classpath, the integration registers Activities for vector stores, embeddings, and MCP tool calls. Inject the matching Spring AI types into your Activities or Workflows and use them as you would in any Spring AI application — each operation is executed through a Temporal Activity.

You can also register these plugins explicitly, without relying on auto-configuration:

```java
new VectorStorePlugin(vectorStore);
new EmbeddingModelPlugin(embeddingModel);
new McpPlugin();
```

`ActivityMcpClient` wraps a Spring AI MCP client so that remote MCP tool calls become durable Activity executions.

## Resources

- [Temporal Spring AI README](https://github.com/temporalio/ai-integrations/tree/main/java/spring-ai) — full reference for the standalone module
- [Spring Boot integration](/develop/java/integrations/spring-boot-integration) — required companion module
- [Plugin system](/develop/plugins-guide) — how integrations are registered with Workers and Clients

---

# Spring Boot integration - Java SDK

Source: https://docs.temporal.io/develop/java/integrations/spring-boot-integration

> Learn about the Temporal Spring Boot integration

This guide introduces the [Temporal Spring Boot](https://central.sonatype.com/artifact/io.temporal/temporal-spring-boot-starter?smo=true) integration. The Temporal Spring Boot integration is the easiest way to get started using the Temporal Java SDK if you are a current [Spring](https://spring.io/) user.

This section includes the following topics:

- [Set up the dependency](#setup-dependency)
- [Connect to your Temporal Service](#connect)
- [Configure Workers](#configure-workers)
- [Customize Options](#customize-options)
- [Interceptors](#interceptors)
- [Integrations](#integrations)
- [Testing](#testing)

## Set up the dependency 

To start using the Temporal Spring Boot integration, you need to add [`io.temporal:temporal-spring-boot-starter`](https://search.maven.org/artifact/io.temporal/temporal-spring-boot-starter)
as a dependency to your Spring project:

> **📝 Note:**
> Temporal's Spring Boot integration currently supports Spring Boot 2.x, 3.x, and 4.x

**[Apache Maven](https://maven.apache.org/):**

```maven
<dependency>
    <groupId>io.temporal</groupId>
    <artifactId>temporal-spring-boot-starter</artifactId>
    <version>1.31.0</version>
</dependency>
```

**[Gradle Groovy DSL](https://gradle.org/):**

```groovy
implementation ("io.temporal:temporal-spring-boot-starter:1.31.0")
```

## Connect 

See the [Temporal Client documentation](/develop/java/client/temporal-client) for more information about connecting to a Temporal Service.

To create an autoconfigured `WorkflowClient`, you need to specify some connection details in your `application.yml` file, as described in the next section.

### Connect to your local Temporal Service

```yaml
spring.temporal:
  connection:
    target: local # you can specify a host:port here for a remote connection
```

This is enough to autowire a `WorkflowClient` in your Spring Boot application:

```java
@SpringBootApplication
class App {
  @Autowire
  private WorkflowClient workflowClient;
}
```

### Connect to a custom Namespace

You can also connect to a custom Namespace by specifying the `spring.temporal.namespace` property.

```yaml
spring.temporal:
  connection:
    target: local # you can specify a host:port here for a remote connection
  namespace: <custom namespace> # you can specify a custom namespace that you are using
```

## Connect to Temporal Cloud 

You can also connect to Temporal Cloud, using either an API key or mTLS for authentication.

See the [Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud) section for more information about connecting to Temporal Cloud.

### Using an API key

```yaml
spring.temporal:
  connection:
    target: <target>
    apiKey: <API key>
  namespace: <namespace>
```

### Using mTLS

```
spring.temporal:
  connection:
    mtls:
      target: <target>
      key-file: /path/to/key.key
      cert-chain-file: /path/to/cert.pem # If you use PKCS12 (.pkcs12, .pfx or .p12), you don't need to set it because the certificates chain is bundled into the key file
  namespace: <namespace>
```

## Configure Workers 

Temporal's Spring Boot integration supports two configuration methods for Workers: explicit configuration and auto-discovery.

### Explicit configuration

```yaml
spring.temporal:
  workers:
    - task-queue: your-task-queue-name
      name: your-worker-name # unique name of the Worker. If not specified, Task Queue is used as the Worker name.
      workflow-classes:
        - your.package.YourWorkflowImpl
      activity-beans:
        - activity-bean-name1
```

### Auto-discovery

Auto Discovery allows you to skip specifying Workflow classes, Activity beans, and Nexus Service beans explicitly in the config by referencing Worker Task Queue names or Worker Names on Workflow, Activity implementations, and Nexus Service implementations. Auto-discovery is applied after and on top of an explicit configuration.

```
spring.temporal:
  workers-auto-discovery:
    packages:
      - your.package # enumerate all the packages that contain your workflow implementations.
```

#### What is auto-discovered:

- Workflow implementation classes annotated with `io.temporal.spring.boot.WorkflowImpl`
- Activity beans present Spring context whose implementations are annotated with `io.temporal.spring.boot.ActivityImpl`
- Nexus Service beans present in Spring context whose implementations are annotated with `io.temporal.spring.boot.NexusServiceImpl`
- Workers if a Task Queue is referenced by the annotations but not explicitly configured. Default configuration will be used.

> **📝 Note:**
> `io.temporal.spring.boot.ActivityImpl` and `io.temporal.spring.boot.NexusServiceImpl` should be applied to beans, one way to do this is to annotate your Activity implementation class with `@Component`

```
@Component
@ActivityImpl(workers = "myWorker")
public class MyActivityImpl implements MyActivity {
  @Override
  public String execute(String input) {
    return input;
  }
}
```

> **📝 Note:**
> Auto-discovered Workflow implementation classes, Activity beans, and Nexus Service beans will be registered with the configured Workers if not already registered.

## Interceptors 

To enable Interceptors, you can create beans by implementing the `io.temporal.common.interceptors.WorkflowClientInterceptor`, `io.temporal.common.interceptors.ScheduleClientInterceptor`, or `io.temporal.common.interceptors.WorkerInterceptor` interface. Interceptors will be registered in the order specified by the `@Order` annotation.

## Integrations 

The Temporal Spring Boot integration also has built in support for various tools in the Spring ecosystem, such as metrics and tracing.

### Metrics

You can set up built-in Spring Boot metrics using [Spring Boot Actuator](https://docs.spring.io/spring-boot/reference/actuator/metrics.html). The Temporal Spring Boot integration will pick up the `MeterRegistry` bean and use it to report Temporal metrics.

Alternatively, you can define a custom `io.micrometer.core.instrument.MeterRegistry` bean in the application context.

### Tracing

You can set up [Spring Cloud Sleuth](https://spring.io/projects/spring-cloud-sleuth) with an OpenTelemetry export. The Temporal Spring Boot integration will pick up the OpenTelemetry bean configured by `spring-cloud-sleuth-otel-autoconfigure` and use it for Temporal traces.

Alternatively, you can define a custom `io.opentelemetry.api.OpenTelemetry` for OpenTelemetry or `io.opentracing.Tracer` for an OpenTracing bean in the application context.

## Customization of options 

To programmatically customize the various options that are created by the Spring Boot integration, you can create beans that implement the `io.temporal.spring.boot.TemporalOptionsCustomizer<OptionsBuilderType>` interface. This will be called after the options in your properties files are applied.

Where `OptionsType` may be one of:

    * `WorkflowServiceStubsOptions.Builder`
    * `WorkflowClientOptions.Builder`
    * `WorkerFactoryOptions.Builder`
    * `WorkerOptions.Builder`
    * `WorkflowImplementationOptions.Builder`
    * `TestEnvironmentOptions.Builder`

`io.temporal.spring.boot.WorkerOptionsCustomizer` may be used instead of `TemporalOptionsCustomizer<WorkerOptions.Builder>` if `WorkerOptions` needs to be customized on the Task Queue or Worker name.

`io.temporal.spring.boot.WorkflowImplementationOptionsCustomizer` may be used instead of `TemporalOptionsCustomizer<WorkflowImplementationOptions.Builder>` if `WorkflowImplementationOptions` needs to be customized on Workflow Type.

## Testing 

The Temporal Spring Boot integration also has easy support for testing your Temporal code. Add the following to your `application.yml` to reconfigure the client to work through `io.temporal.testing.TestWorkflowEnvironment` that uses in-memory Java Test Server:

```
spring.temporal:
  test-server:
    enabled: true
```

When `spring.temporal.test-server.enabled:true` is added, the `spring.temporal.connection` section is ignored. This allows wiring the `TestWorkflowEnvironment` bean in your unit tests:

```
@SpringBootTest(classes = Test.Configuration.class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class Test {
  @Autowired ConfigurableApplicationContext applicationContext;
  @Autowired TestWorkflowEnvironment testWorkflowEnvironment;
  @Autowired WorkflowClient workflowClient;

  @BeforeEach
  void setUp() {
    applicationContext.start();
  }

  @Test
  @Timeout(value = 10)
  public void test() {
    # ...
  }

  @ComponentScan # to discover Activity beans annotated with @Component
  public static class Configuration {}
}
```

See the [Java SDK test frameworks documentation](/develop/java/best-practices/testing-suite#test-frameworks) for more information about testing.

---

# Nexus - Java SDK

Source: https://docs.temporal.io/develop/java/nexus

> This section covers Nexus with the Java SDK

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Temporal Nexus

- [Quickstart](/develop/java/nexus/quickstart)
- [Feature guide](/develop/java/nexus/feature-guide)
- [Standalone Operations](/develop/java/nexus/standalone-operations)
- [Microservice Development Walkthrough](/develop/java/nexus/development-walkthrough)
- [Nexus sync tutorial](https://learn.temporal.io/tutorials/nexus/nexus-sync-tutorial/)

---

# Microservice Development Walkthrough with Nexus in Java - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough

> Build a Nexus Service end to end in Java, starting from a data contract and adding one Nexus capability at a time to solve a purchase approval problem.

> **⚠️ Currently in: Pre-release:**
> This walkthrough uses the new Nexus developer experience: pre-release APIs for the [Temporal Operation Handler](/nexus/temporal-operation-handler) and [Nexus Standalone Activity](/nexus/standalone-activity). These APIs are experimental and subject to change.

This walkthrough builds one Nexus Service from nothing to a complete API, adding one Nexus capability at each step.

A [Nexus Service](/evaluate/features/nexus) is a contract that one team publishes and other teams call across [Namespace](/namespaces) boundaries, without sharing code or a deployment.

## The walkthrough problem

A purchase request needs approval before it can proceed. One team owns that approval process, and
other teams across the company need to start an approval and find out how it went, without taking
on that team's code or running it themselves. That split is what makes this a Nexus problem: the
approval team publishes a Service, and everyone else calls it.

Approval is slow and human-driven, so the system has to survive a wait of minutes or weeks. While a request is pending, other systems might nudge the approver or attach information to the request. When a decision arrives, the requesting system needs the outcome.

The Service needs to:

- Start an approval and eventually return `APPROVED` or `DENIED`
- Accept a nudge that asks the approver again, and count how many have been sent
- Read an approval's progress without changing it
- Accept supporting information for a purchase, whether or not its approval exists yet
- Accept a decision and confirm it was recorded
- Send a notification when the decision is final

Each of those needs a different Nexus capability, introduced one step at a time.

## One contract, every language

**This walkthrough builds the Service in Java.** [NexGen](/nexus/nexgen) generates code from the same contract in Go, Python, and TypeScript, and the reasoning at each step is the same in all of them.

The Java sample is [nexuswalkthrough](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexuswalkthrough).

**Any caller can call any handler, because the contract is the only thing the two sides share.** A Go caller can drive a Python handler, and a TypeScript caller can drive the Java handler built here. [Step 6](/develop/java/nexus/development-walkthrough/call-the-service) builds the Java caller.

## How to follow along

Three kinds of code block appear in this walkthrough. Hover over a block to get a copy icon.

**A terminal window is a command to run.** Run this one now to confirm you have [Temporal CLI](/cli) 1.9.1 or later:

```bash title="Run: check your CLI version"
temporal --version
```

**Run every terminal command, in order**, or later steps may fail.

**A block headed by a file path is sample code** to read. You don't need to type it: it's already in the [sample codebase](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexuswalkthrough), and the file name links to the file.

[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflowId.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflowId.java)

```java
  public static String forItem(String itemId) {
    return "approval-" + itemId;
  }
```

Command output appears in a plain block with no header:

```
temporal version 1.9.1 (Server 1.32.0, UI 2.54.1)
```

## Before you start

### 1. Clone the sample project 

This walkthrough runs inside a clone of the [samples-java](https://github.com/temporalio/samples-java)
repository. Every command and file path is relative to the repository root.

```bash title="Run: clone the sample"
git clone https://github.com/temporalio/samples-java.git
cd samples-java
```

The sample is the finished Service. Each step shows the code it introduces and then runs it.

### 2. Check your CLI version 

This walkthrough needs [Temporal CLI](/cli) 1.9.1 or later and its development server. Each step
runs an Operation from the command line with `temporal nexus operation`, so check that your CLI has
those commands before you start. If `temporal nexus --help` doesn't list `operation`, upgrade.

The development server also needs two dynamic config values that are off by default.
[Step 4](/develop/java/nexus/development-walkthrough/implement-the-service#start-the-development-server)
sets them, and [Debugging and tips](/develop/java/nexus/development-walkthrough/debugging-and-tips)
shows what each one fails with when it's missing.

### 3. Run each Operation as you build it 

[Step 4](/develop/java/nexus/development-walkthrough/implement-the-service)
starts the development server, and [step 5](/develop/java/nexus/development-walkthrough/publish-in-nexus)
makes the Service reachable. From then on, each step that adds an Operation ends by running it, so
you see a result before moving on. Those runs use `temporal nexus operation`, which starts a
[Standalone Nexus Operation](/standalone-nexus-operation): the CLI is the caller, so you need no
caller Workflow or caller Worker until [step 6](/develop/java/nexus/development-walkthrough/call-the-service).

New to Nexus? Read [Nexus Services](/nexus/services) and [Nexus Operations](/nexus/operations), or work through the shorter [Java Nexus quickstart](/develop/java/nexus/quickstart).

## Start the walkthrough

Ten steps build the Service, and the sidebar lists them all. Work through them in order, because
each one runs against what the last one left running.

- [Step 1: Define the data contract](/develop/java/nexus/development-walkthrough/define-the-data-contract): The six Operations callers need, and the types they carry.

[Debugging and tips](/develop/java/nexus/development-walkthrough/debugging-and-tips) collects the
failures each step can produce, indexed by symptom. Keep it open in a tab.

---

# Add a Standalone Activity - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/add-a-standalone-activity

> An Activity-backed Operation sends one notification with retries and no Workflow, keyed on the Nexus request Id so a server retry cannot send it twice.

> **📝 Note:**
> Keep these running
>
> This step uses the processes started earlier. Leave them running:
>
> - The development server, from [step 4](/develop/java/nexus/development-walkthrough/implement-the-service#start-the-development-server). It must have been started with `activity.enableCallbacks=true`, or `notifyRequester` fails with `completion callbacks are not enabled for this namespace`.
> - The handler Worker, from [step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker).
> - The caller Worker, from [step 6](/develop/java/nexus/development-walkthrough/call-the-service#call-the-operations-from-a-caller-workflow).
>

The last Operation, `notifyRequester`, tells the requester whether their approval was `APPROVED` or `DENIED`. It is a single outbound notification with no state and nothing to wait for, so it is backed by a [Standalone Activity](/nexus/standalone-activity), as chosen in [step 3](/develop/java/nexus/development-walkthrough/choose-backing-implementation).

## Write the Activity

The Activity is ordinary and has nothing Nexus-specific in it; a Workflow could call the same Activity unchanged. In this walkthrough it is a placeholder that sends no email. Real logic would call an email provider, push to a notification service, or write to an outbox.

<!--SNIPSTART samples-java-nexus-walkthrough-activities-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalActivities.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalActivities.java)
```java
public interface ApprovalActivities {

  /** STEP 4 placeholder - real logic would apply policy, check limits, or call a risk service. */
  @ActivityMethod
  void evaluateAutoDecision(String itemId, double amount);

  /** STEP 4 placeholder - real logic would page an approver or open a ticket. */
  @ActivityMethod
  void notifyApproverOfPendingRequest(String itemId, String requester);

  /**
   * STEP 9 - The Standalone Activity behind the notifyRequester Operation. One outbound
   * notification, no state, nothing to wait for. In this sample it only logs; real logic would call
   * an email provider, push to a notification service, or write to an outbox.
   */
  @ActivityMethod
  NotifyRequesterOutput notifyRequester(String requester, NotifyRequesterInput.Decision decision);
}
```
<!--SNIPEND-->

## Back the Operation with it

Write it as a `@TemporalOperation` method, but start an Activity on the [Client](/nexus/temporal-operation-handler#the-nexus-aware-client) instead of a Workflow. The Operation starts an Activity Execution with no parent Workflow and completes when the Activity returns.

The Activity provides retries, timeouts, and a record of every attempt. The Operation provides the contract, so other teams can trigger the notification without sharing your code or your Namespace.

<!--SNIPSTART samples-java-nexus-walkthrough-notify-requester-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java)
```java
@TemporalOperation
public TemporalOperationResult<NotifyRequesterOutput> notifyRequester(
    TemporalOperationStartContext ctx, TemporalNexusClient client, NotifyRequesterInput input) {
  return client.startActivity(
      ApprovalActivities.class,
      ApprovalActivities::notifyRequester,
      input.getRequester(),
      input.getDecision(),
      StartActivityOptions.newBuilder()
          // Deriving the Activity Id from the request Id keeps a retried Nexus start
          // request targeting the same Activity Execution instead of sending a second
          // notification.
          .setId("notify-" + ctx.getRequestId())
          .setTaskQueue(HandlerWorker.DEFAULT_TASK_QUEUE_NAME)
          .setStartToCloseTimeout(Duration.ofSeconds(10))
          .setScheduleToCloseTimeout(Duration.ofMinutes(5))
          .setRetryOptions(RetryOptions.newBuilder().setMaximumAttempts(3).build())
          .build());
}
```
<!--SNIPEND-->

### Options an Activity-backed Operation requires

`StartActivityOptions` needs an **Activity Id**, unique within the Namespace, because there is no parent Workflow to scope it.

The Task Queue is optional and defaults to the one the Operation runs on. Set it to run notifications on their own Worker fleet.

**Derive the Activity Id from the Nexus request Id so that server retries don't send a second email.** Retried Nexus start requests carry the same request Id, so every retry lands on the same Activity Id and the notification goes out once. Use the same pattern for any externally visible side effect that can't be undone, such as charging a card, posting to a webhook, or creating a ticket.

To make several Operations share one Activity Execution and all receive its result, derive the Id from the Operation *input* instead. See [Nexus Standalone Activity](/nexus/standalone-activity#required-options).

## Register the Activity on the Worker

Register the Activity implementation on the Worker that hosts the Nexus Service. An Activity-backed Operation needs no Workflow registration.

<!--SNIPSTART samples-java-nexus-walkthrough-handler-worker-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/HandlerWorker.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/HandlerWorker.java)
```java
public static void main(String[] args) {
  WorkflowClient client = ClientOptions.getWorkflowClient(args);

  WorkerFactory factory = WorkerFactory.newInstance(client);
  Worker worker = factory.newWorker(DEFAULT_TASK_QUEUE_NAME);

  worker.registerWorkflowImplementationTypes(ApprovalWorkflowImpl.class);
  worker.registerActivitiesImplementations(new ApprovalActivitiesImpl());
  worker.registerNexusServiceImplementation(new ApprovalServiceImpl());

  factory.start();
}
```
<!--SNIPEND-->

## Cancellation needs heartbeating

This notification finishes immediately, so cancellation doesn't matter here. For a longer Activity-backed Operation it does: a cancellation request doesn't interrupt an Activity, so an Activity that never heartbeats runs until it completes or times out. Before shipping a long-running Activity-backed Operation, read [Activity cancellation](/activity-execution#cancellation).

## Run it

From the command line, this Operation looks like every other one:

```bash title="Run 1 of 2: notify the requester"
temporal nexus operation execute \
  --namespace approval-caller-namespace \
  --endpoint approval-endpoint \
  --service temporal.samples.approval.v1.ApprovalService \
  --operation NotifyRequester \
  --operation-id notify-1 \
  --input '{"requester":"tao@example.com","decision":"APPROVED"}'
```

```
Results:
  Status  COMPLETED
  Result  {"deliveredTo":"tao@example.com"}
```

On the handler side there is no Workflow, only an Activity Execution with no parent:

```bash title="Run 2 of 2: list the Standalone Activity"
temporal activity list --namespace approval-handler-namespace
```

```
  Status      ActivityId                                   Type             StartTime
  Completed   notify-0f816331-6b85-404d-9dfa-dd249041c1ea  NotifyRequester  now
```

The handler derived the Activity Id from the Nexus request Id. Running the command again with a new
`--operation-id` creates a second Activity Execution, but a server retry of the *same* request reuses
the first.

If this fails with `completion callbacks are not enabled for this namespace`, the development
server was started without `activity.enableCallbacks=true`. See
[step 4](/develop/java/nexus/development-walkthrough/implement-the-service#start-the-development-server).

> **💡 Tip:**
> RESOURCES
>
> - [Nexus Standalone Activity](/nexus/standalone-activity) for the full concept and options.
> - [Standalone Activity](/standalone-activity) for Activity Executions outside a Workflow.
> - [Java: Standalone Activities](/develop/java/activities/standalone-activities) for the SDK API.
>

## Next step

- [Step 10: Finish the flow](/develop/java/nexus/development-walkthrough/call-the-standalone-activity): The whole approval end to end.

---

# Add messaging to the Service - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/add-messaging

> A Signal nudges a pending approval, a Query reads its progress, and an Update records the decision, so what the caller needs back decides the message type.

> **📝 Note:**
> Keep these running
>
> This step uses the processes started earlier. Leave them running:
>
> - The development server, from [step 4](/develop/java/nexus/development-walkthrough/implement-the-service#start-the-development-server).
> - The handler Worker, from [step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker).
> - The caller Worker, from [step 6](/develop/java/nexus/development-walkthrough/call-the-service#call-the-operations-from-a-caller-workflow).
>
> The commands at the end of this step also act on the approval that
> [step 5](/develop/java/nexus/development-walkthrough/publish-in-nexus#run-the-operation) left open.
>

The approval blocks waiting for a decision. **Messages** let callers interact with it while it waits.

This step adds three Operations. What the caller needs back decides which [message](/sending-messages) type each one uses:

| Operation | Message type | Why this type |
| --- | --- | --- |
| `remindApprover` | Signal | Fire-and-forget. The caller needs the nudge to happen, not a response. |
| `getApprovalStatus` | Query | Returns the approval's progress without changing it. |
| `submitDecision` | Update | Changes state *and* returns confirmation that the decision was recorded. |

If the caller can proceed without hearing back, use a Signal. If the caller only needs to read state, use a Query. If the caller needs to know what the message did, use an Update.

## Add the handlers to the Workflow

Add a Signal handler that increments the reminder count, a Query handler that reports progress, and an Update handler that records the decision. Recording `APPROVED` or `DENIED` satisfies the condition the Workflow is blocked on, so the Workflow returns that decision as its result.

<!--SNIPSTART samples-java-nexus-walkthrough-approval-workflow-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflow.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflow.java)
```java
public interface ApprovalWorkflow {

  /**
   * The Update handler's name on the wire. The Update-backed Operation has to name the Update
   * explicitly when it starts one, so the name is declared once here and reused there rather than
   * being spelled as a literal in two places.
   */
  String SUBMIT_DECISION_UPDATE = "submitDecision";

  /**
   * STEP 4 - The Workflow method. Its return value is the result of the requestApproval Operation:
   * the Operation completes when this Workflow returns, and the caller receives this value through
   * the Nexus completion callback.
   *
   * <p>Because the Workflow's return value is delivered straight to the caller as the Operation
   * result, it has to be the Operation's declared output type.
   */
  @WorkflowMethod
  RequestApprovalOutput runApproval(RequestApprovalInput input);

  /**
   * STEP 7 - A Signal. Fire-and-forget: the caller gets no result back, which is why a Signal is
   * the right message type for a nudge, and why the contract declares no output for it.
   */
  @SignalMethod
  void remindApprover();

  /**
   * STEP 8 - A Signal that also carries supporting information. Reached through Signal-with-Start,
   * so it may be the message that creates this Workflow.
   */
  @SignalMethod
  void attachContext(String note);

  /**
   * STEP 7 - A Query. Reads in-flight progress without changing it. A Query handler must not block
   * and must not change Workflow state; it returns only what is already in memory.
   *
   * <p>It reports whether a decision is still pending, not what the decision was. The decision is
   * the result of requestApproval, delivered to the caller without anyone asking.
   */
  @QueryMethod
  GetApprovalStatusOutput getApprovalStatus();

  /**
   * STEP 7 - An Update. The caller needs a result back - confirmation that the decision was
   * recorded - which is what makes this an Update rather than a Signal.
   */
  @UpdateMethod(name = ApprovalWorkflow.SUBMIT_DECISION_UPDATE)
  SubmitDecisionOutput submitDecision(SubmitDecisionInput.Decision decision);

  /**
   * STEP 7 - The Update's validator. An Update can reject a request before it changes anything,
   * which a Signal cannot: a Signal has already been accepted by the time the handler runs.
   *
   * <p>Here it rejects a second decision for an approval that has already been decided. Without it
   * the later decision would silently overwrite the earlier one. A rejected Update does not appear
   * in Event History and does not run the handler.
   */
  @UpdateValidatorMethod(updateName = ApprovalWorkflow.SUBMIT_DECISION_UPDATE)
  void validateSubmitDecision(SubmitDecisionInput.Decision decision);
}
```
<!--SNIPEND-->

## Expose them as Nexus Operations

All three are `@TemporalOperation` methods. As described in [The Nexus-aware Client](/nexus/temporal-operation-handler#the-nexus-aware-client), a Signal and a Query are **sync messaging** and an Update is an **async backing**.

### Signal

Send the Signal through the Client and return a synchronous result. The Operation completes during the handler call.

> **⚠️ Caution:**
> The handler has under 10 seconds
>
> A synchronous handler must finish inside the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout), and the real budget is smaller because the clock starts on the caller's side and the request still routes through matching.
>
> One Signal fits easily. A handler that sends several messages or does slow work may not. Overrunning gives the caller a context deadline exceeded error, which it retries with exponential backoff until the schedule-to-close timeout expires.
>

<!--SNIPSTART samples-java-nexus-walkthrough-remind-approver-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java)
```java
@TemporalOperation
public TemporalOperationResult<Void> remindApprover(
    TemporalOperationStartContext ctx, TemporalNexusClient client, RemindApproverInput input) {
  client
      .getWorkflowClient()
      .newWorkflowStub(ApprovalWorkflow.class, ApprovalWorkflowId.forItem(input.getItemId()))
      .remindApprover();
  return TemporalOperationResult.sync(null);
}

```
<!--SNIPEND-->

### Query

Send the Query through the Client and return its answer as a synchronous result. Like the Signal, the Operation completes during the handler call. Unlike the Signal, it returns the approval's progress: whether a decision is pending, and how many reminders and notes it has.

A Worker answers the Query, so the Operation fails if no Worker is polling the approval's Task Queue.

<!--SNIPSTART samples-java-nexus-walkthrough-get-approval-status-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java)
```java
@TemporalOperation
public TemporalOperationResult<GetApprovalStatusOutput> getApprovalStatus(
    TemporalOperationStartContext ctx, TemporalNexusClient client, GetApprovalStatusInput input) {
  return TemporalOperationResult.sync(
      client
          .getWorkflowClient()
          .newWorkflowStub(ApprovalWorkflow.class, ApprovalWorkflowId.forItem(input.getItemId()))
          .getApprovalStatus());
}

```
<!--SNIPEND-->

### Update

Start the Update on the Client. The Operation completes when the Update completes, and the result arrives through the Nexus completion callback. If the Update is already complete when it returns, such as a retried request or one that failed validation, the result returns synchronously.

An Update-backed Operation targets a Workflow that already exists, so `submitDecision` fails for a purchase with no running approval. As an async backing, it is limited to one per Operation invocation, though the handler can combine it with sync messaging.

### Reject a bad Update before it changes anything

An Update can refuse a request; a Signal cannot, because by the time its handler runs, the Signal is already in history.

The approval's **validator**, the `@UpdateValidatorMethod` in the Workflow interface above, rejects a second decision for an approval that is already decided. Without it, the later decision would overwrite the first. A rejected Update never runs the handler or reaches Event History, and the caller sees a failed Operation. A validator takes the same arguments as the handler, returns nothing, and must not change Workflow state.

<!--SNIPSTART samples-java-nexus-walkthrough-submit-decision-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java)
```java
@TemporalOperation
public TemporalOperationResult<SubmitDecisionOutput> submitDecision(
    TemporalOperationStartContext ctx, TemporalNexusClient client, SubmitDecisionInput input)
    throws OperationException {
  return client.startWorkflowUpdate(
      ApprovalWorkflow.class,
      ApprovalWorkflowId.forItem(input.getItemId()),
      ApprovalWorkflow::submitDecision,
      input.getDecision(),
      UpdateOptions.<SubmitDecisionOutput>newBuilder()
          .setResultClass(SubmitDecisionOutput.class)
          // The Update to invoke has to be named explicitly; the method reference above
          // supplies the argument types but not the wire name.
          .setUpdateName(ApprovalWorkflow.SUBMIT_DECISION_UPDATE)
          // An Update-backed Operation must wait for the ACCEPTED stage. The Operation
          // completes later, when the Update completes, through the completion callback.
          // Any other stage is rejected with "nexus op workflow updates only support
          // WorkflowUpdateStageAccepted for async updates".
          .setWaitForStage(WorkflowUpdateStage.ACCEPTED)
          .build());
}

```
<!--SNIPEND-->

## Do not poll for the decision

Don't use a message to fetch the final decision, and don't call `getApprovalStatus` in a loop waiting for `pending` to turn false. The decision is the result of `requestApproval`, and it reaches the caller without anyone asking.

When the handler started the approval, Nexus attached a [completion callback](/glossary#nexus-async-completion-callback) to the Workflow. When the Workflow returns, the handler's Namespace delivers the callback to the caller, which records a `NexusOperationCompleted` event in the caller Workflow's history, and the caller Workflow resumes with the decision. See the [asynchronous Operation lifecycle](/nexus/operations#asynchronous-operation-lifecycle).

The Signal and the Update change a running approval, and the Query reports its progress. None of them returns its outcome. The Signal and the Update also stop working once the approval completes: the Temporal Service accepts a Signal or an Update only while the Workflow is running, and rejects one sent to a closed Workflow with `NOT_FOUND: workflow execution already completed`. That happens as soon as the decision lands, not when the [Retention Period](/temporal-service/temporal-server#retention-period) expires. The Query keeps answering after the approval completes, until the Retention Period expires.

If more than one system needs the outcome, [step 8](/develop/java/nexus/development-walkthrough/send-messages#when-the-approval-already-exists) shows how additional callers attach to a running approval and receive the same decision.

## Run the messaging Operations

The handler Worker from step 4 already serves these Operations. Nudge the approval left running at
the end of [step 5](/develop/java/nexus/development-walkthrough/publish-in-nexus#run-the-operation).
The Signal returns nothing:

```bash title="Run 1 of 4: nudge the approver"
temporal nexus operation execute \
  --namespace approval-caller-namespace \
  --endpoint approval-endpoint \
  --service temporal.samples.approval.v1.ApprovalService \
  --operation RemindApprover \
  --operation-id remind-1 \
  --input '{"itemId":"laptop-42"}'
```

```
Results:
  Status  COMPLETED
  Result  null
```

The Query reports that the approval is still pending, with the reminder just sent:

```bash title="Run 2 of 4: check progress"
temporal nexus operation execute \
  --namespace approval-caller-namespace \
  --endpoint approval-endpoint \
  --service temporal.samples.approval.v1.ApprovalService \
  --operation GetApprovalStatus \
  --operation-id status-1 \
  --input '{"itemId":"laptop-42"}'
```

```
Results:
  Status  COMPLETED
  Result  {"pending":true,"remindersSent":1,"notesAttached":0}
```

The Update returns confirmation, including the Workflow's reminder count:

```bash title="Run 3 of 4: submit the decision"
temporal nexus operation execute \
  --namespace approval-caller-namespace \
  --endpoint approval-endpoint \
  --service temporal.samples.approval.v1.ApprovalService \
  --operation SubmitDecision \
  --operation-id decide-1 \
  --input '{"itemId":"laptop-42","decision":"APPROVED"}'
```

```
Results:
  Status  COMPLETED
  Result  {"recorded":"APPROVED","remindersSent":1}
```

The decision unblocks the approval Workflow, which returns and completes the `requestApproval`
Operation from step 5. Collect its result:

```bash title="Run 4 of 4: collect the approval result"
temporal nexus operation result \
  --namespace approval-caller-namespace --operation-id approval-1
```

```
Results:
  Status  COMPLETED
  Result  {"decision":"APPROVED"}
```

Nothing polled for that decision: the Operation completed because the approval Workflow returned.

> **💡 Tip:**
> RESOURCES
>
> - [Workflow message passing](/encyclopedia/workflow-message-passing) for Signals, Queries, and Updates.
> - [Handling messages](/handling-messages) for handler constraints.
> - [Temporal Operation Handler](/nexus/temporal-operation-handler) for the sync messaging and async backing distinction.
>

## Next step

- [Step 8: Send messages](/develop/java/nexus/development-walkthrough/send-messages): Signal-with-Start, and several callers awaiting one approval.

---

# Call the Service - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/call-the-service

> A caller Workflow invokes Operations through the generated stub, awaits the decision durably, and sets the three timeouts that bound a Nexus Operation.

Call the approval Operations from a Workflow in the caller Namespace. The caller knows only the Endpoint name and the contract from [step 1](/develop/java/nexus/development-walkthrough/define-the-data-contract).

This caller is Java, like the handler, but it doesn't have to be. See [Call it from another language](#call-it-from-another-language).

## What the caller gets from the contract

The caller doesn't hand-write request types, response types, or Operation names. It uses the code generated in [step 2](/develop/java/nexus/development-walkthrough/generate-code):

- **A Service definition**, so an Operation name is a symbol rather than a string you can misspell.
- **Typed models** for every input and output.
- **Runtime validators** that reject a payload that violates the contract before it reaches the wire.

So the contract is enforced twice: a field the contract lacks fails at build time in a typed language, and a payload the contract forbids fails at the boundary rather than inside the handler's Workflow.

## Call the Operations from a caller Workflow

In Java, the generated Service interface works directly as a Nexus Service stub. Create it inside the caller Workflow with Operation options, then call its methods as if they were local. This Workflow runs the whole flow, including Operations that later steps add.

<!--SNIPSTART samples-java-nexus-walkthrough-caller-workflow-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/caller/ApprovalCallerWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/caller/ApprovalCallerWorkflowImpl.java)
```java
public class ApprovalCallerWorkflowImpl implements ApprovalCallerWorkflow {

  private static final Logger logger = Workflow.getLogger(ApprovalCallerWorkflowImpl.class);

  // STEP 6 - In Java the Service interface works directly as a Nexus Service stub. Because the stub
  // is that interface, every call below is type-checked against the contract at compile time.
  //
  // The schedule-to-close timeout bounds the whole Operation. A human approval measured in days
  // would need a timeout in days; this sample decides in seconds, so a short one is fine. The
  // default would not be right for a real approval.
  private final ApprovalService approvalService =
      Workflow.newNexusServiceStub(
          ApprovalService.class,
          NexusServiceOptions.newBuilder()
              .setOperationOptions(
                  NexusOperationOptions.newBuilder()
                      .setScheduleToCloseTimeout(Duration.ofMinutes(2))
                      .build())
              .build());

  @Override
  public String runApprovalFlow(String itemId, String requester, double amount, String note) {

    // -------------------------------------------------------------------------------------------
    // STEP 8 - Attach information before the approval exists.
    //
    // This is deliberately called BEFORE requestApproval, which is the harder ordering. Because
    // attachApprovalContext is Signal-with-Start, this call creates the approval Workflow and
    // delivers the note to it.
    // -------------------------------------------------------------------------------------------
    approvalService.attachApprovalContext(
        new AttachApprovalContextInput(itemId, requester, amount, note));
    logger.info("attachApprovalContext -> note attached, approval now exists");

    // -------------------------------------------------------------------------------------------
    // STEP 6 - Request the approval.
    //
    // The approval Workflow is already running thanks to the call above, so this start would fail
    // under the default conflict policy. The handler sets USE_EXISTING, so instead this attaches
    // the Operation's completion callback to the running Execution.
    //
    // startNexusOperation returns a handle rather than blocking, so this Workflow can keep working
    // while the approval is pending. The wait is durable: this caller can be evicted and its Worker
    // can restart, and the result still arrives.
    // -------------------------------------------------------------------------------------------
    NexusOperationHandle<RequestApprovalOutput> approvalHandle =
        Workflow.startNexusOperation(
            approvalService::requestApproval, new RequestApprovalInput(itemId, requester, amount));

    // Wait for the Operation to be started before messaging it. NexusOperationExecution carries the
    // Operation token for an asynchronous Operation.
    approvalHandle.getExecution().get();
    logger.info("requestApproval -> started and attached to the existing approval");

    // -------------------------------------------------------------------------------------------
    // STEP 8 - Nudge the pending approval. A Signal, so there is no result to collect.
    // -------------------------------------------------------------------------------------------
    approvalService.remindApprover(new RemindApproverInput(itemId));
    logger.info("remindApprover -> approver nudged");

    // -------------------------------------------------------------------------------------------
    // STEP 8 - Check progress. A Query, so the caller gets the approval's state back without
    // changing it. This is a one-off read, not a loop: the decision arrives as the result of
    // requestApproval below.
    // -------------------------------------------------------------------------------------------
    GetApprovalStatusOutput status =
        approvalService.getApprovalStatus(new GetApprovalStatusInput(itemId));
    logger.info(
        "getApprovalStatus -> pending={}, {} reminder(s), {} note(s)",
        status.getPending(),
        status.getRemindersSent(),
        status.getNotesAttached());

    // -------------------------------------------------------------------------------------------
    // STEP 8 - Submit the decision. An Update, so the caller gets confirmation back.
    //
    // In a real system this arrives from a human through a separate caller. The sample submits it
    // here so the flow completes without one.
    // -------------------------------------------------------------------------------------------
    SubmitDecisionOutput ack =
        approvalService.submitDecision(
            new SubmitDecisionInput(itemId, SubmitDecisionInput.Decision.APPROVED));
    logger.info(
        "submitDecision -> recorded={} after {} reminder(s)",
        ack.getRecorded().getValue(),
        ack.getRemindersSent());

    // -------------------------------------------------------------------------------------------
    // STEP 6 - Await the decision.
    //
    // The caller does not poll. The decision is the result of requestApproval, pushed here through
    // the Nexus completion callback the moment the approval Workflow returns. Asking the approval
    // for its status in a loop would be polling for something already on its way.
    // -------------------------------------------------------------------------------------------
    RequestApprovalOutput.Decision decision = approvalHandle.getResult().get().getDecision();
    logger.info("requestApproval -> decision {}", decision.getValue());

    // -------------------------------------------------------------------------------------------
    // STEP 10 - Call the Standalone Activity.
    //
    // From the caller this looks like any other Operation. It does not know that nothing but a
    // single Activity Execution sits behind it.
    // -------------------------------------------------------------------------------------------
    NotifyRequesterOutput notified =
        approvalService.notifyRequester(
            new NotifyRequesterInput(
                requester, NotifyRequesterInput.Decision.fromString(decision.getValue())));
    logger.info("notifyRequester -> delivered to {}", notified.getDeliveredTo());

    return decision.getValue();
  }
}
```
<!--SNIPEND-->

The Endpoint name is bound once, when the caller Worker registers the Workflow, so the Workflow
refers to the Service by its contract alone:

<!--SNIPSTART samples-java-nexus-walkthrough-caller-worker-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/caller/CallerWorker.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/caller/CallerWorker.java)
```java
public static void main(String[] args) {
  WorkflowClient client = ClientOptions.getWorkflowClient(args);

  WorkerFactory factory = WorkerFactory.newInstance(client);
  Worker worker = factory.newWorker(DEFAULT_TASK_QUEUE_NAME);

  worker.registerWorkflowImplementationTypes(
      WorkflowImplementationOptions.newBuilder()
          .setNexusServiceOptions(
              Collections.singletonMap(
                  SERVICE_NAME,
                  NexusServiceOptions.newBuilder().setEndpoint(DEFAULT_ENDPOINT_NAME).build()))
          .build(),
      ApprovalCallerWorkflowImpl.class);

  factory.start();
}
```
<!--SNIPEND-->

The caller doesn't know which Task Queue the handler's Worker polls, or that `requestApproval` is backed by a Workflow while `notifyRequester` is backed by a single Activity. The handler team can change a backing, move the handler to another Namespace, or rewrite it in another language, and this caller keeps working.

Run the caller Worker in another terminal, alongside the handler Worker from
[step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker):

```bash title="Run: start the caller Worker"
./gradlew -q :core:execute \
  -PmainClass=io.temporal.samples.nexuswalkthrough.caller.CallerWorker \
  --args="-target-host localhost:7233 -namespace approval-caller-namespace"
```

It polls the caller Namespace and waits until a caller Workflow starts, which happens in
[Finish](/develop/java/nexus/development-walkthrough/call-the-standalone-activity#run-the-whole-flow).

## Await the decision

`requestApproval` returns the approval Workflow's return value, `APPROVED` or `DENIED`, delivered through the Nexus completion callback when the Workflow finishes.

The caller doesn't poll. It awaits the Operation, and the wait is durable: the caller Workflow can be evicted and its Worker can restart, and the result still arrives.

### Set timeouts

A caller sets three timeouts on a Nexus Operation:

- **Schedule-to-close** bounds the whole Operation. Set it to how long an approval can legitimately take: a human approval measured in days needs a timeout in days.
- **Schedule-to-start** bounds how long the caller waits for the handler to pick the Operation up. Set it so a handler that is down fails fast, even though the approval itself may run for days.
- **Start-to-close** bounds an asynchronous Operation after it has started. Synchronous Operations such as `remindApprover` ignore it, because they complete as part of the start request.

See [Nexus Operations](/nexus/operations#timeouts) for the full timeout model.

## Call it from another language

Nothing about this caller is Java-specific except the language it is written in. Generate a caller from the same contract in Go, Python, or TypeScript, point it at the Endpoint from [step 5](/develop/java/nexus/development-walkthrough/publish-in-nexus), and it drives the Java handler built here with no changes on either side. The reverse also works: this Java caller can call a handler written in any of them.

To generate a caller from this contract, see [Usage](https://github.com/temporalio/nexgen#usage) in the `nexgen` README.

## Calling without a caller Workflow

A caller Workflow gives the call durability and lets you orchestrate around it. To run one Operation with nothing to orchestrate, a Client can start it directly as a [Standalone Nexus Operation](/standalone-nexus-operation), using the same contract, handler, and Endpoint. See [Java: Standalone Operations](/develop/java/nexus/standalone-operations).

> **💡 Tip:**
> RESOURCES
>
> - [Nexus Operations](/nexus/operations) for the Operation lifecycle and timeouts.
> - [Java Nexus feature guide](/develop/java/nexus/feature-guide) for the caller API.
> - [NexGen](/nexus/nexgen) for generating callers in other languages.
>

## Next step

- [Step 7: Add messaging](/develop/java/nexus/development-walkthrough/add-messaging): A Signal to nudge the approval, a Query to check it, an Update to decide it.

---

# Call the Standalone Activity - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/call-the-standalone-activity

> Running the approval end to end shows the caller crossing a Namespace boundary without learning a Workflow Id, a Task Queue, or what backs an Operation.

Call `notifyRequester` once the decision is final.

## The caller cannot tell the difference

The caller Workflow from [step 6](/develop/java/nexus/development-walkthrough/call-the-service#call-the-operations-from-a-caller-workflow) calls `notifyRequester` exactly like `requestApproval`: through the same generated stub and Endpoint, with the same type checking and error handling. Nothing reveals that one is backed by an Activity and the other by a Workflow. The handler team could replace the notification Activity with a Workflow that retries across providers and escalates on failure, and no caller would change.

## Complete the flow

With every step in place, the caller runs the whole approval:

1. Call `requestApproval` and await it. The Operation starts the approval Workflow in the handler Namespace, or attaches to one that another Operation already started.
2. While it is pending, other systems call `remindApprover` to nudge the approver and `attachApprovalContext` to add supporting information.
3. A caller checks progress once with `getApprovalStatus`. The Query reports that a decision is still pending, with the reminder and note counts so far.
4. Someone calls `submitDecision` with `APPROVED` or `DENIED`. The Update records it, confirms to that caller, and unblocks the approval Workflow.
5. The approval Workflow returns the decision, which resolves the `requestApproval` Operation for every attached caller.
6. The caller calls `notifyRequester` with the decision.

<!--SNIPSTART samples-java-nexus-walkthrough-caller-starter-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/caller/CallerStarter.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/caller/CallerStarter.java)
```java
public static void main(String[] args) {
  WorkflowClient client = ClientOptions.getWorkflowClient(args);

  WorkflowOptions options =
      WorkflowOptions.newBuilder().setTaskQueue(CallerWorker.DEFAULT_TASK_QUEUE_NAME).build();

  // Runs the whole flow: context attached first, approval requested, approver reminded, progress
  // checked, decision submitted, decision awaited, requester notified.
  ApprovalCallerWorkflow workflow = client.newWorkflowStub(ApprovalCallerWorkflow.class, options);
  String result =
      workflow.runApprovalFlow(
          "standing-desk-" + UUID.randomUUID(),
          "dana@example.com",
          1250.00,
          "Approved in the Q3 ergonomics budget");
  logger.info("Purchase result: {}", result);
}
```
<!--SNIPEND-->

## Run the whole flow

The handler Worker from
[step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker) and the caller
Worker from [step 6](/develop/java/nexus/development-walkthrough/call-the-service) should already be
running. From the repository root, start the Starter in another terminal:

```bash title="Run: run the whole flow"
./gradlew -q :core:execute \
  -PmainClass=io.temporal.samples.nexuswalkthrough.caller.CallerStarter \
  --args="-target-host localhost:7233 -namespace approval-caller-namespace"
```

```
INFO  i.t.s.n.caller.CallerStarter - Purchase result: APPROVED
```

The Starter exits when the run finishes; the two Workers keep running. The handler Worker shows the
purchase moving through every Operation:

```
INFO  ApprovalWorkflowImpl   - Context attached: Approved in the Q3 ergonomics budget
INFO  ApprovalActivitiesImpl - Evaluating auto-decision rules for standing-desk-... at 1250.0
INFO  ApprovalActivitiesImpl - Approver notified that standing-desk-... is waiting
INFO  ApprovalWorkflowImpl   - Approver reminded, 1 reminder(s) so far
INFO  ApprovalWorkflowImpl   - Approval for standing-desk-... decided APPROVED after 1 reminder(s) and 1 note(s)
INFO  ApprovalActivitiesImpl - Notifying dana@example.com that their request was APPROVED
```

The context is attached *before* the approval is requested, the Signal-with-Start ordering from
[step 8](/develop/java/nexus/development-walkthrough/send-messages#run-it-in-the-harder-order).
`requestApproval` then attached to the approval the note created.

Every call crossed a Namespace boundary, and the caller never learned a Workflow Id, a Task Queue, or what backed any Operation. One Operation started a Workflow, three sent messages to it, one started it if it was not already running, and one started an Activity. To the caller, they were all Operations.

## Trace it end to end

Open the caller Workflow in the UI and follow the links. Because the handlers used the injected [Client](/nexus/temporal-operation-handler#the-nexus-aware-client), each Operation links to the Execution it started or messaged in the handler Namespace, so you can move between the two Namespaces in one view.

## Where to go next

The Service is complete but minimal. Possible extensions:

- **Timeouts and escalation.** Give the approval a deadline and escalate or auto-deny when it passes.
- **Split the Workers.** Run the Nexus Service, the approval Workflow, and the notification Activity on separate Worker fleets. See [Nexus patterns](/nexus/patterns).
- **Callers in other languages.** Generate a caller from the same contract in Go, Python, or TypeScript. See [NexGen](/nexus/nexgen).
- **Standalone invocation.** Call an Operation from a Client with no caller Workflow. See [Standalone Nexus Operation](/standalone-nexus-operation).

Back to the [Microservice Development Walkthrough overview](/develop/java/nexus/development-walkthrough).

> **💡 Tip:**
> RESOURCES
>
> - [Nexus execution debugging](/nexus/execution-debugging) for tracing across Namespaces.
> - [Nexus patterns](/nexus/patterns) for Worker and Service topology.
> - [Nexus Standalone Activity](/nexus/standalone-activity) for Activity-backed Operations.
>

## Next step

- [Debugging and tips](/develop/java/nexus/development-walkthrough/debugging-and-tips): The failures each step can produce, indexed by symptom.

---

# Choose a backing implementation - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/choose-backing-implementation

> A Workflow, Standalone Activity, Update, or sync messaging can sit behind an Operation. Choose by whether the work waits, holds state, or returns a result.

The contract says nothing about what runs behind an Operation. That is the handler's private decision, and it can change later without touching callers.

Picking the wrong backing is the most common source of trouble later. This step chooses one for each Operation in the approval Service.

## Workflow

Use a Workflow for more than one step, any need to wait, or any need for durable intermediate state.

The Operation starts a Workflow and completes when that Workflow returns, so the Workflow's return value is the Operation's result. Use this when the work orchestrates several Activities, needs a timer, receives [messages](/sending-messages) while it runs, or must survive a Worker restart partway through.

A long-lived Workflow that accepts messages is still an ordinary Workflow. What makes it interactive is its message handlers and a Workflow Id you can predict.

## Standalone Activity

Use a Standalone Activity for one step with no waiting and no state: call an external API, run a computation, send a notification.

The Operation starts an Activity Execution with no parent Workflow and completes when the Activity returns. You get retries, timeouts, and a durable record of every attempt without a wrapper Workflow that exists only to call one Activity.

An Activity cannot receive messages or hold state, and cancellation only works if the Activity heartbeats. See [Nexus Standalone Activity](/nexus/standalone-activity).

## The choice for the approval Service

| Operation | Backing | Why |
| --- | --- | --- |
| `requestApproval` | Workflow | Blocks for a human decision, holds the reminder count, and accepts messages while pending |
| `remindApprover` | Sync messaging (Signal) | A Signal to the approval started by `requestApproval` |
| `getApprovalStatus` | Sync messaging (Query) | A Query to that same approval, which reads its progress without changing it |
| `submitDecision` | Update | An Update to that same approval, because the caller needs a result back |
| `attachApprovalContext` | Sync messaging (Signal-with-Start) | A Signal that also starts the approval if it does not exist yet |
| `notifyRequester` | Standalone Activity | One outbound notification, no state, nothing to wait for |

An Update is its own backing: the Operation completes when the Update completes, not when the Workflow returns. Signals and Queries are not a backing. They are [sync messaging](/nexus/temporal-operation-handler#the-nexus-aware-client), which takes effect and completes the Operation during the handler call.

The caller can observe the difference. A Signal-backed Operation completes synchronously, so a Temporal caller's Event History goes straight from `NexusOperationScheduled` to `NexusOperationCompleted`. An Update-backed Operation completes through the Nexus completion callback, which adds a `NexusOperationStarted` event carrying the Operation token used to cancel it.

The approval has to be a Workflow. It exists for a while, has an identity, and other systems interact with it during its lifetime. An Activity cannot block for a human or receive a Signal.

The notification is the opposite: a single side effect with nothing to orchestrate. A Workflow would add an Event History and a Workflow Id for no benefit, but the notification touches the outside world and can fail, so it needs the retries an Activity provides.

## Give the approval Workflow a stable Id

Derive the approval's Workflow Id from the purchase, not a random value, so that later messages can find it. A caller that knows the item id can then reach the right Execution without the handler handing out Workflow Ids.

This also makes the start idempotent: a retried Nexus start request targets the same Workflow Id rather than starting a second approval for one purchase.

Derive it in one place so the two Operations that need it cannot drift apart:

<!--SNIPSTART samples-java-nexus-walkthrough-workflow-id-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflowId.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflowId.java)
```java
public static String forItem(String itemId) {
  return "approval-" + itemId;
}
```
<!--SNIPEND-->

In [step 8](/develop/java/nexus/development-walkthrough/send-messages), `attachApprovalContext` may start the approval before `requestApproval` is called. Both Operations derive the same Workflow Id from the same item id, so they agree on which Execution they mean.

> **💡 Tip:**
> RESOURCES
>
> - [Nexus Standalone Activity](/nexus/standalone-activity) for Activity-backed Operations.
> - [Workflow message passing](/encyclopedia/workflow-message-passing) for what makes a Workflow interactive.
> - [Nexus patterns](/nexus/patterns) for Service and Worker topology choices.
>

## Next step

- [Step 4: Implement the Service](/develop/java/nexus/development-walkthrough/implement-the-service): The approval Workflow, the handler, the development server, and a Worker.

---

# Debug common Nexus problems - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/debugging-and-tips

> Hangs, unauthorized calls, missing UI links, and duplicate side effects in a Nexus Service, each traced from the symptom back to the wiring behind it.

Most Nexus problems are wiring problems with a few recognizable symptoms. Start from the symptom.

## Development server settings

Two errors mean the development server is missing a setting that is off by default. See
[Start the development server](/develop/java/nexus/development-walkthrough/implement-the-service#start-the-development-server).

**`Standalone Nexus operation is disabled`** comes from `temporal nexus operation execute`,
`start`, `describe`, and `result` when the server was started without
`nexusoperation.enableStandalone=true`. Operations called from a caller Workflow are unaffected, so
this only shows up on the command line.

**`completion callbacks are not enabled for this namespace`** fails only `notifyRequester`, when the
server was started without `activity.enableCallbacks=true`. The other five Operations are unaffected.

If `temporal nexus --help` doesn't list `operation`, your CLI is too old for this walkthrough.
Upgrade to 1.9.1 or later.

## The call hangs and nothing happens

Check these causes in order:

1. **No Worker is polling the target Task Queue.** Check that the handler Worker is running and shows as a poller on the Endpoint's target Task Queue.
2. **The Task Queue names don't match.** The Endpoint's target Task Queue and your Worker's Task Queue must be identical strings. A typo queues the request where nobody is listening.
3. **The Operation is waiting as designed.** A human approval with a multi-day schedule-to-close timeout is supposed to sit there. Check in the UI that the Operation is pending rather than stuck.

## The call fails as unauthorized

Add the caller Namespace to the Endpoint's allowed caller list. Creating an Endpoint does not authorize anyone to call it, and in Temporal Cloud the Namespace name includes an Account suffix that is easy to omit. See [Nexus security](/nexus/security).

## The caller and handler are not linked in the UI

Use the [Client](/nexus/temporal-operation-handler#the-nexus-aware-client) that a `@TemporalOperation` method receives for anything that starts or messages an Execution. A Client you construct yourself works, but loses the [bidirectional links](/nexus/execution-debugging#bi-directional-linking) that connect the two Executions.

## The Operation fails because the Workflow already exists

By default, starting a Workflow whose Id is already running is an error, so when two Operations derive the same Workflow Id, the second fails. This is intended: a Workflow-backed Operation has only started once its completion callback is attached, so failing beats reporting success to a caller that would wait forever.

To have the second caller join the running Execution, set the Workflow Id conflict policy to use-existing. See [When the approval already exists](/develop/java/nexus/development-walkthrough/send-messages#when-the-approval-already-exists).

## Pitfalls that are easy to miss

### Polling for a result that is already being delivered

The decision is the result of `requestApproval`, pushed to every caller awaiting the Operation when the Workflow completes. Asking the approval for its status in a loop polls for something already on its way, and stops working once its [Retention Period](/temporal-service/temporal-server#retention-period) expires.

Use the Operation result for outcomes, and messages to change a running approval.

### Expecting a late caller to collect a finished result

Only callers attached to an Operation receive its result. While the approval is running, additional callers can attach with the use-existing conflict policy. After it completes, they can't, so attach before it finishes or have the handler notify them, as `notifyRequester` does.

### An Activity-backed Operation that will not cancel

Cancellation doesn't interrupt an Activity. Without heartbeats and a heartbeat timeout, a cancellation request has no effect and the Operation runs to its timeout. Let the resulting cancellation exception propagate: a cancelled Activity is not retried, but one that swallows the cancellation and throws an ordinary failure is. See [Activity cancellation](/activity-execution#cancellation).

### Duplicate side effects on retry

The server retries Nexus start requests, so a backing Execution whose Id isn't derived from something stable gets started twice. Derive the Workflow Id or Activity Id from the Nexus request Id, or from the Operation input when several Operations should share one Execution. This matters most for external side effects, such as notifications.

### Sending a Signal to a Workflow that may not exist

A Signal to a missing Workflow fails. Use Signal-with-Start when the target may not be running yet, and include in its input whatever the Workflow needs to start. See [Attach information before the approval exists](/develop/java/nexus/development-walkthrough/send-messages#attach-information-before-the-approval-exists).

### More than one async backing per handler invocation

A handler can send any number of sync messages but start at most one async backing. Starting a Workflow and a Workflow Update in the same invocation is not valid. Pick one thing for the caller to await.

### Hand-editing generated code

Generated files are overwritten on the next run. When a generated name is wrong, fix it with a per-language naming override in the contract. See [Naming & overrides](https://github.com/temporalio/nexgen#naming--overrides) in the `nexgen` README.

### Letting the contract drift

Callers and handlers deploy independently, so different contract versions run at the same time. Adding an optional field is safe. Making a field required, removing one, or changing a type breaks whichever side deploys second.

## Tips

**Verify the wiring before writing a caller.** Confirm the Endpoint exists, targets the right Namespace and Task Queue, and has a Worker polling it.

**Set timeouts to match reality.** A human approval measured in days needs a schedule-to-close timeout in days.

**Let contract violations be `BAD_REQUEST`.** The generated validators report every violation in one error, so the caller can fix them all at once. See [Nexus error handling](/nexus/error-handling).

**Use `@TemporalOperation` for every Operation.** An Operation that starts synchronous can later gain an async backing without changing shape.

Back to the [Microservice Development Walkthrough overview](/develop/java/nexus/development-walkthrough).

> **💡 Tip:**
> RESOURCES
>
> - [Nexus execution debugging](/nexus/execution-debugging) for tracing Operations across Namespaces.
> - [Nexus error handling](/nexus/error-handling) for the error model and retry behavior.
> - [Nexus security](/nexus/security) for Endpoint authorization.
> - [Temporal Operation Handler](/nexus/temporal-operation-handler) for current per-SDK capability status.
>

---

# Define the data contract - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/define-the-data-contract

> A contract in JSON Schema 2020-12 declares the six approval Operations, the types they carry, and the rules that keep both sides deployable independently.

> **📝 Note:**
> Start from the Overview
>
> Clone the sample project and check your CLI version in the
> [Overview](/develop/java/nexus/development-walkthrough) before you start this step.
>

Start with the contract, not the code.

The contract is the only thing a caller and a handler share. Everything else is private to one side and can change without the other side knowing: each side's language, whether an Operation is backed by a Workflow or an Activity, and which Task Queue the Worker polls.

## Why the contract comes first

Writing the contract first is what makes the Service polyglot.

**Every sample in this walkthrough, in every language, is generated from this one contract.** A Go caller can call a Java handler, and a TypeScript caller can call a Python handler. The only thing that has to match is the contract both sides were generated from, so each side can use whatever language suits it.

## Plan the Operations

Work backwards from what callers need, not from what your Workflow happens to do.

Callers of the approval Service need to start an approval and learn its outcome, nudge a pending approval, check its progress, attach supporting information to a purchase, submit a decision, and be notified when the decision is final. That produces six Operations:

| Operation | Input | Output | Added in |
| --- | --- | --- | --- |
| `requestApproval` | Item id, requester, amount | `APPROVED` or `DENIED` | Step 4 |
| `remindApprover` | Item id | Nothing | Step 7 |
| `getApprovalStatus` | Item id | Whether a decision is pending, and how many reminders and notes it has so far | Step 7 |
| `submitDecision` | Item id, decision | The decision recorded, and how many reminders preceded it | Step 7 |
| `attachApprovalContext` | Item id, requester, amount, note | Nothing | Step 8 |
| `notifyRequester` | Requester, decision | Where the notification was delivered | Step 9 |

Two of these are easy to get wrong:

**`requestApproval` returns the final decision, not an approval id for the caller to poll.** The Operation is Workflow-backed, so it completes when that Workflow returns, and the Workflow's return value *is* the Operation's result. The caller awaits the Operation and receives `APPROVED` or `DENIED`.

**`attachApprovalContext` does not require the approval to exist.** Supporting information, such as a justification or a manager's note, comes from a different system than the approval request, and the two can arrive in either order. Because either one might have to start the approval Workflow, this Operation's input includes the purchase details the Workflow needs to start. [Step 8](/develop/java/nexus/development-walkthrough/send-messages#attach-information-before-the-approval-exists) covers this in detail.

## Contract design rules

An Operation's input and output are each optional, but when present each must be an **object type**, even if it wraps a single value. That lets you add a field later without breaking the wire format. Returning nothing is fine, as `remindApprover` does.

Keep the types **forward-compatible**. Callers and handlers deploy independently and run different versions of the contract at the same time. Adding an optional field is safe; making an existing field required, or removing one, is not.

## Write the contract

Contracts are modeled with JSON Schema 2020-12. Each definition file is one of two kinds, decided by what sits at its root:

- **Nexus document**: the root carries a `nexusrpc: '1.0.0'` marker, with Services and their Operations at the top level and types under `$defs`. Only this kind can declare a Service.
- **Pure JSON Schema**: the root is itself a type, with reusable types under `$defs`. It declares no Services or Operations, only data models shared across languages.

A file is one or the other. A contract can span several files, with a Nexus document pulling in types from pure-schema files through `$ref`.

The approval contract declares a Service, so its entry file is a Nexus document with two sections. `services` declares the Service and its Operations, each naming its input and output by reference. `$defs` declares the types those references point at, in ordinary JSON Schema with nothing Nexus-specific, which is what lets the same types generate into four languages.

> **📝 Note:**
> How to write a contract of your own
>
> To write a contract yourself, see
> [Definition files](https://github.com/temporalio/nexgen#definition-files) in the `nexgen` README for
> the complete rules the generator enforces.
>

Here is the approval contract in full. In [step 2](/develop/java/nexus/development-walkthrough/generate-code) you pass this file to `nexgen` to generate the Java code:

<!--SNIPSTART samples-java-nexus-walkthrough-contract-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/approval.nexusrpc.yaml](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/approval.nexusrpc.yaml)
```yaml
nexusrpc: "1.0.0"
$schema: https://json-schema.org/draft/2020-12/schema
description: Purchase approval service built by the Nexus Microservice Development Walkthrough.

services:
  ApprovalService:
    fqn: temporal.samples.approval.v1.ApprovalService
    description: Start a purchase approval, message it while it is pending, and learn the outcome.
    operations:

      # Backed by a Workflow. The Workflow's return value is this Operation's result - see step 4.
      requestApproval:
        description: Start an approval and return the decision once it is made.
        input: { $ref: "#/$defs/RequestApprovalInput" }
        output: { $ref: "#/$defs/RequestApprovalOutput" }

      # A Signal. Fire-and-forget, so it declares no output - see step 7.
      remindApprover:
        description: Ask the approver again. Returns nothing.
        input: { $ref: "#/$defs/RemindApproverInput" }

      # A Query. Reads in-flight progress without changing it. It does not return the decision;
      # requestApproval does - see step 7.
      getApprovalStatus:
        description: Report the progress of an approval without changing it.
        input: { $ref: "#/$defs/GetApprovalStatusInput" }
        output: { $ref: "#/$defs/GetApprovalStatusOutput" }

      # An Update. The caller needs confirmation back, which is what makes this an Update rather
      # than a Signal - see step 7.
      submitDecision:
        description: Supply the decision and confirm it was recorded.
        input: { $ref: "#/$defs/SubmitDecisionInput" }
        output: { $ref: "#/$defs/SubmitDecisionOutput" }

      # Signal-with-Start. Its input repeats the purchase details because it may have to create the
      # approval it is messaging - see step 8.
      attachApprovalContext:
        description: Attach supporting information to a purchase, whether or not its approval exists yet.
        input: { $ref: "#/$defs/AttachApprovalContextInput" }

      # Backed by a Standalone Activity - one durable step, no Workflow - see step 9.
      notifyRequester:
        description: Notify the requester once the decision is final.
        input: { $ref: "#/$defs/NotifyRequesterInput" }
        output: { $ref: "#/$defs/NotifyRequesterOutput" }

# The APPROVED | DENIED value set is declared inline on each property that carries it, rather than
# once under $defs. A named enum under $defs is rejected by the generator today, so each Operation
# gets its own nested value class; handler/Decisions.java converts between them.
$defs:

  RequestApprovalInput:
    type: object
    additionalProperties: false
    properties:
      itemId: { type: string }
      requester: { type: string }
      amount: { type: number }
    required: [itemId, requester, amount]

  RequestApprovalOutput:
    type: object
    additionalProperties: false
    properties:
      decision:
        description: The outcome of the approval.
        type: string
        enum: [APPROVED, DENIED]
    required: [decision]

  RemindApproverInput:
    type: object
    additionalProperties: false
    properties:
      itemId: { type: string }
    required: [itemId]

  GetApprovalStatusInput:
    type: object
    additionalProperties: false
    properties:
      itemId: { type: string }
    required: [itemId]

  GetApprovalStatusOutput:
    type: object
    additionalProperties: false
    properties:
      pending: { description: True until a decision is recorded., type: boolean }
      remindersSent: { description: How many reminders have been sent so far., type: integer }
      notesAttached: { description: How many notes have been attached so far., type: integer }
    required: [pending, remindersSent, notesAttached]

  SubmitDecisionInput:
    type: object
    additionalProperties: false
    properties:
      itemId: { type: string }
      decision:
        description: The decision being submitted.
        type: string
        enum: [APPROVED, DENIED]
    required: [itemId, decision]

  SubmitDecisionOutput:
    type: object
    additionalProperties: false
    properties:
      recorded:
        description: The decision that was recorded.
        type: string
        enum: [APPROVED, DENIED]
      remindersSent: { description: How many reminders were sent before the decision., type: integer }
    required: [recorded, remindersSent]

  AttachApprovalContextInput:
    type: object
    additionalProperties: false
    properties:
      itemId: { type: string }
      requester: { type: string }
      amount: { type: number }
      note: { description: The supporting information to attach., type: string }
    required: [itemId, requester, amount, note]

  NotifyRequesterInput:
    type: object
    additionalProperties: false
    properties:
      requester: { type: string }
      decision:
        description: The final decision.
        type: string
        enum: [APPROVED, DENIED]
    required: [requester, decision]

  NotifyRequesterOutput:
    type: object
    additionalProperties: false
    properties:
      deliveredTo: { description: Where the notification was sent., type: string }
    required: [deliveredTo]
```
<!--SNIPEND-->

> **💡 Tip:**
> RESOURCES
>
> - [NexGen](/nexus/nexgen) for the contract format, and [Supported JSON Schema features](https://github.com/temporalio/nexgen#supported-json-schema-features) for the supported subset.
> - [Nexus Services](/nexus/services) for what a Service contract is and how it is shared.
>

## Next step

- [Step 2: Generate code](/develop/java/nexus/development-walkthrough/generate-code): Typed models, validators, and a Service definition from that one contract.

---

# Generate code from the contract - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/generate-code

> NexGen emits typed models, runtime validators, and a Nexus Service definition for each language, so handler and caller agree without hand-written types.

Generate code from the contract before writing any implementation. Both sides use it: the handler implements the generated Service definition, and the caller invokes Operations through it.

## What generation produces

For each type in the contract, [NexGen](/nexus/nexgen) emits a typed model and a runtime validator. For each Service, it also emits a Nexus Service definition with one member per Operation, in the form that is idiomatic for the language.

- The **handler** implements the Service definition, and the Worker registers that implementation.
- The **caller** invokes Operations through it, so calls are type-checked against the contract.

The validators run when a payload is parsed and again when it is serialized, so a request that violates the contract is rejected at the boundary instead of reaching your Workflow. Violations aggregate into one error naming every field that failed, which a handler maps to `BAD_REQUEST`.

## Generate the code

Generation is one command per language. See **[Usage](https://github.com/temporalio/nexgen#usage)** in the `nexgen` README for each language's flags.

> **📝 Note:**
> Installing nexgen
>
> Every `nexgen` release is published as a pre-release, so GitHub's "latest release" URL returns 404.
> Pick a version from the [releases list](https://github.com/temporalio/nexgen/releases) instead, and
> check that it has binaries attached before you download it. Some releases have none.
>

Run this command from the root of the `samples-java` clone you made in
[Before you start](/develop/java/nexus/development-walkthrough#clone-the-sample-project).
It reads
[approval.nexusrpc.yaml](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/approval.nexusrpc.yaml),
the contract from [step 1](/develop/java/nexus/development-walkthrough/define-the-data-contract#write-the-contract).

```bash title="Run: generate the Java code"
nexgen java \
  --output core/src/main/java/io/temporal/samples/nexuswalkthrough/generatedservice \
  --package-name io.temporal.samples.nexuswalkthrough.generatedservice \
  core/src/main/java/io/temporal/samples/nexuswalkthrough/approval.nexusrpc.yaml
```

**Java requires the package name's last segment to match the output directory's name**, which is
why both end in `generatedservice`. The rest of each path is this repository's layout. In your own
project, point `--output` and `--package-name` wherever your source tree keeps generated code.

Generation **overwrites the files it emits but doesn't remove anything else**, so a type you delete
from the contract leaves its generated class behind, still compiling and still importable. Delete
those by hand after a type goes away, and keep the contract outside the output directory so it
stays easy to tell the two apart.

Commit the generated code and regenerate whenever the contract changes. Don't edit the generated files, because the next run overwrites them. When a generated name is wrong for your language, fix it in the contract with a per-language naming override. See **[Naming & overrides](https://github.com/temporalio/nexgen#naming--overrides)** in the `nexgen` README.

## One contract, four languages

Run the generator once per language to get that language's typed models, validators, and Service definition. You still write the handler and the caller yourself, but the part both sides must agree on comes from one source. A handler doesn't need to know which languages its callers use, and a caller doesn't need to know which language implements the handler.

That is what lets the cross-language call in [step 6](/develop/java/nexus/development-walkthrough/call-the-service) work with no coordination beyond the contract.

> **💡 Tip:**
> RESOURCES
>
> - [NexGen](/nexus/nexgen) for what the generator produces.
> - The [`nexgen` README](https://github.com/temporalio/nexgen) for installation, per-language commands, and the supported JSON Schema subset.
> - [Data validation](/nexus/nexgen#data-validation) for how the generated validators behave.
>

## Next step

- [Step 3: Choose a backing](/develop/java/nexus/development-walkthrough/choose-backing-implementation): Workflow, Standalone Activity, Update, or sync messaging, one per Operation.

---

# Implement the Service - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/implement-the-service

> A @TemporalOperation method backs requestApproval with a Workflow that blocks for a human decision, hosted on one Worker with the Activities it runs.

This step implements the Operation at the center of the Service: back `requestApproval` with the approval Workflow, then start the development server and a Worker that hosts the Service, the Workflow, and the Activities.

## Write the approval Workflow

The approval is an ordinary Temporal Workflow, as chosen in [step 3](/develop/java/nexus/development-walkthrough/choose-backing-implementation#workflow). Nothing in it is Nexus-specific, and a Client could start it directly. The message handlers added in [step 7](/develop/java/nexus/development-walkthrough/add-messaging) make it interactive.

The Workflow:

1. Runs an Activity that evaluates whether the request can be auto-decided. In this walkthrough it is a placeholder; real logic would apply policy, check limits, or call a risk service.
2. Runs an Activity that tells a human the request is waiting. Also a placeholder.
3. Blocks until a decision arrives.
4. Returns `APPROVED` or `DENIED`.

The blocking step is why this is a Workflow. It may wait weeks, across Worker restarts and deployments, and costs nothing while idle.

<!--SNIPSTART samples-java-nexus-walkthrough-approval-workflow-impl-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalWorkflowImpl.java)
```java
public class ApprovalWorkflowImpl implements ApprovalWorkflow {

  private static final Logger logger = Workflow.getLogger(ApprovalWorkflowImpl.class);

  private final ApprovalActivities activities =
      Workflow.newActivityStub(
          ApprovalActivities.class,
          ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofSeconds(10)).build());

  // Durable intermediate state. This is the other reason the approval is a Workflow: an Activity
  // could not hold any of it.
  private SubmitDecisionInput.Decision decision;
  private int remindersSent;
  private final List<String> notes = new ArrayList<>();

  @Override
  public RequestApprovalOutput runApproval(RequestApprovalInput input) {
    // Placeholder. Real logic would apply policy, check limits, or call a risk service.
    activities.evaluateAutoDecision(input.getItemId(), input.getAmount());

    // Placeholder. Real logic would page an approver, open a ticket, or send email.
    activities.notifyApproverOfPendingRequest(input.getItemId(), input.getRequester());

    // Block until submitDecision supplies a decision. This wait is durable and unbounded - the
    // Worker can restart and redeploy while it is pending.
    Workflow.await(() -> decision != null);

    logger.info(
        "Approval for {} decided {} after {} reminder(s) and {} note(s)",
        input.getItemId(),
        decision.getValue(),
        remindersSent,
        notes.size());

    // This return value becomes the result of the requestApproval Nexus Operation, delivered to
    // every caller whose completion callback is attached to this Execution.
    return new RequestApprovalOutput(Decisions.toRequestApprovalOutput(decision));
  }

  // STEP 7 - Signal handler. Records the nudge and returns nothing.
  @Override
  public void remindApprover() {
    remindersSent++;
    logger.info("Approver reminded, {} reminder(s) so far", remindersSent);
  }

  // STEP 8 - Signal handler reached through Signal-with-Start. When the note arrives before anyone
  // has called requestApproval, the Signal-with-Start creates this Workflow and this handler runs
  // on the fresh Execution.
  @Override
  public void attachContext(String note) {
    notes.add(note);
    logger.info("Context attached: {}", note);
  }

  // STEP 7 - Query handler. Returns what is already in memory and changes nothing. Once the
  // decision is recorded, pending is false, but the decision itself is not here:
  // it is requestApproval's result.
  @Override
  public GetApprovalStatusOutput getApprovalStatus() {
    return new GetApprovalStatusOutput(decision == null, remindersSent, notes.size());
  }

  // STEP 7 - The Update's validator. Runs before the handler and can reject the request without
  // changing anything or writing to Event History. Throwing here rejects the Update; the Workflow
  // is untouched and the caller's Operation fails.
  @Override
  public void validateSubmitDecision(SubmitDecisionInput.Decision decision) {
    if (this.decision != null) {
      throw new IllegalStateException(
          "approval already decided " + this.decision.getValue() + ", cannot decide again");
    }
  }

  // STEP 7 - Update handler. Records the decision, which satisfies the condition the Workflow
  // method is blocked on, and returns confirmation to the caller. The validator above guarantees
  // this runs at most once.
  @Override
  public SubmitDecisionOutput submitDecision(SubmitDecisionInput.Decision decision) {
    this.decision = decision;
    return new SubmitDecisionOutput(Decisions.toSubmitDecisionOutput(decision), remindersSent);
  }
}
```
<!--SNIPEND-->

## Implement the Operation

Mark every Temporal-backed Operation with `@TemporalOperation`. The annotated method is the Operation's start handler, and the SDK wraps it in the [Temporal Operation Handler](/nexus/temporal-operation-handler) when the Worker registers the Service. Using it for every Operation lets an Operation later gain a Signal or change its backing without changing shape.

The method takes a context, a Nexus-aware Client, and the Operation input, and returns a `TemporalOperationResult`. Call `startWorkflow` on that Client and return its result. The Operation completes when the Workflow returns, delivering the Workflow's return value to the caller.

Use the injected Client for anything that starts or messages an Execution. It propagates bidirectional links and request Ids, so the caller's Execution and the approval Workflow are connected in the UI with no extra code. A Client you create yourself works, but loses that linking.

<!--SNIPSTART samples-java-nexus-walkthrough-request-approval-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java)
```java
@TemporalOperation
public TemporalOperationResult<RequestApprovalOutput> requestApproval(
    TemporalOperationStartContext ctx, TemporalNexusClient client, RequestApprovalInput input) {
  return client.startWorkflow(
      ApprovalWorkflow.class,
      ApprovalWorkflow::runApproval,
      input,
      WorkflowOptions.newBuilder()
          .setWorkflowId(ApprovalWorkflowId.forItem(input.getItemId()))
          .setWorkflowIdConflictPolicy(
              WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING)
          .build());
}

```
<!--SNIPEND-->

Set the Workflow Id from the item id, as decided in [step 3](/develop/java/nexus/development-walkthrough/choose-backing-implementation#give-the-approval-workflow-a-stable-id).

By default, starting a Workflow whose Id is already running **fails the Operation**. The Operation has only started once its completion callback is attached to a Workflow, so failing loudly beats reporting success to a caller that would then wait forever. The code above already sets `USE_EXISTING` instead, because once another Operation can create the approval first, this one must attach to it. [Step 8](/develop/java/nexus/development-walkthrough/send-messages#when-the-approval-already-exists) explains why.

`@TemporalOperation` gives the Operation the default cancel behavior. To change how a cancellation reaches the backing Execution, use `@OperationImpl` with a `TemporalOperationHandler` subclass instead.

## Start the development server

The Worker needs a running Temporal Service and a Namespace to poll. Start the development server
with two dynamic config values that later steps rely on:

```bash title="Run 1 of 3: start the development server"
temporal server start-dev \
  --dynamic-config-value 'nexusoperation.enableStandalone=true' \
  --dynamic-config-value 'activity.enableCallbacks=true'
```

`nexusoperation.enableStandalone` lets a Client start an Operation without a caller Workflow,
which is how you run each Operation from the command line as you build it.
`activity.enableCallbacks` allows a completion callback on a Standalone Activity Execution, which
the `notifyRequester` Operation in [step 9](/develop/java/nexus/development-walkthrough/add-a-standalone-activity)
needs.

In a second terminal, create the Namespace the handler runs in:

```bash title="Run 2 of 3: create the handler Namespace"
temporal operator namespace create --namespace approval-handler-namespace
```

## Run the Worker

One Worker hosts the Nexus Service, Workflow, and Activity implementations. It polls `approval-handler-task-queue`, which the Nexus Endpoint targets in the next step.

<!--SNIPSTART samples-java-nexus-walkthrough-handler-worker-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/HandlerWorker.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/HandlerWorker.java)
```java
public static void main(String[] args) {
  WorkflowClient client = ClientOptions.getWorkflowClient(args);

  WorkerFactory factory = WorkerFactory.newInstance(client);
  Worker worker = factory.newWorker(DEFAULT_TASK_QUEUE_NAME);

  worker.registerWorkflowImplementationTypes(ApprovalWorkflowImpl.class);
  worker.registerActivitiesImplementations(new ApprovalActivitiesImpl());
  worker.registerNexusServiceImplementation(new ApprovalServiceImpl());

  factory.start();
}
```
<!--SNIPEND-->

The Worker that registers a Nexus Service doesn't have to run the backing Workflow. Larger deployments often split them; see [Nexus patterns](/nexus/patterns).

In that second terminal, start it from the repository root and leave it running:

```bash title="Run 3 of 3: start the handler Worker"
./gradlew -q :core:execute \
  -PmainClass=io.temporal.samples.nexuswalkthrough.handler.HandlerWorker \
  --args="-target-host localhost:7233 -namespace approval-handler-namespace"
```

Qualify the task as `:core:execute`. An unqualified `execute` also runs the `lambda-worker:starter`
task, which ignores `-PmainClass` and starts an unrelated sample. The Worker runs until you stop it,
so Gradle keeps reporting the task as executing.

The Worker is polling, but no caller can reach the Service yet.
[Step 5](/develop/java/nexus/development-walkthrough/publish-in-nexus) publishes it and runs
`requestApproval` for the first time.

## Handle failures

Callers can tell two kinds of failure apart:

- A **contract violation**, a payload the generated validator rejects, surfaces as `BAD_REQUEST`. Retrying won't help, and the single error lists every violation.
- An **application failure**, where the approval cannot proceed for a business reason, fails the Operation. Whether it retries depends on the error type you raise. See [Nexus error handling](/nexus/error-handling).

> **💡 Tip:**
> RESOURCES
>
> - [Temporal Operation Handler](/nexus/temporal-operation-handler) for the handler type and the Nexus-aware Client.
> - [Java Nexus feature guide](/develop/java/nexus/feature-guide) for the full handler and Worker API.
> - [Nexus error handling](/nexus/error-handling) for mapping failures to Nexus errors.
>

## Next step

- [Step 5: Publish in Nexus](/develop/java/nexus/development-walkthrough/publish-in-nexus): An Endpoint, and the first run of requestApproval.

---

# Publish the Service in Nexus - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/publish-in-nexus

> A Nexus Endpoint routes Operation requests to the handler Namespace and Task Queue, so callers address it by name and never see what sits behind it.

A Worker is now polling, but callers reach a Service only through a [Nexus Endpoint](/nexus/endpoints), stored in the [Nexus Registry](/nexus/registry).

An Endpoint routes Operation requests to a target Namespace and Task Queue. Callers address the Endpoint by name and never see the Namespace or Task Queue behind it, so you can move the handler later without changing caller code.

## Create the Endpoint

An Endpoint needs a unique name, the target Namespace where the handler runs, and the target Task Queue the handler's Worker polls. The Task Queue must match the one your Worker uses from [step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker), or requests arrive and nothing picks them up.

The caller runs in its own Namespace, so the walkthrough crosses a real Namespace boundary.

The development server and the handler Worker from
[step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker) each hold
a terminal open, so open a third terminal for this step and for the commands in the steps that
follow.

On the development server, create the caller Namespace and the Endpoint with the CLI:

```bash title="Run 1 of 5: create the caller Namespace and the Endpoint"
temporal operator namespace create --namespace approval-caller-namespace

temporal operator nexus endpoint create \
  --name approval-endpoint \
  --target-namespace approval-handler-namespace \
  --target-task-queue approval-handler-task-queue \
  --description-file ./core/src/main/java/io/temporal/samples/nexuswalkthrough/description.md
```

`--description-file` is optional. It attaches Markdown that anyone browsing the Registry sees, so
use it to say what the Service is for and who owns it.

In Temporal Cloud, create the Endpoint in the UI under Nexus, or with `tcld`. See [Create a Nexus Endpoint](/nexus/registry#create-a-nexus-endpoint).

Endpoint names are unique within the Registry. In Temporal Cloud the Registry spans every Namespace in your Account; in a self-hosted deployment it is scoped to the Temporal Service.

## Allow caller Namespaces

> **📝 Note:**
> This section applies to Temporal Cloud only
>
> On a development server or a self-hosted Temporal Service, skip to [Verify it is reachable](#verify-it-is-reachable). There is no allowed-caller list to configure.
>

A Temporal Cloud Endpoint **rejects callers that are not on its allowed list**. Add the caller Namespace, including its Account suffix, when you create or edit the Endpoint in the UI or with `tcld`.

This step is easy to miss because the failure looks like a routing problem. If a call fails as unauthorized and the Endpoint exists, check this list first.

## Set up credentials

A development server needs no credentials. Both Namespaces are local and unauthenticated.

For Temporal Cloud, the caller and handler connect as separate clients, each to its own Namespace, using an API key with access to both or mTLS certificates. With [environment configuration](/develop/environment-configuration), keep one profile per Namespace and select it with an environment variable instead of passing connection options in code.

## Verify it is reachable

Before calling the Service, confirm the wiring. Come back to these checks whenever a later call
stops working.

Check that the Endpoint exists and targets the right Namespace and Task Queue:

```bash title="Run 2 of 5: check the Endpoint exists"
temporal operator nexus endpoint get --name approval-endpoint
```

Then check that the handler Worker is polling that Task Queue:

```bash title="Run 3 of 5: check the Worker is polling"
temporal task-queue describe \
  --namespace approval-handler-namespace \
  --task-queue approval-handler-task-queue
```

A Worker that isn't polling is the other common cause of a call that appears to hang: the request is accepted and queued, and nothing serves it.

The `Pollers` section lists a `workflow` and an `activity` poller, and no Nexus entry, so this
confirms a Worker is alive on the Task Queue rather than that it registered the Nexus Service. If
pollers are listed and calls still fail, check that the Worker calls
`registerNexusServiceImplementation`, as in
[step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker).

## Run the Operation

The Service is now reachable through the Endpoint.
`temporal nexus operation start` starts a [Standalone Nexus Operation](/standalone-nexus-operation),
which makes the CLI the caller, so you don't need a caller Workflow or caller Worker yet.

`requestApproval` blocks until someone decides the approval, so use `start`, which returns as soon as
the Operation has started, rather than `execute`, which waits for the result.
The Operation name is `RequestApproval`, not the `requestApproval` key in the contract: the generator
emits the wire name in Pascal case, and the Endpoint routes on that name.

```bash title="Run 4 of 5: request an approval"
temporal nexus operation start \
  --namespace approval-caller-namespace \
  --endpoint approval-endpoint \
  --service temporal.samples.approval.v1.ApprovalService \
  --operation RequestApproval \
  --operation-id approval-1 \
  --input '{"itemId":"laptop-42","requester":"tao","amount":2500}'
```

`describe` shows the Operation still open, waiting durably on the Workflow:

```bash title="Run 5 of 5: describe the pending Operation"
temporal nexus operation describe \
  --namespace approval-caller-namespace --operation-id approval-1
```

```
  Operation       RequestApproval
  Status          Running
  State           Started
  OperationToken  eyJ0IjoxLCJucyI6ImFwcHJvdmFsLWhhbmRsZXItbmFtZXNwYWNlIiwid2lkIjoiYXBwcm92YWwtbGFwdG9wLTQyIn0

Links: 1

  Link  temporal:///namespaces/approval-handler-namespace/workflows/approval-laptop-42/...
```

The Workflow Id is `approval-laptop-42`, derived from the item id as decided in
[step 3](/develop/java/nexus/development-walkthrough/choose-backing-implementation#give-the-approval-workflow-a-stable-id).
The link points from the Operation to the approval Workflow in the handler Namespace; the
Nexus-aware Client added it automatically.

The approval stays open until something decides it. [Step 7](/develop/java/nexus/development-walkthrough/add-messaging)
adds the Operation that does.

> **💡 Tip:**
> RESOURCES
>
> - [Nexus Endpoints](/nexus/endpoints) and [Nexus Registry](/nexus/registry) for the concepts and management surfaces.
> - [Nexus security](/nexus/security) for the Endpoint authorization model.
> - [Temporal Cloud Nexus](/cloud/nexus) for Cloud-specific setup and limits.
> - [Environment configuration](/develop/environment-configuration) for managing two Namespace profiles.
>

## Next step

- [Step 6: Call the Service](/develop/java/nexus/development-walkthrough/call-the-service): A caller Workflow in its own Namespace.

---

# Send messages to a pending approval - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/development-walkthrough/send-messages

> Signal-with-Start attaches a note whether or not the approval exists, and a use-existing conflict policy lets several callers await the same approval.

> **📝 Note:**
> Keep these running
>
> This step uses the processes started earlier. Leave them running:
>
> - The development server, from [step 4](/develop/java/nexus/development-walkthrough/implement-the-service#start-the-development-server).
> - The handler Worker, from [step 4](/develop/java/nexus/development-walkthrough/implement-the-service#run-the-worker).
> - The caller Worker, from [step 6](/develop/java/nexus/development-walkthrough/call-the-service#call-the-operations-from-a-caller-workflow).
>

To the caller, the messaging Operations are ordinary Operations, called through the same generated stub as `requestApproval` with the same type checking. Whether each one is a Signal, a Query, or an Update is the handler's private detail.

## Nudge, check, and decide

The caller Workflow in [step 6](/develop/java/nexus/development-walkthrough/call-the-service#call-the-operations-from-a-caller-workflow) calls all three. They differ in what comes back:

- `remindApprover` returns nothing and completes as soon as the Signal is accepted. Accepted means durably recorded, not yet handled; a caller that needs confirmation that the nudge took effect needs an Update.
- `getApprovalStatus` returns the approval's progress once, without changing it. The caller checks it once and doesn't loop on it, because the decision arrives as the result of `requestApproval`.
- `submitDecision` returns confirmation that the decision was recorded. The approval Workflow then completes, which resolves the `requestApproval` Operation the original caller is awaiting.

`remindApprover` and `submitDecision` fail if no approval is running for the purchase. `getApprovalStatus` also answers after the approval completes, until the Retention Period expires.

## Attach information before the approval exists

`attachApprovalContext` does not need a running approval.

Supporting information, such as a justification or a manager's note, comes from a different system than the approval request, so it sometimes arrives first. A plain Signal to a Workflow that does not exist fails, which is wrong for a message meant to be accepted whenever it shows up.

`attachApprovalContext` uses **Signal-with-Start** instead. If the approval is running, the note is delivered to it. If not, the approval is started and the note is delivered. The caller doesn't need to know which happened.

<!--SNIPSTART samples-java-nexus-walkthrough-attach-context-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java)
```java
@TemporalOperation
public TemporalOperationResult<Void> attachApprovalContext(
    TemporalOperationStartContext ctx,
    TemporalNexusClient client,
    AttachApprovalContextInput input) {
  WorkflowClient workflowClient = client.getWorkflowClient();
  ApprovalWorkflow stub =
      workflowClient.newWorkflowStub(
          ApprovalWorkflow.class,
          WorkflowOptions.newBuilder()
              .setWorkflowId(ApprovalWorkflowId.forItem(input.getItemId()))
              .setTaskQueue(HandlerWorker.DEFAULT_TASK_QUEUE_NAME)
              .build());

  // signalWithStart delivers the Signal, starting the Workflow first if it is not already
  // running. When the approval already exists, only the Signal is delivered.
  BatchRequest request = workflowClient.newSignalWithStartRequest();
  request.add(stub::attachContext, input.getNote());
  request.add(
      stub::runApproval,
      new RequestApprovalInput(input.getItemId(), input.getRequester(), input.getAmount()));
  workflowClient.signalWithStart(request);

  return TemporalOperationResult.sync(null);
}

```
<!--SNIPEND-->

Signal-with-Start is sync messaging on the [Client](/nexus/temporal-operation-handler#the-nexus-aware-client), so the Operation completes during the handler call and returns nothing. The caller learns only that the note was durably attached.

### Input needs enough to start the Workflow

In the [contract](/develop/java/nexus/development-walkthrough/define-the-data-contract#plan-the-operations), `attachApprovalContext` takes the item id, requester, amount, *and* note, while `remindApprover` takes only the item id. Because the Operation might start the approval Workflow, it must carry what the Workflow needs to start.

In general, a with-Start message's input is the union of what the message needs and what the Workflow's start needs.

## When the approval already exists

Because `attachApprovalContext` can create the approval, a Workflow with that Id may already be running when someone calls `requestApproval`. Both Operations derive the same Workflow Id from the item id, as decided in [step 3](/develop/java/nexus/development-walkthrough/choose-backing-implementation#give-the-approval-workflow-a-stable-id), so that they agree on which approval they mean.

**By default, `requestApproval` fails in this situation**, because starting a Workflow whose Id is already running is an error. A Workflow-backed Operation has only started once its completion callback is attached. If a conflicting start did nothing, the Operation would report success with no callback, and the caller would wait for a decision that never arrives.

That is why `requestApproval` sets the Workflow Id conflict policy to **use-existing**. A start against a running approval then attaches the Operation's completion callback to that Execution, and the caller receives its decision when it completes.

<!--SNIPSTART samples-java-nexus-walkthrough-request-approval-->
[core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexuswalkthrough/handler/ApprovalServiceImpl.java)
```java
@TemporalOperation
public TemporalOperationResult<RequestApprovalOutput> requestApproval(
    TemporalOperationStartContext ctx, TemporalNexusClient client, RequestApprovalInput input) {
  return client.startWorkflow(
      ApprovalWorkflow.class,
      ApprovalWorkflow::runApproval,
      input,
      WorkflowOptions.newBuilder()
          .setWorkflowId(ApprovalWorkflowId.forItem(input.getItemId()))
          .setWorkflowIdConflictPolicy(
              WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING)
          .build());
}

```
<!--SNIPEND-->

This has two effects:

- **More than one caller can await the same approval.** Every attached callback is notified when the Workflow completes, so several systems can call `requestApproval` for one purchase and all receive the same decision.
- **The Operation is idempotent across separate callers**, not only across server retries of one request.

Use-existing attaches only to a *running* Execution. If the approval has completed, the call starts a new approval rather than returning the old decision. A system that needs the outcome of a finished approval must have attached while it was open, or be notified by the handler, which the next step adds.

## Run it in the harder order

Send the note first, for an item id that has no approval yet:

```bash title="Run 1 of 3: attach context before the approval exists"
temporal nexus operation execute \
  --namespace approval-caller-namespace \
  --endpoint approval-endpoint \
  --service temporal.samples.approval.v1.ApprovalService \
  --operation AttachApprovalContext \
  --operation-id attach-1 \
  --input '{"itemId":"desk-7","requester":"tao","amount":1250,"note":"Approved in the Q3 budget"}'
```

```
Results:
  Status  COMPLETED
  Result  null
```

No approval existed for `desk-7`, so the Operation created one and delivered the note:

```bash title="Run 2 of 3: confirm only one approval was created"
temporal workflow list --namespace approval-handler-namespace
```

```
   Status        WorkflowId            Type
  Running    approval-desk-7     ApprovalWorkflow
  Completed  approval-laptop-42  ApprovalWorkflow
```

`approval-laptop-42` is the approval from step 5, decided in step 7.

Now request the approval for the same item. Under the default conflict policy this would fail; with
use-existing it attaches:

```bash title="Run 3 of 3: request the approval that already exists"
temporal nexus operation start \
  --namespace approval-caller-namespace \
  --endpoint approval-endpoint \
  --service temporal.samples.approval.v1.ApprovalService \
  --operation RequestApproval \
  --operation-id approval-desk-7 \
  --input '{"itemId":"desk-7","requester":"tao","amount":1250}'
```

`temporal workflow list` still shows one `approval-desk-7`: the call attached its completion
callback to the approval the note created. Decide it with `SubmitDecision` as in
[step 7](/develop/java/nexus/development-walkthrough/add-messaging#run-the-messaging-operations)
and the result arrives on the Operation started here.

> **📝 Note:**
> The two Workflow Id policies
>
> Two policies govern a start against a Workflow Id already in use:
>
> - The [Workflow Id conflict policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) applies while a Workflow with that Id is **running**. It defaults to failing with `Workflow execution already started`, which this section replaces with use-existing.
> - The [Workflow Id reuse policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy) applies once the previous Workflow with that Id has **closed**. It defaults to Allow Duplicate, which permits a new Execution.
>
> A start against a completed approval falls to the reuse policy and opens a new one. Set the reuse policy too if you don't want a second approval for the same item.
>

> **💡 Tip:**
> RESOURCES
>
> - [Sending messages](/sending-messages) for Signal and Update semantics.
> - [Temporal Operation Handler](/nexus/temporal-operation-handler) for sync messaging and async backings.
> - [Java Nexus feature guide](/develop/java/nexus/feature-guide) for the caller API.
>

## Next step

- [Step 9: Add a Standalone Activity](/develop/java/nexus/development-walkthrough/add-a-standalone-activity): An Operation with no Workflow behind it.

---

# Nexus feature guide - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/feature-guide

> Use Temporal Nexus within the Java SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **⚠️ Currently in: Pre-release:**
> This feature guide includes the new Nexus developer experience: pre-release APIs for the [Temporal Operation Handler](/nexus/temporal-operation-handler) and [Nexus Standalone Activity](/nexus/standalone-activity). These APIs are experimental and subject to change.

Use [Temporal Nexus](/evaluate/features/nexus) to connect Temporal Applications within and across Namespaces using a Nexus
Endpoint, a Nexus Service contract, and Nexus Operations.

> **💡 Tip:**
>
> New to Nexus? Start with the [Nexus Java quickstart](/develop/java/nexus/quickstart).
>

This page shows how to do the following:

- [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server)
- [Create caller and handler Namespaces](#create-caller-handler-namespaces)
- [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint)
- [Define the Nexus Service contract](#define-nexus-service-contract)
- [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers)
- [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service)
- [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server)
- [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code derived from the
> [Java Nexus sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexus) and the [nexusmessagingtemporaloperation sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusmessagingtemporaloperation).
>

## Run the Temporal Development Server with Nexus enabled 

Prerequisites:

- [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/java/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli)
  (v1.9.1 or higher recommended)
- [Install the latest Temporal Java SDK](https://learn.temporal.io/getting_started/java/dev_environment/#add-temporal-java-sdk-dependencies)
  (v1.39.0 or higher recommended)

The first step in working with Temporal Nexus involves starting a Temporal server with Nexus enabled.

```
temporal server start-dev \
  --dynamic-config-value history.enableUpdateCallbacks=true \
  --dynamic-config-value history.enableCHASMSignalBacklinks=true \
  --dynamic-config-value history.enableSignalWithStartFromWorkflow=true
```

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server
should now be available for client connections on `localhost:7233`.

## Create caller and handler Namespaces 

Before setting up Nexus endpoints, create separate Namespaces for the caller and handler.

```
temporal operator namespace create --namespace my-target-namespace
temporal operator namespace create --namespace my-caller-namespace
```

`my-target-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `my-caller-namespace` to
call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls.

## Create a Nexus Endpoint to route requests from caller to handler 

After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests.

```
temporal operator nexus endpoint create \
  --name my-nexus-endpoint-name \
  --target-namespace my-target-namespace \
  --target-task-queue my-handler-task-queue
```

You can also use the Web UI to create the Namespaces and Nexus endpoint.

## Define the Nexus Service contract 

Defining a clear contract for the Nexus Service is crucial for smooth communication.

In this example, there is a service package that describes the Service and Operation names along with input/output types
for caller Workflows to use the Nexus Endpoint.

You can hand-write that package, but the preferred way is to generate it with [NexGen](/nexus/nexgen).
You write the contract once as a JSON or YAML definition file and run `nexgen` against it, and it emits the typed models,
runtime validators, and the Service definition itself.

This is what makes a Nexus Service polyglot. Both sides generate from the same definition file: the handler implements
the Service, the caller invokes its Operations, and neither side hand-writes request or response types. A Java handler
and a Go caller share no code, but they both run off that same service contract, so they interoperate with no
coordination between the teams beyond the contract itself.

The generated validators check every payload against the contract when a value is parsed off the wire and again when
it is serialized onto it, so bad data is rejected at the boundary rather than reaching your Workflow or Activity. A value validates
identically in every language, which is what lets a caller and a handler written in different languages trust the same
contract. See the [`chat.nexusrpc.yaml`](https://github.com/temporalio/nexgen/blob/main/samples/schemas/chat.nexusrpc.yaml)
sample contract and the [Definition files](https://github.com/temporalio/nexgen#definition-files) section of the
`nexgen` README for the file format.

## Develop a Nexus Service and Operation handlers 

Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract.
Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying
Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code.
Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database.
Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive
retryable errors, blocking all Operations from the caller to that Endpoint.

Write each Operation as a method annotated with `@TemporalOperation` on a `@ServiceImpl` class. Java SDK v1.39.0 and
later support it. The method takes three parameters, a `TemporalOperationStartContext`, a `TemporalNexusClient`, and
the Operation input, and returns a `TemporalOperationResult`. When the Worker registers the Service, the SDK wraps the
method in a [`TemporalOperationHandler`](/nexus/temporal-operation-handler). What you do with the Client determines what
backs the Operation:

- **Synchronous.** Return `TemporalOperationResult.sync(...)` and the Operation completes during the handler call. The
  caller has its result as soon as the call returns.
- **Asynchronous.** Call `startWorkflow`, `startActivity`, or `startWorkflowUpdate` on the Client. The handler returns
  as soon as that Execution has started, and the Operation stays open until the Execution finishes, which may be days
  later. Its result is delivered to the caller through the Nexus completion callback. This is what lets an Operation
  outlive the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

A handler can perform any number of synchronous side effects, such as sending a Signal, but at most one asynchronous
backing per invocation.

`@TemporalOperation` gives the Operation the default cancel behavior. To change how a cancellation reaches the backing
Execution, annotate the method with `@OperationImpl` instead and return a `TemporalOperationHandler` subclass that
overrides `cancelWorkflowRun`, `cancelUpdateWorkflow`, or `cancelActivityExecution`. One `@ServiceImpl` class can use
both annotations, but not on the same method.

### Develop a Synchronous Nexus Operation handler

Return a synchronous result when the Operation can answer immediately. The handler computes the answer and returns it,
and the Operation completes during the call.

Handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking), and the whole
call has to finish inside the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

```java
@ServiceImpl(service = SampleNexusService.class)
public class SampleNexusServiceImpl {

  @TemporalOperation
  public TemporalOperationResult<SampleNexusService.EchoOutput> echo(
      TemporalOperationStartContext ctx,
      TemporalNexusClient client,
      SampleNexusService.EchoInput input) {
    return TemporalOperationResult.sync(new SampleNexusService.EchoOutput(input.getMessage()));
  }
}
```

### Use the Temporal Client for Signals, Queries, and Updates

A common pattern is to reach a Workflow that is already running. Query it or Signal it from a synchronous Operation, or
use Signal-With-Start to make sure the Workflow exists before the Signal arrives. Those calls complete during the
handler call, so they have to finish inside the
[Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

Updates are the exception. Do not wait for one inside the handler. Start it with `startWorkflowUpdate` and it backs the
Operation. The handler returns immediately, and the Operation completes when the Update does, however long it takes.

The [nexusmessaging](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusmessaging)
sample shows a Nexus Service that Queries and Signals a running Workflow from synchronous Operations, and backs an
Operation with a Workflow Update.

The Client your handler receives is not an ordinary Temporal Client. It propagates
[bidirectional links](/nexus/execution-debugging#bi-directional-linking) and request IDs on every call, so the
caller-side and handler-side Executions are connected in the UI without wiring anything. Reach the Workflow Client
through `client.getWorkflowClient()` rather than constructing your own.

In this example the Workflow ID is derived from an identifier carried in the Operation input, so the caller only needs
the identifier it cares about:

```java
@TemporalOperation
public TemporalOperationResult<ApproveOutput> approve(
    TemporalOperationStartContext ctx, TemporalNexusClient client, ApproveInput input) {
  client
      .getWorkflowClient()
      .newWorkflowStub(GreetingWorkflow.class, "GreetingWorkflow_for_" + input.getUserId())
      .approve(input);
  return TemporalOperationResult.sync(new ApproveOutput());
}
```

There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusmessaging/callerpattern/) and [on-demand pattern](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusmessaging/ondemandpattern/).
The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it.

### Develop an Asynchronous Nexus Operation handler to start a Workflow

Call `startWorkflow` on the Client. The Operation completes when the Workflow returns, and the Workflow's return
value is delivered to the caller as the Operation's result.

```java
@TemporalOperation
public TemporalOperationResult<SampleNexusService.HelloOutput> hello(
    TemporalOperationStartContext ctx,
    TemporalNexusClient client,
    SampleNexusService.HelloInput input) {
  return client.startWorkflow(
      HelloHandlerWorkflow.class,
      HelloHandlerWorkflow::hello,
      input,
      WorkflowOptions.newBuilder()
          .setWorkflowId(
              String.format(
                  "hello-%s-%s",
                  input.getName(), input.getLanguage().name().toLowerCase(Locale.ROOT)))
          .build());
}
```

Workflow IDs should typically be business-meaningful identifiers and are used to dedupe Workflow starts.
In general, the Workflow ID should be passed in the Operation input as part of the Nexus Service contract.

> **💡 Tip:**
> RESOURCES
>
> [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a
> Conflict-Policy of Use-Existing.
>

#### Map a Nexus Operation input to multiple Workflow arguments

A Nexus Operation can only take one input parameter. To start a Workflow that takes several, pass the arguments
directly to `startWorkflow` between the method reference and the Workflow options:

```java
@TemporalOperation
public TemporalOperationResult<SampleNexusService.HelloOutput> hello(
    TemporalOperationStartContext ctx,
    TemporalNexusClient client,
    SampleNexusService.HelloInput input) {
  return client.startWorkflow(
      HelloHandlerWorkflow.class,
      HelloHandlerWorkflow::hello,
      input.getName(),
      input.getLanguage(),
      WorkflowOptions.newBuilder().setWorkflowId("hello-" + input.getName()).build());
}
```

### Register a Nexus Service in a Worker

After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register a Nexus
Service in a Worker.

<!--SNIPSTART samples-java-nexus-handler-worker-->

[core/src/main/java/io/temporal/samples/nexus/handler/HandlerWorker.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/handler/HandlerWorker.java)

```java
package io.temporal.samples.nexus.handler;

import io.temporal.client.WorkflowClient;
import io.temporal.samples.nexus.options.ClientOptions;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;

public class HandlerWorker {
  public static final String DEFAULT_TASK_QUEUE_NAME = "my-handler-task-queue";

  public static void main(String[] args) {
    WorkflowClient client = ClientOptions.getWorkflowClient(args);

    WorkerFactory factory = WorkerFactory.newInstance(client);

    Worker worker = factory.newWorker(DEFAULT_TASK_QUEUE_NAME);
    worker.registerWorkflowImplementationTypes(HelloHandlerWorkflowImpl.class);
    worker.registerNexusServiceImplementation(new SampleNexusServiceImpl());

    factory.start();
  }
}
```

<!--SNIPEND-->

## Develop a caller Workflow that uses the Nexus Service 

Import the Service API package that has the necessary service and operation names and input/output types to execute a
Nexus Operation from the caller Workflow:

<!--SNIPSTART samples-java-nexus-caller-echo-workflow-->

[core/src/main/java/io/temporal/samples/nexus/caller/EchoCallerWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/EchoCallerWorkflowImpl.java)

```java
package io.temporal.samples.nexus.caller;

import io.temporal.samples.nexus.service.SampleNexusService;
import io.temporal.workflow.NexusOperationOptions;
import io.temporal.workflow.NexusServiceOptions;
import io.temporal.workflow.Workflow;
import java.time.Duration;

public class EchoCallerWorkflowImpl implements EchoCallerWorkflow {
  SampleNexusService sampleNexusService =
      Workflow.newNexusServiceStub(
          SampleNexusService.class,
          NexusServiceOptions.newBuilder()
              .setOperationOptions(
                  NexusOperationOptions.newBuilder()
                      .setScheduleToCloseTimeout(Duration.ofSeconds(10))
                      .build())
              .build());

  @Override
  public String echo(String message) {
    return sampleNexusService.echo(new SampleNexusService.EchoInput(message)).getMessage();
  }
}
```

<!--SNIPEND-->

<!--SNIPSTART samples-java-nexus-caller-hello-workflow-->

[core/src/main/java/io/temporal/samples/nexus/caller/HelloCallerWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/HelloCallerWorkflowImpl.java)

```java
package io.temporal.samples.nexus.caller;

import io.temporal.samples.nexus.service.SampleNexusService;
import io.temporal.workflow.NexusOperationHandle;
import io.temporal.workflow.NexusOperationOptions;
import io.temporal.workflow.NexusServiceOptions;
import io.temporal.workflow.Workflow;
import java.time.Duration;

public class HelloCallerWorkflowImpl implements HelloCallerWorkflow {
  SampleNexusService sampleNexusService =
      Workflow.newNexusServiceStub(
          SampleNexusService.class,
          NexusServiceOptions.newBuilder()
              .setOperationOptions(
                  NexusOperationOptions.newBuilder()
                      .setScheduleToCloseTimeout(Duration.ofSeconds(10))
                      .build())
              .build());

  @Override
  public String hello(String message, SampleNexusService.Language language) {
    NexusOperationHandle<SampleNexusService.HelloOutput> handle =
        Workflow.startNexusOperation(
            sampleNexusService::hello, new SampleNexusService.HelloInput(message, language));
    // Optionally wait for the operation to be started. NexusOperationExecution will contain the
    // operation token in case this operation is asynchronous.
    handle.getExecution().get();
    return handle.getResult().get().getMessage();
  }
}
```

<!--SNIPEND-->

### Register the caller Workflow in a Worker

After developing the caller Workflow, the next step is to register it with a Worker.

<!--SNIPSTART samples-java-nexus-caller-worker-->

[core/src/main/java/io/temporal/samples/nexus/caller/CallerWorker.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/CallerWorker.java)

```java
package io.temporal.samples.nexus.caller;

import io.temporal.client.WorkflowClient;
import io.temporal.samples.nexus.options.ClientOptions;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;
import io.temporal.worker.WorkflowImplementationOptions;
import io.temporal.workflow.NexusServiceOptions;
import java.util.Collections;

public class CallerWorker {
  public static final String DEFAULT_TASK_QUEUE_NAME = "my-caller-workflow-task-queue";

  public static void main(String[] args) {
    WorkflowClient client = ClientOptions.getWorkflowClient(args);

    WorkerFactory factory = WorkerFactory.newInstance(client);

    Worker worker = factory.newWorker(DEFAULT_TASK_QUEUE_NAME);
    worker.registerWorkflowImplementationTypes(
        WorkflowImplementationOptions.newBuilder()
            .setNexusServiceOptions(
                Collections.singletonMap(
                    "SampleNexusService",
                    NexusServiceOptions.newBuilder().setEndpoint("my-nexus-endpoint-name").build()))
            .build(),
        EchoCallerWorkflowImpl.class,
        HelloCallerWorkflowImpl.class);

    factory.start();
  }
}
```

<!--SNIPEND-->

### Develop a starter to start the caller Workflow

To initiate the caller Workflow, a starter program is used.

<!--SNIPSTART samples-java-nexus-caller-starter-->

[core/src/main/java/io/temporal/samples/nexus/caller/CallerStarter.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/CallerStarter.java)

```java
package io.temporal.samples.nexus.caller;

import io.temporal.api.common.v1.WorkflowExecution;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.samples.nexus.options.ClientOptions;
import io.temporal.samples.nexus.service.SampleNexusService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class CallerStarter {
  private static final Logger logger = LoggerFactory.getLogger(CallerStarter.class);

  public static void main(String[] args) {
    WorkflowClient client = ClientOptions.getWorkflowClient(args);

    WorkflowOptions workflowOptions =
        WorkflowOptions.newBuilder().setTaskQueue(CallerWorker.DEFAULT_TASK_QUEUE_NAME).build();
    EchoCallerWorkflow echoWorkflow =
        client.newWorkflowStub(EchoCallerWorkflow.class, workflowOptions);
    WorkflowExecution execution = WorkflowClient.start(echoWorkflow::echo, "Nexus Echo 👋");
    logger.info(
        "Started EchoCallerWorkflow workflowId: {} runId: {}",
        execution.getWorkflowId(),
        execution.getRunId());
    logger.info("Workflow result: {}", echoWorkflow.echo("Nexus Echo 👋"));
    HelloCallerWorkflow helloWorkflow =
        client.newWorkflowStub(HelloCallerWorkflow.class, workflowOptions);
    execution = WorkflowClient.start(helloWorkflow::hello, "Nexus", SampleNexusService.Language.EN);
    logger.info(
        "Started HelloCallerWorkflow workflowId: {} runId: {}",
        execution.getWorkflowId(),
        execution.getRunId());
    logger.info("Workflow result: {}", helloWorkflow.hello("Nexus", SampleNexusService.Language.ES));
  }
}
```

<!--SNIPEND-->

## Make Nexus calls across Namespaces with a development Server 

Follow the steps below to run the Nexus handler Worker, the Nexus caller Worker, and the starter app.

### Run Workers connected to a local development server

Run the Nexus handler Worker:

```bash
./gradlew -q execute -PmainClass=io.temporal.samples.nexus.handler.HandlerWorker \
    --args="-target-host localhost:7233 -namespace my-target-namespace"
```

In another terminal window, run the Nexus caller Worker:

```bash
./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerWorker \
    --args="-target-host localhost:7233 -namespace my-caller-namespace"
```

### Start a caller Workflow

With the Workers running, the final step in the local development process is to start a caller Workflow.

Run the starter:

```bash
./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerStarter \
    --args="-target-host localhost:7233 -namespace my-caller-namespace"
```

This will result in:

```
[main] INFO  i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9b3de8ba-28ae-42fb-8087-bdedf4cecd39 runId: 404a2529-764d-4d1d-9de5-8a9475e40fba
[main] INFO  i.t.s.nexus.caller.CallerStarter - Workflow result: Nexus Echo 👋
[main] INFO  i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9cb29897-356a-4714-87b7-aa2f00784a46 runId: 7e71e62a-db50-49da-b081-24b61016a0fc
[main] INFO  i.t.s.nexus.caller.CallerStarter - Workflow result: ¡Hola! Nexus 👋
```

### Canceling a Nexus Operation 

To cancel a Nexus Operation from within a Workflow, create a `CancellationScope` using the
`Workflow.newCancellationScope` API. `Workflow.newCancellationScope` takes a `Runnable`. Any SDK methods started in this
runnable, such as Nexus operations, will be associated with this scope. `Workflow.newCancellationScope` returns a new
scope that, when the `cancel()` method is called, cancels the context and any SDK method that was started in the scope.
The promise returned by `Workflow.startNexusOperation` is resolved when the operation finishes, whether it succeeds,
fails, times out, or is canceled.

Only asynchronous operations can be canceled in Nexus, as cancelation is sent using an operation token. The Workflow or
other resources backing the operation may choose to ignore the cancelation request. If ignored, the operation may enter
a terminal state.

When a Nexus operation is started the caller can specify different cancellation types that will control how the caller
reacts to cancellation:

- `ABANDON` - Do not request cancellation of the operation.
- `TRY_CANCEL` - Initiate a cancellation request and immediately report cancellation to the caller. Note that this type
  doesn't guarantee that cancellation is delivered to the operation handler if the caller exits before the delivery is
  done.
- `WAIT_REQUESTED` - Request cancellation of the operation and wait for confirmation that the request was received.
  Doesn't wait for actual cancellation.
- `WAIT_COMPLETED` - Wait for operation completion. Operation may or may not complete as cancelled.

The default is `WAIT_COMPLETED`. Users can set a different option on the `NexusServiceOptions` by calling
`.setCancellationType()` on `NexusServiceOptions.Builder`.

Once the caller Workflow completes, the caller's Nexus Machinery stops attempting to cancel operations that have not yet
been canceled, letting them run to completion.

It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending
operations to finish before exiting the Workflow.

See the
[Nexus cancelation sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexuscancellation)
for reference.

## Make Nexus calls across Namespaces in Temporal Cloud 

This section assumes you are already familiar with
[how connect a Worker to Temporal Cloud](/develop/java/client/temporal-client#start-workflow-execution). The same
[source code](https://github.com/temporalio/samples-go/tree/main/nexus) is used in this section, but the Temporal Cloud
CLI will be used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the
caller and handler Workers to their respective Temporal Cloud Namespaces.

### Install `tcld` and generate certificates

Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command
(on macOS):

```
brew install temporalio/brew/tcld
```

If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below:

```
tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key
```

These certificates will be valid for one year.

### Create caller and handler Namespaces

Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler.
If you already have these Namespaces, you don't need to do this.

**Temporal CLI**

```
temporal cloud login

temporal cloud namespace create \
  --name <your-caller-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

temporal cloud namespace create \
  --name <your-target-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

**tcld**

```
tcld login

tcld namespace create \
  --namespace <your-caller-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

tcld namespace create \
  --namespace <your-target-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

Alternatively, you can create Namespaces through the UI:
[https://cloud.temporal.io/namespaces](https://cloud.temporal.io/namespaces).

### Create a Nexus Endpoint to route requests from caller to handler

To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the
`--target-namespace`.

**Temporal CLI**

```
temporal cloud nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file ./core/src/main/java/io/temporal/samples/nexus/service/description.md
```

**tcld**

```
tcld nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file ./core/src/main/java/io/temporal/samples/nexus/service/description.md
```

The `--allow-namespace` flag builds an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as
described in [Runtime Access Controls](/nexus/security#runtime-access-controls).

Alternatively, you can create a Nexus Endpoint through the UI:
[https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus).

### Run Workers Connected to Temporal Cloud

Run the handler Worker:

```
./gradlew -q execute -PmainClass=io.temporal.samples.nexus.handler.HandlerWorker \
  --args="-target-host <your-target-namespace.account>.tmprl.cloud:7233 \
  -namespace <your-target-namespace.account> \
  -client-cert 'path/to/your/ca.pem' \
  -client-key 'path/to/your/ca.key'"
```

Run the caller Worker:

```
./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerWorker \
  --args="-target-host <your-caller-namespace.account>.tmprl.cloud:7233 \
  -namespace <your-caller-namespace.account> \
  -client-cert 'path/to/your/ca.pem' \
  -client-key 'path/to/your/ca.key'"
```

### Start a caller Workflow

```
./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerStarter \
  --args="-target-host <your-caller-namespace.account>.tmprl.cloud:7233 \
  -namespace <your-caller-namespace.account> \
  -client-cert 'path/to/your/ca.pem' \
  -client-key 'path/to/your/ca.key'"
```

This will result in:

```
[main] INFO  i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9b3de8ba-28ae-42fb-8087-bdedf4cecd39 runId: 404a2529-764d-4d1d-9de5-8a9475e40fba
[main] INFO  i.t.s.nexus.caller.CallerStarter - Workflow result: Nexus Echo 👋
[main] INFO  i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9cb29897-356a-4714-87b7-aa2f00784a46 runId: 7e71e62a-db50-49da-b081-24b61016a0fc
[main] INFO  i.t.s.nexus.caller.CallerStarter - Workflow result: ¡Hola! Nexus 👋
```

## Observability

### Web UI

A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and
`NexusOperationCompleted` events in the caller's Event history:

![Observability Sync](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-sync.png)

An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`,
`NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history:

![Observability Async](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-async.png)

### Temporal CLI

Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks
on the handler Workflow:

```
temporal workflow describe -w <ID>
```

Nexus events are included in the caller's Event history:

```
temporal workflow show -w <ID>
```

For **asynchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationStarted`
- `NexusOperationCompleted`

For **synchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationCompleted`

> **📝 Note:**
>
> `NexusOperationStarted` isn't reported in the caller's history for synchronous operations.
>

## Learn more

- Read the high-level description of the [Temporal Nexus feature](/evaluate/features/nexus) and watch the
  [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082).
- Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and
  [Encyclopedia](/nexus).
- Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus).

---

# Nexus Java quickstart - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/quickstart

> Build a Nexus Service that wraps an existing Temporal Workflow using the Java SDK

[Temporal Nexus](/evaluate/features/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow.

> **ℹ️ Info:**
> To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/features/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus).

**Prerequisites:** Complete the [Java SDK Quickstart](/develop/java/set-up-your-local-java) first.
You should have `SayHelloWorkflow`, `SayHelloWorkflowImpl`, `GreetActivities`, `GreetActivitiesImpl`, and `SayHelloWorker` from that guide.

## What you'll build

You have `SayHelloWorkflow` running in the `default` Namespace.
By the end of this guide:

1. A Nexus Service will expose `SayHelloWorkflow` as an Operation.
2. A second Namespace will contain a Workflow that calls that Operation.
3. The caller Workflow will get back `"Hello Temporal"` — the same result, but across Namespaces.

## 1. Define the Nexus Service

Create a file called `SayHelloNexusService.java` that defines the Nexus Service contract.

Creating a Nexus Service establishes the contract between your implementation and any callers.
It provides type safety when invoking Nexus Operations and ensures that Operation Handlers fulfill the contract.

The `@Service` annotation declares this as a Nexus Service.
The `@Operation` annotation marks `sayHello` as a callable Nexus Operation.
`SayHelloWorkflow` returns `String`, so the operation output type is also `String`.

```java
package helloworkflow;

import io.nexusrpc.Operation;
import io.nexusrpc.Service;

@Service
public interface SayHelloNexusService {

    @Operation
    String sayHello(String name);

}
```

## 2. Define the Nexus Operation Handlers

Create a file called `SayHelloNexusServiceImpl.java` that implements the Nexus Operation handler.

Operation handlers contain the logic that runs when a caller invokes a Nexus Operation.

The `@OperationImpl` annotation creates an asynchronous Nexus Operation that starts `SayHelloWorkflow`.
The handler bridges the Nexus `String` input to `SayHelloWorkflow`'s `sayHello` method directly.

```java
package helloworkflow;

import io.nexusrpc.handler.OperationHandler;
import io.nexusrpc.handler.OperationImpl;
import io.nexusrpc.handler.ServiceImpl;
import io.temporal.client.WorkflowOptions;
import io.temporal.nexus.Nexus;
import io.temporal.nexus.WorkflowRunOperation;

@ServiceImpl(service = SayHelloNexusService.class)
public class SayHelloNexusServiceImpl {

    @OperationImpl
    public OperationHandler<String, String> sayHello() {
        return WorkflowRunOperation.fromWorkflowMethod(
            (ctx, details, name) ->
                Nexus.getOperationContext()
                    .getWorkflowClient()
                    .newWorkflowStub(
                        SayHelloWorkflow.class,
                        WorkflowOptions.newBuilder()
                            .setWorkflowId("say-hello-nexus-" + details.getRequestId())
                            .build())
                    ::sayHello
        );
    }

}
```

## 3. Register the Nexus Service Handler in a Worker

Update your existing `SayHelloWorker.java` to register the Nexus Service Handler.

A Worker will only poll for and process incoming Nexus requests if the Nexus Service Handlers are registered.
This is the same Worker concept used for Workflows and Activities.

The registerNexusServiceImplementation parameter registers the handler so it can receive Nexus Operation requests.

```java
package helloworkflow;

import io.temporal.client.WorkflowClient;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;

public class SayHelloWorker {

    public static void main(String[] args) {

        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service);
        WorkerFactory factory = WorkerFactory.newInstance(client);

        Worker worker = factory.newWorker("my-task-queue");
        worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class);
        worker.registerActivitiesImplementations(new GreetActivitiesImpl());
        worker.registerNexusServiceImplementation(new SayHelloNexusServiceImpl());

        System.out.println("Starting SayHelloWorker for task queue 'my-task-queue'...");

        factory.start();

    }

}
```

## 4. Develop the caller Workflow

Create two files — `NexusCallerWorkflow.java` and `NexusCallerWorkflowImpl.java` — that define a Workflow which invokes the Nexus Operation.

The caller Workflow demonstrates the consumer side of Nexus.
Instead of importing handler code directly, the caller only depends on the Service contract.
This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams.

The `Workflow.newNexusServiceStub` method creates a client bound to your Nexus Service and Endpoint.

```java
package helloworkflow;

import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;

@WorkflowInterface
public interface NexusCallerWorkflow {

    @WorkflowMethod
    String greetThroughNexus(String name);

}
```

```java
package helloworkflow;

import io.temporal.workflow.NexusOperationOptions;
import io.temporal.workflow.NexusServiceOptions;
import io.temporal.workflow.Workflow;

import java.time.Duration;

public class NexusCallerWorkflowImpl implements NexusCallerWorkflow {

    private final SayHelloNexusService nexusService = Workflow.newNexusServiceStub(
        SayHelloNexusService.class,
        NexusServiceOptions.newBuilder()
            .setOperationOptions(
                NexusOperationOptions.newBuilder()
                    .setScheduleToCloseTimeout(Duration.ofSeconds(10))
                    .build())
            .build()
    );

    @Override
    public String greetThroughNexus(String name) {
        return nexusService.sayHello(name);
    }

}
```

## 5. Create the caller Namespace and Nexus Endpoint

Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler.
The handler uses the `default` Namespace that was created when you started the dev server.

Namespaces provide isolation between the caller and handler sides.
The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue.

Make sure your local Temporal dev server is running (`temporal server start-dev`).

```bash
temporal operator namespace create --namespace my-caller-namespace
```

```bash
temporal operator nexus endpoint create \\
  --name my-nexus-endpoint-name \\
  --target-namespace default \\
  --target-task-queue my-task-queue
```

## 6. Run and verify

Create `NexusCallerStarter.java` to start the caller Worker and execute the Workflow.

This brings everything together: the caller Worker hosts `NexusCallerWorkflow`, which uses the Nexus stub to invoke `sayHello` on the handler side.
The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `SayHelloWorkflow`, and back to the caller.

**Run the application:**

1. Start the handler Worker in one terminal:

```bash
mvn compile exec:java \
  -Dexec.mainClass="helloworkflow.SayHelloWorker"
```

2. Run the caller in another terminal:

```bash
mvn compile exec:java \
  -Dexec.mainClass="helloworkflow.NexusCallerStarter"
```

You should see:

```
Workflow result: Hello Temporal
```

Open the [Temporal Web UI](http://localhost:8233) and switch between Namespaces to see both Workflow Executions.
In `my-caller-namespace`, find the `NexusCallerWorkflow` execution — you should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in its history.
In `default`, find the `SayHelloWorkflow` execution that was started by the Nexus Operation.

```java
package helloworkflow;

import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.client.WorkflowOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;
import io.temporal.worker.WorkflowImplementationOptions;
import io.temporal.workflow.NexusServiceOptions;

import java.util.Collections;

public class NexusCallerStarter {

    public static void main(String[] args) {

        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service,
            WorkflowClientOptions.newBuilder()
                .setNamespace("my-caller-namespace")
                .build()
        );
        WorkerFactory factory = WorkerFactory.newInstance(client);

        Worker worker = factory.newWorker("my-caller-task-queue");
        worker.registerWorkflowImplementationTypes(
            WorkflowImplementationOptions.newBuilder()
                .setNexusServiceOptions(
                    Collections.singletonMap(
                        "SayHelloNexusService",
                        NexusServiceOptions.newBuilder()
                            .setEndpoint("my-nexus-endpoint-name")
                            .build()))
                .build(),
            NexusCallerWorkflowImpl.class
        );
        factory.start();

        NexusCallerWorkflow workflow = client.newWorkflowStub(
            NexusCallerWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue("my-caller-task-queue")
                .setWorkflowId("nexus-caller-workflow-id")
                .build()
        );

        String result = workflow.greetThroughNexus("Temporal");
        System.out.println("Workflow result: " + result);

        factory.shutdown();
        System.exit(0);

    }

}
```

## Next steps

Now that you have a working Nexus Service, here are some resources to deepen your understanding:

- **[Java Nexus feature guide](/develop/java/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls.
- **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics.
- **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker.
- **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns.
- **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries.
- **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls.
- **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests.
- **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity.

---

# Standalone Nexus Operations - Java SDK

Source: https://docs.temporal.io/develop/java/nexus/standalone-operations

> Execute Nexus Operations independently without a Workflow using the Temporal Java SDK.

> **Public Preview**
> Requires Java SDK `v1.36.1` or above. API breaking changes are kept to a minimum before General Availability.

[Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without
being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using
`Workflow.newNexusServiceStub()`, you execute a Standalone Nexus Operation directly from a Nexus service client created
from a `NexusClient` using `NexusClient.newNexusServiceClient()`.

Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as
Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/java/nexus/feature-guide) for details on
[defining a Service contract](/develop/java/nexus/feature-guide#define-nexus-service-contract),
[developing Operation handlers](/develop/java/nexus/feature-guide#develop-nexus-service-operation-handlers), and
[registering a Service in a Worker](/develop/java/nexus/feature-guide#register-a-nexus-service-in-a-worker).

This page focuses on the client-side APIs that are unique to Standalone Nexus Operations:

- [Execute a Standalone Nexus Operation](#execute-operation)
- [Start a Standalone Nexus Operation and Wait for the Result](#get-operation-result)
- [List Standalone Nexus Operations](#list-operations)
- [Count Standalone Nexus Operations](#count-operations)
- [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud)

> **📝 Note:**
> This documentation uses source code from the
> [Java Nexus Standalone sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusstandalone).
>

## Prerequisites 

Standalone Nexus Operations are a Public Preview feature and require Temporal CLI v1.9.0 or higher.

### 1. Install and verify the Temporal CLI 

The `temporal nexus operation` commands require Temporal CLI v1.9.0 or higher. See
[Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for install options,
then verify:

```bash
temporal --version
```

Which should output v1.9.0 or higher.

### 2. Start a local dev server 

The dev server enables Standalone Nexus Operations by default — no dynamic config is
required. Start it with the caller and handler Namespaces pre-created:

```bash
temporal server start-dev \
  --namespace my-caller-namespace \
  --namespace my-handler-namespace
```

The starter and Worker connect to two different Namespaces (a caller Namespace and a handler
Namespace), mirroring how Nexus crosses Namespace boundaries.

To run the examples on this page against the [Java sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusstandalone),
create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue:

```bash
temporal operator nexus endpoint create \
  --name my-nexus-endpoint \
  --target-namespace my-handler-namespace \
  --target-task-queue nexus-handler-queue
```

Start the sample handler Worker in the handler Namespace:

```bash
TEMPORAL_NAMESPACE=my-handler-namespace \
  ./gradlew -q :core:execute -PmainClass=io.temporal.samples.nexusstandalone.handler.HandlerWorker
```

Run the starter in the caller Namespace (from a separate terminal):

```bash
TEMPORAL_NAMESPACE=my-caller-namespace \
  ./gradlew -q :core:execute -PmainClass=io.temporal.samples.nexusstandalone.StandaloneClientStarter
```

## Execute a Standalone Nexus Operation 

To execute a Standalone Nexus Operation, first create a
[`NexusClient`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusClient.html), then
derive a typed
[`NexusServiceClient`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusServiceClient.html)
from it with `newNexusServiceClient()`, bound to a specific Nexus Endpoint and Service. The endpoint must be
pre-created on the server. Then call `start()` or `execute()` from application code (for example, a starter program),
not from inside a Workflow Definition.

`execute()` waits for the Operation to complete and returns the result.
Both methods take a [`StartNexusOperationOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/StartNexusOperationOptions.html)
whose `id` is required — the SDK never generates one for you. `scheduleToCloseTimeout` is optional and defaults to the
maximum allowed by the Temporal server.

```java
NexusClient nexusClient = NexusClient.newInstance(stubs, options);
NexusServiceClient<GreetingNexusService> greetingClient =
    nexusClient.newNexusServiceClient(GreetingNexusService.class, ENDPOINT_NAME);

// Block until the operation completes and return its result.
GreetingOutput greeting =
    greetingClient.execute(
        GreetingNexusService::greet,
        StartNexusOperationOptions.newBuilder()
            .setId("greet-" + UUID.randomUUID())
            .setScheduleToCloseTimeout(Duration.ofSeconds(10))
            .build(),
        new GreetingInput("World"));
```

`executeAsync()` is the same but returns a `CompletableFuture` instead of blocking.

```java
CompletableFuture<GreetingOutput> future =
    greetingClient.executeAsync(
        GreetingNexusService::greet, options, new GreetingInput("World"));
GreetingOutput greeting = future.get();
```

See the full
[starter sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexusstandalone/StandaloneClientStarter.java)
for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and
counts Operations.

Or use the Temporal CLI to execute a Standalone Nexus Operation:

```bash
temporal nexus operation execute \
  --namespace my-caller-namespace \
  --endpoint my-nexus-endpoint \
  --service GreetingNexusService \
  --operation greet \
  --operation-id my-greet-op \
  --input '{"name":"World"}'
```

## Start a Standalone Nexus Operation and Wait for the Result 

`start()` returns a
[`NexusOperationHandle`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusOperationHandle.html).
Use `NexusOperationHandle.getResult()` to wait until the Operation completes and retrieve its result. This works for
both synchronous and asynchronous Operations.

```java
// Start an operation and get a NexusOperationHandle.
NexusOperationHandle<GreetingOutput> handle =
    greetingClient.start(
        GreetingNexusService::startGreeting, options, new GreetingInput("World"));

// Block until the operation completes and retrieve its result.
GreetingOutput greeting = handle.getResult();
```

If the Operation completed successfully, the result is returned. If the Operation failed, the failure is thrown as a
`NexusOperationException`. Use `getResultAsync()` for a non-blocking `CompletableFuture`, or
`getResult(long timeout, TimeUnit unit)` to bound the wait.

Or use the Temporal CLI to wait for a result by Operation ID:

```bash
temporal nexus operation result --namespace my-caller-namespace --operation-id my-greet-op
```

## List Standalone Nexus Operations 

Use [`NexusClient.listNexusOperationExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusClient.html)
to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. The result is a `Stream`
of operation metadata entries.

Note that `listNexusOperationExecutions()` is called on a `NexusClient`, not on the typed `NexusServiceClient`.

```java
String query = "Endpoint = \"" + ENDPOINT_NAME + "\"";
nexusClient
    .listNexusOperationExecutions(query)
    .forEach(
        op ->
            System.out.printf(
                "OperationId: %s, Operation: %s, Status: %s%n",
                op.getOperationId(), op.getOperation(), op.getStatus()));
```

The `query` parameter accepts [List Filter](/list-filter) syntax. For example,
`"Endpoint = 'my-endpoint' AND ExecutionStatus = 'Running'"`.

Or use the Temporal CLI:

```bash
temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Count Standalone Nexus Operations 

Use [`NexusClient.countNexusOperationExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusClient.html)
to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query.

Note that `countNexusOperationExecutions()` is called on a `NexusClient`, not on the typed `NexusServiceClient`.

```java
String query = "Endpoint = \"" + ENDPOINT_NAME + "\"";
NexusOperationExecutionCount count = nexusClient.countNexusOperationExecutions(query);
System.out.println("Total Nexus operations: " + count.getCount());
```

Passing a `GROUP BY` query (for example, `"GROUP BY ExecutionStatus"`) returns a count per group, available through
`NexusOperationExecutionCount.getGroups()`.

Or use the Temporal CLI:

```bash
temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Run Standalone Nexus Operations with Temporal Cloud 

The code samples referenced on this page build their client from a `ClientConfigProfile` loaded from a TOML profile, so
the same code works against Temporal Cloud — just point the profile at your Cloud Namespace (or override the connection
via `TEMPORAL_*` environment variables). No code changes are needed.

For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint setup, certificate
generation, and authentication options, see
[Make Nexus calls across Namespaces in Temporal Cloud](/develop/java/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud)
and [Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud).

---

# Platform - Java SDK

Source: https://docs.temporal.io/develop/java/platform

> This section explains how to implement platform with the Java SDK

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Platform

- [Observability](/develop/java/platform/observability)
- [Enriching the UI](/develop/java/platform/enriching-ui)

---

# Enriching the user interface - Java SDK

Source: https://docs.temporal.io/develop/java/platform/enriching-ui

> Add contextual information to workflows and events in the Temporal UI using the Java SDK.

Temporal supports adding context to Workflows and Events with metadata. 
This helps users identify and understand Workflows and their operations.

## Adding summary and details to Workflows

### Starting a Workflow

When starting a workflow, you can provide a static summary and details to help identify the Workflow in the UI:

```java
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;

public class Main {
    public static void main(String[] args) {
        // Create service stubs and workflow client
        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient workflowClient = WorkflowClient.newInstance(service);
        
        // Create workflow options with static summary and details
        WorkflowOptions options = WorkflowOptions.newBuilder()
                .setWorkflowId("your-workflow-id")
                .setTaskQueue("your-task-queue")
                .setStaticSummary("Order processing for customer #12345")
                .setStaticDetails("Processing premium order with expedited shipping")
                .build();
        
        // Create the workflow stub
        YourWorkflow workflow = workflowClient.newWorkflowStub(YourWorkflow.class, options);
        
        // Start the workflow
        String result = workflow.yourWorkflowMethod("workflow input");
    }
}
```

`setStaticSummary()` sets a single-line description that appears in the Workflow list view, limited to 200 bytes.
`setStaticDetails()` sets multi-line comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes.

The input format is standard Markdown excluding images, HTML, and scripts.

You can also use `WorkflowClient.start()` for async execution:

```java
// Start workflow asynchronously
WorkflowExecution execution = WorkflowClient.start(workflow::yourWorkflowMethod, "workflow input");
```

### Inside the Workflow

Within a Workflow, you can get and set the _current workflow details_. 
Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. 
Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines.

```java
import io.temporal.workflow.Workflow;

public class YourWorkflowImpl implements YourWorkflow {
    @Override
    public String yourWorkflowMethod(String input) {
        // Get the current details
        String currentDetails = Workflow.getCurrentDetails();
        Workflow.getLogger(YourWorkflowImpl.class).info("Current details: " + currentDetails);
        
        // Set/update the current details
        Workflow.setCurrentDetails("Updated workflow details with new status");
        
        return "Workflow completed";
    }
}
```

### Adding Summary to Activities and Timers

You can attach a `setSummary()` to Activities when starting them from within a Workflow:

```java
import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.Workflow;
import java.time.Duration;

public class YourWorkflowImpl implements YourWorkflow {
    private final YourActivities activities = 
        Workflow.newActivityStub(YourActivities.class,
            ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofSeconds(10))
                .setSummary("Processing user data")
                .build());
    
    @Override
    public String yourWorkflowMethod(String input) {
        // Execute the activity with the summary
        String result = activities.yourActivity(input);
        return result;
    }
}
```

Similarly, you can attach a `setSummary()` to timers within a Workflow:

```java
import io.temporal.workflow.Workflow;
import io.temporal.workflow.TimerOptions;
import java.time.Duration;

public class YourWorkflowImpl implements YourWorkflow {
    @Override
    public String yourWorkflowMethod(String input) {
        // Create a timer with a summary
        Workflow.newTimer(Duration.ofMinutes(5), 
            TimerOptions.newBuilder()
                .setSummary("Waiting for payment confirmation")
                .build())
            .get(); // Wait for the timer to fire
        
        return "Timer completed";
    }
}
```

The input format for `setSummary()` is a string, and limited to 200 bytes.

## Viewing summary and details in the UI

Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. 
Navigate to your Workflow's details page to see the metadata displayed in three key locations:

### User Metadata

On the Workflow Details page, under the **User Metadata** tab, you'll find the workflow-level metadata:

- **Summary & Details** - Displays the static summary and static details set when starting the workflow
- **Current Details** - Displays the dynamic details that can be updated during workflow execution

All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays.

### Timeline

The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance.

Labels longer than 120 characters are truncated with an ellipsis.

Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. 

Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later.

### Event History

Individual events in the Workflow's Event History display their associated summaries when available.

Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the Event details. 
When you do expand an Event, the summary is also prominently displayed in the detailed view.

---

# Observability - Java SDK

Source: https://docs.temporal.io/develop/java/platform/observability

> Explore the observability features of Temporal, including Metrics, Tracing, Logging, and Visibility. Emit Metrics with the Java SDK, set up Tracing, and use Search Attributes.

The observability section of the Temporal Developer's guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

This section covers features related to viewing the state of the application, including:

- [Emit metrics](#metrics)
- [Set up tracing](#tracing)
- [Log from a Workflow](#logging)
- [Visibility APIs](#visibility)

## Emit metrics 

Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process.
For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics).

- For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide.
- For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics).
- For an end-to-end example that exposes metrics with the Java SDK, refer to the [samples-java](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics) repo.

To emit metrics with the Java SDK, use the [`MicrometerClientStatsReporter`](https://github.com/temporalio/sdk-java/blob/69a5d3fa44da4043697b7e57d7ef6691c24287cd/temporal-sdk/src/main/java/io/temporal/common/reporter/MicrometerClientStatsReporter.java#L18) class to integrate with Micrometer MeterRegistry configured for your metrics backend.
[Micrometer](https://micrometer.io/docs) is a popular Java framework that provides integration with Prometheus and other backends.

The following example shows how to use `MicrometerClientStatsReporter` to define the metrics scope and set it with the `WorkflowServiceStubsOptions`.

```java
//...
   // see the Micrometer documentation for configuration details on other supported monitoring systems.
   // in this example shows how to set up Prometheus registry and stats reported.
   PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
   StatsReporter reporter = new MicrometerClientStatsReporter(registry);
    // set up a new scope, report every 10 seconds
     Scope scope = new RootScopeBuilder()
             .reporter(reporter)
             .reportEvery(com.uber.m3.util.Duration.ofSeconds(10));
   // for Prometheus collection, expose a scrape endpoint.
   //...
   // add metrics scope to WorkflowServiceStub options
   WorkflowServiceStubsOptions stubOptions =
       WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build();
//...
```

For more details, see the [Java SDK Samples](https://github.com/temporalio/samples-java/tree/637c2e66fd2dab43d9f3f39e5fd9c55e4f3884f0/core/src/main/java/io/temporal/samples/metrics).
For details on configuring a Prometheus scrape endpoint with Micrometer, see the [Micrometer Prometheus Configuring](https://docs.micrometer.io/micrometer/reference/implementations/prometheus.html#_configuring) documentation.

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Pass the tags to `RootScopeBuilder.tags` when you build the metrics scope.
Every metric reported through that scope carries them, from both the Client and the Worker.

```java
//...
// Set up prometheus registry and stats reported
    PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    StatsReporter reporter = new MicrometerClientStatsReporter(registry);

    Map<String, String> globalTags = new HashMap<>();
    globalTags.put("team", "content-platform");
    globalTags.put("service", "checkout");
    globalTags.put("cost_center", "cc-1042");
    globalTags.put("environment", "production");

    Scope scope = new RootScopeBuilder()
        .tags(globalTags)
        .reporter(reporter)
        .reportEvery(com.uber.m3.util.Duration.ofSeconds(10));

    WorkflowServiceStubsOptions stubOptions =
        WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build();
//...
```

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag           | Example            | Question it answers                                       |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team`        | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service`     | `checkout`         | Which application emits these metrics?                    |
| `cost_center` | `cc-1042`          | Which budget does this Worker fleet belong to?            |
| `environment` | `production`       | Is this production traffic, or staging or test?           |
| `region`      | `us-east-2`        | Where does the Worker fleet run?                          |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Set up tracing 

Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows.

Temporal Web's tracing capabilities mainly track Activity Execution within a Temporal context. If you need custom tracing specific for your use case, you should make use of context propagation to add tracing logic accordingly.

To configure tracing in Java, register the `OpenTracingClientInterceptor()` interceptor.
You can register the interceptors on both the Temporal Client side and the Worker side, or through a [Plugin](/develop/plugins-guide#interceptors) if you're building a reusable library.

The following code examples demonstrate the `OpenTracingClientInterceptor()` on the Temporal Client.

```java
WorkflowClientOptions.newBuilder()
   //...
   .setInterceptors(new OpenTracingClientInterceptor())
   .build();
```

```java
WorkflowClientOptions clientOptions =
    WorkflowClientOptions.newBuilder()
        .setInterceptors(new OpenTracingClientInterceptor(JaegerUtils.getJaegerOptions(type)))
        .build();
WorkflowClient client = WorkflowClient.newInstance(service, clientOptions);
```

The following code examples demonstrate the `OpenTracingClientInterceptor()` on the Worker.

```java
WorkerFactoryOptions.newBuilder()
   //...
   .setWorkerInterceptors(new OpenTracingWorkerInterceptor())
   .build();
```

```java
WorkerFactoryOptions factoryOptions =
    WorkerFactoryOptions.newBuilder()
        .setWorkerInterceptors(
            new OpenTracingWorkerInterceptor(JaegerUtils.getJaegerOptions(type)))
        .build();
WorkerFactory factory = WorkerFactory.newInstance(client, factoryOptions);
```

For more information, see the Temporal [OpenTracing module](https://github.com/temporalio/sdk-java/blob/main/contrib/temporal-opentracing/README.md).

### Context propagation over Nexus Operation calls

Nexus does not use the standard context propagator header structure.
Instead, it relies on a Temporal-agnostic protocol designed to connect arbitrary systems.
To propagate context over Nexus Operation calls, the context is serialized into a `Map<String, String>`.
This map is special as it will normalize all keys to lowercase.

Because Nexus uses this custom format, and because Nexus calls may involve external systems, the `ContextPropagator` interface doesn’t apply to Nexus headers.
Context must be explicitly propagated through interceptors, as shown in the [Nexus Context Propagation sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexuscontextpropagation).

## Log from a Workflow 

Logging enables you to record critical information during code execution.
Loggers create an audit trail and capture information about your Workflow's operation.
An appropriate logging level depends on your specific needs.
During development or troubleshooting, you might use debug or even trace.
In production, you might use info or warn to avoid excessive log volume.

You can find the log levels supported by `slf4j` in [their official documentation](https://www.slf4j.org/apidocs/org/slf4j/ext/XLogger.Level.html). The Temporal SDK core normally uses `WARN` as its default logging level.

To get a standard `slf4j` logger in your Workflow code, use the [`Workflow.getLogger`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html) method.

```java
private static final Logger logger = Workflow.getLogger(DynamicDslWorkflow.class);
```

Logs in replay mode are omitted unless the [`WorkerFactoryOptions.Builder.setEnableLoggingInReplay(boolean)`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerFactoryOptions.Builder.html#setEnableLoggingInReplay(boolean)) method is set to true.

### How to provide a custom logger 

Use a custom logger for logging.

To set a custom logger, supply your own logging implementation and configuration details the same way you would in any other Java application.

## Visibility APIs 

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### How to use Search Attributes 

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI.
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `upsertTypedSearchAttributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `DescribeWorkflow`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [In the Temporal CLI](/cli/command-reference/workflow#list).
  - In code by calling `ListWorkflowExecutions`.

### How to set custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

When starting a Workflow Execution with your Client, include the Custom Search Attribute in the options using `WorkflowOptions.newBuilder().setTypedSearchAttributes()`:

```java
    // In a shared constants file, so all files have access

    public static final SearchAttributeKey<Boolean> IS_ORDER_FAILED = SearchAttributeKey.forBoolean("isOrderFailed");
...
    // In main
    WorkflowOptions options = WorkflowOptions.newBuilder()
        .setWorkflowId(workflowID)
        .setTaskQueue(Constants.TASK_QUEUE_NAME)
        .setTypedSearchAttributes(generateSearchAttributes())
        .build();

    PizzaWorkflow workflow = client.newWorkflowStub(PizzaWorkflow.class, options);
...

    // Further down in the file
    private static SearchAttributes generateSearchAttributes(){
        return SearchAttributes.newBuilder().set(Constants.IS_ORDER_FAILED, false).build();
    }
```

Each `SearchAttribute` object represents a custom attribute name, and the value is a [`SearchAttributeKey`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/SearchAttributeKey.html#forBoolean(java.lang.String)) representing a specific type. Currently the following types are supported:

- Boolean
- Double
- Long
- KeyWord
- KeyWordList
- Text

In this example `isOrderFailed` is set as a Search Attribute. This attribute is
useful for querying Workflows based on the success/failure of customer orders.

### How to upsert Search Attributes 

Within the Workflow code, you can dynamically add or update Search Attributes using [`upsertTypedSearchAttributes`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#upsertTypedSearchAttributes(io.temporal.common.SearchAttributeUpdate...)).
This method is particularly useful for Workflows whose attributes need to change based on internal logic or external events.

```java
  import io.temporal.workflow.Workflow;

  ...

  // Existing Workflow Logic

  Map<String, Object> searchAttribute = new HashMap<>();

  Distance distance;
  try {
    distance = activities.getDistance(address);
    searchAttribute.put("isOrderFailed", false);
    Workflow.upsertTypedSearchAttributes(Constants.IS_ORDER_FAILED.valueSet(false));
  } catch (NullPointerException e) {
    searchAttribute.put("isOrderFailed", true);
    Workflow.upsertTypedSearchAttributes(Constants.IS_ORDER_FAILED.valueSet(true));
    throw new NullPointerException("Unable to get distance");
  }
```

### How to remove a Search Attribute from a Workflow 

To remove a Search Attribute that was previously set, unset its value.

```java
    // In a shared constants file, so all files have access

    public static final SearchAttributeKey<Boolean> IS_ORDER_FAILED = SearchAttributeKey.forBoolean("isOrderFailed");

    ...

    Workflow.upsertTypedSearchAttributes(Constants.IS_ORDER_FAILED.valueUnset());
```

---

# Set up a local development environment with the Java SDK

Source: https://docs.temporal.io/develop/java/set-up-your-local-java

> Configure your local development environment to get started developing with Temporal

# Quickstart

Configure your local development environment to get started developing with Temporal.

## Install the Java JDK

Make sure you have the Java JDK installed. You can either download a copy directly from Oracle or select an OpenJDK
distribution from your preferred vendor.

You'll also need either Maven or Gradle installed.

**If you don't have Maven:** [Download](https://maven.apache.org/download.cgi) and
[install](https://maven.apache.org/install.html) from Apache.org, or use Homebrew: `brew install maven`.

**If you don't have Gradle:** [Download](https://gradle.org/install/) from Gradle.org, use
[IntelliJ IDEA](https://www.jetbrains.com/idea/) (bundled), or use Homebrew: `brew install gradle`.

```bash
java -version
```

## Create a Project

Now that you have your build tool installed, create a project to manage your dependencies and build your Temporal
application.

Choose your build tool to create the appropriate project structure. For Maven, this creates a standard project with the
necessary directories and a basic pom.xml file. For Gradle, this creates a project with build.gradle and the standard
Gradle directory structure.

**Maven**

```bash
mkdir temporal-java-project
```

```bash
cd temporal-java-project
```

```bash
mvn archetype:generate -DgroupId=helloworkflow -DartifactId=temporal-hello-world
-DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
```

```bash
cd temporal-hello-world
```

**Gradle**

```bash
mkdir temporal-hello-world
```

```bash
cd temporal-hello-world
```

```bash
gradle init --type java-application --project-name temporal-hello-world --package helloworkflow
```

## Add Temporal Java SDK Dependencies

Now add the Temporal SDK dependencies to your project configuration file.

For Maven, add the following dependencies to your `pom.xml` file. For Gradle, add the following lines to your
`build.gradle` file.

Next, you'll configure a local Temporal Service for development.

**Maven**

```xml
<dependencies>
  <!--
    Temporal dependencies needed to compile, build,
    test, and run Temporal's Java SDK
  -->

  <!--
    SDK
  -->

<dependency>
  <groupId>io.temporal</groupId>
  <artifactId>temporal-sdk</artifactId>
  <version>1.33.0</version>
</dependency>

  <dependency>
    <!--
      Testing
    -->
    <groupId>io.temporal</groupId>
    <artifactId>temporal-testing</artifactId>
    <version>1.33.0</version>
    <scope>test</scope>
  </dependency>
</dependencies>
```

**Gradle**

```groovy
plugins {
    id 'application'
}

repositories { mavenCentral() }

dependencies {
    implementation 'io.temporal:temporal-sdk:1.33.0'
    testImplementation 'io.temporal:temporal-testing:1.33.0'
}

// Define the main class for the application
application { mainClass = 'helloworkflow.Starter' }

// Helper tasks to run the worker and the starter
tasks.register('runWorker', JavaExec) {
    group = 'application'
    description = 'Run the Temporal worker'
    classpath = sourceSets.main.runtimeClasspath
    mainClass = 'helloworkflow.SayHelloWorker'
}

tasks.register('runStarter', JavaExec) {
    group = 'application'
    description = 'Run the workflow starter'
    classpath = sourceSets.main.runtimeClasspath
    mainClass = 'helloworkflow.Starter'
}
```

```bash
./gradlew build
```

## Install Temporal CLI and start the development server

The fastest way to get a development version of the Temporal Service running on your local machine is to use
[Temporal CLI](/cli).

Choose your operating system to install Temporal CLI:

**macOS**

Install the Temporal CLI using Homebrew:
```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH, for example:
```bash
sudo mv temporal /usr/local/bin
```

## Start the development server

Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory
database.

The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at
http://localhost:8233.

Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal
Service at any time by pressing CTRL+C.

Once you have everything installed, you're ready to build apps with Temporal on your local machine.

After installing, open a new Terminal. Keep this running in the background:
```bash
temporal server start-dev
```

#### Change the Web UI port
The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server:
```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and
Activity.

This test will confirm that:

- The Temporal Java SDK is properly installed
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

### 1. Create the Activity Interface

Create an Activity interface file (GreetActivities.java):

_Note that all files for this quickstart will be created under src/main/java/helloworkflow._

```java
package helloworkflow;

import io.temporal.activity.ActivityInterface;
import io.temporal.activity.ActivityMethod;

@ActivityInterface
public interface GreetActivities {

    @ActivityMethod
    String greet(String name);

}
```

An Activity is a method that executes a single, well-defined action (either short or long running), which often involve
interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling
an API, which are prone to failure. If an Activity fails, Temporal automatically retries it based on your configuration.

You define Activities in Java as an annotated interface, and its implementation.

### 2. Create the Activity Implementation

Create an Activity implementation file (GreetActivitiesImpl.java):

```java
package helloworkflow;

public class GreetActivitiesImpl implements GreetActivities {

    @Override
    public String greet(String name) {
      return "Hello " + name;
    }

}
```

### 3. Create the Workflow

Create a Workflow file (SayHelloWorkflow.java):

```java
package helloworkflow;

import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;

@WorkflowInterface
public interface SayHelloWorkflow {

    @WorkflowMethod
    String sayHello(String name);

}
```

Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run and
keep running for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will
automatically recreate its pre-failure state so it can continue right where it left off.

You define Workflows in Java as an annotated interface, and its implementation.

### 4. Create the Workflow Implementation

Create a Workflow implementation file (SayHelloWorkflowImpl.java):

```java
package helloworkflow;

import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.Workflow;

import java.time.Duration;

public class SayHelloWorkflowImpl implements SayHelloWorkflow {

    private final GreetActivities activities = Workflow.newActivityStub(
      GreetActivities.class,
      ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(5))
        .build()
    );

    @Override
    public String sayHello(String name) {
      return activities.greet(name);
    }

}
```

### 5. Create and Run the Worker

Create a Worker file (SayHelloWorker.java):

```java
package helloworkflow;

import io.temporal.client.WorkflowClient;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;

public class SayHelloWorker {

    public static void main(String[] args) {

      WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
      WorkflowClient client = WorkflowClient.newInstance(service);
      WorkerFactory factory = WorkerFactory.newInstance(client);

      Worker worker = factory.newWorker("my-task-queue");
      worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class);
      worker.registerActivitiesImplementations(new GreetActivitiesImpl());

      System.out.println("Starting SayHelloWorker for task queue 'my-task-queue'...");

      factory.start();

    }

}
```

With your Activity and Workflow defined, you need a Worker to execute them.

Open a new terminal and run the Worker:

**Maven**

```bash
cd temporal-hello-world
mvn compile exec:java -Dexec.mainClass="helloworkflow.SayHelloWorker"
```

**Gradle**

```bash
./gradlew runWorker
```

A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues a Workflow
or Activity task from the Task Queue, it then executes that task.

Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your
Workflows and Activities. For more information on Workers, see
[Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers).

### 6. Execute the Workflow

Now that your Worker is running, it's time to start a Workflow Execution.

This final step will validate that everything is working correctly with your file labeled `Starter.java`.

Create a separate file called `Starter.java`:

```java
package helloworkflow;

import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;

public class Starter {
    public static void main(String[] args) {
        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service);

        SayHelloWorkflow workflow = client.newWorkflowStub(
            SayHelloWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue("my-task-queue")
                .setWorkflowId("say-hello-workflow-id")
                .build()
        );

        String result = workflow.sayHello("Temporal");
        System.out.println("Workflow result: " + result);

        service.shutdown();
        service.awaitTermination(10, java.util.concurrent.TimeUnit.SECONDS);
    }
}
```

While your worker is still running, open a new terminal and run:

**Maven**

```bash
cd temporal-hello-world
mvn compile exec:java -Dexec.mainClass="helloworkflow.Starter"
```

**Gradle**

```bash
./gradlew runStarter
```

### Verify success

If everything is working correctly, you should see:

- Worker processing the workflow and activity
- Output: `Workflow result: Hello Temporal`
- Workflow Execution details in the [Temporal Web UI](http://localhost:8233)

- [Run your first Temporal Application](https://learn.temporal.io/getting_started/java/first_program_in_java/): Create a basic Workflow and run it with the Temporal Java SDK

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Worker Versioning (legacy) - Java SDK

Source: https://docs.temporal.io/develop/java/worker-versioning-legacy

> Learn the Java SDK's outdated Worker Versioning APIs.

## How to use Worker Versioning in Java (Deprecated) 

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>
> See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information.
>

A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp.
To use Worker Versioning, you need to pass a Build ID to your Java Worker and opt in to Worker Versioning.

### Assign a Build ID to your Worker and opt in to Worker Versioning

You should understand assignment rules before completing this step.
See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information.

To enable Worker Versioning for your worker, assign the Build ID--perhaps from an environment variable--and turn it on.

```java
// ...
WorkerOptions workerOptions = WorkerOptions.newBuilder()
    .setBuildId(buildId)
    .setUseBuildIdForVersioning(true)
    // ...
    .build();
Worker w = workerFactory.newWorker("your_task_queue_name", workerOptions);
// ...
```

> **⚠️ Warning:**
>
> Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules.
>

### Specify versions for Activities, Child Workflows, and Continue-as-New

> **⚠️ Caution:**
>
> Java support for this feature is under construction!
>

By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the workflow that created them if they are also configured to run on the same Task Queue.
When configured to run on a separate Task Queue, they will default to using the current assignment rules.

If you want to override this behavior, you can specify your intent via the `setVersioningIntent` method on the `ActivityOptions`, `ChildWorkflowOptions`, or `ContinueAsNewOptions` objects.

For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent:

```java
// ...
private final MyActivity activity =
    Workflow.newActivityStub(
        MyActivity.class,
        ActivityOptions.newBuilder()
          .setScheduleToCloseTimeout(Duration.ofSeconds(10))
          .setVersioningIntent(VersioningIntent.VERSIONING_INTENT_USE_ASSIGNMENT_RULES)
          // ...other options
          .build()
    );
// ...
```

### Tell the Task Queue about your Worker's Build ID (Deprecated)

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID.
You might want to do this as part of your CI deployment process.

```java
// ...
workflowClient.updateWorkerBuildIdCompatability(
    "your_task_queue_name", BuildIdOperation.newIdInNewDefaultSet("deadbeef"));
```

This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue.
New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers.

If, instead, you want to add the Build ID to an existing compatible set, you can do this:

```java
// ...
workflowClient.updateWorkerBuildIdCompatability(
    "your_task_queue_name", BuildIdOperation.newCompatibleVersion("deadbeef", "some-existing-build-id"));
```

This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set.

You can also promote an existing Build ID in a set to be the default for that set:

```java
// ...
workflowClient.updateWorkerBuildIdCompatability(
    "your_task_queue_name", BuildIdOperation.promoteBuildIdWithinSet("deadbeef"));
```

You can also promote an entire set to become the default set for the queue. New Workflows will start using that set's default.

```java
// ...
workflowClient.updateWorkerBuildIdCompatability(
    "your_task_queue_name", BuildIdOperation.promoteSetByBuildId("deadbeef"));
```

---

# Workers - Java SDK

Source: https://docs.temporal.io/develop/java/workers

> This section covers Workers with the Java SDK

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Workers

- [Run a Worker](/develop/java/workers/run-worker-process)
- [Interceptors](/develop/java/workers/interceptors)
- [Serverless Workers](/develop/java/workers/serverless-workers)

---

# Interceptors - Java SDK

Source: https://docs.temporal.io/develop/java/workers/interceptors

Interceptors wrap inbound and outbound Temporal calls so you can apply shared behavior such as tracing, logging, or
authorization. See [Interceptors](/encyclopedia/interceptors) for the inbound vs outbound model.

Register them on the Client (`WorkflowClientOptions.Builder.setInterceptors`) or the Worker Factory
(`WorkerFactoryOptions.Builder.setWorkerInterceptors`). Full API:
[`io.temporal.common.interceptors`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/interceptors/package-summary.html).

Several interceptor APIs are marked `@Experimental`.

## Implement a Client interceptor

Extend `WorkflowClientInterceptorBase` and override `workflowClientCallsInterceptor` to return a
`WorkflowClientCallsInterceptor` that wraps `next`. Override only the calls you want to intercept; forward the rest to
`next` (or to `super`, if you extend `WorkflowClientCallsInterceptorBase`).

```java
public class TracingClientInterceptor extends WorkflowClientInterceptorBase {
  @Override
  public WorkflowClientCallsInterceptor workflowClientCallsInterceptor(
      WorkflowClientCallsInterceptor next) {
    return new WorkflowClientCallsInterceptorBase(next) {
      @Override
      public WorkflowStartOutput start(WorkflowStartInput input) {
        System.out.println("starting workflow " + input.getWorkflowId());
        return super.start(input);
      }
    };
  }
}
```

Register the interceptor when you build the Client:

```java
WorkflowClientOptions clientOptions =
    WorkflowClientOptions.newBuilder().setInterceptors(new TracingClientInterceptor()).build();
WorkflowClient client = WorkflowClient.newInstance(service, clientOptions);
```

## Implement a Worker interceptor

Extend `WorkerInterceptorBase` and override `interceptWorkflow`, `interceptActivity`, or both, for the inbound call
chains you need. Each inbound interceptor must forward every override to `next`. To also intercept outbound calls made
from inside the Workflow, wrap the `WorkflowOutboundCallsInterceptor` passed to `init` and forward that wrapper to
`next.init(...)`:

```java
public class TracingWorkerInterceptor extends WorkerInterceptorBase {
  @Override
  public WorkflowInboundCallsInterceptor interceptWorkflow(WorkflowInboundCallsInterceptor next) {
    return new WorkflowInboundCallsInterceptorBase(next) {
      @Override
      public void init(WorkflowOutboundCallsInterceptor outboundCalls) {
        next.init(
            new WorkflowOutboundCallsInterceptorBase(outboundCalls) {
              @Override
              public <R> ActivityOutput<R> executeActivity(ActivityInput<R> input) {
                Workflow.getLogger(TracingWorkerInterceptor.class)
                    .info("starting activity " + input.getActivityName());
                return super.executeActivity(input);
              }
            });
      }
    };
  }
}
```

Register the interceptor on the Worker Factory:

```java
WorkerFactoryOptions factoryOptions =
    WorkerFactoryOptions.newBuilder().setWorkerInterceptors(new TracingWorkerInterceptor()).build();
WorkerFactory factory = WorkerFactory.newInstance(client, factoryOptions);
```

> **⚠️ Warning:**
> Workflow interceptors and replay
>
> Workflow interceptor methods also run during [replay](/develop/java/best-practices/testing-suite#replay). Use replay-safe
> APIs for logging, randomness, and time.
>

## Related

- [Observability - Tracing](/develop/java/platform/observability#tracing)
- [Error handling - Worker Interceptor](/develop/java/best-practices/error-handling#centralize-with-interceptor)
- [Spring Boot - Interceptors](/develop/java/integrations/spring-boot-integration#interceptors)
- [Plugins - Interceptors](/develop/plugins-guide#interceptors)

---

# Run a Worker - Java SDK

Source: https://docs.temporal.io/develop/java/workers/run-worker-process

> Create and run a Temporal Worker using the Java SDK.

This page covers long-lived Workers that you host and run as persistent processes.
For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/java/workers/serverless-workers).

## Create and run a Worker 

Create a [`WorkerFactory`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerFactory.html) from a [Temporal Client](/develop/java/client/temporal-client), then call `newWorker()` with the Task Queue to poll.
Register the Workflow and Activity types the Worker can execute, then call `start()` on the factory.

<!--SNIPSTART java-create-worker-->
[features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java)
```java
WorkerFactory factory = WorkerFactory.newInstance(client);

Worker worker = factory.newWorker("my-task-queue");
worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class);
worker.registerActivitiesImplementations(new GreetingActivitiesImpl());

factory.start();
```
<!--SNIPEND-->

`start()` starts every Worker the factory created, so one factory can run several Workers in a single process.
The call returns immediately and the Workers poll on background threads, so the process has to stay alive.

## Register Workflows and Activities 

All Workers polling the same Task Queue must register the same Workflow Types and Activity Types.
A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue.
A Worker that receives a Task for a type it did not register fails that Task.

Register Workflows by class with `registerWorkflowImplementationTypes()`.
The Worker creates a new instance for each Workflow Execution, so a Workflow Type can be registered only once per Worker.
Registering two implementations of the same type throws at registration time.

Register Activities by instance with `registerActivitiesImplementations()`.
One instance serves every Workflow Execution that calls it, so the implementation must be thread-safe.
Pass dependencies such as database clients through the constructor.

```java
worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class, OrderWorkflowImpl.class);
worker.registerActivitiesImplementations(new GreetingActivitiesImpl(databaseClient));
```

A Worker can register one implementation of [`DynamicWorkflow`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/DynamicWorkflow.html) and one of `DynamicActivity` alongside any number of type-specific implementations.
The dynamic implementation handles Workflow and Activity Types that no registered type matches.

## Connect to Temporal Cloud 

To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials.
See [Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud) for setup instructions.

## Configure Worker options 

Pass [`WorkerOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.html) to `newWorker()` to set concurrency limits, poller counts, and rate limits for a single Worker.
Pass [`WorkerFactoryOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerFactoryOptions.html) to `newInstance()` for settings shared by every Worker in the process, such as the Workflow cache size.
The defaults work for most cases.

To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference).

## Run a versioned Worker 

Set a Worker Deployment Version and enable versioning in `WorkerOptions`, then set a versioning behavior on each Workflow.

<!--SNIPSTART java-versioned-worker-->
[features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java)
```java
WorkerOptions options =
    WorkerOptions.newBuilder()
        .setDeploymentOptions(
            WorkerDeploymentOptions.newBuilder()
                .setVersion(new WorkerDeploymentVersion("my-app", "1.0"))
                .setUseVersioning(true)
                .build())
        .build();

Worker worker = factory.newWorker("my-task-queue", options);
worker.registerWorkflowImplementationTypes(VersionedGreetingWorkflowImpl.class);
worker.registerActivitiesImplementations(new GreetingActivitiesImpl());
```
<!--SNIPEND-->

Annotate the Workflow method with `@WorkflowVersioningBehavior(VersioningBehavior.PINNED)` or `AUTO_UPGRADE`, or set a default for the whole Worker with `setDefaultVersioningBehavior()`.
The annotation belongs on the implementation, not the interface.

A versioning behavior applies only to a Worker that has versioning enabled.
If a Workflow declares one and its Worker does not enable versioning, the server rejects the Workflow Task and the Task retries instead of failing outright.

See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out.

## Shut down a Worker 

Call `shutdown()` on the factory to stop polling for new Tasks, then `awaitTermination()` to give in-flight Tasks time to finish.

<!--SNIPSTART java-worker-graceful-shutdown-->
[features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java)
```java
factory.shutdown();
factory.awaitTermination(30, TimeUnit.SECONDS);
```
<!--SNIPEND-->

Both calls apply to every Worker the factory created.
See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities.

### Heartbeat during shutdown 

By default, after you call `shutdown()`, an Activity that calls `Activity.getExecutionContext().heartbeat()` doesn't send a Heartbeat.
The call throws `ActivityWorkerShutdownException` instead.
An Activity can catch this exception to detect that the Worker is shutting down.
An Activity that ignores the exception keeps running, but the Temporal Service stops receiving its Heartbeats and can time it out with a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) before it finishes.

To let Activities keep sending Heartbeats while the Worker drains, set `setAllowActivityHeartbeatDuringShutdown(true)` on the Worker options.
This matches the behavior of the other Temporal SDKs.

<!--SNIPSTART java-worker-heartbeat-during-shutdown-->
[java-sandbox/worker-shutdown/src/main/java/io/temporal/docs/workershutdown/HeartbeatDuringShutdownExamples.java](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/java-sandbox/worker-shutdown/src/main/java/io/temporal/docs/workershutdown/HeartbeatDuringShutdownExamples.java)
```java
WorkerOptions options =
    WorkerOptions.newBuilder().setAllowActivityHeartbeatDuringShutdown(true).build();
Worker worker = factory.newWorker("your-task-queue", options);
```
<!--SNIPEND-->

With this option enabled, `heartbeat()` no longer throws `ActivityWorkerShutdownException`, so an Activity can't use that exception to detect shutdown.
Each Activity must finish within the timeout you pass to `awaitTermination()`, or use another signal to detect shutdown.

The option has no effect on `shutdownNow()`, which attempts to stop running Activities by interrupting their threads. An Activity that ignores the interrupt may keep running.
`setAllowActivityHeartbeatDuringShutdown` is experimental and requires Java SDK v1.36.0 or later.

---

# Serverless Workers - Java SDK

Source: https://docs.temporal.io/develop/java/workers/serverless-workers

> Write Temporal Workers that run on serverless compute using the Java SDK.

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview.

Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes.
Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done.

For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers).
For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Supported providers

- [**AWS Lambda**](/develop/java/workers/serverless-workers/aws-lambda) - Use the `temporal-aws-lambda` contrib module to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, observability, and the invocation lifecycle.
- [**GCP Cloud Run**](/develop/java/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, container packaging, and handling scale-in.

---

# Serverless Workers on AWS Lambda - Java SDK

Source: https://docs.temporal.io/develop/java/workers/serverless-workers/aws-lambda

> Write a Temporal Worker that runs on AWS Lambda using the Java SDK temporal-aws-lambda contrib module.

> **Public Preview**

The `temporal-aws-lambda` contrib module lets you run a Temporal Serverless Worker on AWS Lambda.
Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive.
Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline.
You register Workflows and Activities the same way you would with a standard Worker.

For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda).

## Create and run a Worker in Lambda 

Create a class that implements AWS Lambda's `RequestHandler` interface and delegates to `LambdaWorker.define` in a static field.
Pass a `WorkerDeploymentVersion` and a configure callback that registers your Workflows and Activities.
Assign the handler to a static field so it is created once during Lambda cold start and reused across invocations.

```java {12-19}
package com.example;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import io.temporal.aws.lambda.LambdaWorker;
import io.temporal.common.WorkerDeploymentVersion;

/** AWS Lambda entry point for the Temporal Worker. */
public class LambdaFunction implements RequestHandler<Object, Void> {

  private static final RequestHandler<Object, Void> WORKER =
      LambdaWorker.define(
          new WorkerDeploymentVersion("my-app", "build-1"),
          builder -> {
            builder.setTaskQueue("my-task-queue");
            builder.registerWorkflowImplementationTypes(SampleWorkflowImpl.class);
            builder.registerActivitiesImplementations(new GreetingActivitiesImpl());
          });

  @Override
  public Void handleRequest(Object input, Context context) {
    return WORKER.handleRequest(input, context);
  }
}
```

For a complete project, see the [Lambda Worker sample](https://github.com/temporalio/samples-java/tree/main/lambda-worker).

The `WorkerDeploymentVersion` is required.
Worker Deployment Versioning is always enabled for Serverless Workers.
Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or `Pinned`.
Set it per-Workflow with the `@WorkflowVersioningBehavior` annotation on the Workflow method, or set a worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`.

[lambda-worker/worker/src/main/java/io/temporal/samples/lambdaworker/SampleWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/lambda-worker/worker/src/main/java/io/temporal/samples/lambdaworker/SampleWorkflowImpl.java)
```java {21}
package io.temporal.samples.lambdaworker;

import io.temporal.activity.ActivityOptions;
import io.temporal.common.VersioningBehavior;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowVersioningBehavior;
import java.time.Duration;
import org.slf4j.Logger;

/** Workflow implementation that executes a greeting Activity. */
public class SampleWorkflowImpl implements SampleWorkflow {

  private static final Logger logger = Workflow.getLogger(SampleWorkflowImpl.class);

  private final GreetingActivities activities =
      Workflow.newActivityStub(
          GreetingActivities.class,
          ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofSeconds(10)).build());

  @Override
  @WorkflowVersioningBehavior(VersioningBehavior.PINNED)
  public String getGreeting(String name) {
    logger.info("SampleWorkflow started for {}", name);
    String result = activities.createGreeting(name);
    logger.info("SampleWorkflow completed with {}", result);
    return result;
  }
}
```

The configure callback receives a `LambdaWorkerOptions.Builder` with the same [registration methods as a standard Worker](/develop/java/workers/run-worker-process#register-types): `registerWorkflowImplementationTypes`, `registerActivitiesImplementations`, and `registerDynamicWorkflowImplementationType`.

If you need to assemble options outside the callback, call `LambdaWorkerOptions.newBuilderFromEnvironment()`, configure the builder, and pass the built options to `LambdaWorker.newHandler(...)`.

## Configure the Temporal connection 

The `temporal-aws-lambda` module automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details.

Compared with long-lived Workers, the location of the config file is resolved differently, in the following order:

1. `TEMPORAL_CONFIG_FILE` environment variable, if set.
2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`).
3. `temporal.toml` in the current working directory.

The file is optional. If absent, only environment variables are used.

Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options.

## Adjust Worker defaults for Lambda 

The `temporal-aws-lambda` module applies conservative defaults suited to short-lived Lambda invocations.
These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment.

| Setting | Lambda default |
|---|---|
| `MaxConcurrentActivityExecutionSize` | 2 |
| `MaxConcurrentWorkflowTaskExecutionSize` | 10 |
| `MaxConcurrentLocalActivityExecutionSize` | 2 |
| `MaxConcurrentNexusExecutionSize` | 5 |
| `MaxConcurrentWorkflowTaskPollers` | 2 |
| `MaxConcurrentActivityTaskPollers` | 1 |
| `MaxConcurrentNexusTaskPollers` | 1 |
| `WorkflowCacheSize` | 30 |
| `MaxWorkflowThreadCount` | 30 |
| `GracefulShutdownTimeout` | 5 seconds |
| `ShutdownDeadlineBuffer` | 7 seconds |

`ShutdownDeadlineBuffer` is specific to the `temporal-aws-lambda` module.
It controls the full shutdown window reserved at the end of the Lambda invocation, including graceful shutdown time, shutdown hooks, and service stub cleanup.
The default is `GracefulShutdownTimeout` (5s) + 2s.

If you change `GracefulShutdownTimeout` without explicitly setting `ShutdownDeadlineBuffer`, the buffer is recomputed as `GracefulShutdownTimeout` + 2s.
If you explicitly set `ShutdownDeadlineBuffer`, it must be greater than or equal to `GracefulShutdownTimeout`.

If your Worker handles long-running Activities, increase `GracefulShutdownTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together.
For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities).

## Add observability with OpenTelemetry 

The `OtelLambdaWorkerConfigurationHelper` class provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer.
With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions.
The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch.

The underlying metrics and traces are the same ones the Java SDK emits in any environment.
For general observability concepts and the full list of available metrics, see [Observability - Java SDK](/develop/java/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).

> **⚠️ Caution:**
>
> Traces are currently produced through a compatibility shim, not native OpenTelemetry support. The trace structure will change in a future release. Metrics are not affected.
>

```java
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import io.temporal.aws.lambda.LambdaWorker;
import io.temporal.aws.lambda.OtelLambdaWorkerConfigurationHelper;
import io.temporal.common.WorkerDeploymentVersion;

public final class Handler implements RequestHandler<Object, Void> {
  private static final RequestHandler<Object, Void> WORKER =
      LambdaWorker.define(
          new WorkerDeploymentVersion("my-app", "build-1"),
          builder -> {
            OtelLambdaWorkerConfigurationHelper.configure(builder);
            builder.setTaskQueue("serverless-task-queue-1");
            builder.registerWorkflowImplementationTypes(SampleWorkflowImpl.class);
            builder.registerActivitiesImplementations(new SampleActivitiesImpl());
          });

  @Override
  public Void handleRequest(Object input, Context context) {
    return WORKER.handleRequest(input, context);
  }
}
```

`OtelLambdaWorkerConfigurationHelper.configure` configures OpenTelemetry with OTLP trace and metric exporters, uses AWS X-Ray-compatible trace ID generation, installs an OpenTelemetry-backed metrics scope, and registers per-invocation flush hooks.
By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint.
The endpoint can be overridden with the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable.

To collect this telemetry, attach the [ADOT Collector layer](https://aws-otel.github.io/docs/getting-started/lambda) to your Lambda function.
Java does not need a language-specific ADOT layer because the OTel SDK is included as a dependency of the module.

The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline.
You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines.
Bundle the following `otel-collector-config.yaml` in your Lambda deployment package:

[lambda-worker/otel-collector-config.template.yaml](https://github.com/temporalio/samples-java/blob/main/lambda-worker/otel-collector-config.template.yaml)
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "localhost:4317"
      http:
        endpoint: "localhost:4318"

exporters:
  debug:
  awsxray:
    region: ${env:AWS_REGION}
  awsemf:
    namespace: TemporalWorkerMetrics
    log_group_name: /aws/lambda/${env:AWS_LAMBDA_FUNCTION_NAME}
    region: ${env:AWS_REGION}
    dimension_rollup_option: NoDimensionRollup
    resource_to_telemetry_conversion:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray, debug]
    metrics:
      receivers: [otlp]
      exporters: [awsemf]
  telemetry:
    logs:
      level: info
    metrics:
      address: localhost:8888
```

Set the following environment variable on the Lambda function to point the Collector at the bundled config:

- `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml`

Enable X-Ray active tracing on the Lambda function:

```bash
aws lambda update-function-configuration \
  --function-name <your-function-name> \
  --tracing-config Mode=Active
```

The Lambda execution role must have permissions to write to X-Ray and CloudWatch.
Add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions to the execution role.
Without these permissions, the Collector fails silently and no telemetry appears.

If you only need metrics or tracing, use `OtelLambdaWorkerConfigurationHelper.configureMetrics`, `OtelLambdaWorkerConfigurationHelper.configureTracing`, or `OtelLambdaWorkerConfigurationHelper.configureFlushHook` individually.
To use an application-owned OpenTelemetry provider, pass a customizer to the two-argument `configure` overload: `OtelLambdaWorkerConfigurationHelper.configure(builder, b -> b.setOpenTelemetry(openTelemetry))`.
`setOpenTelemetry` is defined on the helper's own builder, not on `LambdaWorkerOptions.Builder`.
In that path, no exporters are created and the helper only installs the metrics scope, interceptors, and per-invocation flush hook.

---

# Serverless Workers on GCP Cloud Run - Java SDK

Source: https://docs.temporal.io/develop/java/workers/serverless-workers/cloud-run

> Run a Temporal Worker on a GCP Cloud Run worker pool using the Java SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Java Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).

## Create a versioned Worker 

Build the Worker as you would any long-running Java Worker, then set `WorkerDeploymentOptions` on [`WorkerOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.html) to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

```java
package example;

import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.common.VersioningBehavior;
import io.temporal.common.WorkerDeploymentVersion;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerDeploymentOptions;
import io.temporal.worker.WorkerFactory;
import io.temporal.worker.WorkerOptions;

String apiKey = System.getenv("TEMPORAL_API_KEY");

WorkflowServiceStubs service =
    WorkflowServiceStubs.newServiceStubs(
        WorkflowServiceStubsOptions.newBuilder()
            .setTarget(System.getenv("TEMPORAL_ADDRESS"))
            .setEnableHttps(true)
            .addApiKey(() -> apiKey)
            .build());

WorkflowClient client =
    WorkflowClient.newInstance(
        service,
        WorkflowClientOptions.newBuilder()
            .setNamespace(System.getenv("TEMPORAL_NAMESPACE"))
            .build());

WorkerFactory factory = WorkerFactory.newInstance(client);

Worker worker =
    factory.newWorker(
        System.getenv("TEMPORAL_TASK_QUEUE"),
        WorkerOptions.newBuilder()
            .setDeploymentOptions(
                WorkerDeploymentOptions.newBuilder()
                    .setUseVersioning(true)
                    .setVersion(new WorkerDeploymentVersion("my-app", "build-1"))
                    .setDefaultVersioningBehavior(VersioningBehavior.PINNED)
                    .build())
            .build());

worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class);
worker.registerActivitiesImplementations(new GreetingActivitiesImpl());

factory.start();
```

The two arguments to `WorkerDeploymentVersion` are the deployment name and the build ID, and together they identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage.

Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`.
Setting `setDefaultVersioningBehavior` as shown above covers every Workflow on the Worker.
To set the behavior per Workflow instead, annotate the Workflow method with `@WorkflowVersioningBehavior`:

```java
import io.temporal.common.VersioningBehavior;
import io.temporal.workflow.WorkflowVersioningBehavior;

public class GreetingWorkflowImpl implements GreetingWorkflow {
  @Override
  @WorkflowVersioningBehavior(VersioningBehavior.PINNED)
  public String run(String name) {
    // ...
  }
}
```

For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/java/workers/run-worker-process).

## Configure the Temporal connection 

Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext.
The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace.

`addApiKey` takes a supplier, which the SDK calls on each request. Rotate the key by returning a new value from the supplier instead of restarting the Worker.

For TLS client certificates instead of an API key, see [Connect to Temporal Cloud](/develop/java/client/temporal-client).

## Package the Worker image 

Cloud Run runs one JVM per instance, so give the JVM a heap sized to the instance rather than to the host.
Java reads the container's memory limit and defaults the maximum heap to a quarter of it, which leaves most of a small instance unused.
Set `-XX:MaxRAMPercentage` to raise that share:

```dockerfile
CMD ["java", "-XX:MaxRAMPercentage=75", "-jar", "/app/worker.jar"]
```

A Cloud Run Worker Pool defaults to 512 MiB per instance. Raise `--memory` when you create the pool if your Worker needs more.

## Keep Activities safe across scale-in 

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing.
An instance running a long Activity can be stopped mid-execution.

Use [Activity Heartbeats](/develop/java/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over:

```java
public class GreetingActivitiesImpl implements GreetingActivities {
  @Override
  public String process(List<String> items) {
    for (int i = 0; i < items.size(); i++) {
      Activity.getExecutionContext().heartbeat(i);
      // ... process items.get(i)
    }
    return "done";
  }
}
```

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Configure OpenTelemetry 

Run an OpenTelemetry Collector as a sidecar in the Worker Pool. The Cloud Run OpenTelemetry plugin exports metrics and traces by OTLP gRPC to the Collector at `localhost:4317`. It configures the endpoint and service name from Cloud Run defaults.

Create both Cloud Run plugins and register them on `WorkflowServiceStubsOptions` before you create the Client:

<!--SNIPSTART java-cloud-run-worker-->
[gcp/cloud-run/src/main/java/io/temporal/samples/gcp/cloudrun/CloudRunWorker.java](https://github.com/temporalio/samples-java/blob/main/gcp/cloud-run/src/main/java/io/temporal/samples/gcp/cloudrun/CloudRunWorker.java)
```java
ClientConfigProfile profile = ClientConfigProfile.load();
CloudRunOpenTelemetryPlugin otelPlugin = CloudRunOpenTelemetryPlugin.newBuilder().build();
CloudRunIdPlugin idPlugin = new CloudRunIdPlugin();

WorkflowServiceStubsOptions serviceOptions =
    WorkflowServiceStubsOptions.newBuilder(profile.toWorkflowServiceStubsOptions())
        .setPlugins(otelPlugin, idPlugin)
        .build();
WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(serviceOptions);
```
<!--SNIPEND-->

Configure the Collector sidecar to receive OTLP gRPC on `localhost:4317`, export traces to Google Cloud, and export metrics to Google Managed Service for Prometheus:

<!--SNIPSTART java-cloud-run-collector-config-->
[gcp/cloud-run/collector-config.yaml](https://github.com/temporalio/samples-java/blob/main/gcp/cloud-run/collector-config.yaml)
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  # Batch traces for throughput. Do not add this processor to the cumulative metrics pipeline:
  # a shutdown flush can otherwise be batched with a recent periodic export of the same series.
  batch/traces:
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s
  memory_limiter:
    # This is the collector's memory polling cadence, not the SDK metric export interval.
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20
  resource_detection:
    detectors: [gcp]
    timeout: 10s
  # Avoid collisions with labels that Google Managed Service for Prometheus adds.
  transform/collision:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["exported_location"], attributes["location"])
          - delete_key(attributes, "location")
          - set(attributes["exported_cluster"], attributes["cluster"])
          - delete_key(attributes, "cluster")
          - set(attributes["exported_namespace"], attributes["namespace"])
          - delete_key(attributes, "namespace")
          - set(attributes["exported_job"], attributes["job"])
          - delete_key(attributes, "job")
          - set(attributes["exported_instance"], attributes["instance"])
          - delete_key(attributes, "instance")
          - set(attributes["exported_project_id"], attributes["project_id"])
          - delete_key(attributes, "project_id")
  # The Telemetry API expects the Google Cloud project in gcp.project_id.
  transform/set_project_id:
    error_mode: ignore
    trace_statements:
      - set(resource.attributes["gcp.project_id"], resource.attributes["gcp.project.id"]) where resource.attributes["gcp.project.id"] != nil
      - set(resource.attributes["gcp.project_id"], resource.attributes["cloud.account.id"]) where resource.attributes["gcp.project_id"] == nil and resource.attributes["cloud.account.id"] != nil

exporters:
  googlemanagedprometheus:
  # Google Cloud's supported OTLP path for traces is the Telemetry API.
  otlp_grpc:
    endpoint: telemetry.googleapis.com:443
    compression: none
    balancer_name: pick_first
    auth:
      authenticator: googleclientauth

extensions:
  # Cloud Run container dependencies require a startup probe. This endpoint is also used for the
  # collector liveness probe in worker-pool.yaml.
  health_check:
    endpoint: 0.0.0.0:13133
  googleclientauth:

service:
  extensions:
    - health_check
    - googleclientauth
  pipelines:
    metrics/otlp:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection, transform/collision]
      exporters: [googlemanagedprometheus]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection, transform/set_project_id, batch/traces]
      exporters: [otlp_grpc]
  # Feed collector self-metrics back through the metrics pipeline.
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: grpc
                endpoint: http://localhost:4317
                insecure: true
```
<!--SNIPEND-->

On shutdown, stop the `WorkerFactory`, then run `otelPlugin.newFlushHook()` before the Cloud Run termination window ends. For a Collector configuration that exports traces to Google Cloud and metrics to Google Managed Service for Prometheus, see the [Java Cloud Run sample](https://github.com/temporalio/samples-java/tree/main/gcp/cloud-run).

## Set a Worker identity 

Use `CloudRunIdPlugin` to identify each Worker instance as `<instance-id>@<revision>`. Register it on
`WorkflowServiceStubsOptions` before creating the service stubs. Clients and Workers created from those service stubs
inherit the identity:

```java
import io.temporal.client.WorkflowClient;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.gcp.cloudrun.id.CloudRunIdPlugin;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;

// ...
    ClientConfigProfile profile = ClientConfigProfile.load();
    WorkflowServiceStubsOptions serviceOptions =
        WorkflowServiceStubsOptions.newBuilder(profile.toWorkflowServiceStubsOptions())
            .setPlugins(new CloudRunIdPlugin())
            .build();
    WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(serviceOptions);
    WorkflowClient client = WorkflowClient.newInstance(service, profile.toWorkflowClientOptions());
    // ...
```

The [preceding example](#opentelemetry) registers `CloudRunIdPlugin` alongside the OpenTelemetry plugin.

The plugin requires Cloud Run metadata, so Client creation fails when the Worker runs outside Cloud Run. For a complete example, see the [Java Cloud Run sample](https://github.com/temporalio/samples-java/tree/main/gcp/cloud-run).

---

# Workflows - Java SDK

Source: https://docs.temporal.io/develop/java/workflows

> This section covers Workflows with the Java SDK

![Java SDK Banner](https://docs.temporal.io/img/assets/banner-java-temporal.png)

## Workflows

- [Workflow basics](/develop/java/workflows/basics)
- [Child Workflows](/develop/java/workflows/child-workflows)
- [Continue-As-New](/develop/java/workflows/continue-as-new)
- [Message passing](/develop/java/workflows/message-passing)
- [Cancellation](/develop/java/workflows/cancellation)
- [Timeouts](/develop/java/workflows/timeouts)
- [Schedules](/develop/java/workflows/schedules)
- [Timers](/develop/java/workflows/timers)
- [Side effects](/develop/java/workflows/side-effects)
- [Versioning](/develop/java/workflows/versioning)
- [Workflow Streams](/develop/java/workflows/workflow-streams)

---

# Workflow basics - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/basics

> This section explains how to implement Workflows with the Java SDK

## Develop a Workflow 

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal Java SDK programming model, a Workflow Definition comprises a Workflow interface annotated with `@WorkflowInterface` and a Workflow implementation that implements the Workflow interface.

The Workflow interface is a Java interface and is annotated with `@WorkflowInterface`.
Each Workflow interface must have only one method annotated with `@WorkflowMethod`.

```java
// Workflow interface
@WorkflowInterface
public interface YourWorkflow {

    @WorkflowMethod
    String yourWFMethod(Arguments args);
}
```

However, when using dynamic Workflows, do not specify a `@WorkflowMethod`, and implement the `DynamicWorkflow` directly in the Workflow implementation code.

The `@WorkflowMethod` identifies the method that is the starting point of the Workflow Execution.
The Workflow Execution completes when this method completes.

You can create [interface inheritance hierarchies](#interface-inheritance) to reuse components across other Workflow interfaces.
The interface inheritance approach does not apply to `@WorkflowMethod` annotations.

A Workflow implementation implements a Workflow interface.

```java
// Define the Workflow implementation which implements our getGreeting Workflow method.
public static class GreetingWorkflowImpl implements GreetingWorkflow {
  ...
}
```

To call Activities in your Workflow, call the Activity implementation.

Use `ExternalWorkflowStub` to start or send Signals from within a Workflow to other running Workflow Executions.

You can also invoke other Workflows as Child Workflows with `Workflow.newChildWorkflowStub()` or `Workflow.newUntypedChildWorkflowStub()` within a Workflow Definition.

## Workflow interface inheritance 

Workflow interfaces can form inheritance hierarchies.
It may be useful for creating reusable components across multiple
Workflow interfaces.
For example imagine a UI or CLI button that allows a `retryNow` Signal on any Workflow. To implement this feature you can redesign an interface like the following:

```java
public interface Retryable {
    @SignalMethod
    void retryNow();
}

@WorkflowInterface
public interface FileProcessingWorkflow extends Retryable {

    @WorkflowMethod
    String processFile(Arguments args);

    @QueryMethod(name="history")
    List<String> getHistory();

    @QueryMethod
    String getStatus();

    @SignalMethod
    void abandon();
}
```

Then some other Workflow interface can extend just `Retryable`, for example:

```java
@WorkflowInterface
public interface MediaProcessingWorkflow extends Retryable {

    @WorkflowMethod
    String processBlob(Arguments args);
}
```

Now if we have two running Workflows, one that implements the `FileProcessingWorkflow` interface and another that implements the `MediaProcessingWorkflow` interface, we can Signal to both using their common interface and knowing their WorkflowIds, for example:

```java
Retryable r1 = client.newWorkflowStub(Retryable.class, firstWorkflowId);
Retryable r2 = client.newWorkflowStub(Retryable.class, secondWorkflowId);
r1.retryNow();
r2.retryNow();
```

The same technique can be used to query Workflows using a base Workflow interface.

Note that this approach does not apply to `@WorkflowMethod` annotations, meaning that when using a base interface, it should not include any `@WorkflowMethod` methods.
To illustrate this, let's say that we define the following **invalid** code:

```java
// INVALID CODE!
public interface BaseWorkflow {
    @WorkflowMethod
    void retryNow();
}

@WorkflowInterface
public interface Workflow1 extends BaseWorkflow {}

@WorkflowInterface
public interface Workflow2 extends BaseWorkflow {}
```

Any attempt to register both implementations with the Worker will fail.
Let's say that we have:

```java
worker.registerWorkflowImplementationTypes(
    Workflow1Impl.class, Workflow2Impl.class);
```

This registration will fail with:

```text
java.lang.IllegalStateException: BaseWorkflow workflow type is already registered with the worker
```

## Define Workflow parameters 

Temporal Workflows may have any number of custom parameters.
However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow.
All Workflow Definition parameters must be serializable.
For more reasons, see [Workflow parameters](/workflow-definition#workflow-parameters).

A method annotated with `@WorkflowMethod` can have any number of parameters.

We recommend passing a single parameter that contains all the input fields to allow for adding fields in a backward-compatible manner.

Note that all inputs should be serializable by the default Jackson JSON Payload Converter.

You can create a custom object and pass it to the Workflow method, as shown in the following example.

```java
//...
@WorkflowInterface
public interface YourWorkflow {
    @WorkflowMethod
    String yourWFMethod(CustomObj customobj);
// ...
}
```

## Define Workflow return parameters 

Workflow return values must also be serializable.
Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported.
However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error.

Workflow method arguments and return values must be serializable and deserializable using the provided [`DataConverter`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html).

The `execute` method for `DynamicWorkflow` can return type Object.
Ensure that your Client can handle an Object type return or is able to convert the Object type response.

Related references:

- [Data Converter](/dataconversion)
- [Java DataConverter reference](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html)

## Customize your Workflow Type 

Workflows have a Type that is referred to as the Workflow name.

The following examples demonstrate how to set a custom name for your Workflow Type.

The Workflow Type defaults to the short name of the Workflow interface.
In the following example, the Workflow Type defaults to `NotifyUserAccounts`.

```java
  @WorkflowInterface

  public interface NotifyUserAccounts {
    @WorkflowMethod
    void notify(String[] accountIds);
}
```

To overwrite this default naming and assign a custom Workflow Type, use the `@WorkflowMethod` annotation with the `name` parameter.
In the following example, the Workflow Type is set to `your-workflow`.

```java
@WorkflowInterface

  public interface NotifyUserAccounts {
  @WorkflowMethod(name = "your-workflow")
  void notify(String[] accountIds);
  }
```

When you set the Workflow Type this way, the value of the `name` parameter does not have to start with an uppercase letter.

## Use Workflow constructors

Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. However, if you use the `@WorkflowInit` annotation on your constructor, you can give it the same [Workflow parameters](/develop/java/workflows/basics#workflow-parameters) as your `@WorkflowMethod`.

The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/java/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `@WorkflowMethod` method. That always happens, whether or not you use the `@WorkflowInit` annotation.

Here's an example. Notice that the constructor and `getGreeting` must have the same parameters:

```java
public class GreetingExample {
    @WorkflowInterface
    public interface GreetingWorkflow {
        @WorkflowMethod
        String getGreeting(String input);
    }

    public static class GreetingWorkflowImpl implements GreetingWorkflow {
        private final String nameWithTitle;
        private boolean titleHasBeenChecked;
        ...
        // Note the annotation is on a public constructor
        @WorkflowInit
        public GreetingWorkflowImpl(String input) {
          this.nameWithTitle = "Knight " + input;
          this.titleHasBeenChecked = false;
        }

        @Override
        public String getGreeting(String input) {
          Workflow.await(() -> titleHasBeenChecked)
          return "Hello " + nameWithTitle;
        }
    }
}
```

## Workflow logic requirements 

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with external (to the Workflow) application code.

When defining Workflows using the Temporal Java SDK, the Workflow code must be written to execute effectively once and to completion.

The following constraints apply when writing Workflow Definitions:

- Do not use mutable global variables in your Workflow implementations. This will ensure that multiple Workflow instances are fully isolated.
- A Workflow Definition must be deterministic. If you need to call non-deterministic functions (such as non-seeded random or `UUID.randomUUID()`) directly from the Workflow code, the Temporal SDK provides replay-safe replacements. See [Random numbers and UUIDs](#random-numbers-and-uuids).
  - For operations like calling external APIs, invoking LLMs, querying databases, or performing I/O, use Activities. Activities run outside Workflow replay and are retried reliably.
- Use Temporal-provided functions instead of that rely on system time. For example, use only `Workflow.currentTimeMillis()` to get the current time inside a Workflow.
- Use `Async.function` or `Async.procedure`, provided by the Temporal SDK, to execute code asynchronously instead of native Java `Thread` or any other multi-threaded classes like `ThreadPoolExecutor`.
- Use only the concurrency features provided by the Workflow class. Multi-threaded code inside a Workflow is executed one thread at a time and under a global lock, so there is no need for explicit synchronization.
  - Call `Workflow.sleep` instead of `Thread.sleep`.
  - Use `Promise` and `CompletablePromise` instead of `Future` and `CompletableFuture`.
  - Use `WorkflowQueue` instead of `BlockingQueue`.
- Use `Workflow.getVersion` when making any changes to the Workflow code to avoid deployment of updated Workflow code interfering with already-running Workflows.
- Do not access configuration APIs directly from a Workflow because changes in the configuration might affect a Workflow Execution path. Instead, pass it as an argument to a Workflow function or use an Activity to load it.
- Use `DynamicWorkflow` when you need a default Workflow that can handle all Workflow Types that are not registered with a Worker. A single implementation can implement a Workflow Type which by definition is dynamically loaded from some external source. All standard `WorkflowOptions` and determinism rules apply to Dynamic Workflow implementations.

The SDK provides replay-safe alternatives for common needs.

### Logging

Use [`Workflow.getLogger()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html)
instead of `System.out.println` or a logger you create yourself. The SDK logger skips log messages during replay to
avoid duplicates:

```java
public class MyWorkflowImpl implements MyWorkflow {
  private static final Logger logger = Workflow.getLogger(MyWorkflowImpl.class);

  @Override
  public String execute(String name) {
    logger.info("Starting workflow for {}", name);
    // ...
  }
}
```

For logger configuration, see [Observability: Log from a Workflow](/develop/java/platform/observability#logging).

### Random numbers and UUIDs

Use [`Workflow.newRandom()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html)
to get a `Random` instance seeded per Workflow Execution, and
[`Workflow.randomUUID()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html)
instead of `UUID.randomUUID()`:

```java
int value = Workflow.newRandom().nextInt(100);
UUID uniqueId = Workflow.randomUUID();
```

### Current time

Use [`Workflow.currentTimeMillis()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html)
instead of `System.currentTimeMillis()` or `Instant.now()`. It returns the time of the last Workflow Task, which is
consistent across replays:

```java
long currentTime = Workflow.currentTimeMillis();
```

To wait, use [`Workflow.sleep()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html)
instead of `Thread.sleep()`.

### Detecting replay (advanced)

Use [`WorkflowUnsafe.isReplaying()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/unsafe/WorkflowUnsafe.html)
to guard code that should only run on the first execution, such as emitting metrics or sending external notifications
from an Interceptor. `Workflow.isReplaying()` is deprecated in favor of this method.

> **⚠️ Caution:**
>
> Never use this to affect Workflow business logic. Branching on replay status breaks determinism.
>

```java
import io.temporal.workflow.unsafe.WorkflowUnsafe;

if (!WorkflowUnsafe.isReplaying()) {
  emitMetric("workflow_started", 1);
}
```

Java Workflow reference: [https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html)

---

# Interrupt a Workflow Execution - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/cancellation

You can interrupt a Workflow Execution in one of the following ways:

- [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution.
- [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution.

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the
Workflow is stuck and cannot be canceled normally.

## Cancel a Workflow Execution 

Canceling a Workflow provides a graceful way to stop Workflow Execution. This action resembles sending a `SIGTERM` to a
process.

- The system records a `WorkflowExecutionCancelRequested` event in the Event History.
- A Workflow Task gets scheduled to process the cancelation.
- The Workflow code can handle the cancelation and execute any cleanup logic.
- The system doesn't forcefully stop the Workflow.

To cancel a Workflow Execution in Java, use the
[cancel()](<https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#cancel()>)
function on the WorkflowStub.

```java
WorkflowStub workflowStub = WorkflowStub.fromTyped(workflow);
workflowStub.cancel();
```

## Cancellation scopes 

In the Java SDK, Workflows are represented internally by a tree of cancellation scopes, each with cancellation behaviors
you can specify. By default, everything runs in the "root" scope.

Scopes are created using the
[Workflow.newCancellationScope](<https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newCancellationScope(io.temporal.workflow.Functions.Proc1)>)
constructor

Cancellations are applied to cancellation scopes, which can encompass an entire Workflow or just part of one. Scopes can
be nested, and cancellation propagates from outer scopes to inner ones. A Workflow's method runs in the outermost scope.
Cancellations are handled by catching `CanceledFailure`s thrown by cancelable operations.

You can also use the following APIs:

- `CancellationScope.current()`: Get the current scope.
- `scope.cancel()`: Cancel all operations inside a `scope`.
- `scope.getCancellationRequest()`: A promise that resolves when a scope cancellation is requested, such as when
  Workflow code calls `cancel()` or the entire Workflow is cancelled by an external client.

When a `CancellationScope` is cancelled, it propagates cancellation in any child scopes and of any cancelable operations
created within it, such as the following:

- Activities
- Timers (created with the
  [sleep](<https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep(java.time.Duration)>)
  function)
- Child Workflows
- Nexus Operations

### Cancel with Workflow error

When you want to interrupt a Workflow Execution using an error, throw the [`DestroyWorkflowThreadError`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/sync/DestroyWorkflowThreadError.html). Make sure there isn't any code on that thread to catch this error or it could cause issues with the Temporal service.

More generally, Workflow code should only ever catch `Exception`, never `Throwable` or `Error` — see [Catch Exception, never Throwable or Error](/develop/java/best-practices/error-handling#catch-exception-not-throwable).

### Cancel an Activity from a Workflow 

Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat
Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate
Activity is executed, the Activity Execution should send Heartbeats and set a
[Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is
still working.

When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic
needs to be performed, it can be done in a `finally` clause or inside a caught cancel error. However, for the Activity
to appear canceled the exception needs to be re-raised.

> **📝 Note:**
>
> Unlike regular Activities, [Local Activities](/local-activity) currently do not support cancellation.
>

To cancel an Activity from a Workflow Execution, call the
[cancel()](<https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/CancellationScope.html#cancel()>)
method on the CancellationScope that the activity was started in.

```java
public class GreetingWorkflowImpl implements GreetingWorkflow {
    @Override
    public String getGreeting(String name) {
      List<Promise<String>> results = new ArrayList<>(greetings.length);

      /*
       * Create our CancellationScope. Within this scope we call the workflow activity
       * composeGreeting method asynchronously for each of our defined greetings in different
       * languages.
       */
      CancellationScope scope =
          Workflow.newCancellationScope(
              () -> {
                for (String greeting : greetings) {
                  results.add(Async.function(activities::composeGreeting, greeting, name));
                }
              });

      /*
       * Execute all activities within the CancellationScope. Note that this execution is
       * non-blocking as the code inside our cancellation scope is also non-blocking.
       */
      scope.run();

      // We use "anyOf" here to wait for one of the activity invocations to return
      String result = Promise.anyOf(results).get();

      // Trigger cancellation of all uncompleted activity invocations within the cancellation scope
      scope.cancel();

      /*
       *  Wait for all activities to perform cleanup if needed.
       *  For the sake of the example we ignore cancellations and
       *  get all the results so that we can print them in the end.
       *
       *  Note that we cannot use "allOf" here as that fails on any Promise failures
       */
      for (Promise<String> activityResult : results) {
        try {
          activityResult.get();
        } catch (ActivityFailure e) {
          if (!(e.getCause() instanceof CanceledFailure)) {
            throw e;
          }
        }
      }
      return result;
    }
}
```

## Terminate a Workflow Execution 

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

To terminate a Workflow Execution in Java, use the
[terminate()](<https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#terminate(java.lang.String,java.lang.Object...)>)
function on the WorkflowStub.

```java
WorkflowStub untyped = WorkflowStub.fromTyped(myWorkflowStub);
untyped.terminate("Sample reason");
```

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Child Workflows - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/child-workflows

> Start a Child Workflow Execution and set a Parent Close Policy using the Java SDK. Manage Child Workflow Events and ensure successful execution.

This page shows how to do the following:

- [Start a Child Workflow Execution](#start-child-workflow)
- [Set a Parent Close Policy](#parent-close-policy)

## Start a Child Workflow Execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In Java, you must explicitly call `Workflow.getWorkflowExecution(child)` to get a `Promise`, then call `.get()` on that Promise to wait for this Event.
See the [Parent Close Policy](#parent-close-policy) section below for a complete example.

### Async Child Workflows

The first call to the Child Workflow stub must always be its Workflow method (method annotated with `@WorkflowMethod`).
Similar to Activities, invoking Child Workflow methods can be made synchronous or asynchronous by using `Async#function` or `Async#procedure`.
The synchronous call blocks until a Child Workflow method completes.
The asynchronous call returns a `Promise` which can be used to wait for the completion of the Child Workflow method, as in the following example:

```java
GreetingChild child = Workflow.newChildWorkflowStub(GreetingChild.class);
Promise<String> greeting = Async.function(child::composeGreeting, "Hello", name);
// ...
greeting.get()
```

To execute an untyped Child Workflow asynchronously, call `executeAsync` on the `ChildWorkflowStub`, as shown in the following example.

```java
//...
ChildWorkflowStub childUntyped =
    Workflow.newUntypedChildWorkflowStub(
        "GreetingChild", // your workflow type
        ChildWorkflowOptions.newBuilder().setWorkflowId("childWorkflow").build());

Promise<String> greeting =
    childUntyped.executeAsync(String.class, String.class, "Hello", name);
String result = greeting.get();
//...
```

The following examples show how to spawn a Child Workflow:

- Spawn a Child Workflow from a Workflow

  ```java
  // Child Workflow interface
  @WorkflowInterface
  public interface GreetingChild {
  @WorkflowMethod
  String composeGreeting(String greeting, String name);
  }
  // Child Workflow implementation not shown

  // Parent Workflow implementation
  public class GreetingWorkflowImpl implements GreetingWorkflow {

  @Override
  public String getGreeting(String name) {
      GreetingChild child = Workflow.newChildWorkflowStub(GreetingChild.class);

      // This is a blocking call that returns only after child has completed.
      return child.composeGreeting("Hello", name );
  }
  }
  ```

- Spawn two Child Workflows (with the same type) in parallel:

  ```java
  // Parent Workflow implementation
  public class GreetingWorkflowImpl implements GreetingWorkflow {

      @Override
      public String getGreeting(String name) {

          // Workflows are stateful, so a new stub must be created for each new child.
          GreetingChild child1 = Workflow.newChildWorkflowStub(GreetingChild.class);
          Promise<String> greeting1 = Async.function(child1::composeGreeting, "Hello", name);

          // Both children will run concurrently.
          GreetingChild child2 = Workflow.newChildWorkflowStub(GreetingChild.class);
          Promise<String> greeting2 = Async.function(child2::composeGreeting, "Bye", name);

          // Do something else here.
          ...
          return "First: " + greeting1.get() + ", second: " + greeting2.get();
      }
  }
  ```

- Send a Signal to a Child Workflow from the parent:

  ```java
  // Child Workflow interface
  @WorkflowInterface
  public interface GreetingChild {
      @WorkflowMethod
      String composeGreeting(String greeting, String name);

      @SignalMethod
      void updateName(String name);
  }

  // Parent Workflow implementation
  public class GreetingWorkflowImpl implements GreetingWorkflow {

      @Override
      public String getGreeting(String name) {
          GreetingChild child = Workflow.newChildWorkflowStub(GreetingChild.class);
          Promise<String> greeting = Async.function(child::composeGreeting, "Hello", name);
          child.updateName("Temporal");
          return greeting.get();
      }
  }
  ```

- Sending a Query to Child Workflows from within the parent Workflow code is not supported. However, you can send a Query to Child Workflows from Activities using `WorkflowClient`.

Related reads:

- [Develop a Workflow Definition](/develop/java/workflows/basics)

- Java Workflow reference: [https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html)

## Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

Set [Parent Close Policy](/parent-close-policy) on an instance of `ChildWorkflowOptions` using [`ChildWorkflowOptions.newBuilder().setParentClosePolicy`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/ChildWorkflowOptions.Builder.html).

- Type: `ChildWorkflowOptions.Builder`
- Default: `PARENT_CLOSE_POLICY_TERMINATE`

```java
 public void parentWorkflow() {
     ChildWorkflowOptions options =
        ChildWorkflowOptions.newBuilder()
            .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON)
            .build();
     MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, options);
     Async.procedure(child::<workflowMethod>, <args>...);
     Promise<WorkflowExecution> childExecution = Workflow.getWorkflowExecution(child);
     // Wait for child to start
     childExecution.get()
}
```

In this example, we are:

1. Setting `ChildWorkflowOptions.ParentClosePolicy` to `ABANDON` when creating a Child Workflow stub.
2. Starting Child Workflow Execution asynchronously using `Async.function` or `Async.procedure`.
3. Calling `Workflow.getWorkflowExecution(…)` on the child stub.
4. Waiting for the `Promise` returned by `getWorkflowExecution` to complete.
   This indicates whether the Child Workflow started successfully (or failed).
5. Completing parent Workflow Execution asynchronously.

Steps 3 and 4 are needed to ensure that a Child Workflow Execution starts before the parent closes.
If the parent initiates a Child Workflow Execution and then completes immediately after, the Child Workflow will never execute.

---

# Continue-As-New - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/continue-as-new

> Use Temporal's Continue-As-New in Java to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters.

This page covers the following for Java developers:

- [What is Continue-As-New?](#what)
- [Use Continue-As-New](#how)
- [When is it right to Continue-As-New?](#when)
- [Test Continue-As-New](#how-to-test)

## What is Continue-As-New? 

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution.
You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History.
It also receives your Workflow's usual parameters.

## Use Continue-As-New with the Java SDK 

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically set to `None` for the original caller of the Workflow.

[View the source code](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/safemessagepassing/ClusterManagerWorkflow.java) in the context of the rest of the application code.

```java
class ClusterManagerInput {
  private final Optional<ClusterManagerState> state;
  private final boolean testContinueAsNew;
}

@WorkflowMethod
ClusterManagerResult run(ClusterManagerInput input);

````
The test hook in the above snippet is covered [below](#how-to-test).

Inside your Workflow, call the [`continueAsNew()`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#continueAsNew(io.temporal.workflow.ContinueAsNewOptions,java.lang.Object...)) function with the same type.
This stops the Workflow right away and starts a new one.

[View the source code](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/safemessagepassing/ClusterManagerWorkflowImpl.java) in the context of the rest of the application code.

```java
Workflow.continueAsNew(
  new ClusterManagerInput(Optional.of(state), input.isTestContinueAsNew()));
````

### Considerations for Workflows with message handlers 

If you use Updates or Signals, don't call Continue-as-New from the handlers.
Instead, wait for your handlers to finish in your main Workflow before you run `continueAsNew`.

## When is it right to Continue-As-New with the Java SDK? 

Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New.
Call `Workflow.getInfo().isContinueAsNewSuggested()` to check if it's time.

## Test Continue-As-New with the Java SDK 

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive.
Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, when `testContinueAsNew == true`, this sample creates a test-only variable called `maxHistoryLength` and sets it to a small value.
A helper method in the Workflow checks it each time it considers using Continue-as-New:

[View the source code](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/safemessagepassing/ClusterManagerWorkflowImpl.java) in the context of the rest of the application code.

```java
private boolean shouldContinueAsNew() {
  if (Workflow.getInfo().isContinueAsNewSuggested()) {
    return true;
  }
  // This is just for ease-of-testing.  In production, we trust temporal to tell us when to
  // continue as new.
  if (maxHistoryLength > 0 && Workflow.getInfo().getHistoryLength() > maxHistoryLength) {
    return true;
  }
  return false;
}
```

---

# Workflow message passing - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/message-passing

> Develop with Queries, Signals, and Updates with the Temporal Java SDK.

A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates.
The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values.
Temporal Clients use messages to read Workflow state and control execution.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic.
This page introduces these features for the Temporal Java SDK.

## Write message handlers 

Follow these guidelines when writing your message handlers:

- Message handlers are defined as methods on the Workflow class, using one of the three annotations: [`@QueryMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/QueryMethod.html), [`@SignalMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/SignalMethod.html), and [`@UpdateMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/UpdateMethod.html).
- The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion).
- Prefer a single class with multiple fields over using multiple input parameters.
  A class allows you to add fields without changing the calling signature.

### Query handlers 

A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution:

```java
public class MessagePassingIntro {

    public enum Language {
        CHINESE,
        ENGLISH,
        FRENCH,
        SPANISH,
        PORTUGUESE,
    }

    public static class GetLanguagesInput {
        public boolean includeUnsupported;

        public GetLanguagesInput() {
            this.includeUnsupported = false;
        }

        public GetLanguagesInput(boolean includeUnsupported) {
            this.includeUnsupported = includeUnsupported;
        }
    }

    @WorkflowInterface
    public interface GreetingWorkflow {
        ...
        // 👉 Use the @QueryMethod annotation to define a Query handler in the
        // Workflow interface.
        @QueryMethod
        List<Language> getLanguages(GetLanguagesInput input);
    }

    public static class GreetingWorkflowImpl implements GreetingWorkflow {
        ...
        @Override
        public List<Language> getLanguages(GetLanguagesInput input) {
            // 👉 The Query handler returns a value: it must not mutate the Workflow state
            // or perform blocking operations.
            if (input.includeUnsupported) {
                return Arrays.asList(Language.values());
            } else {
                return new ArrayList(greetings.keySet());
            }
        }
    }

}
```

- A Query handler must not modify Workflow state.
- You can't perform blocking operations such as executing an Activity in a Query handler.
- The Query annotation accepts an argument (`name`) as described in the API reference docs for [`@QueryMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/QueryMethod.html).

### Signal handlers 

A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow:

```java
public class MessagePassingIntro {

    public static class ApproveInput {
        private String name;

        public ApproveInput() {}

        public ApproveInput(String name) {
            this.name = name;
        }
    }

    @WorkflowInterface
    public interface GreetingWorkflow {
        ...
        // 👉 Use the @SignalMethod annotation to define a Signal handler in the
        // Workflow interface.
        @SignalMethod
        void approve(ApproveInput input);
    }

    public static class GreetingWorkflowImpl implements GreetingWorkflow {
        ...
        private Boolean approvedForRelease;
        private String approverName;

        @Override
        public void approve(ApproveInput input) {
            // 👉 The Signal handler mutates the Workflow state but cannot return a value.
            this.approvedForRelease = true;
            this.approverName = input.name;
        }
    }
}
```

- The handler should not return a value.
  The response is sent immediately from the server, without waiting for the Workflow to process the Signal.

- The Signal annotation accepts arguments (`name`, and `unfinished_policy`) as described in the API reference docs for [`@SignalMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/SignalMethod.html).

- Signal (and Update) handlers can be blocking.
  This allows you to use Activities, Child Workflows, durable [`Workflow.sleep`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep(java.time.Duration)) Timers, [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)), and more.
  See [Blocking handlers](#blocking-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using blocking Signal and Update handlers.

### Update handlers and validators 

An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution.
It can change the Workflow state, control its flow, and return a result.
The sender must wait until the Worker accepts or rejects the Update.
The sender may wait further to receive a returned value or an exception if something goes wrong:

```java
public class MessagePassingIntro {
    @WorkflowInterface
    public interface GreetingWorkflow {
        ...
        // 👉 Use the @UpdateMethod annotation to define an Update handler in the
        // Workflow interface.
        @UpdateMethod
        Language setLanguage(Language language);

        // 👉 Update validators are optional
        @UpdateValidatorMethod(updateName = "setLanguage")
        void setLanguageValidator(Language language);
    }

    public static class GreetingWorkflowImpl implements GreetingWorkflow {
        ...
        @Override
        public Language setLanguage(Language language) {
            // 👉 The Update handler can mutate the Workflow state and return a value.
            Language previousLanguage = this.language;
            this.language = language;
            return previousLanguage;
        }

        @Override
        public void setLanguageValidator(Language language) {
            // 👉 The Update validator performs validation but cannot mutate the Workflow state.
            if (!greetings.containsKey(language)) {
                throw new IllegalArgumentException("Unsupported language: " + language);
            }
        }
    }
}
```

- The Update annotation accepts arguments (`name`, and `unfinished_policy`) as described in the API reference docs for [`@UpdateMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/UpdateMethod.html).

- About validators:
  - Use validators to reject an Update before it is written to History.
    Validators are always optional.
    If you don't need to reject Updates, you can skip them.
  - Define an Update validator with the `@UpdateValidatorMethod` annotation.
    Use the `updateName` argument when declaring the validator to connect it to its Update.
    The validator must return `void` and accept the same argument types as the handler.

- Accepting and rejecting Updates with validators:
  - To reject an Update, throw an exception of any type in the validator.
  - Without a validator, Updates are always accepted.
- Validators and Event History:
  - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic.
  - When a Validator throws an error, the Update is rejected, the Update is not run, and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History.
    The caller receives an "Update failed" error.

- Use [`getCurrentUpdateInfo`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/sync/WorkflowInternal.html#getCurrentUpdateInfo()) to obtain information about the current Update. This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).

- Signal (and Update) handlers can be blocking, letting them use Activities, Child Workflows, durable [`Workflow.sleep`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep(java.time.Duration)) Timers, [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)) conditions, and more.
  See [Blocking handlers](#blocking-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for safe usage guidelines.

## Send messages 

To send Queries, Signals, or Updates you call methods on a `WorkflowInterface`, often called the "WorkflowStub."

Use [newWorkflowStub](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#newWorkflowStub(java.lang.Class,io.temporal.client.WorkflowOptions)) to obtain the WorkflowStub.

For example:

```java
WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();

WorkflowClient client = WorkflowClient.newInstance(service);

WorkflowOptions workflowOptions =
    WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).setWorkflowId(WORKFLOW_ID).build();

// Create the workflow client stub. It is used to start the workflow execution.
GreetingWorkflow workflow = client.newWorkflowStub(GreetingWorkflow.class, workflowOptions);

// Start workflow asynchronously and call its getGreeting workflow method
WorkflowClient.start(workflow::getGreetings);
```

To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition.

> **⚠️ Warning:**
> Using Continue-as-New and Updates
>
> - Temporal _does not_ support Continue-as-New functionality within Update handlers.
> - Complete all handlers _before_ using Continue-as-New.
> - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution.
>

### Send a Query 

Call a Query method defined within a Workflow from a `WorkflowStub` created in Client code to send a Query to a Workflow Execution:

```java
List<Language> languages = workflow.getLanguages(new GetLanguagesInput(false));
System.out.println("Supported languages: " + languages);
```

- Sending a Query doesn’t add events to a Workflow's Event History.

- You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period.
  This includes Workflows that have completed, failed, or timed out.
  Querying terminated Workflows is not safe and, therefore, not supported.

- A Worker must be online and polling the Task Queue to process a Query.

### Send a Signal 

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution.
However, you can only send Signals to Workflow Executions that haven’t closed.

#### Send a Signal from a Client 

To send a Signal from Client code, call a Signal method on the WorkflowStub:

```java
workflow.approve(new ApproveInput("Me"));
```

- The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution.

- The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History.

#### Send a Signal from a Workflow 

A Workflow can send a Signal to another Workflow, known as an _External Signal_.
Use [`Workflow.newExternalWorkflowStub`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newExternalWorkflowStub(java.lang.Class,io.temporal.api.common.v1.WorkflowExecution)) in your _current_ Workflow to create an `ExternalWorkflowStub` for the other Workflow.
Call Signal methods on the external stub to Signal the other Workflow:

```java
OtherWorkflow other = Workflow.newExternalWorkflowStub(OtherWorkflow.class, otherWorkflowID);
other.mySignalMethod();
```

When an External Signal is sent:

- A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History.
- A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History.

#### Signal-With-Start 

Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running.
If there's a Workflow running with the given Workflow Id, it will be signaled.
If there isn't, a new Workflow will be started and immediately signaled.
To use Signal-With-Start, call `signalWithStart` and pass the name of your Signal with its arguments:

```java
public static void signalWithStart() {
    // WorkflowStub is a client-side stub to a single Workflow instance
    WorkflowStub untypedWorkflowStub = client.newUntypedWorkflowStub("GreetingWorkflow",
    WorkflowOptions.newBuilder()
            .setWorkflowId(workflowId)
            .setTaskQueue(taskQueue)
            .build());

    untypedWorkflowStub.signalWithStart("setCustomer", new Object[] {customer2}, new Object[] {customer1});

    String greeting = untypedWorkflowStub.getResult(String.class);
}
```

Here's the `WorkflowInterface` for the previous example.
When using Signal-With-Start, the Signal handler (`setCustomer`) will be executed before the Workflow method (`greet`).

```java
@WorkflowInterface
public interface GreetingWorkflow {
    @WorkflowMethod
    String greet(Customer customer);

    @SignalMethod
    void setCustomer(Customer customer);

    @QueryMethod
    Customer getCustomer();
}
```

### Send an Update 

An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result.

A Client sending an Update must wait until the Server delivers the Update to a Worker.
Workers must be available and responsive.
If you need a response as soon as the Server receives the request, use a Signal instead.
You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity.

- `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation.
- `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished.

To send an Update to a Workflow Execution, you can:

- Call the Update method on a WorkflowStub in Client code and wait for the Update to complete.
  This code fetches an Update result:

  ```java
  Language previousLanguage = workflow.setLanguage(Language.CHINESE);
  ```

- Send [`startUpdate`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#startUpdate(io.temporal.client.UpdateOptions,java.lang.Object...)) to receive an [`WorkflowUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowUpdateHandle.html) as soon as the Update is accepted or rejected.

  - Use this `WorkflowUpdateHandle` later to fetch your results.
  - Blocking Update handlers normally perform long-running asynchronous operations.
  - `startUpdate` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete.

  For example:

  ```java
  WorkflowUpdateHandle<Language> handle =
      WorkflowStub.fromTyped(workflow)
          .startUpdate(
              "setLanguage", WorkflowUpdateStage.ACCEPTED, Language.class, Language.ENGLISH);
  previousLanguage = handle.getResultAsync().get();
  ```

  For more details, see the "Blocking handlers" section.

To obtain an Update handle, you can:

- Use [`startUpdate`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#startUpdate(io.temporal.client.UpdateOptions,java.lang.Object...)) to start an Update and return the handle, as shown in the preceding example.
- Use [`getUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#getUpdateHandle(java.lang.String,java.lang.Class)) to fetch a handle for an in-progress Update using the Update ID and Workflow ID.

You can use the `WorkflowUpdateHandle` to obtain information about the update:

- `getExecution()`: Returns the Workflow Execution that this Update was sent to.
- `getId()`: Returns the Update's unique ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
- `getResultAsync()`: Returns a `CompletableFuture` which can be used to wait for the Update to complete.

#### Update-With-Start 

> **💡 Tip:**
>
> For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended.
>

[Update-with-Start](/sending-messages#update-with-start) lets you
[send an Update](/develop/java/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists:

- If the Workflow exists, the Update is processed.
- If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute.

Use the [`startUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#startUpdateWithStart(io.temporal.workflow.Functions.Func,io.temporal.client.UpdateOptions,io.temporal.client.WithStartWorkflowOperation)) WorkflowClient API.
It returns once the requested Update wait stage has been reached; or when the request times out.
Use the [`WorkflowUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowUpdateHandle.html) to retrieve a result from the Update.

You will need to provide:

- WorkflowStub created from [`WorkflowOptions`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.html).
  The `WorkflowOptions` require [Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) to be specified.
  Choose "Use Existing" and use an idempotent Update handler to ensure your code can be executed again in case of a Client failure.
  Not all `WorkflowOptions` are allowed.
  For example, specifying a Cron Schedule will result in an error.

- [`UpdateOptions`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/UpdateOptions.html).
  Same as for [Update Workflow](/develop/java/workflows/message-passing#send-update-from-client), the update wait stage must be specified.
  For Update-with-Start, the Workflow Id is optional.
  When specified, the Id must match the one used in `WorkflowOptions`.
  Since a running Workflow Execution may not already exist, you can't set a Run Id.

- [`WithStartWorkflowOperation`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WithStartWorkflowOperation.html).
  Specify the workflow method.
  Note that a `WithStartWorkflowOperation` can only be used once.
  Re-using a previously used operation returns an error from `startUpdateWithStart`.

For example:

```java
WorkflowUpdateHandle<Language> handle =
    WorkflowClient.startUpdateWithStart(
        workflow::setLanguage,
        Language.ENGLISH,
        UpdateOptions.<Language>newBuilder().setWaitForStage(WorkflowUpdateStage.ACCEPTED).build(),
        new WithStartWorkflowOperation<>(workflow::getGreetings));

Language previousLanguage = handle.getResultAsync().get();
```

To obtain the update result directly, use the [`executeUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#executeUpdateWithStart(io.temporal.workflow.Functions.Func,io.temporal.client.UpdateOptions,io.temporal.client.WithStartWorkflowOperation)) WorkflowClient API.
It returns once the update result is available; or when the API call times out.
The update wait stage on the `UpdateOptions` is optional.
When specified, it must be `WorkflowUpdateStage.COMPLETED`.

For example:

```java
Language previousLanguage =
    WorkflowClient.executeUpdateWithStart(
        workflow::setLanguage,
        Language.ENGLISH,
        UpdateOptions.<Language>newBuilder().build(),
        new WithStartWorkflowOperation<>(workflow::getGreetings));
```

For more examples, see the [Java sample for early-return pattern](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/earlyreturn).

> **ℹ️ Info:**
> NON-TYPE SAFE API CALLS
>
> In real-world development, sometimes you may be unable to import Workflow Definition method signatures.
> When you don't have access to the Workflow Definition or it isn't written in Java, you can use these non-type safe APIs to obtain an untyped WorkflowStub:
>
> - [`WorkflowClient.newUntypedWorkflowStub`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#newUntypedWorkflowStub(java.lang.String,io.temporal.client.WorkflowOptions))
> - [`Workflow.newUntypedExternalWorkflowStub`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newUntypedExternalWorkflowStub(java.lang.String)).
>
> Pass method names instead of method objects to:
>
> - [`WorkflowStub.query`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#query(java.lang.String,java.lang.Class,java.lang.Object...))
> - [`WorkflowStub.signal`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#signal(java.lang.String,java.lang.Object...))
> - [`WorkflowStub.update`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#update(java.lang.String,java.lang.Class,java.lang.Object...))
> - [`WorkflowStub.startUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#startUpdateWithStart(io.temporal.client.UpdateOptions,java.lang.Object%5B%5D,java.lang.Object%5B%5D))
> - [`WorkflowStub.executeUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#executeUpdateWithStart(io.temporal.client.UpdateOptions,java.lang.Object%5B%5D,java.lang.Object%5B%5D))
>

## Message handler patterns 

This section covers common write operations, such as Signal and Update handlers.
It doesn't apply to pure read operations, like Queries or Update Validators.

> **💡 Tip:**
>
> For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), and [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
>

### Do blocking operations in handlers 

Signal and Update handlers can block.
This allows you to use [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)), Activities, Child Workflows, [`Workflow.sleep`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep(java.time.Duration)) Timers, and so on.
This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls.

To use blocking handlers safely, understand what can go wrong.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of blocking Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below.

The following code modifies the Update handler from earlier on in this page.
The Update handler now makes a blocking call to execute an Activity:

```java
public static class GreetingWorkflowImpl implements GreetingWorkflow {
    ...
    @Override
    public Language setLanguage(Language language) {
        if (!greetings.containsKey(language)) {
            String greeting = activity.greetingService(language);
            if (greeting == null) {
                // 👉 An update validator cannot be blocking, so cannot be used to check that the remote
                // greetingService supports the requested language. Throwing an ApplicationFailure
                // will fail the Update, but the WorkflowExecutionUpdateAccepted event will still be
                // added to history.
                throw ApplicationFailure.newFailure("Greeting service does not support: " + language, "GreetingFailure")
            }
            greetings.put(language, greeting);
        }
        Language previousLanguage = this.language;
        this.language = language;
        return previousLanguage;
    }
}
```

Although a Signal handler can also make blocking calls like this, using an Update handler allows the Client to receive a result or error once the Activity completes.
This lets your Client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, and so on.

### Add blocking wait conditions 

Sometimes, blocking Signal or Update handlers need to meet certain conditions before they should continue.
You can use [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)) to prevent the code from proceeding until a condition is true.
You specify the condition by passing a function that returns `true` or `false`.
This is an important feature that helps you control your handler logic.

Here are two important use cases for `Workflow.await`:

- Waiting in a handler until it is appropriate to continue.
- Waiting in the main Workflow until all active handlers have finished.

#### Wait for conditions in handlers 

It's common to use `Workflow.await` in a handler.
For example, suppose your Workflow class has a `updateReadyToExecute` method that indicates whether your Update handler should be allowed to start executing.
You can use `workflow.wait_condition` in the handler to make the handler pause until the condition is met:

```java
@Override
public String setLanguage(UpdateInput input) {
    Workflow.await(() -> this.updateReadyToExecute(input));
    ...
}
```

Remember: handlers can execute before the main Workflow method starts.

You can also use `Workflow.await` anywhere else in the handler to wait for a specific condition to become true.
This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become true.

#### Ensure your handlers finish before the Workflow completes 

`Workflow.await` can ensure your handler completes before a Workflow finishes.
When your Workflow uses blocking Signal or Update handlers, your main Workflow method can return or Continue-as-New while a handler is still waiting on an async task, such as an Activity.
The Workflow completing may interrupt the handler before it finishes crucial work and cause Client errors when trying to retrieve Update results.
Use `Workflow.await` to wait for [`Workflow.isEveryHandlerFinished`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#isEveryHandlerFinished()) to return `true` to address this problem and allow your Workflow to end smoothly:

```java
public class MyWorkflowImpl implements MyWorkflow {
    ...
    @Override
    public String run() {
        ...
        Workflow.await(() -> Workflow.isEveryHandlerFinished());
        return "workflow-result";
    }
}
```

By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions.
You can silence these warnings on a per-handler basis by passing the `unfinishedPolicy` argument to the [`@SignalMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/SignalMethod.html) / [`@UpdateMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/UpdateMethod.html) annotation:

```java
@WorkflowInterface
public interface MyWorkflow {
    ...
    @UpdateMethod(unfinishedPolicy = HandlerUnfinishedPolicy.ABANDON)
    void myUpdate();
}
```

See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information.

### Use `@WorkflowInit` to operate on Workflow input before any handler executes

Normally, your Workflows constructor won't have any parameters.
However, if you use the `@WorkflowInit` annotation on your constructor, you can give it the same [Workflow parameters](/develop/java/workflows/basics#workflow-parameters) as your `@WorkflowMethod`.
The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/java/client/temporal-client#start-workflow-execution).
The Workflow input arguments are also passed to your `@WorkflowMethod` method -- that always happens, whether or not you use the `@WorkflowInit` annotation.
This is useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers).

> **⚠️ Caution:**
>
> Do not make blocking calls from within your `@WorkflowInit` method. This could result in your Workflow being incompletely initialized at the start, meaning, for example, that Signal, Query, and Update handler registration would be delayed.
>

Here's an example.
Notice that the constructor and `getGreeting` must have the same parameters:

```java
public class GreetingExample {
    @WorkflowInterface
    public interface GreetingWorkflow {
        @WorkflowMethod
        String getGreeting(String input);

        @UpdateMethod
        boolean checkTitleValidity();
    }

    public static class GreetingWorkflowImpl implements GreetingWorkflow {
        private final String nameWithTitle;
        private boolean titleHasBeenChecked;
        ...
        // Note the annotation is on a public constructor
        @WorkflowInit
        public GreetingWorkflowImpl(String input) {
          this.nameWithTitle = "Sir " + input;
          this.titleHasBeenChecked = false;
        }

        @Override
        public String getGreeting(String input) {
          Workflow.await(() -> titleHasBeenChecked)
          return "Hello " + nameWithTitle;
        }

        @Override
        public boolean checkTitleValidity() {
          // 👉 The handler is now guaranteed to see the workflow input
          // after it has been processed by the constructor.
          boolean isValid = activity.checkTitleValidity(nameWithTitle);
          titleHasBeenChecked = true;
          return isValid;
        }
    }
}
```

### Use locks to prevent concurrent handler execution 

Concurrent processes can interact in unpredictable ways.
Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously.
Here's an example of a pathological case:

```java
public class DataWorkflowImpl implements DataWorkflow {
    ...
    @Override
    public void badSignalHandler() {
        Data data = activity.fetchData();
        this.x = data.x;
        // 🐛🐛 Bug!! If multiple instances of this method are executing concurrently, then
        // there may be times when the Workflow has self.x from one Activity execution and self.y from another.
        Workflow.sleep(Duration.ofSeconds(1));
        this.y = data.y;
    }
}
```

Coordinating access with `WorkflowLock` corrects this code.
Locking makes sure that only one handler instance can execute a specific section of code at any given time:

```java
public class DataWorkflowImpl implements DataWorkflow {
    WorkflowLock lock = Workflow.newWorkflowLock();
    ...
    @Override
    public void safeSignalHandler() {
        try {
            lock.lock();
            Data data = activity.fetchData();
            this.x = data.x;
            // ✅ OK: the scheduler may switch now to a different handler execution,
            // or to the main workflow method, but no other execution of this handler
            // can run until this execution finishes.
            Workflow.sleep(Duration.ofSeconds(1));
            this.y = data.y;
        } finally {
            lock.unlock()
        }
    }
}
```

## Message handler troubleshooting 

When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors:

- **The Client can't contact the server**:
  You'll receive a [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) on which the `cause` is a [`StatusRuntimeException`](https://grpc.github.io/grpc-java/javadoc/io/grpc/StatusRuntimeException.html) and `status` of `UNAVAILABLE` (after some retries).

- **The Workflow does not exist**:
  You'll receive a [`WorkflowNotFoundException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowNotFoundException.html).

See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Java-specific discussion of this topic.

### Problems when sending a Signal 

When using Signal, the above `WorkflowException`s are the only types of exception that will result from the request.

In contrast, for Queries and Updates, the client waits for a response from the Worker.
If an issue occurs during the handler execution by the Worker, the Client may receive an exception.

### Problems when sending an Update 

When working with Updates, you may encounter these errors:

- **No Workflow Workers are polling the Task Queue**:
  Your request will be retried by the SDK Client indefinitely.
  You can impose a timeout with `CompletableFuture.get()` method with a timeout parameter.
  This throws a `java.util.concurrent.TimeoutException` exception when it expires.

- **Update failed**: You'll receive a `WorkflowUpdateException` exception.
  There are two ways this can happen:

  - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler.

  - The Update failed after having been accepted.

  Update failures are like [Workflow failures](/references/failures).
  Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.
  These might include:

      - A failed Child Workflow
      - A failed Activity (if the Activity retries have been set to a finite number)
      - The Workflow author throwing `ApplicationFailure`
      - Any error listed in [getFailWorkflowExceptionTypes](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkflowImplementationOptions.html#getFailWorkflowExceptionTypes()) (empty by default)

- **The handler caused the Workflow Task to fail**:
  A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage:
  - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) exception.
  - If the request has been accepted, it is durable.
    Once the Workflow is healthy again after a code deploy, use a [`WorkflowUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowUpdateHandle.html) to fetch the Update result.

- **The Workflow finished while the Update handler execution was in progress**:
  You'll receive a [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) "workflow execution already completed"`.

  This will happen if the Workflow finished while the Update handler execution was in progress, for example because

  - The Workflow was canceled or failed.

  - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers).

### Problems when sending a Query 

When working with Queries, you may encounter these errors:

- **There is no Workflow Worker polling the Task Queue**:
  You'll receive a [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) on which the `cause` is a [`StatusRuntimeException`](https://grpc.github.io/grpc-java/javadoc/io/grpc/StatusRuntimeException.html) with a `status` of `FAILED_PRECONDITION`.

- **Query failed**:
  You'll receive a [`WorkflowQueryException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowQueryException.html) exception if something goes wrong during a Query.
  Any exception in a Query handler will trigger this error.
  This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead.

- **The handler caused the Workflow Task to fail.**
  This would happen, for example, if the Query handler blocks the thread for too long without yielding.

## Dynamic components 

A dynamic Workflow, Activity, Signal, Update, or Query is a kind of unnamed item.
Normally, these items are registered by name with the Worker and invoked at runtime.
When an unregistered or unrecognized Workflow, Activity, or message request arrives with a recognized method signature, the Worker can use a pre-registered dynamic stand-in.

For example, you might send a request to start a Workflow named "MyUnknownWorkflow".
After receiving a Workflow Task, the Worker may find that there's no registered Workflow Definitions of that type.
It then checks to see if there's a registered dynamic Workflow.
If the dynamic Workflow signature matches the incoming Workflow signature, the Worker invokes that just as it would invoke a non-dynamic statically named version.

By registering dynamic versions of your Temporal components, the Worker can fall back to these alternate implementations for name mismatches.

> **⚠️ Caution:**
>
> Use dynamic elements judiciously and as a fallback mechanism, not a primary design.
> They can introduce long-term maintainability and debugging issues.
> Reserve dynamic invocation use for cases where a name is not or can't be known at compile time.
>

### Set a Dynamic Workflow 

Use [`DynamicWorkflow`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/DynamicWorkflow.html) to implement Workflow Types dynamically.
Register a Workflow implementation type that extends `DynamicWorkflow` to implement any Workflow Type that is not explicitly registered with the Worker.

The dynamic Workflow interface is implemented with the `execute` method. This method takes in `EncodedValues` that are inputs to the Workflow Execution.
These inputs can be specified by the Client when invoking the Workflow Execution.

```java
public class MyDynamicWorkflow implements DynamicWorkflow {
   @Override
    public Object execute(EncodedValues args) {
    }
}
```

### Set a Dynamic Activity 

To handle Activity types that do not have an explicitly registered handler, you can directly implement a dynamic Activity.

Use `DynamicActivity` to implement any number of Activity types dynamically.
When an Activity implementation that extends `DynamicActivity` is registered, it is called for any Activity type invocation that doesn't have an explicitly registered handler.

The dynamic Activity interface is implemented with the `execute` method, as shown in the following example.

```java
// Dynamic Activity implementation
 public static class DynamicGreetingActivityImpl implements DynamicActivity {
   @Override
   public Object execute(EncodedValues args) {
     String activityType = Activity.getExecutionContext().getInfo().getActivityType();
     return activityType
         + ": "
         + args.get(0, String.class)
         + " "
         + args.get(1, String.class)
         + " from: "
         + args.get(2, String.class);
   }
 }
```

Use `Activity.getExecutionContext()` to get information about the Activity type that should be implemented dynamically.

### Set a Dynamic Signal 

You can also implement Signal handlers dynamically. This is useful for library-level code and implementation of DSLs.

Use `Workflow.registerListener(Object)` to register an implementation of the `DynamicSignalListener` in the Workflow implementation code.

```java
Workflow.registerListener(
  (DynamicSignalHandler)
      (signalName, encodedArgs) -> name = encodedArgs.get(0, String.class));
```

When registered, any Signals sent to the Workflow without a defined handler will be delivered to the `DynamicSignalHandler`.
Note that you can only register one `Workflow.registerListener(Object)` per Workflow Execution.
`DynamicSignalHandler` can be implemented in both regular and dynamic Workflow implementations.

### Set a Dynamic Query 

You can also implement Query handlers dynamically. This is useful for library-level code and implementation of DSLs.

Use `Workflow.registerListener(Object)` to register an implementation of the `DynamicQueryListener` in the Workflow implementation code.

```java
Workflow.registerListener(
  (DynamicQueryHandler)
      (queryName, encodedArgs) -> name = encodedArgs.get(0, String.class));
```

When registered, any Queries sent to the Workflow without a defined handler will be delivered to the `DynamicQueryHandler`.
Note that you can only register one `Workflow.registerListener(Object)` per Workflow Execution.
`DynamicQueryHandler` can be implemented in both regular and dynamic Workflow implementations.

### Set a Dynamic Update 

You can also implement Update handlers dynamically.
This is useful for library-level code and implementation of DSLs.

```java
Workflow.registerListener(
  (DynamicUpdateHandler)
      (updateName, encodedArgs) -> encodedArgs.get(0, String.class));
```

When registered, any Updates sent to the Workflow without a defined handler will be delivered to the `DynamicUpdateHandler`.
You can only register one `Workflow.registerListener(Object)` per Workflow Execution.
`DynamicUpdateHandler` can be implemented in both regular and dynamic Workflow implementations.

---

# Schedules - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/schedules

> Schedule, Backfill, Delete, Describe, List, Pause, Trigger, Update, and set Cron and Start Delays for Workflow Executions in Java using Temporal's ScheduleClient.

This page shows how to do the following:

- [Schedule a Workflow](#schedule-a-workflow)
  - [Create a Schedule](#create-schedule)
  - [Backfill a Schedule](#backfill-schedule)
  - [Delete a Schedule](#delete-schedule)
  - [Describe a Schedule](#describe-schedule)
  - [List a Schedule](#list-schedule)
  - [Pause a Schedule](#pause-schedule)
  - [Trigger a Schedule](#trigger-schedule)
  - [Update a Schedule](#update-schedule)
- [Set a Cron Schedule](#cron-schedule)
- [Start Delay](#start-delay)

## Schedule a Workflow 

Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes.

Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process.

Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy).
If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions.

### Create a Schedule 

The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands.

To create a Scheduled Workflow Execution in Java, use the `createSchedule()` method on the `ScheduleClient`. When you create the `ScheduleClient`, you can also use any custom Namespace instead of the default by setting the [`ScheduleClientOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/schedules/ScheduleClientOptions.Builder.html). Schedules must be initialized with a Schedule ID.

```java
Schedule schedule =
    Schedule.newBuilder()
        .setAction(
            ScheduleActionStartWorkflow.newBuilder()
                .setWorkflowType(HelloSchedules.GreetingWorkflow.class)
                .setArguments("World")
                .setOptions(
                    WorkflowOptions.newBuilder()
                        .setWorkflowId("WorkflowId")
                        .setTaskQueue("TaskQueue")
                        .build())
                .build())
        .setSpec(ScheduleSpec.newBuilder()
                .setIntervals(
                    Collections.singletonList(new ScheduleIntervalSpec(Duration.ofSeconds(5)))
                ).build()
            )
        .build();

// Create a schedule on the server
ScheduleClient scheduleClient = ScheduleClient
    .newInstance(service, ScheduleClientOptions.newBuilder()
        .setNamespace("custom_namespace")
        .build()
    );

ScheduleHandle handle =
    scheduleClient.createSchedule("ScheduleId", schedule, ScheduleOptions.newBuilder().build());
```

> **💡 Tip:**
> Schedule Auto-Deletion
>
> Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again.
> The Temporal Service doesn't guarantee when this removal will happen.
>

### Backfill a Schedule 

The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time.

To Backfill a Scheduled Workflow Execution in Java, use the `backfill()` method on the `ScheduleHandle`.

```java
ScheduleHandle handle = client.getHandle("schedule-id")

Instant now = Instant.now();
handle.backfill(
    Arrays.asList(
        new ScheduleBackfill(now.minusMillis(5500), now.minusMillis(2500)),
        new ScheduleBackfill(now.minusMillis(2500), now)));
```

### Delete a Schedule 

The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule.

To delete a Scheduled Workflow Execution in Java, use the `delete()` method on the `Schedule Handle`.

```java
ScheduleHandle handle = client.getHandle("schedule-id")
handle.delete();
```

### Describe a Schedule 

The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs.

To describe a Scheduled Workflow Execution in Java, use the `describe()` method on the `ScheduleHandle`.

```java
ScheduleHandle handle = client.getHandle("schedule-id")
ScheduleDescription description = handle.describe();
```

### List a Schedule 

The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs.

To list all schedules, use the `listSchedules()` asynchronous method on the `ScheduleClient`.
If a schedule is added or deleted, it may not be available in the list immediately.

```java
Stream<ScheduleListDescription> scheduleStream = client.listSchedules();
```

### Pause a Schedule 

The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason.

To pause a Scheduled Workflow Execution in Java, use the `pause()` method on the `ScheduleHandle`.
You can pass a `note` to the `pause()` method to provide a reason for pausing the schedule.

```java
ScheduleHandle handle = client.getHandle("schedule-id")
handle.pause("Pausing the schedule for now");
```

### Trigger a Schedule 

The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time.

To trigger a Scheduled Workflow Execution in Java, use the `trigger()` method on the `ScheduleHandle`.

```java
ScheduleHandle handle = client.getHandle("schedule-id")
handle.trigger();
```

### Update a Schedule 

The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval.

Create a function that takes `ScheduleUpdateInput` and returns `ScheduleUpdate`.
To update a Schedule, use a callback to build the update from the description.
The following example updates the Schedule to set a limited number of actions.

```java
ScheduleHandle handle = client.getHandle("schedule-id")
handle.update(
    (ScheduleUpdateInput input) -> {
      Schedule.Builder builder = Schedule.newBuilder(input.getDescription().getSchedule());
      // Make the schedule paused to demonstrate how to unpause a schedule
      builder.setState(
          ScheduleState.newBuilder()
              .setLimitedAction(true)
              .setRemainingActions(10)
              .build());
      return new ScheduleUpdate(builder.build());
    });
```

## Set a Cron Schedule 

> **⚠️ Caution:**
> Cron support is not recommended
>
> We recommend using [Schedules](/schedule) instead of Cron Jobs.
> Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules.
>

A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution.

A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made.

Set the Cron Schedule with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setCronSchedule`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

Setting `setCronSchedule` changes the Workflow Execution into a Temporal Cron Job.
The default timezone for a Cron is UTC.

- Type: `String`
- Default: None

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    YourWorker.yourclient.newWorkflowStub(
        YourWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWF")
                .setTaskQueue(YourWorker.TASK_QUEUE)
                // Set Cron Schedule
                .setCronSchedule("* * * * *")
                .build());
```

Temporal Workflow Schedule Cron strings follow this format:

```
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
* * * * *
```

For more details, see the [Cron Sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloCron.java)

## Start Delay 

Use the `StartDelay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule.

Create an instance of [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) in the Client code and set `StartDelay` using `setStartDelay`.

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWorkflow")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Start the workflow in 12 hours
                .setStartDelay(Duration.ofHours(12))
                .build());
```

---

# Side Effects - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/side-effects

> Side Effects in a Workflow execute non-deterministic code like generating a UUID. The result is saved in Workflow Event History for consistent replays without re-execution.

## Side Effects 

Side Effects are used to execute non-deterministic code, such as generating a UUID or a random number, without compromising determinism in the Workflow.
This is done by storing the non-deterministic results of the Side Effect into the Workflow [Event History](/workflow-execution/event#event-history).

A Side Effect does not re-execute during a Replay. Instead, it returns the recorded result from the Workflow Execution Event History.

Side Effects should not fail. An exception that is thrown from the Side Effect causes failure and retry of the current Workflow Task.

An Activity or a Local Activity may also be used instead of a Side effect, as its result is also persisted in Workflow Execution History.

> **📝 Note:**
>
> You shouldn't modify the Workflow state inside a Side Effect function, because it is not reexecuted during Replay. Side Effect function should be used to return a value.
>

To use a Side Effect in Java, set the [`sideEffect()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sideEffect(java.lang.Class,io.temporal.workflow.Functions.Func)) function in your Workflow Execution and return the non-deterministic code.

```java
int random = Workflow.sideEffect(Integer.class, () -> random.nextInt(100));
if random < 50 {
       ....
} else {
       ....
}
```

Here's another example that uses `sideEffect()`.

```java
// implementation of the @WorkflowMethod
public void execute() {
    int randomInt = Workflow.sideEffect( int.class, () -> {
        Random random = new SecureRandom();
        return random.nextInt();
    });

    String userHome = Workflow.sideEffect(String.class, () -> System.getenv("USER_HOME"));

    if(randomInt % 2 == 0) {
        // ...
    } else {
        // ...
    }
}
```

Java also provides a deterministic method to generate random numbers or random UUIDs.

To generate random numbers in a deterministic method, use [`newRandom()`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newRandom).

```java
// implementation of the @WorkflowMethod
public void execute() {
    int randomInt = Workflow.newRandom().nextInt();
    // ...
}
```

To generate a random UUID in a deterministic method, use [`randomUUID()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newRandom()).

```java
// implementation of the @WorkflowMethod
public void execute() {
    String randomUUID = Workflow.randomUUID().toString();
    // ...
}
```

---

# Workflow Timeouts - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/timeouts

> Optimize Workflow Execution with Temporal Java SDK - Set Workflow Timeouts and Retry Policies efficiently.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts).

Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient.
Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes.
If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer.

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed.
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last.
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task.

Create an instance of [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) in the Client code and set your timeout.

Available timeouts are:

- [setWorkflowExecutionTimeout()](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html#setWorkflowExecutionTimeout(java.time.Duration))
- [setWorkflowRunTimeout()](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html#setWorkflowRunTimeout(java.time.Duration))
- [setWorkflowTaskTimeout()](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html#setWorkflowTaskTimeout(java.time.Duration))

```java
//create Workflow stub for YourWorkflowInterface
YourWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("YourWorkflow")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Workflow Timeout duration
                .setWorkflowExecutionTimeout(Duration.ofSeconds(10))
                // .setWorkflowRunTimeout(Duration.ofSeconds(10))
                // .setWorkflowTaskTimeout(Duration.ofSeconds(10))
                .build());
```

## Workflow retries 

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

To set a Workflow Retry Options in the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance use [`WorkflowOptions.Builder.setWorkflowRetryOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html).

- Type: `RetryOptions`
- Default: `Null` which means no retries will be attempted.

```java
//create Workflow stub for GreetWorkflowInterface
GreetWorkflowInterface workflow1 =
    WorkerGreet.greetclient.newWorkflowStub(
        GreetWorkflowInterface.class,
        WorkflowOptions.newBuilder()
                .setWorkflowId("GreetWF")
                .setTaskQueue(WorkerGreet.TASK_QUEUE)
                // Set Workflow Retry Options
                .setRetryOptions(RetryOptions.newBuilder()
                .build());
```

---

# Timers - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/timers

> A Workflow sets a durable Timer for delayed execution. Even if the Worker or Temporal Service is down, the Timer resumes once back up. Efficient and scalable.

## What is a Timer? 

A Workflow can set a durable Timer for a fixed time period.
In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`.

A Workflow can sleep for months.

Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `Workflow.sleep()` call resolves and your code continues executing.

Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker.

To set a Timer in Java, use [`sleep()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep) and pass the number of seconds you want to wait before continuing.

```java
sleep(5);
```

---

# Versioning - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/versioning

> Update running Workflows without breaking determinism using the Java SDK's Workflow Patching APIs and Worker Build IDs.

Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress.

The Temporal Platform requires that Workflow Definitions are [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably.

With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version.
There are two primary Versioning methods that you can use:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths.
- [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.

> **🚨 Danger:**
> Support for the experimental Worker Versioning method before 2025 was scheduled for removal from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance. You can still refer to the [Worker Versioning Legacy](/develop/java/worker-versioning-legacy) docs if needed.

## Worker Versioning

Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching.

## Versioning with Patching 

### Patching with `getVersion`

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow Executions, create a patch.

Consider the following Workflow Definition:

```java
public void processFile(Arguments args) {
    String localName = null;
    String processedName = null;
    try {
        localName = activities.download(args.getSourceBucketName(), args.getSourceFilename());
        processedName = activities.processFile(localName);
        activities.upload(args.getTargetBucketName(), args.getTargetFilename(), processedName);
    } finally {
        if (localName != null) { // File was downloaded.
            activities.deleteLocalFile(localName);
        }
        if (processedName != null) { // File was processed.
            activities.deleteLocalFile(processedName);
        }
    }
}
```

Imagine you want to revise this Workflow by adding another Activity to calculate a file checksum.
If an existing Workflow Execution was started by the original version of the Workflow code, where there was no `calculateChecksum()` Activity, and then resumed running on a new Worker where this Activity had been added, the server side Event History would be out of sync.
This would cause the Workflow to fail with a nondeterminism error.

To resolve this, you can use `Workflow.getVersion()` to patch your Workflow:

```java
public void processFile(Arguments args) {
    String localName = null;
    String processedName = null;
    try {
        localName = activities.download(args.getSourceBucketName(), args.getSourceFilename());
        processedName = activities.processFile(localName);
        int version = Workflow.getVersion("checksumAdded", Workflow.DEFAULT_VERSION, 1);
        if (version == Workflow.DEFAULT_VERSION) {
            activities.upload(args.getTargetBucketName(), args.getTargetFilename(), processedName);
        } else {
            long checksum = activities.calculateChecksum(processedName);
            activities.uploadWithChecksum(
                args.getTargetBucketName(), args.getTargetFilename(), processedName, checksum);
        }
    } finally {
        if (localName != null) { // File was downloaded.
            activities.deleteLocalFile(localName);
        }
        if (processedName != null) { // File was processed.
            activities.deleteLocalFile(processedName);
        }
    }
}
```

When `Workflow.getVersion()` is run for the new Workflow Execution, it records a marker in the Event History so that all future calls to `getVersion` for this change ID — `checksumAdded` in the example — on this Workflow Execution will always return the given version number, which is `1` in the example.

After all the Workflow Executions prior to version 1 have left retention, you can remove the code for that version.

```java
public void processFile(Arguments args) {
    String localName = null;
    String processedName = null;
    try {
        localName = activities.download(args.getSourceBucketName(), args.getSourceFilename());
        processedName = activities.processFile(localName);
        // getVersion call is left here to ensure that any attempt to replay history
        // for a different version fails. It can be removed later when there is no possibility
        // of this happening.
        Workflow.getVersion("checksumAdded", 1, 1);
        long checksum = activities.calculateChecksum(processedName);
        activities.uploadWithChecksum(
            args.getTargetBucketName(), args.getTargetFilename(), processedName, checksum);
    } finally {
        if (localName != null) { // File was downloaded.
            activities.deleteLocalFile(localName);
        }
        if (processedName != null) { // File was processed.
            activities.deleteLocalFile(processedName);
        }
    }
}
```

### Adding support for versioned Workflow visibility in the Event History

When you invoke `getVersion`, the SDK can record an `UpsertWorkflowSearchAttribute` Event in the history.
This adds a [Search Attribute](/search-attribute#default-search-attribute) named `TemporalChangeVersion` that allows you to filter Workflows based on their version.

#### Enable automatic upsert (Java SDK v1.30.0+)

Starting with Java SDK v1.30.0, you can enable automatic `TemporalChangeVersion` upserts by setting `enableUpsertVersionSearchAttributes` in `WorkflowImplementationOptions`:

```java
WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder()
    .setEnableUpsertVersionSearchAttributes(true)
    .build();

worker.registerWorkflowImplementationTypes(options, YourWorkflowImpl.class);
```

When enabled, each call to `Workflow.getVersion` automatically upserts the `TemporalChangeVersion` Search Attribute with a keyword list of `"<Change ID>-<Version>"` entries.

> **📝 Note:**
>
> This option is backwards compatible and safe to enable on running Workflows.
> However, once enabled, you cannot roll back to a version of the SDK that does not support this option.
>

#### Upsert manually

If you are using a version of the Java SDK earlier than v1.30.0, or prefer to control the upsert yourself, you can set the attribute manually.

```java
import io.temporal.common.SearchAttributeKey;

public static final SearchAttributeKey<List<String>> TEMPORAL_CHANGE_VERSION =
    SearchAttributeKey.forKeywordList("TemporalChangeVersion");
```

Set this attribute when you call `getVersion`:

```java
int version = Workflow.getVersion("MovedThankYouAfterLoop", Workflow.DEFAULT_VERSION, 1);

if (version != Workflow.DEFAULT_VERSION) {
  Workflow.upsertTypedSearchAttributes(TEMPORAL_CHANGE_VERSION
      .valueSet(Arrays.asList("MovedThankYouAfterLoop-" + version)));
}
```

For multiple `getVersion` calls, collect all version changes and set the attribute once:

```java
List<String> list = new ArrayList<String>();
int versionOne = Workflow.getVersion("versionOne", Workflow.DEFAULT_VERSION, 1);
int versionTwo = Workflow.getVersion("versionTwo", Workflow.DEFAULT_VERSION, 1);
if (versionOne != Workflow.DEFAULT_VERSION) {
   list.add("versionOne-" + versionOne);
}
if (versionTwo != Workflow.DEFAULT_VERSION) {
   list.add("versionTwo-" + versionTwo);
}
Workflow.upsertTypedSearchAttributes(TEMPORAL_CHANGE_VERSION.valueSet(list));
```

Patching allows you to make changes to currently running Workflows.
It introduces compatible changes without causing non-determinism errors.

### Workflow cutovers

To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow.

Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes.
To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers.

For example, you would duplicate `PizzaWorkflow` as `PizzaWorkflowV2`:

```java
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;

@WorkflowInterface
public interface PizzaWorkflow {

  @WorkflowMethod
  public OrderConfirmation pizzaWorkflow(PizzaOrder order);
}

public class PizzaWorkflowImpl{

  @Override
  public OrderConfirmation pizzaWorkflow(PizzaOrder order){
      // implementation code omitted for this example
  }
}

@WorkflowInterface
public interface PizzaWorkflowV2 {

  @WorkflowMethod
  public OrderConfirmation pizzaWorkflow(PizzaOrder order);
}

public class PizzaWorkflowImplV2 implements PizzaWorkflowV2{

  @Override
  public OrderConfirmation pizzaWorkflow(PizzaOrder order){
      // implementation code omitted for this example
  }
}
```

It is necessary to create a separate interface because a Workflow Interface can only have one Workflow Method.

You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types:

```java
worker.registerWorkflowImplementationTypes(PizzaWorkflowImpl.class);
worker.registerWorkflowImplementationTypes(PizzaWorkflowImplV2.class);
```

The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow.
This can become impractical over time.
This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching.

### Testing a Workflow for replay safety

To determine whether your Workflow your needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/java/best-practices/testing-suite#replay).

---

# Workflow Streams - Java SDK

Source: https://docs.temporal.io/develop/java/workflows/workflow-streams

> Stream events from a Workflow to subscribers using the Temporal Java SDK Workflow Streams contrib module.

> **Public Preview**

[Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its progress in real time. This page walks through enabling a stream, publishing events from Workflows and Activities, subscribing to a stream with either the blocking iterator or the non-blocking listener, and keeping a stream running across long-lived Workflows.

## Enable streaming on a Workflow

The library ships as the `io.temporal:temporal-workflowstreams` contrib module. Enable streaming by constructing a `WorkflowStream` once via `WorkflowStream.newInstance()`, preferably in a [`@WorkflowInit`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/WorkflowInit.html) constructor. The factory registers the stream's handlers on the current Workflow, and a `@WorkflowInit` constructor runs before any handler dispatch, so polls and offset Queries arriving with the first Workflow Task (for example, from Update-With-Start) are accepted rather than rejected.

```java
import io.temporal.workflow.WorkflowInit;
import io.temporal.workflowstreams.WorkflowStream;
import io.temporal.workflowstreams.WorkflowStreamState;

public class OrderInput {
  public String orderId;
  public WorkflowStreamState streamState;
}

public class OrderWorkflowImpl implements OrderWorkflow {
  private final WorkflowStream stream;

  @WorkflowInit
  public OrderWorkflowImpl(OrderInput input) {
    stream = WorkflowStream.newInstance(input.streamState);
  }

  @Override
  public void execute(OrderInput input) {
    // ... rest of the workflow
  }
}
```

`WorkflowStream.newInstance` creates the in-memory event log and registers the publish Signal, subscribe Update, and offset Query handlers on the current Workflow. The `priorState` argument may be `null` and is only needed for Continue-As-New rollovers. Pass `null` or call the no-argument overload on a fresh start, and the carried `WorkflowStreamState` after a rollover. See [Stream from long-running Workflows](#continue-as-new) for more details.

Constructing the stream at the top of the Workflow method also works. Signals received earlier are buffered by the SDK, but polls and offset Queries are rejected until the stream exists, so use `@WorkflowInit`. Construct exactly one `WorkflowStream` per Workflow.

## Publish from a Workflow

Bind a [topic name](/workflow-streams#topics) with `stream.topic(name)`, then call `publish()` on the returned `WorkflowTopicHandle` to append events. Repeated calls with the same name return the same handle.

```java
import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInit;
import io.temporal.workflowstreams.WorkflowStream;
import io.temporal.workflowstreams.WorkflowTopicHandle;
import java.time.Duration;

public class StatusEvent {
  public String state;
  public int progress;
  public String detail;

  public StatusEvent() {}

  public StatusEvent(String state, int progress, String detail) {
    this.state = state;
    this.progress = progress;
    this.detail = detail;
  }
}

public class OrderWorkflowImpl implements OrderWorkflow {
  private final WorkflowStream stream;
  private final WorkflowTopicHandle status;

  private final OrderActivities activities =
      Workflow.newActivityStub(
          OrderActivities.class,
          ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofMinutes(1)).build());

  @WorkflowInit
  public OrderWorkflowImpl(OrderInput input) {
    stream = WorkflowStream.newInstance(input.streamState);
    status = stream.topic("status");
  }

  @Override
  public void execute(OrderInput input) {
    status.publish(new StatusEvent("validating", 0, "checking inventory"));
    activities.validateOrder(input.orderId);

    status.publish(new StatusEvent("charging", 33, "authorizing payment"));
    activities.chargePayment(input.orderId);

    status.publish(new StatusEvent("shipping", 66, "dispatching to warehouse"));
    activities.dispatchOrder(input.orderId);

    status.publish(new StatusEvent("completed", 100, ""));
  }
}
```

`publish()` runs the payload converter to encode each value. The codec chain (encryption, compression, etc.) runs once on the Signal or Update envelope that carries the batch, never per item, so encryption and compression are applied exactly once each direction.

Unlike the Python and TypeScript SDKs, Java topics carry no per-topic type binding. A topic handle is bound only to a name; published values are `Object` and subscribers decode each item from its raw payload (see [Subscribe](#subscribe)). To customize per-item serialization, pass `WorkflowStreamOptions.newBuilder().setPayloadConverters(...)` to `WorkflowStream.newInstance`, and use the matching converters on the subscriber side.

There is no public accessor for the Worker's configured data converter inside Workflow code, so a custom converter can't be picked up automatically. Pass matching payload converters explicitly to keep Workflow-side publishes consistent with the rest of your Workflow. A pre-built `io.temporal.api.common.v1.Payload` may also be passed to `publish()`, bypassing conversion.

## Publish from a client

Any process that has a Temporal Client and the target Workflow Id can publish to that Workflow's stream by constructing a `WorkflowStreamClient`. This is the general pattern and covers HTTP backends, starters, one-off scripts, other Workflows' Activities, and standalone Activities.

Construct one with:

```java
WorkflowStreamClient.newInstance(workflowClient, workflowId)
```

Then use it the same way you would the Workflow-side handle: bind a topic, publish through it, and let try-with-resources flush on scope exit (`close()` guarantees a final flush of buffered items).

When events originate in an Activity, publish from the Activity directly rather than returning them for the Workflow to forward. The Workflow hosts the stream but doesn't read its own stream. It processes the Activity's return value and emits its own lifecycle events. Keeping Workflow state independent of streamed output is what lets retried Activity attempts surface to subscribers without polluting the Workflow's durable state. See [How events are delivered](/workflow-streams#how-events-are-delivered).

```java
import io.temporal.client.WorkflowClient;
import io.temporal.workflowstreams.TopicHandle;
import io.temporal.workflowstreams.WorkflowStreamClient;
import io.temporal.workflowstreams.WorkflowStreamClientOptions;
import java.time.Duration;

public void publishStatus(WorkflowClient workflowClient, String workflowId) {
  WorkflowStreamClientOptions options =
      WorkflowStreamClientOptions.newBuilder()
          .setBatchInterval(Duration.ofMillis(200))
          .build();
  try (WorkflowStreamClient streamClient =
      WorkflowStreamClient.newInstance(workflowClient, workflowId, options)) {
    TopicHandle status = streamClient.topic("status");
    status.publish(new StatusEvent("started", 0, ""));
    // ...
    // Buffer is flushed automatically on close().
  }
}
```

Inside an Activity scheduled by a Workflow, `WorkflowStreamClient.fromActivity()` infers the Temporal Client and the parent Workflow Id from the Activity context, so you don't have to thread them through the Activity's input:

```java
import io.temporal.activity.Activity;
import io.temporal.workflowstreams.TopicHandle;
import io.temporal.workflowstreams.WorkflowStreamClient;

public class Delta {
  public String text;
}

public void streamDeltas(String orderId) {
  try (WorkflowStreamClient streamClient = WorkflowStreamClient.fromActivity()) {
    TopicHandle deltas = streamClient.topic("delta");
    for (Delta delta : generateDeltas(orderId)) {
      deltas.publish(delta);
      Activity.getExecutionContext().heartbeat(null);
    }
    // Buffer is flushed automatically on close().
  }
}
```

For a [standalone Activity](/develop/java/activities/standalone-activities), there is no parent Workflow context to infer, so `fromActivity()` throws an `IllegalStateException`. Fall back to the general pattern with `Activity.getExecutionContext().getWorkflowClient()` and the target Workflow Id threaded through the Activity's input.

Two operations give the application explicit control over when batches ship: the `forceFlush` argument on a publish for latency, and `client.flush()` for confirmation that prior publications have landed.

Pass `true` as the `forceFlush` argument on a publish to wake the background flusher so the current buffer ships without waiting for the next interval. The flusher only runs while the client is open (between construction and `close()`). The call returns immediately after appending to the buffer and signaling the flusher. It doesn't wait for delivery to the Workflow or to subscribers:

```java
deltas.publish(delta, /* forceFlush */ true);
```

Use it for latency-sensitive events: the first delta of a response so the user sees something fast, or punctuated events like `RETRY` and `STATUS_CHANGE`. See [Tuning](/workflow-streams#tuning) for the trade-off against history pressure.

Use `client.flush()` when you need a mid-stream barrier. Successful completion of the flush is proof that the Temporal server has received all prior publications, so subsequent work that depends on those events being durable can proceed. The client stays open for further publishing afterward. `close()` already flushes on its way out, so the explicit call is only for barriers in the middle:

```java
try (WorkflowStreamClient streamClient =
    WorkflowStreamClient.newInstance(workflowClient, workflowId)) {
  TopicHandle deltas = streamClient.topic("delta");

  for (Delta delta : firstPhase()) {
    deltas.publish(delta);
  }

  streamClient.flush();
  String checkpointId = recordPhaseOneComplete(); // only safe once phase-one events are durable

  for (Delta delta : secondPhase(checkpointId)) {
    deltas.publish(delta);
  }
}
```

Batches also ship early once the buffer reaches `maxBatchSize`, if you set one on `WorkflowStreamClientOptions`; by default only the interval, `forceFlush`, `flush()`, and `close()` trigger a flush.

`publish()` is non-blocking and applies no backpressure. From an Activity or other client, it appends to the client's in-memory buffer and returns. The value goes through the payload converters immediately, so an unconvertible value fails the `publish()` call rather than a later background flush.

From inside a Workflow, it appends synchronously to the in-memory log. [Subscribers](/workflow-streams#subscribing) pull from the Workflow's log on their own schedule, so a slow subscriber doesn't slow down [publishers](/workflow-streams#publishing). If a publisher emits faster than batches can ship to the server, the buffer grows: the process uses more memory, the stream falls further behind real time, and at the limit Signals can't keep up.

If your application needs to bound this (to cap memory, to keep the stream close to real time, or to apply a policy when the publisher overruns the network), apply that policy upstream of `publish()`. The choice (block, drop, error, sample) is application-specific, and Workflow Streams doesn't pick one for you.

## Subscribe

[Subscribing](/workflow-streams#subscribing) uses the same client construction as publishing: `WorkflowStreamClient.newInstance(workflowClient, workflowId)` from any process that has a Temporal Client, or `fromActivity()` inside an Activity. Subscribing from an Activity is less common in practice, so the general client case is the primary example below.

The Java SDK offers two subscriber APIs over one shared poll engine: a blocking iterator for synchronous consumers, and a non-blocking listener that delivers items as callbacks without occupying a caller thread. Neither occupies a thread while a poll is blocked on the server. Polling runs on a small executor shared by all of a client's subscriptions (2 daemon threads by default; see `WorkflowStreamClientOptions.Builder.setPollExecutor`), so many concurrent subscriptions don't mean many threads.

Either way, the subscription ends cleanly when the Workflow reaches a terminal state, automatically follows Continue-As-New chains, recovers from Workflow-side log truncation by restarting from the current base offset, handles pagination when a poll response hits the ~1 MB cap, and also ends when the owning `WorkflowStreamClient` is closed.

Items carry the raw `io.temporal.api.common.v1.Payload` in `item.getPayload()`; decode at the call site with your data converter. Offsets are global across all topics, not per-topic.

### Blocking iterator

`client.subscribe(options)` without a listener returns a `WorkflowStreamSubscription`. A blocking, single-use subscription the consuming thread iterates with a for-each loop. The thread blocks waiting for items while polling still runs on the shared executor.

```java
import io.temporal.client.WorkflowClient;
import io.temporal.common.converter.DefaultDataConverter;
import io.temporal.workflowstreams.SubscribeOptions;
import io.temporal.workflowstreams.WorkflowStreamClient;
import io.temporal.workflowstreams.WorkflowStreamItem;
import io.temporal.workflowstreams.WorkflowStreamSubscription;

public void watchOrder(WorkflowClient workflowClient, String orderId) {
  SubscribeOptions options = SubscribeOptions.newBuilder().setTopics("status").build();
  try (WorkflowStreamClient stream = WorkflowStreamClient.newInstance(workflowClient, orderId);
      WorkflowStreamSubscription subscription = stream.subscribe(options)) {
    for (WorkflowStreamItem item : subscription) {
      StatusEvent evt =
          DefaultDataConverter.STANDARD_INSTANCE.fromPayload(
              item.getPayload(), StatusEvent.class, StatusEvent.class);
      System.out.printf("[%3d%%] %s: %s%n", evt.progress, evt.state, evt.detail);
      if (evt.state.equals("completed")) {
        break;
      }
    }
  }
}
```

`SubscribeOptions` controls the subscription: `setTopics` filters by name (unset means all topics), `setFromOffset` resumes from a stored global offset (zero means the beginning), and `setPollCooldown` sets the minimum interval between polls (default 100 milliseconds). A single-topic convenience method, `streamClient.topic("status").subscribe(fromOffset)`, is equivalent to passing one name in `setTopics`.

`subscription.close()` stops the subscription before the next poll; items already fetched still drain. An unrecoverable poll failure is rethrown from `hasNext()`. A subscriber that never publishes doesn't strictly need to close the `WorkflowStreamClient` for flushing (the background flusher only runs for publishers), but closing it releases the poll executor, so keep both in the try-with-resources as above.

### Non-blocking listener

Unique to the Java SDK, the second subscriber API inverts control. Instead of parking a thread in an iterator, pass a `WorkflowStreamListener` to `subscribe` and items are delivered as callbacks on the poll executor. `subscribe` returns a `WorkflowStreamSubscriptionHandle` immediately. This is the right shape when one process consumes many streams concurrently — all subscriptions share the client's small executor rather than each pinning a thread.

Callbacks are serialized and are never invoked concurrently and must not block. The `CompletionStage` returned by `onNext` is the backpressure boundary. Return `null` or an already-completed stage to receive the next item immediately, or a pending stage to defer both further delivery and the next poll until it completes. A stage that completes exceptionally or an exception thrown directly from `onNext` stops the subscription and is reported to `onError`.

```java
import io.temporal.common.converter.DefaultDataConverter;
import io.temporal.workflowstreams.SubscribeOptions;
import io.temporal.workflowstreams.WorkflowStreamItem;
import io.temporal.workflowstreams.WorkflowStreamListener;
import io.temporal.workflowstreams.WorkflowStreamSubscriptionHandle;
import java.util.concurrent.CompletionStage;

SubscribeOptions options = SubscribeOptions.newBuilder().setTopics("status").build();
WorkflowStreamSubscriptionHandle handle =
    streamClient.subscribe(
        options,
        new WorkflowStreamListener() {
          @Override
          public CompletionStage<Void> onNext(WorkflowStreamItem item) {
            StatusEvent evt =
                DefaultDataConverter.STANDARD_INSTANCE.fromPayload(
                    item.getPayload(), StatusEvent.class, StatusEvent.class);
            System.out.printf(
                "offset=%d topic=%s state=%s%n", item.getOffset(), item.getTopic(), evt.state);
            return null; // or a pending stage to apply backpressure
          }

          @Override
          public void onCompleted() {
            System.out.println("stream ended");
          }
        });

// The calling thread is free; wait on the handle when you need to.
handle.getDoneFuture().join();
```

`onCompleted` fires once when the stream ends cleanly because the Workflow reached a terminal state. `onError` fires once on an unrecoverable failure, including a failure from `onNext`. Its default implementation logs at warn level. `handle.close()` stops the subscription before the next poll without calling `onCompleted`. `handle.getDoneFuture()` completes normally when the stream ends cleanly or the subscription is closed, and exceptionally with the failure passed to `onError`. A single-topic convenience, `streamClient.topic("status").subscribe(fromOffset, listener)`, is also available.

To hand work off a callback without blocking the executor, return a stage that completes when the work is done. For example, `CompletableFuture.runAsync(() -> render(item), renderExecutor)` and the next item is delivered only after it completes.

### Heterogeneous topics

Every item arrives as a raw `Payload` in `item.getPayload()`, so a single subscription naturally consumes multiple topics whose payload types differ. Pass the topic names to `SubscribeOptions.Builder.setTopics` (or leave it unset for every topic on the stream), dispatch on `item.getTopic()`, and decode into the matching type:

```java
SubscribeOptions options =
    SubscribeOptions.newBuilder().setTopics("status", "progress").build();
try (WorkflowStreamSubscription subscription = stream.subscribe(options)) {
  for (WorkflowStreamItem item : subscription) {
    switch (item.getTopic()) {
      case "status":
        StatusEvent status =
            DefaultDataConverter.STANDARD_INSTANCE.fromPayload(
                item.getPayload(), StatusEvent.class, StatusEvent.class);
        System.out.printf("[status] %s: %s%n", status.state, status.detail);
        break;
      case "progress":
        ProgressEvent progress =
            DefaultDataConverter.STANDARD_INSTANCE.fromPayload(
                item.getPayload(), ProgressEvent.class, ProgressEvent.class);
        System.out.printf("[progress] %s%n", progress.message);
        break;
    }
  }
}
```

A single subscription over multiple topics also avoids the cancellation race that two concurrent subscribers would create. Because `item.getPayload()` is the raw payload, it's also the right shape when you want to forward the bytes through to another system without decoding them.

### Closing the stream 

A subscriber's loop or listener doesn't know when the publisher is done. How you [close a stream](/workflow-streams#closing-the-stream) depends on what the application needs. As one example, a common pattern combines two pieces:

1. **An in-band terminator.** The Workflow or its Activity publishes a sentinel event the subscriber recognizes and breaks on. In the `watchOrder` example above, `new StatusEvent("completed", 100, "")` is the minimal form, and the consumer's `if (evt.state.equals("completed")) break` is the matching half. Each subscription decides what its own end-of-stream marker is.
2. **A brief overlap before the Workflow returns.** A poll Update that is still in flight when the Workflow returns is consumed by the subscription transparently, and no new polls can complete after that. If the Workflow returns immediately after publishing the terminator, subscribers may miss it.

There are two ways to provide that overlap.

- [Fixed sleep](/workflow-streams#fixed-sleep). Sleep between the terminator and the return so any in-flight poll has time to fetch the terminator before the Workflow exits:

  ```java
  // at the end of the workflow method
  status.publish(new StatusEvent("completed", 100, ""));
  Workflow.sleep(Duration.ofSeconds(30));
  return result;
  ```

- [Acknowledgment handshake](/workflow-streams#acknowledgment-handshake). The subscriber sends a Signal once it has the terminator; the Workflow waits up to a timeout, returning as soon as the ack arrives:

  ```java
  public class ChatWorkflowImpl implements ChatWorkflow {
    private boolean subscriberDone = false;

    @Override // annotated with @SignalMethod on the workflow interface
    public void subscriberAcknowledgedTerminator() {
      subscriberDone = true;
    }

    @Override
    public String complete(ChatInput input) {
      // ... do work and publish events ...

      // Returns true if the ack arrived, false on timeout. Either way, fall through.
      Workflow.await(Duration.ofSeconds(30), () -> subscriberDone);
      return result;
    }
  }
  ```

The full pattern is wired into the [Stream LLM output](#stream-llm-output) example below.

You can [inspect the terminal status](/workflow-streams#inspecting-terminal-status). A subscription ends cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELED`, `TERMINATED`, or `TIMED_OUT`, but doesn't distinguish among them. If your application needs to know which (to display success or failure to the user, log the outcome, or decide whether to retry), call `workflowClient.newUntypedWorkflowStub(workflowId).describe()` after the subscription ends to inspect the Workflow's status.

## Stream from long-running Workflows 

Workflows that run for hours or accumulate thousands of events need to periodically roll over via [Continue-As-New](/workflow-streams#stream-from-long-running-workflows) to keep history bounded. Subscribers automatically follow these rollovers. To keep a stream running across them without subscribers seeing a gap, carry both your application state and the stream state across the boundary.

Add a `WorkflowStreamState` field to your Workflow input, pass it to `WorkflowStream.newInstance`, and call `stream.continueAsNew(buildArgs)` to invoke the rollover. The helper drains waiting subscribers, waits for in-flight handlers to finish, snapshots the stream state, then Continues-As-New with the arguments built by `buildArgs(postDrainState)`. It never returns:

```java
public class WorkflowInput {
  public int itemsProcessed;
  public WorkflowStreamState streamState;
}

public class LongRunningWorkflowImpl implements LongRunningWorkflow {
  private final WorkflowStream stream;
  private int itemsProcessed;

  @WorkflowInit
  public LongRunningWorkflowImpl(WorkflowInput input) {
    stream = WorkflowStream.newInstance(input.streamState);
    itemsProcessed = input.itemsProcessed;
  }

  @Override
  public void execute(WorkflowInput input) {
    while (true) {
      doOneIteration();
      itemsProcessed++;

      if (Workflow.getInfo().isContinueAsNewSuggested()) {
        stream.continueAsNew(
            state -> {
              WorkflowInput next = new WorkflowInput();
              next.itemsProcessed = itemsProcessed; // your own state, carried across
              next.streamState = state;             // the captured stream state
              return new Object[] {next};
            });
      }
    }
  }
}
```

The `streamState` field is `null` on a fresh start and a populated snapshot after a rollover. The `buildArgs` callback receives the post-detach `WorkflowStreamState` as its only argument, so the snapshot is guaranteed to happen *after* pollers detach.

To pass other Continue-As-New parameters such as a different Task Queue, or to use a custom publisher TTL, use the explicit recipe instead. Drain the pollers, wait for handlers to finish, snapshot the state with your chosen TTL, then call `Workflow.continueAsNew` yourself:

```java
import io.temporal.workflow.ContinueAsNewOptions;

stream.detachPollers();
Workflow.await(() -> Workflow.isEveryHandlerFinished());
WorkflowStreamState state = stream.getState(Duration.ofMinutes(30)); // custom publisher TTL
WorkflowInput next = new WorkflowInput();
next.itemsProcessed = itemsProcessed;
next.streamState = state;
Workflow.continueAsNew(
    ContinueAsNewOptions.newBuilder().setTaskQueue("other-tq").build(), next);
```

The carried `WorkflowStreamState` includes the entire in-memory log of the previous run, so streams that carry large items can hit Temporal's per-payload size limit at the rollover. Offload the bytes via [External Storage](/external-storage) so each item is a small reference rather than the full payload, and combine that with `stream.truncate(upToOffset)` to keep the carried log itself small.

## Deduplication window

See [How events are delivered](/workflow-streams#how-events-are-delivered) for more details on subscriber and publisher behavior. See [Tuning](/workflow-streams#tuning) for more details on how to change your settings to meet the requirements for your Workflow Streams.

There are two limits on the [deduplication window](/workflow-streams#deduplication-window) worth highlighting:

- **Publisher TTL.** At each Continue-As-New, deduplication entries whose last-seen time is older than this are dropped. The last-seen time is updated on each *successful* publish (not on each retry attempt), so a publisher that retries through a long partition without success can still age out. A publisher that returns after a longer pause may produce a duplicate. `stream.continueAsNew(...)` snapshots with a 15-minute default; to tune it, use the explicit recipe above and pass your value to `getState(publisherTtl)`.
- **`maxRetryDuration`.** A `WorkflowStreamClient` retries a failed batch for up to this long (default 10 minutes). If the duration elapses with the batch still pending, the client gives up, the pending batch is dropped, and a `FlushTimeoutException` is raised.

  ```java
  WorkflowStreamClientOptions.newBuilder()
      .setMaxRetryDuration(Duration.ofMinutes(10))
      .build();
  ```

  On timeout, the dropped batch is at-most-once: it may or may not have reached the Workflow. One operational caveat: the `FlushTimeoutException` is raised from inside the background flusher and terminates it. Until you call `client.flush()` or `client.close()` — which surface the deferred exception — subsequent publishes accumulate in the buffer with no flusher to ship them. `maxRetryDuration` must be less than the Workflow's publisher TTL to preserve exactly-once delivery.

## Best practices

There are a few details to note if you're writing custom message handlers or testing the library's capabilities:

- **Construct exactly one `WorkflowStream` per Workflow, preferably in `@WorkflowInit`.** The factory registers the publish Signal, poll Update, and offset Query handlers on the current Workflow. A `@WorkflowInit` constructor runs before any handler dispatch, so polls and offset Queries arriving with the first Workflow Task are accepted; construction at the top of the Workflow method leaves them rejected until the stream exists.
- **`item.getPayload()` is always the raw payload.** Decode it with a converter built from the same payload converters used by the publisher. When publishers and subscribers both rely on the defaults, `DefaultDataConverter.STANDARD_INSTANCE` matches on both sides. If you pass `setPayloadConverters` on the Workflow side or the client side, build a matching converter on the subscriber side.
- **The codec chain runs once on the envelope.** Payload codecs (encryption, compression) configured on the Temporal client run on the Signal or Update envelope that carries each batch, never per item, so items are never double-encoded. `setPayloadConverters` handles only per-item serialization; its `PayloadConverter[]` type makes it impossible to slot a codec in per item.
- **Listener callbacks must not block.** They run serialized on the client's poll executor, which drives every subscription on the client. Blocking in `onNext` stalls the client's other subscriptions. Hand slow work to your own executor and return the resulting `CompletionStage` for backpressure.
- **Size the poll executor for slow Workflows.** The default executor has 2 daemon threads, is created lazily, and is owned (and shut down) by the client; a user-supplied executor is never shut down by the client. The executor runs the short update-admission and delivery steps and poll cooldowns, never the long poll itself, so a small pool serves many subscriptions. The known worst case for pool pressure is a backlogged Workflow pinning a thread in the update-admission call; supply a bigger pool via `setPollExecutor` when running many subscriptions against slow Workflows.
- **Cross-language interop depends on the configured data converter.** The handler names, JSON envelope field names, and per-item payload encoding match the other SDKs' packages exactly, so a Java publisher or subscriber interoperates with a Workflow written in any of them and vice versa. One Java-specific caveat: the protocol envelope types are serialized by the Workflow's and client's *configured* data converter. The default Jackson JSON converter produces the wire-compatible snake_case field names; if you configure a non-Jackson JSON converter, it must produce the same field names for cross-language interop.

## Example: Stream LLM output 

The headline use case fits the publish/subscribe shapes documented above. An Activity calls the model and publishes deltas as they arrive. The Workflow starts the Activity and waits for the consumer to acknowledge end-of-stream. The consumer subscribes, accumulates the deltas, and clears its accumulated state on `RETRY` before continuing. The shape works for a terminal client, a desktop UI, or a Server-Sent Events (SSE) endpoint forwarding to a browser. Anything that holds the displayed state calls `render()` to display it.

If your Activity can retry, the consumer side has to account for it. A retried attempt is a fresh publisher, so its output appears in the stream alongside the output from the previous attempt. In the LLM streaming pattern below, that means the failed attempt's partial deltas and the retried attempt's full output both reach a subscribed UI unless the UI resets on a `RETRY` event. The example wires up that pattern. See [How events are delivered](/workflow-streams#how-events-are-delivered) for the precise guarantees.

**LlmActivitiesImpl.java**

```java
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.http.StreamResponse;
import com.openai.models.chat.completions.ChatCompletionChunk;
import com.openai.models.chat.completions.ChatCompletionCreateParams;
import io.temporal.activity.Activity;
import io.temporal.workflowstreams.TopicHandle;
import io.temporal.workflowstreams.WorkflowStreamClient;
import io.temporal.workflowstreams.WorkflowStreamClientOptions;
import java.time.Duration;
import java.util.concurrent.atomic.AtomicBoolean;

public class TextDelta {
  public String text;

  public TextDelta() {}

  public TextDelta(String text) {
    this.text = text;
  }
}

public class RetryEvent {
  public int attempt;

  public RetryEvent() {}

  public RetryEvent(int attempt) {
    this.attempt = attempt;
  }
}

public class CloseEvent {}

public class LlmActivitiesImpl implements LlmActivities {
  @Override
  public String streamCompletion(String prompt) {
    WorkflowStreamClientOptions options =
        WorkflowStreamClientOptions.newBuilder()
            .setBatchInterval(Duration.ofMillis(200))
            .build();
    try (WorkflowStreamClient streamClient = WorkflowStreamClient.fromActivity(options)) {
      TopicHandle deltas = streamClient.topic("delta");
      TopicHandle retry = streamClient.topic("retry");
      TopicHandle close = streamClient.topic("close");

      // Tell consumers an earlier attempt's deltas are stale.
      int attempt = Activity.getExecutionContext().getInfo().getAttempt();
      if (attempt > 1) {
        retry.publish(new RetryEvent(attempt), /* forceFlush */ true);
      }

      // Disable provider-side retries; let Temporal own retry policy at the Activity layer.
      OpenAIClient openai = OpenAIOkHttpClient.builder().fromEnv().maxRetries(0).build();
      ChatCompletionCreateParams params =
          ChatCompletionCreateParams.builder().model("gpt-4o-mini").addUserMessage(prompt).build();

      StringBuilder full = new StringBuilder();
      AtomicBoolean first = new AtomicBoolean(true);
      try (StreamResponse<ChatCompletionChunk> stream =
          openai.chat().completions().createStreaming(params)) {
        stream.stream()
            .forEach(
                chunk ->
                    chunk.choices().stream()
                        .findFirst()
                        .flatMap(choice -> choice.delta().content())
                        .filter(text -> !text.isEmpty())
                        .ifPresent(
                            text -> {
                              // forceFlush only on the first delta so the user sees something
                              // immediately; subsequent deltas batch at the 200 ms interval.
                              deltas.publish(new TextDelta(text), first.getAndSet(false));
                              full.append(text);
                            }));
      }
      close.publish(new CloseEvent());
      return full.toString();
    }
  }
}
```

**ChatWorkflowImpl.java**

```java
import io.temporal.activity.ActivityOptions;
import io.temporal.workflow.SignalMethod;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInit;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import io.temporal.workflowstreams.WorkflowStream;
import java.time.Duration;

@WorkflowInterface
public interface ChatWorkflow {
  @WorkflowMethod
  String complete(ChatInput input);

  @SignalMethod
  void subscriberAcknowledgedTerminator();
}

public class ChatWorkflowImpl implements ChatWorkflow {
  private final LlmActivities activities =
      Workflow.newActivityStub(
          LlmActivities.class,
          ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofMinutes(5)).build());

  private boolean subscriberDone = false;

  @WorkflowInit
  public ChatWorkflowImpl(ChatInput input) {
    WorkflowStream.newInstance(input.streamState);
  }

  @Override
  public void subscriberAcknowledgedTerminator() {
    subscriberDone = true;
  }

  @Override
  public String complete(ChatInput input) {
    String result = activities.streamCompletion(input.prompt);

    // Wait for the subscriber to ack the terminal `close` event. The timeout
    // is a fallback for when no subscriber is attached; with the ack, the
    // typical case exits as soon as the subscriber confirms.
    Workflow.await(Duration.ofSeconds(30), () -> subscriberDone);
    return result;
  }
}
```

**Consumer.java**

```java
import io.temporal.client.WorkflowClient;
import io.temporal.common.converter.DefaultDataConverter;
import io.temporal.workflowstreams.SubscribeOptions;
import io.temporal.workflowstreams.WorkflowStreamClient;
import io.temporal.workflowstreams.WorkflowStreamItem;
import io.temporal.workflowstreams.WorkflowStreamSubscription;

public String streamChat(WorkflowClient workflowClient, String chatId) {
  StringBuilder output = new StringBuilder();

  Runnable render =
      () -> {
        // ... display the accumulated output (terminal redraw, UI update, etc.)
      };

  SubscribeOptions options =
      SubscribeOptions.newBuilder().setTopics("delta", "retry", "close").build();
  try (WorkflowStreamClient stream = WorkflowStreamClient.newInstance(workflowClient, chatId);
      WorkflowStreamSubscription subscription = stream.subscribe(options)) {
    for (WorkflowStreamItem item : subscription) {
      switch (item.getTopic()) {
        case "retry":
          // Earlier attempt's deltas are stale; drop what we've shown.
          output.setLength(0);
          render.run();
          break;
        case "delta":
          TextDelta delta =
              DefaultDataConverter.STANDARD_INSTANCE.fromPayload(
                  item.getPayload(), TextDelta.class, TextDelta.class);
          output.append(delta.text);
          render.run();
          break;
        case "close":
          // Acknowledge so the Workflow can return without waiting on the fallback timeout.
          workflowClient
              .newUntypedWorkflowStub(chatId)
              .signal("subscriberAcknowledgedTerminator");
          return output.toString();
      }
    }
  }

  return output.toString();
}
```

A few choices in this shape are deliberate:

- The Activity is the publisher because it owns the non-deterministic LLM call. The Workflow processes only the Activity's return value, never reading its own stream. See [Publish from a client](#publish-from-a-client) for why.
- The Activity publishes a `RETRY` event when `Activity.getExecutionContext().getInfo().getAttempt() > 1`. This lets the UI respond appropriately to the failure, typically by clearing accumulated deltas before the next attempt's deltas arrive (see [How events are delivered](/workflow-streams#how-events-are-delivered)).
- Termination uses an *ack handshake*: the consumer signals the Workflow once it has received the `close` event, so the Workflow can return as soon as the subscriber confirms. The `Workflow.await` timeout is the fallback when no subscriber is attached (see [Closing the stream](#closing-the-stream) for the simpler fixed-sleep alternative).
- `forceFlush` is `true` only on the first delta and on the `RETRY` sentinel, where latency matters. Subsequent deltas batch at the 200 ms `batchInterval`; per-delta `forceFlush` would generate one Signal per token (see [Tuning](/workflow-streams#tuning) for the trade-off).

## See also

- [Workflow Streams samples (samples-java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/workflowstreams): six runnable scenarios covering basic publish/subscribe, the non-blocking listener, reconnecting subscribers, external publishers, bounded logs, and LLM streaming.
- [`temporal-workflowstreams` API reference](https://javadoc.io/doc/io.temporal/temporal-workflowstreams).
- [Workflow message passing](/develop/java/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on.
- [Converters and encryption](/develop/java/best-practices/data-handling): converters and codecs.

---

# PHP SDK developer guide

Source: https://docs.temporal.io/develop/php

> Explore Temporal PHP SDK guides to master features for Temporal Applications. Learn Workflows, Activities, Testing, Failure Detection, Messages, Observability, and more.

![PHP SDK Banner](https://docs.temporal.io/img/assets/banner-php-temporal.png)

## Install and get started

You can find detailed installation instructions for the PHP SDK in the [Quickstart](/develop/php/set-up-your-local-php).

There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running.

Once your local Temporal Service is set up, continue building with the following resources:

- [Activity Basics](/develop/php/activities/basics)
- [Workflow basics](/develop/php/workflows/basics)
- [Start an Activity Execution](/develop/php/activities/execution)
- [Run a Worker](/develop/php/workers/run-worker-process)

From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases.

## [Workflows](/develop/php/workflows)

- [Workflow basics](/develop/php/workflows/basics)
- [Child Workflows](/develop/php/workflows/child-workflows)
- [Continue-As-New](/develop/php/workflows/continue-as-new)
- [Cancellation](/develop/php/workflows/cancellation)
- [Timeouts](/develop/php/workflows/timeouts)
- [Message passing](/develop/php/workflows/message-passing)
- [Schedules](/develop/php/workflows/schedules)
- [Timers](/develop/php/workflows/timers)
- [Side effects](/develop/php/workflows/side-effects)
- [Versioning](/develop/php/workflows/versioning)

## [Activities](/develop/php/activities)

- [Activity Basics](/develop/php/activities/basics)
- [Activity Execution](/develop/php/activities/execution)
- [Timeouts](/develop/php/activities/timeouts)
- [Asynchronous Activity Completion](/develop/php/activities/asynchronous-activity)

## [Workers](/develop/php/workers)

- [Run a Worker](/develop/php/workers/run-worker-process)

## [Temporal Client](/develop/php/client)

- [Temporal Client](/develop/php/client/temporal-client)

## [Platform](/develop/php/platform)

- [Observability](/develop/php/platform/observability)
- [Enriching the UI](/develop/php/platform/enriching-ui)

## [Best practices](/develop/php/best-practices)

- [Testing](/develop/php/best-practices/testing-suite)
- [Debugging](/develop/php/best-practices/debugging)

## Temporal PHP technical resources

- [PHP SDK Quickstart - Setup Guide](/develop/php/set-up-your-local-php)
- [PHP API Documentation](https://php.temporal.io)
- [PHP SDK Code Samples](https://github.com/temporalio/samples-php)
- [PHP SDK GitHub](https://github.com/temporalio/sdk-php)

## Get connected with the Temporal PHP community

- [Temporal PHP Community Slack](https://temporalio.slack.com/archives/C01LK9FAMM0)
- [PHP SDK Forum](https://community.temporal.io/tag/php-sdk)

---

# Activities - PHP SDK

Source: https://docs.temporal.io/develop/php/activities

> Activity Definitions, Activity Execution, Activity Timeouts, and Asynchronous Activity Completion in the Temporal PHP SDK.

![PHP SDK Banner](https://docs.temporal.io/img/assets/banner-php-temporal.png)

## Activities

- [Activity basics](/develop/php/activities/basics)
- [Activity Execution](/develop/php/activities/execution)
- [Timeouts](/develop/php/activities/timeouts)
- [Asynchronous Activity Completion](/develop/php/activities/asynchronous-activity)

---

# Asynchronous Activity Completion - PHP SDK

Source: https://docs.temporal.io/develop/php/activities/asynchronous-activity

## Asynchronously complete an Activity 

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function
to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace,
   Workflow Id, and Activity Id.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The Temporal Client is used to Heartbeat and complete the Activity.

Sometimes Workflows need to perform certain operations in parallel.

Invoking activity stub without the use of `yield` will return the Activity result promise which can be resolved at later
moment. Calling `yield` on promise blocks until a result is available.

> Activity promise also exposes `then` method to construct promise chains. Read more about Promises
> [here](https://github.com/reactphp/promise).

Alternatively you can explicitly wrap your code (including `yield` constructs) using `Workflow::async` which will
execute nested code in parallel with main Workflow code. Call `yield` on Promise returned by `Workflow::async` to merge
execution result back to primary Workflow method.

```php
public function greet(string $name): \Generator
{
    // Workflow::async runs it's activities and child workflows in a separate coroutine. Use keyword yield to merge
    // it back to parent process.

    $first = Workflow::async(
        function () use ($name) {
            $hello = yield $this->greetingActivity->composeGreeting('Hello', $name);
            $bye = yield $this->greetingActivity->composeGreeting('Bye', $name);

            return $hello . '; ' . $bye;
        }
    );

    $second = Workflow::async(
        function () use ($name) {
            $hello = yield $this->greetingActivity->composeGreeting('Hola', $name);
            $bye = yield $this->greetingActivity->composeGreeting('Chao', $name);

            return $hello . '; ' . $bye;
        }
    );

    // blocks until $first and $second complete
    return (yield $first) . "\n" . (yield $second);
}
```

**Async completion**

There are certain scenarios when moving on from an Activity upon completion of its function is not possible or
desirable. For example, you might have an application that requires user input to complete the Activity. You could
implement the Activity with a polling mechanism, but a simpler and less resource-intensive implementation is to
asynchronously complete a Temporal Activity.

There are two parts to implementing an asynchronously completed Activity:

1. The Activity provides the information necessary for completion from an external system and notifies the Temporal
   service that it is waiting for that outside callback.
2. The external service calls the Temporal Service to complete the Activity.

The following example demonstrates the first part:

<!--SNIPSTART samples-php-async-activity-completion-activity-class-->
[app/src/AsyncActivityCompletion/GreetingActivity.php](https://github.com/temporalio/samples-php/blob/main/app/src/AsyncActivityCompletion/GreetingActivity.php)
```php
class GreetingActivity implements GreetingActivityInterface
{
    private LoggerInterface $logger;

    public function __construct()
    {
        $this->logger = new Logger();
    }
    /**
     * Demonstrates how to implement an Activity asynchronously.
     * When {@link Activity::doNotCompleteOnReturn()} is called,
     * the Activity implementation function that returns doesn't complete the Activity.
     */
    public function composeGreeting(string $greeting, string $name): string
    {
        // In real life this request can be executed anywhere. By a separate service for example.
        $this->logger->info(sprintf('GreetingActivity token: %s', base64_encode(Activity::getInfo()->taskToken)));
        // Send the taskToken to the external service that will complete the Activity.
        // Return from the Activity a function indicating that Temporal should wait
        // for an async completion message.
        Activity::doNotCompleteOnReturn();
        // When doNotCompleteOnReturn() is invoked the return value is ignored.
        return 'ignored';
    }
}
```
<!--SNIPEND-->

The following code completes the Activity successfully using `WorkflowClient`:

<!--SNIPSTART samples-php-async-activity-completion-completebytoken-->
[app/src/AsyncActivityCompletion/CompleteCommand.php](https://github.com/temporalio/samples-php/blob/main/app/src/AsyncActivityCompletion/CompleteCommand.php)
```php
$client = $this->workflowClient->newActivityCompletionClient();
// Complete the Activity.
$client->completeByToken(
    base64_decode($input->getArgument('token')),
    $input->getArgument('message')
);
```
<!--SNIPEND-->

To fail the Activity, you would do the following:

```php
// Fail the Activity.
$activityClient->completeExceptionallyByToken($taskToken, new \Error("activity failed"));
```

---

# Activity basics - PHP SDK

Source: https://docs.temporal.io/develop/php/activities/basics

> Implement an Activity behind a PHP interface, define its parameters and return values, and set a custom name for the Activity Type.

## Develop a basic Activity 

One of the primary things that Workflows do is orchestrate the execution of Activities.
An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file.
An Activity can interact with the world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service.
For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition).

Activities are defined as methods of a plain PHP interface annotated with `#[ActivityInterface]`.

Following is an example of an interface that defines four Activities:

```php
#[ActivityInterface]
// Defining an interface for the activities.
interface FileProcessingActivities
{
    public function upload(string $bucketName, string $localName, string $targetName): void;

    #[ActivityMethod("transcode_file")]
    public function download(string $bucketName, string $remoteName): void;

    public function processFile(): string;

    public function deleteLocalFile(string $fileName): void;
}
```

### Define Activity parameters 

There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support.
However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB.
And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB.

Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance.
This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task).

Some SDKs require that you pass context objects, others do not.
When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities.
This is so that you can change what data is passed to the Activity without breaking a function or method signature.

Each method defines a single Activity type.
A single Workflow can use more than one Activity interface and call more than one Activity method from the same interface.

The only requirement is that Activity method arguments and return values are serializable to a byte array using the provided [DataConverter](https://github.com/temporalio/sdk-php/blob/master/src/DataConverter/DataConverterInterface.php) interface.
The default implementation uses a JSON serializer, but an alternative implementation can be easily configured.

### Define Activity return values 

All data returned from an Activity must be serializable.

Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/evaluate/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history).

Return values must be serializable to a byte array using the provided [DataConverter](https://github.com/temporalio/sdk-php/blob/master/src/DataConverter/DataConverterInterface.php) interface.
The default implementation uses a JSON serializer, but an alternative implementation can be easily configured.
Thus, you can return both primitive types:

```php
class GreetingActivity implements GreetingActivityInterface
{
    public function composeGreeting(string $greeting, string $name): string
    {
        return $greeting . ' ' . $name;
    }
}
```

And objects:

```php
class GreetingActivity implements GreetingActivityInterface
{
    public function composeGreeting(string $greeting, string $name): Greeting
    {
        return new Greeting($greeting, $name);
    }
}
```

### Customize your Activity Type 

Activities have a Type that are referred to as the Activity name.
The following examples set a custom name for your Activity Type.

An optional `#[ActivityMethod]` attribute can be used to override a default Activity name.

You can define your own prefix for all Activity names by adding the `prefix` option to the `ActivityInterface` attribute.
(The default prefix is empty.)

```php
#[ActivityInterface("file_activities.")]
interface FileProcessingActivities
{
    public function upload(string $bucketName, string $localName, string $targetName);

    #[ActivityMethod("transcode_file")]
    public function download(string $bucketName, string $remoteName);

    public function processFile(): string;

    public function deleteLocalFile(string $fileName);
}
```

The `#[ActivityInterface("file_activities.")]` is an attribute that tells the PHP SDK to generate a class to implement the `FileProcessingActivities` interface. The functions define Activities that are used in the Workflow.

---

# Activity Execution - PHP SDK

Source: https://docs.temporal.io/develop/php/activities/execution

> Call an Activity from a PHP Workflow through a stub created with newActivityStub, set the required Timeout, and get the Activity result.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition).
The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command.
This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History.

A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations.
Activity implementation code should be _idempotent_.

The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history.
The entire Execution history is transferred from the Temporal Service to Workflow Workers when a Workflow state needs to recover.
A large Execution history can thus adversely impact the performance of your Workflow.

Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values.
Otherwise, no additional limitations exist on Activity implementations.

Activity implementation is an implementation of an Activity interface.
The following code example, uses a constructor that takes an Amazon S3 client and a local directory, and uploads a file to the S3 bucket.
Then, the code uses a function to download a file from the S3 bucket passing a bucket name, remote name, and local name as arguments.
Finally, it uses a function that takes a local file name as an argument and returns a string.

```php
// An implementation of an Activity interface.
class FileProcessingActivitiesImpl implements FileProcessingActivities {

    private S3Client $s3Client;

    private string $localDirectory;

    public function __construct(S3Client $s3Client, string $localDirectory) {
        $this->s3Client = $s3Client;
        $this->localDirectory = $localDirectory;
    }

    // Uploading a file to S3.
    public function upload(string $bucketName, string $localName, string $targetName): void
    {
        $this->s3Client->putObject(
            $bucketName,
            $targetName,
            fopen($this->localDirectory . $localName, 'rb+')
        );
    }

// Downloading a file from S3.
    public function download(
        string $bucketName,
        string $remoteName,
        string $localName
    ): void
    {
        $this->s3Client->downloadObject(
            $bucketName,
            $remoteName,
            fopen($this->localDirectory .$localName, 'wb+')
        );
    }

// A function that takes a local file name as an argument and returns a string.
    public function processFile(string $localName): string
    {
        // Implementation omitted for brevity.
        return compressFile($this->localDirectory . $localName);
    }

    public function deleteLocalFile(string $fileName): void
    {
        unlink($this->localDirectory . $fileName);
    }
}
```

### Set the required Activity Timeouts 

Activity Execution semantics rely on several parameters.
The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).
These values are set in the Activity Options.

### Get the results of an Activity Execution 

The call to spawn an [Activity Execution](/activity-execution) generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable.
Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available.

`Workflow::newActivityStub`returns a client-side stub an implements an Activity interface. The client-side stub can be used within the Workflow code. It takes the Activity's type and`ActivityOptions` as arguments.

Calling (via `yield`) a method on this interface invokes an Activity that implements this method.
An Activity invocation synchronously blocks until the Activity completes, fails, or times out.
Even if Activity Execution takes a few months, the Workflow code still sees it as a single synchronous invocation.
It doesn't matter what happens to the processes that host the Workflow.
The business logic code just sees a single method call.

```php
class GreetingWorkflow implements GreetingWorkflowInterface
{
    private $greetingActivity;

    public function __construct()
    {
        $this->greetingActivity = Workflow::newActivityStub(
            GreetingActivityInterface::class,
            ActivityOptions::new()->withStartToCloseTimeout(\DateInterval::createFromDateString('30 seconds'))
        );
    }

    public function greet(string $name): \Generator
    {
        // This is a blocking call that returns only after the activity has completed.
        return yield $this->greetingActivity->composeGreeting('Hello', $name);
    }
}
```

If different Activities need different options, like timeouts or a task queue, multiple client-side stubs can be created with different options.

```php
$greetingActivity = Workflow::newActivityStub(
    GreetingActivityInterface::class,
    ActivityOptions::new()->withStartToCloseTimeout(\DateInterval::createFromDateString('30 seconds'))
);

$greetingActivity = Workflow::newActivityStub(
    GreetingActivityInterface::class,
    ActivityOptions::new()->withStartToCloseTimeout(\DateInterval::createFromDateString('30 minutes'))
);
```

---

# Activity Timeouts - PHP SDK

Source: https://docs.temporal.io/develop/php/activities/timeouts

> Set the Schedule-To-Close, Start-To-Close, and Heartbeat Timeouts on a PHP Activity, tune its Retry Policy, and override the next retry delay.

## Set Activity timeouts 

Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution.

The following timeouts are available in the Activity Options.

- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution).
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution).
- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task.  This timeout is non-retryable by design.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

Because Activities are reentrant, only a single stub can be used for multiple Activity invocations.

Available timeouts are:

- withScheduleToCloseTimeout()
- withStartToCloseTimeout()
- withScheduleToStartTimeout()

```php
$this->greetingActivity = Workflow::newActivityStub(
    GreetingActivityInterface::class,
    // Set Activity Timeout duration
    ActivityOptions::new()
        ->withScheduleToCloseTimeout(CarbonInterval::seconds(2))
        // ->withStartToCloseTimeout(CarbonInterval::seconds(2))
        // ->withScheduleToStartTimeout(CarbonInterval::seconds(10))
);
```

### Set an Activity Retry Policy 

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided.

To set an Activity Retry, set `{@link RetryOptions}` on `{@link ActivityOptions}`.
The follow example creates a new Activity with the given options.

```php
$this->greetingActivity = Workflow::newActivityStub(
    GreetingActivityInterface::class,
    ActivityOptions::new()
        ->withScheduleToCloseTimeout(CarbonInterval::seconds(10))
        ->withRetryOptions(
            RetryOptions::new()
                ->withInitialInterval(CarbonInterval::seconds(1))
                ->withMaximumAttempts(5)
                ->withNonRetryableExceptions([\InvalidArgumentException::class])
        )
);
}
```

For an executable code sample, see [ActivityRetry sample](https://github.com/temporalio/samples-php/tree/master/app/src/ActivityRetry) in the PHP samples repository.

### Set the required Activity Timeouts 

Activity Execution semantics rely on several parameters.
The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).
These values are set in the Activity Options.

### Activity next retry delay 

You may throw an [`ApplicationFailure`](/references/failures#application-failure) with the `nextRetryDelay` field set.
This value will replace and override whatever the retry interval would be on the retry policy.

For example, if in an Activity, you want to base the interval on the number of attempts, you might do:

```php
$attempt = \Temporal\Activity::getInfo()->attempt;

throw new \Temporal\Exception\Failure\ApplicationFailure(
    message: "Something bad happened on attempt $attempt",
    type: 'my_failure_type',
    nonRetryable: false,
    nextRetryDelay: \DateInterval::createFromDateString(\sprintf('%d seconds', $attempt * 3)),
);
```

## Heartbeat an Activity 

An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service).
Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed.
If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy.

Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation.
Heartbeat throttling may lead to Cancellation getting delivered later than expected.

Heartbeats can contain a `details` field describing the Activity's current progress.
If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service.

Some Activities are long-running.
To react to a crash quickly, use the Heartbeat mechanism, `Activity::heartbeat()`, which lets the Temporal Server know that the Activity is still alive.
This acts as a periodic checkpoint mechanism for the progress of an Activity.

You can piggyback `details` on an Activity Heartbeat.
If an Activity times out, the last value of `details` is included in the `TimeoutFailure` delivered to a Workflow.
Then the Workflow can pass the details to the next Activity invocation.
Additionally, you can access the details from within an Activity via `Activity::getHeartbeatDetails`.
When an Activity is retried after a failure `getHeartbeatDetails` enables you to get the value from the last successful Heartbeat.

```php
use Temporal\Activity;

class FileProcessingActivitiesImpl implements FileProcessingActivities
{
    // ...
    public function download(
        string $bucketName,
        string $remoteName,
        string $localName
    ): void
    {
        $this->dowloader->downloadWithProgress(
            $bucketName,
            $remoteName,
            $localName,
            // on progress
            function ($progress) {
                Activity::heartbeat($progress);
            }
        );

        Activity::heartbeat(100); // download complete

        // ...
    }

    // ...
}
```

### Set a Heartbeat Timeout 

A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat).

Some Activities are long-running.
To react to a crash quickly, use the Heartbeat mechanism, `Activity::heartbeat()`, which lets the Temporal Server know that the Activity is still alive.
This acts as a periodic checkpoint mechanism for the progress of an Activity.

You can piggyback `details` on an Activity Heartbeat.
If an Activity times out, the last value of `details` is included in the `TimeoutFailure` delivered to a Workflow.
Then the Workflow can pass the details to the next Activity invocation.
Additionally, you can access the details from within an Activity via `Activity::getHeartbeatDetails`.
When an Activity is retried after a failure `getHeartbeatDetails` enables you to get the value from the last successful Heartbeat.

```php
use Temporal\Activity;

class FileProcessingActivitiesImpl implements FileProcessingActivities
{
    // ...
    public function download(
        string $bucketName,
        string $remoteName,
        string $localName
    ): void
    {
        $this->dowloader->downloadWithProgress(
            $bucketName,
            $remoteName,
            $localName,
            // on progress
            function ($progress) {
                Activity::heartbeat($progress);
            }
        );

        Activity::heartbeat(100); // download complete

        // ...
    }

    // ...
}
```

---

# Best practices - PHP SDK

Source: https://docs.temporal.io/develop/php/best-practices

> This section explains how to implement best practices with the PHP SDK

![PHP SDK Banner](https://docs.temporal.io/img/assets/banner-php-temporal.png)

## Best practices

- [Testing](/develop/php/best-practices/testing-suite)
- [Debugging](/develop/php/best-practices/debugging)

---

# Debugging - PHP SDK

Source: https://docs.temporal.io/develop/php/best-practices/debugging

> Effectively debug your Workflow in both development and production environments using Web UI, Temporal CLI, and performance metrics for optimal Worker and Server performance.

## Debugging 

### How to debug in a development environment 

In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli).

### How to debug in a production environment 

You can debug production Workflows using:

- [Web UI](/web-ui)
- [Temporal CLI](/cli)

You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance).

Debug Server performance with [Cloud metrics](/cloud/metrics/openmetrics) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics).

---

# Testing - PHP SDK

Source: https://docs.temporal.io/develop/php/best-practices/testing-suite

> The Temporal Application Testing section explains frameworks for Workflow and integration testing, including end-to-end, integration, unit tests, and how to mock Activities in a PHP environment.

The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing.

In the context of Temporal, you can create these types of automated tests:

- **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client.
- **Integration:** Anything between end-to-end and unit testing.
  - Running Activities with mocked Context and other SDK imports (and usually network requests).
  - Running Workers with mock Activities, and using a Client to start Workflows.
  - Running Workflows with mocked SDK imports.
- **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls.

We generally recommend writing the majority of your tests as integration tests.

Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers.

## Testing Activities 

An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity.
This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity.

## Testing Workflows 

### How to mock Activities 

Mock the Activity invocation when unit testing your Workflows.

When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker.

**RoadRunner config**

To mock an Activity in PHP, use [RoadRunner Key-Value storage](https://github.com/spiral/roadrunner-kv) and add the following lines to your `tests/.rr.test.yaml` file.

```yaml
# tests/.rr.test.yaml
kv:
  test:
    driver: memory
    config:
      interval: 10
```

If you want to be able to mock Activities, use `WorkerFactory` from the `Temporal\Testing` Namespace
in your PHP Worker:

```php
// worker.test.php
use Temporal\Testing\WorkerFactory;

$factory = WorkerFactory::create();
$worker = $factory->newWorker();

$worker->registerWorkflowTypes(MyWorkflow::class);
$worker->registerActivity(MyActivity::class);
$factory->run();
```

Then, in your tests to mock an Activity, use the`ActivityMocker` class.

Assume we have the following Activity:

```php
#[ActivityInterface(prefix: "SimpleActivity.")]
interface SimpleActivityInterface
{
    #[ActivityMethod('doSomething')]
    public function doSomething(string $input): string;
```

To mock it in the test, you can do this:

```php
final class SimpleWorkflowTestCase extends TestCase
{
    private WorkflowClient $workflowClient;
    private ActivityMocker $activityMocks;

    protected function setUp(): void
    {
        $this->workflowClient = new WorkflowClient(ServiceClient::create('localhost:7233'));
        $this->activityMocks = new ActivityMocker();

        parent::setUp();
    }

    protected function tearDown(): void
    {
        $this->activityMocks->clear();
        parent::tearDown();
    }

    public function testWorkflowReturnsUpperCasedInput(): void
    {
        $this->activityMocks->expectCompletion('SimpleActivity.doSomething', 'world');
        $workflow = $this->workflowClient->newWorkflowStub(SimpleWorkflow::class);
        $run = $this->workflowClient->start($workflow, 'hello');
        $this->assertSame('world', $run->getResult('string'));
    }
}
```

In the preceding test case, we do the following:

1. Instantiate `ActivityMocker` in the `setUp()` method of the test.
2. Clear the cache after each test in `tearDown()`.
3. Mock an Activity call to return a string `world`.

To mock a failure, use the `expectFailure()` method:

```php
$this->activityMocks->expectFailure('SimpleActivity.echo', new \LogicException('something went wrong'));
```

### How to skip time 

Some long-running Workflows can persist for months or even years.
Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount.

For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works.
Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner.

The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time.
Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests.
If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server.
For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel.

#### Set up time skipping 

1. In the `tests` folder, create `bootstrap.php` with the following contents:

```php
declare(strict_types=1);

require __DIR__ . '/../vendor/autoload.php';

use Temporal\Testing\Environment;

$environment = Environment::create();
$environment->start();
register_shutdown_function(fn () => $environment->stop());
```

If you don't want to run the test server with all of your tests, you can add a condition to start a test only if the `RUN_TEMPORAL_TEST_SERVER` environment variable is present:

```php
if (getenv('RUN_TEMPORAL_TEST_SERVER') !== false) {
    $environment = Environment::create();
    $environment->start('./rr serve -c .rr.silent.yaml --workflow-id tests');
    register_shutdown_function(fn() => $environment->stop());
}
```

2. Add `bootstrap.php` and the `TEMPORAL_ADDRESS` environment variable to `phpunit.xml`:

```xml
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.3/phpunit.xsd"
         bootstrap="tests/bootstrap.php"
>
    <php>
        <env name="TEMPORAL_ADDRESS" value="127.0.0.1:7233" />
    </php>
</phpunit>
```

3. Add the test server executable to `.gitignore`:

```gitignore
temporal-test-server
```

## How to Replay a Workflow Execution 

Replay recreates the exact state of a Workflow Execution.
You can replay a Workflow from the beginning of its Event History.

Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view.

When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks:

1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test.
2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI.
3. Run the Event Histories through replay.
4. Fail CI if any error is encountered during replay.

The following are examples of fetching and replaying Event Histories:

To replay Workflow Executions, use the `\Temporal\Testing\Replay\WorkflowReplayer` class.

In the following example, Event Histories are fetching from the Temporal, and then replayed.
If the Workflow Definition is non-deterministic, a `NonDeterministicWorkflowException` will be thrown.
Note that this requires [Advanced Visibility](/visibility#advanced-visibility) to be enabled.

```php
/**
 * We assume you already have a WorkflowClient and WorkflowReplayer in scope.
 * @var \Temporal\Client\WorkflowClientInterface $workflowClient
 * @var \Temporal\Testing\Replay\WorkflowReplayer $replayer
 */

// Find all workflow executions of type "MyWorkflow" and task queue "MyTaskQueue".
$executions = $workflowClient->listWorkflowExecutions(
    "WorkflowType='MyWorkflow' AND TaskQueue='MyTaskQueue'"
);

// Replay each workflow execution.
foreach ($executions as $executionInfo) {
    try {
        $replayer->replayFromServer(
            workflowType: $executionInfo->type->name,
            execution: $executionInfo->execution,
        );
    } catch (\Temporal\Testing\Replay\Exception\ReplayerException $e) {
        // Handle a replay error.
    }
}
```

In the next example, an Event History is loaded from a JSON file, and the maximum number of replayed Events is limited to 42.

```php
$replayer->replayFromJSON(
    workflowType: 'MyWorkflow',
    path: 'history.json',
    lastEventId: 42,  // optional
);
```

You can download a Event History using PHP, and then replay it from a memorized History object:

```php
$history = $this->workflowClient->getWorkflowHistory(
    execution: $run->getExecution(),
)->getHistory();

(new WorkflowReplayer())->replayHistory($history);
```

---

# Client - PHP SDK

Source: https://docs.temporal.io/develop/php/client

![PHP SDK Banner](https://docs.temporal.io/img/assets/banner-php-temporal.png)

## Temporal Client

- [Temporal Client](/develop/php/client/temporal-client)

---

# Temporal Client - PHP SDK

Source: https://docs.temporal.io/develop/php/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the [Temporal Service](/temporal-service).
Communication with a Temporal Service includes, but isn't limited to, the following:

- Scheduling Workflow Executions.
- Starting Workflow Executions.
- Sending Signals to Workflow Executions.
- Sending Queries to Workflow Executions.
- Sending Updates to Workflow Executions.
- Getting the results of a Workflow Execution.
- Providing an Activity Task Token.

This page shows you how to do the following using the PHP SDK with the Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-a-dev-cluster)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Start a Workflow Execution](#start-workflow-execution)
- [Get Workflow results](#get-workflow-results)
- [Advanced connection options](#advanced-connection-options)

> **⚠️ Caution:**
>
> A Temporal Client cannot be initialized and used inside a Workflow.
> However, it is acceptable and common to use a Temporal Client inside an Activity to communicate with a Temporal Service.
>

## Connect to development Temporal Service 

When you are running a Temporal Service locally (such as the [Temporal CLI](/cli/command-reference/server#start-dev)), the number of connection options you must provide is minimal.
Many SDKs default to `127.0.0.1:7233`.

In the PHP SDK, different client classes are responsible for different functional areas.
The [`ServiceClient`](https://php.temporal.io/classes/Temporal-Client-GRPC-ServiceClient.html) is responsible for the low-level API and connection to the Temporal Service.
It is also used in higher-level clients: [`WorkflowClient`](https://php.temporal.io/classes/Temporal-Client-WorkflowClient.html) and [`ScheduleClient`](https://php.temporal.io/classes/Temporal-Client-ScheduleClient.html).

> **📝 Note:**
>
> RoadRunner is not required to work only with the client API; however, the [gRPC extension](https://pecl.php.net/package/grpc) is necessary.
>

Use `create()` factory methods to create clients.

```php
use Temporal\Client\GRPC\ServiceClient;
use Temporal\Client\WorkflowClient;

$serviceClient = ServiceClient::create('localhost:7233');
$workflowClient = WorkflowClient::create($serviceClient);

// Use $workflowClient to work with Workflows ...
```

See the [Advanced connection options](#advanced-connection-options) section for more information on configuring the connection.

## Connect to Temporal Cloud 

When you connect to [Temporal Cloud](/cloud), you need to provide additional connection and client options that include the following:

- The [Temporal Cloud Namespace Id](/cloud/namespaces#temporal-cloud-namespace-id).
- The [Namespace's gRPC endpoint](/cloud/namespaces#temporal-cloud-grpc-endpoint).
  An endpoint listing is available at the [Temporal Cloud Website](https://cloud.temporal.io/namespaces) on each Namespace detail page.
  The endpoint contains the Namespace Id and port.
- mTLS CA certificate.
- mTLS private key.

For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates).

For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see [Temporal Customization Samples](https://github.com/temporalio/samples-server).

Use the [`ServiceClient::createSSL()`](https://php.temporal.io/classes/Temporal-Client-GRPC-BaseClient.html#method_createSSL) method to configure a client connection to the Temporal Service.
The `$clientKey` argument must be combined with the `$clientPem` to authenticate the Client.

```php
use Temporal\Client\ClientOptions;
use Temporal\Client\GRPC\ServiceClient;
use Temporal\Client\WorkflowClient;

$serviceClient = \Temporal\Client\GRPC\ServiceClient::createSSL(
    address: '<your-custom-namespace>.tmprl.cloud:7233',
    // crt: 'certs/server-root-ca-cert.pem', # ROOT CA to validate the server cert
    clientKey: 'certs/client-private-key.pem',
    clientPem: 'certs/client-cert.pem',
    // overrideServerName: 'tls-sample',
);

$workflowClient = WorkflowClient::create(
    serviceClient: $serviceClient,
    options: (new ClientOptions())
        ->withNamespace('<your-custom-namespace>.<id>'),
);
```

To [run Worker processes](/develop/php/workers/run-worker-process#run-a-dev-worker) managed by Temporal Cloud, configure RoadRunner in the same way.

```yml
temporal:
  # ...
  tls:
    # root_ca: 'certs/server-root-ca-cert.pem'
    key: 'certs/client-private-key.pem'
    cert: 'certs/client-cert.pem'
    client_auth_type: require_and_verify_client_cert
    # server_name: 'tls-sample'
```

To set up the [API key](/cloud/api-keys) in the Client, use the [`ServiceClient::withAuthKey()`](https://php.temporal.io/classes/Temporal-Client-GRPC-BaseClient.html#method_withAuthKey) method:

```php
$serviceClient = \Temporal\Client\GRPC\ServiceClient::createSSL(/*...*/)
    ->withAuthKey('your-api-key');
```

Rotating an mTLS client certificate without restarting the Worker isn't currently supported by the PHP SDK or
RoadRunner. The certificate configured on `ServiceClient::createSSL()` or in RoadRunner's `tls:` block is fixed for the
life of the process. To rotate a certificate, stage the new certificate alongside the old one on your Temporal Cloud
Namespace, then restart your Worker (RoadRunner process) with the new certificate before removing the old one. See
[Update certificates using Temporal Cloud UI/tcld](/cloud/certificates#manage-certificates) for the zero-downtime
staging sequence.

## Start a Workflow Execution 

[Workflow Execution](/workflow-execution) semantics rely on several parameters. To start a Workflow Execution, you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters.

In the examples following all Workflow Executions are started using a Temporal Client.
To spawn Workflow Executions from within another Workflow Execution, use either the [Child Workflow](/develop/php/workflows/child-workflows) or External Workflow APIs.

See the [Customize Workflow Type](/develop/php/workflows/basics#workflow-type) section to see how to customize the name of the Workflow Type.

A request to spawn a Workflow Execution causes the Temporal Service to create the first Event ([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History.
The Temporal Service then creates the first Workflow Task, resulting in the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event.

Use Workflow stub to start a Workflow Execution from within a Client.
Workflow stub is a proxy generated by the [`WorkflowClient`](https://php.temporal.io/classes/Temporal-Client-WorkflowClient.html).
You can use a typed or untyped Workflow stub in the client code.

- Typed Workflow stubs are useful because they are type safe and allow you to invoke your Workflow methods such as `#[WorkflowMethod]`, `#[QueryMethod]`, `#[SignalMethod]`, and `#[UpdateMethod]` directly.
- An untyped Workflow stub does not use a Workflow interface. It is more flexible because it has methods from the [`WorkflowStubInterface`](https://php.temporal.io/classes/Temporal-Client-WorkflowStubInterface.html), such as `start`, `signal`, `getResults`, `query`, `update`, `cancel`, and `terminate`
  When using untyped Workflow stub, we rely on the Workflow Type, Activity Type, Child Workflow Type, as well as Query and Signal names.

For example, there is a Workflow defined as follows:

```php
#[WorkflowInterface]
interface AccountTransferWorkflowInterface
{
    #[WorkflowMethod(name: "account.transfer")]
    public function begin(UuidInterface $transactionId);

    #[UpdateMethod(name: "pay")]
    public function move(UuidInterface $from, UuidInterface $to, int $amount);

    #[UpdateMethod(name: "finish")]
    public function commit();

    #[UpdateMethod(name: "cancel")]
    public function rollback(string $reason);
}
```

In case of a **typed** Workflow stub, you can use the `AccountTransferWorkflowInterface` to call the Workflow methods directly:

```php
$stub = $workflowClient->newWorkflowStub(AccountTransferWorkflowInterface::class);

$workflowClient->start($stub, $transactionId);
$stub->move($from1, $to1, $amount1);
$stub->move($from2, $to2, $amount2);
$stub->commit();
```

In case of an **untyped** Workflow stub, you need to specify Workflow Type and method names explicitly:

```php
$stub = $workflowClient->newUntypedWorkflowStub('account.transfer');

$workflowClient->start($stub, $transactionId);
$stub->update('pay', $from1, $to1, $amount1);
$stub->update('pay', $from2, $to2, $amount2);
$stub->update('finish');
```

A Workflow Execution can be started either synchronously or asynchronously.

### Start a Workflow synchronously

A synchronous start initiates a Workflow and then waits for its completion. The started Workflow will not rely on the
invocation process and will continue executing even if the waiting process crashes or stops.

Be sure to acquire the Workflow interface or class name you want to start.
For example:

```php
#[WorkflowInterface]
interface AccountTransferWorkflowInterface
{
    #[WorkflowMethod(name: "MoneyTransfer")]
    #[ReturnType(UuidInterface::class)]
    public function transfer( string $fromAccountId, string $toAccountId, string $referenceId, int $amountCents);
}
```

To start the Workflow in sync mode:

```php
$accountTransfer = $workflowClient->newWorkflowStub(
    AccountTransferWorkflowInterface::class,
);

$result = $accountTransfer->transfer('fromID', 'toID', 'refID', 1000);
```

### Start a Workflow asynchronously

An asynchronous start initiates a Workflow Execution and immediately returns to the caller without waiting for a result.
This is the most common way to start Workflows in a live environment.

To start a Workflow asynchronously, pass the Workflow stub instance and start parameters into the [`WorkflowClient::start()`](https://php.temporal.io/classes/Temporal-Client-WorkflowClientInterface.html#method_start) method.

```php
$accountTransfer = $workflowClient->newWorkflowStub(
    AccountTransferWorkflowInterface::class,
);

$run = $this->workflowClient->start($accountTransfer, 'fromID', 'toID', 'refID', 1000);
```

After the Workflow is started, you can receive details about the Workflow Execution or result via the [`WorkflowRun`](https://php.temporal.io/classes/Temporal-Workflow-WorkflowRunInterface.html) object methods:

```php
$run = $workflowClient->start($accountTransfer, 'fromID', 'toID', 'refID', 1000);

// Get the Workflow ID
var_dump($run->getExecution()->getID());

// Describe the Workflow Execution
var_dump($run->describe());

// Wait for the Workflow to complete and get the result with 10-second timeout
var_dump($run->getResult(timeout: 10));
```

### Start a Workflow on a schedule

You can start a Workflow Execution on a regular schedule with [the CronSchedule option](/develop/php/workflows/schedules#temporal-cron-jobs).

### Set a Workflow's Task Queue 

In most SDKs, the only Workflow Option that must be set is the name of the [Task Queue](/task-queue).
When developing in PHP, the Task Queue name defaults to `"default"`.
While setting a meaningful Task Queue name is recommended for better observability, Workflows can be run without setting this option.

> **📝 Note:**
>
> PHP's default is different from most SDKs, which do require an explicit Task Queue name.
>

For your code to execute, a Worker Process must be running ([how to run Worker Processes](/develop/php/workers/run-worker-process#run-a-dev-worker)).
This process needs a Worker Entity that is polling the same Task Queue name.

Set the Workflow Task Queue with the Workflow stub in the Client code using [`WorkflowOptions::withTaskQueue()`](https://php.temporal.io/classes/Temporal-Client-WorkflowOptions.html#method_withTaskQueue).

```php
$stub = $workflowClient->newWorkflowStub(
    YourWorkflowInterface::class,
    WorkflowOptions::new()
        ->withTaskQueue("Workflow-Task-Queue-1"),
);
```

### Set a Workflow Id 

Although it is not required, we recommend providing your own [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity identifier, such as an order identifier or customer identifier.

Set the Workflow Id with the Workflow stub in the Client code using [`WorkflowOptions::withWorkflowId()`](https://php.temporal.io/classes/Temporal-Client-WorkflowOptions.html#method_withWorkflowId).

```php
$stub = $workflowClient->newWorkflowStub(
    YourWorkflowInterface::class,
    WorkflowOptions::new()
        ->withWorkflowId("Workflow-Id"),
);
```

## Get Workflow results 

If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id.

The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its result.

It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time (asynchronous execution).

In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and results of Workflow Executions.

If you need to wait for the completion of a Workflow after an asynchronous start, make a blocking call to
the `WorkflowRun::getResult()` method.

```php
$stub = $workflowClient->newWorkflowStub(YourWorkflowInterface::class);
$run = $workflowClient->start($stub, 'fromID', 'toID', 'refID', 1000);

var_dump($run->getResult());
```

In case of untyped Workflow stub, you can use the [`WorkflowStub::getResult()`](https://php.temporal.io/classes/Temporal-Workflow-WorkflowRunInterface.html#method_getResult) method:

```php
$stub = $workflowClient->newUntypedWorkflowStub('account.transfer');
$workflowClient->start($stub, 'fromID', 'toID', 'refID', 1000);

var_dump($stub->getResult(timeout: 5.5));
```

Note that you can specify a timeout for the `getResult()` method in seconds.
If the Workflow does not complete within the specified time, a `TimeoutException` will be thrown.
See how to limit all RPC calls in the [RPC timeout](#configure-rpc-timeout) section.

## Advanced connection options 

In PHP, it is common practice to work with resources in blocking mode.
Long blocks can quickly exhaust the pool of available workers and lead to application failure.

This section introduces features and configuration examples of the PHP SDK when working with the Temporal Client API.

### Connection

gRPC connections in the PHP SDK are lazy by default, meaning they are not established until the first call.
To force establishing the connection to the Temporal Service, you can call the [`ConnectionInterface::connect()`](https://php.temporal.io/classes/Temporal-Client-GRPC-Connection-ConnectionInterface.html#method_connect) or [`ServiceClient::getServerCapabilities()`](https://php.temporal.io/classes/Temporal-Client-GRPC-ServiceClientInterface.html#method_getServerCapabilities) method.

```php
// ...
$serviceClient->getConnection()->connect(timeout: 10);

// or
$serviceClient->getServerCapabilities();
```

If, for some reason, the established connection is broken, the SDK will automatically attempt to restore it, taking into account the configured retry policy.

### Retry policy 

Whenever the client fails to connect to the server, an error with a status code is generated.
If the status code is `UNKNOWN`, `UNAVAILABLE`, or `RESOURCE_EXHAUSTED`, the client will make another connection attempt.
By default, the number of attempts is unlimited, and the interval between them will range from 0.5 to 100 seconds with a backoff coefficient of 2.
This means that the client will likely be blocked until it establishes a connection to the server through infinite attempts.

If you want to change the default behavior, use the `withRetryPolicy()` method when creating a client service:

```php
use Temporal\Client\Common\RpcRetryOptions;
use Temporal\Client\GRPC\ServiceClient;
use Temporal\Client\WorkflowClient;

$serviceClient = ServiceClient::create('localhost:7233');
$workflowClient = WorkflowClient::create($serviceClient)
    ->withRetryOptions(
        RpcRetryOptions::new()
            ->withMaximumAttempts(10)
            ->withInitialInterval('1 second')   // The first retry will be in 1 second
            ->withBackoffCoefficient(2.5)       // Each next retry time will be multiplied by 2.5
            ->withMaximumInterval('20 seconds') // The maximum interval between attempts
            ->withMaximumJitterCoefficient(0.2) // Actual retry time can be +/- 20% of the calculated time
    );
```

### RPC timeout 

When the client calls the service's RPC, there is no default time limit for waiting for a response. This can result in the code call `$result = $workflowHandle->getResult();` blocking the PHP worker until the Workflow completes. In some cases, this is not the desired behavior, and there may be a need to set a reasonable timeout for waiting for the RPC to complete.

Use the `withTimeout()` method to build a client with a timeout for all RPC calls.

```php
use Temporal\Client\GRPC\ServiceClient;
use Temporal\Client\WorkflowClient;

$serviceClient = ServiceClient::create('localhost:7233');
$workflowClient = WorkflowClient::create($serviceClient)
    ->withTimeout(5.75);

// Create a Workflow stub
$stub = $workflowClient->newWorkflowStub(AccountTransferWorkflowInterface::class);

// If the Workflow does not complete within 5.75 seconds, a TimeoutException will be thrown
$result = $stub->transfer('fromID', 'toID', 'refID', 1000);
```

> **📝 Note:**
>
> The `withTimeout()` method is immutable.
> If you need to change the timeout for individual operations, create a new client from the existing one with a specific timeout: `$newClient = $workflowClient->withTimeout(0);` (`0` means no timeout).
>

---

# Platform - PHP SDK

Source: https://docs.temporal.io/develop/php/platform

> This section explains how to implement platform with the PHP SDK

![PHP SDK Banner](https://docs.temporal.io/img/assets/banner-php-temporal.png)

## Platform

- [Observability](/develop/php/platform/observability)
- [Enriching the UI](/develop/php/platform/enriching-ui)

---

# Enriching the user interface - PHP SDK

Source: https://docs.temporal.io/develop/php/platform/enriching-ui

> Add contextual information to workflows and events in the Temporal UI using the PHP SDK.

Temporal supports adding context to Workflows and Events with metadata. 
This helps users identify and understand Workflows and their operations.

## Adding summary and details to Workflows

### Starting a Workflow

When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI:

```php
use Temporal\Client\WorkflowClient;
use Temporal\Client\WorkflowOptions;

// Create workflow client
$workflowClient = WorkflowClient::create($serviceClient);

// Start a workflow with static summary and details
$workflow = $workflowClient->newWorkflowStub(
    YourWorkflow::class,
    WorkflowOptions::new()
        ->withWorkflowId('your-workflow-id')
        ->withTaskQueue('your-task-queue')
        ->withStaticSummary('Order processing for customer #12345')
        ->withStaticDetails('Processing premium order with expedited shipping')
);

$result = $workflow->yourWorkflowMethod('workflow input');
```

`withStaticSummary()` sets a single-line description that appears in the Workflow list view, limited to 200 bytes.
`withStaticDetails()` sets multi-line comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes.

The input format is standard Markdown excluding images, HTML, and scripts.

You can also start a Workflow asynchronously:

```php
// Start workflow asynchronously
$workflowClient->start($workflow, 'workflow input');
```

### Adding Summary to Activities and Timers

You can attach a `summary` to timers within a workflow:

```php
use Temporal\Workflow;
use Temporal\Workflow\TimerOptions;

#[WorkflowInterface]
interface YourWorkflow
{
    #[WorkflowMethod]
    public function yourWorkflowMethod(string $input): string;
}

class YourWorkflowImpl implements YourWorkflow
{
    public function yourWorkflowMethod(string $input): \Generator
    {
        // Create a timer with a summary
        yield Workflow::timer(
            300, // 5 minutes in seconds
            TimerOptions::new()->withSummary('Waiting for payment confirmation')
        );
        
        return 'Timer completed';
    }
}
```

For Activities, you can set a summary using Activity options:

```php
use Temporal\Activity\ActivityOptions;
use Temporal\Workflow;

class YourWorkflowImpl implements YourWorkflow
{
    private YourActivitiesInterface $activities;
    
    public function __construct()
    {
        $this->activities = Workflow::newActivityStub(
            YourActivitiesInterface::class,
            ActivityOptions::new()
                ->withStartToCloseTimeout('10 seconds')
                ->withSummary('Processing user data')
        );
    }
    
    public function yourWorkflowMethod(string $input): \Generator
    {
        // Execute the activity with the summary
        $result = yield $this->activities->yourActivity($input);
        
        return $result;
    }
}
```

The input format for `summary` is a string, and limited to 200 bytes.

## Viewing summary and details in the UI

Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. 
Navigate to your Workflow's details page to see the metadata displayed in three key locations:

### User Metadata

On the Workflow Details page, under the **User Metadata** tab, you'll find the workflow-level metadata:

- **Summary & Details** - Displays the static summary and static details set when starting the Workflow
- **Current Details** - Displays the dynamic details that can be updated during Workflow Execution

All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays.

### Timeline

The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance.

Labels longer than 120 characters are truncated with an ellipsis.

Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. 

Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later.

### Event History

Individual events in the Workflow's Event History display their associated summaries when available.

Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. 
When you do expand an Event, the summary is also prominently displayed in the detailed view.

---

# Observability - PHP SDK

Source: https://docs.temporal.io/develop/php/platform/observability

> Explore the Temporal Developer’s guide on observability to learn about Visibility APIs and Search Attributes, helping you manage Workflow Executions efficiently.

The observability section of the Temporal Developer's guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

This section covers features related to viewing the state of the application, including:

- [Log from a Workflow](#logging)
- [Visibility](#visibility)

## Log from a Workflow 

Logging enables you to record critical information during code execution.
Loggers create an audit trail and capture information about your Workflow's operation.
An appropriate logging level depends on your specific needs.
During development or troubleshooting, you might use debug or even trace.
In production, you might use info or warn to avoid excessive log volume.

You can find the log levels supported by `PSR-3` in [their official documentation](https://www.php-fig.org/psr/psr-3/#5-psrlogloglevel). The Temporal SDK core normally uses `WARN` as its default logging level.

To get a PSR-3 compatible logger in your Workflow code, use the [`Workflow::getLogger()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_getLogger) method.

```php
use Temporal\Workflow;

#[Workflow\WorkflowInterface]
class MyWorkflow
{
    #[Workflow\WorkflowMethod]
    public function execute(string $param): \Generator
    {
        Workflow::getLogger()->info('Workflow started', ['parameter' => $param]);

        // Your workflow implementation

        Workflow::getLogger()->info('Workflow completed');
        return 'Done';
    }
}
```

The Workflow logger automatically enriches log context with the current Task Queue name.

Logs in replay mode are omitted unless the [`enableLoggingInReplay`](https://php.temporal.io/classes/Temporal-Worker-WorkerOptions.html#method_withEnableLoggingInReplay) Worker option is set to true.

```php
$factory = WorkerFactory::create();
$worker = $factory->newWorker('your-task-queue', WorkerOptions::new()
    ->withEnableLoggingInReplay(true)
);
```

### Default logger

By default, PHP SDK uses a [`StderrLogger`](https://php.temporal.io/classes/Temporal-Worker-Logger-StderrLogger.html) that outputs log messages to the standard error stream.
These messages are automatically captured by RoadRunner and incorporated into its logging system with the INFO level, ensuring proper log collection in both development and production environments.
For more details on RoadRunner's logging capabilities, see the [RoadRunner Logger documentation](https://docs.roadrunner.dev/docs/logging-and-observability/logger).

### How to provide a custom logger 

You can set a custom PSR-3 compatible logger when creating a Worker:

```php
$myLogger = new MyLogger();

$workerFactory = WorkerFactory::create(converter: $converter);
$worker = $workerFactory->newWorker(
    taskQueue: 'my-task-queue',
    logger: $myLogger,
);
```

## Visibility APIs 

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### How to use Search Attributes 

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI.
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `UpsertSearchAttributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `DescribeWorkflow`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [In the Temporal CLI](/cli/command-reference/workflow#list).
  - In code by calling `ListWorkflowExecutions`.

Here is how to query Workflow Executions:

Use the [listWorkflowExecutions()](https://php.temporal.io/classes/Temporal-Client-WorkflowClientInterface.html#method_listWorkflowExecutions) method on the Client and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows.
The result is an iterable paginator, so you can use the `foreach` loop to iterate over the results.

```php
$paginator = $workflowClient->listWorkflowExecutions('WorkflowType="GreetingWorkflow"');

foreach ($paginator as $info) {
    echo "Workflow ID: {$info->execution->getID()}\n";
}
```

### How to set custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

To set custom Search Attributes, use the `withTypedSearchAttributes` method on `WorkflowOptions` for a Workflow stub.
Typed search attributes are a `TypedSearchAttributes` collection.

```php
$keyDestinationTime = SearchAttributeKey::forDatetime('DestinationTime');
$keyOrderId = SearchAttributeKey::forKeyword('OrderId');

$workflow = $workflowClient->newWorkflowStub(
    OrderWorkflowInterface::class,
    WorkflowOptions::new()
        ->withWorkflowExecutionTimeout('10 minutes')
        ->withTypedSearchAttributes(
            TypedSearchAttributes::empty()
                ->withValue($keyOrderId, $orderid)
                ->withValue($keyDestinationTime, new \DateTimeImmutable('2028-11-05T00:10:07Z'))
        ),
);
```

### How to upsert Search Attributes 

Within the Workflow code, you can dynamically add or update Search Attributes using [`upsertTypedSearchAttributes`](https://php.temporal.io/classes/Temporal-Workflow.html#method_upsertTypedSearchAttributes).
This method is particularly useful for Workflows whose attributes need to change based on internal logic or external events.

```php
#[Workflow\UpdateMethod]
public function postponeDestinationTime(\DateInterval $interval)
{
    // Get the key for the DestinationTime attribute
    $keyDestinationTime = SearchAttributeKey::forDatetime('DestinationTime');

    /** @var DateTimeImmutable $destinationTime */
    $destinationTime = Workflow::getInfo()->typedSearchAttributes->get($keyDestinationTime);

    Workflow::upsertTypedSearchAttributes(
        $keyDestinationTime->valueSet($destinationTime->add($interval)),
    );
}
```

### How to remove a Search Attribute from a Workflow 

To remove a Search Attribute that was previously set, unset its value.

```php
#[Workflow\UpdateMethod]
public function unsetDestinationTime()
{
    // Get the key for the DestinationTime attribute
    $keyDestinationTime = SearchAttributeKey::forDatetime('DestinationTime');

    Workflow::upsertTypedSearchAttributes(
        $keyDestinationTime->valueUnset(),
    );
}
```

---

# Set up a local development environment with the PHP SDK

Source: https://docs.temporal.io/develop/php/set-up-your-local-php

> Configure your local development environment to get started developing with Temporal

# Quickstart

Configure your local development environment to get started developing with Temporal.

## Install PHP

Make sure you have PHP installed.

**If you don't have PHP:** Visit the official website to [download and install](https://www.php.net/downloads.php) it.

### GRPC extension

GRPC extension is required to work with RoadRunner application server.

**If you don't have `ext-grpc` installed:** Visit the official website to
[download and install](https://docs.cloud.google.com/php/docs/reference/help/grpc) it.

> **💡 Tip:**
> GRPC Installation Tip
>
> On macOS with Apple Silicon (M1/M2/M3/M4) and PHP 8.3, `pecl install grpc` may appear to hang or install indefinitely.
> If this happens, try installing a specific version:
>
> ```bash
> pecl install channel://pecl.php.net/grpc-1.78.0RC2
> ```
>
> Note: You can find the latest versions at [pecl.php.net/package/grpc](https://pecl.php.net/package/grpc).
>

```bash
php -v
```

## Create a Project

Now that you have PHP installed, create a project to manage your dependencies and build your Temporal application.

```bash
mkdir temporal-hello-world
```

```bash
cd temporal-hello-world
```

```bash
composer init --name="myproject/quickstart" -n
```

## Add Temporal PHP SDK and configure autoloading

Install the Temporal SDK, then add PSR-4 autoloading to your `composer.json` so PHP can find your Workflow and Activity
classes.

Your final `composer.json` should look like this. After updating, run `composer dump-autoload` to regenerate the
autoloader.

```bash
composer require temporal/sdk
```

```json
{
    "name": "myproject/quickstart",
    "require": {
        "temporal/sdk": "^2.16"
    },
    "autoload": {
        "psr-4": {
            "App\\\\": "src/"
        }
    }
}
```

```bash
composer dump-autoload
```

## Install RoadRunner application server

Install [RoadRunner application server](https://github.com/roadrunner-server/roadrunner). It starts and manages your PHP
processes that run Temporal Workers, and connects them to the Temporal Service over gRPC.

See [RoadRunner installation instructions](https://docs.roadrunner.dev/docs/general/install) to learn about other
installation methods.

**CLI**

Download RoadRunner with the following command:
```bash
./vendor/bin/rr get
```

When prompted "Do you want create default '.rr.yaml' configuration file?", answer **yes**. You'll replace it with the proper config in the next step.

**DLoad**

Install DLoad package manager using Composer
```bash
composer require --dev internal/dload
```

Create a configuration file named `dload.xml` with the following content:
```xml
<?xml version="1.0"?>
<dload xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="vendor/internal/dload/dload.xsd"
       temp-dir="./runtime"
>
    <actions>
        <download software="rr" version="^2025.1.6"/>
    </actions>
</dload>

```

Finally, download the RoadRunner binary:
```bash
./vendor/bin/dload
```

Create a simple configuration file named `.rr.yaml` with the following content:

```yml
version: "3"

rpc: listen: tcp://127.0.0.1:6001

server: command: "php worker.php"

temporal: address: "127.0.0.1:7233"

logs: level: info 
```

## Install Temporal CLI and start the development server

The fastest way to get a development version of the Temporal Service running on your local machine is to use
[Temporal CLI](/cli).

Choose your operating system to install Temporal CLI:

**macOS**

Install the Temporal CLI using Homebrew:
```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH, for example:
```bash
sudo mv temporal /usr/local/bin
```

### DLoad package manager

Consider using DLoad to delegate all installation and updating processes to the package manager.

Add one more download action to the configuration file
```xml
<download software="temporal" version="^1.5"/>
```

The final configuration file should look like this:
```xml
<?xml version="1.0"?>
<dload xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="vendor/internal/dload/dload.xsd"
       temp-dir="./runtime"
>
    <actions>
        <download software="rr" version="^2025.1.6"/>
        <download software="temporal" version="^1.5"/>
    </actions>
</dload>

```

## Start the development server

Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory
database.

The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at
http://localhost:8233.

Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal
Service at any time by pressing CTRL+C.

Once you have everything installed, you're ready to build apps with Temporal on your local machine.

After installing, open a new Terminal. Keep this running in the background:
```bash
temporal server start-dev
```

#### Change the Web UI port
The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server:
```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and
Activity.

This test will confirm that:

- The Temporal PHP SDK is properly installed
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

### 1. Create the Activity

Create an Activity file (`src/GreetingActivity.php`):

```php
<?php
declare(strict_types=1);

namespace App;

use Temporal\Activity\ActivityInterface;
use Temporal\Activity\ActivityMethod;

#[ActivityInterface]
class GreetingActivity
{
    #[ActivityMethod]
    public function greet(string $name): string
    {
        return "Hello, $name!";
    }
}
```

An Activity is a method that executes a single, well-defined action (either short or long-running), which often involves
interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling
an API, which is prone to failure. If an Activity fails, Temporal automatically retries it based on your configuration.

You define Activities in PHP as classes annotated with the `Temporal\Activity\ActivityInterface` attribute.

Each method used in workflow classes should be annotated with `Temporal\Activity\ActivityMethod`.

### 2. Create the Workflow

Create a Workflow file (`src/SayHelloWorkflow.php`):

```php
<?php
declare(strict_types=1);

namespace App;

use Temporal\Activity\ActivityOptions;
use Temporal\Workflow;
use Temporal\Workflow\WorkflowInterface;
use Temporal\Workflow\WorkflowMethod;

#[WorkflowInterface]
class SayHelloWorkflow
{
    #[WorkflowMethod]
    public function sayHello(string $name)
    {
        $activity = Workflow::newActivityStub(
            GreetingActivity::class,
            ActivityOptions::new()
                ->withStartToCloseTimeout(5),
        );

        return yield $activity->greet($name);
    }
}
```

### 3. Create a Worker file

Create a Worker file (`worker.php`, under project root directory):

```php
<?php
declare(strict_types=1);

use Temporal\WorkerFactory;

ini_set('display_errors', 'stderr');
require "vendor/autoload.php";

$factory = WorkerFactory::create();

$worker = $factory->newWorker();

// Register Workflows
$worker->registerWorkflowTypes(\App\SayHelloWorkflow::class);
// Register Activities
$worker->registerActivity(\App\GreetingActivity::class);

$factory->run();
```

#### Run the Worker

Previously, we created a Worker that executes Workflow and Activity tasks.

Now, start the RoadRunner application server to run the Worker by opening up a new terminal window and running this
command:

```bash
./rr serve
```

A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues a Workflow
or Activity task from the Task Queue, it then executes the task.

Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your
Workflows and Activities. For more information on Workers, see
[Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers).

### 5. Execute the Workflow

Now that your Worker is running, it's time to start a Workflow Execution.

This final step will validate that everything is working correctly with your file labeled `client.php`.

Create a separate file called `client.php`:

```php
<?php
declare(strict_types=1);

use Temporal\Client\GRPC\ServiceClient;
use Temporal\Client\WorkflowClient;

ini_set('display_errors', 'stderr');
require "vendor/autoload.php";

$client = new WorkflowClient(
    ServiceClient::create('localhost:7233'),
);

$workflowStub = $client->newWorkflowStub(\App\SayHelloWorkflow::class);
$result = $workflowStub->sayHello('Temporal');

echo "Result: {$result}\n";
```

While your Worker is still running, open a new terminal and run:

```bash
php client.php
```

### Verify success

If everything is working correctly, you should see:

- Worker processing the workflow and activity
- Output: `Result: Hello, Temporal!`
- Workflow Execution details in the [Temporal Web UI](http://localhost:8233)

- [Run your first Temporal Application](https://learn.temporal.io/getting_started/php/hello_world_in_php/): Create a basic Workflow and run it with the Temporal PHP SDK

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Workers - PHP SDK

Source: https://docs.temporal.io/develop/php/workers

> This section explains how to implement Workers with the PHP SDK

![PHP SDK Banner](https://docs.temporal.io/img/assets/banner-php-temporal.png)

## Workers

- [Run a Worker](/develop/php/workers/run-worker-process)

---

# Run a Worker - PHP SDK

Source: https://docs.temporal.io/develop/php/workers/run-worker-process

> Shows how to run Worker processes with the PHP SDK

## How to run Worker Processes 

The [Worker Process](/workers#worker-process) is where Workflow Functions and Activity Functions are executed.

- Each [Worker Entity](/workers#worker-entity) in the Worker Process must register the exact Workflow Types and Activity Types it may execute.
- Each Worker Entity must also associate itself with exactly one [Task Queue](/task-queue).
- Each Worker Entity polling the same Task Queue must be registered with the same Workflow Types and Activity Types.

A [Worker Entity](/workers#worker-entity) is the component within a Worker Process that listens to a specific Task Queue.

Although multiple Worker Entities can be in a single Worker Process, a single Worker Entity Worker Process may be perfectly sufficient.
For more information, see the [Worker tuning guide](/develop/worker-performance).

A Worker Entity contains a Workflow Worker and/or an Activity Worker, which makes progress on Workflow Executions and Activity Executions, respectively.

The [RoadRunner application server](https://roadrunner.dev/) will launch multiple Temporal PHP Worker processes based on provided `.rr.yaml` configuration.

Each Worker might connect to one or multiple Task Queues.
Workers poll the _Temporal Service_ for tasks, perform those tasks, and communicate task execution results back to the _Temporal Service_.

Worker code is developed, deployed, and operated by Temporal customers.
To create a worker use `Temporal\WorkerFactory`:

```php
<?php

declare(strict_types=1);

use Temporal\WorkerFactory;

ini_set('display_errors', 'stderr');
include "vendor/autoload.php";

// factory initiates and runs task queue specific activity and workflow workers
$factory = WorkerFactory::create();

// Worker that listens on a Task Queue and hosts both workflow and activity implementations.
$worker = $factory->newWorker();

// Workflows are stateful. So you need a type to create instances.
$worker->registerWorkflowTypes(App\DemoWorkflow::class);

// Activities are stateless and thread safe. So a shared instance is used.
$worker->registerActivity(App\DemoActivity::class);

// In case an activity class requires some external dependencies provide a callback - factory
// that creates or builds a new activity instance. The factory should be a callable which accepts
// an instance of ReflectionClass with an activity class which should be created.
$worker->registerActivity(App\DemoActivity::class, fn(ReflectionClass $class) => $container->create($class->getName()));

// start primary loop
$factory->run();
```

You can configure task queue name using first argument of `WorkerFactory`->`newWorker`:

```php
$worker = $factory->newWorker('your-task-queue');
```

As mentioned preceding, you can create as many Task Queue connections inside a single Worker Process as you need.

To configure additional WorkerOptions use `Temporal\Worker\WorkerOptions`:

```php
use Temporal\Worker\WorkerOptions;

$worker = $factory->newWorker(
    'your-task-queue',
    WorkerOptions::new()
        ->withMaxConcurrentWorkflowTaskPollers(10)
);
```

Make sure to point the Worker file in application server configuration:

```yaml
rpc:
  listen: tcp://127.0.0.1:6001

server:
  command: 'php worker.php'

temporal:
  address: 'temporal:7233'
  activities:
    num_workers: 10
```

> You can serve HTTP endpoints using the same server setup.

To provide the [API key](/cloud/api-keys) to RoadRunner use a `ServiceCredentials` DTO when creating the WorkerFactory:

```php
use Temporal\Worker\ServiceCredentials;

$workerFactory = \Temporal\WorkerFactory::create(
    credentials: ServiceCredentials::create()->withApiKey('your-api-key'),
);
```

[How to configure connection to a Temporal Cloud](/develop/php/client/temporal-client#connect-to-temporal-cloud)

### How to register types 

All Workers listening to the same Task Queue name must be registered to handle the exact same Workflows Types and Activity Types.

If a Worker polls a Task for a Workflow Type or Activity Type it does not know about, it fails that Task.
However, the failure of the Task does not cause the associated Workflow Execution to fail.

Worker listens on a Task Queue and hosts both Workflow and Activity implementations:

```php
// Workflows are stateful. So you need a type to create instances:
$worker->registerWorkflowTypes(App\DemoWorkflow::class);
// Activities are stateless and thread safe:
$worker->registerActivity(App\DemoActivity::class);
```

In case an activity class requires some external dependencies provide a callback - factory
that creates or builds a new activity instance. The factory should be a callable which accepts
an instance of ReflectionClass with an activity class which should be created.

```php
$worker->registerActivity(
    App\DemoActivity::class,
    fn(ReflectionClass $class) => $container->create($class->getName())
);
```

If you want to clean up some resources after activity is done, you may register a finalizer. This callback is called
after each activity invocation:

```php
$worker->registerActivityFinalizer(fn() => $kernel->shutdown());
```

---

# Workflows - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows

> This section explains how to implement Workflows with the PHP SDK

![PHP SDK Banner](https://docs.temporal.io/img/assets/banner-php-temporal.png)

## Workflows

- [Workflow basics](/develop/php/workflows/basics)
- [Child Workflows](/develop/php/workflows/child-workflows)
- [Continue-As-New](/develop/php/workflows/continue-as-new)
- [Cancellation](/develop/php/workflows/cancellation)
- [Timeouts](/develop/php/workflows/timeouts)
- [Message passing](/develop/php/workflows/message-passing)
- [Schedules](/develop/php/workflows/schedules)
- [Timers](/develop/php/workflows/timers)
- [Side effects](/develop/php/workflows/side-effects)
- [Versioning](/develop/php/workflows/versioning)

---

# Workflow basics - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/basics

> This section explains Workflow basics with the PHP SDK

## Develop a basic Workflow 

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal PHP SDK programming model, Workflows are a class method. Classes must implement interfaces that are annotated with `#[WorkflowInterface]`. The method that is the Workflow must be annotated with `#[WorkflowMethod]`.

```php
use Temporal\Workflow\YourWorkflowInterface;
use Temporal\Workflow\WorkflowMethod;

#[WorkflowInterface]
interface FileProcessingWorkflow
{
    #[WorkflowMethod]
    public function processFile(Argument $args);

}
```

### Define Workflow parameters 

Temporal Workflows may have any number of custom parameters.
However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow.
All Workflow Definition parameters must be serializable.
For more reasons, see [Workflow parameters](/workflow-definition#workflow-parameters).

A method annotated with `#[WorkflowMethod]` can have any number of parameters.

We recommend passing a single parameter that contains all the input fields to allow for adding fields in a backward-compatible manner.

Note that all inputs should be serializable to a byte array using the provided [DataConverter](https://github.com/temporalio/sdk-php/blob/master/src/DataConverter/DataConverterInterface.php) interface.
The default implementation uses a JSON serializer, but an alternative implementation can be easily configured.
You can create a custom object and pass it to the Workflow method, as shown in the following example:

```php
#[WorkflowInterface]
interface FileProcessingWorkflow {
    #[WorkflowMethod]
    public function processFile(Argument $args);
}
```

### Define Workflow return parameters 

Workflow return values must also be serializable.
Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported.
However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error.

A Workflow method returns a Generator.
To properly typecast the Workflow's return value in the client code, use the `#[ReturnType()]` attribute.

```php
#[WorkflowInterface]
interface FileProcessingWorkflow {

    #[WorkflowMethod]
    #[ReturnType("string")]
    public function processFile(Argument $args);
}
```

### Customize your Workflow Type 

Workflows have a Type that are referred to as the Workflow name.

The following examples demonstrate how to set a custom name for your Workflow Type.

To customize a Workflow Type, use the `WorkflowMethod` attribute to specify the name of Workflow.

```php
#[WorkflowMethod(name)]
```

If a Workflow Type is not specified, then Workflow Type defaults to the interface name, which is `YourWorkflowDefinitionInterface` in this case.

```php
#[WorkflowInterface]
interface YourWorkflowDefinitionInterface
{
    #[WorkflowMethod]
    public function processFile(Argument $args);
}
```

### Use Workflow constructors

Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. However, if you use the `#[WorkflowInit]` attribute on your constructor, you can give it the same [Workflow parameters](/develop/php/workflows/basics#workflow-parameters) as your `#[WorkflowMethod]`.

The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/php/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `#[WorkflowMethod]` method. That always happens, whether or not you use the `#[WorkflowInit]` attribute.

Here's an example.
Notice that the constructor and `getGreeting` must have the same parameters:

```php
use Temporal\Workflow;

#[Workflow\WorkflowInterface]
class GreetingExample
{
    private readonly string $nameWithTitle;
    private bool $titleHasBeenChecked;

    // Note the attribute is on a public constructor
    #[Workflow\WorkflowInit]
    public function __construct(string $input)
    {
        $this->nameWithTitle = 'Sir ' . $input;
        $this->titleHasBeenChecked = false;
    }

    #[Workflow\WorkflowMethod]
    public function getGreeting(string $input)
    {
        yield Workflow::await(fn() => $this->titleHasBeenChecked);
        return "Hello " . $this->nameWithTitle;
    }
}
```

### Develop Workflow logic 

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with application code outside the Workflow.

Temporal uses the [Microsoft Azure Event Sourcing pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing) to recover the state of a Workflow object including its local variable values.

In essence, every time a Workflow state has to be restored, its code is re-executed from the beginning.
When replaying, side effects (such as Activity invocations) are ignored because they are already recorded in the Workflow event history.
The replay is not visible while the Workflow runs, so write your Workflow code as if it executes only once.
This design puts the following constraints on the Workflow implementation:

- Do not use any mutable global variables because multiple instances of Workflows are executed in parallel.
- Do not call any non-deterministic functions like non seeded random or `UUID` directly from the Workflow code.

Always do the following in the Workflow implementation code:

- Don't perform any IO or service calls as they are not usually deterministic. Use Activities for this.
- Only use `Workflow::now()` to get the current time inside a Workflow.
- Call `yield Workflow::timer()` instead of `sleep()`.
- Do not use any blocking SPL provided by PHP (that is, `fopen`, `PDO`, etc) in **Workflow code**.
- Use `yield Workflow::getVersion()` when making any changes to the Workflow code. Without this, any deployment of updated Workflow code
  might break already open Workflows.
- Don't access configuration APIs directly from a Workflow because changes in the configuration might affect a Workflow Execution path.
  Pass it as an argument to a Workflow function or use an Activity to load it.

Workflow method arguments and return values are serializable to a byte array using the provided [DataConverter](https://github.com/temporalio/sdk-php/blob/master/src/DataConverter/DataConverterInterface.php) interface.
The default implementation uses JSON serializer, but you can use any alternative serialization mechanism.

Make sure to annotate your `WorkflowMethod` using `ReturnType` to specify concrete return type.

> You can not use the default return type declaration as Workflow methods are generators.

The values passed to Workflows through invocation parameters or returned through a result value are recorded in the execution history.
The entire execution history is transferred from the Temporal service to Workflow workers with every event that the Workflow logic needs to process.
A large execution history can thus adversely impact the performance of your Workflow.
Therefore, be mindful of the amount of data that you transfer via Activity invocation parameters or return values.
Otherwise, no additional limitations exist on Activity implementations.

---

# Cancel a Workflow - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/cancellation

> Cancel a PHP Activity from a Workflow using Heartbeats and a Heartbeat Timeout, and reset a Workflow Execution from its Event History.

## Cancel an Activity from a Workflow 

Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat
Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate
Activity is executed, the Activity Execution should send Heartbeats and set a
[Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is
still working.

When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic
needs to be performed, it can be done in a `finally` clause or inside a caught cancel error. However, for the Activity
to appear canceled the exception needs to be re-raised.

> **📝 Note:**
>
> Unlike regular Activities, [Local Activities](/local-activity) can be canceled if they don't send Heartbeats. Local
> Activities are handled locally, and all the information needed to handle the cancellation logic is available in the same
> Worker process.
>

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Child Workflows - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/child-workflows

> Start a Child Workflow Execution in PHP, configure ChildWorkflowOptions, set a Parent Close Policy, and call children asynchronously with promises.

## Start a Child Workflow Execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In PHP, yielding `$child->start()` or `Workflow::executeChildWorkflow()` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the yield resolves.
See the [Parent Close Policy](#parent-close-policy) section below for an example.

Besides Activities, a Workflow can also start other Workflows.

`Workflow::executeChildWorkflow` and `Workflow::newChildWorkflowStub` enables the scheduling of other Workflows from within a Workflow's implementation.
The parent Workflow has the ability to monitor and impact the lifecycle of the Child Workflow, similar to the way it does for an Activity that it invoked.

```php
// Use one stub per child workflow run
$child = Workflow::newChildWorkflowStub(
    ChildWorkflowInterface::class,
    ChildWorkflowOptions::new()
        // Do not specify WorkflowId if you want Temporal to generate a unique Id
        // for the child execution.
        ->withWorkflowId('BID-SIMPLE-CHILD-WORKFLOW')
        ->withExecutionStartToCloseTimeout(DateInterval::createFromDateString('30 minutes'))
);

// This is a non blocking call that returns immediately.
// Use yield $child->workflowMethod(name) to call synchronously.
$promise = $child->workflowMethod('value');

// Do something else here.
try{
    $value = yield $promise;
} catch(TemporalException $e) {
    $logger->error('child workflow failed');
    throw $e;
}
```

Let's take a look at each component of this call.

Before calling `$child->workflowMethod()`, you must configure `ChildWorkflowOptions` for the invocation.
These options customize various execution timeouts, and are passed into the Workflow stub defined by the `Workflow::newChildWorkflowStub`.
Once stub created you can invoke its Workflow method based on attribute `WorkflowMethod`.

The method call returns immediately and returns a `Promise`.
This allows you to execute more code without having to wait for the scheduled Workflow to complete.

When you are ready to process the results of the Workflow, call the `yield $promise` method on the returned promise object.

When a parent Workflow is cancelled by the user, the Child Workflow can be cancelled or abandoned based on a configurable child policy.

You can also skip the stub part of Child Workflow initiation and use `Workflow::executeChildWorkflow` directly:

```php
// Use one stub per child workflow run
$childResult = yield Workflow::executeChildWorkflow(
    'ChildWorkflowName',
    ['args'],
    ChildWorkflowOptions::new()->withWorkflowId('BID-SIMPLE-CHILD-WORKFLOW'),
    Type::TYPE_STRING // optional: defines the return type
);
```

#### Set a Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

In PHP, a [Parent Close Policy](/parent-close-policy) is set via the `ChildWorkflowOptions` object and `withParentClosePolicy()` method.
The possible values can be obtained from the [`ParentClosePolicy`](https://github.com/temporalio/sdk-php/blob/master/src/Workflow/ParentClosePolicy.php) class.

- `POLICY_TERMINATE`
- `POLICY_ABANDON`
- `POLICY_REQUEST_CANCEL`

Then `ChildWorkflowOptions` object is used to create a new Child Workflow object:

```php
$child = Workflow::newUntypedChildWorkflowStub(
    'child-workflow',
    ChildWorkflowOptions::new()
        ->withParentClosePolicy(ParentClosePolicy::POLICY_ABANDON)
);

yield $child->start();
```

In the snippet above we:

1. Create a new untyped Child Workflow stub with `Workflow::newUntypedChildWorkflowStub`.
2. Provide `ChildWorkflowOptions` object with Parent Close Policy set to `ParentClosePolicy::POLICY_ABANDON`.
3. Start Child Workflow Execution asynchronously using `yield` and method `start()`.

We need `yield` here to ensure that a Child Workflow Execution starts before the parent closes.

---

# Continue-As-New - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/continue-as-new

> Use Temporal's Continue-As-New in PHP to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters.

This page covers the following for PHP developers:

- [What is Continue-As-New?](#what)
- [Use Continue-As-New](#how)
- [When is it right to Continue-As-New?](#when)
- [Test Continue-As-New](#how-to-test)

## What is Continue-As-New? 

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution.
You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History.
It also receives your Workflow's usual parameters.

## Use Continue-As-New with the PHP SDK 

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically set to `None` for the original caller of the Workflow.

[View the source code](https://github.com/temporalio/samples-php/blob/master/app/src/SafeMessageHandlers/MessageHandlerWorkflowInterface.php) in the context of the rest of the application code.

```php
final class ClusterManagerInput {
    public function __construct(
        public ?ClusterManagerState $state = null,
        public bool $testContinueAsNew = false,
    ) {}
}

#[Workflow\WorkflowInterface]
interface MessageHandlerWorkflowInterface
{
#[Workflow\WorkflowMethod]
public function run(ClusterManagerInput $input);
}

````
The test hook in the above snippet is covered [below](#how-to-test).

Inside your Workflow, call the [`continueAsNew()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_continueAsNew) function with the same type.
This stops the Workflow right away and starts a new one.

[View the source code](https://github.com/temporalio/samples-php/blob/master/app/src/SafeMessageHandlers/MessageHandlerWorkflow.php) in the context of the rest of the application code.

```php
Workflow::continueAsNew(
    Workflow::getInfo()->type->name,
    [new ClusterManagerInput($this->state, $input->testContinueAsNew)],
);
````

### Considerations for Workflows with message handlers 

If you use Updates or Signals, don't call Continue-as-New from the handlers.
Instead, wait for your handlers to finish in your main Workflow before you run `continueAsNew`.
See the [`allHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance.

## When is it right to Continue-As-New with the PHP SDK? 

Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New.
Call `Workflow::getInfo()->shouldContinueAsNew` to check if it's time.

## Test Continue-As-New with the PHP SDK 

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive.
Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, when `testContinueAsNew == true`, this sample creates a test-only variable called `$this->maxHistoryLength` and sets it to a small value.
A helper method in the Workflow checks it each time it considers using Continue-as-New:

[View the source code](https://github.com/temporalio/samples-php/blob/master/app/src/SafeMessageHandlers/MessageHandlerWorkflow.php) in the context of the rest of the application code.

```php
private function shouldContinueAsNew(): bool
{
    if (Workflow::getInfo()->shouldContinueAsNew) {
        return true;
    }

    // This is just for ease-of-testing.  In production, we trust temporal to tell us when to continue as new.
    if ($this->maxHistoryLength !== null && Workflow::getInfo()->historyLength > $this->maxHistoryLength) {
        return true;
    }

    return false;
}
```

---

# Workflow message passing - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/message-passing

> Develop with Signals, Queries, and Updates in Temporal Workflows. Define, handle, and send Signals or Queries, and validate updates from a Temporal Client.

## Develop with Signals 

A [Signal](/sending-messages#sending-signals) is a message sent to a running Workflow Execution.

Signals are defined in your code and handled in your Workflow Definition.
Signals can be sent to Workflow Executions from a Temporal Client or from another Workflow Execution.

### Define a Signal 

A Signal has a name and can have arguments.

- The name, also called a Signal type, is a string.
- The arguments must be [serializable](/dataconversion).

Workflows can answer synchronous [Queries](/sending-messages#sending-queries) and receive [Signals](/sending-messages#sending-signals).

All interface methods must have one of the following attributes:

- **#[WorkflowMethod]** indicates an entry point to a Workflow.
  It contains parameters that specify timeouts and a Task Queue name.
  Required parameters (such as `executionStartToCloseTimeoutSeconds`) that are not specified through the attribute must be provided at runtime.
- **#[SignalMethod]** indicates a method that reacts to external signals. It must have a `void` return type.
- **#[QueryMethod]** indicates a method that reacts to synchronous query requests. It must have a non `void` return type.

> It is possible (though not recommended for usability reasons) to annotate concrete class implementation.

You can have more than one method with the same attribute (except #[WorkflowMethod]).

For example:

```php
use Temporal\Workflow\WorkflowInterface;
use Temporal\Workflow\WorkflowMethod;
use Temporal\Workflow\SignalMethod;
use Temporal\Workflow\QueryMethod;

#[WorkflowInterface]
interface FileProcessingWorkflow
{
    #[WorkflowMethod]
    public function processFile(Argument $args);

    #[QueryMethod("history")]
    public function getHistory(): array;

    #[QueryMethod("status")]
    public function getStatus(): string;

    #[SignalMethod]
    public function retryNow(): void;

    #[SignalMethod]
    public function abandon(): void;
}
```

Note that name parameter of Workflow method attributes can be used to specify name of Workflow, Signal and Query types.
If name is not specified the short name of the Workflow interface is used.

In the preceding code the `#[WorkflowMethod(name)]` is not specified, thus the Workflow Type defaults to `"FileProcessingWorkflow"`.

### Handle a Signal 

Workflows listen for Signals by the Signal's name.

Use the `#[SignalMethod]` attribute to handle Signals in the Workflow interface:

```php
use Temporal\Workflow;

#[Workflow\WorkflowInterface]
class YourWorkflow
{
    private bool $value;

    #[Workflow\WorkflowMethod]
    public function run()
    {
        yield Workflow::await(fn()=> $this->value);
        return 'OK';
    }

    #[Workflow\SignalMethod]
    public function setValue(bool $value)
    {
        $this->value = $value;
    }
}
```

In the preceding example, the Workflow updates the protected value.
The main Workflow coroutine waits for the value to change by using the `Workflow::await()` function.

### Send a Signal from a Temporal Client 

When a Signal is sent successfully from the Temporal Client, the [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Event History of the Workflow that receives the Signal.

To send a Signal to a Workflow Execution from a Client, call the Signal method, annotated with `#[SignalMethod]` in the Workflow interface, from the Client code.

To send a Signal to a Workflow, use `WorkflowClient->newWorkflowStub` or `WorkflowClient->newUntypedWorkflowStub`:

```php
$workflow = $workflowClient->newWorkflowStub(YourWorkflow::class);

$run = $workflowClient->start($workflow);

// do something

$workflow->setValue(true);

assert($run->getValue() === true);
```

Use `WorkflowClient->newRunningWorkflowStub` or `WorkflowClient->newUntypedRunningWorkflowStub` with Workflow Id to send Signals to already running Workflows.

```php
$workflow = $workflowClient->newRunningWorkflowStub(YourWorkflow::class, 'workflowID');
$workflow->setValue(true);
```

See [Handle Signal](#handle-signal) for details on how to handle Signals in a Workflow.

### Send a Signal from a Workflow 

A Workflow can send a Signal to another Workflow, in which case it's called an _External Signal_.

When an External Signal is sent:

- A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History.
- A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History.

To send Signal to a Workflow use `WorkflowClient`->`newWorkflowStub` or `WorkflowClient`->`newUntypedWorkflowStub`:

```php
$workflow = $workflowClient->newWorkflowStub(YourWorkflow::class);

$run = $workflowClient->start($workflow);

// do something

$workflow->setValue(true);

assert($run->getValue() === true);
```

Use `WorkflowClient`->`newRunningWorkflowStub` or `WorkflowClient->newUntypedRunningWorkflowStub` with Workflow Id to send
Signals to a running Workflow.

```php
$workflow = $workflowClient->newRunningWorkflowStub(YourWorkflow::class, 'workflowID');
$workflow->setValue(true);
```

### Use Signal-With-Start 

Signal-With-Start is used from the Client.
It takes a Workflow Id, Workflow arguments, a Signal name, and Signal arguments.

If there's a Workflow running with the given Workflow Id, it will be signaled. If there isn't, a new Workflow will be started and immediately signaled.

In cases where you may not know if a Workflow is running, and want to send a Signal to it, use `startwithSignal`.
If a running Workflow exists, the `startwithSignal` API sends the Signal.
If there is no running Workflow, the API starts a new Workflow Run and delivers the Signal to it.

```php
$workflow = $workflowClient->newWorkflowStub(YourWorkflow::class);

$run = $workflowClient->startWithSignal(
    $workflow,
    'setValue',
    [true], // signal arguments
    [] // start arguments
);
```

## Develop with Queries 

A [Query](/sending-messages#sending-queries) is a synchronous operation that is used to get the state of a Workflow Execution.

### Define a Query 

A Query has a name and can have arguments.

- The name, also called a Query type, is a string.
- The arguments must be [serializable](/dataconversion).

Workflows can answer synchronous [Queries](/sending-messages#sending-queries) and receive [Signals](/sending-messages#sending-signals).

All interface methods must have one of the following attributes:

- **#[WorkflowMethod]** indicates an entry point to a Workflow.
  It contains parameters that specify timeouts and a Task Queue name.
  Required parameters (such as `executionStartToCloseTimeoutSeconds`) that are not specified through the attribute must be provided at runtime.
- **#[SignalMethod]** indicates a method that reacts to external signals. It must have a `void` return type.
- **#[QueryMethod]** indicates a method that reacts to synchronous query requests. It must have a non `void` return type.

> It is possible (though not recommended for usability reasons) to annotate concrete class implementation.

You can have more than one method with the same attribute (except #[WorkflowMethod]).

For example:

```php
use Temporal\Workflow\WorkflowInterface;
use Temporal\Workflow\WorkflowMethod;
use Temporal\Workflow\SignalMethod;
use Temporal\Workflow\QueryMethod;

#[WorkflowInterface]
interface FileProcessingWorkflow
{
    #[WorkflowMethod]
    public function processFile(Argument $args);

    #[QueryMethod("history")]
    public function getHistory(): array;

    #[QueryMethod("status")]
    public function getStatus(): string;

    #[SignalMethod]
    public function retryNow(): void;

    #[SignalMethod]
    public function abandon(): void;
}
```

Note that name parameter of Workflow method attributes can be used to specify name of Workflow, Signal and Query types.
If name is not specified the short name of the Workflow interface is used.

In the preceding code the `#[WorkflowMethod(name)]` is not specified, thus the Workflow Type defaults to `"FileProcessingWorkflow"`.

### Handle a Query 

Queries are handled by your Workflow.

Don't include any logic that causes [Command](/workflow-execution#command) generation within a Query handler (such as executing Activities).
Including such logic causes unexpected behavior.

You can add custom Query types to handle Queries such as Querying the current state of a
Workflow, or Querying how many Activities the Workflow has completed. To do this, you need to set
up a Query handler using method attribute `QueryMethod` or `Workflow::registerQuery`.

```php
#[Workflow\WorkflowInterface]
class YourWorkflow
{
    #[Workflow\QueryMethod]
    public function getValue()
    {
        return 42;
    }

    #[Workflow\WorkflowMethod]
    public function run()
    {
        // workflow code
    }
}
```

The handler function can receive any number of input parameters, but all input parameters must be
serializable. The following sample code sets up a Query handler that handles the Query type of
`currentState`:

```php
#[Workflow\WorkflowInterface]
class YourWorkflow
{
    private string $currentState;

    #[Workflow\QueryMethod('current_state')]
    public function getCurrentState(): string
    {
        return $this->currentState;
    }

    #[Workflow\WorkflowMethod]
    public function run()
    {
        // Your normal Workflow code begins here, and you update the currentState
        // as the code makes progress.
        $this->currentState = 'waiting timer';
        try{
            yield Workflow::timer(DateInterval::createFromDateString('1 hour'));
        } catch (\Throwable $e) {
            $this->currentState = 'timer failed';
            throw $e;
        }

        $yourActivity = Workflow::newActivityStub(
            YourActivityInterface::class,
            ActivityOptions::new()->withScheduleToStartTimeout(60)
        );

        $this->currentState = 'waiting activity';
        try{
            yield $yourActivity->doSomething('some input');
        } catch (\Throwable $e) {
            $this->currentState = 'activity failed';
            throw $e;
        }

        $this->currentState = 'done';

        return null;
    }
}
```

You can also issue a Query from code using the `QueryWorkflow()` API on a Temporal Client object.

Use `WorkflowStub` to Query Workflow instances from your Client code (can be applied to both running and closed Workflows):

```php
$workflow = $workflowClient->newWorkflowStub(
    YourWorkflow::class,
    WorkflowOptions::new()
);

$workflowClient->start($workflow);

var_dump($workflow->getCurrentState());
sleep(60);
var_dump($workflow->getCurrentState());
```

### Send a Query 

Queries are sent from a Temporal Client.

## Develop with Updates 

An [Update](/sending-messages#sending-updates) is an operation that can mutate the state of a Workflow Execution and return a response.

### Define Update 

Workflow Updates handlers are methods in your Workflow Definition designed to handle updates.
These updates can be triggered during the lifecycle of a Workflow Execution.

An Update handler has a name, arguments, response, and an optional validator.

- The name, also called an Update type, is a string.
- The arguments and response must be [serializable](/dataconversion).

The [`#[UpdateMethod]`](https://php.temporal.io/classes/Temporal-Workflow-UpdateMethod.html) attribute indicates that the method is used to handle and respond to update requests.

```php
#[UpdateMethod]
public function myUpdate(string $value);
```

### Handle Update 

Workflows listen for Update by the Update's name.

Use the `#[UpdateMethod]` attribute to handle Updates in the Workflow interface.
The handler method can accept multiple serializable input parameters, but it's recommended using only a single parameter.
The function can return a [serializable](/dataconversion) value or `void`.

```php
#[WorkflowInterface]
interface FileProcessingWorkflow {
    #[WorkflowMethod]
    #[ReturnType(ProcessResult::class)]
    public function processFile(File $file);

    #[UpdateMethod]
    public function pauseProcessing(): void;
}
```

Update handlers, unlike Query handlers, can change Workflow state.

The Updates type defaults to the name of the method.
To overwrite this default naming and assign a custom Update type, use the `#[UpdateMethod]` attribute with the `name` parameter.

```php
#[WorkflowInterface]
interface FileProcessingWorkflow {
    #[WorkflowMethod]
    public function processFiles(FileList $files);

    #[UpdateMethod(name: 'pause')]
    public function pauseProcessing(): void;
}
```

**Register Update Handler dynamically**

You can register Update handlers dynamically using the `Workflow::registerUpdate()` method.
The third argument is an optional Update validator. The validator function must have the same parameters as the handler and throw an exception if the validation fails.

```php
Workflow::registerUpdate(
    name: 'pause',
    handler: fn() => $this->paused = true,
    validator: fn() => $this->paused === false or throw new \Exception('Workflow is already paused'),
);
```

### Validate Update 

Validate certain aspects of the data sent to the Workflow using an Update Validator method. For instance, a counter Workflow might never want to accept a non-positive number. Use the [`#[UpdateValidatorMethod]`](https://php.temporal.io/classes/Temporal-Workflow-UpdateValidatorMethod.html) attribute and set the `forUpdate` argument to the name of your Update handler. Your Update Validator should accept the same input parameters as your Update Handler and return `void`.

```php
#[WorkflowInterface]
interface GreetingWorkflow {
    #[WorkflowMethod]
    public function getGreetings(): array;

    #[UpdateMethod]
    public function addGreeting(string $name): int;

    #[UpdateValidatorMethod(forUpdate: 'addGreeting')]
    public function addGreetingValidator(string $name): void;
}
```

### Send Update from a Client 

To send an Update to a Workflow Execution from a Client, call the Update method, annotated with `#[UpdateMethod]` in the Workflow interface, from the Client code.

In the following Client code example, start the Workflow `getGreetings` and call the Update method `addGreeting` that is handled in the Workflow.

```php
// Create a typed Workflow stub for GreetingsWorkflow
$workflow = $workflowClient->newWorkflowStub(GreetingWorkflow::class, $workflowOptions);

// Start the Workflow
$run = $workflowClient->start($workflow);

// Send an update to the Workflow. addGreeting returns
// the number of greetings our workflow has received.
$count = $workflow->addGreeting("World");
```

**Async accept**

In Workflow Update methods, all Workflow features are available, such as executing Activities and child Workflows, and waiting on timers/conditions.
In cases where it's known that the update will take a long time to execute, or you are not interested in the outcome of its execution, you can use the stub method [`startUpdate`](https://php.temporal.io/classes/Temporal-Client-WorkflowStubInterface.html#method_startUpdate) and move on immediately after receiving the validation result.
Note that the processing Workflow Worker must be available.
Otherwise, the request may block indefinitely or fail due to a timeout.

```php
use Ramsey\Uuid\UuidInterface;
use Temporal\Client\Update\UpdateOptions;
use Temporal\Client\Update\WaitPolicy;
use Temporal\Client\Update\LifecycleStage;

// Create an untyped Workflow stub for GreetingsWorkflow
$stub = $client->newUntypedWorkflowStub('GreetingWorkflow', $workflowOptions);

// Start the Workflow
$run = $client->start($stub);

// Send an Update to the Workflow. UpdateHandle returns
$handle = $stub->startUpdate('addGreeting', 'World');

// Use the UpdateHandle to get the Update result with timeout 2.5 seconds
$result = $handle->getResult(timeout: 2.5);

// You can get more control using UpdateOptions
$resultUuid = $stub->startUpdate(
    UpdateOptions::new('storeGreetings', LifecycleStage::StageCompleted)
        ->withResultType(UuidInterface::class)
 )->getResult();
```

#### Update-With-Start 

[Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](#send-update-from-client) that checks whether an already-running Workflow with that ID exists:

- If the Workflow exists, the Update is processed.
- If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute.

You can:

- Use the [`updateWithStart`](https://php.temporal.io/classes/Temporal-Client-WorkflowClientInterface.html#method_updateWithStart) WorkflowClient API.
  It returns once the requested Update wait stage has been reached; or when the request times out.
- Use the [`UpdateHandle`](https://php.temporal.io/classes/Temporal-Client-Update-UpdateHandle.html) to retrieve a result from the Update.

You provide:

- A WorkflowStub created from [`WorkflowOptions`](https://php.temporal.io/classes/Temporal-Client-WorkflowOptions.html).
  - The `WorkflowOptions` require a [Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) to be specified.
  - Choose ["Use Existing"](https://php.temporal.io/classes/Temporal-Common-WorkflowIdConflictPolicy.html#enumcase_UseExisting) and use an idempotent Update handler to ensure your code can be executed again in case of a Client failure.
    Not all `WorkflowOptions` are allowed.
    For example, specifying a Cron Schedule will result in an error.

- Update name or [`UpdateOptions`](https://php.temporal.io/classes/Temporal-Client-Update-UpdateOptions.html).
  This mirrors the approach used for [Update Workflow](#send-update-from-client).
  - For Update-with-Start, the Workflow Id is optional.
  - When specified, the Id must match the one used in `WorkflowOptions`.
  - Since a running Workflow Execution may not already exist, you can't set a Run Id.

For example:

```php
$stub = $workflowClient->newUntypedWorkflowStub(
    ShoppingCartWorkflow::class,
    WorkflowOptions::new()
        ->withTaskQueue('service-queue')
        ->withWorkflowId($cartId)
        ->withWorkflowIdConflictPolicy(WorkflowIdConflictPolicy::UseExisting),
);
$handle = $workflowClient->updateWithStart(
    workflow: $stub,
    update: 'addItem',
    updateArgs: [$itemId, $quantity],
);

$price = $handle->getResult();
```

To wait on the Update result, run the Update with the wait stage set to [`LifecycleStage::StageCompleted`](https://php.temporal.io/classes/Temporal-Client-Update-LifecycleStage.html#enumcase_StageCompleted).
This returns once the update result is available; or when the API call times out.

For example:

```php
$handle = $workflowClient->updateWithStart(
    workflow: $stub,
    update: UpdateOptions::new('addItem', LifecycleStage::StageCompleted),
    updateArgs: [$itemId, $quantity],
);

assert($handle->hasResult() === true);
$price = $handle->getResult();
```

## Message handler patterns 

This section covers common write operations, such as Signal and Update handlers.
It doesn't apply to pure read operations, like Queries or Update Validators.

> **💡 Tip:**
>
> For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing), and [this sample](https://github.com/temporalio/samples-php/tree/master/app/src/SafeMessageHandlers) demonstrating safe `async` message handling.

### Add wait conditions to block

Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue.
You can use a wait condition ([`Workflow::await()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_await)) to set a function that prevents the code from proceeding until the condition returns `true`.
This is an important feature that helps you control your handler logic.

Here are two important use cases for `Workflow::await()`:

- Waiting in a handler until it is appropriate to continue.
- Waiting in the main Workflow until all active handlers have finished.

The condition state you're waiting for can be updated by and reflect any part of the Workflow code.
This includes the main Workflow method, other handlers, or child coroutines spawned by the main Workflow method (see [`Workflow::async()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_async).

### Use wait conditions in handlers

It's common to use a Workflow wait condition to wait until a handler should start.
You can also use wait conditions anywhere else in the handler to wait for a specific condition to become `true`.
This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become `true`.

Consider a `readyForUpdateToExecute` method that runs before your Update handler executes.
The `Workflow::await` method waits until your condition is met:

```php
    #[UpdateMethod]
    public function myUpdate(UpdateInput $input)
    {
        yield Workflow::await(
            fn() => $this->readyForUpdateToExecute($input),
        );

        // ...
    }
```

Remember: Handlers can execute before the main Workflow method starts.

### Ensure your handlers finish before the Workflow completes 

Workflow wait conditions can ensure your handler completes before a Workflow finishes.
When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result.
The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying retrieve Update results.
Use [`Workflow::await()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_await) and [`Workflow::allHandlersFinished()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_allHandlersFinished) to address this problem and allow your Workflow to end smoothly:

```php
#[WorkflowInterface]
class MyWorkflow
{
    #[WorkflowMethod]
    public function run()
    {
        // ...
        yield Workflow::await(fn() => Workflow::allHandlersFinished());
        return "workflow-result";
    }
}
```

By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions.
You can silence these warnings on a per-handler basis by passing the `unfinishedPolicy` argument to the [`UpdateMethod`](https://php.temporal.io/classes/Temporal-Workflow-UpdateMethod.html) / [`SignalMethod`](https://php.temporal.io/classes/Temporal-Workflow-SignalMethod.html) attribute:

```php
#[UpdateMethod(unfinishedPolicy: HandlerUnfinishedPolicy::Abandon)]
public function myUpdate()
{
    // ...
}
```

See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information.

### Use `#[WorkflowInit]` to operate on Workflow input before any handler executes

Normally, your Workflows constructor won't have any parameters.
However, if you use the `#[WorkflowInit]` attribute on your constructor, you can give it the same [Workflow parameters](/develop/php/workflows/basics#workflow-parameters) as your `#[WorkflowMethod]`.
The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/php/client/temporal-client#start-workflow-execution).
The Workflow input arguments are also passed to your `#[WorkflowMethod]` method -- that always happens, whether or not you use the `#[WorkflowInit]` attribute.
This is useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/sending-messages).

Here's an example.
Notice that the constructor and `getGreeting` must have the same parameters:

```php
use Temporal\Workflow;

#[Workflow\WorkflowInterface]
class GreetingExample
{
    private readonly string $nameWithTitle;
    private bool $titleHasBeenChecked;

    // Note the attribute is on a public constructor
    #[Workflow\WorkflowInit]
    public function __construct(string $input)
    {
        $this->nameWithTitle = 'Sir ' . $input;
        $this->titleHasBeenChecked = false;
    }

    #[Workflow\WorkflowMethod]
    public function getGreeting(string $input)
    {
        yield Workflow::await(fn() => $this->titleHasBeenChecked);
        return "Hello " . $this->nameWithTitle;
    }

    #[Workflow\UpdateMethod]
    public function checkTitleValidity()
    {
        // 👉 The handler is now guaranteed to see the workflow input
        // after it has been processed by the constructor.
        $isValid = yield Workflow::executeActivity('activity.checkTitleValidity', [$this->nameWithTitle]);
        $this->titleHasBeenChecked = true;
        return $isValid;
    }
}
```

> **📝 Note:**
>
> By default, the Workflow Handler runs before Signals and Updates in PHP SDK v2. This behavior is incorrect.
> To avoid breaking already written Workflows, since PHP SDK v2.11.0, a [feature flag](https://php.temporal.io/classes/Temporal-Worker-FeatureFlags.html#property_workflowDeferredHandlerStart) was added to enhance the behavior of the Workflow Handler.
> Make sure to set this flag to `true` to enable the correct behavior.
>

### Use `Mutex` to prevent concurrent handler execution 

Concurrent processes can interact in unpredictable ways.
Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously.
Here's an example of a pathological case:

```php
use Temporal\Workflow;

#[Workflow\WorkflowInterface]
class MyWorkflow
{
    // ...

    #[Workflow\SignalMethod]
    public function badAsyncHandler()
    {
        $data = yield Workflow::executeActivity(
            type: 'fetch_data',
            args: ['url' => 'http://example.com'],
            options: ActivityOptions::new()->withStartToCloseTimeout('10 seconds'),
        );
        $this->x = $data->x;
        # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then
        # there may be times when the Workflow has $this->x from one Activity execution and $this->y from another.
        yield Workflow::timer(1);  # or await anything else
        $this->y = $data->y;
    }
}
```

Coordinating access using `Mutex` corrects this code.
Locking makes sure that only one handler instance can execute a specific section of code at any given time:

```php
use Temporal\Workflow;

#[Workflow\WorkflowInterface]
class MyWorkflow
{
    // ...

    private Workflow\Mutex $mutex;

    public function __construct()
    {
        $this->mutex = new Workflow\Mutex();
    }

    #[Workflow\SignalMethod]
    public function safeAsyncHandler()
    {
        $data = yield Workflow::executeActivity(
            type: 'fetch_data',
            args: ['url' => 'http://example.com'],
            options: ActivityOptions::new()->withStartToCloseTimeout('10 seconds'),
        );
        yield Workflow::runLocked($this->mutex, function () use ($data) {
            $this->x = $data->x;
            # ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow
            # method, but no other execution of this handler can run until this execution finishes.
            yield Workflow::timer(1);  # or await anything else
            $this->y = $data->y;
        });
    }
```

## Message handler troubleshooting 

When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors:

- **The client can't contact the server**:
  You'll receive a [`ServiceClientException`](https://php.temporal.io/classes/Temporal-Exception-Client-ServiceClientException.html) in case of a server connection error.
  [How to configure RPC Retry Policy](/develop/php/client/temporal-client#configure-rpc-retry-policy)

- **RPC timeout**:
  You'll receive a [`TimeoutException`](https://php.temporal.io/classes/Temporal-Exception-Client-TimeoutException.html) in case of an RPC timeout.
  [How to configure RPC timeout](/develop/php/client/temporal-client#configure-rpc-timeout)

- **The workflow does not exist**:
  You'll receive a [`WorkflowNotFoundException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowNotFoundException.html) exception.

See [Exceptions in message handlers](/handling-messages#exceptions) for a non–PHP-specific discussion of this topic.

### Problems when sending a Signal 

When using Signal, the only exception that will result from your requests during its execution is `ServiceClientException`.
All handlers may experience additional exceptions during the initial (pre-Worker) part of a handler request lifecycle.

For Queries and Updates, the client waits for a response from the Worker.
If an issue occurs during the handler Execution by the Worker, the client may receive an exception.

### Problems when sending an Update 

When working with Updates, you may encounter these errors:

- **No Workflow Workers are polling the Task Queue**:
  Your request will be retried by the SDK Client indefinitely.
  You can [configure RPC timeout](/develop/php/client/temporal-client#configure-rpc-timeout) to impose a timeout.
  This raises a [`WorkflowUpdateRPCTimeoutOrCanceledException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateRPCTimeoutOrCanceledException.html).

- **Update failed**: You'll receive a [`WorkflowUpdateException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateException.html) exception.
  There are two ways this can happen:

  - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler.

  - The Update failed after having been accepted.

Update failures are like [Workflow failures](/references/failures#errors-in-workflows).
Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.
These might include:

    - A failed Child Workflow
    - A failed Activity (if the Activity retries have been set to a finite number)
    - The Workflow author raising `ApplicationFailure`

- **The handler caused the Workflow Task to fail**:
  A [Workflow Task Failure](/references/failures#errors-in-workflows) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage:
  - If the request hasn't been accepted by the server, you receive a [`WorkflowUpdateException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateException.html).
  - If the request has been accepted, it is durable.
    Once the Workflow is healthy again after a code deploy, use an [`UpdateHandle`](https://php.temporal.io/classes/Temporal-Client-Update-UpdateHandle.html) to fetch the Update result.

- **The Workflow finished while the Update handler execution was in progress**:
  You'll receive a [`WorkflowUpdateException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateException.html).
  This happens if the Workflow finished while the Update handler execution was in progress, for example because

  - The Workflow was canceled or failed.

  - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers).

### Problems when sending a Query 

When working with Queries, you may encounter these errors:

- **There is no Workflow Worker polling the Task Queue**:
  You'll receive a [`WorkflowNotFoundException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowNotFoundException.html).

- **Query failed**:
  You'll receive a [`WorkflowQueryException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowQueryException.html) if something goes wrong during a Query.
  Any exception in a Query handler will trigger this error.
  This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead.

- **The handler caused the Workflow Task to fail.**
  This would happen, for example, if the Query handler blocks the thread for too long without yielding.

## Dynamic components 

Temporal supports Dynamic Queries, Signals, and Updates.
These are unnamed handlers that are invoked if no other statically defined handler with the given name exists.

Dynamic Handlers provide flexibility to handle cases where the names of Queries, Signals, or Updates aren't known at run time.

> **⚠️ Caution:**
>
> Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach.
> Overusing them can lead to maintainability and debugging issues down the line.
>
> Instead, Signals, or Queries should be defined statically whenever possible, with clear names that indicate their purpose.
> Use static definitions as the primary way of structuring your Workflows.
>
> Reserve Dynamic Handlers for cases where the handler names are not known at development time and need to be looked up dynamically at runtime.
> They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic.
>

### Set a Dynamic Query 

A Dynamic Query in Temporal is a Query method that is invoked dynamically at runtime if no other Query with the same name is registered.
Use [`Workflow::registerDynamicQuery()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_registerDynamicQuery) to set a dynamic Query handler.

The Query Handler parameters must accept a `string` name and [`ValuesInterface`](https://php.temporal.io/classes/Temporal-DataConverter-ValuesInterface.html) for the arguments.

```php
Workflow::registerDynamicQuery(function (string $name, ValuesInterface $arguments): string {
    return \sprintf(
        'Got query `%s` with %d arguments',
        $name,
        $arguments->count(),
    );
});
```

### Set a Dynamic Signal 

A Dynamic Signal in Temporal is a Signal that is invoked dynamically at runtime if no other Signal with the same input is registered.
Use [`Workflow::registerDynamicSignal()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_registerDynamicSignal) to set a dynamic Signal handler.

The Signal Handler parameters must accept a `string` name and [`ValuesInterface`](https://php.temporal.io/classes/Temporal-DataConverter-ValuesInterface.html) for the arguments.

```php
Workflow::registerDynamicSignal(function (string $name, ValuesInterface $arguments): void {
     Workflow::getLogger()->info(\sprintf(
         'Executed signal `%s` with %d arguments',
         $name,
         $arguments->count(),
     ));
 });
```

### Set a Dynamic Update 

A Dynamic Update in Temporal is an Update that is invoked dynamically at runtime if no other Update with the same input is registered.
Use [`Workflow::registerDynamicUpdate()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_registerDynamicUpdate) to set a dynamic Update handler.

The method accepts two arguments:

- Update Handler
- Update Validator (optional) that should throw an exception if the validation fails

Both the Handler and the Validator must accept a `string` name and [`ValuesInterface`](https://php.temporal.io/classes/Temporal-DataConverter-ValuesInterface.html) for the arguments.

```php
Workflow::registerDynamicUpdate(
    static fn(string $name, ValuesInterface $arguments): string => \sprintf(
        'Got update `%s` with %d arguments',
        $name,
        $arguments->count(),
    ),
    static fn(string $name, ValuesInterface $arguments) => \str_starts_with(
        $name,
        'update_',
    ) or throw new \InvalidArgumentException('Invalid update name'),
);
```

---

# Schedules - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/schedules

> Use Workflow Start Delay and Temporal Cron Jobs in PHP. Delay Workflow execution or set up recurring tasks with a Cron Schedule using Temporal Client.

This page shows how to do the following:

- [Use Start Delay](#start-delay)
- [Use Temporal Cron Jobs](#temporal-cron-jobs)

For recurring automation, Temporal recommends [Schedules](/schedule) instead of Cron Jobs.
If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and can affect future scheduled starts through the Schedule's [Overlap Policy](/schedule#overlap-policy).

## Use Start Delay 

Use the Workflow [Start Delay](/workflow-execution/timers-delays) functionality if you need to delay the execution of the Workflow without the need for regular launches.
Here you simply specify the time to wait before dispatching the first Workflow task.

```php
$workflow = $workflowClient->newWorkflowStub(
    GreeterWorkflowInterface::class,
    WorkflowOptions::new()
        ->withWorkflowStartDelay(CarbonInterval::minutes(10)),
);
$workflowClient->start($workflow, 'Hello world!');
```

## Use Temporal Cron Jobs 

> **⚠️ Caution:**
> Cron support is not recommended
>
> We recommend using [Schedules](/schedule) instead of Cron Jobs.
> Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules.
>

A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution.

A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made.

Set your Cron Schedule with `CronSchedule('* * * * *')`.
Temporal Workflow Schedule Cron strings follow this format:

```
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
* * * * *
```

The following example sets a Cron Schedule in PHP:

```php
  $workflow = $this->workflowClient->newWorkflowStub(
      CronWorkflowInterface::class,
      WorkflowOptions::new()
          ->withWorkflowId(CronWorkflowInterface::WORKFLOW_ID)
          ->withCronSchedule('* * * * *')
          // Execution timeout limits total time. Cron will stop executing after this timeout.
          ->withWorkflowExecutionTimeout(CarbonInterval::minutes(10))
          // Run timeout limits duration of a single workflow invocation.
          ->withWorkflowRunTimeout(CarbonInterval::minute(1))
  );

  $output->writeln("Starting <comment>CronWorkflow</comment>... ");

  try {
      $run = $this->workflowClient->start($workflow, 'Antony');
      // ...
  }
```

Setting `withCronSchedule` turns the Workflow Execution into a Temporal Cron Job.
For more information, see the [PHP samples](https://github.com/temporalio/samples-php/tree/master/app/src/Cron) for example code or the PHP SDK `WorkflowOptions` [source code](https://github.com/temporalio/sdk-php/blob/master/src/Client/WorkflowOptions.php).

> **💡 Tip:**
> Schedule Auto-Deletion
>
> Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again.
> The Temporal Service doesn't guarantee when this removal will happen.
>

---

# Side Effects - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/side-effects

> Use Side Effects in PHP to execute non-deterministic code like generating UUIDs or random numbers in a Workflow without compromising its determinism.

## Use Side Effects 

Side Effects are used to execute non-deterministic code, such as generating a UUID or a random number, without compromising determinism in the Workflow. This is done by storing the results of the Side Effect into the Workflow [Event History](/workflow-execution/event#event-history).

A Side Effect doesn't re-execute during a Replay. Instead, it returns the recorded result from the Workflow Execution Event History.

Side Effects shouldn't fail. An exception that is thrown from the Side Effect causes failure and retry of the current Workflow Task.

An Activity or a Local Activity can also be used instead of a Side Effect, as its results are also persisted in Workflow Execution History.

> **📝 Note:**
>
> You shouldn't modify the Workflow state inside a Side Effect, because they're not re-executed during Replay. Side Effect functions should only return a value, and that value can be used in Workflow code to alter state.
>

To use a Side Effect in PHP, use the `Workflow::sideEffect()` function in your Workflow Definition to run non-deterministic code and return a value.

```php
#[Workflow\WorkflowMethod]
public function run()
{
    $random = yield Workflow::sideEffect(fn() => random_int(0, 100));
    if ($random < 50) {
        // ...
    } else {
        // ...
    }
}
```

---

# Workflow Timeouts - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/timeouts

> Optimize Workflow Execution with Temporal PHP SDK - Set Workflow Timeouts and Retry Policies efficiently.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient.
Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes.
If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer.

Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts).

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed.
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last.
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task.

Create an instance of `WorkflowOptions` in the Client code and set your timeout.

Available timeouts are:

- `withWorkflowExecutionTimeout()`
- `withWorkflowRunTimeout()`
- `withWorkflowTaskTimeout()`

```php
$workflow = $this->workflowClient->newWorkflowStub(
    DynamicSleepWorkflowInterface::class,
    WorkflowOptions::new()
        ->withWorkflowId(DynamicSleepWorkflow::WORKFLOW_ID)
        ->withWorkflowIdReusePolicy(WorkflowIdReusePolicy::WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE)
        // Set Workflow Timeout duration
        ->withWorkflowExecutionTimeout(CarbonInterval::minutes(2))
        // ->withWorkflowRunTimeout(CarbonInterval::minute(2))
        // ->withWorkflowTaskTimeout(CarbonInterval::minute(2))
);
```

## Workflow retries 

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

A Retry Policy can be configured with an instance of the `RetryOptions` object.
To enable retries for a Workflow, you need to provide a Retry Policy object via `ChildWorkflowOptions` for Child Workflows or via `WorkflowOptions` for top-level Workflows.

```php
$workflow = $this->workflowClient->newWorkflowStub(
      CronWorkflowInterface::class,
      WorkflowOptions::new()->withRetryOptions(
        RetryOptions::new()->withInitialInterval(120)
      )
);
```

---

# Timers - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/timers

> Set a durable Timer in PHP with Workflow::timer(). Timers persist across Worker downtime, and a single Worker can run millions of them.

## What is a Timer? 

A Workflow can set a durable timer for a fixed time period.
In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`.

A Workflow can sleep for months.
Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing.

Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker.

To set a Timer in PHP, use `Workflow::timer()` and pass the number of seconds you want to wait before continuing.

The following example yields a sleep method for 5 minutes.

```php
yield Workflow::timer(300); // sleep for 5 minutes
```

You cannot set a Timer invocation inside the `await` or `awaitWithTimeout` methods.

---

# Versioning - PHP SDK

Source: https://docs.temporal.io/develop/php/workflows/versioning

> Ensure deterministic Temporal Workflow execution and deploy updates with the PHP SDK's patching and Worker Versioning APIs.

Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress.

The Temporal Platform requires that Workflow Definitions are [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably.

With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version.
There are two primary Versioning methods that you can use:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths.
- [Versioning with Patching](#php-sdk-patching-api). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.

## Worker Versioning

Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching.

## Versioning with Patching 

### Patching with `getVersion`

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow Executions, create a patch.

Suppose you have an initial Workflow that runs `prePatchActivity`:

```php
#[WorkflowInterface]
class MyWorkflow
{
    private $activity;

    public function __construct()
    {
        $this->activity = Workflow::newActivityStub(
            YourActivityInterface::class,
            ActivityOptions::new()->withScheduleToStartTimeout(60)
        );
    }

    #[WorkflowMethod]
    public function runAsync()
    {
        $result = yield $this->activity->prePatchActivity();
    }
}
```

Suppose you replaced `prePatchActivity` with `postPatchActivity` and deployed the updated code.

If an existing Workflow Execution was started by the original version of the Workflow code, where `prePatchActivity` was run, and then resumed running on a new Worker where it was replaced with `postPatchActivity`, the server side Event History would be out of sync.
This would cause the Workflow to fail with a nondeterminism error.

To resolve this, you can use [Workflow::getVersion](https://php.temporal.io/classes/Temporal-Workflow.html#method_getVersion) to patch to your Workflow:

```php
#[WorkflowInterface]
class MyWorkflow
{
    // ...

    #[WorkflowMethod]
    public function runAsync()
    {
        $version = yield Workflow::getVersion('Step 1', Workflow::DEFAULT_VERSION, 1);

        $result = $version === Workflow::DEFAULT_VERSION
            ? yield $this->activity->prePatchActivity()
            : yield $this->activity->postPatchActivity();
    }
}
```

When `getVersion()` is run for the new Workflow Execution, it records a marker in the Event History so that all future calls to `getVersion()` for this change Id — `Step 1` in the example — on this Workflow Execution will always return the given version number, which is `1` in the example.

If you make an additional change, such as adding `anotherPatchActivity()`, you need to
add some additional code:

```php
#[WorkflowInterface]
class MyWorkflow
{
    // ...

    #[WorkflowMethod]
    public function runAsync()
    {
        $version = yield Workflow::getVersion('Step 1', Workflow::DEFAULT_VERSION, maxSupported: 2);

        $result = match($version) {
            Workflow::DEFAULT_VERSION => yield $this->activity->prePatchActivity()
            1 => yield $this->activity->postPatchActivity();
            2 => yield $this->activity->anotherPatchActivity();
        };
    }
}
```

Note that we changed `maxSupported` from 1 to 2.
A Workflow that has already passed this `getVersion()` call before it was introduced returns `DEFAULT_VERSION`.
A Workflow that was run with `maxSupported` set to 1 returns 1.
New Workflows return 2.

After all the Workflow Executions prior to version 1 have left retention, you can remove the code for that version:

```php
    #[WorkflowMethod]
    public function runAsync()
    {
        $version = yield Workflow::getVersion('Step 1', minSupported: 1, maxSupported: 2);

        $result = match($version) {
            1 => yield $this->activity->postPatchActivity();
            2 => yield $this->activity->anotherPatchActivity();
        };
    }
```

You'll note that `minSupported` has changed from `DEFAULT_VERSION` to `1`.
If an older version of the Workflow Execution history is replayed on this code, it fails because the minimum expected version is 1.
After all the Workflow Executions for version 1 have left retention, you can remove version 1 so that your code looks like the following:

```php
    #[WorkflowMethod]
    public function runAsync()
    {
        $version = yield Workflow::getVersion('Step 1', minSupported: 2, maxSupported: 2);

        $result = yield $this->activity->anotherPatchActivity();
    }
```

Patching allows you to make changes to currently running Workflows.
It introduces compatible changes without causing non-determinism errors.

### Workflow cutovers

To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow.

Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes.
To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers.

For example, you would duplicate `MyWorkflow` as `MyWorkflowV2`:

```php
#[WorkflowInterface]
class MyWorkflow
{}

#[WorkflowInterface]
class MyWorkflowV2
{}
```

You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types.
The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow.
This can become impractical over time.
This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching.

## Runtime checking 

The Temporal PHP SDK performs a runtime check to help prevent obvious incompatible changes.
Adding, removing, or reordering any of these methods without Versioning triggers the runtime check and results in a nondeterminism error:

- `workflow.ExecuteActivity()`
- `workflow.ExecuteChildWorkflow()`
- `workflow.NewTimer()`
- `workflow.RequestCancelWorkflow()`
- `workflow.SideEffect()`
- `workflow.SignalExternalWorkflow()`
- `workflow.Sleep()`

The runtime check does not perform a thorough check.
For example, it does not check on the Activity's input arguments or the Timer duration.
Each Temporal SDK implements these sanity checks differently, and they are not a complete check for non-deterministic changes.
Instead, you should incorporate [Replay Testing](/develop/php/best-practices/testing-suite#replay) when making revisions.

---

# Plugins guide

Source: https://docs.temporal.io/develop/plugins-guide

> Best practices for creating plugins for AI Agents

# Plugins

A **Plugin** is an abstraction that allows you to customize any aspect of your Temporal Worker setup, including
registering Workflow and Activity definitions, modifying worker and client options, and more. Using plugins, you can
build reusable open-source libraries or build add-ons for engineers at your company.

This guide will teach you how to create plugins and give platform engineers general guidance on using and managing
Temporal's primitives.

Here are some common use cases for plugins:

- AI Agent SDKs
- Observability, tracing, or logging middleware
- Adding reliable built-in functionality such as LLM calls, messaging systems, and payments infrastructure
- Encryption or compliance middleware

## How to build a Plugin

The recommended way to start building plugins is with a `SimplePlugin`. This abstraction will tackle the vast majority
of plugins people want to write.

### Example Plugins

If you prefer to learn by getting hands-on with code, check out some existing plugins.

- Temporal maintains an
  [OpenAI Agents SDK](https://github.com/temporalio/ai-integrations/tree/main/python/openai_agents) plugin for Python
- Temporal maintains a
  [LangGraph](https://github.com/temporalio/ai-integrations/tree/main/python/langgraph) plugin for Python
- [Temporal client and Worker plugin for Pydantic AI](https://github.com/pydantic/pydantic-ai/blob/d9b4b2540183a4426669b2824c87cdfc36144780/pydantic_ai_slim/pydantic_ai/durable_exec/temporal/__init__.py#L142)
- Temporal's TypeScript SDK ships with an
  [OpenTelemetry Plugin](https://github.com/temporalio/sdk-typescript/blob/main/contrib/interceptors-opentelemetry/src/plugin.ts)

## What you can provide to users in a plugin

There are a number of features you can give your users with a plugin. Here's a short list of some of the things you can
do.

- [Built-in Activities](#built-in-activity)
- [Workflow-friendly libraries](#workflow-friendly-libraries)
- [Built-in Workflows](#built-in-workflows)
- [Built-in Nexus Operations](#built-in-nexus-operations)
- [Custom Data Converters](#custom-data-converters)
- [Interceptors](#interceptors)
- [Context Propagators](#context-propagators)

### Built-in Activity

You can provide built-in Activities in a Plugin for users to call from their Workflows. Check out the
[Activities](/activities) page for more details on how they work.

Refer to the [best practices for creating Activities](/activity-definition#best-practices-for-defining-activities) when
you are making Activity plugins.

#### Timeouts and retry policies

Temporal's Activity retry mechanism gives applications the benefits of Durable Execution. See the
[Activity retry policy explanation](/activity-definition#activity-retry-policy) for more details.

**Python**

    [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py)
    ```py
    @activity.defn
    async def some_activity() -> None:
        return None

    plugin = SimplePlugin("organization.PluginName", activities=[some_activity])
    ```

**Go**

    [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go)
    ```go
    func SomeActivity(ctx context.Context) error {
    	// Activity implementation
    	return nil
    }

    func createActivityPlugin() (*temporal.SimplePlugin, error) {
    	return temporal.NewSimplePlugin(temporal.SimplePluginOptions{
    		Name: "organization.PluginName",
    		RunContextBefore: func(ctx context.Context, options temporal.SimplePluginRunContextBeforeOptions) error {
    			options.Registry.RegisterActivityWithOptions(
    				SomeActivity,
    				activity.RegisterOptions{Name: "SomeActivity"},
    			)
    			return nil
    		},
    	})
    }

    ```

**Java**

    [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java)
    ```java
    @ActivityInterface
    public interface SomeActivity {
      @ActivityMethod
      void someActivity();
    }

    public class SomeActivityImpl implements SomeActivity {
      @Override
      public void someActivity() {
        // Activity implementation
      }
    }

    SimplePlugin activityPlugin =
        SimplePlugin.newBuilder("organization.PluginName")
            .registerActivitiesImplementations(new SomeActivityImpl())
            .build();
    ```

**TypeScript**

    [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts)
    ```ts
    const activity = async () => 'activity';
    const plugin = new SimplePlugin({
      name: 'organization.PluginName',
      activities: {
        pluginActivity: activity,
      },
    });
    ```

**.NET**

    [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs)
    ```cs
    [Activity]
    static void SomeActivity() => throw new NotImplementedException();

    SimplePlugin activityPlugin = new SimplePlugin(
        "organization.PluginName",
        new SimplePluginOptions() { }.AddActivity(SomeActivity));
    ```

**Ruby**

    [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb)
    ```rb
    def some_activity
      # Activity implementation
    end

    plugin = Temporalio::SimplePlugin.new(
      name: 'organization.PluginName',
      activities: [method(:some_activity)]
    )
    ```

### Workflow-friendly libraries

You can provide a library for use within a Workflow if you'd like to abstract away some Temporal-specific details for
your users. Your library will call elements you include in your Plugin such as Activities, Child Workflows, Signals,
Updates, Queries, Nexus Operations, Interceptors, Data Converters, and any other code as long as it follows these
requirements:

- Any code that runs in the Workflow context must be [deterministic](/workflow-definition#deterministic-constraints),
  meaning it produces the same commands and results when replayed. For example, don't call system time APIs, generate
  random values, or perform direct network and file I/O from Workflow-context code; move that work to Activities or
  Nexus Operations.
- See [observability](/evaluate/features/observability) to avoid duplicating observation side
  effects when Workflows replay.
- Put other side effects inside of Activities or [Local Activities](/local-activity). This helps your Workflow handle
  being restarted, resumed, or executed in a different process from where it originally began without losing correctness
  or state consistency.
- See [testing your Plugin](#testing-your-plugin) to write tests that check for issues with side effects.
- It should run quickly since it may be replayed many times during a long Workflow execution. More expensive code should
  go in Activities or Nexus Operations.

A Plugin should allow a user to decompose their Workflows into Activities, as well as Child Workflows and Nexus Calls
when needed. This gives users granular control through retries and timeouts, debuggability through the Temporal UI,
operability with resets, pauses, and cancels, memoization for efficiency and resumability, and scalability using task
queues and Workers.

Users use Workflows for:

- Orchestration and decision-making
- Interactivity via [message-passing](/evaluate/features/workflow-message-passing)
- Tracing and observability

#### Making changes to your library

Your users may want to keep their Workflows running across deployments of their Worker code. If their deployment
includes a new version of your Plugin, changes to your Plugin could break Workflow code that started before the new
version was deployed. This can be due to
[non-deterministic behavior from code changes](/workflow-definition#non-deterministic-change) in your Plugin.

See [testing](#testing-your-plugin) to see how to test for this. And, if you make substantive changes, you need to use
[patching](/patching).

#### Example of a Workflow library that uses a Plugin in Python

- [Implementation of the `OpenAIAgentsPlugin`](https://github.com/temporalio/ai-integrations/tree/main/python/openai_agents/src/temporalio/openai_agents)
- [Example of replay testing](https://github.com/temporalio/ai-integrations/blob/main/python/openai_agents/tests/test_openai_replay.py)

### Built-in Workflows

You can provide a built-in Workflow in a `SimplePlugin`. It’s callable as a Child Workflow or standalone. When you want
to provide a piece of functionality that's more complex than an Activity, you can:

- Use a [Workflow Library](#workflow-friendly-libraries) that runs directly in the end user’s Workflow
- Add a Child Workflow

Consider adding a Child Workflow when one or more of these conditions applies:

- That child should outlive the parent.
- The Workflow Event History would otherwise [not scale](/workflow-execution/event#event-history-limits) in parent
  Workflows.
- When you want a separate Workflow ID for the child so that it can be operated independently of the parent's state
  (canceled, terminated, paused).

Any Workflow can be run as a standalone Workflow or as a Child Workflow, so registering a Child Workflow in a
`SimplePlugin` is the same as registering any Workflow.

**Python**

    [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py)
    ```py
    @workflow.defn
    class HelloWorkflow:
        @workflow.run
        async def run(self, name: str) -> str:
            return f"Hello, {name}!"

    plugin = SimplePlugin("organization.PluginName", workflows=[HelloWorkflow])
    ```

**Go**

    [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go)
    ```go
    func HelloWorkflow(ctx workflow.Context, name string) (string, error) {
    	return "Hello, " + name + "!", nil
    }

    func createWorkflowPlugin() (*temporal.SimplePlugin, error) {
    	return temporal.NewSimplePlugin(temporal.SimplePluginOptions{
    		Name: "organization.PluginName",
    		RunContextBefore: func(ctx context.Context, options temporal.SimplePluginRunContextBeforeOptions) error {
    			options.Registry.RegisterWorkflowWithOptions(
    				HelloWorkflow,
    				workflow.RegisterOptions{Name: "HelloWorkflow"},
    			)
    			return nil
    		},
    	})
    }

    ```

**Java**

    [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java)
    ```java
    @WorkflowInterface
    public interface HelloWorkflow {
      @WorkflowMethod
      String run(String name);
    }

    public static class HelloWorkflowImpl implements HelloWorkflow {
      @Override
      public String run(String name) {
        return "Hello, " + name + "!";
      }
    }

    SimplePlugin workflowPlugin =
        SimplePlugin.newBuilder("organization.PluginName")
            .registerWorkflowImplementationTypes(HelloWorkflowImpl.class)
            .build();
    ```

**.NET**

    [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs)
    ```cs
    [Workflow]
    class SimpleWorkflow
    {
        [WorkflowRun]
        public Task<string> RunAsync(string name) => Task.FromResult($"Hello, {name}!");
    }

    SimplePlugin workflowPlugin = new SimplePlugin(
        "organization.PluginName",
        new SimplePluginOptions() { }.AddWorkflow<SimpleWorkflow>());
    ```

**Ruby**

    [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb)
    ```rb
    class HelloWorkflow < Temporalio::Workflow::Definition
      def execute(name)
        "Hello, #{name}!"
      end
    end

    plugin = Temporalio::SimplePlugin.new(
      name: 'organization.PluginName',
      workflows: [HelloWorkflow]
    )
    ```

### Built-in Nexus Operations

Nexus calls are used from Workflows similar to Activities and you can learn more about [Temporal Nexus](/nexus). Like Activities, Nexus operation inputs and return values must be serializable.

**Python**

    [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py)
    ```py
    @nexusrpc.service
    class WeatherService:
        get_weather_nexus_operation: nexusrpc.Operation[WeatherInput, Weather]

    @nexusrpc.handler.service_handler(service=WeatherService)
    class WeatherServiceHandler:
        @nexusrpc.handler.sync_operation
        async def get_weather_nexus_operation(
            self, ctx: nexusrpc.handler.StartOperationContext, input: WeatherInput
        ) -> Weather:
            return Weather(
                city=input.city,
                temperature_range="14-20C",
                conditions="Sunny with wind.",
            )

    plugin = SimplePlugin(
        "organization.PluginName", nexus_service_handlers=[WeatherServiceHandler()]
    )
    ```

**Go**

    [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go)
    ```go
    type WeatherInput struct {
    	City string `json:"city"`
    }

    type Weather struct {
    	City             string `json:"city"`
    	TemperatureRange string `json:"temperatureRange"`
    	Conditions       string `json:"conditions"`
    }

    var WeatherService = nexus.NewService("weather-service")

    var GetWeatherOperation = nexus.NewSyncOperation(
    	"get-weather",
    	func(ctx context.Context, input WeatherInput, options nexus.StartOperationOptions) (Weather, error) {
    		return Weather{
    			City:             input.City,
    			TemperatureRange: "14-20C",
    			Conditions:       "Sunny with wind.",
    		}, nil
    	},
    )

    func createNexusPlugin() (*temporal.SimplePlugin, error) {
    	return temporal.NewSimplePlugin(temporal.SimplePluginOptions{
    		Name: "organization.PluginName",
    		RunContextBefore: func(ctx context.Context, options temporal.SimplePluginRunContextBeforeOptions) error {
    			options.Registry.RegisterNexusService(WeatherService)
    			return nil
    		},
    	})
    }

    ```

**Java**

    [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java)
    ```java
    // Example Nexus service implementation
    public class WeatherService {
      public Weather getWeather(WeatherInput input) {
        return new Weather(input.getCity(), "14-20C", "Sunny with wind.");
      }
    }

    public static class Weather {
      private final String city;
      private final String temperatureRange;
      private final String conditions;

      public Weather(String city, String temperatureRange, String conditions) {
        this.city = city;
        this.temperatureRange = temperatureRange;
        this.conditions = conditions;
      }

      // Getters...
    }

    public static class WeatherInput {
      private final String city;

      public WeatherInput(String city) {
        this.city = city;
      }

      public String getCity() {
        return city;
      }
    }

    SimplePlugin nexusPlugin =
        SimplePlugin.newBuilder("organization.PluginName")
            .registerNexusServiceImplementation(new WeatherService())
            .build();
    ```

**TypeScript**

    [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts)
    ```ts
    const testServiceHandler = nexus.serviceHandler(
      nexus.service('testService', {
        testSyncOp: nexus.operation<string, string>(),
      }),
      {
        async testSyncOp(_, input) {
          return input;
        },
      },
    );
    const plugin = new SimplePlugin({
      name: 'organization.PluginName',
      nexusServices: [testServiceHandler],
    });
    ```

**.NET**

    [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs)
    ```cs
    [NexusService]
    public interface IStringService
    {
        [NexusOperation]
        string DoSomething(string name);
    }

    [NexusServiceHandler(typeof(IStringService))]
    public class HandlerFactoryStringService
    {
        private readonly Func<IOperationHandler<string, string>> handlerFactory;

        public HandlerFactoryStringService(Func<IOperationHandler<string, string>> handlerFactory) =>
            this.handlerFactory = handlerFactory;

        [NexusOperationHandler]
        public IOperationHandler<string, string> DoSomething() => handlerFactory();
    }

    SimplePlugin nexusPlugin = new SimplePlugin(
        "organization.PluginName",
        new SimplePluginOptions() { }.AddNexusService(new HandlerFactoryStringService(() =>
            OperationHandler.Sync<string, string>((ctx, name) => $"Hello, {name}")))
    );
    ```

### Custom Data Converters

A [custom Data Converter](/default-custom-data-converters#custom-data-converter) can alter data formats or provide compression or encryption.

Note that you can use an existing Data Converter such as, in Python, `PydanticPayloadConverter` in your Plugin.

**Python**

    [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py)
    ```py
    def set_converter(converter: DataConverter | None) -> DataConverter:
        if converter is None or converter == DataConverter.default:
            return pydantic_data_converter
        # Should consider interactions with other plugins,
        # as this will override the data converter.
        # This may mean failing, warning, or something else
        return converter

    plugin = SimplePlugin("organization.PluginName", data_converter=set_converter)
    ```

**Go**

    [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go)
    ```go
    func createConverterPlugin() (*temporal.SimplePlugin, error) {
    	customConverter := converter.GetDefaultDataConverter() // Or your custom converter

    	return temporal.NewSimplePlugin(temporal.SimplePluginOptions{
    		Name:          "organization.PluginName",
    		DataConverter: customConverter,
    	})
    }

    ```

**Java**

    [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java)
    ```java
    SimplePlugin converterPlugin =
        SimplePlugin.newBuilder("organization.PluginName")
            .customizeDataConverter(
                existingConverter -> {
                  // Customize the data converter
                  // This example keeps the existing converter unchanged
                  // In practice, you might wrap it with additional functionality
                  return existingConverter;
                })
            .build();
    ```

**TypeScript**

    [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts)
    ```ts
    const codec: PayloadCodec = {
      encode(_payloads: Payload[]): Promise<Payload[]> {
        throw new Error();
      },
      decode(_payloads: Payload[]): Promise<Payload[]> {
        throw new Error();
      },
    };
    const plugin = new SimplePlugin({
      name: 'organization.PluginName',
      dataConverter: (converter: DataConverter | undefined) => ({
        ...converter,
        payloadCodecs: [...(converter?.payloadCodecs ?? []), codec],
      }),
    });
    ```

**.NET**

    [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs)
    ```cs
    private class Codec : IPayloadCodec
    {
        public Task<IReadOnlyCollection<Payload>> EncodeAsync(IReadOnlyCollection<Payload> payloads) => throw new NotImplementedException();
        public Task<IReadOnlyCollection<Payload>> DecodeAsync(IReadOnlyCollection<Payload> payloads) => throw new NotImplementedException();
    }

    SimplePlugin converterPlugin = new SimplePlugin(
        "organization.PluginName",
        new SimplePluginOptions()
        {
            DataConverterOption = new SimplePluginOptions.SimplePluginOption<DataConverter>(
                (converter) => converter with { PayloadCodec = new Codec() }
            ),
        });
    ```

**Ruby**

    [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb)
    ```rb
    custom_converter = Temporalio::Converters::DataConverter.new(
      payload_converter: Temporalio::Converters::PayloadConverter.default
    )

    plugin = Temporalio::SimplePlugin.new(
      name: 'organization.PluginName',
      data_converter: custom_converter
    )
    ```

### Interceptors

Interceptors are middleware that can run before and after various calls such as Activities, Workflows, and Signals. See [Interceptors](/encyclopedia/interceptors) for the interceptor model and links to SDK-specific implementation guides. They're used to:

- Create side effects such as logging and tracing.
- Modify arguments, such as adding headers for authorization or tracing propagation.

**Python**

    [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py)
    ```py
    class SomeWorkerInterceptor(temporalio.worker.Interceptor):
        pass  # Your implementation

    class SomeClientInterceptor(temporalio.client.Interceptor):
        pass  # Your implementation

    plugin = SimplePlugin(
        "organization.PluginName",
        interceptors=[SomeWorkerInterceptor(), SomeClientInterceptor()],
    )
    ```

**Go**

    [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go)
    ```go
    type SomeWorkerInterceptor struct {
    	interceptor.WorkerInterceptorBase
    }

    type SomeClientInterceptor struct {
    	interceptor.ClientInterceptorBase
    }

    func createInterceptorPlugin() (*temporal.SimplePlugin, error) {
    	return temporal.NewSimplePlugin(temporal.SimplePluginOptions{
    		Name:               "organization.PluginName",
    		WorkerInterceptors: []interceptor.WorkerInterceptor{&SomeWorkerInterceptor{}},
    		ClientInterceptors: []interceptor.ClientInterceptor{&SomeClientInterceptor{}},
    	})
    }

    ```

**Java**

    [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java)
    ```java
    public class SomeWorkerInterceptor extends WorkerInterceptorBase {
      // Your worker interceptor implementation
    }

    public class SomeClientInterceptor extends WorkflowClientInterceptorBase {
      // Your client interceptor implementation
    }

    SimplePlugin interceptorPlugin =
        SimplePlugin.newBuilder("organization.PluginName")
            .addWorkerInterceptors(new SomeWorkerInterceptor())
            .addClientInterceptors(new SomeClientInterceptor())
            .build();
    ```

**TypeScript**

    [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts)
    ```ts
    class MyWorkflowClientInterceptor implements WorkflowClientInterceptor {}

    class MyActivityInboundInterceptor implements ActivityInboundCallsInterceptor {}

    class MyActivityOutboundInterceptor implements ActivityOutboundCallsInterceptor {}

    const workflowInterceptorsPath = '';

    const plugin = new SimplePlugin({
      name: 'organization.PluginName',
      clientInterceptors: {
        workflow: [new MyWorkflowClientInterceptor()],
      },
      workerInterceptors: {
        client: {
          workflow: [new MyWorkflowClientInterceptor()],
        },
        workflowModules: [workflowInterceptorsPath],
        activity: [
          (_: Context) => ({
            inbound: new MyActivityInboundInterceptor(),
            outbound: new MyActivityOutboundInterceptor(),
          }),
        ],
      },
    });
    ```

**.NET**

    [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs)
    ```cs
    private class SomeClientInterceptor : IClientInterceptor
    {
        public ClientOutboundInterceptor InterceptClient(
            ClientOutboundInterceptor nextInterceptor) =>
            throw new NotImplementedException();
    }

    private class SomeWorkerInterceptor : IWorkerInterceptor
    {
        public WorkflowInboundInterceptor InterceptWorkflow(
            WorkflowInboundInterceptor nextInterceptor) =>
            throw new NotImplementedException();

        public ActivityInboundInterceptor InterceptActivity(
            ActivityInboundInterceptor nextInterceptor) =>
            throw new NotImplementedException();
    }

    SimplePlugin interceptorPlugin = new SimplePlugin(
        "organization.PluginName",
        new SimplePluginOptions()
        {
            ClientInterceptors = new List<IClientInterceptor>() { new SomeClientInterceptor() },
            WorkerInterceptors = new List<IWorkerInterceptor>() { new SomeWorkerInterceptor() },
        });
    ```

**Ruby**

    [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb)
    ```rb
    class SomeWorkerInterceptor
      include Temporalio::Worker::Interceptor::Workflow

      def intercept_workflow(next_interceptor)
        # Your interceptor implementation
        next_interceptor
      end
    end

    class SomeClientInterceptor
      include Temporalio::Client::Interceptor

      def intercept_client(next_interceptor)
        # Your interceptor implementation
        next_interceptor
      end
    end

    plugin = Temporalio::SimplePlugin.new(
      name: 'organization.PluginName',
      client_interceptors: [SomeClientInterceptor.new],
      worker_interceptors: [SomeWorkerInterceptor.new]
    )
    ```

### Context propagators 

Context propagators pass custom key-value data (such as tracing IDs, tenant IDs, or auth tokens) across Workflow, Activity, and Child Workflow boundaries via Temporal headers. See [Context Propagation](/encyclopedia/context-propagation) for details on how they work.

Propagators registered via a Plugin are appended to any propagators already set by the user or by previous plugins.

**Go**

```go
func createContextPropagatorPlugin() (*temporal.SimplePlugin, error) {
    return temporal.NewSimplePlugin(temporal.SimplePluginOptions{
        Name:               "organization.PluginName",
        ContextPropagators: []workflow.ContextPropagator{NewMyPropagator()},
    })
}
```

### Special considerations for different languages

Each of the SDKs has nuances you should be aware of so you can account for it in your code.

#### Python

You can choose to [run your Workflows in a sandbox in Python](/develop/python/best-practices/python-sdk-sandbox). This lets you run
Workflow code in a sandbox environment to help prevent non-determinism errors in your application. To work for users who
use sandboxing, your Plugin should specify the Workflow runner that it uses.

[features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py)
```py
def workflow_runner(runner: WorkflowRunner | None) -> WorkflowRunner:
    if not runner:
        raise ValueError("No WorkflowRunner provided to the plugin.")

    # If in sandbox, add additional passthrough
    if isinstance(runner, SandboxedWorkflowRunner):
        return dataclasses.replace(
            runner,
            restrictions=runner.restrictions.with_passthrough_modules("module"),
        )
    return runner

plugin = SimplePlugin("organization.PluginName", workflow_runner=workflow_runner)
```

#### TypeScript

TypeScript bundles cannot provide built-in Workflows because the TypeScript SDK bundles all Workflow code from a single
module. Plugin users must import and re-export any Plugin-provided Workflows from their own Workflow module so the
bundle includes them.

Users of a plugin which provides Workflow interceptors should always provide the plugin to the bundler if bundling. If
you aren't aware of the exact function of the plugin, you can always provide it, as it won't have any adverse effects.

[features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts)
```ts
const bundle = await bundleWorkflowCode({
  workflowsPath: require.resolve('./workflows'),
  plugins: [plugin],
});

const worker = await Worker.create({
  connection,
  taskQueue: 'my-task-queue',
  workflowBundle: bundle,
  plugins: [plugin],
});
```

## Testing your Plugin 

To test your Plugin, you'll write a normal Temporal Workflow tests, having included the plugin in your Client.

Two special concerns are versioning tests, for when you're making changes to your plugin, and testing unwanted side
effects.

### Versioning tests

When you make changes to your plugin after it has already shipped to users, we recommend that you set up [replay testing](/develop/python/best-practices/testing-suite#replay) on each important change to make sure that you’re not causing non-determinism errors for your users.

### Side effects tests

Your Plugin should cater to Workflows resuming in different processes than the ones they started on and then replaying
from the beginning, which can happen, for example, after an intermittent failure.

You can ensure you're not depending on local side effects by turning Workflow caching off, which will mean that the
Workflow replays from the top each time it progresses:

**Python**

    [features/snippets/worker/worker.py](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.py)
    ```py
    worker = Worker(client, task_queue="task-queue", max_cached_workflows=0)
    ```

**Go**

    [features/snippets/worker/worker.go](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.go)
    ```go
    worker.SetStickyWorkflowCacheSize(0)
    w := worker.New(c, "task-queue", worker.Options{})
    ```

**Java**

    [features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java)
    ```java
    WorkerFactory factory =
        WorkerFactory.newInstance(
            client, WorkerFactoryOptions.newBuilder().setWorkflowCacheSize(0).build());
    Worker worker = factory.newWorker("task-queue");
    ```

**TypeScript**

    [features/snippets/worker/worker.ts](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.ts)
    ```ts
    const worker = await Worker.create({
      connection,
      taskQueue: 'task-queue',
      maxCachedWorkflows: 0,
    });
    ```

**.NET**

    [features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs)
    ```cs
    using var worker = new TemporalWorker(
        client,
        new TemporalWorkerOptions("task-queue")
        {
            MaxCachedWorkflows = 0
        });
    ```

**Ruby**

    [features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb)
    ```rb
    worker = Temporalio::Worker.new(
      client: client,
      task_queue: 'task-queue',
      max_cached_workflows: 0
    )
    ```

Inside this replay regime, you should test for duplicate side effects or other types of failures. When testing for duplicate side effects, it may not be sufficient to simply have a counter that increments once per effect, as activities may be retried. Instead, consider these patterns:

- count `ActivityTaskScheduled` events of the expected activity type in event history (one per intended call, independent of retries), 
- or accumulate activity IDs in a concurrency-safe set and assert on its size (different scheduled activities get different IDs; retries of the same scheduled activity share one).

It's harder to test against side effects to global variables, so this practice is best avoided entirely.

---

# Python SDK developer guide

Source: https://docs.temporal.io/develop/python

> Explore Temporal Python SDK feature guides to master developing Temporal Applications. Build Workflows, Activities, and Workers, connect to Temporal Services, set up a testing suite, handle failure detection, send messages, complete Activities asynchronously, implement Versioning, use Observability, debug applications, schedule Workflows

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Install and get started

You can find detailed installation instructions for the Python SDK in the [Quickstart](/develop/python/set-up-your-local-python).

There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running.

Once your local Temporal Service is set up, continue building with the following resources:

- [Activity basics](/develop/python/activities/basics)
- [Workflow basics](/develop/python/workflows/basics)
- [Start an Activity Execution](/develop/python/activities/execution)
- [Run a Worker](/develop/python/workers/run-worker-process)

From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases.

## [Workflows](/develop/python/workflows)

- [Workflow basics](/develop/python/workflows/basics)
- [Child Workflows](/develop/python/workflows/child-workflows)
- [Continue-As-New](/develop/python/workflows/continue-as-new)
- [Cancellation](/develop/python/workflows/cancellation)
- [Timeouts](/develop/python/workflows/timeouts)
- [Message passing](/develop/python/workflows/message-passing)
- [Schedules](/develop/python/workflows/schedules)
- [Timers](/develop/python/workflows/timers)
- [Versioning](/develop/python/workflows/versioning)
- [Workflow Streams](/develop/python/workflows/workflow-streams)

## [Activities](/develop/python/activities)

- [Activity basics](/develop/python/activities/basics)
- [Activity Execution](/develop/python/activities/execution)
- [Standalone Activities](/develop/python/activities/standalone-activities-quickstart)
- [Timeouts](/develop/python/activities/timeouts)
- [Asynchronous Activity Completion](/develop/python/activities/asynchronous-activity)
- [Benign exceptions](/develop/python/activities/benign-exceptions)

## [Workers](/develop/python/workers)

- [Run a Worker](/develop/python/workers/run-worker-process)
- [Serverless Workers](/develop/python/workers/serverless-workers)

## [Temporal Client](/develop/python/client)

- [Temporal Client](/develop/python/client/temporal-client)

## [Temporal Nexus](/develop/python/nexus)

- [Quickstart](/develop/python/nexus/quickstart)
- [Feature guide](/develop/python/nexus/feature-guide)
- [Standalone Operations](/develop/python/nexus/standalone-operations)

## [Platform](/develop/python/platform)

- [Observability](/develop/python/platform/observability)
- [Enriching the UI](/develop/python/platform/enriching-ui)

## [Best practices](/develop/python/best-practices)

- [Testing](/develop/python/best-practices/testing-suite)
- [Python SDK sandbox](/develop/python/best-practices/python-sdk-sandbox)
- [Debugging](/develop/python/best-practices/debugging)
- [Data handling](/develop/python/data-handling)
- [Sync vs async](/develop/python/best-practices/python-sdk-sync-vs-async)

## [Integrations](/develop/python/integrations)

- [Baseten integration](/develop/python/integrations/baseten)
- [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#python)
- [Deep Agents integration](/develop/python/integrations/deepagents)
- [Fireworks AI integration](/develop/python/integrations/fireworks-ai)
- [Google ADK integration](/develop/python/integrations/google-adk)
- [Google GenAI integration](/develop/python/integrations/google-genai)
- [Laminar integration](https://laminar.sh/docs/integrations/temporal)
- [Langfuse integration](https://langfuse.com/integrations/frameworks/temporal)
- [LangGraph integration](/develop/python/integrations/langgraph)
- [LangSmith integration](/develop/python/integrations/langsmith)
- [MCP integration](/develop/python/integrations/mcp)
- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents)
- [OpenBox integration](https://docs.openbox.ai/getting-started/temporal)
- [OpenRouter integration](/develop/python/integrations/openrouter)
- [Parseable integration](https://github.com/parseablehq/temporal-plugin-python/blob/main/INTEGRATION.MD)
- [Pydantic AI integration](https://pydantic.dev/docs/ai/capabilities/durable_execution/temporal/)
- [Strands Agents integration](/develop/python/integrations/strands-agents)
- [Tenuo integration](https://tenuo.ai/temporal)
- [Together AI integration](/develop/python/integrations/together-ai)

## Temporal Python technical resources

- [Python SDK Quickstart - Setup Guide](/develop/python/set-up-your-local-python)
- [Python API Documentation](https://python.temporal.io)
- [Python SDK Code Samples](https://github.com/temporalio/samples-python)
- [Python SDK GitHub](https://github.com/temporalio/sdk-python)
- [Temporal 101 in Python Free Course](https://learn.temporal.io/courses/temporal_101/python/)
- [Build a Job Queue with Standalone Activities Tutorial](https://learn.temporal.io/tutorials/python/standalone-activities/)

## Get connected with the Temporal Python community

- [Temporal Python Community Slack](https://app.slack.com/client/TNWA8QCGZ)
- [Python SDK Forum](https://community.temporal.io/tag/python-sdk)

---

# Activities - Python SDK

Source: https://docs.temporal.io/develop/python/activities

> Activity Definitions, Activity Execution, Timeouts, Standalone Activities, and benign exceptions in the Temporal Python SDK.

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Activities

- [Activity basics](/develop/python/activities/basics)
- [Activity Execution](/develop/python/activities/execution)
- [Standalone Activities quickstart](/develop/python/activities/standalone-activities-quickstart)
- [Standalone Activities feature guide](/develop/python/activities/standalone-activities)
- [Timeouts](/develop/python/activities/timeouts)
- [Asynchronous Activity Completion](/develop/python/activities/asynchronous-activity)
- [Benign exceptions](/develop/python/activities/benign-exceptions)

---

# Asynchronous Activity Completion - Python SDK

Source: https://docs.temporal.io/develop/python/activities/asynchronous-activity

> Asynchronously complete an Activity using the Temporal Python SDK. Follow three steps for Activity completion and use the Temporal Client for Heartbeat and updates.

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The Temporal Client is used to Heartbeat and complete the Activity.

To mark an Activity as completing asynchronously, do the following inside the Activity.

<!--SNIPSTART python-async-activity-completion-raise-->
[python-sandbox/activities/async_activity_completion/activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/async_activity_completion/activities.py)
```py
# Capture token for later completion
captured_token = activity.info().task_token
pending_completions.put_nowait(captured_token)  # Give the external system the token
activity.raise_complete_async()
```
<!--SNIPEND-->

To update an Activity outside the Activity, use the [get_async_activity_handle()](https://python.temporal.io/temporalio.client.Client.html#get_async_activity_handle) method to get the handle of the Activity.

<!--SNIPSTART python-async-activity-completion-get-handle-->
[python-sandbox/activities/async_activity_completion/external_system.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/async_activity_completion/external_system.py)
```py
handle = my_client.get_async_activity_handle(task_token=captured_token)
```
<!--SNIPEND-->

Then, on that handle, you can call the results of the Activity, `heartbeat`, `complete`, `fail`, or `report_cancellation` method to update the Activity.

<!--SNIPSTART python-async-activity-completion-complete-->
[python-sandbox/activities/async_activity_completion/external_system.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/async_activity_completion/external_system.py)
```py
await handle.complete("Completion value.")
```
<!--SNIPEND-->

---

# Activity basics - Python SDK

Source: https://docs.temporal.io/develop/python/activities/basics

> Define an Activity with the activity.defn decorator, pass a single dataclass parameter, return serializable data, and set a custom Activity Type.

## Develop a basic Activity 

One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function
or method execution that's intended to execute a single, well-defined action (either short or long-running), such as
querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with the world outside
the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to
execute the Activity, we must define the [Activity Definition](/activity-definition).

Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/python/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/python/activities/standalone-activities-quickstart).

You can develop an Activity Definition by using the `@activity.defn` decorator. Register the function as an Activity
with a custom name through a decorator argument, for example `@activity.defn(name="your_activity")`.

> **📝 Note:**
>
> The Temporal Python SDK supports multiple ways of implementing an Activity:
>
> - Asynchronously using [`asyncio`](https://docs.python.org/3/library/asyncio.html)
> - Synchronously multithreaded using
>   [`concurrent.futures.ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor)
> - Synchronously multiprocess using
>   [`concurrent.futures.ProcessPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor)
>   and
>   [`multiprocessing.managers.SyncManager`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.SyncManager)
>
> Blocking the async event loop in Python would turn your asynchronous program into a synchronous program that executes
> serially, defeating the entire purpose of using `asyncio`. This can also lead to potential deadlock, and unpredictable
> behavior that causes tasks to be unable to execute. Debugging these issues can be difficult and time consuming, as
> locating the source of the blocking call might not always be immediately obvious.
>
> Due to this, consider not making blocking calls from within an asynchronous Activity, or use an async safe library to
> perform these actions. If you must use a blocking library, consider using a synchronous Activity instead.
>

<!--SNIPSTART python-activity-basics-activity-->
[python-sandbox/activities/activity_basics/your_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_basics/your_activities.py)
```py
from temporalio import activity
from activity_basics.your_dataobject import YourParams

@activity.defn(name="your_activity")
async def your_activity(input: YourParams) -> str:
    return f"{input.greeting}, {input.name}!"
```
<!--SNIPEND-->

### Define Activity parameters 

There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may
support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the
arguments, is limited to a maximum of 4 MB.

Also, keep in mind that all Payload data is recorded in the
[Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker
performance. This is because the entire Event History could be transferred to a Worker Process with a
[Workflow Task](/tasks#workflow-task).

Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data
that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the
application data passed to Activities. This is so that you can change what data is passed to the Activity without
breaking a function or method signature.

Activity parameters are the function parameters of the function decorated with `@activity.defn`. These can be any data
type Temporal can convert, including dataclasses when properly type-annotated. Technically this can be multiple
parameters, but Temporal strongly encourages a single dataclass parameter containing all input fields.

<!--SNIPSTART python-activity-basics-activity {"highlightedLines": "2,6"}-->
[python-sandbox/activities/activity_basics/your_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_basics/your_activities.py)
```py {2,6}
from temporalio import activity
from activity_basics.your_dataobject import YourParams

@activity.defn(name="your_activity")
async def your_activity(input: YourParams) -> str:
    return f"{input.greeting}, {input.name}!"
```
<!--SNIPEND-->

### Define Activity return values 

All data returned from an Activity must be serializable.

Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there
is a hard limit of 4MB for any gRPC message size in the Event History transaction
([see Cloud limits here](/evaluate/cloud/limits#per-message-grpc-limit)). Keep in mind that all return
values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history).

An Activity Execution can return inputs and other Activity values.

The following example defines an Activity that takes an object as input and returns a string.

<!--SNIPSTART python-activity-basics-activity {"highlightedLines": "6,7"}-->
[python-sandbox/activities/activity_basics/your_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_basics/your_activities.py)
```py {6,7}
from temporalio import activity
from activity_basics.your_dataobject import YourParams

@activity.defn(name="your_activity")
async def your_activity(input: YourParams) -> str:
    return f"{input.greeting}, {input.name}!"
```
<!--SNIPEND-->

### Customize your Activity Type 

Activities have a Type that are referred to as the Activity name. The following examples set a custom name for
your Activity Type.

You can customize the Activity name with a custom name in the decorator argument. For example,
`@activity.defn(name="your-activity")`. If the name parameter is not specified, the Activity name defaults to the
function name.

<!--SNIPSTART python-activity-basics-activity {"highlightedLines": "5"}-->
[python-sandbox/activities/activity_basics/your_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_basics/your_activities.py)
```py {5}
from temporalio import activity
from activity_basics.your_dataobject import YourParams

@activity.defn(name="your_activity")
async def your_activity(input: YourParams) -> str:
    return f"{input.greeting}, {input.name}!"
```
<!--SNIPEND-->

---

# Benign exceptions - Python SDK

Source: https://docs.temporal.io/develop/python/activities/benign-exceptions

> Mark expected or non-severe Activity errors as benign in Python to reduce noise in logs, metrics, and OpenTelemetry traces.

When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues.
By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic.

To mark an error as benign, set the `category` parameter to `ApplicationErrorCategory.BENIGN` when raising an [`ApplicationError`](https://python.temporal.io/temporalio.exceptions.ApplicationError.html).

Benign errors:
- Have Activity failure logs downgraded to DEBUG level
- Do not emit Activity failure metrics
- Do not set the OpenTelemetry failure status to ERROR

<!--SNIPSTART python-benign-exceptions-activity-->
[python-sandbox/activities/benign_exceptions/activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/benign_exceptions/activities.py)
```py
from temporalio import activity
from temporalio.exceptions import ApplicationError, ApplicationErrorCategory

@activity.defn
async def my_activity() -> str:
    try:
        return await call_external_service()
    except Exception as err:
        raise ApplicationError(
            message=str(err),
            # Mark this error as benign since it's expected
            category=ApplicationErrorCategory.BENIGN,
        )
```
<!--SNIPEND-->

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried.

---

# Activity Execution - Python SDK

Source: https://docs.temporal.io/develop/python/activities/execution

> Call an Activity from a Python Workflow with workflow.execute_activity, set the required Timeout, and await the Activity result.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a
[Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the
[ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in the set of three
[Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled),
[ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event
History.

A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations. Activity
implementation code should be _idempotent_.

The values passed to Activities through invocation parameters or returned through a result value are recorded in the
Execution history. The entire Execution history is transferred from the Temporal Service to Workflow Workers when a
Workflow state needs to recover. A large Execution history can thus adversely impact the performance of your Workflow.

Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values.
Otherwise, no additional limitations exist on Activity implementations.

To spawn an Activity Execution, use the
[`execute_activity()`](https://python.temporal.io/temporalio.workflow.html#execute_activity) operation from within your
Workflow Definition.

`execute_activity()` is a shortcut for
[`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) that waits on its result.

To get just the handle to wait and cancel separately, use `start_activity()`. In most cases, use `execute_activity()`
unless advanced task capabilities are needed.

A single argument to the Activity is positional. Multiple arguments are not supported in the type-safe form of
`start_activity()` or `execute_activity()` and must be supplied by the `args` keyword argument.

<!--SNIPSTART python-workflow-basics-workflow-->
[python-sandbox/workflows/workflow_basics/your_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/your_workflow.py)
```py
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from workflow_basics.your_activities import your_activity
    from workflow_basics.your_dataobject import YourParams

@workflow.defn(name="YourWorkflow")
class YourWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            your_activity,
            YourParams("Hello", name),
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

### Set the required Activity Timeouts 

Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a
[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a
[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the
Activity Options.

Activity options are set as keyword arguments after the Activity arguments.

Available timeouts are:

- schedule_to_close_timeout
- schedule_to_start_timeout
- start_to_close_timeout

<!--SNIPSTART python-activity-execution-timeouts {"highlightedLines": "8-15"}-->
[python-sandbox/activities/activity_execution/timeout_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_execution/timeout_workflow.py)
```py {8-15}
with workflow.unsafe.imports_passed_through():
    from activity_execution.activities import your_activity, YourParams

@workflow.defn
class YourWorkflow:
    @workflow.run
    async def run(self, greeting: str) -> list[str]:
        activity_timeout_result = await workflow.execute_activity(
            your_activity,
            YourParams(greeting, "Activity Timeout option"),
            # Activity Execution Timeout
            start_to_close_timeout=timedelta(seconds=10),
            # schedule_to_start_timeout=timedelta(seconds=10),
            # schedule_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

### Get the results of an Activity Execution 

The call to spawn an [Activity Execution](/activity-execution) generates the
[ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable.
Workflow Executions can either block progress until the result is available through the Awaitable or continue
progressing, making use of the result when it becomes available.

Use [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) to start an Activity and
return its handle, [`ActivityHandle`](https://python.temporal.io/temporalio.workflow.ActivityHandle.html). Use
[`execute_activity()`](https://python.temporal.io/temporalio.workflow.html#execute_activity) to return the results.

You must provide either `schedule_to_close_timeout` or `start_to_close_timeout`.

`execute_activity()` is a shortcut for `await start_activity()`. An asynchronous `execute_activity()` helper is provided
which takes the same arguments as `start_activity()` and `await`s on the result. `execute_activity()` should be used in
most cases unless advanced task capabilities are needed.

<!--SNIPSTART python-workflow-basics-workflow-->
[python-sandbox/workflows/workflow_basics/your_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/your_workflow.py)
```py
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from workflow_basics.your_activities import your_activity
    from workflow_basics.your_dataobject import YourParams

@workflow.defn(name="YourWorkflow")
class YourWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            your_activity,
            YourParams("Hello", name),
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

---

# Standalone Activities feature guide - Python SDK

Source: https://docs.temporal.io/develop/python/activities/standalone-activities

> Execute Activities independently without a Workflow using the Temporal Python SDK.

[Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated
by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/python/activities/basics). The only difference is that you execute a
Standalone Activity directly from your Temporal Client.

> **💡 Tip:**
>
> New to Standalone Activities? Start with the [Standalone Activities quickstart](/develop/python/activities/standalone-activities-quickstart).
>

This page covers the following:

- [Prerequisites](#prerequisites)
- [Start a Standalone Activity without waiting for the result](#start-activity)
- [Get a handle to an existing Standalone Activity](#get-activity-handle)
- [Wait for the result of a Standalone Activity](#get-activity-result)
- [List Standalone Activities](#list-activities)
- [Count Standalone Activities](#count-activities)
- [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from the [hello_standalone_activity](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/activities/hello_standalone_activity) sample.
>

## Prerequisites 

Standalone Activities require:

- **Python** 3.10+
- **Temporal Python SDK** v1.33.0 or higher
- **[Temporal CLI](/cli/setup-cli)** v1.9.1 or higher

The [Standalone Activities quickstart](/develop/python/activities/standalone-activities-quickstart)
walks through installing these.

## Start a Standalone Activity without waiting for the result 

Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue
your Activity job, without waiting for it to be executed by your Worker.

Use
[`client.start_activity()`](https://python.temporal.io/temporalio.client.Client.html#start_activity)
to start your Standalone Activity and get a handle:

<!--SNIPSTART python-standalone-activity-start-->
[python-sandbox/activities/hello_standalone_activity/start_activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/start_activity.py)
```py
activity_handle = await client.start_activity(
    compose_greeting,
    args=[ComposeGreetingInput("Hello", "World")],
    id="my-standalone-activity-id",
    task_queue="my-standalone-activity-task-queue",
    start_to_close_timeout=timedelta(seconds=10),
)
```
<!--SNIPEND-->

With the Temporal Server and Worker running, open a new terminal in the `python-sandbox/activities` directory and run:

```bash
uv run --no-project --with "temporalio>=1.33.0" python -m hello_standalone_activity.start_activity
```

Or use the Temporal CLI:

```bash
temporal activity start \
  --type compose_greeting \
  --activity-id my-standalone-activity-id \
  --task-queue my-standalone-activity-task-queue \
  --start-to-close-timeout 10s \
  --input '{"greeting": "Hello", "name": "World"}'
```

## Get a handle to an existing Standalone Activity 

You can also use `client.get_activity_handle()` to create a handle to a previously started Standalone Activity:

<!--SNIPSTART python-standalone-activity-get-handle-->
[python-sandbox/activities/hello_standalone_activity/get_activity_handle.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/get_activity_handle.py)
```py
activity_handle = client.get_activity_handle(
    activity_id="my-standalone-activity-id",
    run_id=run_id,
)
```
<!--SNIPEND-->

You can now use the handle to wait for the result, describe, cancel, or terminate the Activity.

## Wait for the result of a Standalone Activity 

Under the hood, calling `client.execute_activity()` is the same as calling
[`client.start_activity()`](https://python.temporal.io/temporalio.client.Client.html#start_activity)
to durably enqueue the Standalone Activity, and then calling  `await activity_handle.result()` to
wait for the activity to be executed and fetch the result:

<!--SNIPSTART python-standalone-activity-result-->
[python-sandbox/activities/hello_standalone_activity/start_activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/start_activity.py)
```py
activity_result = await activity_handle.result()
```
<!--SNIPEND-->

Or use the Temporal CLI to wait for a result by Activity ID:

```bash
temporal activity result --activity-id my-standalone-activity-id
```

## List Standalone Activities 

Use
[`client.list_activities()`](https://python.temporal.io/temporalio.client.Client.html#list_activities)
to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is
an async iterator that yields ActivityExecution entries.

These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included.

<!--SNIPSTART python-standalone-activity-list-->
[python-sandbox/activities/hello_standalone_activity/list_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/list_activities.py)
```py
import asyncio

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

async def my_application():
    connect_config = ClientConfig.load_client_connect_config()
    connect_config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(**connect_config)

    activities = client.list_activities(
        query="TaskQueue = 'my-standalone-activity-task-queue'",
    )

    async for info in activities:
        print(
            f"ActivityID: {info.activity_id}, Type: {info.activity_type}, Status: {info.status}"
        )

if __name__ == "__main__":
    asyncio.run(my_application())
```
<!--SNIPEND-->

Run it:

```bash
uv run --no-project --with "temporalio>=1.33.0" python -m hello_standalone_activity.list_activities
```

Or use the Temporal CLI:

```bash
temporal activity list
```

The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow
Visibility](/visibility). For example, "ActivityType = 'MyActivity' AND ExecutionStatus = 'Running'".

## Count Standalone Activities 

Use [`client.count_activities()`](https://python.temporal.io/temporalio.client.Client.html#count_activities) to count
Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the total
count of executions (running, completed, failed, etc.) - not the number of queued tasks. It works the
same way as counting Workflow Executions.

<!--SNIPSTART python-standalone-activity-count-->
[python-sandbox/activities/hello_standalone_activity/count_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/count_activities.py)
```py
import asyncio

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

async def my_application():
    connect_config = ClientConfig.load_client_connect_config()
    connect_config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(**connect_config)

    resp = await client.count_activities(
        query="TaskQueue = 'my-standalone-activity-task-queue'",
    )

    print("Total activities:", resp.count)

    for group in resp.groups:
        print(f"Group {group.group_values}: {group.count}")

if __name__ == "__main__":
    asyncio.run(my_application())
```
<!--SNIPEND-->

Run it:

```bash
uv run --no-project --with "temporalio>=1.33.0" python -m hello_standalone_activity.count_activities
```

Or use the Temporal CLI:

```bash
temporal activity count
```

## Run Standalone Activities with Temporal Cloud 

The code samples on this page use `ClientConfig.load_client_connect_config()`, so the same code
works against Temporal Cloud - just configure the connection via environment variables or a TOML
profile. No code changes are needed.

For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate
generation, and authentication setup in the Cloud UI, see
[Connect to Temporal Cloud](/develop/python/client/temporal-client#connect-to-temporal-cloud).

### Connect with mTLS

Set these environment variables with values from your Temporal Cloud Namespace settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem'
export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key'
```

### Connect with an API key

Set these environment variables with values from your Temporal Cloud API key settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_API_KEY=<your-api-key>
```

Then run the Worker and starter code as shown in the [Standalone Activities quickstart](/develop/python/activities/standalone-activities-quickstart).

---

# Standalone Activities Python quickstart - Python SDK

Source: https://docs.temporal.io/develop/python/activities/standalone-activities-quickstart

> Execute a Standalone Activity with the Temporal Python SDK without writing a Workflow.

# Quickstart

Standalone Activities are Activities that run independently, without being orchestrated by a
Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/python/activities/basics). The only difference is that you execute a
Standalone Activity directly from your Temporal Client.

> **📝 Note:**
>
> This documentation uses source code from the [hello_standalone_activity](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/activities/hello_standalone_activity) sample.
>

## Get started with Standalone Activities

Prerequisites:

- **Python 3.10+**

- **[uv](https://docs.astral.sh/uv/)** - Python package manager. Install with Homebrew, or see the [uv installation guide](https://docs.astral.sh/uv/getting-started/installation/) for other platforms.

- **Temporal Python SDK** (v1.33.0 or higher)

- **Temporal CLI** v1.9.1 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`.

Start the Temporal development server with `temporal server start-dev`.

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Server will now be available for client connections on `localhost:7233`, and the
Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233).

```bash
brew install uv
```

```bash
brew install temporal
```

```bash
temporal --version
```

```bash
temporal server start-dev
```

## Clone the sample

Clone the [documentation-sdk-code-examples](https://github.com/temporalio/documentation-sdk-code-examples) repository and switch to the `python-sandbox/activities` directory to follow along.
Run every command on this page from that directory.

```bash
git clone https://github.com/temporalio/documentation-sdk-code-examples.git
cd documentation-sdk-code-examples/python-sandbox/activities
```

The sample is structured as follows:

```
hello_standalone_activity/
├── __init__.py
├── my_activity.py
├── worker.py
└── execute_activity.py
```

<!--SNIPSTART python-standalone-activity-quickstart-activity-->
[python-sandbox/activities/hello_standalone_activity/my_activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/my_activity.py)
```py
# my_activity.py
from dataclasses import dataclass

from temporalio import activity

@dataclass
class ComposeGreetingInput:
    greeting: str
    name: str

@activity.defn
def compose_greeting(input: ComposeGreetingInput) -> str:
    activity.logger.info("Running activity with parameter %s" % input)
    return f"{input.greeting}, {input.name}!"
```
<!--SNIPEND-->

## Write an Activity function

An Activity in the Temporal Python SDK is just a normal function with the `@activity.defn`
decorator. It can optionally be an `async def`. The way you write a Standalone Activity is identical
to how you write an Activity to be orchestrated by a Workflow. In fact, an Activity can be executed
both as a Standalone Activity and as a Workflow Activity.

Create [hello_standalone_activity/my_activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/my_activity.py).

<!--SNIPSTART python-standalone-activity-quickstart-worker-->
[python-sandbox/activities/hello_standalone_activity/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/worker.py)
```py
import asyncio
from concurrent.futures import ThreadPoolExecutor

from temporalio.client import Client
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker

from hello_standalone_activity.my_activity import compose_greeting

async def main():
    connect_config = ClientConfig.load_client_connect_config()
    connect_config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(**connect_config)
    worker = Worker(
        client,
        task_queue="my-standalone-activity-task-queue",
        activities=[compose_greeting],
        activity_executor=ThreadPoolExecutor(5),
    )
    print("worker running...", end="", flush=True)
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

```bash
uv run --no-project --with "temporalio>=1.33.0" python -m hello_standalone_activity.worker
```

## Run a Worker with the Activity registered

Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities —
you create a Worker, register the Activity, and run the Worker. The Worker doesn't need to know
whether the Activity will be invoked from a Workflow or as a Standalone Activity. See
[Run a Worker](/develop/python/workers/run-worker-process#run-a-dev-worker) for more details on Worker setup and
configuration options.

Create [hello_standalone_activity/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/worker.py).

Open a new terminal, switch to the `python-sandbox/activities` directory, and run the Worker.
Leave this terminal running - the Worker needs to stay up to process activities.

<!--SNIPSTART python-standalone-activity-quickstart-execute-->
[python-sandbox/activities/hello_standalone_activity/execute_activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/execute_activity.py)
```py
import asyncio
from datetime import timedelta

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

from hello_standalone_activity.my_activity import ComposeGreetingInput, compose_greeting

async def my_application():
    connect_config = ClientConfig.load_client_connect_config()
    connect_config.setdefault("target_host", "localhost:7233")
    client = await Client.connect(**connect_config)

    activity_result = await client.execute_activity(
        compose_greeting,
        args=[ComposeGreetingInput("Hello", "World")],
        id="my-standalone-activity-id",
        task_queue="my-standalone-activity-task-queue",
        start_to_close_timeout=timedelta(seconds=10),
    )
    print(f"Activity result: {activity_result}")

if __name__ == "__main__":
    asyncio.run(my_application())
```
<!--SNIPEND-->

```bash
uv run --no-project --with "temporalio>=1.33.0" python -m hello_standalone_activity.execute_activity
```

```bash
temporal activity execute \\
  --type compose_greeting \\
  --activity-id my-standalone-activity-id \\
  --task-queue my-standalone-activity-task-queue \\
  --start-to-close-timeout 10s \\
  --input '{"greeting": "Hello", "name": "World"}'
```

## Execute a Standalone Activity

Use
[`client.execute_activity()`](https://python.temporal.io/temporalio.client.Client.html#execute_activity)
to execute a Standalone Activity. Call this from your application code, not from inside a Workflow
Definition. This durably enqueues your Standalone Activity in the Temporal Server, waits for it to
be executed on your Worker, and then fetches the result.

Create [hello_standalone_activity/execute_activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/hello_standalone_activity/execute_activity.py).

To run it:

1. Make sure the Temporal Server is running (from the Get Started step above).
2. Make sure the Worker is running (from the Run a Worker step above).
3. Open a new terminal, switch to the `python-sandbox/activities` directory, and run:
    ```
    uv run --no-project --with "temporalio>=1.33.0" python -m hello_standalone_activity.execute_activity
    ```

Or use the Temporal CLI.

You should see:

```
Activity result: Hello, World!
```

## Run with Temporal Cloud

All code samples on this page use
[`ClientConfig.load_client_connect_config()`](https://python.temporal.io/temporalio.envconfig.ClientConfig.html)
to configure the Temporal Client connection. It responds to [environment
variables](/references/client-environment-configuration) and [TOML configuration
files](/references/client-environment-configuration), so the same code works against a local dev
server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal
Cloud](/develop/python/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the feature guide
for mTLS and API key setup.

## Next steps

- **[Standalone Activities feature guide](/develop/python/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud.
- **[Build a Job Queue with Standalone Activities](https://learn.temporal.io/tutorials/python/standalone-activities/)**: Tutorial that builds a durable job queue, with idempotency keys, server-side deduplication, throughput limits, [Priority and Fairness](/develop/task-queue-priority-fairness), and heartbeat checkpoints.
- **[Activity basics](/develop/python/activities/basics)**: Write and register Activities with the Python SDK.

---

# Activity Timeouts - Python SDK

Source: https://docs.temporal.io/develop/python/activities/timeouts

> Set the Schedule-To-Close, Start-To-Close, and Heartbeat Timeouts on a Python Activity, tune its Retry Policy, and override the next retry delay.

## Set Activity timeouts 

Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution.

The following timeouts are available in the Activity Options.

- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution).
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution).
- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

Activity options are set as keyword arguments after the Activity arguments.

Available timeouts are:

- `schedule_to_close_timeout`
- `schedule_to_start_timeout`
- `start_to_close_timeout`

<!--SNIPSTART python-activity-timeouts-timeout-workflow {"highlightedLines": "12-19"}-->
[python-sandbox/activities/activity_timeouts/timeout_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_timeouts/timeout_workflow.py)
```py {12-19}
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activity_timeouts.activities import your_activity, YourParams

@workflow.defn
class YourWorkflow:
    @workflow.run
    async def run(self, greeting: str) -> str:
        activity_timeout_result = await workflow.execute_activity(
            your_activity,
            YourParams(greeting, "Activity Timeout option"),
            # Activity Execution Timeout
            start_to_close_timeout=timedelta(seconds=10),
            # schedule_to_start_timeout=timedelta(seconds=10),
            # schedule_to_close_timeout=timedelta(seconds=10),
        )
        return activity_timeout_result
```
<!--SNIPEND-->

### Set an Activity Retry Policy 

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided.

To create an Activity Retry Policy in Python, set the [RetryPolicy](https://python.temporal.io/temporalio.common.RetryPolicy.html) class within the [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) or [`execute_activity()`](https://python.temporal.io/temporalio.workflow.html#execute_activity) function.

<!--SNIPSTART python-activity-timeouts-retry-workflow {"highlightedLines": "13-25"}-->
[python-sandbox/activities/activity_timeouts/retry_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_timeouts/retry_workflow.py)
```py {13-25}
from datetime import timedelta

from temporalio import workflow
from temporalio.common import RetryPolicy

with workflow.unsafe.imports_passed_through():
    from activity_timeouts.activities import your_activity, YourParams

@workflow.defn
class YourWorkflow:
    @workflow.run
    async def run(self, greeting: str) -> str:
        activity_result = await workflow.execute_activity(
            your_activity,
            YourParams(greeting, "Retry Policy options"),
            start_to_close_timeout=timedelta(seconds=10),
            # Retry Policy
            retry_policy=RetryPolicy(
                backoff_coefficient=2.0,
                maximum_attempts=5,
                initial_interval=timedelta(seconds=1),
                maximum_interval=timedelta(seconds=2),
                # non_retryable_error_types=["ValueError"],
            ),
        )
        return activity_result
```
<!--SNIPEND-->

### Override the retry interval with `next_retry_delay` 

To override the next retry interval set by the current policy, pass `next_retry_delay` when raising an [ApplicationError](/references/failures#application-failure) in an Activity.
This value replaces and overrides whatever the retry interval would normally be on the retry policy.

For example, you can set the delay interval based on an Activity's attempt count.
In the following example, the retry delay starts at 3 seconds after the first attempt.
It increases to 6 seconds for the second attempt, 9 seconds for the third attempt, and so forth.
This creates a steadily increasing backoff, versus the exponential approach used by [backoff coefficients](/encyclopedia/retry-policies#backoff-coefficient):

<!--SNIPSTART python-activity-timeouts-next-retry-delay-->
[python-sandbox/activities/activity_timeouts/next_retry_delay.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_timeouts/next_retry_delay.py)
```py
from temporalio.exceptions import ApplicationError
from datetime import timedelta

@activity.defn
async def my_activity(input: MyActivityInput) -> str:
    try:
        # Your activity logic goes here
        return await call_remote_service(input)
    except Exception as e:
        attempt = activity.info().attempt
        raise ApplicationError(
            f"Error encountered on attempt {attempt}",
            next_retry_delay=timedelta(seconds=3 * attempt),
        ) from e
```
<!--SNIPEND-->

## Heartbeat an Activity 

An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service).
Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed.
If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy.

Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation.
Heartbeat throttling may lead to Cancellation getting delivered later than expected.

Heartbeats can contain a `details` field describing the Activity's current progress.
If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service.

To Heartbeat an Activity Execution in Python, use the [`heartbeat()`](https://python.temporal.io/temporalio.activity.html#heartbeat) API.

<!--SNIPSTART python-activity-timeouts-heartbeat-->
[python-sandbox/activities/activity_timeouts/heartbeat_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_timeouts/heartbeat_activities.py)
```py
@activity.defn
async def your_activity_definition() -> str:
    activity.heartbeat("heartbeat details!")
```
<!--SNIPEND-->

In addition to obtaining cancellation information, Heartbeats also support detail data that persists on the server for retrieval during Activity retry.
If an Activity calls `heartbeat(123, 456)` and then fails and is retried, `heartbeat_details` returns an iterable containing `123` and `456` on the next Run.

### Set a Heartbeat Timeout 

A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat).

[`heartbeat_timeout`](https://python.temporal.io/temporalio.worker.StartActivityInput.html#heartbeat_timeout) is a class variable for the [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) function used to set the maximum time between Activity Heartbeats.

<!--SNIPSTART python-activity-timeouts-start-activity-heartbeat-timeout-->
[python-sandbox/activities/activity_timeouts/heartbeat_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_timeouts/heartbeat_workflow.py)
```py
handle = workflow.start_activity(
    activity="your-activity",
    schedule_to_close_timeout=timedelta(seconds=5),
    heartbeat_timeout=timedelta(seconds=1),
)
```
<!--SNIPEND-->

`execute_activity()` is a shortcut for [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) that waits on its result.

To get just the handle to wait and cancel separately, use `start_activity()`. `execute_activity()` should be used in most cases unless advanced task capabilities are needed.

<!--SNIPSTART python-activity-timeouts-execute-activity-heartbeat-timeout-->
[python-sandbox/activities/activity_timeouts/heartbeat_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities/activity_timeouts/heartbeat_workflow.py)
```py
executed_result = await workflow.execute_activity(
    activity="your-activity",
    arg=name,
    schedule_to_close_timeout=timedelta(seconds=5),
    heartbeat_timeout=timedelta(seconds=1),
)
```
<!--SNIPEND-->

---

# Best practices - Python SDK

Source: https://docs.temporal.io/develop/python/best-practices

> This section explains how to implement best practices with the Python SDK

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Best practices

- [Error handling](/develop/python/best-practices/error-handling)
- [Testing](/develop/python/best-practices/testing-suite)
- [Python SDK sandbox](/develop/python/best-practices/python-sdk-sandbox)
- [Debugging](/develop/python/best-practices/debugging)
- [Data handling](/develop/python/data-handling)
- [Sync vs async](/develop/python/best-practices/python-sdk-sync-vs-async)

---

# Debugging - Python SDK

Source: https://docs.temporal.io/develop/python/best-practices/debugging

> Debug Workflows in development and production environments using the Temporal Python SDK, Web UI, Temporal CLI, replay, tracing, logging, and performance metrics.

This page shows how to do the following:

- [Debug in a development environment](#debug-in-a-development-environment)
- [Debug in a production environment](#debug-in-a-production-environment)

### Debug in a development environment 

When developing Workflows, you can use the normal development tools of logging and a debugger to see what’s happening in your Workflow.
In addition to the normal development tools of logging and a debugger, you can also see what’s happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli).

### Debug in a production environment 

You can debug production Workflows using:

- [Web UI](/web-ui)
- [Temporal CLI](/cli)
- [Replay](/develop/python/best-practices/testing-suite#replay)
- [Tracing](/develop/python/platform/observability#tracing)
- [Logging](/develop/python/platform/observability#logging)

You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance).
For more information, see [Observability ▶️ Metrics](/develop/python/platform/observability#metrics) for setting up SDK metrics.

Debug Server performance with [Cloud metrics](/cloud/metrics/openmetrics) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics).

---

# Error handling - Python SDK

Source: https://docs.temporal.io/develop/python/best-practices/error-handling

> Learn how to handle errors in Temporal Python applications with retry policies, idempotent Activities, and recovery patterns.

Temporal automatically handles many types of failures through retries and Durable Execution.
This page shows you how to build on these capabilities to create robust error handling for your applications.

**Key concepts:**

Not all failures should be handled the same way.
**Transient failures** (like brief network hiccups) resolve on their own and should be retried immediately.
**Intermittent failures** (like rate limiting) need increasing delays between retries.
**Permanent failures** (like invalid input) won't resolve through retries and need different data or code changes.

Temporal distinguishes between **Workflow Task failures** (bugs that can be fixed with redeployment) and **Workflow Execution failures** (business logic failures that should stop the Workflow).
Task failures retry automatically so you can fix and redeploy without losing state.
Execution failures require you to explicitly raise an `ApplicationError`.

This page shows how to:

- [Make Activities idempotent](#make-activities-idempotent)
- [Raise exceptions from Activities](#raise-exceptions-from-activities)
- [Raise exceptions from Workflows](#raise-exceptions-from-workflows)
- [Handle exceptions in Workflows](#handle-exceptions-in-workflows)
- [Configure custom Retry Policies](#configure-custom-retry-policies)
- [Mark specific errors as non-retryable](#mark-errors-as-non-retryable)
- [Specify non-retryable error types in Retry Policies](#specify-non-retryable-error-types)
- [Implement rollback logic with the Saga pattern](#implement-saga-pattern)
- [Understand Temporal's failure types](#understand-failure-types)

## Make Activities idempotent 

Because Activities may be retried due to failures, it's strongly recommended to make them idempotent.
An idempotent operation produces the same result whether executed once or multiple times.

Activities follow an at-least-once execution model.
If a Worker executes an Activity successfully but crashes before notifying the Temporal Service, the Activity will be retried.
Without idempotence, this could cause duplicate charges in payment processing or create duplicate resources in infrastructure provisioning.

### Use idempotency keys

Most external services support idempotency keys—unique identifiers that prevent duplicate operations.
When the service receives a request with a key it has already processed, it returns the original result instead of performing the operation again.

Create an idempotency key by combining the Workflow Run ID and Activity ID:

<!--SNIPSTART python-error-handling-idempotency-key-->
[python-sandbox/best_practices/error_handling/idempotency.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/idempotency.py)
```py
from temporalio import activity

@activity.defn
async def process_payment(amount: float, account: str):
    info = activity.info()
    idempotency_key = f"{info.workflow_run_id}-{info.activity_id}"

    # Pass idempotency_key to your payment service
    result = await payment_service.charge(
        amount=amount,
        account=account,
        idempotency_key=idempotency_key
    )
    return result
```
<!--SNIPEND-->

This value remains constant across Activity retries but is unique among all Workflow Executions.

### Design Activities to be atomic

Activities are atomic—they either complete successfully or not.
If an Activity performs multiple steps and the last step fails, the entire Activity is retried.

Consider this Activity:
1. Look up data in database
2. Call microservice with the data
3. Write result to filesystem

If step 3 fails, all three steps execute again on retry.
You might split this into three separate Activities so only the failed step retries, but balance this against having a larger Event History with more Activity Executions.

## Raise exceptions from Activities 

Use `ApplicationError` to communicate application-specific failures from Activities. Raising it explicitly gives you more control.

> **📝 Note:**
>
> *Any* other exceptions that are raised from your Python code in a Temporal Activity will be converted to an `ApplicationError` internally. This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries.
>

<!--SNIPSTART python-error-handling-raise-from-activity-->
[python-sandbox/best_practices/error_handling/activity_errors.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/activity_errors.py)
```py
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def validate_charge(credit_card_number: str, amount: float):
    if not is_valid_card(credit_card_number):
        raise ApplicationError(
            f"Invalid credit card number: {credit_card_number}",
            type="InvalidCreditCard",
        )

    if amount <= 0:
        raise ApplicationError(
            f"Amount must be positive, got {amount}",
            type="InvalidAmount",
        )

    return True
```
<!--SNIPEND-->

When raising an `ApplicationError`:
- Provide a descriptive `message`
- Optionally provide a `type` string to categorize the failure
- The error appears in the Event History as an `ActivityTaskFailed` event

When an Activity fails, Temporal wraps the exception in an `ActivityError` before surfacing it to the Workflow.
The `ActivityError` provides context including:
- Activity type that failed
- Number of retry attempts
- Original cause (the [`ApplicationError`](https://python.temporal.io/temporalio.exceptions.ApplicationError.html) you raised, or `TimeoutError`, [`CancelledError`](https://python.temporal.io/temporalio.exceptions.CancelledError.html), etc.)

## Raise exceptions from Workflows 

The behavior depends on what exception you raise:

### Fail a Workflow Execution

To deliberately fail a Workflow Execution, raise an `ApplicationError`:

<!--SNIPSTART python-error-handling-fail-workflow-->
[python-sandbox/best_practices/error_handling/pizza_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/pizza_workflow.py)
```py
from datetime import timedelta

from temporalio import workflow
from temporalio.exceptions import ApplicationError

@workflow.defn
class PizzaDeliveryWorkflow:
    @workflow.run
    async def run(self, order: PizzaOrder):
        distance = await workflow.execute_activity(
            calculate_distance,
            order.address,
            start_to_close_timeout=timedelta(seconds=10)
        )

        if order.is_delivery and distance.kilometers > 25:
            workflow.logger.error("Customer outside service area")
            raise ApplicationError(
                "Customer lives outside the service area",
                type="CustomerOutsideServiceArea"
            )

        # Continue with order...
```
<!--SNIPEND-->

One reason to use the Temporal `ApplicationError` class is because it allows you to set an additional `non_retryable` parameter.
This way, you can decide whether an error should not be retried automatically by Temporal.
This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse. You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](/develop/python/activities/timeouts#activity-retries).

This puts the Workflow Execution in "Failed" state with no automatic retries.
Use this for permanent failures where retrying won't help—like the customer being too far away.

### Trigger a Workflow Task retry

Raising any other Python exception (like `ValueError` or `TypeError`) causes a Workflow Task failure, which retries automatically:

<!--SNIPSTART python-error-handling-workflow-task-failure-->
[python-sandbox/best_practices/error_handling/pizza_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/pizza_workflow.py)
```py
# This causes a Workflow Task failure (retries automatically)
raise ValueError("Unexpected condition")
```
<!--SNIPEND-->

This is intentional.
Regular Python exceptions are treated as bugs that can be fixed with a code deployment, not business logic failures.
The Workflow Task retries indefinitely, letting you fix the bug and redeploy without losing Workflow state.

## Handle exceptions in Workflows 

Use Python's `try/except` blocks to handle Activity failures in your Workflow:

<!--SNIPSTART python-error-handling-handle-activity-errors-->
[python-sandbox/best_practices/error_handling/money_transfer.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/money_transfer.py)
```py
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError, ApplicationError
from datetime import timedelta

@workflow.defn
class MoneyTransferWorkflow:
    @workflow.run
    async def run(self, details: TransferDetails):
        # Withdraw money
        try:
            withdraw_result = await workflow.execute_activity(
                withdraw,
                details,
                start_to_close_timeout=timedelta(seconds=10)
            )
        except ActivityError as e:
            raise ApplicationError(
                f"Withdrawal failed: {e.cause}",
                type="WithdrawalError"
            )

        # Deposit money
        try:
            deposit_result = await workflow.execute_activity(
                deposit,
                details,
                start_to_close_timeout=timedelta(seconds=10),
                # Limit retries so a deposit that keeps failing reaches the refund below
                retry_policy=RetryPolicy(
                    initial_interval=timedelta(seconds=1),
                    maximum_attempts=3,
                ),
            )
        except ActivityError as e:
            # Deposit failed - attempt refund
            try:
                await workflow.execute_activity(
                    refund,
                    withdraw_result,
                    start_to_close_timeout=timedelta(seconds=10)
                )
                raise ApplicationError(
                    f"Deposit failed but money refunded to source account",
                    type="DepositError"
                )
            except ActivityError as refund_err:
                raise ApplicationError(
                    f"Deposit failed and refund also failed: {refund_err.cause}",
                    type="CriticalTransferError"
                )

        return f"Transfer complete: {withdraw_result}, {deposit_result}"
```
<!--SNIPEND-->

Common Temporal exceptions you can catch in Workflows:
- `ActivityError` - Activity failed after exhausting retries
- `ChildWorkflowError` - Child Workflow failed
- `CancelledError` - Workflow, Activity, or Timer was canceled
- `TimeoutError` - Operation exceeded timeout

If these exceptions propagate unhandled, the Workflow Execution fails (or enters "Canceled" state for `CancelledError`).

## Configure custom Retry Policies 

Activities have a default Retry Policy with unlimited attempts and exponential backoff.
Customize this to match your expected failure patterns.

<!--SNIPSTART python-error-handling-custom-retry-policy-->
[python-sandbox/best_practices/error_handling/retry_policies.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/retry_policies.py)
```py
from temporalio import workflow
from temporalio.common import RetryPolicy
from datetime import timedelta

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order: Order):
        # Custom retry for rate-limited service
        retry_policy = RetryPolicy(
            initial_interval=timedelta(seconds=10),
            backoff_coefficient=3.0,
            maximum_interval=timedelta(minutes=5),
            maximum_attempts=20,
        )

        result = await workflow.execute_activity(
            call_external_service,
            order,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=retry_policy,
        )
        return result
```
<!--SNIPEND-->

Retry Policy attributes:
- **`initial_interval`**: Delay before first retry (default: 1 second)
- **`backoff_coefficient`**: Multiplier for subsequent delays (default: 2.0)
- **`maximum_interval`**: Cap on retry delay (default: 100× initial interval)
- **`maximum_attempts`**: Maximum retry attempts (default: unlimited)
- **`non_retryable_error_types`**: Error types that shouldn't retry (default: empty)

### Match your Retry Policy to failure types

**For transient failures** (brief network issues): Use the defaults or a low `initial_interval` and `backoff_coefficient`.

**For intermittent failures** (rate limiting): Increase `initial_interval` and `backoff_coefficient` to space out retries and let the condition resolve.

**For cost-sensitive APIs**: Set `maximum_attempts` to limit retries (rare—usually prefer timeouts).

### Use different policies for different Activities

You can use different Retry Policies for different Activities, or even multiple policies for the same Activity:

<!--SNIPSTART python-error-handling-different-policies-->
[python-sandbox/best_practices/error_handling/retry_policies.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/retry_policies.py)
```py
fast_retry = RetryPolicy(
    initial_interval=timedelta(seconds=1),
    backoff_coefficient=1.5,
)

slow_retry = RetryPolicy(
    initial_interval=timedelta(seconds=30),
    backoff_coefficient=3.0,
)

# Same Activity, different policies
await workflow.execute_activity(
    process_order,
    order,
    start_to_close_timeout=timedelta(seconds=10),
    retry_policy=fast_retry,
)

# Later, with different circumstances...
await workflow.execute_activity(
    process_order,
    order,
    start_to_close_timeout=timedelta(seconds=10),
    retry_policy=slow_retry,
)
```
<!--SNIPEND-->

### Don't use Workflow Retry Policies

Unlike Activities, Workflows don't retry by default, and you usually shouldn't add a Retry Policy.
A Workflow Definition produces the same result every time, so it isn't designed for failure-prone operations.
A Workflow failure typically indicates a code bug or bad input data—retrying the entire Workflow repeats the same logic without fixing the underlying issue.

If you need retry logic for specific Workflow operations, implement it in your Workflow code rather than using a Workflow Retry Policy.

## Mark specific errors as non-retryable 

Some failures are permanent and won't resolve through retries.
Mark these as non-retryable to fail fast instead of waiting for timeouts.

Set the `non_retryable` flag when raising an `ApplicationError`:

<!--SNIPSTART python-error-handling-non-retryable-->
[python-sandbox/best_practices/error_handling/non_retryable.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/non_retryable.py)
```py
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def process_payment(card_number: str, amount: float):
    if not is_valid_card_format(card_number):
        # Invalid format will never become valid through retries
        raise ApplicationError(
            f"Invalid credit card format: {card_number}",
            type="InvalidCardFormat",
            non_retryable=True,
        )

    if amount <= 0:
        # Invalid amount won't be fixed by retrying
        raise ApplicationError(
            f"Amount must be positive: {amount}",
            type="InvalidAmount",
            non_retryable=True,
        )

    # Process payment...
```
<!--SNIPEND-->

An `ApplicationError` with `non_retryable=True` will never retry, regardless of the Retry Policy.

Use non-retryable errors for:
- Invalid input data that prevents the Activity from proceeding
- Business rule violations
- Authorization failures

**Use this sparingly.**
In most cases, it's better to let the Retry Policy handle when to stop retrying based on time or attempts.

## Specify non-retryable error types 

Sometimes you want the Workflow (caller) to decide which error types shouldn't retry, rather than the Activity (implementer).

List error types that shouldn't retry in your Retry Policy:

<!--SNIPSTART python-error-handling-non-retryable-error-types-->
[python-sandbox/best_practices/error_handling/checkout_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/checkout_workflow.py)
```py
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError
from datetime import timedelta

@workflow.defn
class CheckoutWorkflow:
    @workflow.run
    async def run(self, payment_details: PaymentDetails):
        retry_policy = RetryPolicy(
            non_retryable_error_types=[
                "InvalidCardFormat",
                "InsufficientFunds",
                "AccountClosed",
            ]
        )

        try:
            result = await workflow.execute_activity(
                process_payment,
                payment_details,
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=retry_policy,
            )
            return result
        except ActivityError as e:
            workflow.logger.error(f"Payment failed: {e.cause}")
            # Handle the non-retryable error...
```
<!--SNIPEND-->

When an Activity raises an `ApplicationError`, Temporal checks if its `type` is in `non_retryable_error_types`.
If it matches, the Activity fails immediately without retries.

### When to use each approach

**`non_retryable=True` in the Activity**: Use when the Activity implementer knows the error is permanently unrecoverable.
This enforces the constraint for all callers.

**`non_retryable_error_types` in the Retry Policy**: Use when the caller wants to decide which errors are unrecoverable based on their business logic.
This lets different Workflows make different decisions about the same Activity.

## Implement rollback logic with the Saga pattern 

The [Saga pattern](/design-patterns/saga-pattern) coordinates a sequence of operations where each operation has a compensating action to undo its effects.
If any operation fails, execute compensating actions in reverse order to roll back previous operations.

Use this for multi-step processes like:
- E-commerce checkout (payment, inventory, shipping)
- Distributed transactions across services
- Multi-stage data updates

<!--SNIPSTART python-error-handling-saga-->
[python-sandbox/best_practices/error_handling/saga_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/saga_workflow.py)
```py
from temporalio import workflow
from temporalio.exceptions import ActivityError, ApplicationError
from datetime import timedelta

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order: Order):
        compensations = []

        try:
            # Reserve inventory
            compensations.append({
                "activity": revert_inventory,
                "input": order
            })
            await workflow.execute_activity(
                reserve_inventory,
                order,
                start_to_close_timeout=timedelta(seconds=10),
            )

            # Charge payment
            compensations.append({
                "activity": refund_payment,
                "input": order
            })
            payment_id = await workflow.execute_activity(
                charge_payment,
                order,
                start_to_close_timeout=timedelta(seconds=10),
            )

            # Create shipment
            compensations.append({
                "activity": cancel_shipment,
                "input": payment_id
            })
            shipment_id = await workflow.execute_activity(
                create_shipment,
                order,
                start_to_close_timeout=timedelta(seconds=10),
            )

            return {"payment_id": payment_id, "shipment_id": shipment_id}

        except ActivityError as e:
            workflow.logger.error(f"Order failed: {e.cause}, rolling back...")

            # Execute compensations in reverse order
            for compensation in reversed(compensations):
                try:
                    await workflow.execute_activity(
                        compensation["activity"],
                        compensation["input"],
                        start_to_close_timeout=timedelta(seconds=10),
                    )
                except ActivityError as comp_err:
                    # Log compensation failure but continue with others
                    workflow.logger.error(f"Compensation failed: {comp_err.cause}")

            # Re-raise the original error
            raise ApplicationError(
                f"Order failed: {e.cause}",
                type="OrderFailed"
            )
```
<!--SNIPEND-->

Key points:
- Add compensating actions to a list **before** executing each Activity
- Use `reversed(compensations)` to undo operations in the correct order
- Handle compensation failures gracefully (they might fail too)
- Temporal manages all state and retry logic, making Saga implementation straightforward

## Understand Temporal's failure types 

Temporal uses specialized exception types to represent different failure scenarios.
All exceptions inherit from [`TemporalError`](https://python.temporal.io/temporalio.exceptions.TemporalError.html).

**Do not extend `TemporalError` or its children.**
Use the provided exception types to ensure:
- Consistent behavior across process and language boundaries
- Compatibility with the Temporal Service
- Proper serialization via Protocol Buffers

### Common failure types

**`ApplicationError`**: Raised by your code to indicate application-specific failures.
This is the only Temporal exception you should raise manually.
When you raise an `ApplicationError`, you can optionally provide a `type` string and mark it as `non_retryable`.

**`ActivityError`**: Wraps exceptions raised from Activities.
The `cause` field contains the original error (`ApplicationError`, `TimeoutError`, `CancelledError`, etc.).
Catch this in Workflows to handle Activity failures.

**`TimeoutError`**: Occurs when an Activity or Workflow exceeds its configured timeout.

**`CancelledError`**: Results from cancellation of a Workflow, Activity, or Timer.
You can catch and ignore this to continue execution despite cancellation.

**`TerminatedError`**: Occurs when a Workflow Execution is forcefully terminated.

**`ChildWorkflowError`**: Raised when a Child Workflow Execution fails.

**`WorkflowAlreadyStartedError`**: Raised when attempting to start a Workflow with an ID that's already running.

**`ServerError`**: Used for exceptions from the Temporal Service itself (like database failures).

### Workflow Task vs Workflow Execution failures

**Workflow Task failures** occur when Workflow code raises a non-Temporal exception (like `ValueError`, `TypeError`, or non-determinism errors).
These retry automatically, letting you fix bugs and redeploy without losing Workflow state.

**Workflow Execution failures** occur when Workflow code raises a Temporal exception like `ApplicationError`.
These put the Workflow in "Failed" state with no automatic retries.

Example of a permanent failure that should fail the Workflow:

<!--SNIPSTART python-error-handling-permanent-failure-->
[python-sandbox/best_practices/error_handling/pizza_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/error_handling/pizza_workflow.py)
```py
if distance.kilometers > MAX_DELIVERY_DISTANCE:
    # Retrying won't change the distance - this is permanent
    raise ApplicationError(
        "Customer lives outside service area",
        type="OutsideServiceArea"
    )
```
<!--SNIPEND-->

### Protecting sensitive information

The default Failure Converter copies exception messages and stack traces as plain text visible in the Web UI.
If your exceptions might contain sensitive information, configure a custom Failure Converter to encrypt this data.
See the [Securing Application Data course](https://learn.temporal.io/courses/appdatasec/) for details.

---

# Temporal Python SDK sandbox environment

Source: https://docs.temporal.io/develop/python/best-practices/python-sdk-sandbox

> The Temporal Python SDK offers a sandbox environment to run Workflow code, aiming to prevent non-determinism errors in applications by isolating global state and applying restrictions.

The Temporal Python SDK enables you to run Workflow code in a sandbox environment to help prevent non-determinism errors in your application.
The Temporal Workflow Sandbox for Python is not completely isolated, and some libraries can internally mutate state, which can result in breaking determinism.

## Benefits

Temporal's Python SDK uses a sandbox environment for Workflow runs to make developing Workflow code safer.

If a Workflow Execution performs a non-deterministic event, an exception is thrown, which results in failing the Task Worker.
The Workflow will not progress until the code is fixed.

The Temporal Python sandbox offers a mechanism to _pass through modules_ from outside the sandbox. By default, this includes all standard library modules and Temporal modules. For performance and behavior reasons, users should pass through all models, Activities, Nexus services, or other modules that are in separate files whose calls will be deterministic. For more information, see [Passthrough modules](#passthrough-modules).

## How it works

The Sandbox environment consists of two main components.

- [Global state isolation](#global-state-isolation)
- [Restrictions](#restrictions)

### Global state isolation

The first component of the Sandbox is a global state isolation.
Global state isolation uses `exec` to compile and evaluate statements.

Upon the start of a Workflow, the file in which the Workflow is defined is imported into a newly created sandbox.

If a module is imported by the file, a known set, which includes all of Python's standard library, is _passed through_ from outside the sandbox.

These modules are expected to be free of side effects and have their non-deterministic aspects restricted.

For a full list of modules imported, see [Customize the Sandbox](#customize-the-sandbox).

### Restrictions

Restrictions prevent known non-deterministic library calls.
This is achieved by using proxy objects on modules wrapped around the custom importer set in the sandbox.

Restrictions apply at both the Workflow import level and the Workflow run time.

A default set of restrictions that prevents most dangerous standard library calls.

## Skip Workflow sandboxing

The following techniques aren't recommended, but they allow you to avoid, skip, or break through the sandbox environment.

Skipping Workflow Sandboxing results in a lack of determinism checks. Using the Workflow Sandboxing environment helps prevent non-determinism errors but doesn't completely negate the risk.

### Skip Sandboxing for a block of code

To skip a sandbox environment for a specific block of code in a Workflow, use [`sandbox_unrestricted()`](https://python.temporal.io/temporalio.workflow.unsafe.html#sandbox_unrestricted). The Workflow will run without sandbox restrictions.

<!--SNIPSTART python-sdk-sandbox-unrestricted-block-->
[python-sandbox/best_practices/sdk_sandbox/unrestricted_block.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/unrestricted_block.py)
```py
with temporalio.workflow.unsafe.sandbox_unrestricted():
    # Your code
    today = datetime.date.today()
```
<!--SNIPEND-->

### Skip Sandboxing for an entire Workflow

To skip a sandbox environment for a Workflow, set the `sandboxed` argument in the [`@workflow.defn`](https://python.temporal.io/temporalio.workflow.html#defn) decorator to false.
The entire Workflow will run without sandbox restrictions.

<!--SNIPSTART python-sdk-sandbox-unsandboxed-workflow-->
[python-sandbox/best_practices/sdk_sandbox/unsandboxed_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/unsandboxed_workflow.py)
```py
@workflow.defn(sandboxed=False)
```
<!--SNIPEND-->

### Skip Sandboxing for a Worker

To skip a sandbox environment for a Worker, set the `workflow_runner` keyword argument of the `Worker` init to [`UnsandboxedWorkflowRunner()`](https://python.temporal.io/temporalio.worker.UnsandboxedWorkflowRunner.html).

## Customize the sandbox

When creating the Worker, the `workflow_runner` defaults to [`SandboxedWorkflowRunner()`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxedWorkflowRunner.html).
The `SandboxedWorkflowRunner` init accepts a `restrictions` keyword argument that defines a set of restrictions to apply to this sandbox.

The [`SandboxRestrictions`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html) dataclass is immutable and contains four fields that can be customized, but only three have notable values.

- [`passthrough_modules`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#passthrough_modules)
- [`invalid_modules_members`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#invalid_module_members)
- [`import_notification_policy`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#import_notificaton_policy)

### Passthrough modules

By default, the sandbox completely reloads non-standard-library and non-Temporal modules for every Workflow run. Passing through a module means that the module will not be reloaded every time the Workflow runs. Instead, the module will be imported from outside the sandbox and used directly in the Workflow. This can improve performance because importing a module can be a time-consuming process, and passing through a module can avoid this overhead.

> **📝 Note:**
> It is important to note that you should only import _known-side-effect-free_ third-party modules: meaning they don't have any unintended consequences when imported and used multiple times. This is because passing through a module means that it will be used multiple times in a Workflow without being reloaded, so any side effects it has won't be repeated. For this reason, it's recommended to only pass through modules that are known to be deterministic, meaning they will always produce the same output given the same input.

One way to pass through a module is at import time in the Workflow file using the [`imports_passed_through`](https://python.temporal.io/temporalio.workflow.unsafe.html#imports_passed_through) context manager.

<!--SNIPSTART python-sdk-sandbox-passthrough-workflow-->
[python-sandbox/best_practices/sdk_sandbox/passthrough_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/passthrough_workflow.py)
```py
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    import pydantic

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        greeting = pydantic.TypeAdapter(str).validate_python(f"Hello, {name}!")
        return greeting
```
<!--SNIPEND-->

Alternatively, this can be done at Worker creation time by customizing the runner's restrictions.

<!--SNIPSTART python-sdk-sandbox-passthrough-worker-->
[python-sandbox/best_practices/sdk_sandbox/workers.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/workers.py)
```py
from temporalio.worker import Worker
from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions

my_worker = Worker(
  client,
  task_queue=task_queue,
  workflows=[MyWorkflow],
  workflow_runner=SandboxedWorkflowRunner(
    restrictions=SandboxRestrictions.default.with_passthrough_modules("pydantic")
  )
)
```
<!--SNIPEND-->

In both of these cases, now the `pydantic` module will be passed through from outside the sandbox instead of being reloaded for every Workflow run.

### Invalid module members

`invalid_module_members` includes modules that cannot be accessed.

Checks are compared against the fully qualified path to the item.

For example, to remove a restriction on `datetime.date.today()`, see the following example.

<!--SNIPSTART python-sdk-sandbox-unrestrict-member-->
[python-sandbox/best_practices/sdk_sandbox/workers.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/workers.py)
```py
import dataclasses
from temporalio.worker import Worker
from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions

my_restrictions = dataclasses.replace(
    SandboxRestrictions.default,
    invalid_module_members=SandboxRestrictions.invalid_module_members_default.with_child_unrestricted(
      "datetime", "date", "today",
    ),
)
my_worker = Worker(
    client,
    task_queue=task_queue,
    workflows=[TodayWorkflow],
    workflow_runner=SandboxedWorkflowRunner(restrictions=my_restrictions),
)
```
<!--SNIPEND-->

Restrictions can also be added by piping (`|`) together [`SandboxMatcher`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxMatcher.html) instances.

The following example restricts the `datetime.date` class from being used.

<!--SNIPSTART python-sdk-sandbox-restrict-member-->
[python-sandbox/best_practices/sdk_sandbox/workers.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/workers.py)
```py
import dataclasses
from temporalio.worker import Worker
from temporalio.worker.workflow_sandbox import (
  SandboxedWorkflowRunner,
  SandboxMatcher,
  SandboxRestrictions,
)

my_restrictions = dataclasses.replace(
    SandboxRestrictions.default,
    invalid_module_members=SandboxRestrictions.invalid_module_members_default | SandboxMatcher(
      children={"datetime": SandboxMatcher(use={"date"})},
    ),
)
my_worker = Worker(
    client,
    task_queue=task_queue,
    workflows=[TodayWorkflow],
    workflow_runner=SandboxedWorkflowRunner(restrictions=my_restrictions),
)
```
<!--SNIPEND-->

### Import notification policy

The sandbox's import notification policy specifies how the sandbox behaves when it imports modules in a way that may be unintentional. It covers two common scenarios: dynamic imports and enforcing module passthrough. Each can be controlled independently.

A dynamic import occurs when a module is imported after the Workflow is loaded into the sandbox. These imports are often invisible and, if they don't do anything restricted by the sandbox, cause memory overhead. By default the [`WARN_ON_DYNAMIC_IMPORT`](https://python.temporal.io/temporalio.workflow.SandboxImportNotificationPolicy.html#WARN_ON_DYNAMIC_IMPORT) policy setting is enabled and a warning will be emitted when a module that is not in the [passthrough modules](#passthrough-modules) list is dynamically imported.

The other notable policy settings apply when a module is imported into the sandbox that was not passed through. These settings are disabled by default and must be explicitly turned on.
The [`WARN_ON_UNINTENTIONAL_PASSTHROUGH`](https://python.temporal.io/temporalio.workflow.SandboxImportNotificationPolicy.html#WARN_ON_UNINTENTIONAL_PASSTHROUGH) setting emits a warning when a module not included in the [passthrough modules](#passthrough-modules) list is imported.

Similarly, the [`RAISE_ON_UNINTENTIONAL_PASSTHROUGH`](https://python.temporal.io/temporalio.workflow.SandboxImportNotificationPolicy.html#RAISE_ON_UNINTENTIONAL_PASSTHROUGH) setting will raise an error when a non-passed-through module is imported.

The import notification policy can be set for specific imports by using [`sandbox_import_notification_policy`](https://python.temporal.io/temporalio.workflow.unsafe.html#sandbox_import_notification_policy) context manager.

<!--SNIPSTART python-sdk-sandbox-notification-workflow-->
[python-sandbox/best_practices/sdk_sandbox/notification_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/notification_workflow.py)
```py
from temporalio import workflow

with workflow.unsafe.sandbox_import_notification_policy(
    workflow.SandboxImportNotificationPolicy.SILENT
):
    import pydantic

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        greeting = pydantic.TypeAdapter(str).validate_python(f"Hello, {name}!")
        return greeting
```
<!--SNIPEND-->

This can also be done at worker creation time by customizing the runner's restrictions.

<!--SNIPSTART python-sdk-sandbox-notification-worker-->
[python-sandbox/best_practices/sdk_sandbox/workers.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sdk_sandbox/workers.py)
```py
from temporalio import workflow
from temporalio.worker import Worker
from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions

my_worker = Worker(
  client,
  task_queue=task_queue,
  workflows=[MyWorkflow],
  workflow_runner=SandboxedWorkflowRunner(
    restrictions=SandboxRestrictions.default.with_import_notification_policy(
      workflow.SandboxImportNotificationPolicy.WARN_ON_DYNAMIC_IMPORT | workflow.SandboxImportNotificationPolicy.WARN_ON_UNINTENTIONAL_PASSTHROUGH
    )
  )
)
```
<!--SNIPEND-->

The [`sandbox_import_notification_policy`](https://python.temporal.io/temporalio.workflow.unsafe.html#sandbox_import_notification_policy) context manager will always be respected if used in combination with the restrictions customization. 

For more information on the Python sandbox, see the following resources.

- [Python SDK README](https://github.com/temporalio/sdk-python)
- [Python API docs](https://python.temporal.io/index.html)

---

# Temporal Python SDK synchronous versus asynchronous Activity implementations

Source: https://docs.temporal.io/develop/python/best-practices/python-sdk-sync-vs-async

> The Temporal Python SDK supports implementing Activities asynchronously with asyncio, synchronously with ThreadPoolExecutor or ProcessPoolExecutor. Choose the correct method to avoid application errors.

The Temporal Python SDK supports multiple ways of implementing an Activity:

- Asynchronously using [`asyncio`](https://docs.python.org/3/library/asyncio.html)
- Synchronously multithreaded using [`concurrent.futures.ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor)
- Synchronously multiprocess using [`concurrent.futures.ProcessPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) and [`multiprocessing.managers.SyncManager`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.SyncManager)

It is important to implement your Activities using the correct method, otherwise
your application may fail in sporadic and unexpected ways. Which one you should
use depends on your use case. This section provides guidance to help you choose
the best approach.

## The Python asynchronous event loop and blocking calls

First, let's look at how async event loops work in Python. The Python async
event loop runs in a thread and executes all tasks in its thread. When any
task is running in the event loop, the loop is blocked and no other tasks can be
running at the same time within that event loop. Whenever a task executes an
`await` expression, the task is suspended, and the event loop begins or resumes
execution of another task.

This means that the event loop can only pass the flow of control when the `await`
keyword is executed. If a program makes a blocking call, such as one that reads
from a file, makes a synchronous request to a network service, waits for user input,
or anything else that blocks the execution, the entire event loop must wait until
that execution has completed.

Blocking the async event loop in Python would turn your asynchronous program
into a synchronous program that executes serially, defeating the entire purpose
of using `asyncio`. This can also lead to potential deadlock, and unpredictable behavior
that causes tasks to be unable to execute. Debugging these issues can be difficult
and time consuming, as locating the source of the blocking call might not always
be immediately evident.

Due to this, Python developers must be extra careful to not make blocking calls
from within an asynchronous Activity, or use an async safe library to perform
these actions.

For example, making an HTTP call with the popular `requests` library within an
asynchronous Activity would lead to blocking your event loop. If you want to make
an HTTP call from within an asynchronous Activity, you should use an async-safe HTTP library
such as `aiohttp` or `httpx`. Otherwise, use a synchronous Activity.

## Python SDK Worker Execution Architecture

Python workers have the following components for executing code:

- Your event loop, which runs Tasks from async Activities **plus the rest of the Temporal Worker, such as communicating with the server**.
- An executor for executing Activity Tasks from synchronous Activities. A thread pool executor is recommended.
- A thread pool executor for executing Workflow Tasks.

> See Also: [docs for](https://python.temporal.io/temporalio.worker.Worker.html#__init__) `worker.__init__()`

### Activities

- Async Activities and the temporal worker SDK code both run the default asyncio event loop or whatever event loop you give the Worker.
- Synchronous Activities run in the `activity_executor`.

### Workflows

Since Workflow Tasks have the following three properties, they're run in threads.

- are CPU bound
- need to be timed out for deadlock detection
- need to not block other Workflow Tasks

The `workflow_task_executor` is the thread pool these Tasks are run on.
The fact that Workflow Tasks run in a thread pool can be confusing at first because Workflow Definitions are `async`.
The key differentiator is that the `async` in Workflow Definitions isn't referring to the standard event loop -- it's referring to the Workflow's own event loop.
Each Workflow gets its own “Workflow event loop,” which is deterministic, and described in [the Python SDK blog](https://temporal.io/blog/durable-distributed-asyncio-event-loop#temporal-workflows-are-asyncio-event-loops).
The Workflow event loop doesn't constantly loop -- it just gets cycled through during a Workflow Task to make as much progress as possible on all of its futures.
When it can no longer make progress on any of its futures, then the Workflow Task is complete.

### Number of CPU cores

The only ways to use more than one core in a python Worker (considering Python's GIL) are:

- Run more than one Worker Process.
- Run the synchronous Activities in a process pool executor, but a thread pool executor is recommended.

### A Worker infrastructure option: Separate Activity and Workflow Workers

To reduce the risk of event loops or executors getting blocked,
some users choose to deploy separate Workers for Workflow Tasks and Activity Tasks.

## Activity Definition

**By default, Activities should be synchronous rather than asynchronous**.
You should only make an Activity asynchronous if you are
certain that it doesn't block the event loop.

This is because if you have blocking code in an `async def` function,
it blocks your event loop and the rest of Temporal, which can cause bugs that are
hard to diagnose, including freezing your worker and blocking Workflow progress
(because Temporal can't tell the server that Workflow Tasks are completing).
The reason synchronous Activities help is because they
run in the `activity_executor` ([docs for](https://python.temporal.io/temporalio.worker.Worker.html#__init__) `worker.__init__()`)
rather than in the global event loop,
which helps because:

- There's no risk of accidentally blocking the global event loop.
- If you have multiple
  Activity Tasks running in a thread pool rather than an event loop, one bad
  Activity Task can't slow down the others; this is because the OS scheduler preemptively
  switches between threads, which the event loop coordinator doesn't do.

> See Also:
> ["Types of Activities" section of Python SDK README](https://github.com/temporalio/sdk-python#types-of-activities)

## How to implement Synchronous Activities

The following code is a synchronous Activity Definition.
It takes a name (`str`) as input and returns a
customized greeting (`str`) as output.

It makes a call to a microservice, and when making this call,
you'll notice that it uses the `requests` library. This is safe to do in
synchronous Activities.

<!--SNIPSTART python-sync-vs-async-sync-class-->
[python-sandbox/best_practices/sync_vs_async/sync_class_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sync_vs_async/sync_class_activities.py)
```py
import urllib.parse
import requests
from temporalio import activity

class TranslateActivities:

    @activity.defn
    def greet_in_spanish(self, name: str) -> str:
        greeting = self.call_service("get-spanish-greeting", name)
        return greeting

    # Utility method for making calls to the microservices
    def call_service(self, stem: str, name: str) -> str:
        base = f"http://localhost:9999/{stem}"
        url = f"{base}?name={urllib.parse.quote(name)}"

        response = requests.get(url)
        return response.text
```
<!--SNIPEND-->

The preceding example doesn't share a session across the Activity, so
`__init__` was removed. While `requests` does have the ability to create sessions,
it's currently unknown if they're thread safe. Due to no longer having or needing
`__init__`, the case could be made here to not implement the Activities as a class,
but just as decorated functions as shown here:

<!--SNIPSTART python-sync-vs-async-sync-functions-->
[python-sandbox/best_practices/sync_vs_async/sync_function_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sync_vs_async/sync_function_activities.py)
```py
@activity.defn
def greet_in_spanish(name: str) -> str:
    greeting = call_service("get-spanish-greeting", name)
    return greeting

# Utility method for making calls to the microservices
def call_service(stem: str, name: str) -> str:
    base = f"http://localhost:9999/{stem}"
    url = f"{base}?name={urllib.parse.quote(name)}"

    response = requests.get(url)
    return response.text
```
<!--SNIPEND-->

Whether to implement Activities as class methods or functions is a design
choice left up to the developer when cross-activity state isn't needed. Both are
equally valid implementations.

### How to run Synchronous Activities on a Worker

When running synchronous Activities, the Worker
needs to have an `activity_executor`. Temporal
recommends using a `ThreadPoolExecutor` as shown here:

<!--SNIPSTART python-sync-vs-async-sync-worker-->
[python-sandbox/best_practices/sync_vs_async/sync_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sync_vs_async/sync_worker.py)
```py
with ThreadPoolExecutor(max_workers=42) as executor:
    worker = Worker(
        client,
        task_queue="greeting-tasks",
        workflows=[GreetingWorkflow],
        activities=[activities.greet_in_spanish],
        activity_executor=executor,
    )
```
<!--SNIPEND-->

## How to implement asynchronous Activities

The following code is an implementation of the preceding Activity, but as an
asynchronous Activity Definition.

It makes
a call to a microservice, accessed through HTTP, to request this
greeting in Spanish. This Activity uses the `aiohttp` library to make an async
safe HTTP request. Using the `requests` library here would have resulted in
blocking code within the async event loop, which will block the entire async
event loop. For more in-depth information about this issue, refer to the
[Python asyncio documentation](https://docs.python.org/3/library/asyncio-dev.html#running-blocking-code).

The following code also implements the Activity Definition as a class, rather than a
function. The `aiohttp` library requires an established `Session` to perform the
HTTP request. It would be inefficient to establish a `Session` every time an
Activity is invoked, so instead this code accepts a `Session` object as an instance
parameter and makes it available to the methods. This approach will also be
beneficial when the execution is over and the `Session` needs to be closed.

In this example, the Activity supplies the name in the URL and retrieves
the greeting from the body of the response.

<!--SNIPSTART python-sync-vs-async-async-class-->
[python-sandbox/best_practices/sync_vs_async/async_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/sync_vs_async/async_activities.py)
```py
import aiohttp
import urllib.parse
from temporalio import activity
from temporalio.exceptions import ApplicationError

class TranslateActivities:
    def __init__(self, session: aiohttp.ClientSession):
        self.session = session

    @activity.defn
    async def greet_in_spanish(self, name: str) -> str:
        greeting = await self.call_service("get-spanish-greeting", name)
        return greeting

    # Utility method for making calls to the microservices
    async def call_service(self, stem: str, name: str) -> str:
        base = f"http://localhost:9999/{stem}"
        url = f"{base}?name={urllib.parse.quote(name)}"

        async with self.session.get(url) as response:
            translation = await response.text()

            if response.status >= 400:
                raise ApplicationError(
                    f"HTTP Error {response.status}: {translation}",
                    # We want to have Temporal automatically retry 5xx but not 4xx
                    non_retryable=response.status < 500,
                )

            return translation
```
<!--SNIPEND-->

### How to run synchronous code from an asynchronous activity

If your Activity is asynchronous and you don't want to change it to synchronous,
but you need to run blocking code inside it,
then you can use python utility functions to run synchronous code
in an asynchronous function:

- [`loop.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor), which is also mentioned in the ["running blocking code" section of the "developing with asyncio" guide](https://docs.python.org/3/library/asyncio-dev.html#running-blocking-code)
- [`asyncio.to_thread()`](https://docs.python.org/3/library/asyncio-task.html#running-in-threads)

## When should you use async Activities

Asynchronous Activities have many advantages, such as potential speed up of execution.
However, as discussed above, making unsafe calls within the async event loop
can cause sporadic and difficult to diagnose bugs. For this reason, we recommend
using asynchronous Activities _only_ when you are certain that your Activities
are async safe and don't make blocking calls.

If you experience bugs that you think may be a result of an unsafe call being made in an asynchronous Activity, convert it to a synchronous Activity and see if the issue resolves.

---

# Testing - Python SDK

Source: https://docs.temporal.io/develop/python/best-practices/testing-suite

> The Temporal Application Testing guide covers Frameworks facilitating Workflow and integration testing, including end-to-end, integration, and unit tests. Use mocked Activities, skip time in tests, and replay Workflow Executions.

The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing.

In the context of Temporal, you can create these types of automated tests:

- **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client.
- **Integration:** Anything between end-to-end and unit testing.
  - Running Activities with mocked Context and other SDK imports (and usually network requests).
  - Running Workers with mock Activities, and using a Client to start Workflows.
  - Running Workflows with mocked SDK imports.
- **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls.

We generally recommend writing the majority of your tests as integration tests.

Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers.

## Test frameworks 

Some SDKs have support or examples for popular test frameworks, runners, or libraries.

One recommended framework for testing in Python for the Temporal SDK is [pytest](https://docs.pytest.org/), which can help with fixtures to stand up and tear down test environments, provide useful test discovery, and make it easy to write parameterized tests.

If you do use `pytest`, consider using `-s` (`--show-capture=no`) so you can see the logs live.

## Testing Activities 

An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity.
This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity.

### Run an Activity 

If an Activity references its context, you need to mock that context when testing in isolation.

To run an Activity in a test, use the [`ActivityEnvironment`](https://python.temporal.io/temporalio.testing.ActivityEnvironment.html) class.

This class allows you to run any callable inside an Activity context.
Use it to test the behavior of your code under various conditions.

### Listen to Heartbeats 

When an Activity sends a Heartbeat, be sure that you can see the Heartbeats in your test code so that you can verify them.

To test a Heartbeat in an Activity, use the [`on_heartbeat()`](https://python.temporal.io/temporalio.testing.ActivityEnvironment.html#on_heartbeat) property of the [`ActivityEnvironment`](https://python.temporal.io/temporalio.testing.ActivityEnvironment.html) class.
This property sets a custom function that is called every time the `activity.heartbeat()` function is called within the Activity.

<!--SNIPSTART python-testing-heartbeats-->
[python-sandbox/best_practices/test_testing_suite.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/test_testing_suite.py)
```py
@activity.defn
async def activity_with_heartbeats(param: str):
    activity.heartbeat(f"param: {param}")
    activity.heartbeat("second heartbeat")

env = ActivityEnvironment()
heartbeats = []
# Set the `on_heartbeat` property to a callback function that will be called for each Heartbeat sent by the Activity.
env.on_heartbeat = lambda *args: heartbeats.append(args[0])
# Use the run method to start the Activity, passing in the function that contains the Heartbeats and any necessary parameters.
await env.run(activity_with_heartbeats, "test")
# Verify that the expected Heartbeats are received by the callback function.
assert heartbeats == ["param: test", "second heartbeat"]
```
<!--SNIPEND-->

## Testing Workflows 

The simplest test case we can write is to have the test environment execute the Workflow and then evaluate the results. `WorkflowEnvironment.start_local` configures a local environment for running and testing Workflows:

<!--SNIPSTART python-testing-workflow-->
[python-sandbox/test_workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/test_workflows.py)
```py
import uuid

import pytest
from temporalio import activity
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

from activities import greet
from workflows import SayHelloWorkflow

@pytest.mark.asyncio
async def test_say_hello_workflow():
    """Execute the workflow end-to-end with its real activity."""
    task_queue_name = str(uuid.uuid4())
    async with await WorkflowEnvironment.start_local(ui=True, ui_port=8233) as env:
        async with Worker(
            env.client,
            task_queue=task_queue_name,
            workflows=[SayHelloWorkflow],
            activities=[greet],
        ):
            result = await env.client.execute_workflow(
                SayHelloWorkflow.run,
                "Temporal",
                id=str(uuid.uuid4()),
                task_queue=task_queue_name,
            )
            assert result == "Hello Temporal"
```
<!--SNIPEND-->

You can also pass `ui=True` to `start_local` to see the UI.

### How to mock Activities 

When running unit tests on Workflows, many times you want to test the Workflow logic in isolation. When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker.

<!--SNIPSTART python-testing-mock-activity-->
[python-sandbox/test_workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/test_workflows.py)
```py
@pytest.mark.asyncio
async def test_say_hello_workflow_with_mocked_activity():
    """Run the workflow with the greet activity mocked out."""

    @activity.defn(name="greet")
    async def greet_mocked(name: str) -> str:
        return "mocked greeting"

    task_queue_name = str(uuid.uuid4())
    async with await WorkflowEnvironment.start_local() as env:
        async with Worker(
            env.client,
            task_queue=task_queue_name,
            workflows=[SayHelloWorkflow],
            activities=[greet_mocked],
        ):
            result = await env.client.execute_workflow(
                SayHelloWorkflow.run,
                "Temporal",
                id=str(uuid.uuid4()),
                task_queue=task_queue_name,
            )
            assert result == "mocked greeting"
```
<!--SNIPEND-->

The mocked Activity implementation should have the same signature as the real implementation (including the input and output types) and the same name.
When the Workflow invokes the Activity, it invokes the mocked implementation instead of the real one, allowing you to test your Workflow isolated.

### How to skip time 

Some long-running Workflows can persist for months or even years.
Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount.

For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works.
Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner.

The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time.
Time is a global property of an instance of `WorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests.
If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server.
For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel.

#### Skip time automatically 

Use the [`start_time_skipping()`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#start_time_skipping) method to start a test server process and skip time automatically. In time-skipping mode, Timers, which include sleeps and conditional timeouts, are fast-forwarded except when Activities are running.

Use the [`start_local()`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#start_local) method for testing a full local Temporal Server. There is no time skipping in this environment, so any Timers will wait the actual amount of time. You can find an example of this being used in [the code above](/develop/python/best-practices/testing-suite#test-workflows).

Use the [`from_client()`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#from_client) method for an existing Temporal Server.

#### Skip time manually 

To implement time skipping manually, use the [`sleep`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#sleep) method inside the `WorkflowEnvironment`. This will manually advance time by the duration you specify.

<!--SNIPSTART python-testing-manual-time-skipping-->
[python-sandbox/best_practices/test_testing_suite.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/test_testing_suite.py)
```py
async def test_manual_time_skipping():
    async with await WorkflowEnvironment.start_time_skipping() as env:
        start = await env.get_current_time()
        # You can use the env.sleep(seconds) method to manually advance time
        await env.sleep(3) # This will advance time by 3 seconds
        assert await env.get_current_time() - start >= timedelta(seconds=3)
```
<!--SNIPEND-->

## How to Replay a Workflow Execution 

Replay recreates the exact state of a Workflow Execution.
You can replay a Workflow from the beginning of its Event History.

Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view.

When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks:

1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test.
2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI.
3. Run the Event Histories through replay.
4. Fail CI if any error is encountered during replay.

The following are examples of fetching and replaying Event Histories:

To replay Workflow Executions, use the [`replay_workflows`](https://python.temporal.io/temporalio.worker.Replayer.html#replay_workflows) or [`replay_workflow`](https://python.temporal.io/temporalio.worker.Replayer.html#replay_workflow) methods, passing one or more Event Histories as arguments.

In the following example (which, as of server v1.18, requires Advanced Visibility to be enabled), Event Histories are downloaded from the server and then replayed.
If any replay fails, the code raises an exception.

<!--SNIPSTART python-testing-replay-from-server-->
[python-sandbox/best_practices/test_testing_suite.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/test_testing_suite.py)
```py
workflows = client.list_workflows("TaskQueue='foo' and StartTime > '2022-01-01T12:00:00Z'")
histories = workflows.map_histories()
replayer = Replayer(
    workflows=[MyWorkflowA, MyWorkflowB, MyWorkflowC]
)
await replayer.replay_workflows(histories)
```
<!--SNIPEND-->

In the next example, a single history is loaded from a JSON string:

<!--SNIPSTART python-testing-replay-from-json-->
[python-sandbox/best_practices/test_testing_suite.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/test_testing_suite.py)
```py
replayer = Replayer(workflows=[YourWorkflow])
await replayer.replay_workflow(WorkflowHistory.from_json(workflow_id, history_json_str))
```
<!--SNIPEND-->

In both examples, if Event History is non-deterministic, an error is thrown.
You can choose to wait until all histories have been replayed with `replay_workflows` by setting the `fail_fast` option to `false`.

> **📝 Note:**
>
> If the Event History is exported by [Temporal Web UI](/web-ui) or through [Temporal CLI](/cli), you can pass the JSON file history object as a JSON string or as a Python dictionary through the `json.load()` function, which takes a file object and returns the JSON object.
>
> :::tip
> When fetching event histories directly from the server or exporting them, be aware that the data can be protobuf-encoded (`bytes`). The `Replayer`, however, often works with decoded histories (like a `dict`).
>
> If you encounter `TypeError` exceptions related to `dict` versus `bytes` mismatches during replay, ensure your event history is properly decoded before passing it to the Replayer.

---

# Client - Python SDK

Source: https://docs.temporal.io/develop/python/client

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Temporal Client

- [Temporal Client](/develop/python/client/temporal-client)

---

# Temporal Client - Python SDK

Source: https://docs.temporal.io/develop/python/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service.
Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and
Queries to Workflow Executions, getting Workflow results, and more.

For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow.

This page shows you how to do the following using the Python SDK with the Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-development-service)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Start a Workflow Execution](#start-workflow-execution)
- [Get Workflow results](#get-workflow-results)

A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a
Temporal Client inside an Activity to communicate with a Temporal Service.

## Connect to development Temporal Service 

Use [`Client.connect`](https://python.temporal.io/temporalio.client.Client.html#connect) to create a client. Connection
options include the Temporal Server address, Namespace, and (optionally) TLS configuration. You can provide these
options directly in code, load them from **environment variables**, or a **TOML configuration file** using the
[`envconfig`](https://python.temporal.io/temporalio.envconfig.html) helpers. We recommend environment variables or a
configuration file for secure, repeatable configuration.

When you're running a Temporal Service locally (such as with the
[Temporal CLI dev server](/cli/command-reference/server#start-dev)), the required options are minimal. If you
don't specify a host/port, most connections default to `127.0.0.1:7233` and the `default` Namespace.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the
TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK
looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to
[Environment Configuration](/develop/environment-configuration#configuration-methods) for more details about
configuration files and profiles.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own
set of connection options.

<!--SNIPSTART python-client-config-toml-->
[python-sandbox/client/temporal_client/config.toml](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/config.toml)
```toml
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Optional: Add custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
# TLS auto-enables when TLS config or an API key is present
# disabled = false
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path  = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment     = "production"
service-version = "v1.2.3"
```
<!--SNIPEND-->

You can create a Temporal Client using a profile from the configuration file using the
`ClientConfig.load_client_connect_config` function as follows. In this example, you load the `default` profile for local
development:

<!--SNIPSTART python-client-load-default-profile {"highlightedLines": "23-25"}-->
[python-sandbox/client/temporal_client/load_default.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/load_default.py)
```py {23-25}
import asyncio
from pathlib import Path

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

async def main():
    """
    Loads the default profile from the config.toml file in this directory.
    """
    print("--- Loading default profile from config.toml ---")

    # For this sample to be self-contained, we explicitly provide the path to
    # the config.toml file included in this directory.
    # By default though, the config.toml file will be loaded from
    # ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
    config_file = Path(__file__).parent / "config.toml"

    # load_client_connect_config is a helper that loads a profile and prepares
    # the config dictionary for Client.connect. By default, it loads the
    # "default" profile.
    connect_config = ClientConfig.load_client_connect_config(
        config_file=str(config_file)
    )

    print(f"Loaded 'default' profile from {config_file}.")
    print(f"  Address: {connect_config.get('target_host')}")
    print(f"  Namespace: {connect_config.get('namespace')}")
    print(f"  gRPC Metadata: {connect_config.get('rpc_metadata')}")

    print("\nAttempting to connect to client...")
    try:
        await Client.connect(**connect_config)  # type: ignore
        print("✅ Client connected successfully!")
    except Exception as e:
        print(f"❌ Failed to connect: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

**Environment Variables**

Use the `envconfig` package to set connection options for the Temporal Client using environment variables. For a list of
all available environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

Set the following environment variables before running your Python application. Replace the placeholder values with your
actual configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and
the `default` Namespace. You may omit these variables entirely since they're the defaults.

```bash
export TEMPORAL_NAMESPACE="default"
export TEMPORAL_ADDRESS="localhost:7233"
```

After setting the environment variables, you can create a Temporal Client as follows:

<!--SNIPSTART python-client-load-from-env {"highlightedLines": "10,18"}-->
[python-sandbox/client/temporal_client/load_from_env.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/load_from_env.py)
```py {10,18}
import asyncio

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

async def main():
    # load_client_connect_config is a helper that loads a profile and prepares
    # the config dictionary for Client.connect. By default, it loads the
    # "default" profile.
    connect_config = ClientConfig.load_client_connect_config()

    print(f"  Address: {connect_config.get('target_host')}")
    print(f"  Namespace: {connect_config.get('namespace')}")
    print(f"  gRPC Metadata: {connect_config.get('rpc_metadata')}")

    print("\nAttempting to connect to client...")
    try:
        await Client.connect(**connect_config)  # type: ignore
        print("✅ Client connected successfully!")
    except Exception as e:
        print(f"❌ Failed to connect: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

**Code**

If you don't want to use environment variables or a configuration file, you can specify connection options directly in
code. This is convenient for local development and testing. You can also load a base configuration from environment
variables or a configuration file, and then override specific options in code.

Use the `connect()` method on the `Client` class to create and connect to a Temporal Client to the Temporal Service.

<!--SNIPSTART python-client-start-workflow-->
[python-sandbox/client/temporal_client/start_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/start_workflow.py)
```py
import asyncio
from temporalio.client import Client
from temporal_client.your_workflows import YourWorkflow

async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_workflow(
        YourWorkflow.run,
        "your name",
        id="your-workflow-id",
        task_queue="your-task-queue",
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Connect to Temporal Cloud 

You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal
Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured
local development instance:

- Your credentials for authentication.
  - If you are using an API key, provide the API key value.
  - If you are using mTLS, provide the mTLS CA certificate and mTLS private key.
- Your _Namespace and Account ID_ combination, which follows the format `<namespace_id>.<account_id>`.
- The recommended _endpoint_ is the gRPC Namespace endpoint: `<namespace>.<account>.tmprl.cloud:7233`.
  This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability).
  See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options.

You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab.

For more information about managing and generating client certificates for Temporal Cloud, see
[How to manage certificates in Temporal Cloud](/cloud/certificates).

You can provide these connection options using environment variables, a configuration file, or directly in code.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to
[Environment Configuration](/references/client-environment-configuration).

You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path
to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the
default path `~/.config/temporalio/temporal.toml`.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to
connect to Temporal Cloud via an API key:

<!--SNIPSTART python-client-cloud-api-key-toml-->
[python-sandbox/client/temporal_client/cloud_api_key.toml](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/cloud_api_key.toml)
```toml
# Cloud profile for Temporal Cloud
[profile.staging]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```
<!--SNIPEND-->

If you want to use mTLS authentication instead of an API key, replace the `api_key` field with a `tls` table that holds
your mTLS certificate and private key:

<!--SNIPSTART python-client-cloud-mtls-toml-->
[python-sandbox/client/temporal_client/cloud_mtls.toml](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/cloud_mtls.toml)
```toml
# Cloud profile for Temporal Cloud
[profile.staging]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"

[profile.staging.tls]
client_cert_data = "your-tls-client-cert-data"
client_key_path = "your-tls-client-key-path"
```
<!--SNIPEND-->

With the connections options defined in the configuration file, use the
[`connect` method](https://python.temporal.io/temporalio.client.Client.html#connect) on the `Client` class to create a
Temporal Client using the `staging` profile as follows. After loading the profile, you can also programmatically
override specific connection options before creating the client.

<!--SNIPSTART python-client-load-profile-with-overrides {"highlightedLines": "14,23-25"}-->
[python-sandbox/client/temporal_client/load_profile_with_overrides.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/load_profile_with_overrides.py)
```py {14,23-25}
import asyncio
from pathlib import Path

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

async def main():
    """
    Demonstrates loading a named profile and overriding values programmatically.
    """
    print("--- Loading 'staging' profile with programmatic overrides ---")

    config_file = Path(__file__).parent / "config.toml"
    profile_name = "staging"

    print(
        "The 'staging' profile in config.toml has an incorrect address (localhost:9999)."
    )
    print("We'll programmatically override it to the correct address.")

    # Load the 'staging' profile.
    connect_config = ClientConfig.load_client_connect_config(
        profile=profile_name,
        config_file=str(config_file),
    )

    # Override the target host to the correct address.
    # This is the recommended way to override configuration values.
    connect_config["target_host"] = "localhost:7233"

    print(f"\nLoaded '{profile_name}' profile from {config_file} with overrides.")
    print(
        f"  Address: {connect_config.get('target_host')} (overridden from localhost:9999)"
    )
    print(f"  Namespace: {connect_config.get('namespace')}")

    print("\nAttempting to connect to client...")
    try:
        await Client.connect(**connect_config)  # type: ignore
        print("✅ Client connected successfully!")
    except Exception as e:
        print(f"❌ Failed to connect: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

**Environment Variables**

The following environment variables are required to connect to Temporal Cloud:

- `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `<namespace_id>.<account_id>`.
- `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace.
- `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication.
- `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path.
  Required if you are using mTLS authentication.
- `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path.
  Required if you are using mTLS authentication.

Ensure these environment variables exist in your environment before running your Python application.

Import the `temporalio.envconfig` package to set connection options for the Temporal Client using environment variables.
The `ClientConfig.load_client_connect_config` function will automatically load all environment variables. For a list of all available
environment variables and their default values, refer to [Environment Configuration](/develop/environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

After setting the environment variables, use the following code to create the Temporal Client:

<!--SNIPSTART python-client-load-from-env {"highlightedLines": "10,18"}-->
[python-sandbox/client/temporal_client/load_from_env.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/load_from_env.py)
```py {10,18}
import asyncio

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

async def main():
    # load_client_connect_config is a helper that loads a profile and prepares
    # the config dictionary for Client.connect. By default, it loads the
    # "default" profile.
    connect_config = ClientConfig.load_client_connect_config()

    print(f"  Address: {connect_config.get('target_host')}")
    print(f"  Namespace: {connect_config.get('namespace')}")
    print(f"  gRPC Metadata: {connect_config.get('rpc_metadata')}")

    print("\nAttempting to connect to client...")
    try:
        await Client.connect(**connect_config)  # type: ignore
        print("✅ Client connected successfully!")
    except Exception as e:
        print(f"❌ Failed to connect: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

**Code**

You can also specify connection options directly in code to connect to Temporal Cloud. To create an initial connection,
provide the endpoint, Namespace and Account ID combination, and API key values to the `Client.connect` method.

<!--SNIPSTART python-client-cloud-api-key-->
[python-sandbox/client/temporal_client/cloud_connect.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/cloud_connect.py)
```py
client = await Client.connect(
    endpoint,
    namespace=namespace,
    api_key=api_key,
    tls=True,
)
```
<!--SNIPEND-->

To connect using mTLS instead of an API key, provide the mTLS certificate and private key as follows:

<!--SNIPSTART python-client-cloud-mtls-->
[python-sandbox/client/temporal_client/mtls_connect.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/mtls_connect.py)
```py
from temporalio.client import Client, TLSConfig

async def main():
    with open("client-cert.pem", "rb") as f:
        client_cert = f.read()
    with open("client-private-key.pem", "rb") as f:
        client_private_key = f.read()
    client = await Client.connect(
        "your-custom-namespace.tmprl.cloud:7233",
        namespace="<your-custom-namespace>.<account-id>",
        tls=TLSConfig(
            client_cert=client_cert,
            client_private_key=client_private_key,
            # domain=domain, # TLS domain
            # server_root_ca_cert=server_root_ca_cert, # ROOT CA to validate the server cert
        ),
    )
```
<!--SNIPEND-->

For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see
[Temporal Customization Samples](https://github.com/temporalio/samples-server).

Unlike an API key, `TLSConfig` is static: the certificate bytes you pass to `Client.connect` are fixed for the lifetime
of that `Client`. To rotate an mTLS client certificate without restarting your Worker, connect a new `Client` with the
new certificate, then assign it to the running `Worker`'s `client` property:

<!--SNIPSTART python-client-rotate-mtls-cert-->
[python-sandbox/client/temporal_client/cloud_connect.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/cloud_connect.py)
```py
with open("client-cert-new.pem", "rb") as f:
    client_cert = f.read()
with open("client-private-key-new.pem", "rb") as f:
    client_private_key = f.read()

new_client = await Client.connect(
    "your-custom-namespace.tmprl.cloud:7233",
    namespace="<your-custom-namespace>.<account-id>",
    tls=TLSConfig(
        client_cert=client_cert,
        client_private_key=client_private_key,
    ),
)

# worker is the Worker instance already running against the old client
worker.client = new_client
```
<!--SNIPEND-->

The Worker starts using `new_client` for subsequent calls to the Temporal Service (Workflow Task completion, Activity
Heartbeats, and so on); calls already in flight on the old client finish normally. The new client must use the same
`Runtime` as the Worker's current client.

## Start a Workflow Execution 

[Workflow Execution](/workflow-execution) semantics rely on several parameters. To start a Workflow Execution, you must
supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific
contextual data, and Workflow Function parameters.

In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from
within another Workflow Execution, use either the [Child Workflow](/develop/python/workflows/child-workflows) or External Workflow
APIs.

See the [Customize Workflow Type](/develop/python/workflows/basics#workflow-type) section to see how to customize the
name of the Workflow Type.

A request to spawn a Workflow Execution causes the Temporal Service to create the first Event
([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The
Temporal Service then creates the first Workflow Task, resulting in the first
[WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event.

To start a Workflow Execution in Python, use either the
[`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or
[`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods
in the Client.

<!--SNIPSTART python-client-start-workflow {"selectedLines": ["6-20"]}-->
[python-sandbox/client/temporal_client/start_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/start_workflow.py)
```py
# ...
async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_workflow(
        YourWorkflow.run,
        "your name",
        id="your-workflow-id",
        task_queue="your-task-queue",
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Set a Workflow's Task Queue 

The only Workflow Option that you must set is the name of the [Task Queue](/task-queue).

For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task
Queue name.

To set a Task Queue in Python, specify the `task_queue` argument when executing a Workflow with either
[`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or
[`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) methods.

<!--SNIPSTART python-client-start-workflow {"selectedLines": ["6-20"]}-->
[python-sandbox/client/temporal_client/start_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/start_workflow.py)
```py
# ...
async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_workflow(
        YourWorkflow.run,
        "your name",
        id="your-workflow-id",
        task_queue="your-task-queue",
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Set a Workflow Id 

You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id).

When setting a Workflow Id, we recommend mapping it to a business process or business entity identifier, such as an
order identifier or customer identifier.

To set a Workflow Id in Python, specify the `id` argument when executing a Workflow with either
[`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or
[`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) methods.

The `id` argument should be a unique identifier for the Workflow Execution.

<!--SNIPSTART python-client-start-workflow {"selectedLines": ["6-20"]}-->
[python-sandbox/client/temporal_client/start_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/start_workflow.py)
```py
# ...
async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_workflow(
        YourWorkflow.run,
        "your name",
        id="your-workflow-id",
        task_queue="your-task-queue",
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Get Workflow results 

If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id.

The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its
result.

It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time
(asynchronous execution).

In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and
results of Workflow Executions.

Use [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or
[`get_workflow_handle()`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle) to return a
Workflow handle. Then use the [`result`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#result) method
to await on the result of the Workflow.

To get a handle for an existing Workflow by its Id, you can use
[`get_workflow_handle()`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle), or use
[`get_workflow_handle_for()`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle_for) for type
safety.

Then use [`describe()`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#describe) to get the current
status of the Workflow. If the Workflow does not exist, this call fails.

<!--SNIPSTART python-client-get-workflow-result {"highlightedLines": "8-11"}-->
[python-sandbox/client/temporal_client/get_workflow_result.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/client/temporal_client/get_workflow_result.py)
```py {8-11}
import asyncio
from temporalio.client import Client

async def main():
    client = await Client.connect("localhost:7233")

    handle = client.get_workflow_handle(
        workflow_id="your-workflow-id",
    )
    results = await handle.result()
    print(f"Result: {results}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

---

# Data handling - Python SDK

Source: https://docs.temporal.io/develop/python/data-handling

All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three
layers that handle different concerns:

![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)

Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON
serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when
your application requires non-JSON types, encryption, or payload offloading.

|                           | [PayloadConverter](/develop/python/data-handling/data-conversion) | [PayloadCodec](/develop/python/data-handling/data-encryption) | [ExternalStorage](/develop/python/data-handling/external-storage) |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------- |
| **Purpose**               | Serialize application data to bytes                               | Transform encoded payloads (encrypt, compress)                | Offload large payloads to external store                               |
| **Default**               | JSON serialization                                                | None (passthrough)                                            | None (all payloads are stored in Event History)                      |

For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage).

---

# Payload conversion - Python SDK

Source: https://docs.temporal.io/develop/python/data-handling/data-conversion

> Customize how Temporal serializes application objects using Payload Converters in the Python SDK, including Pydantic and custom type examples.

Payload Converters serialize your application objects into a `Payload` and deserialize them back.
A `Payload` is a binary form with metadata that Temporal uses to transport data.

By default, Temporal uses a `DefaultPayloadConverter` that handles `None`, `bytes`, protobuf messages, and anything JSON-serializable.
You only need a custom Payload Converter when your application uses types that aren't natively supported.

## Default supported types

The default Data Converter supports converting multiple types including:

- `None`
- `bytes`
- `google.protobuf.message.Message` — As JSON when encoding, but can decode binary proto from other languages
- Anything that can be converted to JSON including:
  - Anything that [`json.dump`](https://docs.python.org/3/library/json.html#json.dump) supports natively
  - [dataclasses](https://docs.python.org/3/library/dataclasses.html)
  - Iterables including ones JSON dump may not support by default, for example `set`
  - [IntEnum, StrEnum](https://docs.python.org/3/library/enum.html) based enumerates
  - [UUID](https://docs.python.org/3/library/uuid.html)

Although Workflows, Updates, Signals, and Queries can all be defined with multiple input parameters, users are strongly
encouraged to use a single `dataclass` or Pydantic model parameter so that fields with defaults can be easily added
without breaking compatibility.
Similar advice applies to return values.

Classes with generics may not have the generics properly resolved.
The current implementation does not have generic type resolution.
Users should use concrete types.

## Use Pydantic models

To use Pydantic model instances, install Pydantic and set the Pydantic Data Converter when creating Client instances:

<!--SNIPSTART python-data-conversion-pydantic-->
[python-sandbox/best_practices/data_conversion/pydantic_client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/data_conversion/pydantic_client.py)
```py
from temporalio.contrib.pydantic import pydantic_data_converter

client = await Client.connect("localhost:7233", data_converter=pydantic_data_converter)
```
<!--SNIPEND-->

This Data Converter supports conversion of all [types supported by Pydantic](https://docs.pydantic.dev/latest/api/standard_library_types/) to and from JSON.
In addition to Pydantic models, supported types include:

- Everything that [`json.dumps()`](https://docs.python.org/3/library/json.html#py-to-json-table) supports by default.
- Several standard library types that `json.dumps()` does not support, including dataclasses, types from the datetime module, sets, and UUID.
- Custom types composed of any of these, with any degree of nesting.
  For example, a list of Pydantic models with `datetime` fields.

See the [Pydantic documentation](https://docs.pydantic.dev/latest/api/standard_library_types/) for full details.

> **📝 Note:**
>
> Pydantic v1 isn't supported by this Data Converter.
> If you aren't yet able to upgrade from Pydantic v1, see https://github.com/temporalio/samples-python/tree/main/pydantic_converter_v1 for limited v1 support.
>

`datetime.date`, `datetime.time`, and `datetime.datetime` can only be used with the Pydantic Data Converter.

## How the default converter works

The default converter is a `CompositePayloadConverter` that tries each encoding converter in order until one handles the value.
Upon serialization, each `EncodingPayloadConverter` is used in order until one succeeds.

Payload Converters can be customized independently of a Payload Codec.

## Custom Payload Converters

To handle custom data types, create a new `EncodingPayloadConverter`.
For example, to support `IPv4Address` types:

<!--SNIPSTART python-data-conversion-encoding-converter-->
[python-sandbox/best_practices/data_conversion/ipv4_encoding_converter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/data_conversion/ipv4_encoding_converter.py)
```py
import dataclasses
import ipaddress
from typing import Any, Optional, Type

from temporalio.api.common.v1 import Payload
from temporalio.converter import (
    CompositePayloadConverter,
    DataConverter,
    DefaultPayloadConverter,
    EncodingPayloadConverter,
)

class IPv4AddressEncodingPayloadConverter(EncodingPayloadConverter):
    @property
    def encoding(self) -> str:
        return "text/ipv4-address"

    def to_payload(self, value: Any) -> Optional[Payload]:
        if isinstance(value, ipaddress.IPv4Address):
            return Payload(
                metadata={"encoding": self.encoding.encode()},
                data=str(value).encode(),
            )
        else:
            return None

    def from_payload(self, payload: Payload, type_hint: Optional[Type] = None) -> Any:
        assert not type_hint or type_hint is ipaddress.IPv4Address
        return ipaddress.IPv4Address(payload.data.decode())

class IPv4AddressPayloadConverter(CompositePayloadConverter):
    def __init__(self) -> None:
        # Just add ours as first before the defaults
        super().__init__(
            IPv4AddressEncodingPayloadConverter(),
            *DefaultPayloadConverter.default_encoding_payload_converters,
        )

my_data_converter = dataclasses.replace(
    DataConverter.default,
    payload_converter_class=IPv4AddressPayloadConverter,
)
```
<!--SNIPEND-->

### Customize the JSON converter for custom types

If you need your custom type to work in lists, unions, and other collections, customize the existing JSON converter instead of adding a new encoding converter.
The JSON converter is the last in the list, so it handles any otherwise unknown type.

Customize serialization with a custom `json.JSONEncoder` and deserialization with a custom `JSONTypeConverter`:

<!--SNIPSTART python-data-conversion-json-converter-->
[python-sandbox/best_practices/data_conversion/ipv4_json_converter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/data_conversion/ipv4_json_converter.py)
```py
import dataclasses
import inspect
import ipaddress
from typing import Any, Optional, Type, Union

from temporalio.converter import (
    AdvancedJSONEncoder,
    CompositePayloadConverter,
    DataConverter,
    DefaultPayloadConverter,
    JSONPlainPayloadConverter,
    JSONTypeConverter,
    JSONTypeConverterUnhandled,
)

class IPv4AddressJSONEncoder(AdvancedJSONEncoder):
    def default(self, o: Any) -> Any:
        if isinstance(o, ipaddress.IPv4Address):
            return str(o)
        return super().default(o)

class IPv4AddressJSONTypeConverter(JSONTypeConverter):
    def to_typed_value(
        self, hint: Type, value: Any
    ) -> Union[Optional[Any], JSONTypeConverterUnhandled]:
        # Collection hints such as list[IPv4Address] aren't classes, so check first
        if inspect.isclass(hint) and issubclass(hint, ipaddress.IPv4Address):
            return ipaddress.IPv4Address(value)
        return JSONTypeConverter.Unhandled

class IPv4AddressPayloadConverter(CompositePayloadConverter):
    def __init__(self) -> None:
        # Replace default JSON plain with our own that has our encoder and type
        # converter
        json_converter = JSONPlainPayloadConverter(
            encoder=IPv4AddressJSONEncoder,
            custom_type_converters=[IPv4AddressJSONTypeConverter()],
        )
        super().__init__(
            *[
                c if not isinstance(c, JSONPlainPayloadConverter) else json_converter
                for c in DefaultPayloadConverter.default_encoding_payload_converters
            ]
        )

my_data_converter = dataclasses.replace(
    DataConverter.default,
    payload_converter_class=IPv4AddressPayloadConverter,
)
```
<!--SNIPEND-->

Now `IPv4Address` can be used in type hints, including collections and optionals.

---

# Payload encryption - Python SDK

Source: https://docs.temporal.io/develop/python/data-handling/data-encryption

> Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Python SDK.

Payload Codecs transform `Payload` bytes after serialization (by the Payload Converter) and before the data is sent to the Temporal Service.
Unlike Payload Converters, codecs run outside the Workflow sandbox, so they can use non-deterministic operations and call external services.

The most common use case is encryption: encrypting payloads before they reach the Temporal Service so that sensitive data is never stored in plaintext.

## PayloadCodec interface

Implement a `PayloadCodec` with `encode()` and `decode()` methods.
These should loop through all of a Workflow's payloads, perform your marshaling, compression, or encryption steps in order, and set an `"encoding"` metadata field.

In this example, the `encode` method compresses a payload using Python's [cramjam](https://github.com/milesgranger/cramjam) library to provide `snappy` compression.
The `decode()` function implements the `encode()` logic in reverse:

<!--SNIPSTART python-data-encryption-codec-->
[python-sandbox/best_practices/data_encryption/codec.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/data_encryption/codec.py)
```py
from typing import Iterable, List

import cramjam
from temporalio.api.common.v1 import Payload
from temporalio.converter import PayloadCodec

class CompressionCodec(PayloadCodec):
    async def encode(self, payloads: Iterable[Payload]) -> List[Payload]:
        return [
            Payload(
                metadata={
                    "encoding": b"binary/snappy",
                },
                data=(bytes(cramjam.snappy.compress(p.SerializeToString()))),
            )
            for p in payloads
        ]

    async def decode(self, payloads: Iterable[Payload]) -> List[Payload]:
        ret: List[Payload] = []
        for p in payloads:
            if p.metadata.get("encoding", b"").decode() != "binary/snappy":
                ret.append(p)
                continue
            ret.append(Payload.FromString(bytes(cramjam.snappy.decompress(p.data))))
        return ret
```
<!--SNIPEND-->

## Configure the codec on the Data Converter

Add a `data_converter` parameter to your `Client.connect()` options that overrides the default converter with your Payload Codec:

<!--SNIPSTART python-data-encryption-client-->
[python-sandbox/best_practices/data_encryption/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/data_encryption/client.py)
```py
import dataclasses

import temporalio.converter
from temporalio.client import Client

from data_encryption.codec import CompressionCodec

async def connect() -> Client:
    client = await Client.connect(
        "localhost:7233",
        data_converter=dataclasses.replace(
            temporalio.converter.default(), payload_codec=CompressionCodec()
        ),
    )
```
<!--SNIPEND-->

For reference, see the [Encryption](https://github.com/temporalio/samples-python/tree/main/encryption) sample.

## Codec Server

A Codec Server is an HTTP server that runs your `PayloadCodec` remotely, so that the Temporal Web UI and CLI can decode encrypted payloads for display.

For more information, see [Codec Server](/codec-server).

---

# External Storage - Python SDK

Source: https://docs.temporal.io/develop/python/data-handling/external-storage

> Offload large payloads to external storage using the claim check pattern in the Python SDK.

> **Public Preview**
> APIs and configuration may change before General Availability. Join the
> [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for
> help.

The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted
deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external
storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how
to set up External Storage with Amazon S3 and how to implement a custom storage driver.

For a conceptual overview of External Storage and its use cases, see [External Storage](/external-storage).

## Store and retrieve large payloads with Amazon S3

The Python SDK includes an S3 storage driver. Follow these steps to set it up:

### Prerequisites

- An Amazon S3 bucket that you have read and write access to. Refer to
  [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire
  lifetime of the Workflow. For multi-region durability, see
  [Durable External Storage](/external-storage#durable-external-storage).
- Install the `aioboto3` extra: `python -m pip install "temporalio[aioboto3]"`

### Procedure

1. Create an S3 client using `aioboto3` and pass it to the `S3StorageDriver`. The driver uses your standard AWS
   credentials from the environment (environment variables, IAM role, or AWS config file):

   <!--SNIPSTART python-external-storage-s3-driver-create-->
   [python-sandbox/best_practices/external_storage/s3_driver.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/external_storage/s3_driver.py)
   ```py
   session = aioboto3.Session(profile_name=AWS_PROFILE, region_name=AWS_REGION)
   async with session.client("s3") as s3_client:
       driver = S3StorageDriver(
           client=new_aioboto3_client(s3_client),
           bucket="my-temporal-payloads",
       )
   ```
   <!--SNIPEND-->

2. Configure the driver on your `DataConverter` and pass the converter to your Client and Worker:

   <!--SNIPSTART python-external-storage-s3-setup-->
   [python-sandbox/best_practices/external_storage/s3_driver.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/external_storage/s3_driver.py)
   ```py
   data_converter = dataclasses.replace(
       DataConverter.default,
       external_storage=ExternalStorage(drivers=[driver]),
   )

   client_config = ClientConfig.load_client_connect_config()

   client = await Client.connect(**client_config, data_converter=data_converter)

   worker = Worker(
       client,
       task_queue="my-task-queue",
       workflows=[LargePayloadWorkflow],
   )
   ```
   <!--SNIPEND-->

By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the
`payload_size_threshold` parameter, even setting it to 0 to externalize all payloads regardless of size. Refer to
[Configure payload size threshold](#configure-payload-size-threshold) for more information.

All Workflows and Activities running on the Worker use the storage driver automatically without changes to your business
logic. The driver uploads and downloads payloads concurrently and validates payload integrity on retrieve.

The S3 driver includes diagnostic metadata, such as the AWS region, in error messages to help troubleshoot storage
failures. For a complete working example that includes a Worker, Codec Server, and S3 driver, see the
[External Storage sample](https://github.com/temporalio/samples-python/tree/main/external_storage).

## Implement a custom storage driver

If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver.
Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and [Lifecycle management](/external-storage#lifecycle) for retention requirements.

The following example shows a custom driver that uses local disk as the backing store. This example is for local
development and testing only. In production, use a durable storage system that is accessible to all Workers.
For example, see the [Redis storage driver sample](https://github.com/temporalio/samples-python/tree/main/external_storage_redis).

<!--SNIPSTART python-external-storage-custom-driver-->
[python-sandbox/best_practices/external_storage/local_disk_driver.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/external_storage/local_disk_driver.py)
```py
class LocalDiskStorageDriver(StorageDriver):
    def __init__(self, store_dir: str = "/tmp/temporal-payload-store") -> None:
        self._store_dir = store_dir

    def name(self) -> str:
        return "local-disk"

    async def store(
        self,
        context: StorageDriverStoreContext,
        payloads: Sequence[Payload],
    ) -> list[StorageDriverClaim]:
        os.makedirs(self._store_dir, exist_ok=True)

        prefix = self._store_dir
        target = context.target
        if isinstance(target, StorageDriverWorkflowInfo) and target.id:
            prefix = os.path.join(self._store_dir, target.namespace, target.id)
            os.makedirs(prefix, exist_ok=True)

        claims = []
        for payload in payloads:
            key = f"{uuid.uuid4()}.bin"
            file_path = os.path.join(prefix, key)
            with open(file_path, "wb") as f:
                f.write(payload.SerializeToString())
            claims.append(StorageDriverClaim(claim_data={"path": file_path}))
        return claims

    async def retrieve(
        self,
        context: StorageDriverRetrieveContext,
        claims: Sequence[StorageDriverClaim],
    ) -> list[Payload]:
        payloads = []
        for claim in claims:
            file_path = claim.claim_data["path"]
            with open(file_path, "rb") as f:
                raw = f.read()
            payload = Payload()
            payload.ParseFromString(raw)
            payloads.append(payload)
        return payloads
```
<!--SNIPEND-->

The following sections walk through the key parts of the driver implementation.

### 1. Extend the StorageDriver class

A custom driver extends the `StorageDriver` abstract class and implements three methods:

- `name()` returns a unique string that identifies the driver. The SDK stores this name in the claim check reference so
  it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks
  retrieval.
- `store()` receives a list of payloads and returns one `StorageDriverClaim` per payload. A claim is a set of string
  key-value pairs that the driver uses to locate the payload later.
- `retrieve()` receives the claims that `store()` produced and returns the original payloads.

### 2. Store payloads

In `store()`, convert each Payload protobuf message to bytes with `payload.SerializeToString()` and write the bytes to
your storage system. The application data has already been serialized by the
[Payload Converter](/develop/python/data-handling/data-conversion) and
[Payload Codec](/develop/python/data-handling/data-encryption) before it reaches the driver. See the
[data conversion pipeline](/external-storage#data-pipeline) for more details.

Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `context.target`
provides identity information (namespace, Workflow ID, or Activity ID) depending on the operation. Consider structuring
your storage keys to include this information so that you can identify which Workflow owns each payload. Within that
scope, content-addressable keys (such as a SHA-256 hash of the payload bytes) can help deduplicate identical payloads.

### 3. Retrieve payloads

In `retrieve()`, download the bytes using the claim data, then reconstruct the Payload protobuf message with
`payload.ParseFromString(data)`. The Payload Converter handles deserializing the application data after the driver
returns the payload.

### 4. Configure the Data Converter

Pass an `ExternalStorage` instance to your `DataConverter` and use the converter when creating your Client and Worker.
You can also package your driver as a [plugin](/develop/plugins-guide) for easier reuse across services:

<!--SNIPSTART python-external-storage-custom-driver-data-converter-->
[python-sandbox/best_practices/external_storage/local_disk_driver.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/external_storage/local_disk_driver.py)
```py
data_converter = dataclasses.replace(
    DataConverter.default,
    external_storage=ExternalStorage(
        drivers=[LocalDiskStorageDriver()],
    ),
)
```
<!--SNIPEND-->

## Configure payload size threshold

You can configure the payload size threshold that triggers external storage. By default, payloads of 256 KiB or larger
are offloaded to external storage. You can adjust this with the `payload_size_threshold` parameter, or set it to 0 to
externalize all payloads regardless of size. Payloads smaller than the threshold stay inline in Event History. The size
compared against the threshold is that of the serialized Payload, which includes its metadata, not just your data.

<!--SNIPSTART python-external-storage-threshold-->
[features/snippets/external_storage/threshold/threshold_config.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/threshold/threshold_config.py)
```py
data_converter = dataclasses.replace(
    DataConverter.default,
    external_storage=ExternalStorage(
        drivers=[driver],
        payload_size_threshold=0,
    ),
)
```
<!--SNIPEND-->

## Use multiple storage drivers

When you register multiple drivers, you must provide a `driver_selector` function that chooses which driver stores each
payload. Any driver in the list that is not selected for storing is still available for retrieval, which is useful when
migrating between storage backends. Return `None` from the selector to keep a specific payload inline in Event History.

Multiple drivers are useful in scenarios such as:

- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one
  you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old
  driver remains available for retrieving existing claims.
- Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3
  for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver
  based on the runtime environment.

The following example registers two drivers but always selects `preferred_driver` for new payloads. The `legacy_driver`
is only registered so the Worker can retrieve payloads that were previously stored with it:

<!--SNIPSTART python-external-storage-multiple-drivers-->
[features/snippets/external_storage/multiple_drivers/multiple_drivers.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/multiple_drivers/multiple_drivers.py)
```py
preferred_driver = S3StorageDriver(client=s3_client, bucket="my-bucket")
legacy_driver = LegacyStorageDriver()

ExternalStorage(
    drivers=[preferred_driver, legacy_driver],
    driver_selector=lambda context, payload: preferred_driver,
)
```
<!--SNIPEND-->

## Multi-region durability

To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with
[Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) and an
[S3 Multi-Region Access Point (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/), then point the
driver at the MRAP ARN instead of a bucket name. See
[Durable External Storage](/external-storage#durable-external-storage) for the full pattern and trade-offs.

In code, the only change is the value you pass as `bucket`:

<!--SNIPSTART python-external-storage-s3-mrap-->
[python-sandbox/best_practices/external_storage/s3_driver.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/best_practices/external_storage/s3_driver.py)
```py
session = aioboto3.Session(profile_name=AWS_PROFILE, region_name=AWS_REGION)
async with session.client("s3") as s3_client:
    driver = S3StorageDriver(
        client=new_aioboto3_client(s3_client),
        bucket="arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap",
    )
```
<!--SNIPEND-->

`aioboto3` (via `botocore`) automatically uses [SigV4A](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html)
signing when the bucket value is an MRAP ARN. Make sure your `botocore` version is recent enough to support SigV4A.

---

# Integrations - Python SDK

Source: https://docs.temporal.io/develop/python/integrations

> Integrations with other tools and services.

The following integrations are available for the Temporal Python SDK.
Most of these integrations are built on the Temporal Python SDK's [Plugin system](/develop/plugins-guide), which you
can also use to build your own integrations. A few, such as OpenRouter, are sample patterns that need nothing inside
Workflow code.

Temporal-maintained AI integrations are distributed as [standalone packages](https://github.com/temporalio/ai-integrations),
such as `temporalio-strands-agents`. Each integration guide lists its package and import paths.

- [Baseten](/develop/python/integrations/baseten) — Run OpenAI Agents SDK agents on open-weight models served by Baseten, with every model call as a durable Temporal Activity. _(Python · Model provider)_
- [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#python) — Monitor and evaluate AI application performance with Braintrust observability. _(Python · Agent observability)_
- [Deep Agents](/develop/python/integrations/deepagents) — Make LangChain Deep Agents durable with Temporal Workflows and Activities. _(Python · Agent framework)_
- [Fireworks AI](/develop/python/integrations/fireworks-ai) — Run OpenAI Agents SDK agents on open-weight models served by Fireworks AI, with every model call as a durable Temporal Activity. _(Python · Model provider)_
- [Google ADK](/develop/python/integrations/google-adk) — Run Google ADK agents as durable Temporal Workflows. _(Python · Agent framework)_
- [Google GenAI](/develop/python/integrations/google-genai) — Call Google Gemini models durably from Temporal Workflows with the Google Gen AI SDK. _(Python · Agent framework)_
- [Laminar](https://laminar.sh/docs/integrations/temporal) — Trace Temporal Workflows, Activities, and the LLM calls inside them with Laminar. _(Python · Agent observability)_
- [Langfuse](https://langfuse.com/integrations/frameworks/temporal) — Trace and debug LLM calls in Temporal Workflows with Langfuse. _(Python · Agent observability)_
- [LangGraph](/develop/python/integrations/langgraph) — Run LangGraph agent graphs as durable, resumable Temporal Workflows. _(Python · Agent framework)_
- [LangSmith](/develop/python/integrations/langsmith) — Trace and debug LLM calls in Temporal Workflows with LangSmith. _(Python · Agent observability)_
- [Model Context Protocol (MCP)](/develop/python/integrations/mcp) — Call MCP tools, prompts, and resources from Python Workflows as durable Temporal Activities. _(Python · Framework)_
- [OpenAI Agents SDK](/develop/python/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(Python · Agent framework)_
- [OpenBox](https://docs.openbox.ai/getting-started/temporal) — Add governance and observability guardrails to AI-powered Temporal Workflows with OpenBox. _(Python · Governance · Agent observability)_
- [OpenRouter](/develop/python/integrations/openrouter) — Call hundreds of models through OpenRouter from Temporal Activities, with durable retries, cost tracking, and budgets. _(Python · Model gateway)_
- [Parseable](https://github.com/parseablehq/temporal-plugin-python/blob/main/INTEGRATION.MD) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(Python · Agent observability)_
- [Pydantic AI](https://pydantic.dev/docs/ai/capabilities/durable_execution/temporal/) — Build type-safe AI agents with Durable Execution through Pydantic AI. _(Python · Agent framework)_
- [Strands Agents](/develop/python/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(Python · Agent framework)_
- [Tenuo](https://tenuo.ai/temporal) — Add governance and compliance guardrails to AI-powered Temporal Workflows. _(Python · Governance)_
- [Together AI](/develop/python/integrations/together-ai) — Run OpenAI Agents SDK agents on open-weight models served by Together AI, with every model call as a durable Temporal Activity. _(Python · Model provider)_
- [xMemory](https://xmemory.ai/temporal/#quickstart--python) — Store and recall agent memory across Workflow Executions with xMemory. _(Python · Agent memory)_

---

# Baseten integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/baseten

[Baseten](https://www.baseten.co/) serves open-weight models such as gpt-oss, DeepSeek, GLM, and Kimi, both on its
shared Model APIs and on dedicated deployments, through an API compatible with the OpenAI Chat Completions API.

Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an
Activity. A call to Baseten gets retries with backoff, a timeout, and a durable record in Event History, and the agent
loop around it survives Worker restarts. Because Baseten speaks the OpenAI Chat Completions API, the integration's model
provider hook is all you need: a model provider that maps a model name such as `baseten/openai/gpt-oss-120b` to
Baseten's endpoint. One Worker can serve Baseten alongside other providers, and each Workflow picks its provider through
the model name it gives its agent.

This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the
[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers),
which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker.

## Prerequisites

- This guide assumes you are already familiar with the
  [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Baseten's
  [OpenAI-compatible API](https://docs.baseten.co/inference/model-apis/overview).
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following
  [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal
  development server running.
- Create a [Baseten API key](https://app.baseten.co/settings/api_keys) and export it as `BASETEN_API_KEY` in the
  Worker's environment. The key stays in the Worker process; it is never part of Workflow input or Event History.

## Install

```bash
uv add temporalio-openai-agents
```

The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. The Baseten CLI and the
`truss` package are for deploying models to Baseten; calling Model APIs needs neither.

## Configure the model provider

The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model
with tool calling:

<!--SNIPSTART python-openai-agents-baseten-provider-->
[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py)
```py
BASETEN = OpenAICompatibleProvider(
    name="baseten",
    base_url="https://inference.baseten.co/v1",
    api_key_env="BASETEN_API_KEY",
    default_model="openai/gpt-oss-120b",
)
```
<!--SNIPEND-->

Model APIs name models the way Hugging Face does, as `<organization>/<model>`, so the example here is
`openai/gpt-oss-120b`, named `baseten/openai/gpt-oss-120b` from a Workflow: an open-weight model under the Apache 2.0
license that Baseten serves on Model APIs, where tool calling is supported for every model. Its
[model library page](https://www.baseten.co/library/gpt-oss-120b/) and `GET https://inference.baseten.co/v1/models`
describe the catalog. A [dedicated deployment](https://docs.baseten.co/inference/calling-your-model) has its own base
URL, `https://model-<id>.api.baseten.co/environments/production/sync/v1`, takes the same API key, and expects the model
name that the deployment serves. The sample's `baseten/` prefix only reaches Model APIs: to call a deployment, add a
`PROVIDERS` entry with a distinct name and that base URL, such as `baseten-dedicated`, and use its prefix in the model
name.

A `ModelProvider` routes a `<provider>/<model>` name to the matching endpoint and returns the Chat Completions model
that the Agents SDK calls:

<!--SNIPSTART python-openai-agents-openai-compatible-model-provider-->
[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py)
```py
class OpenAICompatibleModelProvider(ModelProvider):
    """Routes ``<provider>/<model>`` names to the matching OpenAI-compatible endpoint.

    One Worker serves every provider whose API key is set, so Workflows on
    different providers run side by side on the same Task Queue. Client
    retries are off: the plugin runs each model call as a Temporal Activity,
    and the Activity's Retry Policy owns the retries.
    """

    def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None:
        self._providers = {provider.name: provider for provider in providers}
        self._clients: dict[str, AsyncOpenAI] = {}
        for provider in self._providers.values():
            api_key = os.environ.get(provider.api_key_env)
            if api_key:
                self._clients[provider.name] = AsyncOpenAI(
                    base_url=provider.base_url, api_key=api_key, max_retries=0
                )

    @property
    def configured(self) -> list[str]:
        """Names of the providers whose API key is set."""
        return sorted(self._clients)

    def get_model(self, model_name: str | None) -> Model:
        provider_name, _, model = (model_name or "").partition("/")
        provider = self._providers.get(provider_name)
        if provider is None or not model:
            # A mistake in the Workflow's input: fail the Activity without retrying.
            raise ApplicationError(
                f"Model name {model_name!r} must be <provider>/<model> with a "
                f"provider in {sorted(self._providers)}",
                non_retryable=True,
            )
        client = self._clients.get(provider_name)
        if client is None:
            raise ApplicationError(
                f"Set {provider.api_key_env} and restart the Worker to use {provider_name}",
                non_retryable=True,
            )
        # Chat Completions is the API all of these providers serve. The Agents
        # SDK defaults to the Responses API, which not every provider offers.
        return OpenAIChatCompletionsModel(model=model, openai_client=client)

```
<!--SNIPEND-->

Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name
for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off
(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a
hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents
SDK's default Responses API model, which Baseten does not serve: Model APIs expose Chat Completions and, in beta, the
Anthropic Messages API.

## Configure the Worker

Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A
custom model provider must set a model Activity timeout; the sample uses 60 seconds. Model APIs need no deployment of
your own to warm up; a dedicated deployment that has scaled to zero parks a request until a replica starts, for up to
1200 seconds by default, so cover that cold start in the timeout or keep `min_replica` above zero. The Heartbeat Timeout
lets Temporal notice a Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the
plugin heartbeats on its own.

<!--SNIPSTART python-openai-agents-openai-compatible-worker-->
[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py)
```py
# One provider for every endpoint whose API key is set. The model name on
# each Workflow's agent selects the endpoint.
model_provider = OpenAICompatibleModelProvider(PROVIDERS.values())
if not model_provider.configured:
    env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values())
    sys.exit(f"No provider API key is set; export at least one of {env_vars}.")
print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True)

client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                # A custom model provider must set a timeout. Raise it for
                # slow models or deployments that scale up from zero.
                start_to_close_timeout=timedelta(seconds=60),
                # The plugin heartbeats, so a Worker that dies mid-call is
                # noticed in seconds rather than after the full timeout.
                heartbeat_timeout=timedelta(seconds=10),
            ),
            model_provider=model_provider,
        ),
    ],
)

worker = Worker(
    client,
    task_queue="openai-agents-model-providers-task-queue",
    workflows=[OpenAICompatibleAgentWorkflow],
)
await worker.run()
```
<!--SNIPEND-->

The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider:
only the Worker talks to Baseten.

## Write the Workflow

The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one
Worker runs Workflows on Baseten and on other providers at the same time. Any model Baseten serves works, prefixed with
`baseten/`.

<!--SNIPSTART python-openai-agents-openai-compatible-workflow-->
[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py)
```py
@dataclass
class OpenAICompatibleAgentInput:
    prompt: str
    # "<provider>/<model>", such as "together/openai/gpt-oss-120b". The Worker's
    # model provider routes the agent's model calls to that provider.
    model: str

@workflow.defn
class OpenAICompatibleAgentWorkflow:
    @workflow.run
    async def run(self, input: OpenAICompatibleAgentInput) -> str:
        # Tools that run inside the Workflow must be async: the Agents SDK runs
        # sync tools in a thread, which the Workflow sandbox does not allow.
        @function_tool
        async def get_weather(city: str) -> str:
            workflow.logger.debug(f"Getting weather for {city}")
            return f"The weather in {city} is sunny."

        agent = Agent(
            name="Assistant",
            instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.",
            model=input.model,
            tools=[get_weather],
        )

        result = await Runner.run(agent, input.prompt)
        return result.final_output

```
<!--SNIPEND-->

Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings
for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0
forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release,
give such an agent an explicit `model_settings=ModelSettings()`.

Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow
sandbox does not allow. Tools that do I/O belong in Activities; see
[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools).

## Run the sample

Export your Baseten key and start the Worker:

```bash
export BASETEN_API_KEY="your_api_key"
uv run openai_agents/model_providers/run_openai_compatible_worker.py
```

Then start a Workflow on Baseten in a second terminal:

```bash
uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model baseten/openai/gpt-oss-120b
```

Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on
the same Worker.

## Retries, rate limits, and timeouts

The plugin classifies Baseten's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy,
and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as well. Other
4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as does a model
name whose provider has no key on the Worker. Event History records the Activity's result, its attempt count, and its
last failure, not every attempt. Individual failed attempts appear in the Worker's logs and, while the Activity is still
retrying, as the pending Activity's last failure when you describe the Workflow Execution.

Baseten enforces requests-per-minute and tokens-per-minute limits that depend on your
[account tier](https://docs.baseten.co/inference/model-apis/pricing-and-limits) and answers `429` when either is
exceeded; the `x-ratelimit-remaining-requests` and `x-ratelimit-remaining-tokens` headers show which one hit zero. A
`529` means Baseten had no serving capacity for the request even within your limits. Both retry under the Activity's
Retry Policy. Baseten's [error reference](https://docs.baseten.co/inference/errors) asks for exponential backoff and
mentions a `Retry-After` header only as a possibility on `529`, so the Retry Policy's backoff, not the provider, usually
sets the pace. Sustained `429` responses mean the limit itself needs to go up, through Baseten's
[rate limit increase form](https://www.baseten.co/talk-to-us/increase-rate-limits/).

## What works through Baseten

Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the
Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter,
and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat
Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the
[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual.

## Baseten notes

- A dedicated deployment is called at `https://model-<id>.api.baseten.co/environments/production/sync/v1` with the same
  `Authorization: Bearer` key; the legacy `Api-Key` scheme is still accepted. The model name must match what the
  deployment serves. A deployment whose `min_replica` is zero releases its replicas when idle, and the first request
  after that waits for a replica to start; see Baseten's
  [request lifecycle](https://docs.baseten.co/deployment/autoscaling/request-lifecycle) page.
- `GET https://inference.baseten.co/v1/models` lists the Model APIs catalog with context windows, pricing, and supported
  features, as does `baseten model-api list` from the Baseten CLI. Baseten notes that the catalog can report different
  limits from the ones the endpoint enforces.
- gpt-oss-120b reasons by default at `medium` effort. The reasoning arrives in `reasoning_content` on the assistant
  message, and its tokens count as output tokens. Set `ModelSettings(reasoning=Reasoning(effort="low"))` on the agent to
  change the effort; Baseten's [reasoning](https://docs.baseten.co/inference/model-apis/reasoning) page lists the
  accepted values per model.
- Tool calling, structured outputs, and JSON mode are available on every Model API through `tools` and
  `response_format`; see Baseten's [function calling](https://docs.baseten.co/inference/function-calling) and
  [structured outputs](https://docs.baseten.co/inference/structured-outputs) guides. Only one completion per request is
  supported, so leave `n` at its default.
- Baseten reuses cached input tokens across requests that share an `x-session-affinity` header. An agent run resends the
  growing conversation on every turn, so sending the Workflow ID as the session, through
  `ModelSettings(extra_headers=...)`, lets later turns reuse the cache. Baseten recommends at most about 60 requests per
  minute per session.
- Two Baseten features sit outside this pattern: async inference, which delivers results to a webhook and works only on
  dedicated deployments, and server-side web search tools on Model APIs, which are in preview.

## Samples

- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers):
  the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name.
- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history,
  and tracing with the plugin.

---

# Deep Agents integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/deepagents

Temporal's integration with [Deep Agents](https://github.com/langchain-ai/deepagents) makes an existing LangChain
Deep Agent durable by adding one plugin. Build your agent with `create_deep_agent(...)` inside a Workflow, add
`DeepAgentsPlugin` to your Client or Worker, and each LLM call and each I/O tool call becomes a Temporal Activity —
while the agent's control loop runs, and deterministically replays, inside the Workflow.

The code you already wrote against `deepagents` doesn't change. Sub-agents, planning and todo state, the filesystem
middleware, human-in-the-loop interrupts, and `agent.ainvoke(...)` all keep working. On top of that you get
crash durability, automatic retries and timeouts on every model and tool call, resumable human-in-the-loop, and
bounded Workflow history for long-running agents.

> **Pre-release**

Code snippets in this guide are taken from the
[Deep Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/deepagents_plugin). Refer to the
samples for the complete code.

## Prerequisites

- This guide assumes you are already familiar with Deep Agents. If you aren't, refer to the
  [Deep Agents documentation](https://github.com/langchain-ai/deepagents) for more details.
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Ensure you have set up your local development environment by following the
  [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave the
  Temporal development server running if you want to test your code locally.

## Install the plugin

Install the standalone `temporalio-deepagents` package:

```bash
uv add temporalio-deepagents
```

or with pip:

```bash
pip install temporalio-deepagents
```

The package installs the Temporal Python SDK and Deep Agents dependencies. Refer to the
[package requirements](https://github.com/temporalio/ai-integrations/blob/main/python/deepagents/pyproject.toml)
for supported Python and dependency versions.

Import `DeepAgentsPlugin`, `TemporalBackend`, and other integration helpers from `temporalio.deepagents`.

## Hello world

A Deep Agent becomes durable without changing the agent code itself. The Workflow below builds a vanilla
`create_deep_agent(...)` and drives it with `await agent.ainvoke(...)` — exactly the code you would write outside
Temporal. Because `model=` is a bare `"provider:name"` string, the plugin auto-routes the model call through the
`deepagents.invoke_model` Activity, so the LLM call gets Temporal-managed retries and timeouts while the agent's control
loop replays deterministically in the Workflow.

<!--SNIPSTART python-deepagents-hello-world-workflow-->
[deepagents_plugin/hello_world/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/hello_world/workflow.py)
```py
# No `workflow.unsafe.imports_passed_through()` guard is needed: the plugin
# configures the workflow sandbox to pass the deepagents / LangChain import
# tree through, so workflow files import them like any other module.
from deepagents import create_deep_agent
from temporalio import workflow

@workflow.defn
class HelloWorldAgent:
    @workflow.run
    async def run(self, question: str) -> str:
        agent = create_deep_agent(
            model="anthropic:claude-sonnet-4-5",
            system_prompt="You are a helpful assistant. Answer concisely.",
        )
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": question}]}
        )
        return result["messages"][-1].content

```
<!--SNIPEND-->

`DeepAgentsPlugin` is a **client-level** plugin: add it to `Client.connect(...)` and the SDK propagates it to any Worker
built from that Client. Add it on exactly one side. The plugin registers the `deepagents.*` Activities and installs the
LangChain-aware data converter, so the Worker needs no other wiring.

<!--SNIPSTART python-deepagents-hello-world-worker-->
[deepagents_plugin/hello_world/run_worker.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/hello_world/run_worker.py)
```py
import asyncio
import os

from temporalio.client import Client
from temporalio.deepagents import DeepAgentsPlugin
from temporalio.worker import Worker

from deepagents_plugin.hello_world.workflow import HelloWorldAgent

async def main() -> None:
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        plugins=[DeepAgentsPlugin()],
    )

    worker = Worker(
        client,
        task_queue="deepagents-hello-world",
        workflows=[HelloWorldAgent],
    )
    print("Worker started. Ctrl+C to exit.")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

Use `model_activity_options` on the plugin to control the timeout and retry policy for each model call. API keys live on
the Worker via the model provider (LangChain's `init_chat_model` by default), never in Workflow inputs or history —
the Workflow ships only the model name, and the Worker builds the real client. LLM-SDK-side retries are disabled so that
Temporal owns retries and timeouts.

Start the Workflow like any other:

<!--SNIPSTART python-deepagents-hello-world-run-workflow-->
[deepagents_plugin/hello_world/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/hello_world/run_workflow.py)
```py
import asyncio
import os

from temporalio.client import Client

from deepagents_plugin.hello_world.workflow import HelloWorldAgent

async def main() -> None:
    client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"))

    result = await client.execute_workflow(
        HelloWorldAgent.run,
        "What is Temporal in one sentence?",
        id="deepagents-hello-world",
        task_queue="deepagents-hello-world",
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Choose Workflow or Activity execution per tool

A Deep Agent holds its tools in-Workflow. A tool that only reads or writes agent state is pure and belongs there; a tool
that does real I/O must not run in Workflow code. The plugin gives you two explicit ways to move a tool's work into an
Activity:

- `activity_as_tool(my_activity, ...)` surfaces an existing `@activity.defn` function as a Deep Agents tool without
  re-declaring it.
- `tool_as_activity(tool, ...)` wraps a LangChain tool whose body does I/O so its execution runs as a
  `deepagents.invoke_tool` Activity.

An unwrapped, non-builtin tool runs in-Workflow, and the plugin emits a best-effort warning at agent construction
(the warning fires through the plugin's `create_deep_agent` seam, so code paths that bypass that seam construct
without it).
Deep Agents' pure built-ins (`write_todos`, the state-backed file tools) stay in-Workflow by design.

This sample wraps an existing activity and an I/O tool, and builds the agent with `create_temporal_deep_agent`,
which wraps the model name in a durable `TemporalModel` and scopes `activity_options` to this agent's model calls:

<!--SNIPSTART python-deepagents-react-agent-activity-->
[deepagents_plugin/react_agent/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/react_agent/workflow.py)
```py
@activity.defn
async def get_weather(city: str) -> str:
    """Return the current weather for a city."""
    # A real implementation would call a weather API here; this is a stand-in.
    return f"It is sunny and 22C in {city}."

```
<!--SNIPEND-->

<!--SNIPSTART python-deepagents-react-agent-workflow-->
[deepagents_plugin/react_agent/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/react_agent/workflow.py)
```py
@tool
def web_search(query: str) -> str:
    """Search the web for a query and return a short result."""
    # Real I/O (an HTTP call) would go here; wrapped with tool_as_activity so it
    # runs in an activity, not in workflow code.
    return f"Top result for {query!r}: Temporal makes code durable."

@workflow.defn
class ReactAgent:
    @workflow.run
    async def run(self, question: str) -> str:
        weather_tool = activity_as_tool(
            get_weather,
            start_to_close_timeout=timedelta(seconds=30),
        )
        search_tool = tool_as_activity(
            web_search,
            start_to_close_timeout=timedelta(seconds=30),
        )
        agent = create_temporal_deep_agent(
            model="anthropic:claude-sonnet-4-5",
            tools=[weather_tool, search_tool],
            system_prompt=(
                "You are a research assistant. Use the get_weather and "
                "web_search tools when they help answer the question."
            ),
            # Scopes the model-call activity options to this agent — the
            # recommended way to set model timeouts/retries per agent.
            activity_options={"start_to_close_timeout": timedelta(minutes=2)},
        )
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": question}]}
        )
        return result["messages"][-1].content

```
<!--SNIPEND-->

## Durable backends

A Deep Agent's built-in file tools (`write_file`, `read_file`, `ls`, and so on) delegate to a *backend*. The default
`StateBackend` keeps files in agent state — pure Workflow state, replay-safe, and needs no wrapping. A `FilesystemBackend`,
`LocalShellBackend`, or `StoreBackend` touches real resources, which must not happen from Workflow code.

Wrap such a backend with `TemporalBackend(inner, activity_options=...)` so each file or shell operation the agent's tools
invoke becomes a `deepagents.backend_op` Activity instead of running in the Workflow. The agent code is unchanged; only
the backend is wrapped.

<!--SNIPSTART python-deepagents-filesystem-backend-workflow-->
[deepagents_plugin/filesystem_backend/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/filesystem_backend/workflow.py)
```py
from datetime import timedelta

from deepagents import create_deep_agent
from deepagents.backends import FilesystemBackend
from temporalio import workflow
from temporalio.deepagents import TemporalBackend

@workflow.defn
class FilesystemAgent:
    @workflow.run
    async def run(self, root_dir: str, instruction: str) -> str:
        # Wrap the real-I/O backend so every file op runs in an activity.
        backend = TemporalBackend(
            # virtual_mode roots every path the agent uses under root_dir, so the
            # agent's file tools stay sandboxed to this working directory.
            FilesystemBackend(root_dir=root_dir, virtual_mode=True),
            activity_options={"start_to_close_timeout": timedelta(seconds=30)},
        )
        agent = create_deep_agent(
            model="anthropic:claude-sonnet-4-5",
            # TemporalBackend delegates the backend protocol to the wrapped
            # backend at runtime, which the static type can't see through.
            backend=backend,  # type: ignore[arg-type]
            system_prompt=(
                "You are a file-savvy assistant. Use the write_file and "
                "read_file tools to complete the task."
            ),
        )
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": instruction}]}
        )
        return result["messages"][-1].content

```
<!--SNIPEND-->

## Sub-agents

A coordinator built with `create_deep_agent(..., subagents=[...])` delegates to its sub-agents via the built-in `task`
tool. Deep Agents builds each sub-agent as a separate graph, but they inherit the parent's `model` object by default.
Because the plugin makes that model object durable, every sub-agent's model call is automatically durable too — you wire
the plugin once and the whole agent tree is covered, with no per-sub-agent wiring.

<!--SNIPSTART python-deepagents-subagents-workflow-->
[deepagents_plugin/subagents/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/subagents/workflow.py)
```py
from deepagents import create_deep_agent
from temporalio import workflow

@workflow.defn
class SubagentsWorkflow:
    @workflow.run
    async def run(self, question: str) -> str:
        agent = create_deep_agent(
            model="anthropic:claude-sonnet-4-5",
            system_prompt=(
                "You are a research coordinator. Delegate deep investigation to "
                "the researcher sub-agent via the task tool, then synthesize a "
                "final answer."
            ),
            subagents=[
                {
                    "name": "researcher",
                    "description": "Researches a topic in depth and reports findings.",
                    "system_prompt": "You research topics thoroughly and report back.",
                }
            ],
        )
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": question}]}
        )
        return result["messages"][-1].content

```
<!--SNIPEND-->

## Human-in-the-loop

`create_deep_agent(..., interrupt_on=...)` makes the agent pause before a guarded tool runs. With an in-Workflow
`InMemorySaver` checkpointer, LangGraph does *not* raise out of `ainvoke` — it returns the current state with an
`__interrupt__` entry describing the pending approval. Because the agent loop runs in the Workflow, that pause surfaces
directly in Workflow code.

The plugin adds no shim here; the native LangGraph resume protocol is used as-is. The recommended Temporal mapping is to
expose the pending approval via a [Query](/develop/python/workflows/message-passing#queries) and resume via an
[Update](/develop/python/workflows/message-passing#updates) that feeds the human's decision back with
`Command(resume={"decisions": [...]})`. The `InMemorySaver` is replay-safe because its state lives in the Workflow's own
memory, rehydrated by deterministic replay, and the stable Workflow ID is used as the `thread_id`.

<!--SNIPSTART python-deepagents-human-in-the-loop-workflow-->
[deepagents_plugin/human_in_the_loop/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/human_in_the_loop/workflow.py)
```py
from datetime import timedelta

from deepagents import create_deep_agent
from langchain_core.runnables import RunnableConfig
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command
from temporalio import workflow
from temporalio.deepagents import tool_as_activity

@workflow.defn
class HumanInTheLoopAgent:
    def __init__(self) -> None:
        self._pending: str | None = None
        self._decision: str | None = None
        self._resumed = False

    @workflow.run
    async def run(self, city: str) -> str:
        def book_trip(city: str) -> str:
            """Book a trip to a city (requires human approval)."""
            return f"Booked a trip to {city}."

        trip_tool = tool_as_activity(
            book_trip, start_to_close_timeout=timedelta(seconds=30)
        )
        agent = create_deep_agent(
            model="anthropic:claude-sonnet-4-5",
            tools=[trip_tool],
            interrupt_on={"book_trip": True},
            checkpointer=InMemorySaver(),
        )
        config = RunnableConfig(configurable={"thread_id": workflow.info().workflow_id})

        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": f"Book a trip to {city}."}]},
            config=config,
        )
        # LangGraph returns (not raises) the pending approval under __interrupt__.
        pending = result.get("__interrupt__")
        if pending:
            self._pending = str(getattr(pending[0], "value", pending[0]))
            # Block until a client approves/rejects via the `resume` update.
            await workflow.wait_condition(lambda: self._resumed)
            # No longer paused: the query goes back to reporting None.
            self._pending = None
            result = await agent.ainvoke(
                Command(resume={"decisions": [{"type": self._decision}]}),
                config=config,
            )
        return result["messages"][-1].content

    @workflow.query
    def pending_approval(self) -> str | None:
        """Return the pending approval prompt, or ``None`` if not paused."""
        return self._pending

    @workflow.update
    async def resume(self, decision: str) -> None:
        """Resume the paused agent with ``"approve"`` or ``"reject"``."""
        self._decision = decision
        self._resumed = True

    @resume.validator
    def validate_resume(self, decision: str) -> None:
        # Runs before the update is accepted, keeping invalid decisions out of
        # workflow history entirely. Only the decisions this workflow feeds to
        # `Command(resume=...)` are allowed.
        if decision not in ("approve", "reject"):
            raise ValueError('decision must be "approve" or "reject"')

```
<!--SNIPEND-->

## Continue-as-new

Long conversations bloat Workflow history until it hits Temporal's limits. `run_deep_agent(agent, input,
state_snapshot=...)` snapshots state and continues into a fresh run when a turn ends with pending todos and the
server recommends continuing (`workflow.info().is_continue_as_new_suggested()`, the default and recommended mode).
Pass `continue_as_new_after=N` to trigger on a fixed history-event count instead.

- **Carries forward:** the accumulated messages and the model/tool result cache, so an LLM or tool call completed before
  the continue-as-new is *not* re-run afterward.
- **Does not carry forward:** anything else — the snapshot is exactly the messages plus the result cache. Pending
  todos only gate *whether* to continue; the todo list itself (and any in-memory checkpointer state) starts fresh in
  the continued run and is re-derived by the agent from the carried conversation.

Your `@workflow.run` method must accept the carried state — its signature is `run(self, input, state_snapshot=None)`.
On a continue-as-new, `run_deep_agent` re-invokes the method with `args=[input, snapshot]`, so `input` must be passed
straight through, not re-wrapped, or the carried conversation is corrupted.

<!--SNIPSTART python-deepagents-continue-as-new-workflow-->
[deepagents_plugin/continue_as_new/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/continue_as_new/workflow.py)
```py
from typing import Any

from deepagents import create_deep_agent
from temporalio import workflow
from temporalio.deepagents import run_deep_agent

@workflow.defn
class LongResearchAgent:
    @workflow.run
    async def run(
        self, input: dict[str, Any], state_snapshot: dict | None = None
    ) -> str:
        agent = create_deep_agent(
            model="anthropic:claude-sonnet-4-5",
            system_prompt=(
                "You are a research agent. Break large tasks into todos and work "
                "through them until the research is complete."
            ),
        )
        result = await run_deep_agent(
            agent,
            # ``input`` is the messages mapping. Pass it through unchanged: on a
            # continue-as-new, run_deep_agent re-invokes this method with the
            # carried input as its first arg, so re-wrapping it here would nest a
            # dict where a message is expected and corrupt the conversation.
            input,
            # No threshold: continue-as-new fires when the agent still has
            # pending todos and the server suggests continuing — the recommended
            # mode. Pass continue_as_new_after=N to use a fixed event count.
            state_snapshot=state_snapshot,
        )
        return result["messages"][-1].content

```
<!--SNIPEND-->

## Streaming

Constructing the plugin with `DeepAgentsPlugin(streaming_topic="...")` flips model dispatch from
`deepagents.invoke_model` to `deepagents.invoke_model_streaming`: the streaming Activity coalesces chunk batches and
publishes them to a [`WorkflowStream`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/workflow_streams)
topic for external subscribers, while the aggregated final message is still returned to the Workflow. The durable result
is identical to the non-streaming path.

Streaming is async-only, so the Workflow drives an explicit `TemporalModel.astream(...)` and hosts a `WorkflowStream`
so external subscribers can attach by Workflow ID.

<!--SNIPSTART python-deepagents-streaming-workflow-->
[deepagents_plugin/streaming/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/streaming/workflow.py)
```py
from langchain_core.messages import HumanMessage
from temporalio import workflow
from temporalio.contrib.workflow_streams import WorkflowStream
from temporalio.deepagents import TemporalModel

STREAMING_TOPIC = "model-chunks"

@workflow.defn
class StreamingWorkflow:
    def __init__(self) -> None:
        # Host the stream so the publish-Signal handler is registered before the
        # streaming activity (the external publisher) starts publishing.
        self.stream = WorkflowStream()

    @workflow.run
    async def run(self, prompt: str) -> str:
        model = TemporalModel(model="anthropic:claude-sonnet-4-5")
        parts: list[str] = []
        async for chunk in model.astream([HumanMessage(content=prompt)]):
            parts.append(str(chunk.content))
        return "".join(parts)

```
<!--SNIPEND-->

## Compose with an observability plugin

This plugin carries no tracing context of its own. For observability, compose it with an observability plugin such as
[`temporalio.langsmith`](/develop/python/integrations/langsmith) or
`temporalio.contrib.opentelemetry`. Registration order of the two plugins does not matter; the observability plugin
captures the LLM calls that `DeepAgentsPlugin` runs as Activities. The Workflow itself is an ordinary Deep Agent — the tracing
comes entirely from composing the plugins on the Client.

<!--SNIPSTART python-deepagents-langsmith-tracing-workflow-->
[deepagents_plugin/langsmith_tracing/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/langsmith_tracing/workflow.py)
```py
from deepagents import create_deep_agent
from temporalio import workflow

@workflow.defn
class TracedAgent:
    @workflow.run
    async def run(self, question: str) -> str:
        agent = create_deep_agent(
            model="anthropic:claude-sonnet-4-5",
            system_prompt="You are a helpful assistant.",
        )
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": question}]}
        )
        return result["messages"][-1].content

```
<!--SNIPEND-->

For agents built directly as LangGraph graphs rather than a compiled Deep Agent, see the
[LangGraph integration](/develop/python/integrations/langgraph).

## Runtime behavior and limitations

- **Auto-routing is Workflow-scoped.** While a Worker built with this plugin is running, the plugin patches Deep Agents'
  model-resolution seam so a bare `model="provider:name"` string is auto-routed through an Activity, regardless of how you
  imported `create_deep_agent`. This seam is shared by the agent and every sub-agent, and it only rewrites the model when
  resolved *inside a Workflow*, so importing `deepagents` on a plain client or Activity Worker is unaffected, and the
  patched seam is restored when the Worker stops. Pass `TemporalModel("provider:name")` yourself if you would rather be
  explicit.
- **Built-ins stay in-Workflow.** Deep Agents' pure built-in tools (`write_todos`, state-backed file tools) run
  in-Workflow by design and do not need wrapping. Only tools and backends that do real I/O should be moved to Activities.
- **Durable checkpointers are not replay-safe.** The default in-Workflow `InMemorySaver` is rehydrated for free by
  deterministic replay. A durable checkpointer that does its own I/O is not replay-safe from inside a Workflow, and the
  plugin warns if you pass one — prefer the snapshot plus continue-as-new path above.

## Samples

The [Deep Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/deepagents_plugin) demonstrate
all supported patterns, including tool choice, durable backends, sub-agents, human-in-the-loop, continue-as-new,
streaming, and observability composition.

---

# Fireworks AI integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/fireworks-ai

[Fireworks AI](https://fireworks.ai/) serves open-weight models such as gpt-oss, Kimi K2, Qwen, and GLM on serverless
endpoints and on-demand deployments through an API compatible with the OpenAI Chat Completions API.

Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an
Activity. A call to Fireworks AI gets retries with backoff, a timeout, and a durable record in Event History, and the
agent loop around it survives Worker restarts. Because Fireworks AI speaks the OpenAI Chat Completions API, the
integration's model provider hook is all you need: a model provider that maps a model name such as
`fireworks/accounts/fireworks/models/gpt-oss-120b` to Fireworks AI's endpoint. One Worker can serve Fireworks AI
alongside other providers, and each Workflow picks its provider through the model name it gives its agent.

This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the
[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers),
which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker.

## Prerequisites

- This guide assumes you are already familiar with the
  [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Fireworks AI's
  [OpenAI-compatible API](https://docs.fireworks.ai/tools-sdks/openai-compatibility).
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following
  [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal
  development server running.
- Create a [Fireworks AI API key](https://app.fireworks.ai/settings/users/api-keys) and export it as `FIREWORKS_API_KEY`
  in the Worker's environment. The key stays in the Worker process; it is never part of Workflow input or Event History.

## Install

```bash
uv add temporalio-openai-agents
```

The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. Fireworks AI's own
[`fireworks-ai` package](https://docs.fireworks.ai/tools-sdks/python-sdk) is not needed: Fireworks AI documents the
`openai` client as a supported way to call its API.

## Configure the model provider

The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model
with tool calling:

<!--SNIPSTART python-openai-agents-fireworks-provider-->
[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py)
```py
FIREWORKS = OpenAICompatibleProvider(
    name="fireworks",
    base_url="https://api.fireworks.ai/inference/v1",
    api_key_env="FIREWORKS_API_KEY",
    default_model="accounts/fireworks/models/gpt-oss-120b",
)
```
<!--SNIPEND-->

Serverless models on Fireworks AI are named `accounts/fireworks/models/<model>`, and the
[Model Library](https://fireworks.ai/models) shows each model's ID and whether it supports tool calling. The example
here is [gpt-oss-120b](https://fireworks.ai/models/fireworks/gpt-oss-120b), an open-weight reasoning model with tool
calling that Fireworks AI serves serverless; a Workflow names it `fireworks/accounts/fireworks/models/gpt-oss-120b`. An
[on-demand deployment](https://docs.fireworks.ai/guides/ondemand-deployments) uses the same base URL; pass the
deployment's resource name, `accounts/<account>/deployments/<deployment-id>`, as the model instead, behind the same
`fireworks/` prefix.

A `ModelProvider` routes a `<provider>/<model>` name to the matching endpoint and returns the Chat Completions model
that the Agents SDK calls:

<!--SNIPSTART python-openai-agents-openai-compatible-model-provider-->
[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py)
```py
class OpenAICompatibleModelProvider(ModelProvider):
    """Routes ``<provider>/<model>`` names to the matching OpenAI-compatible endpoint.

    One Worker serves every provider whose API key is set, so Workflows on
    different providers run side by side on the same Task Queue. Client
    retries are off: the plugin runs each model call as a Temporal Activity,
    and the Activity's Retry Policy owns the retries.
    """

    def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None:
        self._providers = {provider.name: provider for provider in providers}
        self._clients: dict[str, AsyncOpenAI] = {}
        for provider in self._providers.values():
            api_key = os.environ.get(provider.api_key_env)
            if api_key:
                self._clients[provider.name] = AsyncOpenAI(
                    base_url=provider.base_url, api_key=api_key, max_retries=0
                )

    @property
    def configured(self) -> list[str]:
        """Names of the providers whose API key is set."""
        return sorted(self._clients)

    def get_model(self, model_name: str | None) -> Model:
        provider_name, _, model = (model_name or "").partition("/")
        provider = self._providers.get(provider_name)
        if provider is None or not model:
            # A mistake in the Workflow's input: fail the Activity without retrying.
            raise ApplicationError(
                f"Model name {model_name!r} must be <provider>/<model> with a "
                f"provider in {sorted(self._providers)}",
                non_retryable=True,
            )
        client = self._clients.get(provider_name)
        if client is None:
            raise ApplicationError(
                f"Set {provider.api_key_env} and restart the Worker to use {provider_name}",
                non_retryable=True,
            )
        # Chat Completions is the API all of these providers serve. The Agents
        # SDK defaults to the Responses API, which not every provider offers.
        return OpenAIChatCompletionsModel(model=model, openai_client=client)

```
<!--SNIPEND-->

Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name
for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off
(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a
hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents
SDK's default Responses API model, which Fireworks AI also offers, because Chat Completions is the one API every
provider in the sample serves.

## Configure the Worker

Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A
custom model provider must set a model Activity timeout; the sample uses 60 seconds. Fireworks AI answers a
scaled-to-zero on-demand deployment with an immediate 503 rather than a slow first response, so the Retry Policy covers
scale-up and the timeout only has to cover your longest generation. The Heartbeat Timeout lets Temporal notice a Worker
that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its own.

<!--SNIPSTART python-openai-agents-openai-compatible-worker-->
[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py)
```py
# One provider for every endpoint whose API key is set. The model name on
# each Workflow's agent selects the endpoint.
model_provider = OpenAICompatibleModelProvider(PROVIDERS.values())
if not model_provider.configured:
    env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values())
    sys.exit(f"No provider API key is set; export at least one of {env_vars}.")
print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True)

client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                # A custom model provider must set a timeout. Raise it for
                # slow models or deployments that scale up from zero.
                start_to_close_timeout=timedelta(seconds=60),
                # The plugin heartbeats, so a Worker that dies mid-call is
                # noticed in seconds rather than after the full timeout.
                heartbeat_timeout=timedelta(seconds=10),
            ),
            model_provider=model_provider,
        ),
    ],
)

worker = Worker(
    client,
    task_queue="openai-agents-model-providers-task-queue",
    workflows=[OpenAICompatibleAgentWorkflow],
)
await worker.run()
```
<!--SNIPEND-->

The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider:
only the Worker talks to Fireworks AI.

## Write the Workflow

The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one
Worker runs Workflows on Fireworks AI and on other providers at the same time. Any model Fireworks AI serves works,
prefixed with `fireworks/`.

<!--SNIPSTART python-openai-agents-openai-compatible-workflow-->
[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py)
```py
@dataclass
class OpenAICompatibleAgentInput:
    prompt: str
    # "<provider>/<model>", such as "together/openai/gpt-oss-120b". The Worker's
    # model provider routes the agent's model calls to that provider.
    model: str

@workflow.defn
class OpenAICompatibleAgentWorkflow:
    @workflow.run
    async def run(self, input: OpenAICompatibleAgentInput) -> str:
        # Tools that run inside the Workflow must be async: the Agents SDK runs
        # sync tools in a thread, which the Workflow sandbox does not allow.
        @function_tool
        async def get_weather(city: str) -> str:
            workflow.logger.debug(f"Getting weather for {city}")
            return f"The weather in {city} is sunny."

        agent = Agent(
            name="Assistant",
            instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.",
            model=input.model,
            tools=[get_weather],
        )

        result = await Runner.run(agent, input.prompt)
        return result.final_output

```
<!--SNIPEND-->

Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings
for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0
forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release,
give such an agent an explicit `model_settings=ModelSettings()`.

Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow
sandbox does not allow. Tools that do I/O belong in Activities; see
[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools).

## Run the sample

Export your Fireworks AI key and start the Worker:

```bash
export FIREWORKS_API_KEY="your_api_key"
uv run openai_agents/model_providers/run_openai_compatible_worker.py
```

Then start a Workflow on Fireworks AI in a second terminal:

```bash
uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model fireworks/accounts/fireworks/models/gpt-oss-120b
```

Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on
the same Worker.

## Retries, rate limits, and timeouts

The plugin classifies Fireworks AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry
Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as
well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as
does a model name whose provider has no key on the Worker. Event History records the Activity's result, its attempt
count, and its last failure, not every attempt. Individual failed attempts appear in the Worker's logs and, while the
Activity is still retrying, as the pending Activity's last failure when you describe the Workflow Execution.

On serverless, Fireworks AI enforces
[adaptive token-per-minute limits](https://docs.fireworks.ai/serverless/rate-limits) per model that grow with steady
traffic and shrink when it stops. A burst beyond them returns 429, and Fireworks AI asks clients to honor `Retry-After`
when it is present. Under high load Fireworks AI can also shed traffic with `503 Service Overloaded`, which the plugin
retries like any 5xx; the `priority` service tier lowers that chance.
[On-demand deployments](https://docs.fireworks.ai/guides/inference-error-codes) have no token-per-minute limit, so a 429
there means the deployment is saturated: add replicas or lower concurrency rather than wait. Both paths count against
the account-wide requests-per-minute cap described on the
[account quotas page](https://docs.fireworks.ai/guides/quotas_usage/account-quotas), which also sets the spending tiers
that raise the serverless ceilings.

## What works through Fireworks AI

Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the
Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter,
and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat
Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the
[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual.

## Fireworks AI notes

- Create an on-demand deployment with `firectl deployment create accounts/fireworks/models/gpt-oss-120b ...` and pass
  the deployment name it returns as the model. By default a deployment scales to zero after an hour without traffic and
  answers the next request with an immediate 503 while it scales back up, which the Activity's Retry Policy absorbs; a
  deployment with zero minimum replicas is deleted after seven days without traffic. See the
  [deployments quickstart](https://docs.fireworks.ai/getting-started/ondemand-quickstart).
- Fireworks AI updates and retires serverless models over time and gives
  [at least two weeks' notice](https://docs.fireworks.ai/faq-new/deployment-infrastructure/do-you-provide-notice-before-removing-model-availability)
  before removing one. For a version that must not change under a long-running Workflow, pin it on an on-demand
  deployment.
- An agent's `output_type` is sent as a JSON schema in `response_format`, which Fireworks AI
  [supports](https://docs.fireworks.ai/structured-responses/structured-response-formatting). Fireworks AI turns off
  reasoning output while it enforces a schema, and turns JSON mode on by itself for tool calls.
- Reasoning models return their thinking in a `reasoning_content` field, and for models that interleave thinking with
  tool calls Fireworks AI
  [expects each earlier assistant turn's `reasoning_content` back](https://docs.fireworks.ai/guides/reasoning) in the
  next request. The Agents SDK replays it only for DeepSeek models by default; pass `should_replay_reasoning_content` to
  `OpenAIChatCompletionsModel` to replay it for the model you use.
- [Prompt caching](https://docs.fireworks.ai/guides/prompt-caching) is on by default, so a retried attempt that re-sends
  the same prompt pays the discounted cached rate for the prefix. On serverless the cache lives in one replica, so hits
  need a session identifier, which you can send as an `x-session-affinity` header through
  `ModelSettings(extra_headers=...)`; dedicated deployments share their cache across requests.
- Fireworks AI also serves the [Responses API](https://docs.fireworks.ai/guides/response-api) at the same base URL, with
  stored conversation state and server-executed tools of its own: remote MCP servers, web search, and a Python tool.
  Build an `OpenAIResponsesModel` instead of `OpenAIChatCompletionsModel` to use it. The sample does not, and those
  tools were not exercised with the Agents SDK's hosted tool types.

## Samples

- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers):
  the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name.
- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history,
  and tracing with the plugin.

---

# Google ADK integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/google-adk

Temporal's Google ADK integration lets you run [Google ADK](https://google.github.io/adk-docs/) agents inside Temporal
Workflows, so an agent keeps its place across Worker restarts, deploys, and transient failures.

Temporal gives your agent code [Durable Execution](/temporal#durable-execution). The Google ADK
gives you the agent itself: model calls, tools, multi-agent handoffs, and MCP. The integration connects the two so that
you write an ordinary ADK agent and run it as a Workflow, without managing sessions, a database, or your own retry logic.

The `GoogleAdkPlugin` is what ties them together. It runs each model call and tool call as a Temporal Activity,
configures the payload converter that serializes ADK objects, and makes ADK's runtime deterministic so the Workflow can
replay. You add the same plugin to your Client and your Worker, and the rest of your agent code stays standard ADK.

> **Pre-release**

## Prerequisites

- This guide assumes you are already familiar with the Google ADK. If you aren't, refer to the
  [Google ADK documentation](https://google.github.io/adk-docs/) for more details.
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Ensure you have set up your local development environment by following the
  [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave
  the Temporal Development Server running if you want to test your code locally.

## Install

Install the standalone `temporalio-google-adk` package:

```bash
uv add temporalio-google-adk
```

If you use pip:

```bash
pip install temporalio-google-adk
```

The package installs the Temporal Python SDK and Google ADK dependencies. Refer to the
[package requirements](https://github.com/temporalio/ai-integrations/blob/main/python/google_adk/pyproject.toml)
for supported Python and dependency versions.

Import `GoogleAdkPlugin` and `TemporalModel` from `temporalio.google_adk`. Import Workflow helpers such as `activity_as_tool`
from `temporalio.google_adk.workflow`.

## Define an agent in a Workflow

Write your agent the way you normally would with the ADK. The one Temporal-specific piece is `TemporalModel`, which you
pass in place of a model name. It runs each model call as an `invoke_model` Activity, so every turn is durable and shows
up in the Event history.

[google_adk_agents/basic/workflows/hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/basic/workflows/hello_world_workflow.py)
```py
@workflow.defn
class HelloWorldAgentWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        # TemporalModel runs each model call as an `invoke_model` activity.
        agent = Agent(
            name="hello_world_agent",
            model=TemporalModel("gemini-2.5-flash"),
            instruction="You only respond in haikus.",
        )

        # The plugin points ADK's session-id generation at workflow.uuid4(), so
        # creating a session here is replay-safe.
        runner = InMemoryRunner(agent=agent, app_name="hello_world_app")
        session = await runner.session_service.create_session(
            app_name="hello_world_app", user_id="user"
        )

        final_text = ""
        async with Aclosing(
            runner.run_async(
                user_id="user",
                session_id=session.id,
                new_message=types.Content(role="user", parts=[types.Part(text=prompt)]),
            )
        ) as agen:
            async for event in agen:
                if event.content and event.content.parts:
                    for part in event.content.parts:
                        if part.text:
                            final_text = part.text

        return final_text

```

Everything other than `TemporalModel` is normal ADK. You build an `Agent`, drive it with a runner, and read the events
it produces.

## Add the plugin to your Worker and Client

Build one `GoogleAdkPlugin` and pass the same instance to both the Client and the Worker. The Client side links the code
that starts a Workflow to the Workflow itself, and the Worker side runs the agent.

[google_adk_agents/basic/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/basic/run_worker.py)
```py
plugin = GoogleAdkPlugin()

client = await Client.connect(
    os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin]
)

worker = Worker(
    client,
    task_queue="google-adk-agents-basic",
    workflows=[HelloWorldAgentWorkflow],
    plugins=[plugin],
)
await worker.run()
```

Model calls run as Activities on the Worker, so the Worker process is the one that needs your model provider
credentials. The samples use Gemini, which reads its key from the `GOOGLE_API_KEY` environment variable.

```bash
export GOOGLE_API_KEY="your-api-key"
python run_worker.py
```

The ADK supports other model providers as well, for example non-Gemini models through LiteLLM. Change the model name you
pass to `TemporalModel` to use one.

Start the Workflow the way you would start any other Temporal Workflow. Use a Client that has the plugin so the starting
code is linked to the run.

[google_adk_agents/basic/run_hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/basic/run_hello_world_workflow.py)
```py
client = await Client.connect(
    os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
    plugins=[GoogleAdkPlugin()],
)

result = await client.execute_workflow(
    HelloWorldAgentWorkflow.run,
    "Tell me about recursion in programming.",
    id="google-adk-agents-basic-workflow-id",
    task_queue="google-adk-agents-basic",
)
print(f"Result: {result}")
```

## Run tools as Activities

To expose a Temporal Activity as a tool the agent can call, wrap it with `activity_as_tool`. When the model calls the tool,
it runs as its own Activity, so it gets its own retries and timeouts and appears in the Event history.

[google_adk_agents/tools/workflows/weather_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/tools/workflows/weather_workflow.py)
```py
@workflow.defn
class WeatherAgentWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        # activity_as_tool runs the tool call as a real Temporal activity, so it's
        # retryable and shows up in history.
        weather_tool = temporalio.google_adk.workflow.activity_as_tool(
            get_weather, start_to_close_timeout=timedelta(seconds=60)
        )

        agent = Agent(
            name="weather_agent",
            model=TemporalModel("gemini-2.5-flash"),
            instruction="Use the get_weather tool to answer weather questions.",
            tools=[weather_tool],
        )
```

`get_weather` is a plain Temporal Activity. Register it on the Worker alongside the Workflow.

[google_adk_agents/tools/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/tools/run_worker.py)
```py
plugin = GoogleAdkPlugin()

client = await Client.connect(
    os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin]
)

worker = Worker(
    client,
    task_queue="google-adk-agents-tools",
    workflows=[WeatherAgentWorkflow],
    activities=[get_weather],
    plugins=[plugin],
)
await worker.run()
```

## Coordinate multiple agents

The ADK's multi-agent patterns work inside a Workflow. Give each agent its own `TemporalModel`, and pass an
`ActivityConfig` when you want to name its model turns or set per-agent timeouts. A coordinator delegates to its
`sub_agents` using the ADK's built-in handoff.

[google_adk_agents/agent_patterns/workflows/multi_agent_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/agent_patterns/workflows/multi_agent_workflow.py)
```py
@workflow.defn
class MultiAgentWorkflow:
    @workflow.run
    async def run(self, topic: str) -> str:
        session_service = InMemorySessionService()
        session = await session_service.create_session(
            app_name="multi_agent_app", user_id="user"
        )

        # The ActivityConfig summary makes each model turn a named activity in
        # history.
        researcher = LlmAgent(
            name="researcher",
            model=TemporalModel(
                "gemini-2.5-flash",
                activity_config=ActivityConfig(summary="Researcher Agent"),
            ),
            instruction="You are a researcher. Find information about the topic.",
        )

        writer = LlmAgent(
            name="writer",
            model=TemporalModel(
                "gemini-2.5-flash",
                activity_config=ActivityConfig(summary="Writer Agent"),
            ),
            instruction="You are a poet. Write a haiku based on the research.",
        )

        # ADK's transfer_to_agent handoff runs durably here.
        coordinator = LlmAgent(
            name="coordinator",
            model=TemporalModel(
                "gemini-2.5-flash",
                activity_config=ActivityConfig(
                    start_to_close_timeout=timedelta(seconds=30),
                    summary="Coordinator Agent",
                ),
            ),
            instruction="You are a coordinator. Delegate to researcher then writer.",
            sub_agents=[researcher, writer],
        )
```

The `summary` you set on each `ActivityConfig` becomes the Activity name in the Event history, which makes it easy to see
which agent ran and when.

## Use MCP tool servers

Install the package's `mcp` extra:

```bash
uv add "temporalio-google-adk[mcp]"
```

To give an agent tools from an [MCP](https://modelcontextprotocol.io/) server, use `TemporalMcpToolSet`. It runs the
server's list-tools and call-tool operations as Activities because connecting to an MCP server is external I/O. You define a factory function for the toolset and register it with the plugin
through a `TemporalMcpToolSetProvider`.

[google_adk_agents/mcp/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/mcp/run_worker.py)
```py
plugin = GoogleAdkPlugin(
    toolset_providers=[TemporalMcpToolSetProvider("echo", echo_toolset)]
)
```

In the Workflow, give the agent a `TemporalMcpToolSet` with the same name. The `not_in_workflow_toolset` factory lets you
run the same agent locally, outside Temporal, by connecting to the MCP server directly.

[google_adk_agents/mcp/workflows/echo_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/mcp/workflows/echo_workflow.py)
```py
@workflow.defn
class EchoMcpWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        # TemporalMcpToolSet runs the MCP server's list-tools and call-tool
        # calls as activities.
        agent = Agent(
            name="echo_agent",
            model=TemporalModel("gemini-2.5-flash"),
            instruction="Use the echo tool to echo back the user's message.",
            tools=[TemporalMcpToolSet("echo", not_in_workflow_toolset=echo_toolset)],
        )
```

## Stream model output

`TemporalModel` can stream a model's output as it is generated, using a Workflow stream from
`temporalio.contrib.workflow_streams`. Give the model a `streaming_topic`, host a `WorkflowStream` on the Workflow, and
the streaming model call publishes each chunk to the topic as it arrives. A Client subscribes to the topic to read
chunks while the Workflow is still running.

> **⚠️ Caution:**
>
> Streaming is an early, experimental part of this integration and is more likely to change than the rest of the API.
> Refer to the [streaming sample](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/streaming) for a
> complete, working example.
>

[google_adk_agents/streaming/workflows/streaming_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/streaming/workflows/streaming_workflow.py)
```py
@workflow.defn
class StreamingAgentWorkflow:
    @workflow.init
    def __init__(self, prompt: str) -> None:
        # The streaming activity publishes LlmResponse chunks to this stream as
        # they come back from the model.
        self.stream = WorkflowStream()

    @workflow.run
    async def run(self, prompt: str) -> str:
        # streaming_mode=SSE routes the call through the invoke_model_streaming
        # activity.
        model = TemporalModel("gemini-2.5-flash", streaming_topic="responses")
        agent = Agent(
            name="streaming_agent",
            model=model,
            instruction="You are a helpful assistant.",
        )

        runner = InMemoryRunner(agent=agent, app_name="streaming_app")
        session = await runner.session_service.create_session(
            app_name="streaming_app", user_id="user"
        )

        final_text = ""
        async for event in runner.run_async(
            user_id="user",
            session_id=session.id,
            new_message=types.Content(role="user", parts=[types.Part(text=prompt)]),
            run_config=RunConfig(streaming_mode=StreamingMode.SSE),
        ):
            if event.content and event.content.parts:
                for part in event.content.parts:
                    if part.text:
                        final_text = part.text

        return final_text
```

## Samples

The [Google ADK samples](https://github.com/temporalio/samples-python/tree/main/google_adk_agents) cover each pattern in
a self-contained, runnable scenario:

- [`basic`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/basic): a single agent with `TemporalModel` and one model call.
- [`tools`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/tools): a Temporal Activity exposed as a tool with `activity_as_tool`.
- [`agent_patterns`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/agent_patterns): a coordinator agent that delegates to sub-agents.
- [`mcp`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/mcp): MCP tools run as Activities, with a self-contained echo server.
- [`streaming`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/streaming): token streaming with `WorkflowStream`.

---

# Google GenAI integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/google-genai

Temporal's Google GenAI integration lets you call [Gemini](https://ai.google.dev/gemini-api/docs) models from inside
Temporal Workflows, so a sequence of model calls keeps its place across Worker restarts, deploys, and transient failures.

Temporal gives your code [Durable Execution](/temporal#durable-execution). The
[Google Gen AI SDK](https://googleapis.github.io/python-genai/) gives you the model API: content generation, automatic
function calling, chat sessions, structured output, files, and MCP. The integration connects the two so that you write
ordinary Gemini SDK code and run it as a Workflow, without writing your own retry loop or checkpointing.

`GoogleGenAIPlugin` is what ties them together. You build a `genai.Client` with your credentials on the Worker and hand
it to the plugin. Inside the Workflow you construct a `TemporalAsyncClient`, which has the same shape as the SDK's async
client but routes every API call through a Temporal Activity. Each call gets its own timeout, retry policy, and Event
History entry, and your credentials stay on the Worker.

> **Public Preview**

Code snippets in this guide come from the
[Google GenAI plugin samples](https://github.com/temporalio/samples-python/tree/main/google_genai). Refer to the
samples for the complete code.

## Prerequisites

- This guide assumes you are already familiar with the Google Gen AI SDK. If you aren't, refer to the
  [Gemini API documentation](https://ai.google.dev/gemini-api/docs) for more details.
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Ensure you have set up your local development environment by following the
  [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave
  the Temporal development server running if you want to test your code locally.

## Install the plugin

Install the standalone `temporalio-google-genai` package:

```bash
uv add temporalio-google-genai
```

If you use pip:

```bash
pip install temporalio-google-genai
```

The package installs the Temporal Python SDK and Google GenAI dependencies. Refer to the
[package requirements](https://github.com/temporalio/ai-integrations/blob/main/python/google_genai/pyproject.toml)
for supported Python and dependency versions.

Import `GoogleGenAIPlugin`, `TemporalAsyncClient`, and `TemporalMcpClientSession` from `temporalio.google_genai`.
For [MCP tool servers](#use-mcp-tool-servers), install the package's `mcp` extra.

## Call a model from a Workflow

Construct a `TemporalAsyncClient` in your Workflow and call it the way you would call `genai.Client.aio`. The client
takes no credentials — it resolves each call to an Activity that runs on the Worker.

[google_genai/hello_world/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/hello_world/workflow.py)
```py
@workflow.defn
class HelloWorldWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        client = TemporalAsyncClient()
        response = await client.models.generate_content(
            model="gemini-2.5-flash",
            contents=prompt,
        )
        return response.text or ""

```

On the Worker, build a real `genai.Client` with your credentials, wrap it in a `GoogleGenAIPlugin`, and pass the plugin
to `Client.connect`. A Worker created from that Temporal Client picks up the plugin, which registers the Activity that
makes the API calls and swaps in the Pydantic payload converter that serializes Gemini types.

[google_genai/hello_world/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/hello_world/run_worker.py)
```py
import asyncio
import os

from google import genai
from temporalio.client import Client
from temporalio.google_genai import GoogleGenAIPlugin
from temporalio.worker import Worker

from google_genai.hello_world.workflow import HelloWorldWorkflow

async def main() -> None:
    # The real genai.Client (with credentials) lives only on the worker.
    genai_client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])
    plugin = GoogleGenAIPlugin(genai_client)

    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        plugins=[plugin],
    )

    worker = Worker(
        client,
        task_queue="google-genai-hello-world",
        workflows=[HelloWorldWorkflow],
    )
    print("Worker started. Ctrl+C to exit.")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

Because API calls run as Activities, the Worker process is the one that needs credentials. The samples use the Gemini
Developer API, which reads its key from the `GOOGLE_API_KEY` environment variable.

```bash
export GOOGLE_API_KEY="your-api-key"
uv run google_genai/hello_world/run_worker.py
```

Start the Workflow the way you start any other Temporal Workflow. The starting Client does not need the plugin.

[google_genai/hello_world/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/hello_world/run_workflow.py)
```py
import asyncio
import os

from temporalio.client import Client

from google_genai.hello_world.workflow import HelloWorldWorkflow

async def main() -> None:
    client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"))

    result = await client.execute_workflow(
        HelloWorldWorkflow.run,
        "Write a haiku about durable execution.",
        id="google-genai-hello-world",
        task_queue="google-genai-hello-world",
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```

## Run tools as Activities

The Gemini SDK's automatic function calling loop runs inside the Workflow, so you don't write the tool loop yourself.
A tool can be either of the following:

- A Temporal Activity wrapped with `activity_as_tool`. The model's call to it runs as its own Activity, with its own
  timeout and retries, and appears in the Event History. Use this for anything that does I/O or is otherwise
  non-deterministic.
- A plain Workflow method passed directly. It runs in the Workflow with no Activity dispatch, so it must be
  [deterministic](/develop/python/workflows/basics#workflow-logic-requirements).

This Workflow passes one of each on a single call.

[google_genai/tools/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/tools/workflow.py)
```py
@workflow.defn
class ToolsWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        client = TemporalAsyncClient()
        response = await client.models.generate_content(
            model="gemini-2.5-flash",
            contents=prompt,
            config=types.GenerateContentConfig(
                tools=[
                    activity_as_tool(
                        get_weather,
                        activity_config=ActivityConfig(
                            start_to_close_timeout=timedelta(seconds=30),
                        ),
                    ),
                    self.recommend_thing_to_do,
                ],
            ),
        )
        return response.text or ""

    async def recommend_thing_to_do(self, weather: str) -> str:
        """Recommend something to do given a weather description."""
        if "sunny" in weather.lower():
            return "Go for a hike."
        return "Visit a museum."

```

`activity_as_tool` keeps the wrapped function's name, docstring, and type signature, which is what the model uses to
decide when to call it. Its `activity_config` must set `start_to_close_timeout` or `schedule_to_close_timeout`; there is
no default, and the tool call fails without one.

Register the Activity on the Worker alongside the Workflow.

[google_genai/tools/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/tools/run_worker.py)
```py
worker = Worker(
    client,
    task_queue="google-genai-tools",
    workflows=[ToolsWorkflow],
    activities=[get_weather],
)
```

## Hold a multi-turn conversation

`client.chats` works inside a Workflow. The chat session keeps its history in Workflow state, and each `send_message`
call runs as its own Activity, so a conversation that spans hours or days survives a Worker restart.

[google_genai/chat/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/chat/workflow.py)
```py
@workflow.defn
class ChatWorkflow:
    @workflow.run
    async def run(self, prompts: list[str]) -> list[str]:
        client = TemporalAsyncClient()
        chat = client.chats.create(model="gemini-2.5-flash")
        replies: list[str] = []
        for prompt in prompts:
            response = await chat.send_message(prompt)
            replies.append(response.text or "")
        return replies

```

To drive the turns from the outside instead of from a list, take each prompt as a
[Signal](/develop/python/workflows/message-passing#signals) and wait on it with `workflow.wait_condition`.

## Return structured output

The plugin installs Temporal's Pydantic payload converter, so a Pydantic model passes through Temporal payloads
unchanged. Pass the model as `response_schema` and read the parsed result from `response.parsed`.

[google_genai/structured_output/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/structured_output/workflow.py)
```py
class Recipe(BaseModel):
    name: str
    ingredients: list[str]
    steps: list[str]

@workflow.defn
class StructuredOutputWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> Recipe:
        client = TemporalAsyncClient()
        response = await client.models.generate_content(
            model="gemini-2.5-flash",
            contents=prompt,
            config=types.GenerateContentConfig(
                response_mime_type="application/json",
                response_schema=Recipe,
            ),
        )
        recipe = response.parsed
        if not isinstance(recipe, Recipe):
            # ``parsed`` is None when the model returns malformed JSON. Fail the
            # workflow with an ApplicationError rather than asserting: an
            # assertion is a workflow task failure, which Temporal retries
            # forever, so the run would hang instead of failing visibly.
            raise ApplicationError(
                f"Gemini did not return a valid Recipe: {response.text!r}",
                non_retryable=True,
            )
        return recipe

```

## Use MCP tool servers

Install the package's `mcp` extra:

```bash
uv add "temporalio-google-genai[mcp]"
```

To give a model tools from an [MCP](https://modelcontextprotocol.io/) server, register the server on the Worker and
reference it by name in the Workflow. Connecting to an MCP server is external I/O, so the plugin holds the connection on
the Worker and runs `list_tools` and `call_tool` as Activities against it.

Register each server with a factory that yields a connected, initialized `mcp.ClientSession`.

[google_genai/mcp/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/mcp/run_worker.py)
```py
@asynccontextmanager
async def echo_session() -> AsyncIterator[ClientSession]:
    """Yield a connected, initialized session to the stdio echo MCP server."""
    params = StdioServerParameters(command=sys.executable, args=[ECHO_SERVER])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            yield session

async def main() -> None:
    genai_client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])
    plugin = GoogleGenAIPlugin(genai_client, mcp_servers={"echo": echo_session})
```

In the Workflow, pass a `TemporalMcpClientSession` with the same name in the `tools` list. Automatic function calling
discovers and calls the server's tools from there.

[google_genai/mcp/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/mcp/workflow.py)
```py
@workflow.defn
class McpWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        client = TemporalAsyncClient()
        session = TemporalMcpClientSession(
            "echo",
            cache_tools=True,
            activity_config=ActivityConfig(
                start_to_close_timeout=timedelta(seconds=30),
            ),
        )
        response = await client.models.generate_content(
            model="gemini-2.5-flash",
            contents=prompt,
            config=types.GenerateContentConfig(tools=[session]),
        )
        return response.text or ""

```

`cache_tools=True` reuses the first `list_tools` result for the rest of the run instead of listing tools before every
call. The Worker keeps each MCP connection open between uses and disconnects it after five minutes idle; change that
with `mcp_connection_idle_timeout` on the plugin.

Server-side MCP needs no wiring. Vertex AI's `Tool(mcp_servers=[McpServer(...)])` and the Interactions API's MCP steps
run on Google's backend, so they pass through as ordinary request and response data.

## Stream model output

`generate_content_stream` can forward chunks to an external subscriber while the Workflow is still running. Set
`streaming_topic` on the client and host a `WorkflowStream` in the Workflow's `@workflow.init`; each chunk is published
to that topic as it arrives. The Workflow's own iteration over the stream is unchanged.

[google_genai/streaming/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/streaming/workflow.py)
```py
@workflow.defn
class StreamingWorkflow:
    @workflow.init
    def __init__(self, prompt: str) -> None:
        # Hosting a WorkflowStream is required when streaming_topic is set.
        self.stream = WorkflowStream()
        self._done = False

    @workflow.run
    async def run(self, prompt: str) -> str:
        client = TemporalAsyncClient(streaming_topic="gemini")
        chunks: list[str] = []
        async for chunk in await client.models.generate_content_stream(
            model="gemini-2.5-flash",
            contents=prompt,
        ):
            chunks.append(chunk.text or "")
        # Bound the wait: if the subscriber dies without signaling, complete
        # anyway instead of waiting forever.
        try:
            await workflow.wait_condition(lambda: self._done, timeout=FINISH_TIMEOUT)
        except asyncio.TimeoutError:
            workflow.logger.warning(
                "No finish signal after %s; completing without a subscriber.",
                FINISH_TIMEOUT,
            )
        return "".join(chunks)

    @workflow.signal
    def finish(self) -> None:
        self._done = True

```

A consumer subscribes to the topic with `WorkflowStreamClient`. The published chunks are Pydantic
`GenerateContentResponse` objects, so the subscribing Client needs `pydantic_data_converter`.

[google_genai/streaming/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/streaming/run_workflow.py)
```py
async def consume(client: Client, workflow_id: str) -> None:
    """Subscribe to the "gemini" topic and print chunks as the model produces them."""
    stream = WorkflowStreamClient.create(client, workflow_id)
    async for item in stream.subscribe(
        ["gemini"],
        from_offset=0,
        result_type=types.GenerateContentResponse,
        poll_cooldown=timedelta(milliseconds=50),
    ):
        chunk: types.GenerateContentResponse = item.data
        if chunk.text:
            print(chunk.text, end="", flush=True)
        if chunk.candidates and chunk.candidates[0].finish_reason:
            print()
            return

async def main() -> None:
    # The stream publishes Pydantic GenerateContentResponse chunks, so the
    # consumer needs the Pydantic data converter to decode them.
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        data_converter=pydantic_data_converter,
    )
```

The streaming Activity batches published chunks and flushes them every 100 milliseconds by default. Adjust that with
`streaming_batch_interval`. Delivery is at-least-once per Activity attempt: if the streaming Activity retries, the model
call re-runs and republishes, so subscribers should tolerate duplicates and treat the Workflow result as the source of
truth.

## Upload files and reference them in a prompt

`client.files` runs as Activities too, so the file is read on the Worker rather than in the Workflow. Upload it, then
pass the returned handle in `contents`.

[google_genai/files/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/files/workflow.py)
```py
@workflow.defn
class FilesWorkflow:
    @workflow.run
    async def run(self, file_path: str, prompt: str) -> str:
        client = TemporalAsyncClient()
        uploaded = await client.files.upload(
            file=file_path,
            config=types.UploadFileConfig(mime_type="text/plain"),
        )
        contents = cast(types.ContentListUnion, [prompt, uploaded])
        response = await client.models.generate_content(
            model="gemini-2.5-flash",
            contents=contents,
        )
        return response.text or ""

```

The file path resolves on the Worker, so the Worker needs access to it. Operations that require separate Google Cloud
credentials, such as `files.register_files`, use the `extra_credentials` you pass to the plugin.

## Use the Interactions API and managed agents

`client.interactions` and `client.agents` are server-managed: the state lives on Google's backend and each operation
runs as its own Activity.

[google_genai/interactions/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/interactions/workflow.py)
```py
@workflow.defn
class InteractionsWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> dict[str, Any]:
        client = TemporalAsyncClient()

        # create/get return either an Interaction or a streaming response; without
        # stream=True the result is always an Interaction.
        interaction: Any = await client.interactions.create(
            model="gemini-2.5-flash",
            input=prompt,
        )
        fetched: Any = await client.interactions.get(interaction.id)
        await client.interactions.delete(interaction.id)

        return {"id": interaction.id, "status": str(fetched.status)}

```

Two limits apply to the Interactions API:

- It has no automatic function calling. Declare tools as `{"type": "function", ...}` dicts and drive the tool loop
  yourself, running each call with `workflow.execute_activity` or an `activity_as_tool` callable.
- Streamed interactions are batched. The Activity drains the server-sent event stream and the Workflow iterates the
  collected events.

`client.webhooks` is not supported in Workflows.

## Run against Vertex AI

To use Vertex AI instead of the Gemini Developer API, set `vertexai=True` on both sides. In the Workflow, pass the
project and location as Workflow arguments rather than reading environment variables, which keeps the Workflow
deterministic.

[google_genai/vertex_ai/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/vertex_ai/workflow.py)
```py
@workflow.defn
class VertexAIWorkflow:
    @workflow.run
    async def run(self, prompt: str, project: str, location: str) -> str:
        client = TemporalAsyncClient(
            vertexai=True,
            project=project,
            location=location,
        )
        response = await client.models.generate_content(
            model="gemini-2.5-flash",
            contents=prompt,
        )
        return response.text or ""

```

The Worker's `genai.Client` uses Application Default Credentials instead of an API key. Run
`gcloud auth application-default login`, or set `GOOGLE_APPLICATION_CREDENTIALS` to a service account key file.

[google_genai/vertex_ai/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/vertex_ai/run_worker.py)
```py
genai_client = genai.Client(
    vertexai=True,
    project=os.environ["GOOGLE_CLOUD_PROJECT"],
    location=os.environ.get("GOOGLE_CLOUD_LOCATION", "us-central1"),
)
plugin = GoogleGenAIPlugin(genai_client)
```

The `vertexai` setting must match on both sides. A Workflow that sets `vertexai=True` against a Worker configured for
the Gemini Developer API sends requests the backend can't serve.

## Set timeouts and retries

Every API call defaults to a 60-second `start_to_close_timeout` and Temporal's default retry policy. Override that for
all of a client's calls with `activity_config`:

```python
from datetime import timedelta

from temporalio.common import RetryPolicy
from temporalio.google_genai import TemporalAsyncClient
from temporalio.workflow import ActivityConfig

client = TemporalAsyncClient(
    activity_config=ActivityConfig(
        start_to_close_timeout=timedelta(minutes=5),
        retry_policy=RetryPolicy(maximum_attempts=3),
    ),
)
```

`activity_as_tool` and `TemporalMcpClientSession` take their own `activity_config`, so tool calls and MCP calls can use
different limits than model calls.

Let Temporal own retries. The plugin rejects a `genai.Client` configured with `http_options.retry_options`, because an
SDK-internal retry loop hides its attempts inside a single Activity and compounds with the Temporal retry policy.

## Samples

The [Google GenAI plugin samples](https://github.com/temporalio/samples-python/tree/main/google_genai) cover each
pattern as a self-contained, runnable scenario:

- [`hello_world`](https://github.com/temporalio/samples-python/tree/main/google_genai/hello_world): one
  `generate_content` call.
- [`tools`](https://github.com/temporalio/samples-python/tree/main/google_genai/tools): an Activity tool and a
  Workflow-method tool on the same call.
- [`streaming`](https://github.com/temporalio/samples-python/tree/main/google_genai/streaming): chunks forwarded
  to an external subscriber with `WorkflowStream`.
- [`chat`](https://github.com/temporalio/samples-python/tree/main/google_genai/chat): a multi-turn conversation
  with `client.chats`.
- [`structured_output`](https://github.com/temporalio/samples-python/tree/main/google_genai/structured_output):
  typed JSON output through a Pydantic model.
- [`mcp`](https://github.com/temporalio/samples-python/tree/main/google_genai/mcp): MCP tools run as Activities,
  with a self-contained echo server.
- [`files`](https://github.com/temporalio/samples-python/tree/main/google_genai/files): a file uploaded with
  `client.files` and referenced in a prompt.
- [`interactions`](https://github.com/temporalio/samples-python/tree/main/google_genai/interactions):
  server-managed conversations with `client.interactions`.
- [`agents`](https://github.com/temporalio/samples-python/tree/main/google_genai/agents): managed agent create,
  get, list, and delete with `client.agents`.
- [`vertex_ai`](https://github.com/temporalio/samples-python/tree/main/google_genai/vertex_ai): the same
  hello-world flow against Vertex AI.

---

# LangGraph integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/langgraph

Temporal's integration with [LangGraph](https://www.langchain.com/langgraph) gives your LangGraph AI agent workflows
durable execution, automatic retries, and timeouts via the Temporal platform.

The plugin supports both the LangGraph **Graph API** (`StateGraph` with nodes and edges) and the **Functional API**
(`@entrypoint` / `@task` decorators). Each graph node and task must specify whether it runs as a Temporal Activity or
directly inside the Workflow — Activity nodes get configurable timeouts and retry policies, while Workflow nodes run
inline and must be deterministic.

> **Public Preview**

Code snippets in this guide are taken from the
[LangGraph plugin samples](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin). Refer to the
samples for the complete code.

## Prerequisites

- This guide assumes you are already familiar with LangGraph. If you aren't, refer to the
  [LangGraph documentation](https://langchain-ai.github.io/langgraph/) for more details.
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Ensure you have set up your local development environment by following the
  [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave the
  Temporal development server running if you want to test your code locally.

## Install the plugin

Install the standalone `temporalio-langgraph` package:

```bash
uv add temporalio-langgraph
```

or with pip:

```bash
pip install temporalio-langgraph
```

The package installs the Temporal Python SDK and LangGraph dependencies. Refer to the
[package requirements](https://github.com/temporalio/ai-integrations/blob/main/python/langgraph/pyproject.toml)
for supported Python and dependency versions.

> **📝 Note:**
>
> Use a Python runtime that supports the `context` parameter of
> [`asyncio.create_task()`](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task) for the Functional API
> (`@entrypoint` / `@task`), `interrupt()`, and streaming from a node running in the Workflow. LangGraph uses this parameter
> to propagate `contextvars`. The plugin warns if your runtime does not support these features.
>

## Graph API

The Graph API uses `StateGraph` to define nodes and edges declaratively.

### Define a graph and Workflow

Build a `StateGraph`, then retrieve it inside your Workflow with the `graph()` helper:

```python
from datetime import timedelta

from langgraph.graph import START, StateGraph
from temporalio import workflow
from temporalio.langgraph import graph

async def process_query(query: str) -> str:
    """Process a query and return a response."""
    return f"Processed: {query}"

def build_graph() -> StateGraph:
    """Construct a single-node graph."""
    g = StateGraph(str)
    g.add_node(
        "process_query",
        process_query,
        metadata={
            "execute_in": "activity",
            "start_to_close_timeout": timedelta(seconds=10),
        },
    )
    g.add_edge(START, "process_query")
    return g

@workflow.defn
class HelloWorldWorkflow:
    @workflow.run
    async def run(self, query: str) -> str:
        return await graph("hello-world").compile().ainvoke(query)
```

### Configure the Worker

Create a `LangGraphPlugin` with your graphs and pass it to the Worker:

```python
import asyncio

from temporalio.client import Client
from temporalio.langgraph import LangGraphPlugin
from temporalio.worker import Worker

async def main() -> None:
    client = await Client.connect("localhost:7233")
    plugin = LangGraphPlugin(graphs={"hello-world": build_graph()})

    worker = Worker(
        client,
        task_queue="langgraph-hello-world",
        workflows=[HelloWorldWorkflow],
        plugins=[plugin],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

### Set Activity options

Pass Activity options as node `metadata` when calling `add_node`. Every node must include `"execute_in"` set to either
`"activity"` or `"workflow"`; the plugin raises an error if it's missing.

```python
from datetime import timedelta
from temporalio.common import RetryPolicy

g = StateGraph(str)
g.add_node(
    "my_node",
    my_node,
    metadata={
        "execute_in": "activity",
        "start_to_close_timeout": timedelta(seconds=30),
        "retry_policy": RetryPolicy(maximum_attempts=3),
    },
)
```

> **⚠️ Warning:**
>
> Don't pass a LangGraph `retry_policy=` to `add_node` (or `@task(retry_policy=...)`) — the plugin raises an error if you
> do. Use Temporal's `RetryPolicy` via the node's `metadata` (Graph API) or `activity_options` (Functional API) instead.
>

### Shared defaults

To apply the same Activity options across every node and task, pass `default_activity_options` to `LangGraphPlugin`.
Per-node `metadata` (Graph API) and per-task `activity_options` (Functional API) override these defaults key by key:

```python
plugin = LangGraphPlugin(
    graphs={"my-graph": g},
    default_activity_options={
        "start_to_close_timeout": timedelta(seconds=30),
        "retry_policy": RetryPolicy(maximum_attempts=3),
    },
)
```

To mitigate potential determinism bugs, `execute_in` cannot be set in `default_activity_options` — you must set it on each node or task individually.  See [Activity versus Workflow execution](#activity-vs-workflow-execution).

## Functional API

The Functional API uses `@entrypoint` and `@task` decorators, letting you orchestrate tasks with native Python
control flow (`while`, `if/else`, `for`) rather than declaring nodes and edges.

### Define tasks and a Workflow

```python
from datetime import timedelta

from langgraph.func import entrypoint as lg_entrypoint
from langgraph.func import task
from temporalio import workflow
from temporalio.langgraph import entrypoint

@task
def agent_think(query: str, history: list[str]) -> dict:
    """Decide the next action based on query and tool history."""
    tool_results = [h for h in history if h.startswith("[Tool]")]
    if len(tool_results) < 2:
        return {"action": "tool", "tool_name": "search", "tool_input": query}
    return {"action": "final", "answer": f"Found: {'; '.join(tool_results)}"}

@task
def execute_tool(tool_name: str, tool_input: str) -> str:
    """Execute a tool by name."""
    return f"[Tool] Result for {tool_name}({tool_input})"

@lg_entrypoint()
async def react_agent(query: str) -> dict:
    """ReAct agent loop: think -> act -> observe -> repeat."""
    history: list[str] = []
    while True:
        decision = await agent_think(query, history)
        if decision["action"] == "final":
            return {"answer": decision["answer"], "steps": len(history)}
        result = await execute_tool(decision["tool_name"], decision["tool_input"])
        history.append(result)

all_tasks = [agent_think, execute_tool]

activity_options = {
    t.func.__name__: {
        "execute_in": "activity",
        "start_to_close_timeout": timedelta(seconds=30),
    }
    for t in all_tasks
}

@workflow.defn
class ReactAgentWorkflow:
    @workflow.run
    async def run(self, query: str) -> dict:
        return await entrypoint("react-agent").ainvoke(query)
```

### Configure the Worker with the Functional API

```python
from temporalio.langgraph import LangGraphPlugin

plugin = LangGraphPlugin(
    entrypoints={"react-agent": react_agent},
    tasks=all_tasks,
    activity_options=activity_options,
)

worker = Worker(
    client,
    task_queue="langgraph-react-agent",
    workflows=[ReactAgentWorkflow],
    plugins=[plugin],
)
```

## Checkpointer

If your LangGraph code requires a checkpointer (for example, if you're using interrupts), use `InMemorySaver`. Temporal
handles durability, so third-party checkpointers (like PostgreSQL or Redis) are not needed.

```python
import langgraph.checkpoint.memory

g = graph("my-graph").compile(
    checkpointer=langgraph.checkpoint.memory.InMemorySaver(),
)
```

## Runtime context

LangGraph's run-scoped context (`context_schema`) is reconstructed on the Activity side, so nodes and tasks can read
from `runtime.context`:

```python
from langgraph.runtime import Runtime
from typing_extensions import TypedDict

from temporalio.langgraph import graph

class Context(TypedDict):
    user_id: str

async def my_node(state: State, runtime: Runtime[Context]) -> dict:
    return {"user": runtime.context["user_id"]}

# In the Workflow:
g = graph("my-graph").compile()
await g.ainvoke({...}, context=Context(user_id="alice"))
```

Your `context` object must be serializable by the configured Temporal payload converter, since it crosses the Activity
boundary.

## Continue-as-new

Long-running graphs can hit Temporal's per-Event history size limit. Use Temporal's
[continue-as-new](/develop/python/workflows/continue-as-new) to start a fresh execution while preserving the results of nodes
and tasks that have already completed.

The `cache()` helper returns the current task-result cache as a serializable dict. Pass it to `graph(name, cache=...)`
or `entrypoint(name, cache=...)` in the new run to skip re-executing nodes that already produced a result.

```python
from temporalio import workflow
from temporalio.langgraph import cache, graph

@workflow.defn
class LongRunningWorkflow:
    @workflow.run
    async def run(self, state: State, prior_cache: dict | None = None) -> State:
        g = graph("my-graph", cache=prior_cache).compile()
        # ... run some steps, then continue-as-new before history grows too large ...
        workflow.continue_as_new(args=[state, cache()])
```

## Streaming

To stream intermediate values (such as LLM tokens or progress updates) out of a running
graph, set `streaming_topic` on `LangGraphPlugin`. Calls to LangGraph's
`get_stream_writer()` inside a node then publish to the named topic on the Workflow's
[`WorkflowStream`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/workflow_streams),
and external subscribers consume the stream with `WorkflowStreamClient`.

Activity nodes publish via a batched Temporal signal, controlled by
`streaming_batch_interval` (default 100ms). Workflow nodes publish synchronously to the
in-Workflow stream, with no signal.

When `streaming_topic` is set, your Workflow **must** construct a `WorkflowStream()` in its
`@workflow.init` (its `__init__`); otherwise the plugin raises an error.

```python
from datetime import timedelta

from langgraph.config import get_stream_writer
from langgraph.graph import START, StateGraph
from typing_extensions import TypedDict

from temporalio import workflow
from temporalio.langgraph import LangGraphPlugin, graph
from temporalio.contrib.workflow_streams import WorkflowStream, WorkflowStreamClient

class State(TypedDict):
    value: str

async def token_node(state: State) -> dict[str, str]:
    writer = get_stream_writer()
    for token in ["hello", " ", "world"]:
        writer({"token": token})
    writer({"done": True})
    return {"value": "hello world"}

@workflow.defn
class StreamingWorkflow:
    def __init__(self) -> None:
        # Required when streaming_topic is set on the plugin.
        _ = WorkflowStream()
        self.app = graph("streaming").compile()

    @workflow.run
    async def run(self) -> str:
        result = await self.app.ainvoke({"value": ""})
        return result["value"]
```

Configure the plugin with `streaming_topic`:

```python
g = StateGraph(State)
g.add_node("token_node", token_node, metadata={"execute_in": "activity"})
g.add_edge(START, "token_node")

plugin = LangGraphPlugin(
    graphs={"streaming": g},
    default_activity_options={"start_to_close_timeout": timedelta(seconds=10)},
    streaming_topic="tokens",
)
```

An external client subscribes to the topic to consume items as they're published:

```python
handle = await client.start_workflow(
    StreamingWorkflow.run, id="streaming-wf", task_queue="streaming-tq"
)

ws_client = WorkflowStreamClient.create(client, handle.id)
async for item in ws_client.topic("tokens", type=dict).subscribe(from_offset=0):
    print(item.data)
    if item.data.get("done"):
        break

print(await handle.result())
```

> **📝 Note:**
>
> For streaming from a node running in the Workflow (`execute_in: "workflow"`), check the Python runtime requirements
> in the [installation note](#install-the-plugin).
>

### What's covered, and what isn't

`streaming_topic` wires up exactly one LangGraph stream mode: `stream_mode="custom"` — the
values written through `get_stream_writer()`. The other modes (`"messages"`, `"values"`,
`"updates"`, and `"debug"`) are **not** captured by `streaming_topic`, because they aren't
produced by node-side writers; LangGraph's orchestrator emits them as it walks the graph.

To stream one of those modes, bridge `astream()` in the Workflow and republish each yielded
chunk to a `WorkflowStream` topic yourself:

```python
@workflow.defn
class AstreamBridge:
    def __init__(self) -> None:
        self.stream = WorkflowStream()
        self.app = graph("g").compile()

    @workflow.run
    async def run(self) -> None:
        topic = self.stream.topic("astream")
        async for chunk in self.app.astream({...}, stream_mode="messages"):
            topic.publish(chunk)
        topic.publish({"done": True})
```

### Retry semantics

Streaming has **at-least-once** delivery per Activity attempt. When an Activity-wrapped node
retries (transient failure, worker crash, and so on), the node function re-runs from scratch
and re-publishes its writes — earlier publishes from the failed attempt are not rolled back.
Subscribers should be ready to see duplicates and recover idempotently: dedupe on a sequence
ID you include in each chunk, or treat the stream as advisory and rely on the Workflow's final
result for state.

For a complete working example, see the
[streaming sample](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin/graph_api/streaming).

## Tracing

For tracing your LangGraph Workflows and Activities, we recommend the
[Temporal LangSmith plugin](/develop/python/integrations/langsmith). It
composes with `LangGraphPlugin` — pass both plugins to your Worker.

## Stores are not supported

LangGraph's `Store` (for example, `InMemoryStore` passed via `graph.compile(store=...)` or `@entrypoint(store=...)`)
isn't accessible inside Activity-wrapped nodes: the Store holds live state that can't cross the Activity boundary, and
Activities may run on a different worker than the Workflow. If you pass a store, the plugin logs a warning on first use
and `runtime.store` is `None` inside nodes.

Use Workflow state for per-run memory, or an external database (Postgres, Redis, etc.) configured on each worker if you
need shared memory across runs.

## Activity versus Workflow execution 

Every graph node and `@task` must specify `execute_in` — set it to `"activity"` to run as a Temporal Activity, or
`"workflow"` to run directly inside the Workflow. The plugin raises an error if you forget to set it.

`execute_in` must be set per node or task; it cannot be set in `default_activity_options`.

Understanding when to use each mode is important for correctness and durability.

### When to use an Activity

Use `execute_in: "activity"` when a node does any of the following:

- **Makes network calls** — LLM calls, HTTP requests, database queries, or any I/O. Activities can do I/O; Workflows
  cannot.
- **Has non-deterministic behavior** — anything that can return different results on re-execution (random numbers,
  current time, external data).
- **Is long-running or may fail** — Activities get configurable timeouts, automatic retries, and heartbeating. If an LLM
  call times out or a service is unavailable, Temporal retries the Activity without re-running the entire Workflow.
- **Calls `interrupt()`** — LangGraph's `interrupt()` is supported in Activity nodes. The plugin serializes the
  interrupt and propagates it back to the Workflow for human-in-the-loop patterns.

### When to run in the Workflow

Use `execute_in: "workflow"` when a node:

- **Orchestrates other graphs** — a node that calls `graph("child").compile().ainvoke(state)` to dispatch to a subgraph.
  The subgraph's own nodes still run as Activities, but the orchestration logic runs in the Workflow.
- **Performs pure state transformations** — deterministic data reshaping, merging, or filtering with no I/O.
- **Is a lightweight routing step** — when a node's only job is to decide what happens next and you want to avoid the
  overhead of an Activity round-trip.

> **⚠️ Warning:**
>
> A Workflow Definition must be [deterministic](/develop/python/workflows/basics#workflow-logic-requirements). A node running in
> the Workflow **must not** make network calls, use `random`, read the system clock, or do file I/O. Violating this causes
> non-determinism errors on replay.
>

### Where LangGraph primitives run

Not all LangGraph primitives are node functions. Some run in the Workflow context regardless of the `execute_in` setting:

| Primitive | Runs in | Notes |
| --- | --- | --- |
| Node functions | Activity or Workflow | Controlled by `execute_in` in node `metadata` (required) |
| `@task` functions | Activity or Workflow | Controlled by `execute_in` in `activity_options` (required) |
| Conditional edge functions (`add_conditional_edges`) | Workflow | Always runs in the Workflow. Must be **deterministic** and **async** (sync functions trigger `run_in_executor`, which is not allowed in the Temporal sandbox). |
| `interrupt()` | Activity | Call `interrupt()` inside Activity nodes. The plugin serializes the interrupt and propagates it to the Workflow. |
| `Command(resume=...)` | Workflow | Used from Workflow code to resume after an interrupt. |
| `InMemorySaver` checkpointer | Workflow | Runs in-process. Temporal handles durability — external checkpointers are not needed. |

> **💡 Tip:**
>
> Conditional edge functions like `should_continue` must be `async def`, not plain `def`. Synchronous functions cause
> LangGraph to use `run_in_executor`, which is not supported inside Temporal's Workflow sandbox.
>
> ```python
> # ✅ Correct: async conditional edge function
> async def should_continue(state: AgentState) -> str:
>     if state["messages"][-1].startswith("[Agent]") and "Calling" in state["messages"][-1]:
>         return "tools"
>     return END
>
> g.add_conditional_edges("agent", should_continue)
> ```
>

### Syntax

```python
# Graph API
g.add_node("my_node", my_node, metadata={"execute_in": "workflow"})

# Functional API
plugin = LangGraphPlugin(
    tasks=[my_task],
    activity_options={"my_task": {"execute_in": "workflow"}},
)
```

### Example: Subgraph orchestration

A common pattern is a parent node that runs in the Workflow and dispatches to a child graph whose nodes run as
Activities:

```python
async def parent_node(state: State) -> dict[str, str]:
    return await graph("child").compile().ainvoke(state)

parent = StateGraph(State)
parent.add_node("parent_node", parent_node, metadata={"execute_in": "workflow"})
parent.add_edge(START, "parent_node")

plugin = LangGraphPlugin(graphs={"parent": parent, "child": child})
```

## Human-in-the-loop

LangGraph's `interrupt()` works with Temporal signals and queries to support human-in-the-loop patterns:

1. A graph node calls `interrupt(draft)`, pausing execution.
2. The Workflow exposes the pending draft via a Temporal query.
3. An external process (UI, CLI) queries the draft and sends approval via a Temporal signal.
4. The graph resumes — `interrupt()` returns the signal value and the node completes.

See the [human-in-the-loop samples](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin/graph_api/human_in_the_loop) for
complete working examples using both Graph and Functional APIs.

## Samples

The [LangGraph plugin samples](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin)
demonstrate all supported patterns across both APIs.

---

# LangSmith integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/langsmith

Temporal's LangSmith integration lets you trace AI agent Workflows in [LangSmith](https://smith.langchain.com/)
alongside every LLM call, tool execution, and Temporal operation.

Temporal gives your agent code [durable execution](/temporal#durable-execution). 
LangSmith adds the observability side, so you can inspect LLM inputs and outputs, follow a
request from the Client through to the model, and compare runs over time.

The `LangSmithPlugin` is what connects the two. It propagates trace context across Temporal boundaries so that runs
started on the Client nest correctly under Workflow and Activity runs on the Worker. It can also create LangSmith
runs for Temporal operations themselves: Workflow executions, Activity executions, Signals, Updates, and Queries.

> **Public Preview**

All code snippets in this guide are taken from the
[LangSmith tracing sample](https://github.com/temporalio/samples-python/tree/main/langsmith_tracing). Refer to the
sample for complete code.

## Prerequisites

- This guide assumes you are already familiar with LangSmith. If you aren't, refer to the
  [LangSmith documentation](https://docs.smith.langchain.com/) for more details.
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Ensure you have set up your local development environment by following the
  [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave
  the Temporal Development Server running if you want to test your code locally.

## Configure Workers to use LangSmith

Workers execute the code that defines your Workflows and Activities. To trace Workflow and Activity execution in
LangSmith, add the `LangSmithPlugin` to your Worker.

Follow the steps below to configure your Worker.

1. Install the standalone `temporalio-langsmith` package. It installs the Temporal Python SDK and LangSmith dependencies.
   Refer to the [package requirements](https://github.com/temporalio/ai-integrations/blob/main/python/langsmith/pyproject.toml)
   for supported Python and dependency versions.

   ```bash
   uv add temporalio-langsmith
   ```

2. Add the `LangSmithPlugin` to your Worker. Set `project_name` to the LangSmith project where you want traces to
   appear.

   ```python
   from temporalio.langsmith import LangSmithPlugin
   from temporalio.worker import Worker

   worker = Worker(
       client,
       task_queue="my-task-queue",
       workflows=[MyWorkflow],
       activities=[my_activity],
       plugins=[LangSmithPlugin(project_name="my-project")],
   )
   ```

3. Run the Worker. Ensure the Worker process has access to your LangSmith API key via the `LANGSMITH_API_KEY`
   environment variable, and enable tracing with `LANGCHAIN_TRACING_V2`.

   ```bash
   export LANGSMITH_API_KEY="your-api-key"
   export LANGCHAIN_TRACING_V2=true
   python worker.py
   ```

## Configure Clients to use LangSmith

Add the plugin to any Temporal Client you use on the Client side (typically a starter or API that calls into
your Workflows) so that client-side operations like starting a Workflow or sending an Update get linked to the
Workflows they trigger.

```python
from temporalio.client import Client
from temporalio.langsmith import LangSmithPlugin

client = await Client.connect(
    "localhost:7233",
    plugins=[LangSmithPlugin(project_name="my-project")],
)
```

> **💡 Tip:**
>
> Use the same `project_name` on both the Worker and the Client so their traces land in the same LangSmith project.
>

> **📝 Note:**
>
> `@traceable` functions on the Client side run outside the plugin's interceptor scope, so they don't pick up
> `project_name` from the plugin. If you have a client-side `@traceable` that wraps a call into your Workflow, pass
> `project_name` to it explicitly so it lands in the same LangSmith project as the rest of the trace.
>

## Trace Activities

Any non-deterministic work in a Temporal Workflow (LLM calls, tool executions, database queries, external API calls,
and so on) must run inside an Activity. That makes Activities an important place to add LangSmith runs. When you
decorate an Activity function with `@traceable`, the run shows up in LangSmith nested under the Workflow that
scheduled it.

```python
from dataclasses import dataclass
from langsmith import traceable
from temporalio import activity

@traceable(name="Fetch Weather", run_type="tool")
@activity.defn
async def fetch_weather(city: str) -> str:
    # Call an external weather API here.
    ...
```

You can combine `@traceable` with provider-specific LangSmith wrappers to capture more detail. For OpenAI, for
example, `wrap_openai` patches the client so that every API call creates its own child run with the model name,
prompt, completion, token counts, and latency. You can access this by wrapping the client:

```python
from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import AsyncOpenAI
from temporalio import activity

@dataclass
class OpenAIRequest:
    model: str
    input: str

# wrap_openai patches the client so that every API call adds a ChatOpenAI run under the @traceable.
# Set max_retries=0 and use Temporal's Activity retry policy instead.
@traceable(name="Call OpenAI", run_type="llm")
@activity.defn
async def call_openai(request: OpenAIRequest) -> str:
    client = wrap_openai(AsyncOpenAI(max_retries=0))
    response = await client.responses.create(
        model=request.model,
        input=request.input,
    )
    return response.output_text
```

LangSmith ships similar wrappers for
[Anthropic](https://docs.smith.langchain.com/observability/how-to/integrations#anthropic) and other providers; refer
to the LangSmith documentation for the full list.

## Add custom runs with @traceable

Decorate functions with `@traceable` to create named runs for your business logic. You control the run name, tags,
metadata, and `run_type` (`chain`, `llm`, `tool`, `retriever`).

Put `@traceable` on Activities and on private helper methods within your Workflow class that get called from Workflow
code. For example:

```python
from langsmith import traceable
from temporalio import workflow

@workflow.defn
class ChatbotWorkflow:
    # Private helper methods can be decorated directly.
    @traceable(name="Save Note", run_type="tool")
    def _save_note(self, name: str, content: str) -> str:
        ...
```

> **⚠️ Warning:**
>
> Do not put `@traceable` directly on any `@workflow` method (for example, `@workflow.run`, `@workflow.signal`,
> `@workflow.update`, `@workflow.query`). Doing so can produce duplicate or orphaned (unknown parent) runs in LangSmith.
> If you want to trace the body of one of these methods, move the logic into an inner function and decorate that:
>
> ```python
> @workflow.defn
> class MyWorkflow:
>     @workflow.run
>     async def run(self, prompt: str) -> str:
>         # Option 1: Use the @traceable decorator
>         @traceable(name=f"Ask: {prompt[:60]}", run_type="chain")
>         async def _run() -> str:
>             ...
>         return await _run()
>
>     @workflow.update
>     async def message_from_user(self, message: str) -> str:
>         async def _handle_message(self, message: str) -> str:
>             ...
>         # Option 2: Use the traceable() function
>         return await traceable(
>             name=f"Update: {message[:60]}",
>             run_type="chain",
>         )(self._handle_message)(message)
> ```
>

## Include Temporal operations as runs

By default, `LangSmithPlugin(add_temporal_runs=False)` only propagates LangSmith context so that `@traceable` and
`wrap_openai` calls nest correctly. The plugin does not create its own runs.

Set `add_temporal_runs=True` if you want runs for the Temporal operations themselves: Workflow executions, Activity
executions, Signals, Updates, Queries, and Child Workflows.

```python
plugin = LangSmithPlugin(
    project_name="my-project",
    add_temporal_runs=True,
)
```

With this on, your LangSmith traces include runs like `StartWorkflow:MyWorkflow`, `RunWorkflow:MyWorkflow`,
`StartActivity:call_openai`, and `RunActivity:call_openai`. `Start*` and `Run*` pairs appear as siblings: the `Start*`
run is emitted by the side scheduling the operation (for example, the Client), and the `Run*` run is emitted by the
side executing it (for example, the Worker).

## Trace hierarchy example

With the plugin configured on both Client and Worker, and `add_temporal_runs=True`, a trace for a simple LLM call looks
like this:

```
Run Agent                               (@traceable, client-side)
├── StartWorkflow:MyWorkflow             (automatic, LangSmithPlugin)
└── RunWorkflow:MyWorkflow               (automatic, LangSmithPlugin)
    └── Ask: What is Temporal?          (@traceable, Workflow)
        ├── StartActivity:call_openai    (automatic, LangSmithPlugin)
        └── RunActivity:call_openai      (automatic, LangSmithPlugin)
            └── Call OpenAI              (@traceable, Activity)
                └── ChatOpenAI           (automatic via wrap_openai)
```

Without `add_temporal_runs` (the default), only the `@traceable` and `wrap_openai` runs appear. Context still
propagates, so they nest correctly under the client-side run:

```
Run Agent                               (@traceable, client-side)
└── Ask: What is Temporal?              (@traceable, Workflow-side)
    └── Call OpenAI                     (@traceable, Activity-side)
        └── ChatOpenAI                  (automatic via wrap_openai)
```

## Example sample

The [LangSmith tracing sample](https://github.com/temporalio/samples-python/tree/main/langsmith_tracing) puts 
these patterns together in two working examples:

- **`basic/`**: a one-shot Workflow that sends a prompt to OpenAI and returns the response.
- **`chatbot/`**: a long-running conversational Workflow with tool calls (save and read notes), Update handlers, and
  dynamic trace names per message.

Each example shows the `LangSmithPlugin` configuration, `@traceable` runs on the Client, Workflow, and Activity, and
expected trace output for both `add_temporal_runs=False` and `add_temporal_runs=True`.

---

# Model Context Protocol (MCP) integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/mcp

> Call MCP tools, prompts, and resources from Python Workflows, with Activity retries, timeouts, and results preserved across Worker restarts.

The `temporalio-mcp` integration runs [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) client operations
as Temporal Activities. Workflow code calls `TemporalMCPClient`, while `MCPPlugin` manages the MCP clients, transports,
and credentials on the Worker. Completed operation results are recorded in Event History and reused during Workflow
replay.

> **Pre-release**

This guide uses the standalone MCP integration, without an agent framework. To give MCP tools to an OpenAI agent,
use the [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents#mcp-servers).

## Install the integration

Use Python 3.10 or later and install the package:

```bash
uv add temporalio-mcp
```

The package installs the Temporal Python SDK (version 1.32.0 or later), MCP Python SDK v2, and Pydantic dependencies.

Before running the example, follow the [Python development environment setup](/develop/python/set-up-your-local-python)
and leave the Temporal Development Server running at `localhost:7233`.
Save the following four Python files in the same directory.

## Create an MCP server

Define an MCP server with an `echo` tool that returns its input. The Worker starts this server as a subprocess and
communicates with it over stdio.

```python title="mcp_server.py"
from mcp.server.mcpserver import MCPServer

server = MCPServer("echo")

@server.tool()
def echo(value: str) -> str:
    """Return the input unchanged."""
    return value

if __name__ == "__main__":
    server.run()
```

## Call a tool from a Workflow

Create a `TemporalMCPClient` with a name you will register on the Worker. Call `call_tool()` with the tool name and
arguments, then read the returned MCP content.

```python title="workflows.py"
from mcp.types import TextContent
from temporalio import workflow
from temporalio.exceptions import ApplicationError
from temporalio.mcp import TemporalMCPClient

@workflow.defn
class EchoWorkflow:
    @workflow.run
    async def run(self, message: str) -> str:
        mcp = TemporalMCPClient("echo")
        result = await mcp.call_tool("echo", {"value": message})
        if result.is_error:
            raise ApplicationError("Echo tool failed", non_retryable=True)
        return "\n".join(
            content.text
            for content in result.content
            if isinstance(content, TextContent)
        )
```

Keep MCP client creation and network I/O on the Worker. The Workflow-side proxy schedules Activities and does not
open an MCP connection itself.

## Register the MCP client on a Worker

Pass `MCPPlugin` a mapping of names to factory functions. Each factory must return a fresh MCP `Client` and transport.
The name must match the one passed to `TemporalMCPClient`.

```python title="run_worker.py"
import asyncio
import sys
from pathlib import Path

from mcp import Client as MCPClient, StdioServerParameters, stdio_client
from temporalio.client import Client
from temporalio.mcp import MCPPlugin
from temporalio.worker import Worker

from workflows import EchoWorkflow

async def main() -> None:
    parameters = StdioServerParameters(
        command=sys.executable,
        args=[str(Path(__file__).with_name("mcp_server.py"))],
    )
    plugin = MCPPlugin(
        clients={"echo": lambda: MCPClient(stdio_client(parameters))},
    )
    client = await Client.connect("localhost:7233")
    async with Worker(
        client,
        task_queue="mcp-demo",
        workflows=[EchoWorkflow],
        plugins=[plugin],
    ):
        await asyncio.Event().wait()

if __name__ == "__main__":
    asyncio.run(main())
```

The plugin registers the MCP Activities and configures the Workflow sandbox to pass through the MCP modules.
You don't need to add `MCPPlugin` to the Temporal Client that starts the Workflow.

For a server available over Streamable HTTP, replace the stdio factory with an MCP client that points to your server:

```python
from mcp import Client as MCPClient
from temporalio.mcp import MCPPlugin

plugin = MCPPlugin(
    clients={"echo": lambda: MCPClient("http://localhost:8000/mcp")},
)
```

That server must expose the `echo` tool used in this example. In-process MCP servers and custom MCP v2 transports
are also supported through the MCP `Client` API.

## Run the Workflow

Start the Worker:

```bash
uv run run_worker.py
```

In another terminal, run a Client that starts the Workflow on the same Task Queue:

```python title="run_workflow.py"
import asyncio

from temporalio.client import Client

from workflows import EchoWorkflow

async def main() -> None:
    client = await Client.connect("localhost:7233")
    result = await client.execute_workflow(
        EchoWorkflow.run,
        "Hello from Temporal",
        id="mcp-echo-workflow",
        task_queue="mcp-demo",
    )
    print(result)

if __name__ == "__main__":
    asyncio.run(main())
```

```bash
uv run run_workflow.py
```

The script prints `Hello from Temporal`. Open the Workflow in the [Web UI](http://localhost:8233) to see the
`temporalio.mcp.echo.call-tool` Activity in its Event History.

## Use tools, prompts, and resources

`TemporalMCPClient` exposes the following request/response operations. Each schedules an Activity named
`temporalio.mcp.<name>.<operation>`, where `<name>` is the registered client name.

| Workflow method | Activity operation | Return type |
| --- | --- | --- |
| `list_tools()` | `list-tools` | `ListToolsResult` |
| `call_tool(name, arguments)` | `call-tool` | `CallToolResult` |
| `list_prompts()` | `list-prompts` | `ListPromptsResult` |
| `get_prompt(name, arguments)` | `get-prompt` | `GetPromptResult` |
| `list_resources()` | `list-resources` | `ListResourcesResult` |
| `list_resource_templates()` | `list-resource-templates` | `ListResourceTemplatesResult` |
| `read_resource(uri)` | `read-resource` | `ReadResourceResult` |

List operations fetch every page within one Activity and return a complete result with `next_cursor=None`.
`list_tools()` caches its result for the lifetime of the Workflow-side client object, so repeated calls on that object
do not schedule another Activity. Pass `cache_tools_list=False` to `TemporalMCPClient` to fetch tools on every call.

## Configure timeouts and retries

Pass an `activity_config` to `TemporalMCPClient` to configure the Activities it schedules:

```python
from datetime import timedelta

from temporalio.common import RetryPolicy
from temporalio.mcp import TemporalMCPClient

mcp = TemporalMCPClient(
    "echo",
    activity_config={
        "start_to_close_timeout": timedelta(seconds=20),
        "schedule_to_close_timeout": timedelta(minutes=1),
        "retry_policy": RetryPolicy(maximum_attempts=3),
    },
)
```

By default, each operation has a one-minute Start-To-Close Timeout. The integration adds this default only when you
omit both `start_to_close_timeout` and `schedule_to_close_timeout`, or set both to `None`.
Set a Schedule-To-Close Timeout to bound the total time, including retries. Retryable errors can retry indefinitely
with the default Retry Policy and no Schedule-To-Close Timeout.

Handle errors according to how the MCP server reports them:

- A tool result with `is_error=True` completes the Activity successfully. Inspect the result in Workflow code and
  decide whether to retry, return an error, or fail the Workflow. The example above fails the Workflow.
- JSON-RPC error responses fail the Activity with an `ApplicationError` whose type is `MCPProtocolError`.
  Its `details[0]` contains the JSON-RPC error code, and `details[1]` contains error data when provided.
  Invalid response schemas also use `MCPProtocolError`, with validation errors in `details[0]`.
- Invalid requests, invalid arguments, and invalid response schemas are non-retryable. Internal server errors,
  closed connections, request timeouts, and transport exceptions remain retryable.

Make tools with side effects [idempotent](/activity-definition#idempotency). An Activity can execute more than
once, such as when a Worker loses the response after the MCP server has completed a tool call.

MCP Activities do not heartbeat. Setting `heartbeat_timeout` does not add heartbeats or enable cancellation delivery.
An Activity timeout also does not cancel an MCP request already running on the Worker. Configure the MCP client's
`read_timeout_seconds` to bound the request itself.

## Manage connections and credentials

For factories with no arguments, the Worker reuses connections that negotiate the modern, sessionless MCP protocol
until they have been idle for five minutes. Set `connection_idle_timeout` on `MCPPlugin` to control reuse:

- Set it to `None` to keep connections until plugin shutdown.
- Set it to `timedelta(0)` to close connections as soon as they become idle.
- Set it to another `timedelta` to change the idle period.

Connections that negotiate a legacy MCP handshake are not shared between Activities.
Keep any state needed across calls in the Workflow or in persistent storage on the MCP server. A Worker restart
creates a new connection; it does not restore MCP server state.

To select configuration for a tenant or another caller, register a factory that accepts one positional parameter and
pass `factory_argument` from the Workflow. For example, given a Worker-side `clients` mapping containing
`{"echo": create_client_for_tenant}`, construct the proxy as:

```python
from temporalio.mcp import TemporalMCPClient

mcp = TemporalMCPClient("echo", factory_argument="tenant-a")
```

The factory receives `"tenant-a"`. A non-`None` argument creates a fresh client for each Activity, without reusing a
connection. Pass only a stable, non-secret identifier: the argument is recorded in Event History.
Resolve credentials inside the Worker-side factory.

---

# OpenAI Agents SDK integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/openai-agents

> Run OpenAI Agents SDK agents as durable Temporal Workflows in Python, with model calls executed as Activities.

Temporal's integration with the [OpenAI Agents SDK for Python](https://openai.github.io/openai-agents-python/) lets you
run agents as Temporal Workflows. Agent orchestration—the agent loop, tool selection, and handoffs—runs inside the
Workflow, while model calls run as [Activities](/glossary#activity).

Like with other types of API calls, in a [Temporal Application](/glossary#temporal-application), you make LLM calls in
your Activities. This integration handles that for you: model calls are executed as Activities, so they retry durably
and are not repeated during Workflow replay. Your agents survive Worker restarts and can run for extended periods
without losing state.

## Prerequisites

- This guide assumes you are already familiar with the OpenAI Agents SDK. If you aren't, refer to the
  [OpenAI Agents SDK documentation](https://openai.github.io/openai-agents-python/) for more details.
- If you are new to Temporal, we recommend you read the [Understanding Temporal](/evaluate/understanding-temporal)
  document or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course to understand the basics
  of Temporal.
- Ensure you have set up your local development environment by following the
  [Set up a local development environment with the Python SDK](/develop/python/set-up-your-local-python) guide. When you are done, leave the
  Temporal Development Server running if you want to test your code locally.

## Install

```bash
uv add temporalio-openai-agents
```

The `temporalio-openai-agents` package installs the Temporal Python SDK and OpenAI Agents SDK dependencies. It requires
Python 3.10 or later and Temporal Python SDK 1.33.0 or later.

Two import paths cover most applications. `temporalio.openai_agents` holds what you configure on the Worker and
Client—including `OpenAIAgentsPlugin`, `ModelActivityParameters`, and sandbox providers.
`temporalio.openai_agents.workflow` holds what you call from inside a Workflow, such as `activity_as_tool` and MCP
server proxies.

Version 1.0.0 moves the integration out of the Temporal Python SDK. If you used the earlier contrib module, replace the
`temporalio[openai-agents]` dependency with `temporalio-openai-agents` and change imports from
`temporalio.contrib.openai_agents` to `temporalio.openai_agents`.

With Temporal Python SDK 1.33, install both distributions into the same physical `site-packages/temporalio` directory,
as a standard non-editable virtual environment does. Editable installs, layered deployments, `--target` installs, and
other split-directory installations require Temporal Python SDK 1.34 or later.

## Run your first durable agent

A Temporal-backed agent needs three pieces: a Workflow that runs the agent, a Worker configured with the integration
plugin, and a Client configured with the same plugin.

### Write the Workflow

Inside the Workflow, write ordinary OpenAI Agents SDK code. The plugin redirects `Runner.run` so that each model call
becomes an Activity—there is no Temporal-specific runner to learn.

<!--SNIPSTART python-openai-agents-hello-world-workflow-->
[openai_agents/basic/workflows/hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/workflows/hello_world_workflow.py)
```py
from agents import Agent, Runner
from temporalio import workflow

@workflow.defn
class HelloWorldAgent:
    @workflow.run
    async def run(self, prompt: str) -> str:
        agent = Agent(
            name="Assistant",
            instructions="You only respond in haikus.",
        )

        result = await Runner.run(agent, input=prompt)
        return result.final_output

```
<!--SNIPEND-->

### Configure the Worker

Register `OpenAIAgentsPlugin` on the Client. The plugin registers the model Activity, configures the Pydantic data
converter, propagates tracing context, and registers any configured MCP server or sandbox providers.

<!--SNIPSTART python-openai-agents-hello-world-worker-->
[openai_agents/basic/run_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/run_worker.py)
```py
client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                start_to_close_timeout=timedelta(seconds=30)
            )
        ),
    ],
)
```
<!--SNIPEND-->

Workers built from that Client pick up the plugin automatically, so all a Worker has to do is register the Workflow.

`ModelActivityParameters` controls how the model Activity is scheduled. Alongside `start_to_close_timeout`, which
defaults to 60 seconds, it accepts `schedule_to_close_timeout`, `schedule_to_start_timeout`, `heartbeat_timeout`,
`retry_policy`, `cancellation_type`, `versioning_intent`, `task_queue`, `priority`, `summary_override`,
`use_local_activity`, `streaming_topic`, and `streaming_batch_interval`.

You must ensure the Worker process has access to your model-provider credentials. Most provider SDKs read credentials
from environment variables.

### Start the Workflow

Attach the same plugin to the Client that starts the Workflow, so payloads are converted the same way on both sides.

<!--SNIPSTART python-openai-agents-hello-world-client-->
[openai_agents/basic/run_hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/run_hello_world_workflow.py)
```py
client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(),
    ],
)

# Execute a workflow
result = await client.execute_workflow(
    HelloWorldAgent.run,
    "Tell me about recursion in programming.",
    id="my-workflow-id",
    task_queue="openai-agents-basic-task-queue",
)
print(f"Result: {result}")
```
<!--SNIPEND-->

## Tools

Where a tool runs depends on how you define it.

| Tool                                | Runs in          | Use for                                                    |
| :---------------------------------- | :--------------- | :--------------------------------------------------------- |
| `activity_as_tool()`                | Temporal Activity | External I/O and other non-deterministic work               |
| `FunctionTool` / `@function_tool`   | Workflow          | Deterministic, Workflow-safe computation                    |
| OpenAI-hosted tool                  | Model provider    | Provider-hosted features run as part of the model call      |

Model calls are always routed through Activities. Tools are not: a `@function_tool` runs in the Workflow unless you back
it with an Activity, so any tool that performs I/O needs `activity_as_tool()` or a Nexus Operation.

### Activity-backed tools

Use `activity_as_tool` for HTTP calls, database access, file system work, or other I/O. Write an ordinary Temporal
Activity:

<!--SNIPSTART python-openai-agents-weather-activity-->
[openai_agents/basic/activities/get_weather_activity.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/activities/get_weather_activity.py)
```py
from dataclasses import dataclass

from temporalio import activity

@dataclass
class Weather:
    city: str
    temperature_range: str
    conditions: str

@activity.defn
async def get_weather(city: str) -> Weather:
    """
    Get the weather for a given city.
    """
    return Weather(city=city, temperature_range="14-20C", conditions="Sunny with wind.")

```
<!--SNIPEND-->

Then pass it through `activity_as_tool` when you build the agent:

<!--SNIPSTART python-openai-agents-activity-tool-workflow-->
[openai_agents/basic/workflows/tools_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/workflows/tools_workflow.py)
```py
@workflow.defn
class ToolsWorkflow:
    @workflow.run
    async def run(self, question: str) -> str:
        agent = Agent(
            name="Hello world",
            instructions="You are a helpful agent.",
            tools=[
                temporal_agents.workflow.activity_as_tool(
                    get_weather, start_to_close_timeout=timedelta(seconds=10)
                )
            ],
        )

        result = await Runner.run(agent, input=question)
        return result.final_output

```
<!--SNIPEND-->

`activity_as_tool` controls how the agent invokes the Activity; it does not register the Activity with the Worker. Pass
the Activity function to the Worker's `activities` argument as well.

Because the Activity may not run in the same process as the Workflow, an Activity-backed tool receives a copy of the
agent context and cannot mutate it. A tool that runs in the Workflow can.

### Inline and hosted tools

For deterministic computation, use the standard `@function_tool` decorator and call it directly from the Workflow. Do
not perform network, database, or file system I/O from these tools—use `activity_as_tool` instead.

Hosted tools such as `WebSearchTool`, `FileSearchTool`, `CodeInterpreterTool`, and `ImageGenerationTool` are executed by
the model provider during the model Activity, so they need no extra wiring:

<!--SNIPSTART python-openai-agents-hosted-tool-workflow-->
[openai_agents/tools/workflows/web_search_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/tools/workflows/web_search_workflow.py)
```py
@workflow.defn
class WebSearchWorkflow:
    @workflow.run
    async def run(self, question: str, user_city: str = "New York") -> str:
        agent = Agent(
            name="Web searcher",
            instructions="You are a helpful agent.",
            tools=[
                WebSearchTool(user_location={"type": "approximate", "city": user_city})
            ],
        )

        result = await Runner.run(agent, question)
        return result.final_output

```
<!--SNIPEND-->

`LocalShellTool` and `ComputerTool` are not supported, because they assume a single long-lived local process.

### Nexus operation tools

Use `nexus_operation_as_tool` to expose a [Nexus](/nexus) Operation as an agent tool. The Workflow starts the Operation
through a Nexus client and feeds the result back to the agent, which lets an agent call across a Namespace boundary:

```python
from temporalio.openai_agents.workflow import nexus_operation_as_tool

weather_tool = nexus_operation_as_tool(
    WeatherService.get_weather,
    service=WeatherService,
    endpoint="weather-endpoint",
)
```

### Nested agent tools

`Agent.as_tool()` from the OpenAI Agents SDK works unchanged. The nested agent's model calls become Activities like any
others, so a multi-agent run stays durable throughout:

<!--SNIPSTART python-openai-agents-agent-as-tool-workflow-->
[openai_agents/agent_patterns/workflows/agents_as_tools_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/agent_patterns/workflows/agents_as_tools_workflow.py)
```py
def orchestrator_agent() -> Agent:
    spanish_agent = Agent(
        name="spanish_agent",
        instructions="You translate the user's message to Spanish",
        handoff_description="An english to spanish translator",
    )

    french_agent = Agent(
        name="french_agent",
        instructions="You translate the user's message to French",
        handoff_description="An english to french translator",
    )

    italian_agent = Agent(
        name="italian_agent",
        instructions="You translate the user's message to Italian",
        handoff_description="An english to italian translator",
    )

    orchestrator_agent = Agent(
        name="orchestrator_agent",
        instructions=(
            "You are a translation agent. You use the tools given to you to translate."
            "If asked for multiple translations, you call the relevant tools in order."
            "You never translate on your own, you always use the provided tools."
        ),
        tools=[
            spanish_agent.as_tool(
                tool_name="translate_to_spanish",
                tool_description="Translate the user's message to Spanish",
            ),
            french_agent.as_tool(
                tool_name="translate_to_french",
                tool_description="Translate the user's message to French",
            ),
            italian_agent.as_tool(
                tool_name="translate_to_italian",
                tool_description="Translate the user's message to Italian",
            ),
        ],
    )
    return orchestrator_agent

```
<!--SNIPEND-->

## MCP servers

Temporal's durability does not extend to [MCP](https://modelcontextprotocol.io/) servers, which run independently of the
Workflow. The integration runs each MCP operation as an Activity and gives Workflow code a durable proxy for an MCP
Python SDK v2 server.

Install the optional MCP dependencies:

```bash
uv add "temporalio-openai-agents[mcp]"
```

### Register an MCP server

On the Worker, pass a dictionary of named OpenAI Agents SDK server factories to `OpenAIAgentsPlugin.mcp_servers`. The
factory and server transport run on the Worker, outside the Workflow sandbox.

[openai_agents/mcp_v2/run_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/mcp_v2/run_worker.py)
```python
from datetime import timedelta

from agents.mcp import MCPServerStreamableHttp
from temporalio.client import Client
from temporalio.openai_agents import ModelActivityParameters, OpenAIAgentsPlugin

MCP_SERVER_NAME = "StreamableHttpV2Server"

def streamable_http_server() -> MCPServerStreamableHttp:
    return MCPServerStreamableHttp(
        name=MCP_SERVER_NAME,
        params={"url": "http://localhost:8000/mcp"},
    )

client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                start_to_close_timeout=timedelta(seconds=60)
            ),
            mcp_servers={MCP_SERVER_NAME: streamable_http_server},
        ),
    ],
)
```

You can use stdio, streamable HTTP, in-process, or custom MCP v2 transports. Configure transport, connection, retry,
and message-handling behavior on the Worker-side `MCPServer`.

### Use the MCP server in a Workflow

Call `temporal_mcp_server()` with the registered name, then pass the returned proxy to the agent.

[openai_agents/mcp_v2/workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/mcp_v2/workflow.py)
```python
from agents import Agent, Runner
from temporalio.openai_agents.workflow import temporal_mcp_server

server = temporal_mcp_server("StreamableHttpV2Server")
agent = Agent(
    name="Assistant",
    instructions="Use the tools to answer the questions.",
    mcp_servers=[server],
)
result = await Runner.run(starting_agent=agent, input="What's the weather in Tokyo?")
```

Configure Workflow-facing behavior, including `tool_filter`, `require_approval`, `failure_error_function`, and metadata
resolvers, on `temporal_mcp_server()`. A callable `tool_filter` runs during Workflow replay, so it must be deterministic.
Do not perform I/O or depend on the system clock, randomness, mutable global state, or other external state from the
filter.

The proxy caches the tool list in Workflow state by default. Pass `cache_tools_list=False` to refresh it each time the
OpenAI Agents SDK lists tools. For a parameterless factory using the modern sessionless protocol, the Worker can reuse
an idle connection for up to five minutes. Set `mcp_connection_idle_timeout` on `OpenAIAgentsPlugin` to change that
period.

> **⚠️ Warning:**
>
> `temporal_mcp_server()` accepts a `factory_argument` that is passed to the registered factory. It is an Activity
> argument, so it is recorded in Workflow history and, without a payload codec, visible in the Web UI. Do not pass
> secrets, credentials, or API keys through it. Resolve them on the Worker inside the server factory.
>

Version 1.0.0 deprecates `StatelessMCPServerProvider`, `StatefulMCPServerProvider`, `stateless_mcp_server()`, and
`stateful_mcp_server()`. Replace provider objects passed through `mcp_server_providers` with named factories passed
through `mcp_servers`, and replace both Workflow helpers with `temporal_mcp_server()`.

### Hosted MCP tool

For a network-accessible server, `HostedMCPTool` uses an MCP client hosted by OpenAI, so there is nothing to register
on the Worker:

<!--SNIPSTART python-openai-agents-hosted-mcp-workflow-->
[openai_agents/hosted_mcp/workflows/simple_mcp_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/hosted_mcp/workflows/simple_mcp_workflow.py)
```py
@workflow.defn
class SimpleMCPWorkflow:
    @workflow.run
    async def run(
        self, question: str, server_url: str = "https://gitmcp.io/openai/codex"
    ) -> str:
        agent = Agent(
            name="Assistant",
            tools=[
                HostedMCPTool(
                    tool_config={
                        "type": "mcp",
                        "server_label": "gitmcp",
                        "server_url": server_url,
                        "require_approval": "never",
                    }
                )
            ],
        )

        result = await Runner.run(agent, question)
        return result.final_output

```
<!--SNIPEND-->

## Conversation history and human-in-the-loop

Because the agent loop runs inside a Workflow, conversation history and pending approvals have to be replay safe.

### Carry conversation history across turns

Hold the history in Workflow state and rebuild each turn's input from the previous result. `result.to_input_list()`
returns the conversation as input items, and `result.last_agent` records which agent a handoff left you on:

```python
self.input_items.append({"content": user_input, "role": "user"})
result = await Runner.run(self.current_agent, self.input_items)
self.input_items = result.to_input_list()
self.current_agent = result.last_agent
```

Workflow state is replay safe, so history survives Worker restarts within a run. `SQLiteSession` is not supported: it
keeps history in a local file, which no longer identifies one conversation once Workers are distributed.

### Handle long-running conversations

A chat-style Workflow accumulates history with every turn, and over a long session the event history can grow large
enough to hit Temporal's per-Workflow limit. Use
[Continue-as-New](/develop/python/workflows/continue-as-new) to start a fresh Execution carrying the history forward.

In this example each user turn arrives as a Workflow
[Update](/develop/python/workflows/message-passing#updates), so the caller gets the agent's reply back from the same
call. The `run` method waits until Temporal suggests continuing, drains in-flight handlers, then continues as new with
the accumulated state:

<!--SNIPSTART python-openai-agents-continue-as-new-workflow-->
[openai_agents/customer_service/workflows/customer_service_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/customer_service/workflows/customer_service_workflow.py)
```py
@workflow.run
async def run(
    self, customer_service_state: CustomerServiceWorkflowState | None = None
):
    await workflow.wait_condition(
        lambda: workflow.info().is_continue_as_new_suggested()
        and workflow.all_handlers_finished()
    )
    workflow.continue_as_new(
        CustomerServiceWorkflowState(
            printed_history=self.printed_history,
            current_agent_name=self.current_agent.name,
            context=self.context,
            input_items=self.input_items,
        )
    )

```
<!--SNIPEND-->

### Add human approval

An agent action that should not proceed unattended can pause for a person. A `HostedMCPTool` configured with
`require_approval` calls your `on_approval_request` callback, which runs in Workflow context—so it must be
deterministic, and it can wait on a Signal or Update to get the answer from outside:

<!--SNIPSTART python-openai-agents-hosted-mcp-approval-workflow-->
[openai_agents/hosted_mcp/workflows/approval_mcp_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/hosted_mcp/workflows/approval_mcp_workflow.py)
```py
def approval_callback(request: MCPToolApprovalRequest) -> MCPToolApprovalFunctionResult:
    """Simple approval callback that logs the request and approves by default.

    In a real application, user input would be provided through a UI or API.
    The approval callback executes within the Temporal workflow, so the application
    can use signals or updates to receive user input.
    """
    workflow.logger.info(f"MCP tool approval requested for: {request.data.name}")

    result: MCPToolApprovalFunctionResult = {"approve": True}
    return result

```
<!--SNIPEND-->

## Sandbox

> **⚠️ Caution:**
>
> Sandbox support is pre-release and may change before general availability.
>

`SandboxAgent` gives an agent a machine to work on: a shell to run commands in and a filesystem to read and write. The
plugin dispatches every sandbox operation—creating the session, each command, each read and write, PTY interaction, and
teardown—as its own Temporal Activity. Each one is individually retryable and visible in Workflow history, and the
session state is serialized with the Workflow, so a Worker restart part-way through a run resumes against the same
session.

Register a `SandboxClientProvider` for each backend you want to reach, under a unique name:

<!--SNIPSTART python-openai-agents-sandbox-worker-->
[openai_agents/sandbox/run_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/sandbox/run_worker.py)
```py
client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                start_to_close_timeout=timedelta(seconds=60)
            ),
            # The plugin registers one set of sandbox activities per
            # provider, prefixed with the provider name. Register several
            # providers to let one worker serve several backends.
            sandbox_clients=[
                SandboxClientProvider(SANDBOX_PROVIDER, UnixLocalSandboxClient()),
            ],
        ),
    ],
)
```
<!--SNIPEND-->

In the Workflow, `temporal_sandbox_client()` resolves a name to that backend and goes in the `RunConfig`:

<!--SNIPSTART python-openai-agents-sandbox-workflow-->
[openai_agents/sandbox/workflows/local_sandbox_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/sandbox/workflows/local_sandbox_workflow.py)
```py
@workflow.defn
class LocalSandboxWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        # A default SandboxAgent already carries the Filesystem, Shell, and
        # Compaction capabilities, so there are no tools to declare here.
        agent = SandboxAgent[None](
            name="Sandbox Assistant",
            instructions=(
                "You have a sandbox with a shell and a filesystem. Use it to do "
                "the work rather than answering from memory, then report what "
                "the commands returned."
            ),
        )

        result = await Runner.run(
            starting_agent=agent,
            input=prompt,
            run_config=RunConfig(
                sandbox=SandboxRunConfig(
                    # Must match the name registered on the worker.
                    client=temporal_sandbox_client(SANDBOX_PROVIDER),
                    options=UnixLocalSandboxClientOptions(),
                ),
            ),
        )
        return result.final_output_as(str, raise_if_incorrect_type=True)

```
<!--SNIPEND-->

The name becomes the prefix of that backend's Activity names, which is what lets several backends share one Worker. A
single Workflow can target more than one by calling `temporal_sandbox_client()` once per name. Names must match the
Worker's registration exactly.

The sample above uses `UnixLocalSandboxClient`, which runs commands on the Worker host and copies the Worker's entire
environment into each agent-run process. An agent can read values such as `OPENAI_API_KEY`. Command output returns as an
Activity result and is stored in Workflow history unless you protect payloads with a codec.

Use `UnixLocalSandboxClient` only for local development with trusted prompts. In production, register a remote client
such as `DaytonaSandboxClient` or `E2BSandboxClient` from `agents.extensions.sandbox` instead. Only the Worker changes;
the Workflow still names a provider.

## Streaming

> **⚠️ Caution:**
>
> Streaming is experimental and may change before general availability.
>

`Runner.run_streamed` works inside a Workflow. The model call runs as a streaming Activity that consumes
`Model.stream_response` and publishes each event to a [Workflow Stream](/workflow-streams) topic as the model produces
it, so an external client can watch a run live while it stays durable.

Set the topic on `ModelActivityParameters.streaming_topic` and host a `WorkflowStream` in the Workflow. The topic is
required: without it, `run_streamed` raises before scheduling any Activity.

<!--SNIPSTART python-openai-agents-streaming-workflow-->
[openai_agents/streaming/workflows/stream_text_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/streaming/workflows/stream_text_workflow.py)
```py
@workflow.defn
class StreamTextWorkflow:
    @workflow.init
    def __init__(self, input: StreamTextInput) -> None:
        # WorkflowStream requires construction from a method named __init__
        # (it checks its caller's frame and raises otherwise), and
        # @workflow.init is what makes the run argument — and the
        # stream_state it carries across continue-as-new — available here.
        self.stream = WorkflowStream(prior_state=input.stream_state)
        self.done = self.stream.topic(TOPIC_DONE, type=bool)

    @workflow.run
    async def run(self, input: StreamTextInput) -> str:
        agent = Agent(
            name="Joker",
            instructions="You are a helpful assistant.",
        )
        result = Runner.run_streamed(agent, input=input.prompt)

        # The workflow only sees these events once the activity returns, so
        # the loop just counts them. External subscribers receive them as the
        # activity publishes them.
        deltas = 0
        async for event in result.stream_events():
            if event.type == "raw_response_event" and isinstance(
                event.data, ResponseTextDeltaEvent
            ):
                deltas += 1
        workflow.logger.info("collected %d delta events", deltas)

        # In-band terminator so the subscriber can stop without racing the
        # workflow's completion, then a brief pause to let its next poll
        # deliver the tail of the stream — the log lives in workflow memory
        # and is gone once this run completes.
        self.done.publish(True)
        await workflow.sleep(DRAIN_INTERVAL)
        # final_output is typed Any and is None when a run ends without
        # message output, so assert the str this signature promises rather
        # than letting a None through.
        return result.final_output_as(str, raise_if_incorrect_type=True)

```
<!--SNIPEND-->

Subscribe from outside with `WorkflowStreamClient`:

<!--SNIPSTART python-openai-agents-streaming-client-->
[openai_agents/streaming/run_stream_text_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/streaming/run_stream_text_workflow.py)
```py
stream = WorkflowStreamClient.create(client, workflow_id)
converter = client.data_converter.payload_converter

# A single iterator over both topics — one subscriber, no cancellation race
# between concurrent ones. result_type=RawValue delivers the underlying
# Payload so heterogeneous topics can be decoded per item.topic. The loop
# ends on the in-band terminator, or by the iterator exhausting if the
# workflow reaches a terminal state without publishing one (e.g. on
# failure); either way handle.result() below surfaces the outcome.
last_sequence = -1
response_in_flight = False
async for item in stream.subscribe(
    [TOPIC_EVENTS, TOPIC_DONE], result_type=RawValue
):
    if item.topic == TOPIC_DONE:
        break
    # Subscribers receive native OpenAI events, not the agents-SDK
    # StreamEvent wrappers that stream_events() yields in the workflow.
    event: Any = converter.from_payload(item.data.payload, EVENT_TYPE)

    # Every event carries a sequence_number that starts at 0 per response,
    # so a number that does not advance means a new response is streaming.
    # That is a retry only if the previous one never completed: each turn
    # of a multi-turn run is its own response and restarts the count too.
    # The retry is an independently sampled answer rather than a
    # continuation, so mark the seam instead of letting the failed
    # attempt's partial text run into the new one. The workflow's return
    # value is unaffected — stream_events() there sees only the attempt
    # that succeeded.
    sequence = event.sequence_number
    if sequence <= last_sequence and response_in_flight:
        print("\n\n[model activity retried — output restarts here]\n")
    last_sequence = sequence
    response_in_flight = not isinstance(event, ResponseCompletedEvent)

    if isinstance(event, ResponseTextDeltaEvent):
        print(event.delta, end="", flush=True)
```
<!--SNIPEND-->

Two things to know:

- Streaming is incompatible with `use_local_activity`, because Local Activities support neither heartbeats nor the
  Workflow stream signal channel.
- Activity retries are visible to stream subscribers but not to `stream_events()`. An attempt that fails mid-response
  leaves its events on the stream and the retry publishes a second sequence, while `stream_events()` sees only the
  successful attempt's collected events.

## Tracing

OpenAI Agents SDK tracing works across Client, Workflow, and Activity boundaries, with the plugin propagating trace
context for you.

### OpenAI hosted traces

Hosted tracing needs no setup beyond the plugin. To start a trace on the Client—so the whole Workflow Execution is part
of a larger trace—open `plugin.tracing_context()` first:

```python
plugin = OpenAIAgentsPlugin()
client = await Client.connect("localhost:7233", plugins=[plugin])

with plugin.tracing_context():
    with trace("Customer support workflow"):
        result = await client.execute_workflow(
            CustomerSupportAgent.run,
            "Help me with my order",
            id="customer-support-123",
            task_queue="my-task-queue",
        )
```

`plugin.tracing_context()` is required when starting traces outside a Worker; without it the trace does not propagate
into the Workflow.

### OpenTelemetry

> **⚠️ Caution:**
>
> The OpenTelemetry integration is Public Preview and may change before general availability.
>

If you already collect traces with OpenTelemetry, the integration can emit the agent's spans through your pipeline, so
model calls, tools, and orchestration land in the same backend as the rest of your traces.

Install the additional dependencies:

```bash
uv add "temporalio-openai-agents[openinference]" opentelemetry-exporter-otlp
```

Then set a global replay-safe tracer provider before connecting the Client, and turn on instrumentation in the plugin:

```python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from temporalio.contrib.opentelemetry import create_tracer_provider

tracer_provider = create_tracer_provider()
tracer_provider.add_span_processor(
    SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
)
trace.set_tracer_provider(tracer_provider)

client = await Client.connect(
    "localhost:7233",
    plugins=[OpenAIAgentsPlugin(use_otel_instrumentation=True)],
)
```

The provider must come from `create_tracer_provider()`. It replays safely, exporting spans only when a Workflow actually
completes rather than on every replay, and generates deterministic span identifiers so they correlate across replays.
Passing any other provider raises a `ValueError`.

To call the OpenTelemetry API directly from Workflow code, allow the module through the Workflow sandbox and open an
agents-SDK span first, so your spans are parented rather than becoming roots:

```python
import opentelemetry.trace

from agents import custom_span
from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions

worker = Worker(
    client,
    task_queue="my-task-queue",
    workflows=[MyWorkflow],
    workflow_runner=SandboxedWorkflowRunner(
        SandboxRestrictions.default.with_passthrough_modules("opentelemetry")
    ),
)

# Inside the Workflow:
with custom_span("Workflow coordination"):
    tracer = opentelemetry.trace.get_tracer(__name__)
    with tracer.start_as_current_span("Custom workflow span"):
        ...
```

## Resources

- [OpenAI Agents SDK samples](https://github.com/temporalio/samples-python/tree/main/openai_agents) — runnable examples
  for the patterns in this guide.
- [`temporalio-openai-agents` README](https://github.com/temporalio/ai-integrations/tree/main/python/openai_agents)
  — the full plugin reference, including the complete feature-support matrix.
- [OpenAI Agents SDK for Python](https://openai.github.io/openai-agents-python/)
- [Temporal Plugins guide](/develop/plugins-guide) — the Plugin system this integration is built on, which you can also
  use to build your own integrations.

---

# OpenRouter integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/openrouter

[OpenRouter](https://openrouter.ai/) is a model gateway: one OpenAI-compatible API and one API key in front of hundreds
of models from many providers. It picks the provider and model for each request, falls back between them, and reports
what every response cost.

Temporal handles everything around those calls. Each call runs as an Activity, so it gets retries with backoff, a
timeout, and a durable record in Event History of its result, its cost, and how many attempts it took. The Workflow around the
Activities can fan out over a batch with bounded concurrency, survive a Worker crash without re-running finished
calls, and pause for hours until a person acts.

The division of labor is simple. OpenRouter decides *which provider and model* serve a request, in milliseconds.
Temporal decides *what happens over time*: waiting out a rate limit, surviving a crash, parking until someone raises a
budget, and keeping the record of what happened.

This integration is a sample pattern rather than a plugin: OpenRouter needs nothing inside Workflow code. Code snippets
in this guide are taken from the [OpenRouter samples](https://github.com/temporalio/samples-python/tree/main/openrouter).
Refer to the samples for the complete code, and to the
[TypeScript sample](https://github.com/temporalio/samples-typescript/tree/main/openrouter) for the same pattern in
TypeScript.

## Prerequisites

- This guide assumes you are already familiar with OpenRouter's
  [chat completions API](https://openrouter.ai/docs/quickstart).
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following
  [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal
  development server running.
- Create an [OpenRouter API key](https://openrouter.ai/settings/keys) and export it as `OPENROUTER_API_KEY` in the
  Worker's environment. The key stays in the Worker process; it is never part of Workflow input or Event History.

## Install

The samples call OpenRouter with the `openai` package pointed at OpenRouter's base URL, which is the setup OpenRouter
documents for OpenAI-compatible clients:

```bash
pip install temporalio openai
```

OpenRouter's official [`openrouter`](https://pypi.org/project/openrouter/) package works the same way. If you use it,
construct the client with `retry_config=RetryConfig("none", ...)`. By default it retries 5xx and connection errors for
up to an hour, which hides attempts from Temporal.

## Call OpenRouter from an Activity

Build one client for the Worker's lifetime with client-side retries turned off, and pass it to the Activity class. The
Activity makes exactly one HTTP call per attempt, so the Activity's Retry Policy is the only retry policy in play.
Event History records each Activity's result, attempt count, and last failure; the sample logs each attempt's model,
cost, and cache status from the Worker.

```python
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
    max_retries=0,
    timeout=60.0,
)
```

<!--SNIPSTART python-openrouter-call-activity-->
[openrouter/activities.py](https://github.com/temporalio/samples-python/blob/main/openrouter/activities.py)
```py
@activity.defn
async def call_openrouter(self, request: OpenRouterRequest) -> OpenRouterResult:
    """One chat completion. One HTTP call per attempt; Temporal retries."""
    # Heartbeat so a killed Worker is noticed after heartbeat_timeout
    # rather than after the full start_to_close_timeout.
    heartbeat_timeout = activity.info().heartbeat_timeout
    heartbeat_task = (
        asyncio.create_task(_heartbeat_forever(heartbeat_timeout / 2))
        if heartbeat_timeout
        else None
    )
    try:
        return await self._send(request)
    finally:
        if heartbeat_task:
            heartbeat_task.cancel()

async def _send(self, request: OpenRouterRequest) -> OpenRouterResult:
    extra_body: dict[str, Any] = {}
    if request.fallback_models:
        # OpenRouter tries these in order within the same request.
        extra_body["models"] = request.fallback_models
    elif request.model == "openrouter/auto":
        extra_body["plugins"] = [
            {"id": "auto-router", "cost_tier": request.cost_tier}
        ]
    model = request.fallback_models[0] if request.fallback_models else request.model

    try:
        raw = await self._client.chat.completions.with_raw_response.create(
            model=model,
            messages=[{"role": "user", "content": request.prompt}],
            extra_body=extra_body or None,
            extra_headers={
                # Ask OpenRouter to cache the successful response. A retry
                # of the byte-identical request within the TTL is served
                # from cache and billed at $0.
                "X-OpenRouter-Cache": "true",
                "X-OpenRouter-Cache-TTL": str(request.cache_ttl_seconds),
            },
        )
    except APIStatusError as e:
        error = _error_object(e.body)
        error.setdefault("message", e.message)
        raise_for_status(e.status_code, error, e.response.headers)
    # Connection errors and timeouts propagate as-is: Temporal retries them.

    payload = json.loads(raw.text)
    if isinstance(payload.get("error"), dict):
        # OpenRouter can return HTTP 200 with an error body and no choices
        # when the upstream provider failed after the request was accepted.
        error = _error_object(payload)
        raise_for_status(_error_code(error, 500), error, raw.headers)
    choices = payload.get("choices") or []
    choice_error = choices[0].get("error") if choices else None
    if isinstance(choice_error, dict):
        # Or a 200 with a partial answer and the provider's error on the
        # choice itself; a partial answer is not an answer.
        raise_for_status(_error_code(choice_error, 500), choice_error, raw.headers)
    if not choices:
        # No error and no answer: treat like a server error and retry.
        raise_for_status(500, {"message": "Response has no choices"}, raw.headers)

    usage = payload.get("usage") or {}
    cost = usage.get("cost")
    if not isinstance(cost, (int, float)):
        # OpenRouter reports cost on every response; if it is ever missing,
        # say so rather than pretending the call was free.
        activity.logger.warning("OpenRouter response has no usage.cost")
    result = OpenRouterResult(
        prompt=request.prompt,
        model=str(payload.get("model", model)),
        answer=_content_to_text((choices[0].get("message") or {}).get("content"))
        if choices
        else "",
        cost_usd=float(cost) if isinstance(cost, (int, float)) else None,
        generation_id=str(payload.get("id", "")),
        cache_status=raw.headers.get("x-openrouter-cache-status", ""),
    )
    activity.logger.info(
        "OpenRouter call completed: attempt=%d model=%s cost_usd=%s cache=%s id=%s",
        activity.info().attempt,
        result.model,
        "unknown" if result.cost_usd is None else f"{result.cost_usd:.6f}",
        result.cache_status or "-",
        result.generation_id,
    )

    if request.fail_once_after_call and activity.info().attempt == 1:
        # Demo hook: the Worker "crashes" after the response arrived. The
        # retry re-sends the identical request and gets a cache hit.
        raise ApplicationError(
            "Simulated failure after the response was received",
            type="SimulatedFailure",
        )

    return result

```
<!--SNIPEND-->

Three details in that Activity carry most of the value.

### Classify errors

OpenRouter's error codes tell you whether a retry can help. The Activity turns them into an `ApplicationError` with the
matching retry posture, and passes a `Retry-After` header through as the next retry delay:

| Status | Meaning | Retry? |
|---|---|---|
| 408, 429 | Timeout, rate limited (`Retry-After` may be set) | Yes, honoring `Retry-After` up to five minutes |
| 500, 502, 503, 524, 529 | Server error, model down, no provider available, edge timeout, provider overloaded | Yes |
| 400 | Bad request | No |
| 401 | Invalid API key | No |
| 402 with `error.metadata.limit_source` of `openrouter_in_flight_budget` | In-flight budget cap, transient | Yes, honoring `Retry-After` |
| 402, other | The account or API key is out of credits | No; raised as `OpenRouterOutOfCredits`, see [Pause when money runs out](#pause-when-money-runs-out) |
| 403 with `Key limit exceeded` | A per-key limit, as observed in practice | No; raised as `OpenRouterOutOfCredits` |
| 403, other | Moderation or permission block | No |

OpenRouter can also return HTTP 200 with an `error` object in the body and no `choices` when the upstream provider
failed after the request was accepted, or with a partial answer and an `error` on the choice. The Activity reads the
raw body first and classifies both cases by the code inside the error.

### Make retries free with response caching

An Activity is at-least-once. If a Worker dies after OpenRouter answered but before Temporal recorded the result,
Temporal retries the Activity and sends the request again. The Activity sends OpenRouter's
`X-OpenRouter-Cache: true` header, so OpenRouter serves the retried, byte-identical request from its response cache
and bills $0 for it. OpenRouter writes the cache shortly after the response completes; a retry that arrives before
that write lands is a `MISS` and is billed. The response header `X-OpenRouter-Cache-Status` reports `HIT` or `MISS`, and the sample returns it
with each result.

For this to work, nothing per-attempt may go in the request body. The attempt number belongs in heartbeat details and
logs, not in the prompt. Two identical requests in flight at the same time both miss the cache and both bill.

### Heartbeat

A 60-second model call with a 90-second `start_to_close_timeout` would take 90 seconds to fail over after a Worker
crash. The Activity heartbeats, and the Workflow sets `heartbeat_timeout=timedelta(seconds=10)`, so Temporal notices
the crash in seconds.

## Fan out a prompt batch

The `prompt_batch` sample runs one Activity per prompt under a semaphore, so a slow or failing prompt never blocks the
rest. The Workflow records a prompt whose Activity fails with a non-retryable error as skipped rather than failing the batch:

<!--SNIPSTART python-openrouter-prompt-batch-fan-out-->
[openrouter/prompt_batch/workflow.py](https://github.com/temporalio/samples-python/blob/main/openrouter/prompt_batch/workflow.py)
```py
semaphore = asyncio.Semaphore(batch.max_concurrency)
outcomes = await asyncio.gather(
    *(self._answer(prompt, batch, semaphore) for prompt in batch.prompts)
)
```
<!--SNIPEND-->

Each result carries the concrete model OpenRouter's [Auto Router](https://openrouter.ai/docs/guides/routing/routers/auto-router)
chose, OpenRouter's reported `usage.cost`, the generation ID, and the cache status. The batch's `reported_cost_usd`
sums what OpenRouter reported on each prompt's final, successful attempt, and `unknown_cost_count` is how many results
had no cost. It is not a bill: an attempt that was billed but whose response never reached Temporal is not in it.
OpenRouter's dashboard is the source of truth for spend.

Each Activity adds a few events to Event History, and every answer is part of the Workflow result payload. The sample
caps a batch at 100 prompts. For larger batches, start one Workflow per slice, or use the
[sliding window](https://github.com/temporalio/samples-python/tree/main/batch_sliding_window) pattern with
Continue-As-New.

## Pause when money runs out

The `budget_gate` sample is the same batch, except that it pauses instead of failing when money runs out, and resumes
when a person raises the budget. Two things can pause it:

- A soft budget in the Workflow input, checked against the cost OpenRouter reports on every response. The Workflow
  reserves an estimate per in-flight call and parks a prompt when `spent + reserved + estimate` would exceed the
  budget.
- OpenRouter refusing the call for lack of credits: a 402 for the account or the API key, or the 403 `Key limit
  exceeded` seen in practice for a per-key limit. The Activity raises both as the non-retryable
  `OpenRouterOutOfCredits`; the Workflow catches that type and parks the prompt instead of skipping it.

Either way the prompt waits on `workflow.wait_condition`, which costs nothing while it waits and survives Worker
restarts. A `raise_budget` Update wakes every parked prompt. A `spend_report` Query shows the spend so far, the
reservations, the per-prompt ledger, and the parked prompts with the reason for each:

<!--SNIPSTART python-openrouter-budget-gate-handlers-->
[openrouter/budget_gate/workflow.py](https://github.com/temporalio/samples-python/blob/main/openrouter/budget_gate/workflow.py)
```py
@workflow.update
def raise_budget(self, new_budget_usd: float) -> SpendReport:
    """Raise the soft budget and wake every parked prompt.

    Send the current budget unchanged to resume after topping up credits
    in the OpenRouter dashboard.
    """
    self._budget_usd = new_budget_usd
    self._budget_version += 1
    return self.spend_report()

@raise_budget.validator
def validate_raise_budget(self, new_budget_usd: float) -> None:
    if not math.isfinite(new_budget_usd):
        raise ValueError("The budget must be a finite number.")
    if new_budget_usd < self._budget_usd:
        raise ValueError(
            f"New budget ${new_budget_usd} is below the current budget "
            f"${self._budget_usd}; the budget can only go up."
        )

@workflow.query
def spend_report(self) -> SpendReport:
    return SpendReport(
        budget_usd=self._budget_usd,
        spent_usd=round(self._spent_usd, 6),
        reserved_usd=round(self._reserved_usd, 6),
        completed=len(self._ledger),
        paused=dict(self._paused),
        ledger=list(self._ledger),
    )

```
<!--SNIPEND-->

<!--SNIPSTART python-openrouter-budget-gate-pause-->
[openrouter/budget_gate/workflow.py](https://github.com/temporalio/samples-python/blob/main/openrouter/budget_gate/workflow.py)
```py
async def _reserve(self, prompt: str, estimate: float) -> bool:
    """Reserve `estimate` against the budget, parking until it fits."""

    def fits() -> bool:
        # Tolerance absorbs float accumulation; see BUDGET_TOLERANCE_USD.
        return (
            self._spent_usd + self._reserved_usd + estimate
            <= self._budget_usd + BUDGET_TOLERANCE_USD
        )

    # Loop rather than check once: when the budget is raised, every parked
    # prompt is woken before any of them runs, so each must re-check after
    # waking in case an earlier one already took the new headroom. (Each
    # re-park starts a new timer for the remaining time; fine at this scale.)
    while not fits():
        workflow.logger.info(
            "Soft budget reached (spent $%.6f of $%.6f); pausing %r",
            self._spent_usd,
            self._budget_usd,
            prompt,
        )
        if not await self._park(prompt, "soft_budget_exhausted", fits):
            return False
    self._reserved_usd += estimate
    return True

async def _park(self, prompt: str, reason: str, until: Callable[[], bool]) -> bool:
    """Durable pause until `until()` holds or the batch deadline passes.

    Survives Worker restarts and can wait for hours. Returns False when the
    deadline passed first.
    """
    if until():
        # Nothing to wait for (a top-up already landed during the call).
        return True
    remaining = self._deadline - workflow.now()
    if remaining <= timedelta(0):
        return False
    self._paused[prompt] = reason
    try:
        await workflow.wait_condition(until, timeout=remaining)
        return True
    except asyncio.TimeoutError:
        return False
    finally:
        self._paused.pop(prompt, None)

```
<!--SNIPEND-->

The validator rejects lowering the budget. Sending the current budget unchanged is how an operator says "I topped up
credits at OpenRouter"; it bumps a version counter that prompts parked on an out-of-credits error are waiting for. If
nobody acts before the batch's approval deadline (one deadline shared by every parked prompt), the batch completes
with the remaining prompts listed as skipped.

The soft budget is a soft budget. It counts what OpenRouter reported on final attempts, charging the estimate when a
response carries no cost, so it tracks reported spend rather than the bill. The cost of a call is only known after the
response, and reservations count against the budget, so overshoot is at most `max_concurrency` times how far a
call's real cost exceeds the estimate.
To bound one call's cost, set `provider.max_price` in the request. The hard cap is the credit limit on the OpenRouter
API key, which is what produces the out-of-credits error.

Interacting with a paused batch from the Temporal CLI:

```bash
temporal workflow query --workflow-id <id> --type spend_report
temporal workflow update execute --workflow-id <id> --name raise_budget --input '0.05'
```

## Routing: OpenRouter fallbacks and Temporal retries

OpenRouter and Temporal both retry, at different time scales, for different reasons. Use both.

| Concern | Where it lives |
|---|---|
| Provider outage or rate limit on one provider, within a request | OpenRouter: Auto Router, provider preferences, or a `models` list tried in order |
| Which model answers a given prompt | OpenRouter: `openrouter/auto` with a `cost_tier`, or an explicit model slug |
| A 429 with a 30-second `Retry-After` | Temporal: the Activity retries after 30 seconds, visibly |
| Worker crash mid-call | Temporal: Heartbeat Timeout, retry, cache hit |
| Waiting hours for a human to raise a budget or add credits | Temporal: `wait_condition` and an Update |
| Record of each call's result, cost, and retries | Temporal Event History (result, attempt count, last failure), Worker logs per attempt, OpenRouter's generation IDs |

Passing `models: ["a/first", "b/second"]` replaces the Auto Router: OpenRouter tries the list in order within one
request, and the response's `model` field reports which one answered.

## Use OpenRouter with the OpenAI Agents SDK plugin

OpenRouter speaks the OpenAI Chat Completions API, so the [OpenAI Agents SDK integration](openai-agents) can use it as
its model provider with no custom code. Point the stock `OpenAIProvider` at OpenRouter, turn off client retries, and
select Chat Completions:

<!--SNIPSTART python-openai-agents-openrouter-provider-->
[openai_agents/model_providers/run_openrouter_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openrouter_worker.py)
```py
def openrouter_provider() -> OpenAIProvider:
    """OpenAI Agents SDK model provider backed by OpenRouter.

    OpenRouter speaks the OpenAI Chat Completions API, so the stock provider
    works once it is pointed at OpenRouter's base URL. Client retries are off:
    the plugin runs each model call as a Temporal Activity, and Temporal owns
    the retries.
    """
    default_headers: dict[str, str] = {}
    # Optional app attribution for OpenRouter's rankings.
    if referer := os.getenv("OPENROUTER_HTTP_REFERER"):
        default_headers["HTTP-Referer"] = referer
    if title := os.getenv("OPENROUTER_APP_TITLE"):
        default_headers["X-OpenRouter-Title"] = title

    client = AsyncOpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=os.environ["OPENROUTER_API_KEY"],
        max_retries=0,
        default_headers=default_headers or None,
    )
    # These samples use Chat Completions, OpenRouter's primary endpoint; the
    # Agents SDK defaults to the Responses API, which OpenRouter also offers.
    return OpenAIProvider(openai_client=client, use_responses=False)

```
<!--SNIPEND-->

Pass the provider to `OpenAIAgentsPlugin(model_provider=...)` and set the agent's `model` to `openrouter/auto` or any
OpenRouter model slug. The plugin runs each model call as an Activity, so the Activity Retry Policy applies to
OpenRouter calls the same way it does to OpenAI calls. Tools that run inside the Workflow must be `async`: the Agents
SDK runs sync tools in a thread, which the Workflow sandbox does not allow.

## Samples

- [openrouter/prompt_batch](https://github.com/temporalio/samples-python/tree/main/openrouter/prompt_batch): fan out a
  batch with the Auto Router, with a `--fail-once` flag that shows a retry served from cache at $0.
- [openrouter/budget_gate](https://github.com/temporalio/samples-python/tree/main/openrouter/budget_gate): the batch
  that pauses on a soft budget or on 402 and resumes on `raise_budget`.
- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers#openrouter):
  OpenRouter as the model provider for an OpenAI Agents SDK agent.
- [samples-typescript/openrouter](https://github.com/temporalio/samples-typescript/tree/main/openrouter): the prompt
  batch in TypeScript.

---

# Strands Agents integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/strands-agents

> Run Strands Agents AI workflows with durable execution using the Temporal Python SDK and Strands plugin.

Temporal's integration with [Strands Agents](https://strandsagents.com/) is an [SDK Plugin](/develop/plugins-guide) that
gives your Strands agents [Durable Execution](/temporal#durable-execution) via the Temporal platform. The plugin routes
model invocations, tool calls, MCP tool calls, and hooks through Temporal Activities, so every step your agent takes is
recorded in Workflow history and can survive crashes, restarts, and infrastructure failures.

> **Public Preview**

Code snippets in this guide are taken from the
[Strands Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/strands_plugin). Refer to the
samples for the complete code.

## Get started

Install the plugin, then run a minimal Strands agent inside a Temporal Workflow.

### Prerequisites

- This guide assumes you are already familiar with Strands Agents. If you are not, refer to the
  [Strands Agents documentation](https://strandsagents.com/) for more details.
- If you are new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the
  [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following the
  [Set up your local development environment](/develop/python/set-up-your-local-python) guide. Leave the Temporal
  development server running if you want to test your code locally.

### Install the plugin

Install the standalone `temporalio-strands-agents` package:

```bash
uv add temporalio-strands-agents
```

or with pip:

```bash
pip install temporalio-strands-agents
```

The package installs the Temporal Python SDK and Strands Agents dependencies. Refer to the
[package requirements](https://github.com/temporalio/ai-integrations/blob/main/python/strands_agents/pyproject.toml)
for supported Python and dependency versions.

Import `StrandsPlugin`, `TemporalAgent`, and `TemporalMCPClient` from `temporalio.strands_agents`. Import Workflow helpers
such as `activity_as_tool` and `activity_as_hook` from `temporalio.strands_agents.workflow`.

### Run a Strands agent with Durable Execution

The following example runs a Strands agent inside a Temporal Workflow. Model calls execute as Temporal Activities, which
means they get automatic retries, timeouts, and durable execution. If the Worker process crashes mid-conversation,
Temporal replays the Workflow and resumes from the last completed Activity.

**1. Define the Workflow**

Create a Workflow that holds a `TemporalAgent` and invokes it with a prompt. The `start_to_close_timeout` sets the
maximum time each model call Activity can run:

<!--SNIPSTART python-strands-hello-world-workflow-->
[strands_plugin/hello_world/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hello_world/workflow.py)
```py
from datetime import timedelta

from temporalio import workflow
from temporalio.strands_agents import TemporalAgent

@workflow.defn
class HelloWorldWorkflow:
    def __init__(self) -> None:
        self.agent = TemporalAgent(start_to_close_timeout=timedelta(seconds=60))

    @workflow.run
    async def run(self, prompt: str) -> str:
        result = await self.agent.invoke_async(prompt)
        return str(result)

```
<!--SNIPEND-->

> **⚠️ Caution:**
>
> Inside a Workflow, always call `agent.invoke_async(message)`, not `agent(message)`. The synchronous form spawns a worker
> thread, which the Workflow sandbox blocks.
>

**2. Start a Worker**

Create a Worker that registers the Workflow and the `StrandsPlugin`. The plugin automatically registers the Activities
that handle model calls:

<!--SNIPSTART python-strands-hello-world-worker-->
[strands_plugin/hello_world/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hello_world/run_worker.py)
```py
import asyncio
import os

from temporalio.client import Client
from temporalio.strands_agents import StrandsPlugin
from temporalio.worker import Worker

from strands_plugin.hello_world.workflow import HelloWorldWorkflow

async def main() -> None:
    plugin = StrandsPlugin()
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        plugins=[plugin],
    )

    worker = Worker(
        client,
        task_queue="strands-hello-world",
        workflows=[HelloWorldWorkflow],
    )
    print("Worker started. Ctrl+C to exit.")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

**3. Run the Workflow**

Start the Workflow from a separate client script. This example sends the prompt "Write a haiku about durable execution"
and prints the agent's response:

<!--SNIPSTART python-strands-hello-world-run-workflow-->
[strands_plugin/hello_world/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hello_world/run_workflow.py)
```py
import asyncio
import os

from temporalio.client import Client

from strands_plugin.hello_world.workflow import HelloWorldWorkflow

async def main() -> None:
    client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"))

    result = await client.execute_workflow(
        HelloWorldWorkflow.run,
        "Write a haiku about durable execution.",
        id="strands-hello-world",
        task_queue="strands-hello-world",
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Build the agent

Customize which model provider your agent uses, add tools that run as Activities, subscribe to lifecycle events with
hooks, and connect to MCP servers.

### Choose and configure models

By default, `StrandsPlugin` uses Strands' own default model (`BedrockModel`). To use a different model, pass a `models`
mapping to `StrandsPlugin` on the Worker. When you provide a custom `models` mapping, each `TemporalAgent` must specify
which model to use by name.

Each entry in the mapping pairs a name with a factory function that creates a model provider (such as `AnthropicModel`
or `BedrockModel`). The provider is created on first use and reused for the Worker's lifetime:

```python
from strands.models.anthropic import AnthropicModel
from strands.models.bedrock import BedrockModel

# Workflow
@workflow.defn
class MultiModelWorkflow:
    def __init__(self) -> None:
        self.agent_a = TemporalAgent(
            model="claude",
            start_to_close_timeout=timedelta(seconds=60),
        )
        self.agent_b = TemporalAgent(
            model="bedrock",
            start_to_close_timeout=timedelta(seconds=60),
        )

# Worker
Worker(..., plugins=[StrandsPlugin(models={
    "claude": lambda: AnthropicModel(client_args={"api_key": "..."}),
    "bedrock": lambda: BedrockModel(),
})])
```

Each `TemporalAgent` carries its own Activity options (timeouts, retry policy, task queue, streaming topic) and
dispatches to a shared model Activity, which resolves the model name against the registered factories at runtime. A
model name not present in the `models` mapping raises `ValueError` inside the Activity.

### Run non-deterministic tools as Activities

Strands tools that perform I/O, access external services, or produce non-deterministic results need to run as Temporal
Activities rather than inline in the Workflow. Wrap each tool in an `@activity.defn` function, register the Activities
on the Worker, and pass them to the agent using `activity_as_tool`.

Define an Activity for the tool:

<!--SNIPSTART python-strands-tools-activity-->
[strands_plugin/tools/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/tools/workflow.py)
```py
@activity.defn
async def fetch_weather(city: str) -> dict:
    """Stub weather lookup — replace with a real HTTP call in production."""
    return {
        "city": city,
        "temperature_f": 72,
        "conditions": "sunny",
    }

```
<!--SNIPEND-->

Pass the Activity to the agent in the Workflow using `activity_as_tool`:

<!--SNIPSTART python-strands-tools-workflow-->
[strands_plugin/tools/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/tools/workflow.py)
```py
@workflow.defn
class ToolsWorkflow:
    def __init__(self) -> None:
        self.agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=60),
            tools=[
                letter_counter,
                activity_as_tool(
                    fetch_weather,
                    start_to_close_timeout=timedelta(seconds=30),
                ),
                activity_as_tool(
                    environment_activity,
                    start_to_close_timeout=timedelta(seconds=30),
                ),
            ],
        )

    @workflow.run
    async def run(self, prompt: str) -> str:
        result = await self.agent.invoke_async(prompt)
        return str(result)

```
<!--SNIPEND-->

Register the Activity functions on the Worker:

<!--SNIPSTART python-strands-tools-worker-->
[strands_plugin/tools/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/tools/run_worker.py)
```py
import asyncio
import os

from temporalio.client import Client
from temporalio.strands_agents import StrandsPlugin
from temporalio.worker import Worker

from strands_plugin.tools.workflow import (
    ToolsWorkflow,
    environment_activity,
    fetch_weather,
)

async def main() -> None:
    plugin = StrandsPlugin()
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        plugins=[plugin],
    )

    worker = Worker(
        client,
        task_queue="strands-tools",
        workflows=[ToolsWorkflow],
        activities=[fetch_weather, environment_activity],
    )
    print("Worker started. Ctrl+C to exit.")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

If you are using built-in `strands_tools`, wrap them in a thin async function decorated with `@activity.defn` so they
run as Temporal Activities.

### React to agent lifecycle events

Strands' [hook system](https://strandsagents.com/docs/user-guide/concepts/agents/hooks/) lets you subscribe callbacks to events in the agent lifecycle, such
as invocation start/end, model call before/after, tool call before/after, and message added. Use hooks to add logging,
metrics, or custom logic at each stage.

Pass `hooks=[MyHookProvider()]` to `TemporalAgent`. Hook callbacks fire in Workflow context, so deterministic callbacks
work without any extra setup.

For callbacks that need I/O (audit logging, metrics, alerting), use `activity_as_hook` to dispatch the work as a
Temporal Activity. The following example shows both patterns in one `HookProvider`. The `_record` callback runs in
Workflow context (deterministic), while `persist_tool_call` runs as an Activity (I/O-safe):

<!--SNIPSTART python-strands-hooks-activity-->
[strands_plugin/hooks/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hooks/workflow.py)
```py
@activity.defn
async def persist_tool_call(tool_name: str) -> None:
    # In production, write to a database / S3 / your audit pipeline.
    activity.logger.info(f"audit: tool {tool_name} completed")

```
<!--SNIPEND-->

<!--SNIPSTART python-strands-hooks-provider-->
[strands_plugin/hooks/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hooks/workflow.py)
```py
class AuditHook(HookProvider):
    def __init__(self) -> None:
        self.fired: list[str] = []

    def register_hooks(self, registry: HookRegistry, **kwargs: object) -> None:
        registry.add_callback(AfterToolCallEvent, self._record)
        registry.add_callback(
            AfterToolCallEvent,
            activity_as_hook(
                persist_tool_call,
                activity_input=lambda event: event.tool_use["name"],
                start_to_close_timeout=timedelta(seconds=15),
            ),
        )

    def _record(self, event: AfterToolCallEvent) -> None:
        self.fired.append(event.tool_use["name"])

```
<!--SNIPEND-->

> **⚠️ Caution:**
>
> Hook callbacks run in Workflow context, so they must be
> [deterministic](/develop/python/workflows/basics#workflow-logic-requirements). Do not use `time.time()`, `uuid.uuid4()`,
> or I/O inside hook callbacks. Use `activity_as_hook` for anything that requires I/O.
>

The `activity_input` parameter extracts serializable values from the event to pass as the Activity's input. Use a
dataclass or Pydantic model for multiple values. This is needed because hook events hold references to `Agent`,
`AgentTool` instances, and other objects that cannot cross the Activity boundary.

### Connect to MCP servers

If your agent needs access to tools provided by an [MCP](https://modelcontextprotocol.io/) server, configure the MCP
clients on the Worker and reference them by name in the Workflow.

`StrandsPlugin(mcp_clients=...)` takes a mapping of `name` to `MCPClient` factory, mirroring the `models` pattern. The
plugin registers a per-server Activity and connects at Worker startup to enumerate available tools. In the Workflow,
`TemporalMCPClient(server="name")` is a handle that references the server by name and carries per-call Activity options.

Define the Workflow with a `TemporalMCPClient`:

<!--SNIPSTART python-strands-mcp-workflow-->
[strands_plugin/mcp/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/mcp/workflow.py)
```py
from datetime import timedelta

from temporalio import workflow
from temporalio.strands_agents import TemporalAgent, TemporalMCPClient

@workflow.defn
class MCPWorkflow:
    def __init__(self) -> None:
        echo = TemporalMCPClient(
            server="echo",
            start_to_close_timeout=timedelta(seconds=30),
        )
        self.agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=60),
            tools=[echo],
        )

    @workflow.run
    async def run(self, prompt: str) -> str:
        result = await self.agent.invoke_async(prompt)
        return str(result)

```
<!--SNIPEND-->

Register the MCP client factory on the Worker:

<!--SNIPSTART python-strands-mcp-worker {"selectedLines": ["6-10", "17-25", "28-41"]}-->
[strands_plugin/mcp/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/mcp/run_worker.py)
```py
# ...
from mcp import StdioServerParameters, stdio_client
from strands.tools.mcp.mcp_client import MCPClient
from temporalio.client import Client
from temporalio.strands_agents import StrandsPlugin
from temporalio.worker import Worker
# ...
def _make_echo_client() -> MCPClient:
    return MCPClient(
        lambda: stdio_client(
            StdioServerParameters(
                command=sys.executable,
                args=[str(ECHO_SERVER)],
            )
        )
    )
# ...
async def main() -> None:
    plugin = StrandsPlugin(mcp_clients={"echo": _make_echo_client})
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        plugins=[plugin],
    )

    worker = Worker(
        client,
        task_queue="strands-mcp",
        workflows=[MCPWorkflow],
    )
    print("Worker started. Ctrl+C to exit.")
    await worker.run()
```
<!--SNIPEND-->

Each factory returns a fully configured `MCPClient`, so you can pass options like `tool_filters`, `prefix`,
`elicitation_callback`, or `tasks_config` to it.

> **ℹ️ Info:**
>
> The plugin connects to each MCP server once at Worker startup to enumerate tools. The schema is frozen for the Worker's
> lifetime. Restart Workers to pick up MCP server changes. If a server is unavailable at startup, the Worker fails to
> start.
>

## Interact with the agent

Control the shape of agent responses, stream output in real time, and pause the agent for human approval.

### Add human approval gates

Some agent actions, such as deleting resources or sending messages, may require human approval before proceeding.
Strands offers two ways to interrupt an agent and wait for a response. Both work with the plugin.

In each case, `agent.invoke_async()` returns `AgentResult(stop_reason="interrupt", interrupts=[...])` instead of
raising. Pair this with a Signal handler that supplies responses, then resume by calling
`agent.invoke_async(responses)`.

#### Interrupt from a hook

A hook on an interruptible event such as `BeforeToolCallEvent` can pause the agent by calling
`event.interrupt(name, reason=...)`. The hook runs in Workflow context, so it must be deterministic.

Define the approval hook:

<!--SNIPSTART python-strands-human-in-the-loop-hook-->
[strands_plugin/human_in_the_loop/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/human_in_the_loop/workflow.py)
```py
class ApprovalHook(HookProvider):
    def register_hooks(self, registry: HookRegistry, **kwargs: object) -> None:
        registry.add_callback(BeforeToolCallEvent, self._gate)

    def _gate(self, event: BeforeToolCallEvent) -> None:
        if event.tool_use["name"] != "delete_file":
            return
        approval = event.interrupt(
            "approval",
            reason=f"approve delete of {event.tool_use['input']['path']}?",
        )
        if approval != "approve":
            event.cancel_tool = "denied"

```
<!--SNIPEND-->

The Workflow waits for a Signal carrying the approval response, then resumes the agent:

<!--SNIPSTART python-strands-human-in-the-loop-workflow-->
[strands_plugin/human_in_the_loop/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/human_in_the_loop/workflow.py)
```py
@workflow.defn
class HumanInTheLoopWorkflow:
    def __init__(self) -> None:
        self.agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=60),
            tools=[delete_file],
            hooks=[ApprovalHook()],
        )
        self._approval: str | None = None
        self._pending_reason: str | None = None

    @workflow.signal
    def approve(self, response: str) -> None:
        self._approval = response

    @workflow.query
    def pending_approval(self) -> str | None:
        return self._pending_reason

    @workflow.run
    async def run(self, prompt: str) -> str:
        result = await self.agent.invoke_async(prompt)
        while result.stop_reason == "interrupt":
            interrupts = list(result.interrupts or [])
            self._pending_reason = interrupts[0].reason if interrupts else None
            await workflow.wait_condition(lambda: self._approval is not None)
            response = self._approval
            self._approval = None
            self._pending_reason = None
            responses: list[InterruptResponseContent] = [
                {"interruptResponse": {"interruptId": i.id, "response": response}}
                for i in interrupts
            ]
            result = await self.agent.invoke_async(responses)
        return str(result)

```
<!--SNIPEND-->

#### Interrupt from a tool

A `@strands.tool` function can raise `InterruptException(Interrupt(...))` directly. The agent stops with the interrupt,
and the Workflow handles the resume the same way as for hooks:

```python
from strands import tool
from strands.interrupt import Interrupt, InterruptException

@tool
def delete_thing(name: str) -> str:
    raise InterruptException(
        Interrupt(id=f"delete:{name}", name="approval", reason=f"delete {name}?")
    )
```

The same approach works from an `activity_as_tool`-wrapped Activity. The plugin's failure converter preserves the
`Interrupt` payload across the Activity boundary, so `AgentResult.interrupts` is populated the same way.

Define the Activity that raises the interrupt:

<!--SNIPSTART python-strands-activity-interrupt-activity-->
[strands_plugin/activity_interrupt/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/activity_interrupt/workflow.py)
```py
@activity.defn
async def delete_thing(name: str) -> str:
    if name not in _APPROVED:
        _APPROVED.add(name)
        raise InterruptException(
            Interrupt(
                id=f"delete:{name}",
                name="approval",
                reason=f"approve delete of protected resource '{name}'?",
            )
        )
    return f"deleted {name}"

```
<!--SNIPEND-->

> **⚠️ Caution:**
>
> Activity-tool interrupts rely on the plugin's failure converter, which is installed via the client's data converter.
> Attach `StrandsPlugin` to the **client** (not just the Worker) for Activity-tool interrupts to work.
>

Workers built from that client pick up the plugin automatically:

<!--SNIPSTART python-strands-activity-interrupt-worker-->
[strands_plugin/activity_interrupt/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/activity_interrupt/run_worker.py)
```py
import asyncio
import os

from temporalio.client import Client
from temporalio.strands_agents import StrandsPlugin
from temporalio.worker import Worker

from strands_plugin.activity_interrupt.workflow import (
    ActivityInterruptWorkflow,
    delete_thing,
)

async def main() -> None:
    plugin = StrandsPlugin()
    # The plugin MUST be on the client so its failure converter is installed.
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        plugins=[plugin],
    )

    worker = Worker(
        client,
        task_queue="strands-activity-interrupt",
        workflows=[ActivityInterruptWorkflow],
        activities=[delete_thing],
    )
    print("Worker started. Ctrl+C to exit.")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Return structured data from an agent

To have the agent return a typed object instead of free-form text, pass a `structured_output_model` to `TemporalAgent`.
The plugin defaults to the [`pydantic_data_converter`](/develop/python/data-handling/data-conversion), so Pydantic types
serialize cleanly across the Activity and Workflow boundary:

<!--SNIPSTART python-strands-structured-output-workflow-->
[strands_plugin/structured_output/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/structured_output/workflow.py)
```py
from datetime import timedelta

from pydantic import BaseModel, Field
from temporalio import workflow
from temporalio.strands_agents import TemporalAgent

class PersonInfo(BaseModel):
    name: str = Field(description="Name of the person")
    age: int = Field(description="Age of the person")
    occupation: str = Field(description="Occupation of the person")

@workflow.defn
class StructuredOutputWorkflow:
    def __init__(self) -> None:
        self.agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=60),
            structured_output_model=PersonInfo,
        )

    @workflow.run
    async def run(self, prompt: str) -> PersonInfo:
        result = await self.agent.invoke_async(prompt)
        assert isinstance(result.structured_output, PersonInfo)
        return result.structured_output

```
<!--SNIPEND-->

### Stream agent output to clients

For long-running agent calls, you may want to forward model output chunks to an external consumer as they arrive rather
than waiting for the full response.

Pass `streaming_topic="..."` to `TemporalAgent` and host a `WorkflowStream` on the Workflow. Each `StreamEvent` is
published from inside the model Activity. Subscribers read events through `WorkflowStreamClient`. Chunks are batched on
`streaming_batch_interval` (default 100 ms).

Define the Workflow with a `WorkflowStream` and a streaming topic:

<!--SNIPSTART python-strands-streaming-workflow-->
[strands_plugin/streaming/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/streaming/workflow.py)
```py
from datetime import timedelta

from temporalio import workflow
from temporalio.contrib.workflow_streams import WorkflowStream
from temporalio.strands_agents import TemporalAgent

@workflow.defn
class StreamingWorkflow:
    def __init__(self) -> None:
        self.stream = WorkflowStream()
        self.agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=60),
            streaming_topic="events",
        )

    @workflow.run
    async def run(self, prompt: str) -> str:
        result = await self.agent.invoke_async(prompt)
        return str(result)

```
<!--SNIPEND-->

Subscribe to the stream from a client:

<!--SNIPSTART python-strands-streaming-client-->
[strands_plugin/streaming/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/streaming/run_workflow.py)
```py
import asyncio
import os
from datetime import timedelta

from strands.types.streaming import StreamEvent
from temporalio.client import Client
from temporalio.contrib.workflow_streams import WorkflowStreamClient

from strands_plugin.streaming.workflow import StreamingWorkflow

async def main() -> None:
    client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"))
    workflow_id = "strands-streaming"

    handle = await client.start_workflow(
        StreamingWorkflow.run,
        "Count from 1 to 5, one number per sentence.",
        id=workflow_id,
        task_queue="strands-streaming",
    )

    async def consume() -> None:
        stream = WorkflowStreamClient.create(client, workflow_id)
        async for item in stream.subscribe(
            ["events"],
            from_offset=0,
            result_type=StreamEvent,
            poll_cooldown=timedelta(milliseconds=50),
        ):
            event: StreamEvent = item.data
            if "contentBlockDelta" in event:
                delta = event["contentBlockDelta"].get("delta", {})
                if "text" in delta:
                    print(delta["text"], end="", flush=True)
            elif "messageStop" in event:
                print()
                return

    consume_task = asyncio.create_task(consume())
    result = await handle.result()
    await asyncio.wait_for(consume_task, timeout=10.0)
    print(f"Final result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Run in production

Configure retry policies, handle long-running chat sessions, and add distributed tracing.

### Configure retries

`TemporalAgent` disables Strands' built-in `ModelRetryStrategy` so that retries are handled exclusively by Temporal.
Configure retries with `retry_policy` on `TemporalAgent` for model calls, and on the Activity options accepted by
`activity_as_tool`, `activity_as_hook`, and `TemporalMCPClient` for their respective calls:

```python
from temporalio.common import RetryPolicy

TemporalAgent(
    start_to_close_timeout=timedelta(seconds=60),
    retry_policy=RetryPolicy(maximum_attempts=3),
)
```

Passing `retry_strategy=...` to `TemporalAgent(...)` raises `ValueError`. Remove the argument (or pass
`retry_strategy=None`) and use `retry_policy` instead.

### Handle long-running chat sessions

A chat-style Workflow accumulates message history with every turn. Over a long session, the Workflow's event history can
grow large enough to hit Temporal's per-Workflow history limit. To avoid this, use
[Continue-as-New](/develop/python/workflows/continue-as-new) to start a fresh Workflow execution while carrying the
agent's message history forward as input.

In this example, each user turn arrives as a Workflow [Update](/develop/python/workflows/message-passing#updates), so
the caller gets the agent's reply back from the same call. The `run` method creates the agent, then waits until either
the chat ends or Temporal suggests continue-as-new. When it does, the Workflow drains any in-flight updates and starts a
fresh execution with the agent's accumulated messages:

<!--SNIPSTART python-strands-continue-as-new-workflow-->
[strands_plugin/continue_as_new/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/continue_as_new/workflow.py)
```py
import asyncio
from dataclasses import dataclass, field
from datetime import timedelta

from strands.types.content import Messages
from temporalio import workflow
from temporalio.strands_agents import TemporalAgent

@dataclass
class ChatInput:
    messages: Messages = field(default_factory=list)

@workflow.defn
class ChatWorkflow:
    def __init__(self) -> None:
        self._done = False
        self._lock = asyncio.Lock()
        self._agent: TemporalAgent | None = None

    @workflow.update
    async def turn(self, prompt: str) -> str:
        await workflow.wait_condition(lambda: self._agent is not None)
        async with self._lock:
            assert self._agent is not None
            result = await self._agent.invoke_async(prompt)
            return str(result).strip()

    @workflow.signal
    def end_chat(self) -> None:
        self._done = True

    @workflow.query
    def messages(self) -> Messages:
        return list(self._agent.messages) if self._agent else []

    @workflow.run
    async def run(self, input: ChatInput) -> None:
        self._agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=60),
            messages=list(input.messages),
        )

        await workflow.wait_condition(
            lambda: self._done or workflow.info().is_continue_as_new_suggested()
        )

        await workflow.wait_condition(workflow.all_handlers_finished)

        if not self._done:
            workflow.continue_as_new(ChatInput(messages=self._agent.messages))

```
<!--SNIPEND-->

### Add tracing with OpenTelemetry

To get distributed traces across model, tool, and MCP Activities, combine `StrandsPlugin` with the
[OpenTelemetry plugin](/develop/python/platform/observability#tracing). Register `OpenTelemetryPlugin` on the client and
`StrandsPlugin` on the Worker. Workers built from that client pick up the OpenTelemetry plugin automatically:

```python
import opentelemetry.trace
from temporalio.client import Client
from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider
from temporalio.strands_agents import StrandsPlugin
from temporalio.worker import Worker

opentelemetry.trace.set_tracer_provider(create_tracer_provider())

client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()])

Worker(
    client,
    task_queue="strands",
    workflows=[MyWorkflow],
    plugins=[StrandsPlugin()],
)
```

Set the tracer provider before connecting the client.

### Snapshots are not supported

`TemporalAgent.take_snapshot()` and `TemporalAgent.load_snapshot()` raise `NotImplementedError`. Temporal's event
history already persists Workflow state durably at a finer granularity than Strands snapshots, so snapshots are
redundant inside a Workflow.

### Samples

The [Strands Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/strands_plugin) demonstrate
all supported patterns end-to-end.

---

# Together AI integration - Python SDK

Source: https://docs.temporal.io/develop/python/integrations/together-ai

[Together AI](https://www.together.ai/) serves open-weight models such as gpt-oss, DeepSeek, Qwen, and GLM on a
serverless per-token API and on dedicated endpoints through an API compatible with the OpenAI Chat Completions API.

Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an
Activity. A call to Together AI gets retries with backoff, a timeout, and a durable record in Event History, and the
agent loop around it survives Worker restarts. Because Together AI speaks the OpenAI Chat Completions API, the
integration's model provider hook is all you need: a model provider that maps a model name such as
`together/openai/gpt-oss-120b` to Together AI's endpoint. One Worker can serve Together AI alongside other providers,
and each Workflow picks its provider through the model name it gives its agent.

This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the
[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers),
which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker.

## Prerequisites

- This guide assumes you are already familiar with the
  [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Together AI's
  [OpenAI-compatible API](https://docs.together.ai/docs/inference/openai-compatibility).
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following
  [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal
  development server running.
- Create a [Together AI API key](https://api.together.ai/settings/projects/~current/api-keys) and export it as
  `TOGETHER_API_KEY` in the Worker's environment. The key stays in the Worker process; it is never part of Workflow
  input or Event History.

## Install

```bash
uv add temporalio-openai-agents
```

The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. Together AI's own
[`together`](https://pypi.org/project/together/) package is not needed; the sample talks to Together AI through the
`openai` client.

## Configure the model provider

The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model
with tool calling:

<!--SNIPSTART python-openai-agents-together-provider-->
[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py)
```py
TOGETHER = OpenAICompatibleProvider(
    name="together",
    base_url="https://api.together.ai/v1",
    api_key_env="TOGETHER_API_KEY",
    default_model="openai/gpt-oss-120b",
)
```
<!--SNIPEND-->

Together AI model IDs follow the `<organization>/<model-name>` convention, for example `openai/gpt-oss-120b`; OpenAI's
own model names return a 404. The [serverless models](https://docs.together.ai/docs/serverless/models) catalog lists
each model's ID and whether it supports function calling and structured outputs. The sample's example,
`openai/gpt-oss-120b`, is an open-weight reasoning model that supports both; a Workflow names it
`together/openai/gpt-oss-120b`. A [dedicated endpoint](https://docs.together.ai/docs/dedicated-endpoints/overview) is
addressed the same way: its endpoint string, `<project-slug>/<endpoint-name>`, goes in the `model` field, and requests
go to `https://api-inference.together.ai/v1` instead of the serverless base URL. The sample's `together/` prefix only
reaches the serverless API: to call a dedicated endpoint, add a `PROVIDERS` entry with a distinct name and that base
URL, such as `together-dedicated`, and use its prefix in the model name.

A `ModelProvider` routes a `<provider>/<model>` name to the matching endpoint and returns the Chat Completions model
that the Agents SDK calls:

<!--SNIPSTART python-openai-agents-openai-compatible-model-provider-->
[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py)
```py
class OpenAICompatibleModelProvider(ModelProvider):
    """Routes ``<provider>/<model>`` names to the matching OpenAI-compatible endpoint.

    One Worker serves every provider whose API key is set, so Workflows on
    different providers run side by side on the same Task Queue. Client
    retries are off: the plugin runs each model call as a Temporal Activity,
    and the Activity's Retry Policy owns the retries.
    """

    def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None:
        self._providers = {provider.name: provider for provider in providers}
        self._clients: dict[str, AsyncOpenAI] = {}
        for provider in self._providers.values():
            api_key = os.environ.get(provider.api_key_env)
            if api_key:
                self._clients[provider.name] = AsyncOpenAI(
                    base_url=provider.base_url, api_key=api_key, max_retries=0
                )

    @property
    def configured(self) -> list[str]:
        """Names of the providers whose API key is set."""
        return sorted(self._clients)

    def get_model(self, model_name: str | None) -> Model:
        provider_name, _, model = (model_name or "").partition("/")
        provider = self._providers.get(provider_name)
        if provider is None or not model:
            # A mistake in the Workflow's input: fail the Activity without retrying.
            raise ApplicationError(
                f"Model name {model_name!r} must be <provider>/<model> with a "
                f"provider in {sorted(self._providers)}",
                non_retryable=True,
            )
        client = self._clients.get(provider_name)
        if client is None:
            raise ApplicationError(
                f"Set {provider.api_key_env} and restart the Worker to use {provider_name}",
                non_retryable=True,
            )
        # Chat Completions is the API all of these providers serve. The Agents
        # SDK defaults to the Responses API, which not every provider offers.
        return OpenAIChatCompletionsModel(model=model, openai_client=client)

```
<!--SNIPEND-->

Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name
for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off
(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a
hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents
SDK's default Responses API model, which Together AI's OpenAI compatibility guide does not cover.

## Configure the Worker

Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A
custom model provider must set a model Activity timeout; the sample uses 60 seconds. Serverless performance on Together
AI is best-effort, and a dedicated endpoint that was stopped or scaled down pays a cold start on its next request, so
size the timeout for the slowest call you expect rather than the median. The Heartbeat Timeout lets Temporal notice a
Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its
own.

<!--SNIPSTART python-openai-agents-openai-compatible-worker-->
[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py)
```py
# One provider for every endpoint whose API key is set. The model name on
# each Workflow's agent selects the endpoint.
model_provider = OpenAICompatibleModelProvider(PROVIDERS.values())
if not model_provider.configured:
    env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values())
    sys.exit(f"No provider API key is set; export at least one of {env_vars}.")
print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True)

client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                # A custom model provider must set a timeout. Raise it for
                # slow models or deployments that scale up from zero.
                start_to_close_timeout=timedelta(seconds=60),
                # The plugin heartbeats, so a Worker that dies mid-call is
                # noticed in seconds rather than after the full timeout.
                heartbeat_timeout=timedelta(seconds=10),
            ),
            model_provider=model_provider,
        ),
    ],
)

worker = Worker(
    client,
    task_queue="openai-agents-model-providers-task-queue",
    workflows=[OpenAICompatibleAgentWorkflow],
)
await worker.run()
```
<!--SNIPEND-->

The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider:
only the Worker talks to Together AI.

## Write the Workflow

The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one
Worker runs Workflows on Together AI and on other providers at the same time. Any model Together AI serves works,
prefixed with `together/`.

<!--SNIPSTART python-openai-agents-openai-compatible-workflow-->
[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py)
```py
@dataclass
class OpenAICompatibleAgentInput:
    prompt: str
    # "<provider>/<model>", such as "together/openai/gpt-oss-120b". The Worker's
    # model provider routes the agent's model calls to that provider.
    model: str

@workflow.defn
class OpenAICompatibleAgentWorkflow:
    @workflow.run
    async def run(self, input: OpenAICompatibleAgentInput) -> str:
        # Tools that run inside the Workflow must be async: the Agents SDK runs
        # sync tools in a thread, which the Workflow sandbox does not allow.
        @function_tool
        async def get_weather(city: str) -> str:
            workflow.logger.debug(f"Getting weather for {city}")
            return f"The weather in {city} is sunny."

        agent = Agent(
            name="Assistant",
            instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.",
            model=input.model,
            tools=[get_weather],
        )

        result = await Runner.run(agent, input.prompt)
        return result.final_output

```
<!--SNIPEND-->

Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings
for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0
forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release,
give such an agent an explicit `model_settings=ModelSettings()`.

Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow
sandbox does not allow. Tools that do I/O belong in Activities; see
[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools).

## Run the sample

Export your Together AI key and start the Worker:

```bash
export TOGETHER_API_KEY="your_api_key"
uv run openai_agents/model_providers/run_openai_compatible_worker.py
```

Then start a Workflow on Together AI in a second terminal:

```bash
uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model together/openai/gpt-oss-120b
```

Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on
the same Worker.

## Retries, rate limits, and timeouts

The plugin classifies Together AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry
Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as
well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as
does a model name whose provider has no key on the Worker. Event History records the Activity's result, its attempt
count, and its last failure, not every attempt. Individual failed attempts appear in the Worker's logs and, while the
Activity is still retrying, as the pending Activity's last failure when you describe the Workflow Execution.

Under high demand, Together AI's serverless API answers `429 Too Many Requests` or `503 Service Unavailable` and asks
clients to retry with exponential backoff, which is what the default Activity Retry Policy does; Together AI's
documentation does not describe a `Retry-After` header. Its [error codes](https://docs.together.ai/docs/error-codes)
page lists 500, 503, 504, 524, and 529 as retryable and the rest as request, credential, or billing problems to fix
first, including 402 when the account reaches its monthly spending limit and 403 when the prompt plus `max_tokens`
exceeds the model's context length. Serverless performance is best-effort; see
[rate limits](https://docs.together.ai/docs/serverless/rate-limits) for the current policy and
[provisioned throughput](https://docs.together.ai/docs/inference/provisioned-throughput) for reserved capacity with a
throughput and reliability commitment.

## What works through Together AI

Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the
Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter,
and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat
Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the
[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual.

## Together AI notes

- Dedicated endpoints are served at `https://api-inference.together.ai/v1`, not the serverless base URL. Deploy a model,
  wait for the deployment to reach `DEPLOYMENT_STATE_READY` (first-time provisioning takes minutes), then use the
  endpoint string as the model name. A running deployment keeps at least one replica; stopping it ends the cost and
  makes the next request pay a cold start. See [autoscaling](https://docs.together.ai/docs/dedicated-endpoints/scaling)
  for the replica bounds.
- Regional inference: `https://us-api.together.ai/v1` keeps serverless processing in the United States and
  `https://ca-api.together.ai/v1` in Canada, each with its own model list. See
  [geo-specific inference](https://docs.together.ai/docs/serverless/geo-specific-inference).
- `openai/gpt-oss-120b` is a reasoning model. Together AI accepts `reasoning_effort` (`low`, `medium`, or `high`; the
  default is `medium`), which the Agents SDK sends when an agent sets
  `ModelSettings(reasoning=Reasoning(effort="low"))`.
  Together AI returns the model's chain of thought in a `reasoning` field, which the Agents SDK's Chat Completions model
  does not read, so prior reasoning is not passed back between tool calls. See
  [reasoning](https://docs.together.ai/docs/inference/chat/reasoning).
- Structured outputs: an agent's `output_type` becomes a `response_format` JSON schema, which
  [Together AI supports](https://docs.together.ai/docs/inference/chat/structured-outputs) on models marked for it in the
  catalog. Together AI recommends also describing the schema in the prompt, and a response that stops with
  `finish_reason` of `length` leaves the JSON truncated, so give structured answers enough `max_tokens`.
- Model discovery: the `openai` client's `models.list()` works against Together AI, and the serverless catalog marks
  function calling and structured outputs per model. Together AI retires serverless models over time (for example,
  `openai/gpt-oss-20b` in September 2026), so check the [changelog](https://docs.together.ai/docs/changelog) before
  pinning a model in production.

## Samples

- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers):
  the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name.
- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history,
  and tracing with the plugin.

---

# Nexus - Python SDK

Source: https://docs.temporal.io/develop/python/nexus

> This section explains how to use Temporal Nexus with the Python SDK

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Temporal Nexus

- [Quickstart](/develop/python/nexus/quickstart)
- [Feature guide](/develop/python/nexus/feature-guide)
- [Standalone Operations](/develop/python/nexus/standalone-operations)

---

# Nexus feature guide - Python SDK

Source: https://docs.temporal.io/develop/python/nexus/feature-guide

> Use Temporal Nexus within the Python SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **⚠️ Currently in: Pre-release:**
> This feature guide includes the new Nexus developer experience: pre-release APIs for the [Temporal Operation Handler](/nexus/temporal-operation-handler) and [Nexus Standalone Activity](/nexus/standalone-activity). These APIs are experimental and subject to change.

Use [Temporal Nexus](/evaluate/features/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **💡 Tip:**
>
> New to Nexus? Start with the [Nexus Python quickstart](/develop/python/nexus/quickstart).
>

This page shows how to do the following:

- [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server)
- [Create caller and handler Namespaces](#create-caller-handler-namespaces)
- [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint)
- [Define the Nexus Service contract](#define-nexus-service-contract)
- [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers)
- [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service)
- [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server)
- [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from the [nexus_feature_guide](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/nexus/nexus_feature_guide) sample.

## Run the Temporal Development Server with Nexus enabled 

Prerequisites:

- [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/python/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli)
  (`v1.9.2` or higher recommended)
- [Install the latest Temporal Python SDK](https://learn.temporal.io/getting_started/python/dev_environment/#add-temporal-python-sdk-dependencies)
  (`v1.33.0` or higher recommended)

The first step in working with Temporal Nexus involves starting a Temporal Server with Nexus enabled.

```
temporal server start-dev \
  --dynamic-config-value history.enableUpdateCallbacks=true \
  --dynamic-config-value history.enableCHASMSignalBacklinks=true \
  --dynamic-config-value history.enableSignalWithStartFromWorkflow=true
```

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases.

The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`.

## Create caller and handler Namespaces 

Before setting up Nexus endpoints, create separate Namespaces for the caller and handler.

```
temporal operator namespace create --namespace my-target-namespace
temporal operator namespace create --namespace my-caller-namespace
```

`my-target-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `my-caller-namespace` to
call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls.

## Create a Nexus Endpoint to route requests from caller to handler 

After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests.

```
temporal operator nexus endpoint create \
  --name my-nexus-endpoint-name \
  --target-namespace my-target-namespace \
  --target-task-queue my-handler-task-queue
```

You can also use the Web UI to create the Namespaces and Nexus endpoint.

## Define the Nexus Service contract 

Defining a clear contract for the Nexus Service is crucial for smooth communication.

In this example, there is a service module that describes the Service and Operation names along with input/output types
for caller Workflows to use the Nexus Endpoint.

You can hand-write that module, but the preferred way is to generate it with [NexGen](/nexus/nexgen).
You write the contract once as a JSON or YAML definition file and run `nexgen` against it, and it emits the typed models,
runtime validators, and the Service definition itself.

This is what makes a Nexus Service polyglot. Both sides generate from the same definition file: the handler implements
the Service, the caller invokes its Operations, and neither side hand-writes request or response types. A Python handler
and a Go caller share no code, but they both run off that same service contract, so they interoperate with no
coordination between the teams beyond the contract itself.

The generated validators check every payload against the contract when a value is parsed off the wire and again when
it is serialized onto it, so bad data is rejected at the boundary rather than reaching your Workflow or Activity. A value validates
identically in every language, which is what lets a caller and a handler written in different languages trust the same
contract. See the [`chat.nexusrpc.yaml`](https://github.com/temporalio/nexgen/blob/main/samples/schemas/chat.nexusrpc.yaml)
sample contract and the [Definition files](https://github.com/temporalio/nexgen#definition-files) section of the
`nexgen` README for the file format.

## Develop a Nexus Service and Operation handlers 

Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract.
Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying
Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code.
Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database.
Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive
retryable errors, blocking all Operations from the caller to that Endpoint.

Every Operation is written with a [Temporal Operation Handler](/nexus/temporal-operation-handler). The
`@nexus.temporal_operation` decorator hands your start method three things: a context, a Client, and the Operation
input. What you do with the Client determines what backs the Operation:

- **Synchronous.** Return `nexus.TemporalOperationResult.sync(...)` and the Operation completes during the handler call.
  The caller has its result as soon as the call returns.
- **Asynchronous.** Call `start_workflow`, `start_activity`, or `start_workflow_update` on the Client. The handler
  returns as soon as that Execution has started, and the Operation stays open until the Execution finishes, which may be
  days later. Its result is delivered to the caller through the Nexus completion callback. This is what lets an
  Operation outlive the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

A handler can perform any number of synchronous side effects, such as sending a Signal, but at most one asynchronous
backing per invocation.

### Develop a Synchronous Nexus Operation handler

Return a synchronous result when the Operation can answer immediately. The handler computes the answer and returns it,
and the Operation completes during the call.

Handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking), and the whole
call has to finish inside the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

<!--SNIPSTART python-nexus-feature-guide-service-handler {"selectedLines": ["1-2", "17-26"]}-->
[python-sandbox/nexus/nexus_feature_guide/service_handler.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_feature_guide/service_handler.py)
```py
import nexusrpc
from temporalio import nexus
# ...
@nexusrpc.handler.service_handler(service=MyNexusService)
class MyNexusServiceHandler:
    @nexus.temporal_operation
    async def echo(
        self,
        _ctx: nexus.TemporalStartOperationContext,
        client: nexus.TemporalNexusClient,
        input: EchoInput,
    ) -> nexus.TemporalOperationResult[EchoOutput]:
        return nexus.TemporalOperationResult.sync(EchoOutput(message=input.message))
```
<!--SNIPEND-->

### Use the Temporal Client for Signals, Queries, and Updates

A common pattern is to reach a Workflow that is already running. Query it or Signal it from a synchronous Operation, or
use Signal-With-Start to make sure the Workflow exists before the Signal arrives. Those calls complete during the
handler call, so they have to finish inside the
[Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

Updates are the exception. Do not wait for one inside the handler. Start it with `start_workflow_update` and it backs
the Operation. The handler returns immediately, and the Operation completes when the Update does, however long it
takes.

The [nexus_messaging](https://github.com/temporalio/samples-python/tree/main/nexus_messaging)
sample shows a Nexus Service that Queries and Signals a running Workflow from synchronous Operations, and backs an
Operation with a Workflow Update.

The Client your handler receives is not an ordinary Temporal Client. It propagates
[bidirectional links](/nexus/execution-debugging#bi-directional-linking) and request IDs on every call, so the
caller-side and handler-side Executions are connected in the UI without wiring anything. Reach the Workflow Client
through `client.client` rather than constructing your own.

In this example the Workflow ID is derived from an identifier carried in the Operation input, so the caller only needs
the identifier it cares about:

<!--SNIPSTART python-nexus-feature-guide-messaging-->
[python-sandbox/nexus/nexus_feature_guide/messaging.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_feature_guide/messaging.py)
```py
@nexusrpc.handler.service_handler(service=NexusGreetingService)
class NexusGreetingServiceHandler:
    def _get_workflow_handle(
        self, client: Client, user_id: str
    ) -> WorkflowHandle[GreetingWorkflow, str]:
        return client.get_workflow_handle_for(
            GreetingWorkflow.run, f"GreetingWorkflow_for_{user_id}"
        )

    @nexus.temporal_operation
    async def approve(
        self,
        _ctx: nexus.TemporalStartOperationContext,
        client: nexus.TemporalNexusClient,
        input: ApproveInput,
    ) -> nexus.TemporalOperationResult[ApproveOutput]:
        await self._get_workflow_handle(client.client, input.user_id).signal(
            GreetingWorkflow.approve, input
        )
        return nexus.TemporalOperationResult.sync(ApproveOutput())
```
<!--SNIPEND-->

There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-python/tree/main/nexus_messaging/callerpattern/) and [on-demand pattern](https://github.com/temporalio/samples-python/tree/main/nexus_messaging/ondemandpattern/).
The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it.

### Develop an Asynchronous Nexus Operation handler to start a Workflow

Call `start_workflow` on the Client. The Operation completes when the Workflow returns, and the Workflow's return value
is delivered to the caller as the Operation's result.

<!--SNIPSTART python-nexus-feature-guide-service-handler {"selectedLines": ["17-18", "28-39"]}-->
[python-sandbox/nexus/nexus_feature_guide/service_handler.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_feature_guide/service_handler.py)
```py
# ...
@nexusrpc.handler.service_handler(service=MyNexusService)
class MyNexusServiceHandler:
# ...
    @nexus.temporal_operation
    async def hello(
        self,
        _ctx: nexus.TemporalStartOperationContext,
        client: nexus.TemporalNexusClient,
        input: HelloInput,
    ) -> nexus.TemporalOperationResult[HelloOutput]:
        return await client.start_workflow(
            HelloHandlerWorkflow.run,
            input,
            id=f"hello-{input.name}-{input.language}",
        )
```
<!--SNIPEND-->

Workflow IDs should typically be business-meaningful identifiers and are used to dedupe Workflow starts. In general, the Workflow ID should be passed in the Operation input as part of the Nexus Service contract.

> **💡 Tip:**
> RESOURCES
>
> [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing.
>

#### Map a Nexus Operation input to multiple Workflow arguments

A Nexus Operation can only take one input parameter. To start a Workflow that takes several, pass them to
`start_workflow` as `args` instead of a single positional argument:

<!--SNIPSTART python-nexus-feature-guide-service-handler {"selectedLines": ["48-52"]}-->
[python-sandbox/nexus/nexus_feature_guide/service_handler.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_feature_guide/service_handler.py)
```py
# ...
        return await client.start_workflow(
            HelloArgsHandlerWorkflow.run,
            args=[input.name, input.language],
            id=f"hello-{input.name}-{input.language}",
        )
```
<!--SNIPEND-->

### Register a Nexus Service in a Worker

After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register your Nexus Service handler in a Worker.
At this stage you can pass any arguments you need to your service handler's `__init__` method.

<!--SNIPSTART python-nexus-feature-guide-handler-worker-->
[python-sandbox/nexus/nexus_feature_guide/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_feature_guide/worker.py)
```py
async def main():
    client = await Client.connect("localhost:7233", namespace=NAMESPACE)
    worker = Worker(
        client,
        task_queue=TASK_QUEUE,
        workflows=[HelloHandlerWorkflow, HelloArgsHandlerWorkflow],
        nexus_service_handlers=[MyNexusServiceHandler()],
    )
    await worker.run()
```
<!--SNIPEND-->

## Develop a caller Workflow that uses the Nexus Service 

To execute a Nexus Operation from the caller Workflow, import the necessary service definition and operation input/output types:

<!--SNIPSTART python-nexus-feature-guide-caller-workflow-->
[python-sandbox/nexus/nexus_feature_guide/caller_workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_feature_guide/caller_workflows.py)
```py
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from nexus_feature_guide.service import (
        EchoInput,
        EchoOutput,
        HelloInput,
        HelloOutput,
        MyNexusService,
    )

NEXUS_ENDPOINT = "my-nexus-endpoint-name"

@workflow.defn
class CallerWorkflow:
    @workflow.run
    async def run(self, name: str) -> tuple[EchoOutput, HelloOutput]:
        nexus_client = workflow.create_nexus_client(
            service=MyNexusService,
            endpoint=NEXUS_ENDPOINT,
        )
        # Start the nexus operation and wait for the result in one go, using execute_operation.
        wf_result = await nexus_client.execute_operation(
            MyNexusService.hello,
            HelloInput(name=name, language="en"),
        )
        # Alternatively, you can use start_operation to obtain the operation handle and
        # then `await` the handle to obtain the result.
        sync_operation_handle = await nexus_client.start_operation(
            MyNexusService.echo,
            EchoInput(message=f"Hello {name}"),
        )
        sync_result = await sync_operation_handle
        return sync_result, wf_result
```
<!--SNIPEND-->

### Register the caller Workflow in a Worker and start the caller Workflow

After developing the caller Workflow, the next step is to register it with a Worker.

Finally, the caller Workflow must be started using `client.start_workflow()` or `client.execute_workflow()`.

These steps are the same as for any normal Workflow.
The Python sample combines them in a single application.
See [nexus_feature_guide/caller_app.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_feature_guide/caller_app.py) for reference.

## Make Nexus calls across Namespaces with a development Server 

Run these commands from the `python-sandbox/nexus` directory of the sample repository.

In one terminal, run the Temporal Worker in the handler Namespace:
```
uv run --no-project --with "temporalio>=1.34.0" python -m nexus_feature_guide.worker
```

In another terminal, run the Temporal Worker in the caller Namespace and start the caller Workflow:
```
uv run --no-project --with "temporalio>=1.34.0" python -m nexus_feature_guide.caller_app
```

### Canceling a Nexus Operation 

To cancel a Nexus Operation from within a Workflow, call `handle.cancel()` on the operation handle. Only asynchronous operations can be canceled in Nexus, since cancellation is sent using an operation token.
The Workflow or other resources backing the operation may choose to ignore the cancellation request.
If ignored, the operation may enter a terminal state.

When a Nexus operation is started, the caller can specify different cancellation types that control how the caller reacts to cancellation:

- `ABANDON` - Do not request cancellation of the operation.
- `TRY_CANCEL` - Initiate a cancellation request and immediately report cancellation to the caller. Note that this type
  doesn't guarantee that cancellation is delivered to the operation handler if the caller exits before the delivery is
  done.
- `WAIT_REQUESTED` - Request cancellation of the operation and wait for confirmation that the request was received.
  Doesn't wait for actual cancellation.
- `WAIT_COMPLETED` - Wait for operation completion. Operation may or may not complete as cancelled.

The default is `WAIT_COMPLETED`. Users can set a different option for `cancellation_type` when starting or executing an operation.

Once the caller Workflow completes, the caller's Nexus Machinery stops attempting to cancel operations that have not yet
been canceled, letting them run to completion.

It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending
operations to finish before exiting the Workflow.

See the [Nexus cancellation sample](https://github.com/temporalio/samples-python/tree/main/nexus_cancel) for reference.

## Make Nexus calls across Namespaces in Temporal Cloud 

This section assumes you are already familiar with how to connect a Worker to Temporal Cloud.
The Temporal Cloud CLI is used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces.

### Install `tcld` and generate certificates

Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command
(on macOS):

```
brew install temporalio/brew/tcld
```

If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below:

```
tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key
```

These certificates will be valid for one year.

### Create caller and handler Namespaces

Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler.
If you already have these Namespaces, you don't need to do this.

**Temporal CLI**

```
temporal cloud login

temporal cloud namespace create \
  --name <your-caller-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

temporal cloud namespace create \
  --name <your-target-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

**tcld**

```
tcld login

tcld namespace create \
  --namespace <your-caller-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

tcld namespace create \
  --namespace <your-target-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/namespaces](https://cloud.temporal.io/namespaces).

### Create a Nexus Endpoint to route requests from caller to handler

To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`.

**Temporal CLI**

```
temporal cloud nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file hello_nexus/endpoint_description.md
```

**tcld**

```
tcld nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file hello_nexus/endpoint_description.md
```

The `--allow-namespace` flag builds an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in [Runtime Access Controls](/nexus/security#runtime-access-controls).

Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus).

## Observability

### Web UI

A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history:

![Observability Sync](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-sync.png)

An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history:

![Observability Async](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-async.png)

### Temporal CLI

Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow:

```
temporal workflow describe -w <ID>
```

Nexus events are included in the caller's Event history:

```
temporal workflow show -w <ID>
```

For **asynchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationStarted`
- `NexusOperationCompleted`

For **synchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationCompleted`

> **📝 Note:**
>
> `NexusOperationStarted` isn't reported in the caller's history for synchronous operations.
>

## Learn more

- Read the high-level description of the [Temporal Nexus feature](/evaluate/features/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082).
- Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus).
- Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus).

---

# Nexus Python quickstart - Python SDK

Source: https://docs.temporal.io/develop/python/nexus/quickstart

> Build a Nexus Service that wraps an existing Temporal Workflow using the Python SDK

[Temporal Nexus](/evaluate/features/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow.

> **ℹ️ Info:**
> To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/features/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus).

**Prerequisites:** Complete the [Python SDK Quickstart](/develop/python/set-up-your-local-python) first.
You should have `activities.py`, `workflows.py`, `worker.py`, and `starter.py` from that guide.
From the project root, create a `nexus_quickstart/` directory with an empty `__init__.py`.
Put the quickstart's `activities.py` and `workflows.py`, along with every file created in this guide, in that directory.
The `worker.py` in this guide replaces the quickstart's `worker.py`.

## What you'll build

You have `SayHelloWorkflow` running in the `default` Namespace.
By the end of this guide:

1. A Nexus Service will expose `SayHelloWorkflow` as an Operation.
2. A second Namespace will contain a Workflow that calls that Operation.
3. The caller Workflow will get back `"Hello Temporal"` — the same result, but across Namespaces.

<!--SNIPSTART python-nexus-quickstart-service-->
[python-sandbox/nexus/nexus_quickstart/service.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_quickstart/service.py)
```py
from dataclasses import dataclass

import nexusrpc

@dataclass
class MyInput:
    name: str

@nexusrpc.service
class SayHelloNexusService:
    say_hello: nexusrpc.Operation[MyInput, str]
```
<!--SNIPEND-->

## 1. Define the Nexus Service

Create a file called `service.py` that defines the Nexus Service contract.

Creating a Nexus Service establishes the contract between your implementation and any callers.
It provides type safety when invoking Nexus Operations and ensures that operation handlers fulfill the contract.

The `@nexusrpc.service` decorator declares a service with typed operations.
`SayHelloWorkflow` returns `str`, so the operation output type is `str`.

<!--SNIPSTART python-nexus-quickstart-handler-->
[python-sandbox/nexus/nexus_quickstart/handler.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_quickstart/handler.py)
```py
import uuid

import nexusrpc.handler
from temporalio import nexus

from nexus_quickstart.service import MyInput, SayHelloNexusService
from nexus_quickstart.workflows import SayHelloWorkflow

@nexusrpc.handler.service_handler(service=SayHelloNexusService)
class SayHelloNexusServiceHandler:
    @nexus.workflow_run_operation
    async def say_hello(
        self, ctx: nexus.WorkflowRunOperationContext, input: MyInput
    ) -> nexus.WorkflowHandle[str]:
        return await ctx.start_workflow(
            SayHelloWorkflow.run,
            input.name,
            id=f"say-hello-nexus-{uuid.uuid4()}",

            # Task queue defaults to the task queue this Operation is handled on.
        )
```
<!--SNIPEND-->

## 2. Define the Nexus Operation handlers

Create a file called `handler.py` that implements the Nexus Operation handler.

Operation handlers contain the logic that runs when a caller invokes a Nexus Operation.

The `@nexus.workflow_run_operation` decorator creates an asynchronous Nexus Operation that starts a Workflow.
The handler bridges the Nexus `MyInput` dataclass to `SayHelloWorkflow`'s `str` parameter by extracting `input.name`.

<!--SNIPSTART python-nexus-quickstart-worker-->
[python-sandbox/nexus/nexus_quickstart/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_quickstart/worker.py)
```py
import asyncio

from temporalio.client import Client
from temporalio.worker import Worker

from nexus_quickstart.workflows import SayHelloWorkflow
from nexus_quickstart.activities import greet
from nexus_quickstart.handler import SayHelloNexusServiceHandler

async def main():
    client = await Client.connect("localhost:7233")
    worker = Worker(
        client,
        task_queue="my-task-queue",
        workflows=[SayHelloWorkflow],
        activities=[greet],
        nexus_service_handlers=[SayHelloNexusServiceHandler()],
    )
    print("Worker started.")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## 3. Register the Nexus Service handler in a Worker

Update your existing `worker.py` to register the Nexus Service handler.

A Worker will only poll for and process incoming Nexus requests if the Nexus Service handlers are registered.
This is the same Worker concept used for Workflows and Activities.

The `nexus_service_handlers` parameter registers the handler so it can receive Nexus Operation requests.

<!--SNIPSTART python-nexus-quickstart-caller-->
[python-sandbox/nexus/nexus_quickstart/caller.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_quickstart/caller.py)
```py
from datetime import timedelta
from temporalio import workflow
from nexus_quickstart.service import MyInput, SayHelloNexusService

NEXUS_ENDPOINT = "my-nexus-endpoint-name"

@workflow.defn
class CallerWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        nexus_client = workflow.create_nexus_client(
            service=SayHelloNexusService,
            endpoint=NEXUS_ENDPOINT,
        )
        return await nexus_client.execute_operation(
            SayHelloNexusService.say_hello,
            MyInput(name=name),
            schedule_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

## 4. Develop the caller Workflow

Create a file called `caller.py` that defines a Workflow which invokes the Nexus Operation.

The caller Workflow demonstrates the consumer side of Nexus.
Instead of importing handler code directly, the caller only depends on the Service contract.
This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams.

The `workflow.create_nexus_client()` method creates a client bound to your Nexus Service and Endpoint.
`execute_operation` starts the operation and waits for the result.

## 5. Create the caller Namespace and Nexus Endpoint

Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler.
The handler uses the `default` Namespace that was created when you started the dev server.

Namespaces provide isolation between the caller and handler sides.
The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue.
The endpoint name must match the variable defined in `caller.py` from step 4.

Make sure your local Temporal dev server is running (`temporal server start-dev`).

```bash
temporal operator namespace create --namespace my-caller-namespace
```

```bash
temporal operator nexus endpoint create \\
  --name my-nexus-endpoint-name \\
  --target-namespace default \\
  --target-task-queue my-task-queue
```

<!--SNIPSTART python-nexus-quickstart-caller-starter-->
[python-sandbox/nexus/nexus_quickstart/caller_starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_quickstart/caller_starter.py)
```py
import asyncio
import uuid

from temporalio.client import Client
from temporalio.worker import Worker

from nexus_quickstart.caller import CallerWorkflow

CALLER_TASK_QUEUE = "my-caller-task-queue"
NAMESPACE = "my-caller-namespace"

async def main():
    client = await Client.connect("localhost:7233", namespace=NAMESPACE)

    async with Worker(
        client,
        task_queue=CALLER_TASK_QUEUE,
        workflows=[CallerWorkflow],
    ):
        result = await client.execute_workflow(
            CallerWorkflow.run,
            "Temporal",
            id=f"caller-workflow-{uuid.uuid4()}",
            task_queue=CALLER_TASK_QUEUE,
        )
        print("Workflow result:", result)

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## 6. Run and verify

Create a file called `caller_starter.py` to start the caller Worker and execute the Workflow.

This step brings everything together: the caller Worker hosts `CallerWorkflow`, which uses the Nexus client to invoke `say_hello` on the handler side.
The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `SayHelloWorkflow`, and back to the caller.

**Run the application:**

1. From the project root, start the handler Worker in one terminal:

```bash
source env/bin/activate
python3 -m nexus_quickstart.worker
```

2. From the project root, run the caller in another terminal:

```bash
source env/bin/activate
python3 -m nexus_quickstart.caller_starter
```

You should see:

```
Workflow result: Hello Temporal
```

Open the [Temporal Web UI](http://localhost:8233) and find the `CallerWorkflow` execution.
You should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in the Event history.

## Next steps

Now that you have a working Nexus Service, here are some resources to deepen your understanding:

- **[Python Nexus feature guide](/develop/python/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls.
- **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics.
- **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker.
- **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns.
- **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries.
- **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls.
- **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests.
- **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity.

---

# Standalone Nexus Operations - Python SDK

Source: https://docs.temporal.io/develop/python/nexus/standalone-operations

> Execute Nexus Operations independently without a Workflow using the Temporal Python SDK.

> **Public Preview**
> Requires Python SDK `1.30.0` or above. API breaking changes are kept to a minimum before General Availability.

[Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without
being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using
`workflow.create_nexus_client()`, you execute a Standalone Nexus Operation directly from a Nexus Client created using
`client.create_nexus_client()`.

Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as
Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/python/nexus/feature-guide) for details on
[defining a Service contract](/develop/python/nexus/feature-guide#define-nexus-service-contract),
[developing Operation handlers](/develop/python/nexus/feature-guide#develop-nexus-service-operation-handlers), and
[registering a Service in a Worker](/develop/python/nexus/feature-guide#register-a-nexus-service-in-a-worker).

This page focuses on the client-side APIs that are unique to Standalone Nexus Operations:

- [Execute a Standalone Nexus Operation](#execute-operation)
- [Start a Standalone Nexus Operation and Wait for the Result](#get-operation-result)
- [List Standalone Nexus Operations](#list-operations)
- [Count Standalone Nexus Operations](#count-operations)
- [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud)

> **📝 Note:**
> This documentation uses source code from the
> [nexus_standalone_operations](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/nexus/nexus_standalone_operations) sample.
>

## Prerequisites 

Standalone Nexus Operations are a Public Preview feature and require Temporal CLI v1.9.0 or higher.

### 1. Install and verify the Temporal CLI 

The `temporal nexus operation` commands require Temporal CLI v1.9.0 or higher. See
[Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for install options,
then verify:

```bash
temporal --version
```

Which should output v1.9.0 or higher.

### 2. Start a local dev server 

The dev server enables Standalone Nexus Operations by default — no dynamic config is
required. Start it with the caller and handler Namespaces pre-created:

```bash
temporal server start-dev \
  --namespace my-caller-namespace \
  --namespace my-handler-namespace
```

The starter and Worker connect to two different Namespaces (a caller Namespace and a handler
Namespace), mirroring how Nexus crosses Namespace boundaries.

To run the examples on this page against the [Python sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/nexus/nexus_standalone_operations),
create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue:

```bash
temporal operator nexus endpoint create \
  --name my-nexus-endpoint \
  --target-namespace my-handler-namespace \
  --target-task-queue nexus-handler-queue
```

Run the remaining commands from the `python-sandbox/nexus` directory of the sample repository.
Start the sample Worker in the handler Namespace:

```bash
TEMPORAL_NAMESPACE=my-handler-namespace uv run --no-project --with "temporalio>=1.34.0" python -m nexus_standalone_operations.worker
```

Run the starter in the caller Namespace (from a separate terminal):

```bash
TEMPORAL_NAMESPACE=my-caller-namespace uv run --no-project --with "temporalio>=1.34.0" python -m nexus_standalone_operations.starter
```

## Execute a Standalone Nexus Operation 

To execute a Standalone Nexus Operation, first create a
[`NexusClient`](https://python.temporal.io/temporalio.client.NexusClient.html) using `client.create_nexus_client()`, bound to a
specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call `start_operation()` or `execute_operation()` from application code (for example, a starter program), not from inside a Workflow Definition.

`execute_operation` waits for the Operation to complete and returns the result.
Both methods require `id`. `schedule_to_close_timeout` is optional and defaults to the maximum allowed by the Temporal server.

<!--SNIPSTART python-nexus-standalone-execute-operation-->
[python-sandbox/nexus/nexus_standalone_operations/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_standalone_operations/starter.py)
```py
nexus_client = client.create_nexus_client(
    service=MyNexusService, endpoint=ENDPOINT_NAME
)

# Await the result of the operation immediately.
echo_result = await nexus_client.execute_operation(
    MyNexusService.echo,
    EchoInput(message="hello"),
    id=f"echo-{uuid.uuid4()}",
    schedule_to_close_timeout=timedelta(seconds=10),
)
```
<!--SNIPEND-->

See the full
[starter sample](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_standalone_operations/starter.py)
for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and
counts Operations.

Or use the Temporal CLI to execute a Standalone Nexus Operation:

```bash
temporal nexus operation execute \
  --namespace my-caller-namespace \
  --endpoint my-nexus-endpoint \
  --service MyNexusService \
  --operation echo \
  --operation-id my-echo-op \
  --input '{"message":"hello"}'
```

## Start a Standalone Nexus Operation and Wait for the Result 

`start_operation` returns a [`NexusOperationHandle`](https://python.temporal.io/temporalio.client.NexusOperationHandle.html).
Use `NexusOperationHandle.result()` to wait until the Operation completes and retrieve its result. This works for both
synchronous and asynchronous Operations.

<!--SNIPSTART python-nexus-standalone-start-operation-->
[python-sandbox/nexus/nexus_standalone_operations/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_standalone_operations/starter.py)
```py
# Start an operation and get a NexusOperationHandle
handle = await nexus_client.start_operation(
    MyNexusService.hello,
    HelloInput(name="World"),
    id=f"hello-{uuid.uuid4()}",
    schedule_to_close_timeout=timedelta(seconds=10),
)
# Await the result
try:
    hello_result = await handle.result()
    print(hello_result)
except Exception as err:
    print(err)
    raise
```
<!--SNIPEND-->

If the Operation completed successfully, the result is returned. If the Operation failed, the failure is raised as an error.

Or use the Temporal CLI to wait for a result by Operation ID:

```bash
temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op
```

## List Standalone Nexus Operations 

Use [`client.list_nexus_operations()`](https://python.temporal.io/temporalio.client.Client.html#list_nexus_operations) to list Standalone Nexus
Operation Executions that match a [List Filter](/list-filter) query. The result contains an iterator that yields
operation metadata entries.

Note that `list_nexus_operations` is called on the base `client.Client`, not on the `NexusClient`.

<!--SNIPSTART python-nexus-standalone-list-operations-->
[python-sandbox/nexus/nexus_standalone_operations/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_standalone_operations/starter.py)
```py
query = f'Endpoint = "{ENDPOINT_NAME}"'
async for op in client.list_nexus_operations(query):
    print(
        f" OperationId: {op.operation_id},",
        f" Operation: {op.operation},",
        f" Status: {op.status.name}",
    )
```
<!--SNIPEND-->

The `query` parameter accepts [List Filter](/list-filter) syntax. For example,
`"Endpoint = 'my-endpoint' AND ExecutionStatus = 'Running'"`.

Or use the Temporal CLI:

```bash
temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Count Standalone Nexus Operations 

Use [`client.count_nexus_operations()`](https://python.temporal.io/temporalio.client.Client.html#count_nexus_operations) to count Standalone Nexus
Operation Executions that match a [List Filter](/list-filter) query.

Note that `count_nexus_operations` is called on the base `client.Client`, not on the `NexusClient`.

<!--SNIPSTART python-nexus-standalone-count-operations-->
[python-sandbox/nexus/nexus_standalone_operations/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/nexus/nexus_standalone_operations/starter.py)
```py
query = f'Endpoint = "{ENDPOINT_NAME}"'
count = await client.count_nexus_operations(query)
print(f"Total Nexus operations: {count.count}")
```
<!--SNIPEND-->

Or use the Temporal CLI:

```bash
temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Run Standalone Nexus Operations with Temporal Cloud 

The code samples referenced on this page use [`ClientConfig.load_client_connect_config()`](https://python.temporal.io/temporalio.envconfig.ClientConfig.html#load_client_connect_config), so the same code
works against Temporal Cloud — just configure the connection via environment variables or a TOML
profile. No code changes are needed.

For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint
setup, certificate generation, and authentication options, see
[Make Nexus calls across Namespaces in Temporal Cloud](/develop/python/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud)
and [Connect to Temporal Cloud](/develop/python/client/temporal-client#connect-to-temporal-cloud).

---

# Platform - Python SDK

Source: https://docs.temporal.io/develop/python/platform

> This section explains how to implement platform with the Python SDK

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Platform

- [Observability](/develop/python/platform/observability)
- [Enriching the UI](/develop/python/platform/enriching-ui)

---

# Enriching the user interface - Python SDK

Source: https://docs.temporal.io/develop/python/platform/enriching-ui

> Add contextual information to workflows and events in the Temporal UI using the Python SDK.

Temporal supports adding context to Workflows and events with metadata. 
This helps users identify and understand Workflows and their operations.

## Adding summary and details to Workflows

### Starting a Workflow

When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI:

<!--SNIPSTART python-enriching-ui-start-workflow-->
[python-sandbox/platform/enriching_ui/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/enriching_ui/starter.py)
```py
# Start a Workflow with static summary and details
handle = await client.start_workflow(
    YourWorkflow.run,
    "workflow input",
    id="your-workflow-id",
    task_queue="your-task-queue",
    static_summary="Order processing for customer #12345",
    static_details="Processing premium order with expedited shipping"
)
```
<!--SNIPEND-->

`static_summary` is a single-line description that appears in the Workflow list view, limited to 200 bytes.
`static_details` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes.

The input format is standard Markdown excluding images, HTML, and scripts.

You can also use the `execute_workflow` method with the same parameters:

<!--SNIPSTART python-enriching-ui-execute-workflow-->
[python-sandbox/platform/enriching_ui/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/enriching_ui/starter.py)
```py
result = await client.execute_workflow(
    YourWorkflow.run,
    "workflow input",
    id="your-workflow-id",
    task_queue="your-task-queue",
    static_summary="Order processing for customer #12345",
    static_details="Processing premium order with expedited shipping"
)
```
<!--SNIPEND-->

### Inside the Workflow

Within a Workflow, you can get and set the _current Workflow details_. 
Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. 
Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines.

<!--SNIPSTART python-enriching-ui-current-details-->
[python-sandbox/platform/enriching_ui/details_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/enriching_ui/details_workflow.py)
```py
@workflow.defn
class YourWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        # Get the current details
        current_details = workflow.get_current_details()
        print(f"Current details: {current_details}")

        # Set/update the current details
        workflow.set_current_details("Updated workflow details with new status")

        return "Workflow completed"
```
<!--SNIPEND-->

### Adding Summary to Activities and Timers

You can attach a metadata parameter `summary` to Activities when starting them from within a Workflow:

<!--SNIPSTART python-enriching-ui-activity-summary-->
[python-sandbox/platform/enriching_ui/activity_summary_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/enriching_ui/activity_summary_workflow.py)
```py
@workflow.defn
class YourWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        # Start an activity with a summary
        result = await workflow.execute_activity(
            your_activity,
            input,
            start_to_close_timeout=timedelta(seconds=10),
            summary="Processing user data"
        )
        return result
```
<!--SNIPEND-->

Similarly, you can attach a `summary` to Timers within a Workflow:

<!--SNIPSTART python-enriching-ui-timer-summary-->
[python-sandbox/platform/enriching_ui/timer_summary_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/enriching_ui/timer_summary_workflow.py)
```py
@workflow.defn
class YourWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        # Create a timer with a summary
        await workflow.sleep(timedelta(minutes=5), summary="Waiting for payment confirmation")
        return "Timer completed"
```
<!--SNIPEND-->

The input format for `summary` is a string, and limited to 200 bytes.

## Viewing summary and details in the UI

Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. 
Navigate to your Workflow's details page to see the metadata displayed in three key locations:

### User Metadata

On the Workflow Details page, under the **User Metadata** tab, you'll find the workflow-level metadata:

- **Summary & Details** - Displays the static summary and static details set when starting the Workflow
- **Current Details** - Displays the dynamic details that can be updated during Workflow execution

All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays.

### Timeline

The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance.

Labels longer than 120 characters are truncated with an ellipsis.

Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. 

Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later.

### Event History

Individual events in the Workflow's Event History display their associated summaries when available.

Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the Event details. 
When you do expand an Event, the summary is also prominently displayed in the detailed view.

---

# Observability - Python SDK

Source: https://docs.temporal.io/develop/python/platform/observability

> Discover how to monitor your Temporal Application using metrics, tracing, logging, and visibility APIs. Emit metrics, set up tracing, log from Workflows, and use custom Search Attributes.

The observability section of the Temporal Developer's guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

This section covers features related to viewing the state of the application, including:

- [Emit metrics](#metrics)
- [Set up tracing](#tracing)
- [Log from a Workflow](#logging)
- [Visibility APIs](#visibility)

## Emit metrics 

Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process.
For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics).

- For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide.
- For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics).
- For an end-to-end example that exposes metrics with the Python SDK, refer to the [samples-python](https://github.com/temporalio/samples-python/tree/main/prometheus) repo.

Metrics in Python are configured globally; therefore, you should set a Prometheus endpoint before any other Temporal code.

### Set a Prometheus endpoint

The following example exposes a Prometheus endpoint on port `9000`.

<!--SNIPSTART python-observability-prometheus-->
[python-sandbox/platform/observability/prometheus.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/prometheus.py)
```py
from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig

# Create a new runtime that has telemetry enabled. Create this first to avoid
# the default Runtime from being lazily created.
new_runtime = Runtime(telemetry=TelemetryConfig(metrics=PrometheusConfig(bind_address="0.0.0.0:9000")))
my_client = await Client.connect("my.temporal.host:7233", runtime=new_runtime)
```
<!--SNIPEND-->

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Set [`global_tags`](https://python.temporal.io/temporalio.runtime.TelemetryConfig.html#global_tags) on `TelemetryConfig` to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker.

<!--SNIPSTART python-observability-global-tags {"selectedLines": ["5", "12-24"]}-->
[python-sandbox/telemetry-worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/telemetry-worker.py)
```py
# ...
from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig
# ...
    new_runtime = Runtime(
        telemetry=TelemetryConfig(
            metrics=PrometheusConfig(bind_address="0.0.0.0:9000"),
            global_tags={
                "team": "content-platform",
                "service": "checkout",
                "cost_center": "cc-1042",
                "environment": "production",
            },
        )
    )

    client = await Client.connect("localhost:7233", runtime=new_runtime)
```
<!--SNIPEND-->

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag           | Example            | Question it answers                                    |
| ------------- | ------------------ | ------------------------------------------------------ |
| `team`        | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service`     | `checkout`         | Which application emits these metrics?                 |
| `cost_center` | `cc-1042`          | Which budget does this Worker fleet belong to?         |
| `environment` | `production`       | Is this production traffic, or staging or test?        |
| `region`      | `us-east-2`        | Where does the Worker fleet run?                       |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Set up tracing 

Tracing allows you to view the call graph of a Workflow along with its Activities and any Child Workflows.

Temporal Web's tracing capabilities mainly track Activity Execution within a Temporal context. If you need custom tracing specific for your use case, you should make use of context propagation to add tracing logic accordingly.

To configure tracing in Python, install the `opentelemetry` dependencies and an exporter for your tracing backend.

```bash
# This command installs the `opentelemetry` dependencies.
pip install temporalio[opentelemetry]
# Any OpenTelemetry exporter works; this one speaks OTLP.
pip install opentelemetry-exporter-otlp
```

The Python SDK offers two ways to emit OpenTelemetry spans: the `OpenTelemetryPlugin`, which supports spans with real
durations and the standard OpenTelemetry API inside Workflow code, and the earlier `TracingInterceptor`.

### Trace with the OpenTelemetry plugin

The [`OpenTelemetryPlugin`](https://python.temporal.io/temporalio.contrib.opentelemetry.OpenTelemetryPlugin.html)
propagates trace context across Client, Workflow, and Activity boundaries and lets Workflow code use the standard
OpenTelemetry API. It requires a replay-safe tracer provider from
[`create_tracer_provider()`](https://python.temporal.io/temporalio.contrib.opentelemetry.html#create_tracer_provider),
set as the global tracer provider before you connect the Client.

<!--SNIPSTART python-observability-opentelemetry-plugin-->
[python-sandbox/platform/observability/tracing_plugin.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/tracing_plugin.py)
```py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from temporalio.client import Client
from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider

async def main():
    provider = create_tracer_provider()
    provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
    )
    trace.set_tracer_provider(provider)

    client = await Client.connect(
        "localhost:7233",
        plugins=[OpenTelemetryPlugin(add_temporal_spans=True)],
    )
```
<!--SNIPEND-->

Register the plugin on the Client only. Workers created from that Client inherit it. With `add_temporal_spans=True`,
the plugin creates spans for Temporal operations: `StartWorkflow`, `RunWorkflow`, `StartActivity`, `RunActivity`,
Signal, Query, and Update handlers, and Child Workflows. With the default `add_temporal_spans=False`, the plugin only
propagates trace context, so spans you create yourself nest correctly without additional Temporal spans.

Inside Workflow code, create spans with the regular tracer. The plugin allows the `opentelemetry` module through
the Workflow sandbox.

<!--SNIPSTART python-observability-workflow-span-->
[python-sandbox/platform/observability/traced_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/traced_workflow.py)
```py
from datetime import timedelta

from opentelemetry import trace
from temporalio import workflow

from observability.my_activities import my_activity

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        with trace.get_tracer(__name__).start_as_current_span("prepare"):
            return await workflow.execute_activity(
                my_activity, name, start_to_close_timeout=timedelta(seconds=10)
            )
```
<!--SNIPEND-->

Replay is safe. The provider generates span identifiers deterministically from Workflow state and does not export
spans while a Workflow replays, so Worker restarts and cache evictions never produce duplicate spans. A span that is
open when a Worker stops is exported once, when the Workflow finishes it on another Worker. Each Activity attempt
produces its own `RunActivity` span, so retries stay visible.

> **⚠️ Caution:**
>
> `OpenTelemetryPlugin` and `create_tracer_provider()` are experimental and may change in future versions.
>

For a complete example that sends agent traces to an observability backend, see the
[OpenTelemetry section of the OpenAI Agents SDK guide](/develop/python/integrations/openai-agents#opentelemetry).

### Trace with the interceptor

The [`temporalio.contrib.opentelemetry.TracingInterceptor`](https://python.temporal.io/temporalio.contrib.opentelemetry.TracingInterceptor.html)
class is the earlier integration. Set it as an interceptor as an argument of
[`Client.connect()`](https://python.temporal.io/temporalio.client.Client.html#connect).

<!--SNIPSTART python-observability-tracing-interceptor-->
[python-sandbox/platform/observability/tracing_interceptor.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/tracing_interceptor.py)
```py
from temporalio.client import Client
from temporalio.contrib.opentelemetry import TracingInterceptor

async def main():
    client = await Client.connect("localhost:7233", interceptors=[TracingInterceptor()])
```
<!--SNIPEND-->

When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the
Worker. Spans are created and serialized through the server to give one trace for a Workflow Execution.

The interceptor creates Workflow-side spans as completed spans with no duration, because an open span cannot
survive replay. To add a custom span from Workflow code with the interceptor, use
[`temporalio.contrib.opentelemetry.workflow.completed_span()`](https://python.temporal.io/temporalio.contrib.opentelemetry.workflow.html).
For spans with real durations inside Workflows, use the plugin instead.

## Log from a Workflow 

Logging enables you to record critical information during code execution.
Loggers create an audit trail and capture information about your Workflow's operation.
An appropriate logging level depends on your specific needs.
During development or troubleshooting, you might use debug or even trace.
In production, you might use info or warn to avoid excessive log volume.

You can log from a Workflow using Python's standard library, by importing the logging module `logging`. You can find the log levels supported by the `logging` module in [their official documentation](https://docs.python.org/3/library/logging.html#logging-levels). The Temporal SDK core normally uses `WARN` as its default logging level.

Set your logging configuration to a level you want to expose logs to.
The following example sets the logging information level to `INFO`.

<!--SNIPSTART python-observability-logging-config-->
[python-sandbox/platform/observability/logging_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/logging_workflow.py)
```py
logging.basicConfig(level=logging.INFO)
```
<!--SNIPEND-->

Then in your Workflow, set your [`logger`](https://python.temporal.io/temporalio.workflow.html#logger) and level on the Workflow. The following example logs the Workflow.

<!--SNIPSTART python-observability-logging-workflow {"highlightedLines": "11"}-->
[python-sandbox/platform/observability/logging_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/logging_workflow.py)
```py {11}
from temporalio import workflow

@workflow.defn
class GreetingWorkflow:
    def __init__(self) -> None:
        self._greeting = "<no greeting>"

    @workflow.run
    async def run(self, name: str) -> None:
        workflow.logger.info("Workflow input parameter: %s" % name)
        self._greeting = f"Hello, {name}!"

    @workflow.query
    def greeting(self) -> str:
        return self._greeting
```
<!--SNIPEND-->

### Custom logger 

Use a custom logger for logging.

Use the built-in [Logging facility for Python](https://docs.python.org/3/library/logging.html) to set a custom logger.

## Visibility APIs 

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### Use Search Attributes 

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service in the Temporal CLI or Web UI.
  - For example: `temporal operator search-attribute create --name CustomKeywordField --type Text`
    - Replace `CustomKeywordField` with the name of your Search Attribute.
    - Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList`
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `upsert_search_attributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `DescribeWorkflow`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [In the Temporal CLI](/cli/command-reference/operator#list-2)
  - In code by calling `ListWorkflowExecutions`.

Here is how to query Workflow Executions:

Use the [list_workflows()](https://python.temporal.io/temporalio.client.Client.html#list_workflows) method on the Client handle and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows.

<!--SNIPSTART python-observability-list-workflows {"highlightedLines": "30-31"}-->
[python-sandbox/platform/observability/list_workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/list_workflows.py)
```py {30-31}
import asyncio
from temporalio.client import Client
from observability.search_attributes_workflow import GreetingWorkflow

async def main():
    client = await Client.connect("localhost:7233")
    handle = await client.start_workflow(
        GreetingWorkflow.run,
        id="search-attributes-workflow-id",
        task_queue="search-attributes-task-queue",
        search_attributes={"CustomKeywordField": ["old-value"]},
    )

    print(
        "First search attribute values: ",
        (await handle.describe()).search_attributes.get("CustomKeywordField"),
    )
    await asyncio.sleep(3)
    print(
        "Second search attribute values: ",
        (await handle.describe()).search_attributes.get("CustomKeywordField"),
    )
    await asyncio.sleep(3)
    print(
        "Empty search attribute values: ",
        (await handle.describe()).search_attributes.get("CustomKeywordField"),
    )

    async for workflow in client.list_workflows('WorkflowType="GreetingWorkflow"'):
        print(f"Workflow: {workflow.id}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### How to set custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

Use `SearchAttributeKey` to create your Search Attributes. Then, when starting a Workflow execution using `client.start_workflow()`, include the Custom Search Attributes by passing instances of `SearchAttributePair()` containing each of your keys and starting values to a parameter called `search_attributes`.
If you had Custom Search Attributes `CustomerId` of type `Keyword` and `MiscData` of type `Text`, you could provide these starting values:

<!--SNIPSTART python-observability-typed-search-attributes-->
[python-sandbox/platform/observability/typed_search_attributes.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/typed_search_attributes.py)
```py
customer_id_key = SearchAttributeKey.for_keyword("CustomerId")
misc_data_key = SearchAttributeKey.for_text("MiscData")

handle = await client.start_workflow(
    CustomerWorkflow.run,
    id="search-attributes-workflow-id",
    task_queue="search-attributes-task-queue",
    search_attributes=TypedSearchAttributes([
        SearchAttributePair(customer_id_key, "customer_1"),
        SearchAttributePair(misc_data_key, "customer_1_data")
    ]),
)
```
<!--SNIPEND-->

In this example, `CustomerId` and `MiscData` are set as Search Attributes.
These attributes are useful for querying Workflows based on the customer ID or the date the order was placed.

### Upsert Search Attributes 

You can upsert Search Attributes to add or update Search Attributes from within Workflow code.

To upsert custom Search Attributes, use the [`upsert_search_attributes()`](https://python.temporal.io/temporalio.workflow.html#upsert_search_attributes) method to pass a list of `SearchAttributeUpdate()`.
These can be created via value_set calls on Search Attribute keys:

<!--SNIPSTART python-observability-upsert-search-attributes-->
[python-sandbox/platform/observability/search_attributes_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/search_attributes_workflow.py)
```py
workflow.upsert_search_attributes([
    customer_id_key.value_set("customer_2")
])
```
<!--SNIPEND-->

### Remove a Search Attribute from a Workflow 

To remove a Search Attribute that was previously set, call `value_unset()` on the Search Attribute key.

<!--SNIPSTART python-observability-remove-search-attribute-->
[python-sandbox/platform/observability/search_attributes_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/platform/observability/search_attributes_workflow.py)
```py
workflow.upsert_search_attributes([
    customer_id_key.value_unset()
])
```
<!--SNIPEND-->

---

# Set up a local development environment with the Python SDK

Source: https://docs.temporal.io/develop/python/set-up-your-local-python

> Configure your local development environment to get started developing with Temporal

# Quickstart

Configure your local development environment to get started developing with Temporal.

## Install Python

Make sure you have Python installed. Check your version of Python with the following command.

```bash
python3 -V
```

```bash
python 3.13.3
```

## Install the Temporal Python SDK

You should install the Temporal Python SDK in your project using a virtual environment. Create a directory for your
Temporal project, switch to the new directory, create a Python virtual environment, activate it, and then install the
Temporal SDK.

Next, you'll configure a local Temporal Service for development.

```bash
mkdir temporal-project
```

```bash
cd temporal-project
```

```bash
python3 -m venv env
```

```bash
source env/bin/activate
```

```bash
pip install temporalio
```

## Install Temporal CLI

The fastest way to get a development version of the Temporal Service running on your local machine is to use
[Temporal CLI](/cli).

Choose your operating system to install Temporal CLI.

**macOS**

Install the Temporal CLI using Homebrew:
```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH, for example:
```bash
sudo mv temporal /usr/local/bin
```

## Start the development server

Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory
database.

The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at
http://localhost:8233.

Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal
Service at any time by pressing CTRL+C.

Once you have everything installed, you're ready to build apps with Temporal on your local machine.

After installing, open a new Terminal window and start the development server:

```bash
temporal server start-dev
```

#### Change the Web UI port
The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server:
```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and
Activity.

This test will confirm that:

- The Temporal Python SDK is properly installed
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

### 1. Create the Activity

Create an Activity file (activities.py):

<!--SNIPSTART python-quickstart-activity-->
[python-sandbox/activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/activities.py)
```py
from temporalio import activity

@activity.defn
async def greet(name: str) -> str:
    return f"Hello {name}"
```
<!--SNIPEND-->

An Activity is a normal function or method that executes a single, well-defined action (either short or long running),
which often involve interacting with the outside world, such as sending emails, making network requests, writing to a
database, or calling an API, which are prone to failure. If an Activity fails, Temporal automatically retries it based
on your configuration.

### 2. Create the Workflow

Create a Workflow file (workflows.py):

<!--SNIPSTART python-quickstart-workflow-->
[python-sandbox/workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows.py)
```py
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import greet

@workflow.defn
class SayHelloWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            greet,
            name,
            schedule_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run and
keep running for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will
automatically recreate its pre-failure state so it can continue right where it left off.

### 3. Create the Worker

Create a Worker file (worker.py):

<!--SNIPSTART python-quickstart-worker-->
[python-sandbox/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/worker.py)
```py
import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from workflows import SayHelloWorkflow
    from activities import greet

async def main():
    client = await Client.connect("localhost:7233")
    worker = Worker(
        client,
        task_queue="my-task-queue",
        workflows=[SayHelloWorkflow],
        activities=[greet],
    )
    print("Worker started.")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

Run the Worker by opening up a new terminal:

```bash
source env/bin/activate
python3 worker.py
```

Keep this terminal running - you should see "Worker started" displayed.

With your Activity and Workflow defined, you need a Worker to execute them. A Worker polls a Task Queue, that you
configure it to poll, looking for work to do. Once the Worker dequeues the Workflow or Activity task from the Task
Queue, it then executes that task.

Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your
Workflows and Activities. For more information on Workers, see
[Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers).

### 4. Execute the Workflow

Now that your Worker is running, it's time to start a Workflow Execution.

This final step will validate that everything is working correctly with your file labeled `starter.py`.

Create a separate file called `starter.py`:

<!--SNIPSTART python-quickstart-start-workflow-->
[python-sandbox/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/starter.py)
```py
import asyncio
import uuid
from temporalio.client import Client

async def main():
    client = await Client.connect("localhost:7233")
    result = await client.execute_workflow(
        "SayHelloWorkflow",
        "Temporal",
        id=f"say-hello-workflow-{uuid.uuid4()}",
        task_queue="my-task-queue",
    )
    print("Workflow result:", result)

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

While the Worker is still running, run the following command in a new terminal:

```bash
source env/bin/activate
python3 starter.py
```

### Verify success

If everything is working correctly, you should see:

- Worker processing the Workflow and Activity
- Output: `Workflow result: Hello Temporal`
- Workflow Execution details in the [Temporal Web UI](http://localhost:8233)

- [Run your first Temporal Application](https://learn.temporal.io/getting_started/python/first_program_in_python/): Create a basic Workflow and run it with the Temporal Python SDK

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Workers - Python SDK

Source: https://docs.temporal.io/develop/python/workers

> This section explains how to implement Workers with the Python SDK

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Workers

- [Run a Worker](/develop/python/workers/run-worker-process)
- [Interceptors](/develop/python/workers/interceptors)
- [Serverless Workers](/develop/python/workers/serverless-workers)

---

# Interceptors - Python SDK

Source: https://docs.temporal.io/develop/python/workers/interceptors

Interceptors are SDK hooks that let you intercept inbound and outbound Temporal calls. You use them to add common
behavior across many calls, such as tracing and context propagation. 
This is similar to using middleware in web frameworks such as
[Django](https://docs.djangoproject.com/en/5.2/topics/http/middleware/),
[Starlette](https://www.starlette.io/middleware/), and
[Flask](https://flask.palletsprojects.com/en/stable/lifecycle/#middleware).

There are two types of interceptors--inbound and outbound.

* Outbound interceptors wrap network calls, running before they reach the network and after they return.
* Inbound interceptors run after the network hop, wrapping application code and running before it starts and after it returns.

Concretely, there are five categories of inbound and outbound calls that you can modify in this way:

| | [Outbound Client](https://python.temporal.io/temporalio.client.OutboundInterceptor.html) | [Inbound Workflow](https://python.temporal.io/temporalio.worker.WorkflowInboundInterceptor.html) | [Outbound Workflow](https://python.temporal.io/temporalio.worker.WorkflowOutboundInterceptor.html) | [Inbound Activity](https://python.temporal.io/temporalio.worker.ActivityInboundInterceptor.html) | [Outbound Activity](https://python.temporal.io/temporalio.worker.ActivityOutboundInterceptor.html) |
| --- | --- | --- | --- | --- | --- |
| **Description** | Wraps calls from your application to the Temporal Client to start a Workflow or send [Messages](/encyclopedia/workflow-message-passing) to it | Wraps calls arriving into a [Workflow Execution](/workflow-execution), such as executing the Workflow, handling [Messages](/encyclopedia/workflow-message-passing) | Wraps calls a [Workflow](/workflow-definition) makes to the SDK, such as scheduling [Activities](/activities), starting [Child Workflows](/child-workflows), and invoking [Nexus Operations](/nexus) | Wraps calls arriving into an [Activity Execution](/activity-execution) | Wraps calls an [Activity](/activities) makes to the SDK, such as sending [Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) and reading Activity info |
| **Runs on** | Client | Worker (Workflow sandbox) | Worker (Workflow sandbox) | Worker (Activity context) | Worker (Activity context) |
| **Example methods** | `start_workflow()`, `signal_workflow()`, `list_workflows()` | `execute_workflow()`, `handle_query()`, `handle_signal()`, `handle_update_handler()` | `start_activity()`, `start_child_workflow()`, `signal_child_workflow()`, `start_nexus_operation()` | `execute_activity()` | `info()`, `heartbeat()` |

These are not exhaustive lists; refer to the linked API docs for each category.

> **⚠️ Warning:**
> Workflow interceptors and replay
>
> Workflow inbound and outbound interceptor methods also execute during [replay](/develop/python/best-practices/testing-suite#replay). Use replay-safe APIs for logging, randomness, and time in these interceptors.
> See [Develop Workflow logic](/develop/python/workflows/basics#workflow-logic-requirements) for details.
>
> If you want to write generic code shared by all inbound Workflow call handlers but want to skip read-only operations, check `workflow.unsafe.is_read_only()`.
>
> Activity and Client interceptors are not affected by replay.
>

## Register an Interceptor 

Registering an interceptor means supplying an interceptor instance to the SDK so Temporal can invoke it when matching
Client or Worker calls occur. Once registered, the interceptor runs as part of the call path and can observe or modify
request and response data.

### Register on the Client 

Pass interceptors in the `interceptors` argument of `Client.connect()`. Client interceptors modify outbound calls such
as starting and signaling Workflows.

<!--SNIPSTART python-interceptors-register-client-->
[python-sandbox/workers/interceptors/register.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/interceptors/register.py)
```py
client = await Client.connect(
    "localhost:7233",
    interceptors=[TracingInterceptor()],
)
```
<!--SNIPEND-->

The `interceptors` list can contain multiple interceptors.
In this case they form a chain: a method implemented on an interceptor instance in the list can perform side effects, and modify the data, before passing it on to the corresponding method on the next interceptor in the list.

### Register via a Plugin

If you're building a reusable library or want to bundle interceptors with other primitives, you can register them through a [Plugin](/develop/plugins-guide#interceptors).

### Register on the Worker only 

If your interceptor doesn't affect the Client, you can pass interceptors in the `interceptors` argument of `Worker()`.
Worker interceptors modify inbound and outbound Workflow and Activity calls.

<!--SNIPSTART python-interceptors-register-worker-->
[python-sandbox/workers/interceptors/register.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/interceptors/register.py)
```py
worker = Worker(
    client,
    task_queue="my-task-queue",
    interceptors=[SomeWorkerInterceptor()],
    workflows=[GreetingWorkflow],
)
```
<!--SNIPEND-->

> **📝 Note:**
>
> If your interceptor class inherits from both `client.Interceptor` and `worker.Interceptor`, pass it to
> `Client.connect()` rather than the `Worker()` constructor. The Worker will use interceptors from its underlying Client
> automatically.
>

## How to implement Interceptors in Python

Interceptors run as a chain.  Each interceptor wraps the entire inner call: your code runs before the call, invokes `next` to execute the rest of the chain, and then runs after the call completes. This means you can inspect or modify both the `input` and the result, handle errors, and perform side effects at either stage.

### Implementing Client call Interceptors

To modify outbound Client calls, define a class inheriting from
[`client.Interceptor`](https://python.temporal.io/temporalio.client.Interceptor.html), and implement the method
`intercept_client()` to return an instance of
[`OutboundInterceptor`](https://python.temporal.io/temporalio.client.OutboundInterceptor.html) that implements the
subset of outbound Client calls that you wish to modify.

This example implements an Interceptor on outbound Client calls that sets a certain key in the outbound `headers` field.
A User ID is context-propagated by being sent in a header field with outbound requests:

<!--SNIPSTART python-interceptors-context-propagation-->
[python-sandbox/workers/interceptors/context_propagation.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/interceptors/context_propagation.py)
```py
class ContextPropagationInterceptor(
    temporalio.client.Interceptor, temporalio.worker.Interceptor
):
    def __init__(
        self,
        payload_converter: temporalio.converter.PayloadConverter = temporalio.converter.default().payload_converter,
    ) -> None:
        self._payload_converter = payload_converter

    def intercept_client(
        self, next: temporalio.client.OutboundInterceptor
    ) -> temporalio.client.OutboundInterceptor:
        return _ContextPropagationClientOutboundInterceptor(
            next, self._payload_converter
        )

def set_header_from_context(
    input: _InputWithHeaders, payload_converter: temporalio.converter.PayloadConverter
) -> None:
    user_id_val = user_id.get()
    if user_id_val:
        input.headers = {
            **input.headers,
            HEADER_KEY: payload_converter.to_payload(user_id_val),
        }

class _ContextPropagationClientOutboundInterceptor(
    temporalio.client.OutboundInterceptor
):
    def __init__(
        self,
        next: temporalio.client.OutboundInterceptor,
        payload_converter: temporalio.converter.PayloadConverter,
    ) -> None:
        super().__init__(next)
        self._payload_converter = payload_converter

    async def start_workflow(
        self, input: temporalio.client.StartWorkflowInput
    ) -> temporalio.client.WorkflowHandle[Any, Any]:
        set_header_from_context(input, self._payload_converter)
        return await super().start_workflow(input)
```
<!--SNIPEND-->

It often happens that your Worker and Client interceptors will share code because they implement closely related logic.
In the Python SDK, you will typically want to create an interceptor class that inherits from _both_ `client.Interceptor`
and `worker.Interceptor` as above, since their method sets do not overlap.

You can then [register](#register) this interceptor in your client/starter code.

Your interceptor classes need not implement every method; the default implementation is always to pass the data on to the next method in the interceptor chain.
During execution, when the SDK encounters an Inbound Activity call, it will look to the first Interceptor instance, get hold of the appropriate intercepted method, and call it.
The intercepted method will perform its function then call the same method on the next Interceptor in the chain.
At the end of the chain the SDK will call the "real" SDK method.

### Implementing Worker call Interceptors

To modify inbound and outbound Workflow and Activity calls, define a class inheriting from `worker.Interceptor`. This is
an interface with two methods named `intercept_activity` and `workflow_interceptor_class`, which you can use to
configure interceptions of Activity and Workflow calls, respectively. `intercept_activity` returns an
`ActivityInboundInterceptor`.

This example demonstrates using an interceptor to measure [Schedule-To-Start](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) and Schedule-To-Close latency.
Notice how the interceptor wraps the call: it records Schedule-To-Start before `execute_activity`, then records Schedule-To-Close after it completes:

<!--SNIPSTART python-interceptors-activity-metrics-->
[python-sandbox/workers/interceptors/activity_metrics.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/interceptors/activity_metrics.py)
```py
from datetime import datetime, timezone
from temporalio import activity
from temporalio.client import Client
from temporalio.worker import (
    ActivityInboundInterceptor,
    ExecuteActivityInput,
    Interceptor,
    Worker,
)

from interceptors.workflows import GreetingWorkflow, compose_greeting

class SimpleWorkerInterceptor(Interceptor):
    def intercept_activity(
        self, next: ActivityInboundInterceptor
    ) -> ActivityInboundInterceptor:
        return ActivityMetricsInterceptor(next)

class ActivityMetricsInterceptor(ActivityInboundInterceptor):
    async def execute_activity(self, input: ExecuteActivityInput):
        info = activity.info()
        meter = activity.metric_meter()
        attrs = {"workflow_type": info.workflow_type}

        # Before the activity executes: record Schedule-To-Start
        schedule_to_start = info.started_time - info.current_attempt_scheduled_time
        meter.create_histogram_timedelta(
            "custom_activity_schedule_to_start_latency",
            description="Time between activity scheduling and start",
            unit="duration",
        ).record(schedule_to_start, attrs)

        # Execute the activity
        result = await self.next.execute_activity(input)

        # After the activity completes: record Schedule-To-Close
        elapsed = datetime.now(timezone.utc) - info.current_attempt_scheduled_time
        meter.create_histogram_timedelta(
            "custom_activity_schedule_to_close_latency",
            description="Time between activity scheduling and completion",
            unit="duration",
        ).record(elapsed, attrs)

        return result

async def main():
    client = await Client.connect(
        "localhost:7233",
    )
    worker = Worker(
        client,
        task_queue="my-task-queue",
        workflows=[GreetingWorkflow],
        activities=[compose_greeting],
        interceptors=[SimpleWorkerInterceptor()],
    )
    await worker.run()
```
<!--SNIPEND-->

The `workflow_interceptor_class` returns a `WorkflowInboundInterceptor` that works similarly to
`ActivityInboundInterceptor`.

---

# Run a Worker - Python SDK

Source: https://docs.temporal.io/develop/python/workers/run-worker-process

> Create and run a Temporal Worker using the Python SDK.

This page covers long-lived Workers that you host and run as persistent processes.
For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/python/workers/serverless-workers).

## Create and run a Worker 

Create a `Worker` with a [Temporal Client](/develop/python/client/temporal-client), the Task Queue to poll, and the Workflows and Activities it can execute.
Call `run()` to start polling.

<!--SNIPSTART python-run-worker-create-->
[python-sandbox/workers/run_worker/create_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/run_worker/create_worker.py)
```py
client = await Client.connect("localhost:7233")

worker = Worker(
    client,
    task_queue="my-task-queue",
    workflows=[HelloWorkflow],
    activities=[some_activity],
)
await worker.run()
```
<!--SNIPEND-->

`run()` does not return on its own. It polls until you call `shutdown()`, then returns once shutdown is complete.
A Worker is also an async context manager: `async with worker:` starts it on entry and shuts it down on exit. See [Shut down a Worker](#shut-down-a-worker).

## Register Workflows and Activities 

All Workers polling the same Task Queue must register the same Workflow Types and Activity Types.
A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue.
A Worker that receives a Task for a type it did not register fails that Task.

Pass a list of Workflows in `workflows`, a list of Activities in `activities`, or both.

Activities defined with `async def` run on the Worker's event loop. Activities defined with a plain `def` are synchronous and require an executor, so pass one in `activity_executor`:

<!--SNIPSTART python-run-worker-sync-activities-->
[python-sandbox/workers/run_worker/sync_activity_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/run_worker/sync_activity_worker.py)
```py
worker = Worker(
    client,
    task_queue="my-task-queue",
    workflows=[MyWorkflow],
    activities=[my_sync_activity],
    activity_executor=ThreadPoolExecutor(5),
)
```
<!--SNIPEND-->

The same executor can be shared across multiple Workers.

## Connect to Temporal Cloud 

To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials.
See [Connect to Temporal Cloud](/develop/python/client/temporal-client#connect-to-temporal-cloud) for setup instructions.

## Configure Worker options 

The `Worker` constructor takes keyword arguments that control concurrency limits, pollers, timeouts, and caching, including `max_concurrent_activities`, `max_concurrent_workflow_tasks`, and `max_cached_workflows`.
The defaults work for most cases.

To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference).

## Run a versioned Worker 

Set a Worker Deployment Version and enable versioning in `deployment_config`, then set a default versioning behavior for the Workflows on the Worker.

<!--SNIPSTART python-run-worker-versioned-->
[python-sandbox/workers/run_worker/versioned_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/run_worker/versioned_worker.py)
```py
worker = Worker(
    client,
    task_queue="my-task-queue",
    workflows=[HelloWorkflow],
    activities=[some_activity],
    deployment_config=WorkerDeploymentConfig(
        version=WorkerDeploymentVersion(
            deployment_name="my-app",
            build_id="1.0",
        ),
        use_worker_versioning=True,
        default_versioning_behavior=VersioningBehavior.PINNED,
    ),
)
```
<!--SNIPEND-->

To set the behavior per Workflow instead of on the Worker, pass `versioning_behavior` to `@workflow.defn`.
See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out.

## Shut down a Worker 

Shut a Worker down by leaving the `async with` block, which calls `shutdown()` for you.
To keep the Worker running until the process is interrupted, create an `asyncio.Event` and wait on it inside the block.

Set that event from the entry point that catches `KeyboardInterrupt`:

<!--SNIPSTART python-run-worker-interrupt-entrypoint-->
[python-sandbox/workers/run_worker/graceful_shutdown.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/run_worker/graceful_shutdown.py)
```py
interrupt_event = asyncio.Event()

if __name__ == "__main__":
    loop = asyncio.new_event_loop()
    try:
        loop.run_until_complete(main())
    except KeyboardInterrupt:
        interrupt_event.set()
        loop.run_until_complete(loop.shutdown_asyncgens())
```
<!--SNIPEND-->

Waiting on `interrupt_event` inside the block holds the Worker open.
Once the event is set, the block exits, and the Worker stops polling for new Tasks and waits for in-flight Tasks to finish, up to `graceful_shutdown_timeout`.

<!--SNIPSTART python-run-worker-graceful-shutdown-->
[python-sandbox/workers/run_worker/graceful_shutdown.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/run_worker/graceful_shutdown.py)
```py
worker = Worker(
    client,
    task_queue="my-task-queue",
    workflows=[HelloWorkflow],
    activities=[some_activity],
    graceful_shutdown_timeout=timedelta(seconds=30),
)
async with worker:
    await interrupt_event.wait()
```
<!--SNIPEND-->

See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities.

---

# Serverless Workers - Python SDK

Source: https://docs.temporal.io/develop/python/workers/serverless-workers

> Write Temporal Workers that run on serverless compute using the Python SDK.

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview. Amazon Bedrock AgentCore Runtime support is in
> Pre-release. To request AgentCore Runtime access on Temporal Cloud, create a
> [support ticket](/evaluate/cloud/support#support-ticket) or contact your account team. You can also
> [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or try them through self-hosting.

Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes.
Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done.

For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers).
For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Supported providers

- [**AWS Lambda**](/develop/python/workers/serverless-workers/aws-lambda) - Use the `lambda_worker` contrib package to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, and observability.
- [**Amazon Bedrock AgentCore Runtime**](/develop/python/workers/serverless-workers/agentcore) - Run a standard Worker from an AgentCore Runtime handler. Covers the handler, Worker Versioning, connection configuration, and Worker shutdown.
- [**GCP Cloud Run**](/develop/python/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in.

---

# Serverless Workers on Amazon Bedrock AgentCore Runtime - Python SDK

Source: https://docs.temporal.io/develop/python/workers/serverless-workers/agentcore

> Run a Temporal Worker on Amazon Bedrock AgentCore Runtime using the Python SDK.

> **Pre-release**
> Amazon Bedrock AgentCore Runtime support is in Pre-release, and its APIs may change in backwards-incompatible ways.
> To request Temporal Cloud access, create a [support ticket](/evaluate/cloud/support#support-ticket) or contact your
> account team. You can also [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or
> [try it through self-hosting](/production-deployment/worker-deployments/serverless-workers/agentcore/self-hosted-setup).

On Amazon Bedrock AgentCore Runtime, you run a standard long-lived Python Worker inside an AgentCore Runtime handler.
Temporal starts the handler when the Worker Controller Instance needs capacity. The handler starts a Worker that polls
the Task Queue, then stops it when your idle policy decides to release capacity.

The Worker uses the normal Python SDK. The handler uses the `bedrock-agentcore` package to receive AgentCore Runtime
invocations.

For the provider behavior, including autoscaling, Worker Versioning, and the Runtime session lifecycle, see
[Serverless Workers on Amazon Bedrock AgentCore Runtime](/serverless-workers/agentcore).
For the infrastructure procedure, see
[Deploy a Serverless Worker on Amazon Bedrock AgentCore Runtime](/production-deployment/worker-deployments/serverless-workers/agentcore).

## Install the AgentCore Runtime SDK 

Install the AgentCore Runtime SDK alongside the Temporal Python SDK:

```bash
pip install bedrock-agentcore
```

## Create a versioned Worker 

Serverless Workers require [Worker Versioning](/worker-versioning). Create the Worker as you would any long-lived
Python Worker, then set `deployment_config` to declare its Worker Deployment Version and enable versioning:

<!--SNIPSTART python-agentcore-sandbox-runtime-handler {"selectedLines": ["13", "19-26", "28"]}-->
[python-sandbox/workers/agentcore_worker/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/agentcore_worker/worker.py)
```py
# ...
    worker = Worker(
# ...
        deployment_config=WorkerDeploymentConfig(
            version=WorkerDeploymentVersion(
                deployment_name=DEPLOYMENT_NAME,
                build_id=BUILD_ID,
            ),
            use_worker_versioning=True,
            default_versioning_behavior=VersioningBehavior.PINNED,
        ),
# ...
    )
```
<!--SNIPEND-->

`TEMPORAL_DEPLOYMENT_NAME` and `TEMPORAL_BUILD_ID` must match the Worker Deployment Version that you create with
`temporal worker deployment create-version`. Configure that Worker Deployment Version with the AgentCore Runtime
endpoint that Temporal invokes. For the endpoint configuration, see
[Worker Versioning](/serverless-workers/agentcore#worker-versioning).

Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or
`AUTO_UPGRADE`. Setting `default_versioning_behavior` as shown applies `PINNED` behavior to every Workflow on the
Worker. To set the behavior per Workflow instead, pass `versioning_behavior` to the `@workflow.defn` decorator.

## Start the Worker from the Runtime handler 

AgentCore Runtime invokes an HTTP handler. Use `BedrockAgentCoreApp` to provide that handler. Register the Worker as an
asynchronous AgentCore task, then return an acknowledgment while the Worker continues polling in the background. The
sample stores the background task in `_worker` and uses it to prevent another invocation from starting a duplicate
Worker in the same Runtime session:

<!--SNIPSTART python-agentcore-sandbox-runtime-handler-->
[python-sandbox/workers/agentcore_worker/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/agentcore_worker/worker.py)
```py
async def run_worker() -> None:
    """Poll until idle, then drain."""
    api_key = os.environ.get("TEMPORAL_API_KEY") or None
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        namespace=os.environ.get("TEMPORAL_NAMESPACE", "default"),
        api_key=api_key,
        tls=bool(api_key),
    )

    tracker = ActivityTracker()
    log.info("polling %s as %s/%s", TASK_QUEUE, DEPLOYMENT_NAME, BUILD_ID)
    worker = Worker(
        client,
        task_queue=TASK_QUEUE,
        workflows=[workflows.ProcessItemsWorkflow],
        activities=[my_activity],
        interceptors=[tracker],
        deployment_config=WorkerDeploymentConfig(
            version=WorkerDeploymentVersion(
                deployment_name=DEPLOYMENT_NAME,
                build_id=BUILD_ID,
            ),
            use_worker_versioning=True,
            default_versioning_behavior=VersioningBehavior.PINNED,
        ),
        graceful_shutdown_timeout=DRAIN,
    )
    async with worker:
        await tracker.wait_until_idle(DEBOUNCE)
    log.info("worker idle for %ss; drained", DEBOUNCE)

async def _run_until_idle(task_id: int) -> None:
    """Own the Worker's whole life, and always release the async task."""
    try:
        await run_worker()
    except Exception:
        # Nothing awaits this task, so an error would otherwise be swallowed.
        log.exception("worker failed in async task")
    finally:
        # Without this the session stays HealthyBusy until MaxLifetime.
        app.complete_async_task(task_id)

@app.entrypoint
async def invoke(payload: dict) -> dict:
    """Start the Worker and acknowledge. The payload is unused."""
    # Prevent duplicate workers since we exit early
    global _worker
    if _worker is not None and not _worker.done():
        log.info("worker already polling %s", TASK_QUEUE)
        return {"message": "worker already polling", "task_queue": TASK_QUEUE}

    task_id = app.add_async_task("temporal-worker")
    _worker = asyncio.create_task(_run_until_idle(task_id))

    return {"message": "worker starting", "task_queue": TASK_QUEUE}
```
<!--SNIPEND-->

The payload does not represent a Workflow input. The Worker Controller Instance invokes the endpoint to add Worker
capacity. Applications start Workflows through the Temporal Client, as usual. `add_async_task` causes AgentCore to
report the Runtime as busy while the Worker polls. `complete_async_task` releases that status after the Worker drains
or fails.

## Configure the Temporal connection 

The `temporalio.envconfig` package loads [Temporal Client](/develop/python/client/temporal-client) configuration from
environment variables and an optional TOML configuration file. Set the Temporal address, Namespace, Task Queue, and
Worker Deployment Version values as Runtime environment variables. Store a Temporal Cloud API key or TLS material in a
secret store rather than in the Runtime definition.

For the supported connection variables, config-file format, and profiles, see
[Environment configuration](/develop/environment-configuration).

## Stop and drain the Worker 

AgentCore cannot tell when a Worker that is still polling has no Temporal work. The Runtime remains busy while the
asynchronous task is registered, so it can remain active until its eight-hour maximum lifetime. To release capacity
sooner, have the handler detect when the Worker has no useful work and complete the asynchronous task.

When the condition remains true for an idle period, leave the `async with worker` block. The Worker stops polling for
new Tasks and gives in-flight Activities time to complete before the Runtime handler returns.

The following example from the
[AgentCore sample Worker](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/agentcore_worker/worker.py)
defines an `ActivityTracker`. It uses an [Activity inbound Interceptor](/develop/python/workers/interceptors) to count
running Activities.

<!--SNIPSTART python-agentcore-sandbox-activity-tracker-->
[python-sandbox/workers/agentcore_worker/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/agentcore_worker/worker.py)
```py
# How long the Worker keeps polling after it goes idle.
DEBOUNCE = float(os.environ.get("AGENTCORE_DEBOUNCE_SECONDS", "60"))
# How long the drain waits for in-flight Activities.
DRAIN = timedelta(seconds=120)

class ActivityTracker(Interceptor):
    """Tracks in-flight activities and blocks until AGENTCORE_DEBOUNCE_SECONDS elapses with no events."""

    def __init__(self) -> None:
        self.inflight = 0
        self.changed = asyncio.Event()

    def intercept_activity(
        self, next: ActivityInboundInterceptor
    ) -> ActivityInboundInterceptor:
        return _TrackedActivity(next, self)

    async def wait_until_idle(self, debounce: float) -> None:
        """Return once no Activity has run for ``debounce`` seconds."""
        while True:
            self.changed.clear()
            try:
                # Wake the moment an Activity starts or finishes; a timeout
                # instead means nothing has happened for the whole window.
                await asyncio.wait_for(self.changed.wait(), timeout=debounce)
            except asyncio.TimeoutError:
                if self.inflight == 0:
                    return

class _TrackedActivity(ActivityInboundInterceptor):
    def __init__(
        self, next: ActivityInboundInterceptor, tracker: ActivityTracker
    ) -> None:
        super().__init__(next)
        self._tracker = tracker

    async def execute_activity(self, input: ExecuteActivityInput):
        self._tracker.inflight += 1
        self._tracker.changed.set()
        log.info("activity in flight: %d", self._tracker.inflight)
        try:
            return await self.next.execute_activity(input)
        finally:
            self._tracker.inflight -= 1
            self._tracker.changed.set()
```
<!--SNIPEND-->

Register the tracker as a Worker Interceptor and wait for it inside the Worker context:

<!--SNIPSTART python-agentcore-sandbox-runtime-handler {"selectedLines": ["11", "13-14", "18", "27-30"]}-->
[python-sandbox/workers/agentcore_worker/worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/agentcore_worker/worker.py)
```py
# ...
    tracker = ActivityTracker()
# ...
    worker = Worker(
        client,
# ...
        interceptors=[tracker],
# ...
        graceful_shutdown_timeout=DRAIN,
    )
    async with worker:
        await tracker.wait_until_idle(DEBOUNCE)
```
<!--SNIPEND-->

`ActivityTracker` retires the Worker only after 60 seconds without an Activity starting or completing and with no
Activity running. A long-running Activity keeps the count above zero, so the idle policy does not interrupt it. The
two-minute `graceful_shutdown_timeout` is a safety limit for any Activity still in flight when shutdown starts.

Memory pressure can be another retirement condition. For example, the Runtime handler can monitor process memory and
initiate the same graceful shutdown when usage crosses a threshold. Memory usage is not an idle signal. It tells you
when to recycle a Worker, not whether it has work to do. Test any memory-based policy against the Runtime's memory
limit and your Activity retry behavior.

`AGENTCORE_DEBOUNCE_SECONDS` controls the idle period. `graceful_shutdown_timeout` controls how long the Worker waits
for in-flight Activities after it stops polling. Choose both values for your workload, and account for AgentCore's
maximum Runtime lifetime. For the AgentCore lifecycle settings, see
[Lifecycle](/serverless-workers/agentcore#lifecycle).

## Keep Activities safe across Worker termination 

AgentCore can end the compute that runs a Worker. An Activity running at that time can be interrupted and retried.
Use [Activity Heartbeats](/develop/python/activities/timeouts#activity-heartbeats) so a retry resumes from its last
recorded progress instead of starting over:

<!--SNIPSTART python-agentcore-heartbeat-activity-->
[python-sandbox/workers/agentcore_worker/activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/agentcore_worker/activities.py)
```py
from temporalio import activity

@activity.defn
async def my_activity(items: list[str]) -> str:
    # Resume after the last item a previous attempt recorded, if there was one.
    details = activity.info().heartbeat_details
    start = details[0] + 1 if details else 0
    for i, item in enumerate(items[start:], start=start):
        activity.heartbeat(i)
        # ... process item
    return "done"
```
<!--SNIPEND-->

## Add observability 

An AgentCore Runtime Worker emits the same traces and metrics as a Worker on other compute. For metrics export and
OpenTelemetry tracing interceptors, see [Observability - Python SDK](/develop/python/platform/observability) and the
[SDK metrics reference](/references/sdk-metrics).

---

# Serverless Workers on AWS Lambda - Python SDK

Source: https://docs.temporal.io/develop/python/workers/serverless-workers/aws-lambda

> Write a Temporal Worker that runs on AWS Lambda using the Python SDK lambda_worker package.

> **Public Preview**

The `lambda_worker` contrib package lets you run a Temporal Serverless Worker on AWS Lambda.
Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive.
Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline.
You register Workflows and Activities the same way you would with a standard Worker.

For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda).

## Create and run a Worker in Lambda 

Use the `run_worker` function to create a Lambda handler that runs a Temporal Worker.
Pass a `WorkerDeploymentVersion` and a configure callback that registers your Workflows and Activities.

<!--SNIPSTART python-lambda-worker-sandbox {"selectedLines": ["1-2", "4-12", "14-19"], "highlightedLines": "16-19"}-->
[python-sandbox/workers/aws_lambda_worker/lambda_function.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/aws_lambda_worker/lambda_function.py)
```py {16-19}
from temporalio.common import WorkerDeploymentVersion
from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker
# ...

from aws_lambda_worker.activities import hello_activity
from aws_lambda_worker.workflows import TASK_QUEUE, SampleWorkflow

def configure(config: LambdaWorkerConfig) -> None:
    config.worker_config["task_queue"] = TASK_QUEUE
    config.worker_config["workflows"] = [SampleWorkflow]
    config.worker_config["activities"] = [hello_activity]
# ...

lambda_handler = run_worker(
    WorkerDeploymentVersion(deployment_name="my-app", build_id="build-1"),
    configure,
)
```
<!--SNIPEND-->

`run_worker` takes a `WorkerDeploymentVersion` and a configure callback, and returns a Lambda handler.
The `WorkerDeploymentVersion` identifies the [Worker Deployment](/worker-versioning#deployments) and [Build ID](/worker-versioning#deployment-versions) for this Worker.
The deployment name groups related Workers across versions, and the Build Id identifies a specific release of your Worker code.
Worker Versioning is required for Serverless Workers.

The `configure` callback receives a `LambdaWorkerConfig` dataclass with fields pre-populated with Lambda-appropriate defaults.
Set the Task Queue, Workflows, and Activities through `worker_config`, which accepts the same keyword arguments as the `Worker` constructor.

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`.
Set it per-Workflow in the `@workflow.defn` decorator, or set a worker-level default with `default_versioning_behavior` in the worker config.

<!--SNIPSTART python-lambda-worker-sandbox-workflow {"highlightedLines": "12"}-->
[python-sandbox/workers/aws_lambda_worker/workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/aws_lambda_worker/workflows.py)
```py {12}
from datetime import timedelta

from temporalio import workflow
from temporalio.common import VersioningBehavior

with workflow.unsafe.imports_passed_through():
    from aws_lambda_worker.activities import hello_activity

TASK_QUEUE = "serverless-task-queue-python"

@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class SampleWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            hello_activity,
            name,
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

## Configure the Temporal connection 

The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details.

Compared with long-lived Workers, the location of the config file is resolved differently, in the following order:

1. `TEMPORAL_CONFIG_FILE` environment variable, if set.
2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`).
3. `temporal.toml` in the current working directory.

The file is optional. If absent, only environment variables are used.

Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options.

## Adjust Worker defaults for Lambda 

The `lambda_worker` package applies conservative defaults suited to short-lived Lambda invocations.
These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment.

| Setting | Lambda default |
|---|---|
| `max_concurrent_activities` | 2 |
| `max_concurrent_workflow_tasks` | 10 |
| `max_concurrent_local_activities` | 2 |
| `max_concurrent_nexus_tasks` | 5 |
| `workflow_task_poller_behavior` | `SimpleMaximum(2)` |
| `activity_task_poller_behavior` | `SimpleMaximum(1)` |
| `nexus_task_poller_behavior` | `SimpleMaximum(1)` |
| `graceful_shutdown_timeout` | 5 seconds |
| `max_cached_workflows` | 30 |
| `disable_eager_activity_execution` | Always `True` |
| `shutdown_deadline_buffer` | 7 seconds |

`disable_eager_activity_execution` is always `True` and cannot be overridden.
Eager Activities require a persistent connection, which Lambda invocations don't maintain.

`shutdown_deadline_buffer` is specific to the `lambda_worker` package.
It controls how much time before the Lambda deadline the Worker begins its graceful shutdown.
The default is `graceful_shutdown_timeout` + 2 seconds.

If your Worker handles long-running Activities, increase `graceful_shutdown_timeout`, `shutdown_deadline_buffer`, and the Lambda invocation deadline (`--timeout`) together.
For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities).

## Add observability with OpenTelemetry 

The `lambda_worker.otel` module provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer.
With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions.
The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch.

The underlying metrics and traces are the same ones the Python SDK emits in any environment.
For general observability concepts and the full list of available metrics, see [Observability - Python SDK](/develop/python/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).

<!--SNIPSTART python-lambda-worker-sandbox {"highlightedLines": "3,13"}-->
[python-sandbox/workers/aws_lambda_worker/lambda_function.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/aws_lambda_worker/lambda_function.py)
```py {3,13}
from temporalio.common import WorkerDeploymentVersion
from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker
from temporalio.contrib.aws.lambda_worker.otel import apply_defaults

from aws_lambda_worker.activities import hello_activity
from aws_lambda_worker.workflows import TASK_QUEUE, SampleWorkflow

def configure(config: LambdaWorkerConfig) -> None:
    config.worker_config["task_queue"] = TASK_QUEUE
    config.worker_config["workflows"] = [SampleWorkflow]
    config.worker_config["activities"] = [hello_activity]
    apply_defaults(config)

lambda_handler = run_worker(
    WorkerDeploymentVersion(deployment_name="my-app", build_id="build-1"),
    configure,
)
```
<!--SNIPEND-->

`apply_defaults` configures both metrics and tracing.
By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint.

To collect this telemetry, attach the [ADOT Python Lambda layer](https://aws-otel.github.io/docs/getting-started/lambda/lambda-python) to your Lambda function.
The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on `localhost:4317` and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch.

The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline.
You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines.
Bundle the following `otel-collector-config.yaml` in your Lambda deployment package:

<!--SNIPSTART python-lambda-worker-sandbox-otel-collector-config-->
[python-sandbox/workers/aws_lambda_worker/otel-collector-config.yaml](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/aws_lambda_worker/otel-collector-config.yaml)
```yaml
receivers:
    otlp:
        protocols:
            grpc:
                endpoint: "localhost:4317"
            http:
                endpoint: "localhost:4318"

exporters:
    debug:
    awsxray:
        region: us-west-2
    awsemf:
        # AWS EMF exporter for metrics
        # These are example configurations
        namespace: TemporalWorkerMetrics
        log_group_name: /aws/lambda/<your-function-name>
        region: us-west-2
        dimension_rollup_option: NoDimensionRollup
        resource_to_telemetry_conversion:
            enabled: true

service:
    pipelines:
        traces:
            receivers: [otlp]
            exporters: [awsxray, debug]
        metrics:
            receivers: [otlp]
            exporters: [awsemf]
    telemetry:
        logs:
            level: debug
        metrics:
            address: localhost:8888
```
<!--SNIPEND-->

Set the following environment variable on the Lambda function:

- `OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml`

Enable X-Ray active tracing on the Lambda function:

```bash
aws lambda update-function-configuration \
  --function-name <your-function-name> \
  --tracing-config Mode=Active
```

The Lambda execution role must have permissions to write to X-Ray and CloudWatch.
Attach the [`AWSXRayDaemonWriteAccess`](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSXRayDaemonWriteAccess.html) managed policy, or add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions.
Without these permissions, the Collector fails silently and no telemetry appears.

If you only need metrics or tracing, use `build_metrics_telemetry_config` or `apply_tracing` individually.

---

# Serverless Workers on GCP Cloud Run - Python SDK

Source: https://docs.temporal.io/develop/python/workers/serverless-workers/cloud-run

> Run a Temporal Worker on a GCP Cloud Run worker pool using the Python SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Python Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).

## Create a versioned Worker 

Build the Worker as you would any long-running Python Worker, then pass `deployment_config` to `Worker()` to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

<!--SNIPSTART python-cloud-run-versioned-worker-->
[python-sandbox/workers/cloud_run_worker/versioned_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/cloud_run_worker/versioned_worker.py)
```py
import asyncio
import os

from temporalio.client import Client
from temporalio.common import VersioningBehavior, WorkerDeploymentVersion
from temporalio.envconfig import ClientConfig
from temporalio.worker import Worker, WorkerDeploymentConfig

from cloud_run_worker.my_activities import my_activity
from cloud_run_worker.my_workflows import MyWorkflow

async def main() -> None:
    client = await Client.connect(**ClientConfig.load_client_connect_config())

    worker = Worker(
        client,
        task_queue=os.environ["TEMPORAL_TASK_QUEUE"],
        workflows=[MyWorkflow],
        activities=[my_activity],
        deployment_config=WorkerDeploymentConfig(
            version=WorkerDeploymentVersion(
                deployment_name="my-app",
                build_id="build-1",
            ),
            use_worker_versioning=True,
            default_versioning_behavior=VersioningBehavior.PINNED,
        ),
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

`deployment_name` and `build_id` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage.

Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`.
Setting `default_versioning_behavior` as shown above covers every Workflow on the Worker.
To set the behavior per Workflow instead, pass `versioning_behavior` to the `@workflow.defn` decorator:

<!--SNIPSTART python-cloud-run-workflow-versioning-behavior-->
[python-sandbox/workers/cloud_run_worker/my_workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/cloud_run_worker/my_workflows.py)
```py
from datetime import timedelta

from temporalio import workflow
from temporalio.common import VersioningBehavior

with workflow.unsafe.imports_passed_through():
    from cloud_run_worker.my_activities import my_activity

@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class MyWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            my_activity,
            [name],
            start_to_close_timeout=timedelta(minutes=5),
            heartbeat_timeout=timedelta(seconds=30),
        )
```
<!--SNIPEND-->

For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/python/workers/run-worker-process).

## Configure the Temporal connection 

The `temporalio.envconfig` package loads [Temporal Client](/develop/python/client/temporal-client) configuration from environment variables and an optional TOML config file, so the Worker code carries no Namespace or credentials.
Set the non-secret values as environment variables on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager.
For the full list of supported variables, the config file format, and profiles, see [Environment configuration](/develop/environment-configuration).

`ClientConfig.load_client_connect_config()` returns the keyword arguments for `Client.connect`, which is why the Worker above unpacks it with `**`.
To inspect or change values before connecting, load the profile instead and convert it yourself:

<!--SNIPSTART python-cloud-run-connect-profile-->
[python-sandbox/workers/cloud_run_worker/connect_profile.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/cloud_run_worker/connect_profile.py)
```py
from temporalio.envconfig import ClientConfigProfile

profile = ClientConfigProfile.load()
connect_config = profile.to_client_connect_config()
client = await Client.connect(**connect_config)
```
<!--SNIPEND-->

## Keep Activities safe across scale-in 

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing.
An instance running a long Activity can be stopped mid-execution.

Use [Activity Heartbeats](/develop/python/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over:

<!--SNIPSTART python-cloud-run-heartbeat-activity-->
[python-sandbox/workers/cloud_run_worker/my_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/cloud_run_worker/my_activities.py)
```py
from temporalio import activity

@activity.defn
async def my_activity(items: list[str]) -> str:
    # Resume after the last item a previous attempt recorded, if there was one.
    details = activity.info().heartbeat_details
    start = details[0] + 1 if details else 0
    for i, item in enumerate(items[start:], start=start):
        activity.heartbeat(i)
        # ... process item
    return "done"
```
<!--SNIPEND-->

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Configure OpenTelemetry 

Run an OpenTelemetry Collector as a sidecar in the Worker Pool. The Cloud Run OpenTelemetry plugin exports metrics and traces by OTLP gRPC to the Collector at `localhost:4317`. It configures the endpoint, service name, Core SDK metrics, and tracer provider from Cloud Run defaults.

Create the plugin and pass it to `Client.connect`. The sample also registers `CloudRunIdPlugin` on the same Client and
reads its connection settings from the environment:

<!--SNIPSTART python-cloud-run-sandbox-otel-->
[python-sandbox/workers/cloud_run_worker/otel_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/cloud_run_worker/otel_worker.py)
```py
# OpenTelemetryPlugin exports Core metrics and traces to the collector
# sidecar; CloudRunIdPlugin sets the client identity from instance metadata.
plugin = OpenTelemetryPlugin(add_temporal_spans=True)
client = await Client.connect(
    settings.address,
    namespace=settings.namespace,
    api_key=settings.api_key,
    tls=settings.tls,
    plugins=[plugin, CloudRunIdPlugin()],
)
```
<!--SNIPEND-->

Configure the Collector sidecar to receive OTLP gRPC on `localhost:4317`, export traces to Google Cloud, and export metrics to Google Managed Service for Prometheus:

<!--SNIPSTART python-cloud-run-sandbox-collector-config-->
[python-sandbox/workers/cloud_run_worker/collector-config.yaml](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/cloud_run_worker/collector-config.yaml)
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  batch/traces:
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20
  resource_detection:
    detectors: [gcp]
    timeout: 10s
  # Rename Temporal datapoint labels that collide with the target labels Managed Service for Prometheus injects (e.g. namespace).
  transform/collision:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["exported_location"], attributes["location"])
          - delete_key(attributes, "location")
          - set(attributes["exported_cluster"], attributes["cluster"])
          - delete_key(attributes, "cluster")
          - set(attributes["exported_namespace"], attributes["namespace"])
          - delete_key(attributes, "namespace")
          - set(attributes["exported_job"], attributes["job"])
          - delete_key(attributes, "job")
          - set(attributes["exported_instance"], attributes["instance"])
          - delete_key(attributes, "instance")
          - set(attributes["exported_project_id"], attributes["project_id"])
          - delete_key(attributes, "project_id")
  # The Telemetry API expects the Google Cloud project in gcp.project_id.
  transform/set_project_id:
    error_mode: ignore
    trace_statements:
      - set(resource.attributes["gcp.project_id"], resource.attributes["gcp.project.id"]) where resource.attributes["gcp.project.id"] != nil
      - set(resource.attributes["gcp.project_id"], resource.attributes["cloud.account.id"]) where resource.attributes["gcp.project_id"] == nil and resource.attributes["cloud.account.id"] != nil

exporters:
  googlemanagedprometheus:
  otlp_grpc:
    endpoint: telemetry.googleapis.com:443
    compression: none
    balancer_name: pick_first
    auth:
      authenticator: googleclientauth

extensions:
  googleclientauth:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [googleclientauth, health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection, transform/collision]
      exporters: [googlemanagedprometheus]
    traces:
      receivers: [otlp]
      processors:
        [memory_limiter, resource_detection, transform/set_project_id, batch/traces]
      exporters: [otlp_grpc]
```
<!--SNIPEND-->

Wait until the Collector accepts connections before starting the Worker. On shutdown, call `plugin.shutdown()` after the Worker stops so traces can flush. For a Collector configuration that exports traces to Google Cloud and metrics to Google Managed Service for Prometheus, see the [Python Cloud Run sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/workers/cloud_run_worker).

## Set a Worker identity 

Use `CloudRunIdPlugin` to identify each Worker instance as `<instance-id>@<revision>`. It reads Cloud Run environment variables and instance metadata when
the Client connects, then Workers created from that Client inherit the identity.

Register the plugin when connecting the Client:

<!--SNIPSTART python-cloud-run-identity {"selectedLines": ["3-4", "8", "12-19"]}-->
[python-sandbox/workers/cloud_run_worker/identity_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workers/cloud_run_worker/identity_worker.py)
```py
# ...
from temporalio.client import Client
from temporalio.contrib.gcp.cloud_run.id import CloudRunIdPlugin
# ...
from cloud_run_worker.settings import load_settings
# ...
    settings = load_settings()
    client = await Client.connect(
        settings.address,
        namespace=settings.namespace,
        api_key=settings.api_key,
        tls=settings.tls,
        plugins=[CloudRunIdPlugin()],
    )
```
<!--SNIPEND-->

The [preceding example](#opentelemetry) registers `CloudRunIdPlugin` alongside the OpenTelemetry plugin.

The ID plugin requires the Cloud Run metadata server, so it fails when the Worker runs outside Cloud Run. For a complete
example, see the [Python Cloud Run sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/workers/cloud_run_worker).

---

# Workflows - Python SDK

Source: https://docs.temporal.io/develop/python/workflows

> This section explains how to implement Workflows with the Python SDK

![Python SDK Banner](https://docs.temporal.io/img/assets/banner-python-temporal.png)

## Workflows

- [Workflow basics](/develop/python/workflows/basics)
- [Child Workflows](/develop/python/workflows/child-workflows)
- [Continue-As-New](/develop/python/workflows/continue-as-new)
- [Cancellation](/develop/python/workflows/cancellation)
- [Timeouts](/develop/python/workflows/timeouts)
- [Message passing](/develop/python/workflows/message-passing)
- [Schedules](/develop/python/workflows/schedules)
- [Timers](/develop/python/workflows/timers)
- [Versioning](/develop/python/workflows/versioning)
- [Workflow Streams](/develop/python/workflows/workflow-streams)

---

# Workflow basics - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/basics

> This section explains Workflow basics with the Python SDK

## Develop a basic Workflow 

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a
[Workflow Definition](/workflow-definition).

In the Temporal Python SDK programming model, Workflows are defined as classes.

Specify the `@workflow.defn` decorator on the Workflow class to identify a Workflow.

Use the `@workflow.run` to mark the entry point method to be invoked. This must be set on one asynchronous method
defined on the same class as `@workflow.defn`. Run methods have positional parameters.

<!--SNIPSTART python-workflow-basics-workflow {"highlightedLines": "9,11"}-->
[python-sandbox/workflows/workflow_basics/your_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/your_workflow.py)
```py {9,11}
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from workflow_basics.your_activities import your_activity
    from workflow_basics.your_dataobject import YourParams

@workflow.defn(name="YourWorkflow")
class YourWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            your_activity,
            YourParams("Hello", name),
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

### Define Workflow parameters 

Temporal Workflows may have any number of custom parameters. However, we strongly recommend that objects are used as
parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All
Workflow Definition parameters must be serializable. For more reasons, see [Workflow parameters](/workflow-definition#workflow-parameters).

Workflow parameters are the method parameters of the singular method decorated with `@workflow.run`. These can be any
data type Temporal can convert, including [`dataclasses`](https://docs.python.org/3/library/dataclasses.html) when
properly type-annotated. Technically this can be multiple parameters, but Temporal strongly encourages a single
`dataclass` parameter containing all input fields.

<!--SNIPSTART python-workflow-basics-params {"highlightedLines": "3"}-->
[python-sandbox/workflows/workflow_basics/your_dataobject.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/your_dataobject.py)
```py {3}
from dataclasses import dataclass

@dataclass
class YourParams:
    greeting: str
    name: str
```
<!--SNIPEND-->

### Define Workflow return parameters 

Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly
idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow
Execution will only ever receive one of either the result or the error.

To return a value of the Workflow, use `return` to return an object.

To return the results of a Workflow Execution, use either `start_workflow()` or `execute_workflow()` asynchronous
methods.

For performance and behavior reasons, users should pass through all modules, including Activities, Nexus services, and
third-party plugins, whose calls will be deterministic using
[`imports_passed_through`](https://python.temporal.io/temporalio.workflow.unsafe.html#imports_passed_through) or at
Worker creation time by customizing the runner's restrictions with
[`with_passthrough_modules`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#with_passthrough_modules).

<!--SNIPSTART python-workflow-basics-workflow {"highlightedLines": "5-7"}-->
[python-sandbox/workflows/workflow_basics/your_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/your_workflow.py)
```py {5-7}
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from workflow_basics.your_activities import your_activity
    from workflow_basics.your_dataobject import YourParams

@workflow.defn(name="YourWorkflow")
class YourWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            your_activity,
            YourParams("Hello", name),
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

### Customize your Workflow Type 

Workflows have a Type that are referred to as the Workflow name.

The following examples demonstrate how to set a custom name for your Workflow Type.

You can customize the Workflow name with a custom name in the decorator argument. For example,
`@workflow.defn(name="your-workflow-name")`. If the name parameter is not specified, the Workflow name defaults to the
unqualified class name.

<!--SNIPSTART python-workflow-basics-workflow-->
[python-sandbox/workflows/workflow_basics/your_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/your_workflow.py)
```py
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from workflow_basics.your_activities import your_activity
    from workflow_basics.your_dataobject import YourParams

@workflow.defn(name="YourWorkflow")
class YourWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            your_activity,
            YourParams("Hello", name),
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

### Use Workflow constructors

Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflow `__init__` method won't have any parameters. However, if you use the `@workflow.init` decorator on your `__init__` method, you can give it the same Workflow parameters as your `@workflow.run` method.

The SDK will then ensure that your `__init__` method receives the Workflow input arguments that the Client sent. The Workflow input arguments are also passed to your `@workflow.run` method. That always happens, whether or not you use the `@workflow.init` decorator.

Here's an example. Notice that `__init__` and `get_greeting` must have the same parameters, with the same type annotations:

<!--SNIPSTART python-workflow-basics-init-->
[python-sandbox/workflows/workflow_basics/workflow_init.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/workflow_init.py)
```py
@dataclass
class MyWorkflowInput:
    name: str

@workflow.defn
class WorkflowRunSeesWorkflowInitWorkflow:
    @workflow.init
    def __init__(self, workflow_input: MyWorkflowInput) -> None:
        self.name_with_title = f"Knight {workflow_input.name}"
        self.title_has_been_checked = False

    @workflow.run
    async def get_greeting(self, workflow_input: MyWorkflowInput) -> str:
        await workflow.wait_condition(lambda: self.title_has_been_checked)
        return f"Hello, {self.name_with_title}"
```
<!--SNIPEND-->

### Develop Workflow logic 

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints).
Each Temporal SDK provides a
set of APIs that can be used inside your Workflow to interact with external (to the Workflow) application code.

A Workflow Definition must be deterministic because the Temporal Server may [replay](/develop/python/best-practices/testing-suite#replay) your Workflow to reconstruct its state. This means:

- no threading
- no randomness
- no external calls to processes
- no network I/O
- no global state mutation
- no system date or time

All API safe for Workflows used in the [`temporalio.workflow`](https://python.temporal.io/temporalio.workflow.html) must
run in the implicit [`asyncio` event loop](https://docs.python.org/3/library/asyncio-eventloop.html) and be
_deterministic_.

The SDK provides replay-safe alternatives for common needs:

#### Logging

Use [`workflow.logger`](https://python.temporal.io/temporalio.workflow.html#logger) instead of `print()` or the standard `logging` module.
The SDK logger automatically suppresses log messages during replay to avoid duplicates:

<!--SNIPSTART python-workflow-basics-logger-->
[python-sandbox/workflows/workflow_basics/deterministic_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/deterministic_workflow.py)
```py
@workflow.defn
class DeterministicApisWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        workflow.logger.info("Starting workflow for %s", name)
```
<!--SNIPEND-->

For logger configuration, see [Observability: Log from a Workflow](/develop/python/platform/observability#logging).

#### Random numbers and UUIDs

Use [`workflow.random()`](https://python.temporal.io/temporalio.workflow.html#random) to get a deterministic `random.Random` instance seeded per Workflow Execution. Never use `random.random()` or other `random` module functions directly.
For UUIDs, use [`workflow.uuid4()`](https://python.temporal.io/temporalio.workflow.html#uuid4) instead of `uuid.uuid4()`:

<!--SNIPSTART python-workflow-basics-random-->
[python-sandbox/workflows/workflow_basics/deterministic_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/deterministic_workflow.py)
```py
value = workflow.random().randint(1, 100)
unique_id = workflow.uuid4()
```
<!--SNIPEND-->

#### Current time

Use [`workflow.now()`](https://python.temporal.io/temporalio.workflow.html#now) instead of `datetime.now()` or `time.time()`. The SDK returns the time of the last Workflow Task, which is consistent across replays:

<!--SNIPSTART python-workflow-basics-now-->
[python-sandbox/workflows/workflow_basics/deterministic_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/deterministic_workflow.py)
```py
current_time = workflow.now()
```
<!--SNIPEND-->

#### Detecting replay (advanced)

Use [`workflow.unsafe.is_replaying`](https://python.temporal.io/temporalio.workflow.html#is_replaying) to guard code that should only run on the first execution, such as emitting metrics or sending external notifications from an [Interceptor](/develop/python/workers/interceptors).
> **⚠️ Caution:**
>
> Never use this to affect Workflow business logic — branching on replay status breaks determinism.
>

<!--SNIPSTART python-workflow-basics-is-replaying-->
[python-sandbox/workflows/workflow_basics/deterministic_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_basics/deterministic_workflow.py)
```py
if not workflow.unsafe.is_replaying():
    emit_metric("workflow_started", 1)
```
<!--SNIPEND-->

If your goal is to always take action when something new is happening, check that `workflow.unsafe.is_replaying_history_events()` is false instead. This will be false during read-only operations like queries and update validators.  This is what the SDK's built-in logger and tracing interceptors use internally.

---

# Cancellation - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/cancellation

You can interrupt a Workflow Execution in one of the following ways:

- [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution.
- [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution.

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the
Workflow is stuck and cannot be canceled normally.

## Cancel a Workflow Execution 

Canceling a Workflow provides a graceful way to stop Workflow Execution. This action resembles sending a `SIGTERM` to a
process.

- The system records a `WorkflowExecutionCancelRequested` event in the Event History.
- A Workflow Task gets scheduled to process the cancelation.
- The Workflow code can handle the cancelation and execute any cleanup logic.
- The system doesn't forcefully stop the Workflow.

To cancel a Workflow Execution in Python, use the
[cancel()](https://python.temporal.io/temporalio.client.WorkflowHandle.html#cancel) function on the Workflow handle.

<!--SNIPSTART python-cancellation-cancel-workflow-->
[python-sandbox/workflows/cancellation/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/cancellation/client.py)
```py
await client.get_workflow_handle(workflow_id).cancel()
```
<!--SNIPEND-->

### Cancel an Activity from a Workflow 

Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat
Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate
Activity is executed, the Activity Execution should send Heartbeats and set a
[Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is
still working.

When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic
needs to be performed, it can be done in a `finally` clause or inside a caught cancel error. However, for the Activity
to appear canceled the exception needs to be re-raised.

> **📝 Note:**
>
> Unlike regular Activities, [Local Activities](/local-activity) can be canceled if they don't send Heartbeats. Local
> Activities are handled locally, and all the information needed to handle the cancellation logic is available in the same
> Worker process.
>

To cancel an Activity from a Workflow Execution, call the
[cancel()](https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel) method on the Activity handle that
is returned from [start_activity()](https://python.temporal.io/temporalio.workflow.html#start_activity).

<!--SNIPSTART python-cancellation-activity-->
[python-sandbox/workflows/cancellation/activity_cancellation.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/cancellation/activity_cancellation.py)
```py
@activity.defn
async def cancellable_activity(input: ComposeArgsInput) -> NoReturn:
    try:
        while True:
            print("Heartbeating cancel activity")
            await asyncio.sleep(0.5)
            activity.heartbeat("some details")
    except asyncio.CancelledError:
        print("Activity cancelled")
        raise

@activity.defn
async def run_activity(input: ComposeArgsInput):
    print("Executing activity")
    return input.arg1 + input.arg2

@workflow.defn
class GreetingWorkflow:
    @workflow.run
    async def run(self, input: ComposeArgsInput) -> None:
        activity_handle = workflow.start_activity(
            cancellable_activity,
            ComposeArgsInput(input.arg1, input.arg2),
            start_to_close_timeout=timedelta(minutes=5),
            heartbeat_timeout=timedelta(seconds=30),
        )

        await asyncio.sleep(3)
        activity_handle.cancel()
```
<!--SNIPEND-->

> **📝 Note:**
>
> The Activity handle is a Python task. By calling `cancel()`, you're essentially requesting the task to be canceled.
>

## Terminate a Workflow Execution 

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

To terminate a Workflow Execution in Python, use the
[terminate()](https://python.temporal.io/temporalio.client.WorkflowHandle.html#terminate) function on the Workflow
handle.

<!--SNIPSTART python-cancellation-terminate-workflow-->
[python-sandbox/workflows/cancellation/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/cancellation/client.py)
```py
await client.get_workflow_handle(workflow_id).terminate()
```
<!--SNIPEND-->

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Child Workflows - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/child-workflows

> Start a Child Workflow Execution and set a Parent Close Policy using the Temporal Python SDK. Ensure proper progress logging and specify Parent Workflow behavior upon closure.

This page shows how to do the following:

- [Start a Child Workflow Execution](#child-workflows)
- [Set a Parent Close Policy](#parent-close-policy)

## Start a Child Workflow Execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In Python, awaiting `start_child_workflow()` or `execute_child_workflow()` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call resolves.
If you start a Child Workflow from a non-main coroutine (for example, a Signal or Update handler), make sure the Parent Workflow doesn't complete before that call resolves.

To spawn a Child Workflow Execution in Python, use the [`execute_child_workflow()`](https://python.temporal.io/temporalio.workflow.html#execute_child_workflow) function which starts the Child Workflow and waits for completion or
use the [`start_child_workflow()`](https://python.temporal.io/temporalio.workflow.html#start_child_workflow) function to start a Child Workflow and return its handle.
This is useful if you want to do something after it has only started, or to get the Workflow/Run ID, or to be able to signal it while running.

> **📝 Note:**
>
> `execute_child_workflow()` is a helper function for `start_child_workflow()` plus `await handle`.
>

<!--SNIPSTART python-child-workflows-execute-->
[python-sandbox/workflows/child_workflows/workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/child_workflows/workflows.py)
```py
from temporalio import workflow
from temporalio.workflow import ParentClosePolicy

from child_workflows.dataobject import ComposeGreetingInput

@workflow.defn
class ComposeGreetingWorkflow:
    @workflow.run
    async def run(self, input: ComposeGreetingInput) -> str:
        return f"{input.greeting}, {input.name}!"

@workflow.defn
class GreetingWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_child_workflow(
            ComposeGreetingWorkflow.run,
            ComposeGreetingInput("Hello", name),
            id="hello-child-workflow-workflow-child-id",
            parent_close_policy=ParentClosePolicy.ABANDON,
        )
```
<!--SNIPEND-->

### Set a Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

Set the `parent_close_policy` parameter inside the [`start_child_workflow`](https://python.temporal.io/temporalio.workflow.html#start_child_workflow) function or the [`execute_child_workflow()`](https://python.temporal.io/temporalio.workflow.html#execute_child_workflow) function to specify the behavior of the Child Workflow when the Parent Workflow closes.

<!--SNIPSTART python-child-workflows-execute {"highlightedLines": "2,22"}-->
[python-sandbox/workflows/child_workflows/workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/child_workflows/workflows.py)
```py {2,22}
from temporalio import workflow
from temporalio.workflow import ParentClosePolicy

from child_workflows.dataobject import ComposeGreetingInput

@workflow.defn
class ComposeGreetingWorkflow:
    @workflow.run
    async def run(self, input: ComposeGreetingInput) -> str:
        return f"{input.greeting}, {input.name}!"

@workflow.defn
class GreetingWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_child_workflow(
            ComposeGreetingWorkflow.run,
            ComposeGreetingInput("Hello", name),
            id="hello-child-workflow-workflow-child-id",
            parent_close_policy=ParentClosePolicy.ABANDON,
        )
```
<!--SNIPEND-->

---

# Continue-As-New - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/continue-as-new

> Use Temporal's Continue-As-New in Python to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters.

This page covers the following for Python developers:

- [What is Continue-As-New?](#what)
- [Use Continue-As-New](#how)
- [When is it right to Continue-As-New?](#when)
- [Test Continue-As-New](#how-to-test)

## What is Continue-As-New? 

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution.
You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History.
It also receives your Workflow's usual parameters.

## Use Continue-As-New with the Python SDK 

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically set to `None` for the original caller of the Workflow.

<!--SNIPSTART python-continue-as-new-input-->
[python-sandbox/workflows/continue_as_new/workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/continue_as_new/workflow.py)
```py
@dataclass
class ClusterManagerInput:
    state: Optional[ClusterManagerState] = None
    test_continue_as_new: bool = False

@workflow.defn
class ClusterManagerWorkflow:
    @workflow.init
    def __init__(self, input: ClusterManagerInput) -> None:
        # The original caller passes state=None. Later runs receive the state from the previous run.
        self.state = input.state or ClusterManagerState()
        # Test hook: use a small history limit so tests reach Continue-As-New quickly.
        self.max_history_length: Optional[int] = (
            120 if input.test_continue_as_new else None
        )

    @workflow.run
    async def run(self, input: ClusterManagerInput) -> ClusterManagerResult:
```
<!--SNIPEND-->

The test hook in the above snippet is covered [below](#how-to-test).

Inside your Workflow, call the [`continue_as_new()`](https://python.temporal.io/temporalio.workflow.html#continue_as_new) function with the same type.
This stops the Workflow right away and starts a new one.

<!--SNIPSTART python-continue-as-new-call-->
[python-sandbox/workflows/continue_as_new/workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/continue_as_new/workflow.py)
```py
workflow.continue_as_new(
    ClusterManagerInput(
        state=self.state,
        test_continue_as_new=input.test_continue_as_new,
    )
)
```
<!--SNIPEND-->

### Considerations for Workflows with message handlers 

If you use Updates or Signals, don't call Continue-as-New from the handlers.
Instead, wait for your handlers to finish in your main Workflow before you run `continue_as_new`.
See the [`all_handlers_finished`](message-passing#wait-for-message-handlers) example for guidance.

## When is it right to Continue-As-New with the Python SDK? 

Use Continue-as-New when your Workflow might encounter degraded performance or [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New.
Call `workflow.info().is_continue_as_new_suggested()` to check if it's time.

## Test Continue-As-New with the Python SDK 

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive.
Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, when `test_continue_as_new == True`, this sample creates a test-only variable called `self.max_history_length` and sets it to a small value.
A helper method in the Workflow checks it each time it considers using Continue-as-New:

<!--SNIPSTART python-continue-as-new-should-continue-->
[python-sandbox/workflows/continue_as_new/workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/continue_as_new/workflow.py)
```py
def should_continue_as_new(self) -> bool:
    if workflow.info().is_continue_as_new_suggested():
        return True
    # For testing
    if (
        self.max_history_length
        and workflow.info().get_current_history_length() > self.max_history_length
    ):
        return True
    return False
```
<!--SNIPEND-->

---

# Workflow message passing - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/message-passing

> Develop with Queries, Signals, and Updates with the Temporal Python SDK.

A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates.
The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values.
Temporal Clients use messages to read Workflow state and control its execution.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic.
This page introduces these features for the Temporal Python SDK.

## Write message handlers 

> **ℹ️ Info:**
> The code that follows is part of a working message passing [sample](https://github.com/temporalio/documentation-sdk-code-examples/tree/main/python-sandbox/message_passing).

Follow these guidelines when writing your message handlers:

- Message handlers are defined as methods on the Workflow class, using one of the three decorators: [`@workflow.query`](https://python.temporal.io/temporalio.workflow.html#query), [`@workflow.signal`](https://python.temporal.io/temporalio.workflow.html#signal), and [`@workflow.update`](https://python.temporal.io/temporalio.workflow.html#update).
- The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion).
- Prefer [data classes](https://docs.python.org/3/library/dataclasses.html) to multiple input parameters. Data class parameters allow you to add fields without changing the calling signature. Keep in mind that serialization and deserialization can fail with the default data converter if the new field does not have a default value.

### Query handlers 

A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution:

<!--SNIPSTART python-message-passing-greeting-workflow {"selectedLines": ["1-13", "21-30", "38-44"]}-->
[python-sandbox/workflows/message_passing/greeting_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/greeting_workflow.py)
```py
class Language(IntEnum):
    ARABIC = 1
    CHINESE = 2
    ENGLISH = 3
    FRENCH = 4
    HINDI = 5
    PORTUGUESE = 6
    SPANISH = 7

@dataclass
class GetLanguagesInput:
    include_unsupported: bool
# ...
@workflow.defn
class GreetingWorkflow:
    def __init__(self) -> None:
        self.approved_for_release = False
        self.approver_name: Optional[str] = None
        self.greetings = {
            Language.CHINESE: "你好，世界",
            Language.ENGLISH: "Hello, world",
        }
        self.language = Language.ENGLISH
# ...
    @workflow.query
    def get_languages(self, input: GetLanguagesInput) -> list[Language]:
        # 👉 A Query handler returns a value: it can inspect but must not mutate the Workflow state.
        if input.include_unsupported:
            return list(Language)
        else:
            return list(self.greetings)
```
<!--SNIPEND-->

- The Query decorator can accept arguments.
  Refer to the API docs: [`@workflow.query`](https://python.temporal.io/temporalio.workflow.html#query).

- A Query handler uses `def`, not `async def`.
  You can't perform async operations like executing an Activity in a Query handler.

### Signal handlers 

A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow:

<!--SNIPSTART python-message-passing-greeting-workflow {"selectedLines": ["16-18", "21-22", "46-50"]}-->
[python-sandbox/workflows/message_passing/greeting_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/greeting_workflow.py)
```py
# ...
@dataclass
class ApproveInput:
    name: str
# ...
@workflow.defn
class GreetingWorkflow:
# ...
    @workflow.signal
    def approve(self, input: ApproveInput) -> None:
        # 👉 A Signal handler mutates the Workflow state but cannot return a value.
        self.approved_for_release = True
        self.approver_name = input.name
```
<!--SNIPEND-->

- The Signal decorator can accept arguments.
  Refer to the API docs: [`@workflow.signal`](https://python.temporal.io/temporalio.workflow.html#signal).

- The handler should not return a value.
  The response is sent immediately from the server, without waiting for the Workflow to process the Signal.

- Signal (and Update) handlers can be `async def`.
  This allows you to use Activities, Child Workflows, durable [`asyncio.sleep`](https://docs.python.org/3/library/asyncio-task.html#asyncio.sleep) Timers, [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Signal and Update handlers.

### Update handlers and validators 

An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution.
It can change the Workflow state, control its flow, and return a result.
The sender must wait until the Worker accepts or rejects the Update.
The sender may wait further to receive a returned value or an exception if something goes wrong:

<!--SNIPSTART python-message-passing-greeting-workflow {"selectedLines": ["1-8", "21-22", "52-62"]}-->
[python-sandbox/workflows/message_passing/greeting_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/greeting_workflow.py)
```py
class Language(IntEnum):
    ARABIC = 1
    CHINESE = 2
    ENGLISH = 3
    FRENCH = 4
    HINDI = 5
    PORTUGUESE = 6
    SPANISH = 7
# ...
@workflow.defn
class GreetingWorkflow:
# ...
    @workflow.update
    def set_language(self, language: Language) -> Language:
        # 👉 An Update handler can mutate the Workflow state and return a value.
        previous_language, self.language = self.language, language
        return previous_language

    @set_language.validator
    def validate_language(self, language: Language) -> None:
        if language not in self.greetings:
            # 👉 In an Update validator you raise any exception to reject the Update.
            raise ValueError(f"{language.name} is not supported")
```
<!--SNIPEND-->

- The Update decorator can take arguments (like, `name`, `dynamic` and `unfinished_policy`) as described in the API reference docs for [`workflow.update`](https://python.temporal.io/temporalio.workflow.html#update).

- About validators:
  - Use validators to reject an Update before it is written to History.
    Validators are always optional.
    If you don't need to reject Updates, you can skip them.
  - The SDK automatically provides a validator decorator named `@<update-handler-name>.validator`.
    The validator must accept the same argument types as the handler and return `None`.

- Accepting and rejecting Updates with validators:
  - To reject an Update, raise an exception of any type in the validator.
  - Without a validator, Updates are always accepted.
- Validators and Event History:
  - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic.
  - When a Validator raises an error, the Update is rejected and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History.
    The caller receives an "Update failed" error.

- Use [`workflow.current_update_info`](https://python.temporal.io/temporalio.workflow.html#current_update_info) to obtain information about the current Update.
  This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
- Update (and Signal) handlers can be `async def`, letting them use Activities, Child Workflows, durable [`asyncio.sleep`](https://docs.python.org/3/library/asyncio-task.html#asyncio.sleep) Timers, [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for safe usage guidelines.

## Send messages 

To send Queries, Signals, or Updates, you call methods on a [WorkflowHandle](https://python.temporal.io/temporalio.client.WorkflowHandle.html) object:

- Use [start_workflow](https://python.temporal.io/temporalio.client.Client.html#start_workflow) to start a Workflow and return its handle.

- Use [get_workflow_handle_for](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle_for) to retrieve a typed Workflow handle by its Workflow Id.

For example:

<!--SNIPSTART python-message-passing-start-workflow-->
[python-sandbox/workflows/message_passing/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/client.py)
```py
client = await Client.connect("localhost:7233")
workflow_handle = await client.start_workflow(
    GreetingWorkflow.run, id="greeting-workflow-1234", task_queue="my-task-queue"
)
```
<!--SNIPEND-->

To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition.

> **⚠️ Warning:**
> Using Continue-as-New and Updates
>
> - Temporal _does not_ support Continue-as-New functionality within Update handlers.
> - Complete all handlers _before_ using Continue-as-New.
> - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution.
>

### Send a Query 

Use [`WorkflowHandle.query`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#query) to send a Query to a Workflow Execution:

<!--SNIPSTART python-message-passing-send-query-->
[python-sandbox/workflows/message_passing/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/client.py)
```py
supported_languages = await workflow_handle.query(
    GreetingWorkflow.get_languages, GetLanguagesInput(include_unsupported=False)
)
```
<!--SNIPEND-->

- Sending a Query doesn’t add events to a Workflow's Event History.

- You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period.
  This includes Workflows that have completed, failed, or timed out.
  Querying terminated Workflows is not safe and, therefore, not supported.

- A Worker must be online and polling the Task Queue to process a Query.

### Send a Signal 

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution.
However, you can only send Signals to Workflow Executions that haven’t closed.

#### Send a Signal from a Client 

Use [`WorkflowHandle.signal`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#signal) to send a Signal:

<!--SNIPSTART python-message-passing-send-signal-->
[python-sandbox/workflows/message_passing/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/client.py)
```py
await workflow_handle.signal(GreetingWorkflow.approve, ApproveInput(name="me"))
```
<!--SNIPEND-->

- The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution.

- The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History.

#### Send a Signal from a Workflow 

A Workflow can send a Signal to another Workflow, known as an _External Signal_.
You'll need a Workflow handle for the external Workflow.
Use [`get_external_workflow_handle_for`](https://python.temporal.io/temporalio.workflow.html#get_external_workflow_handle_for):

<!--SNIPSTART python-message-passing-external-signal {"highlightedLines": "20-25"}-->
[python-sandbox/workflows/message_passing/external_signal.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/external_signal.py)
```py {20-25}
from typing import Optional

from temporalio import workflow

@workflow.defn
class WorkflowA:
    def __init__(self) -> None:
        self._signal: Optional[str] = None

    @workflow.run
    async def run(self) -> str:
        await workflow.wait_condition(lambda: self._signal is not None)
        return self._signal

    @workflow.signal
    def your_signal(self, value: str) -> None:
        self._signal = value

@workflow.defn
class WorkflowB:
    @workflow.run
    async def run(self) -> None:
        handle = workflow.get_external_workflow_handle_for(WorkflowA.run, "workflow-a")
        await handle.signal(WorkflowA.your_signal, "signal argument")
```
<!--SNIPEND-->

When an External Signal is sent:

- A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History.
- A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History.

#### Signal-With-Start 

Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running.
To use Signal-With-Start, call the [`start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) method and pass the `start_signal` argument with the name of your Signal:

<!--SNIPSTART python-message-passing-signal-with-start {"highlightedLines": "6-14"}-->
[python-sandbox/workflows/message_passing/signal_with_start.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/signal_with_start.py)
```py {6-14}
import asyncio

from temporalio.client import Client
from message_passing.wf_signal import GreetingWorkflow

async def main():
    client = await Client.connect("localhost:7233")
    await client.start_workflow(
        GreetingWorkflow.run,
        id="your-signal-with-start-workflow",
        task_queue="signal-tq",
        start_signal="submit_greeting",
        start_signal_args=["User Signal with Start"],
    )
    handle = client.get_workflow_handle_for(
        GreetingWorkflow, "your-signal-with-start-workflow"
    )
    await handle.signal(GreetingWorkflow.exit)
    result = await handle.result()
    print(result)
```
<!--SNIPEND-->

### Send an Update 

An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result.

A client sending an Update must wait until the Server delivers the Update to a Worker.
Workers must be available and responsive.
If you need a response as soon as the Server receives the request, use a Signal instead.
You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity.

- `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation.
- `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished.

To send an Update to a Workflow Execution, you can:

- Call [`execute_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#execute_update) and wait for the Update to complete.
  This code fetches an Update result:

  <!--SNIPSTART python-message-passing-execute-update-->
  [python-sandbox/workflows/message_passing/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/client.py)
  ```py
  previous_language = await workflow_handle.execute_update(
      GreetingWorkflow.set_language, Language.CHINESE
  )
  ```
  <!--SNIPEND-->

- Send [`start_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#start_update) to receive an [`UpdateHandle`](https://python.temporal.io/temporalio.client.WorkflowUpdateHandle.html) as soon as the Update is accepted.

  - Use this `UpdateHandle` later to fetch your results.
  - `async def` Update handlers normally perform long-running asynchronous operations, such as executing an Activity.
  - `start_update` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete.

  For example:

  <!--SNIPSTART python-message-passing-start-update-->
  [python-sandbox/workflows/message_passing/client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/client.py)
  ```py
  # Wait until the update is accepted
  update_handle = await workflow_handle.start_update(
      GreetingWorkflow.set_language,
      Language.ENGLISH,
      wait_for_stage=WorkflowUpdateStage.ACCEPTED,
  )
  # Wait until the update is completed
  update_result = await update_handle.result()
  ```
  <!--SNIPEND-->

  For more details, see the "Async handlers" section.

To obtain an Update handle, you can:

- Use [`start_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#start_update) to start an Update and return the handle, as shown in the preceding example.
- Use [`get_update_handle_for`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#get_update_handle_for) to fetch a handle for an in-progress Update using the Update ID.

#### Update-With-Start 

> **💡 Tip:**
>
> For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended.
>

[Update-with-Start](/sending-messages#update-with-start) lets you
[send an Update](/develop/python/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists:

- If the Workflow exists, the Update is processed.
- If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute.

Use [`execute_update_with_start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_update_with_start_workflow) to start the Update and wait for the result in one go.

Alternatively, use [`start_update_with_start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_update_with_start_workflow) to start the Update and receive a [`WorkflowUpdateHandle`](https://python.temporal.io/temporalio.client.WorkflowUpdateHandle.html), and then use `await update_handle.result()` to retrieve the result from the Update.

These calls return once the requested Update wait stage has been reached, or when the request times out.

You will need to provide a [`WithStartWorkflowOperation`](https://python.temporal.io/temporalio.client.WithStartWorkflowOperation.html) to define the Workflow that will be started if necessary, and its arguments.
You must specify a [WorkflowIdConflictPolicy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) when creating the `WithStartWorkflowOperation`.
Note that a `WithStartWorkflowOperation` can only be used once.

For example:

<!--SNIPSTART python-message-passing-update-with-start-->
[python-sandbox/workflows/message_passing/shopping_cart_starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/shopping_cart_starter.py)
```py
start_op = WithStartWorkflowOperation(
    ShoppingCartWorkflow.run,
    id=cart_id,
    id_conflict_policy=common.WorkflowIDConflictPolicy.USE_EXISTING,
    task_queue="my-task-queue",
)
try:
    price = Decimal(
        await temporal_client.execute_update_with_start_workflow(
            ShoppingCartWorkflow.add_item,
            ShoppingCartItem(sku=item_id, quantity=quantity),
            start_workflow_operation=start_op,
        )
    )
except WorkflowUpdateFailedError:
    price = None

workflow_handle = await start_op.workflow_handle()

return price, workflow_handle
```
<!--SNIPEND-->

> **ℹ️ Info:**
> SEND MESSAGES WITHOUT TYPE SAFETY
>
> In real-world development, sometimes you may be unable to import Workflow Definition method signatures.
> When you don't have access to the Workflow Definition or it isn't written in Python, you can still use APIs that aren't type-safe, and dynamic method invocation.
> Pass method names instead of method objects to:
>
> - [`Client.start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_workflow)
> - [`WorkflowHandle.query`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#query)
> - [`WorkflowHandle.signal`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#signal)
> - [`WorkflowHandle.execute_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#execute_update)
> - [`WorkflowHandle.start_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#start_update)
>
> Use these non-type safe APIs:
>
> - [`get_workflow_handle`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle)
> - [`get_external_workflow_handle`](https://python.temporal.io/temporalio.workflow.html#get_external_workflow_handle).
>

## Message handler patterns 

This section covers common write operations, such as Signal and Update handlers.
It doesn't apply to pure read operations, like Queries or Update Validators.

> **💡 Tip:**
>
> For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing), and [this sample](https://github.com/temporalio/samples-python/blob/message-passing/message_passing/safe_message_handlers/README.md) demonstrating safe `async` message handling.
>

### Use async handlers 

Signal and Update handlers can be `async def` as well as `def`.
Using `async def` allows you to use `await` with Activities, Child Workflows, [`asyncio.sleep`](https://docs.python.org/3/library/asyncio-task.html#asyncio.sleep) Timers, [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) conditions, and so on.
This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at `await` calls.
To use `async def` handlers safely, understand what can go wrong.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, the [Safe message handlers](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) sample, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below.

The following code executes an Activity that makes a network call to a remote service.
It modifies the Update handler from earlier on this page, turning it into an `async def`:

<!--SNIPSTART python-message-passing-async-handler {"selectedLines": ["12-23", "26-29", "47-67"]}-->
[python-sandbox/workflows/message_passing/async_greeting_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/async_greeting_workflow.py)
```py
# ...
@activity.defn
async def call_greeting_service(to_language: Language) -> Optional[str]:
    await asyncio.sleep(0.2)  # Pretend that we are calling a remote service.
    greetings = {
        Language.ARABIC: "مرحبا بالعالم",
        Language.CHINESE: "你好，世界",
        Language.ENGLISH: "Hello, world",
        Language.FRENCH: "Bonjour, monde",
        Language.HINDI: "नमस्ते दुनिया",
        Language.SPANISH: "Hola mundo",
    }
    return greetings.get(to_language)
# ...
@workflow.defn
class GreetingWorkflow:
    def __init__(self) -> None:
        self.lock = asyncio.Lock()
# ...
    @workflow.update
    async def set_language(self, language: Language) -> Language:
        if language not in self.greetings:
            # 👉 Use a lock here to ensure that multiple calls to set_language are processed in order.
            async with self.lock:
                greeting = await workflow.execute_activity(
                    call_greeting_service,
                    language,
                    start_to_close_timeout=timedelta(seconds=10),
                )
                if greeting is None:
                    # 👉 An update validator cannot be async, so cannot be used to check that the remote
                    # call_greeting_service supports the requested language. Raising ApplicationError
                    # will fail the Update, but the WorkflowExecutionUpdateAccepted event will still be
                    # added to history.
                    raise ApplicationError(
                        f"Greeting service does not support {language.name}"
                    )
                self.greetings[language] = greeting
        previous_language, self.language = self.language, language
        return previous_language
```
<!--SNIPEND-->

After updating the code to use a `async def`, your Update handler can schedule an Activity and await the result.
Although an `async def` Signal handler can also execute an Activity, using an Update handler allows the Client to receive a result or error once the Activity completes.
This lets your client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, and so on.

### Add wait conditions to block

Sometimes, `async def` Signal or Update handlers need to meet certain conditions before they should continue.
You can use [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) to prevent the code from proceeding until a condition is true.
You specify the condition by passing a function that returns `True` or `False`, and you can optionally set a timeout.
This is an important feature that helps you control your handler logic.

Here are three important use cases for `workflow.wait_condition`:

- Wait for a Signal or Update to arrive.
- Wait in a handler until it's appropriate to continue.
- Wait in the main Workflow until all active handlers have finished.

#### Wait for a Signal or Update to arrive

It's common to use `workflow.condition` to wait for a particular Signal or Update to be sent by a Client:

<!--SNIPSTART python-message-passing-greeting-workflow {"selectedLines": ["21-36", "46-50"]}-->
[python-sandbox/workflows/message_passing/greeting_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/greeting_workflow.py)
```py
# ...
@workflow.defn
class GreetingWorkflow:
    def __init__(self) -> None:
        self.approved_for_release = False
        self.approver_name: Optional[str] = None
        self.greetings = {
            Language.CHINESE: "你好，世界",
            Language.ENGLISH: "Hello, world",
        }
        self.language = Language.ENGLISH

    @workflow.run
    async def run(self) -> str:
        await workflow.wait_condition(lambda: self.approved_for_release)
        await workflow.wait_condition(workflow.all_handlers_finished)
        return self.greetings[self.language]
# ...
    @workflow.signal
    def approve(self, input: ApproveInput) -> None:
        # 👉 A Signal handler mutates the Workflow state but cannot return a value.
        self.approved_for_release = True
        self.approver_name = input.name
```
<!--SNIPEND-->

#### Use wait conditions in handlers 

It's common to use a Workflow wait condition to wait until a handler should start.
You can also use wait conditions anywhere else in the handler to wait for a specific condition to become `True`.
This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become `True`.

Consider a `ready_for_update_to_execute` method that runs before your Update handler executes.
The `workflow.wait_condition` method waits until your condition is met:

<!--SNIPSTART python-message-passing-wait-in-handler-->
[python-sandbox/workflows/message_passing/wait_conditions.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/wait_conditions.py)
```py
@workflow.update
async def my_update(self, update_input: UpdateInput) -> str:
    await workflow.wait_condition(
        lambda: self.ready_for_update_to_execute(update_input)
    )
```
<!--SNIPEND-->

You can also use wait conditions anywhere else in the handler to wait for a specific condition to become true.
This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become true.

#### Ensure your handlers finish before the Workflow completes 

Workflow wait conditions can ensure your handler completes before a Workflow finishes.
When your Workflow uses `async def` Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result.
The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying retrieve Update results.
Use [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) and [`all_handlers_finished`](https://python.temporal.io/temporalio.workflow.html#all_handlers_finished) to address this problem and allow your Workflow to end smoothly:

<!--SNIPSTART python-message-passing-wait-all-handlers-->
[python-sandbox/workflows/message_passing/wait_conditions.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/wait_conditions.py)
```py
@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> str:
        await workflow.wait_condition(lambda: self.ready)
        await workflow.wait_condition(workflow.all_handlers_finished)
        return "workflow-result"
```
<!--SNIPEND-->

By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions.
You can silence these warnings on a per-handler basis by passing the `unfinished_policy` argument to the [`@workflow.signal`](https://python.temporal.io/temporalio.workflow.html#signal) / [`workflow.update`](https://python.temporal.io/temporalio.workflow.html#update) decorator:

<!--SNIPSTART python-message-passing-unfinished-policy-->
[python-sandbox/workflows/message_passing/wait_conditions.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/wait_conditions.py)
```py
@workflow.update(unfinished_policy=workflow.HandlerUnfinishedPolicy.ABANDON)
async def my_update(self) -> None:
    await workflow.sleep(timedelta(hours=1))
```
<!--SNIPEND-->

See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information.

### Use `@workflow.init` to operate on Workflow input before any handler executes

Normally, your Workflow `__init__` method won't have any parameters.
However, if you use the `@workflow.init` decorator on your `__init__` method, you can give it the same [Workflow parameters](/develop/python/workflows/basics#workflow-parameters) as your `@workflow.run` method.
The SDK will then ensure that your `__init__` method receives the Workflow input arguments that the [Client sent](/develop/python/client/temporal-client#start-workflow-execution).
(The Workflow input arguments are also passed to your `@workflow.run` method -- that always happens, whether or not you use the `@workflow.init` decorator.)
This is useful if you have message handlers that need access to workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers).

Here's an example.
Notice that `__init__` and `get_greeting` must have the same parameters, with the same type annotations:

<!--SNIPSTART python-message-passing-workflow-init-->
[python-sandbox/workflows/message_passing/workflow_init.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/workflow_init.py)
```py
@dataclass
class MyWorkflowInput:
    name: str

@workflow.defn
class WorkflowRunSeesWorkflowInitWorkflow:
    @workflow.init
    def __init__(self, workflow_input: MyWorkflowInput) -> None:
        self.name_with_title = f"Sir {workflow_input.name}"
        self.title_has_been_checked = False

    @workflow.run
    async def get_greeting(self, workflow_input: MyWorkflowInput) -> str:
        await workflow.wait_condition(lambda: self.title_has_been_checked)
        return f"Hello, {self.name_with_title}"

    @workflow.update
    async def check_title_validity(self) -> bool:
        # 👉 The handler is now guaranteed to see the workflow input
        # after it has been processed by __init__.
        is_valid = await workflow.execute_activity(
            check_title_validity,
            self.name_with_title,
            schedule_to_close_timeout=timedelta(seconds=10),
        )
        self.title_has_been_checked = True
        return is_valid
```
<!--SNIPEND-->

### Use `asyncio.Lock` to prevent concurrent handler execution 

Concurrent processes can interact in unpredictable ways.
Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously.
Here's an example of a pathological case:

<!--SNIPSTART python-message-passing-handler-concurrency {"selectedLines": ["20-21", "33-42"]}-->
[python-sandbox/workflows/message_passing/handler_concurrency.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/handler_concurrency.py)
```py
# ...
@workflow.defn
class MyWorkflow:
# ...
    @workflow.signal
    async def bad_async_handler(self):
        data = await workflow.execute_activity(
            fetch_data, start_to_close_timeout=timedelta(seconds=10)
        )
        self.x = data.x
        # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then
        # there may be times when the Workflow has self.x from one Activity execution and self.y from another.
        await asyncio.sleep(1)  # or await anything else
        self.y = data.y
```
<!--SNIPEND-->

Coordinating access using `asyncio.Lock` corrects this code.
Locking makes sure that only one handler instance can execute a specific section of code at any given time:

<!--SNIPSTART python-message-passing-handler-concurrency {"selectedLines": ["20-25", "44-54"]}-->
[python-sandbox/workflows/message_passing/handler_concurrency.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/handler_concurrency.py)
```py
# ...
@workflow.defn
class MyWorkflow:
    def __init__(self) -> None:
        self.x: Optional[int] = None
        self.y: Optional[int] = None
        self.lock = asyncio.Lock()
# ...
    @workflow.signal
    async def safe_async_handler(self):
        async with self.lock:
            data = await workflow.execute_activity(
                fetch_data, start_to_close_timeout=timedelta(seconds=10)
            )
            self.x = data.x
            # ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow
            # method, but no other execution of this handler can run until this execution finishes.
            await asyncio.sleep(1)  # or await anything else
            self.y = data.y
```
<!--SNIPEND-->

## Message handler troubleshooting 

When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors:

- **The client can't contact the server**:
  You'll receive a [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) on which the `status` attribute is [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `UNAVAILABLE` (after some retries; see the `retry_config` argument to [`Client.connect`](https://python.temporal.io/temporalio.client.Client.html#connect)).

- **The workflow does not exist**:
  You'll receive an [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception on which the `status` attribute is [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `NOT_FOUND`.

See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Python-specific discussion of this topic.

### Problems when sending a Signal 

When using Signal, the only exceptions that will result from your requests during its execution are the `RPCError`s described above.

For Queries and Updates, the Client waits for a response from the Worker, and therefore additional errors may occur during the handler Execution by the Worker.

### Problems when sending an Update 

When working with Updates, in addition to the `RPCError`s described above, you may encounter these errors:

- **No Workflow Workers are polling the Task Queue**:
  Your request will be retried by the SDK Client indefinitely.
  You can use [`asyncio.timeout`](https://docs.python.org/3/library/asyncio-task.html#timeouts) to impose a timeout.
  This raises a [`temporalio.client.WorkflowUpdateRPCTimeoutOrCancelledError`](https://python.temporal.io/temporalio.client.WorkflowUpdateRPCTimeoutOrCancelledError.html) exception.

- **Update failed**: You'll receive a [`temporalio.client.WorkflowUpdateFailedError`](https://python.temporal.io/temporalio.client.WorkflowUpdateFailedError.html) exception.
  There are two ways this can happen:

  - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler.

  - The Update failed after having been accepted.

  Update failures are like [Workflow failures](/references/failures#errors-in-workflows).
  Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.
  These might include:

      - A failed Child Workflow
      - A failed Activity (if the Activity retries have been set to a finite number)
      - The Workflow author raising [`ApplicationError`](/references/failures#application-failure)
      - Any error listed in [workflow_failure_exception_types](https://python.temporal.io/temporalio.worker.Worker.html) (empty by default)

- **The handler caused the Workflow Task to fail**:
  A [Workflow Task Failure](/references/failures#errors-in-workflows) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage:
  - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception.
  - If the request has been accepted, it is durable.
    Once the Workflow is healthy again after a code deploy, use an [`UpdateHandle`](https://python.temporal.io/temporalio.client.WorkflowUpdateHandle.html) to fetch the Update result.

- **The Workflow finished while the Update handler execution was in progress**:
  You'll receive a [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception with a `status` attribute of [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `NOT_FOUND`.
  This happens if the Workflow finished while the Update handler execution was in progress, for example because

  - The Workflow was canceled or failed.

  - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers).

### Problems when sending a Query 

When working with Queries, in addition to the `RPCError`s described above, you may encounter these errors:

- **There is no Workflow Worker polling the Task Queue**:
  You'll receive a [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception on which the `status` attribute is [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `FAILED_PRECONDITION`.

- **Query failed**:
  You'll receive a [`temporalio.client.WorkflowQueryFailedError`](https://python.temporal.io/temporalio.client.WorkflowQueryFailedError.html) exception if something goes wrong during a Query.
  Any exception in a Query handler will trigger this error.
  This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead.

- **The handler caused the Workflow Task to fail.**
  This would happen, for example, if the Query handler blocks the thread for too long without yielding.

## Dynamic components 

A dynamic Workflow, Activity, Signal, Update, or Query is a kind of unnamed item.
Normally, these items are registered by name with the Worker and invoked at runtime.
When an unregistered or unrecognized Workflow, Activity, or message request arrives with a recognized method signature, the Worker can use a pre-registered dynamic stand-in.

For example, you might send a request to start a Workflow named "MyUnknownWorkflow".
After receiving a Workflow Task, the Worker may find that there's no registered Workflow Definitions of that type.
It then checks to see if there's a registered dynamic Workflow.
If the dynamic Workflow signature matches the incoming Workflow signature, the Worker invokes that just as it would invoke a non-dynamic statically named version.

By registering dynamic versions of your Temporal components, the Worker can fall back to these alternate implementations for name mismatches.

> **⚠️ Caution:**
>
> Use dynamic elements judiciously and as a fallback mechanism, not a primary design.
> They can introduce long-term maintainability and debugging issues.
> Reserve dynamic invocation use for cases where a name is not or can't be known at compile time.
>

### Set a dynamic Signal, Query, or Update handler 

A dynamic Signal, Query, or Update refers to a special stand-in handler.
It's used when an unregistered handler request arrives.

Consider a Signal, where you might send something like `workflow.signal(MyWorkflow.my_signal_method, my_arg)`.
This is a type-safe compiler-checked approach that guarantees a method exists.
There's also a non-type-safe string-based form: `workflow.signal('some-name', my_arg)`.
When sent to the server, the name is checked only after arriving at the Worker.
This is where "dynamic handlers" come in.

After failing to find a handler with a matching name and type, the Worker checks for a registered dynamic stand-in handler.
If found, the Worker uses that instead.

You must opt handlers into dynamic access.
Add `dynamic=True` to the handler decorator (for example, `@workflow.signal(dynamic=True)`) to make a handler dynamic.
The handler's signature must accept `(self, name: str, args: Sequence[RawValue])`.
Use a [payload_converter](https://python.temporal.io/temporalio.workflow.html#payload_converter) function to convert `RawValue` objects to your required type.
For example:

<!--SNIPSTART python-message-passing-dynamic-signal {"selectedLines": ["1-3", "17-20"]}-->
[python-sandbox/workflows/message_passing/dynamic_handlers.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/dynamic_handlers.py)
```py
from collections.abc import Sequence

from temporalio.common import RawValue
# ...
    @workflow.signal(dynamic=True)
    async def dynamic_signal(self, name: str, args: Sequence[RawValue]) -> None:
        value = workflow.payload_converter().from_payload(args[0].payload, str)
        self.received.append(f"{name}: {value}")
```
<!--SNIPEND-->

This sample creates a `dynamic_signal` Signal.
When an unregistered or unrecognized Signal arrives with a matching signature, dynamic assignment uses this handler to manage the Signal.
It is responsible for transforming the sequence contents into usable data in a form that the method's logic can process and act on.

### Set a dynamic Workflow 

A dynamic Workflow refers to a special stand-in Workflow Definition.
It's used when an unknown Workflow Execution request arrives.

Consider the "MyUnknownWorkflow" example described earlier.
The Worker may find there's no registered Workflow Definitions of that name or type.
After failing to find a Workflow Definition with a matching type, the Worker looks for a dynamic stand-in.
If found, it invokes that instead.

To participate, your Workflow must opt into dynamic access.
Adding `dynamic=True` to the `@workflow.defn` decorator makes the Workflow Definition eligible to participate in dynamic invocation.
You must register the Workflow with the [Worker](https://python.temporal.io/temporalio.worker.html) before it can be invoked.

The Workflow Definition's primary Workflow method must accept a single argument of type `Sequence[temporalio.common.RawValue]`.
Use a [payload_converter](https://python.temporal.io/temporalio.workflow.html#payload_converter) function to convert `RawValue` objects to your required type.
For example:

<!--SNIPSTART python-message-passing-dynamic-workflow {"highlightedLines": "19-28"}-->
[python-sandbox/workflows/message_passing/dynamic_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/dynamic_workflow.py)
```py {19-28}
from dataclasses import dataclass
from datetime import timedelta
from collections.abc import Sequence

from temporalio import activity, workflow
from temporalio.common import RawValue

@dataclass
class YourDataClass:
    greeting: str
    name: str

@activity.defn()
async def default_greeting(input: YourDataClass) -> str:
    return f"{input.greeting}, {input.name}!\nActivity Type: {activity.info().activity_type}"

@workflow.defn(dynamic=True)
class DynamicWorkflow:
    @workflow.run
    async def run(self, args: Sequence[RawValue]) -> str:
        name = workflow.payload_converter().from_payload(args[0].payload, str)
        return await workflow.execute_activity(
            default_greeting,
            YourDataClass("Hello", name),
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

This Workflow converts the first `Sequence` element to a string, and uses that to execute an Activity.

### Set a dynamic Activity 

A dynamic Activity is a stand-in implementation.
It's used when an Activity Task with an unknown Activity type is received by the Worker.

To participate, your Activity must opt into dynamic access.
Adding `dynamic=True` to the `@activity.defn` decorator makes the Activity Definition eligible to participate in dynamic invocation.
You must register the Activity with the [Worker](https://python.temporal.io/temporalio.worker.html) before it can be invoked.

The Activity Definition must then accept a single argument of type `Sequence[temporalio.common.RawValue]`.
Use a [payload_converter](https://python.temporal.io/temporalio.activity.html#payload_converter) function to convert `RawValue` objects to your required types.
For example:

<!--SNIPSTART python-message-passing-dynamic-activity {"highlightedLines": "14-19,27-35"}-->
[python-sandbox/workflows/message_passing/dynamic_activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/message_passing/dynamic_activity.py)
```py {14-19,27-35}
from dataclasses import dataclass
from datetime import timedelta
from collections.abc import Sequence

from temporalio import activity, workflow
from temporalio.common import RawValue

@dataclass
class YourDataClass:
    greeting: str
    name: str

@activity.defn(dynamic=True)
async def dynamic_greeting(args: Sequence[RawValue]) -> str:
    arg1 = activity.payload_converter().from_payload(args[0].payload, YourDataClass)
    return (
        f"{arg1.greeting}, {arg1.name}!\nActivity Type: {activity.info().activity_type}"
    )

@activity.defn()
async def default_greeting(input: YourDataClass) -> str:
    return f"{input.greeting}, {input.name}!\nActivity Type: {activity.info().activity_type}"

@workflow.defn
class GreetingWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            "unregistered_activity",
            YourDataClass("Hello", name),
            start_to_close_timeout=timedelta(seconds=10),
        )
```
<!--SNIPEND-->

This example invokes an unregistered Activity by name.
The Worker resolves it using the registered dynamic Activity instead.
When possible, prefer to use compiler-checked type-safe arguments rather than Activity name strings.

---

# Schedules - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/schedules

> Schedule, Create, Backfill, Delete, Describe, List, Pause, Trigger, and Update a Scheduled Workflow, along with Temporal Cron Jobs and Start Delay options.

This page shows how to do the following:

- [Schedule a Workflow](#schedule-a-workflow)
  - [Create a Scheduled Workflow](#create)
  - [Backfill a Scheduled Workflow](#backfill)
  - [Delete a Scheduled Workflow](#delete)
  - [Describe a Scheduled Workflow](#describe)
  - [List a Scheduled Workflow](#list)
  - [Pause a Scheduled Workflow](#pause)
  - [Trigger a Scheduled Workflow](#trigger)
  - [Update a Scheduled Workflow](#update)
- [Temporal Cron Jobs](#temporal-cron-jobs)
- [Start Delay](#start-delay)

## Schedule a Workflow 

Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes.

Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process.

Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy).
If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions.

### Create a Scheduled Workflow 

The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands.

To create a Scheduled Workflow Execution in Python, use the [create_schedule()](https://python.temporal.io/temporalio.client.Client.html#create_schedule)
asynchronous method on the Client.
Then pass the Schedule ID and the Schedule object to the method to create a Scheduled Workflow Execution.
Set the `action` parameter to `ScheduleActionStartWorkflow` to start a Workflow Execution.
Optionally, you can set the `spec` parameter to `ScheduleSpec` to specify the schedule or set the `intervals` parameter to `ScheduleIntervalSpec` to specify the interval.
Other options include: `cron_expressions`, `skip`, `start_at`, and `jitter`.

<!--SNIPSTART python-schedules-create {"highlightedLines": "18-32"}-->
[python-sandbox/workflows/schedules/create_schedule.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/create_schedule.py)
```py {18-32}
import asyncio
from datetime import timedelta

from temporalio.client import (
    Client,
    Schedule,
    ScheduleActionStartWorkflow,
    ScheduleIntervalSpec,
    ScheduleSpec,
    ScheduleState,
)
from schedules.your_workflows import YourSchedulesWorkflow

async def main():
    client = await Client.connect("localhost:7233")

    await client.create_schedule(
        "workflow-schedule-id",
        Schedule(
            action=ScheduleActionStartWorkflow(
                YourSchedulesWorkflow.run,
                "my schedule arg",
                id="schedules-workflow-id",
                task_queue="schedules-task-queue",
            ),
            spec=ScheduleSpec(
                intervals=[ScheduleIntervalSpec(every=timedelta(minutes=2))]
            ),
            state=ScheduleState(note="Here's a note on my Schedule."),
        ),
    )

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

> **💡 Tip:**
> Schedule Auto-Deletion
>
> Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again.
> The Temporal Service doesn't guarantee when this removal will happen.
>

### Backfill a Scheduled Workflow 

The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time.

To Backfill a Scheduled Workflow Execution in Python, use the [backfill()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#backfill) asynchronous
method on the Schedule Handle.

<!--SNIPSTART python-schedules-backfill {"highlightedLines": "12-18"}-->
[python-sandbox/workflows/schedules/backfill_schedule.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/backfill_schedule.py)
```py {12-18}
import asyncio
from datetime import datetime, timedelta, timezone

from temporalio.client import Client, ScheduleBackfill, ScheduleOverlapPolicy

async def main():
    client = await Client.connect("localhost:7233")
    handle = client.get_schedule_handle(
        "workflow-schedule-id",
    )
    now = datetime.now(timezone.utc)
    await handle.backfill(
        ScheduleBackfill(
            start_at=now - timedelta(minutes=10),
            end_at=now - timedelta(minutes=9),
            overlap=ScheduleOverlapPolicy.ALLOW_ALL,
        ),
    )

    print(f"Backfilled Schedule: {handle.id}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Delete a Scheduled Workflow 

The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule.

To delete a Scheduled Workflow Execution in Python, use the [delete()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#delete) asynchronous method on the Schedule Handle.

<!--SNIPSTART python-schedules-delete {"highlightedLines": "12"}-->
[python-sandbox/workflows/schedules/delete_schedule.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/delete_schedule.py)
```py {12}
import asyncio

from temporalio.client import Client

async def main():
    client = await Client.connect("localhost:7233")
    handle = client.get_schedule_handle(
        "workflow-schedule-id",
    )

    await handle.delete()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Describe a Scheduled Workflow 

The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs.

To describe a Scheduled Workflow Execution in Python, use the [describe()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#describe) asynchronous method on the Schedule Handle.
You can get a complete list of the attributes of the Scheduled Workflow Execution from the [ScheduleDescription](https://python.temporal.io/temporalio.client.ScheduleDescription.html) class.

<!--SNIPSTART python-schedules-describe {"highlightedLines": "12-14"}-->
[python-sandbox/workflows/schedules/describe_schedule.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/describe_schedule.py)
```py {12-14}
import asyncio

from temporalio.client import Client

async def main():
    client = await Client.connect("localhost:7233")
    handle = client.get_schedule_handle(
        "workflow-schedule-id",
    )

    desc = await handle.describe()

    print(f"Returns the note: {desc.schedule.state.note}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### List a Scheduled Workflow 

The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs.

To list all schedules, use the [list_schedules()](https://python.temporal.io/temporalio.client.Client.html#list_schedules) asynchronous method on the Client.
If a schedule is added or deleted, it may not be available in the list immediately.

<!--SNIPSTART python-schedules-list {"highlightedLines": "9"}-->
[python-sandbox/workflows/schedules/list_schedules.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/list_schedules.py)
```py {9}
import asyncio

from temporalio.client import Client

async def main() -> None:
    client = await Client.connect("localhost:7233")

    async for schedule in await client.list_schedules():
        print(f"List Schedule Info: {schedule.info}.")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Pause a Scheduled Workflow 

The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason.

To pause a Scheduled Workflow Execution in Python, use the [pause()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#pause) asynchronous method on the Schedule Handle.
You can pass a `note` to the `pause()` method to provide a reason for pausing the schedule.

<!--SNIPSTART python-schedules-pause {"highlightedLines": "12"}-->
[python-sandbox/workflows/schedules/pause_schedule.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/pause_schedule.py)
```py {12}
import asyncio

from temporalio.client import Client

async def main():
    client = await Client.connect("localhost:7233")
    handle = client.get_schedule_handle(
        "workflow-schedule-id",
    )

    await handle.pause(note="Pausing the schedule for now")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Trigger a Scheduled Workflow 

The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time.

To trigger a Scheduled Workflow Execution in Python, use the [trigger()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#trigger) asynchronous method on the Schedule Handle.

<!--SNIPSTART python-schedules-trigger {"highlightedLines": "12"}-->
[python-sandbox/workflows/schedules/trigger_schedule.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/trigger_schedule.py)
```py {12}
import asyncio

from temporalio.client import Client

async def main():
    client = await Client.connect("localhost:7233")
    handle = client.get_schedule_handle(
        "workflow-schedule-id",
    )

    await handle.trigger()

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

### Update a Scheduled Workflow 

The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval.

Create a function that takes `ScheduleUpdateInput` and returns `ScheduleUpdate`.
To update a Schedule, use a callback to build the update from the description.
The following example updates the Schedule to use a new argument.

<!--SNIPSTART python-schedules-update {"highlightedLines": "17-24"}-->
[python-sandbox/workflows/schedules/update_schedule.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/update_schedule.py)
```py {17-24}
import asyncio

from temporalio.client import (
    Client,
    ScheduleActionStartWorkflow,
    ScheduleUpdate,
    ScheduleUpdateInput,
)

async def main():
    client = await Client.connect("localhost:7233")
    handle = client.get_schedule_handle(
        "workflow-schedule-id",
    )

    async def update_schedule_simple(input: ScheduleUpdateInput) -> ScheduleUpdate:
        schedule_action = input.description.schedule.action

        if isinstance(schedule_action, ScheduleActionStartWorkflow):
            schedule_action.args = ["my new schedule arg"]
        return ScheduleUpdate(schedule=input.description.schedule)

    await handle.update(update_schedule_simple)

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Temporal Cron Jobs 

> **⚠️ Caution:**
> Cron support is not recommended
>
> We recommend using [Schedules](/schedule) instead of Cron Jobs.
> Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules.
>

A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution.

A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made.

You can set each Workflow to repeat on a schedule with the `cron_schedule` option from either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods.

<!--SNIPSTART python-schedules-cron {"highlightedLines": "11-17"}-->
[python-sandbox/workflows/schedules/cron_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/cron_workflow.py)
```py {11-17}
import asyncio

from temporalio.client import Client

from schedules.your_workflows import CronWorkflow

async def main():
    client = await Client.connect("localhost:7233")

    handle = await client.start_workflow(
        CronWorkflow.run,
        id="your-workflow-id",
        task_queue="your-task-queue",
        cron_schedule="* * * * *",
    )
    print(f"Started Cron Workflow: {handle.id}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

Temporal Workflow Schedule Cron strings follow this format:

```
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
* * * * *
```

## Start Delay 

Use the `start_delay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule.

Use the `start_delay` option in either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods in the Client.

<!--SNIPSTART python-schedules-start-delay-->
[python-sandbox/workflows/schedules/start_delay.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/schedules/start_delay.py)
```py
import asyncio
from datetime import timedelta

from temporalio.client import Client

from schedules.your_workflows import YourWorkflow

async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_workflow(
        YourWorkflow.run,
        "your name",
        id="your-workflow-id",
        task_queue="your-task-queue",
        start_delay=timedelta(hours=1, minutes=20, seconds=30)
    )

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

---

# Workflow Timeouts - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/timeouts

> Optimize Workflow Execution with Temporal Python SDK - Set Workflow Timeouts and Retry Policies efficiently.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient.
Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes.
If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer.

Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts).

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed.
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last.
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task.

Set the timeout to either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods.

Available timeouts are:

- `execution_timeout`
- `run_timeout`
- `task_timeout`

<!--SNIPSTART python-workflow-timeouts-starter {"highlightedLines": "11-20"}-->
[python-sandbox/workflows/workflow_timeouts/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_timeouts/starter.py)
```py {11-20}
import asyncio
from datetime import timedelta
from temporalio.client import Client
from workflow_timeouts.your_workflows import YourWorkflow
from temporalio.common import RetryPolicy

async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_workflow(
        YourWorkflow.run,
        "your timeout argument",
        id="your-workflow-id",
        task_queue="your-task-queue",
        # Set Workflow Timeout duration
        execution_timeout=timedelta(seconds=2),
        # run_timeout=timedelta(seconds=2),
        # task_timeout=timedelta(seconds=2),
    )
    handle = await client.start_workflow(
        YourWorkflow.run,
        "your retry policy argument",
        id="your-workflow-id",
        task_queue="your-task-queue",
        retry_policy=RetryPolicy(maximum_interval=timedelta(seconds=2)),
    )

    print(f"Result: {result}")
    print(f"Handle: {handle.id}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

## Workflow retries 

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

Set the Retry Policy to either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods.

<!--SNIPSTART python-workflow-timeouts-starter {"highlightedLines": "21-27"}-->
[python-sandbox/workflows/workflow_timeouts/starter.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_timeouts/starter.py)
```py {21-27}
import asyncio
from datetime import timedelta
from temporalio.client import Client
from workflow_timeouts.your_workflows import YourWorkflow
from temporalio.common import RetryPolicy

async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_workflow(
        YourWorkflow.run,
        "your timeout argument",
        id="your-workflow-id",
        task_queue="your-task-queue",
        # Set Workflow Timeout duration
        execution_timeout=timedelta(seconds=2),
        # run_timeout=timedelta(seconds=2),
        # task_timeout=timedelta(seconds=2),
    )
    handle = await client.start_workflow(
        YourWorkflow.run,
        "your retry policy argument",
        id="your-workflow-id",
        task_queue="your-task-queue",
        retry_policy=RetryPolicy(maximum_interval=timedelta(seconds=2)),
    )

    print(f"Result: {result}")
    print(f"Handle: {handle.id}")

if __name__ == "__main__":
    asyncio.run(main())
```
<!--SNIPEND-->

---

# Timers - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/timers

> Set a durable Timer in Python with asyncio.sleep(). Timers persist across Worker downtime, and a single Worker can run millions of them.

A Workflow can set a durable Timer for a fixed time period.
In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`.

A Workflow can sleep for months.
Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing.

Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker.

To set a Timer in Python, call the [`asyncio.sleep()`](https://docs.python.org/3/library/asyncio-task.html#sleeping) function and pass the duration in seconds you want to wait before continuing.

<!--SNIPSTART python-timers-looping-workflow {"highlightedLines": "11"}-->
[python-sandbox/workflows/timers/looping_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/timers/looping_workflow.py)
```py {11}
import asyncio

from temporalio import workflow

@workflow.defn
class LoopingWorkflow:
    @workflow.run
    async def run(self, iteration: int) -> None:
        if iteration == 5:
            return
        await asyncio.sleep(10)
        workflow.continue_as_new(iteration + 1)
```
<!--SNIPEND-->

---

# Versioning - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/versioning

> Ensure deterministic Temporal Workflow execution and safely deploy updates using the Python SDK's patching and Worker Versioning APIs, for scalable long-running Workflows.

Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress.

The Temporal Platform requires that Workflow Definitions are [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably.

With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version.
There are two primary Versioning methods that you can use:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths.
- [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.

> **🚨 Danger:**
> Support for the experimental Worker Versioning method before 2025 was scheduled for removal from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance. You can still refer to the [Worker Versioning Legacy](/develop/python/workflows/worker-versioning-legacy) docs if needed.

## Worker Versioning

Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching.

## Versioning with Patching 

### Adding a patch

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow Executions, create a patch.

Suppose you have an initial Workflow version called `pre_patch_activity`:

<!--SNIPSTART python-versioning-initial {"highlightedLines": "5,11-14"}-->
[python-sandbox/workflows/versioning/workflow_1_initial.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/versioning/workflow_1_initial.py)
```py {5,11-14}
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from versioning.activities import pre_patch_activity

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        self._result = await workflow.execute_activity(
            pre_patch_activity,
            schedule_to_close_timeout=timedelta(minutes=5),
        )

    @workflow.query
    def result(self) -> str:
        return self._result
```
<!--SNIPEND-->

Now, you want to update your code to run `post_patch_activity` instead. This represents your desired end state.

<!--SNIPSTART python-versioning-desired {"highlightedLines": "5,11-14"}-->
[python-sandbox/workflows/versioning/workflow_2_desired.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/versioning/workflow_2_desired.py)
```py {5,11-14}
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from versioning.activities import post_patch_activity

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        self._result = await workflow.execute_activity(
            post_patch_activity,
            schedule_to_close_timeout=timedelta(minutes=5),
        )

    @workflow.query
    def result(self) -> str:
        return self._result
```
<!--SNIPEND-->

The problem is that you cannot deploy `post_patch_activity` directly until you're certain there are no more running Workflows created using the `pre_patch_activity` code, otherwise you are likely to cause a nondeterminism error.
Instead, you'll need to deploy `post_patched_activity` and use the [patched](https://python.temporal.io/temporalio.workflow.html#patched) function to determine which version of the code to execute.

Patching is a three-step process:

1. Patch in any new, updated code using the `patched()` function. Run the new patched code alongside old code.
2. Remove old code and use `deprecate_patch()` to mark a particular patch as deprecated.
3. Once there are no longer any open Workflow Executions of the previous version of the code, remove `deprecate_patch()`.
   Let's walk through this process in sequence.

### Patching in new code 

Using `patched` inserts a marker into the Event History.
During Replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker (in this case, `my-patch`).
This ensures you can safely deploy code from `post_patch_activity` as a "feature flag" alongside the original version (`pre_patch_activity`).

<!--SNIPSTART python-versioning-patched {"highlightedLines": "12-21"}-->
[python-sandbox/workflows/versioning/workflow_3_patched.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/versioning/workflow_3_patched.py)
```py {12-21}
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from versioning.activities import post_patch_activity, pre_patch_activity

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        if workflow.patched("my-patch"):
            self._result = await workflow.execute_activity(
                post_patch_activity,
                schedule_to_close_timeout=timedelta(minutes=5),
            )
        else:
            self._result = await workflow.execute_activity(
                pre_patch_activity,
                schedule_to_close_timeout=timedelta(minutes=5),
            )

    @workflow.query
    def result(self) -> str:
        return self._result
```
<!--SNIPEND-->

### Deprecating patches 

After ensuring that all Workflows started with `pre_patch_activity` code have left retention, you can [deprecate the patch](https://python.temporal.io/temporalio.workflow.html#deprecate_patch).

Once your Workflows are no longer running the pre-patch code paths, you can deploy your code with `deprecate_patch()`.
These Workers will be running the most up-to-date version of the Workflow code, which no longer requires the patch.
Deprecated patches serve as a bridge between the final stage of the patching process and the final state that no longer has patches. They function similarly to regular patches by adding a marker to the Event History. However, this marker won't cause a replay failure when the Workflow code doesn't produce it.

<!--SNIPSTART python-versioning-deprecated {"highlightedLines": "12"}-->
[python-sandbox/workflows/versioning/workflow_4_deprecated.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/versioning/workflow_4_deprecated.py)
```py {12}
from datetime import timedelta
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from versioning.activities import post_patch_activity

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        workflow.deprecate_patch("my-patch")
        self._result = await workflow.execute_activity(
            post_patch_activity,
            schedule_to_close_timeout=timedelta(minutes=5),
        )

    @workflow.query
    def result(self) -> str:
        return self._result
```
<!--SNIPEND-->

### Removing a patch 

Once your pre-patch Workflows have left retention, you can then safely deploy Workers that no longer use either the `patched()` or `deprecate_patch()` calls:

<!--SNIPSTART python-versioning-desired {"selectedLines": ["7-14"]}-->
[python-sandbox/workflows/versioning/workflow_2_desired.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/versioning/workflow_2_desired.py)
```py
# ...
@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        self._result = await workflow.execute_activity(
            post_patch_activity,
            schedule_to_close_timeout=timedelta(minutes=5),
        )
```
<!--SNIPEND-->

Patching allows you to make changes to currently running Workflows.
It introduces compatible changes without causing non-determinism errors.

### Detailed description of the `patched()` function

This video provides an overview of how the `patched()` function works:

<br/>
For a more in-depth explanation, refer to the [Patching](/patching) Encyclopedia entry.

### Workflow cutovers

To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow.

Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes.
To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers.

For example, you would duplicate `PizzaWorkflow` as `PizzaWorkflowV2`:

<!--SNIPSTART python-versioning-cutover-workflows-->
[python-sandbox/workflows/versioning/pizza_workflows.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/versioning/pizza_workflows.py)
```py
@workflow.defn(name="PizzaWorkflow")
class PizzaWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        # this function contains the original code
        return f"Pizza for {name}"

@workflow.defn(name="PizzaWorkflowV2")
class PizzaWorkflowV2:
    @workflow.run
    async def run(self, name: str) -> str:
        # this function contains the updated code
        return f"Pizza with extra cheese for {name}"
```
<!--SNIPEND-->

You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types:

<!--SNIPSTART python-versioning-cutover-worker-->
[python-sandbox/workflows/versioning/pizza_worker.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/versioning/pizza_worker.py)
```py
worker = Worker(
    client,
    task_queue="your-task-queue",
    workflows=[PizzaWorkflow, PizzaWorkflowV2],
)
```
<!--SNIPEND-->

The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow.
This can become impractical over time.
This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching.

### Testing a Workflow for replay safety

To determine whether your Workflow needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/python/best-practices/testing-suite#replay).

---

# Worker Versioning (legacy) - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy

> Learn the Python SDK's outdated Worker Versioning APIs.

## (Deprecated) Use Worker Versioning in Python 

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to
> [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>
> See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more
> information.
>

A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp.
To use Worker Versioning, you need to pass a Build ID to your Python Worker and opt in to Worker Versioning.

### Assign a Build ID to your Worker and opt in to Worker Versioning

You should understand assignment rules before completing this step. See the
[Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for
more information.

To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on.

```python
# ...
worker = Worker(
  task_queue="your_task_queue_name",
  build_id=build_id,
  use_worker_versioning=True,
  # ... register workflows & activities, etc
)
# ...
```

> **⚠️ Warning:**
>
> Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules.
>

### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to
> [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the workflow that created
them if they are also configured to run on the same Task Queue. When configured to run on a separate Task Queue, they
will default to using the current assignment rules.

If you want to override this behavior, you can specify your intent via the `versioning_intent` argument available on the
methods you use to invoke these commands.

For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent:

```python
# ...
await workflow.execute_activity(
    say_hello,
    "hi",
    versioning_intent=VersioningIntent.USE_ASSIGNMENT_RULES,
    start_to_close_timeout=timedelta(seconds=5),
)
# ...
```

### Tell the Task Queue about your Worker's Build ID (Deprecated)

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to
> [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do
this as part of your CI deployment process.

```python
# ...
await client.update_worker_build_id_compatibility(
    "your_task_queue_name", BuildIdOpAddNewDefault("deadbeef")
)
```

This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the
default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process
by appropriately compatible Workers.

If, instead, you want to add the Build ID to an existing compatible set, you can do this:

```python
# ...
await client.update_worker_build_id_compatibility(
    "your_task_queue_name", BuildIdOpAddNewCompatible("deadbeef", "some-existing-build-id")
)
```

This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new
default Build ID for that set.

You can also promote an existing Build ID in a set to be the default for that set:

```python
# ...
await client.update_worker_build_id_compatibility(
    "your_task_queue_name", BuildIdOpPromoteBuildIdWithinSet("deadbeef")
)
```

You can also promote an entire set to become the default set for the queue. New Workflows will start using that set's
default build.

```python
# ...
await client.update_worker_build_id_compatibility(
    "your_task_queue_name", BuildIdOpPromoteSetByBuildId("deadbeef")
)
```

---

# Workflow Streams - Python SDK

Source: https://docs.temporal.io/develop/python/workflows/workflow-streams

> Stream events from a Workflow to subscribers using the Temporal Python SDK Workflow Streams contrib module.

> **Public Preview**

[Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its progress in real time. This page walks through enabling a stream, publishing events from Workflows and Activities, subscribing to a stream, and keeping a stream running across long-lived Workflows.

## Enable streaming on a Workflow

You can import Workflow Streams from `temporalio.contrib.workflow_streams`. Enable streaming by constructing a `WorkflowStream` from your Workflow's `@workflow.init` method because the stream's handlers have to be registered before the first publish Signal arrives. Doing it from `@workflow.run` raises a `RuntimeError` and would miss any publishes that arrived before the run body started executing.

<!--SNIPSTART python-workflow-streams-order-workflow {"selectedLines": ["1-5", "15-17", "27-31"]}-->
[python-sandbox/workflows/workflow_streams/order_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/order_workflow.py)
```py
from dataclasses import dataclass
from datetime import timedelta

from temporalio import workflow
from temporalio.contrib.workflow_streams import WorkflowStream
# ...
@dataclass
class OrderInput:
    order_id: str
# ...
@workflow.defn
class OrderWorkflow:
    @workflow.init
    def __init__(self, input: OrderInput) -> None:
        self.stream = WorkflowStream()
```
<!--SNIPEND-->

Constructing `WorkflowStream` creates the in-memory event log and dynamically registers the publish Signal, subscribe Update, and offset Query handlers on the current Workflow. Constructing more than one stream on the same Workflow also raises a `RuntimeError`.

If your Workflow is long-running, see [Stream from long-running Workflows](#continue-as-new) for how to carry stream state across rollovers so subscribers don't see gaps.

## Publish from a Workflow

You can [publish](/workflow-streams#publishing) when you bind a [topic name](/workflow-streams#topics) to its event type once via `self.stream.topic("name", type=Type)`, then call `publish()` on the returned handle to append events. The handle records the per-stream binding from topic name to value type so call sites don't have to repeat the type on every publish and so subscribers reading the same handle decode to the matching type.

<!--SNIPSTART python-workflow-streams-order-workflow {"selectedLines": ["20-47"]}-->
[python-sandbox/workflows/workflow_streams/order_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/order_workflow.py)
```py
# ...
@dataclass
class StatusEvent:
    state: str
    progress: int = 0
    detail: str = ""

@workflow.defn
class OrderWorkflow:
    @workflow.init
    def __init__(self, input: OrderInput) -> None:
        self.stream = WorkflowStream()
        self.status = self.stream.topic("status", type=StatusEvent)

    @workflow.run
    async def run(self, input: OrderInput) -> str:
        self.status.publish(StatusEvent(state="validating", detail="checking inventory"))
        await validate_order(input.order_id)

        self.status.publish(StatusEvent(state="charging", progress=33, detail="authorizing payment"))
        await charge_payment(input.order_id)

        self.status.publish(StatusEvent(state="shipping", progress=66, detail="dispatching to warehouse"))
        result = await dispatch_order(input.order_id)

        self.status.publish(StatusEvent(state="completed", progress=100))
        await workflow.sleep(timedelta(seconds=30))
        return result
```
<!--SNIPEND-->

`publish()` runs the payload converter to encode each value. The codec chain (encryption, compression, etc.) runs once on the Signal or Update envelope that carries the batch, never per item, so encryption and compression are applied exactly once each direction.

The `type=` argument is optional and defaults to `Any`. Pass it when you want the binding recorded so re-binding the same name to an unequal type raises and exception or so subscribers can pick up the type from the same handle.

## Publish from a client

Any process that has a Temporal Client and the target Workflow Id can publish to that Workflow's stream by constructing a `WorkflowStreamClient`. This is the general pattern and covers HTTP backends, starters, one-off scripts, other Workflows' Activities, and standalone Activities.

Construct one with:

```python
WorkflowStreamClient.create(client, workflow_id)
```

Then use it the same way you would the Workflow-side handle: bind a topic, publish through it, and let the async context manager flush on exit.

When events originate in an Activity, publish from the Activity directly rather than returning them for the Workflow to forward. The Workflow hosts the stream but doesn't read its own stream; it processes the Activity's return value and emits its own lifecycle events. Keeping Workflow state independent of streamed output is what lets retried Activity attempts surface to subscribers without polluting the Workflow's durable state. See [How events are delivered](/workflow-streams#how-events-are-delivered).

<!--SNIPSTART python-workflow-streams-publish-client-->
[python-sandbox/workflows/workflow_streams/publish_client.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/publish_client.py)
```py
from datetime import timedelta

from temporalio.client import Client
from temporalio.contrib.workflow_streams import WorkflowStreamClient

from workflow_streams.order_workflow import StatusEvent

async def publish_status(workflow_id: str) -> None:
    temporal_client = await Client.connect("localhost:7233")
    stream_client = WorkflowStreamClient.create(
        temporal_client,
        workflow_id=workflow_id,
        batch_interval=timedelta(milliseconds=200),
    )
    async with stream_client:
        status = stream_client.topic("status", type=StatusEvent)
        status.publish(StatusEvent(state="started"))
        ...
        # Buffer is flushed on context manager exit.
```
<!--SNIPEND-->

Inside an Activity scheduled by a Workflow, `WorkflowStreamClient.from_within_activity()` is used to infer the Temporal Client and the parent Workflow Id from the Activity context, so you don't have to thread them through the Activity's input:

<!--SNIPSTART python-workflow-streams-activity-publish-->
[python-sandbox/workflows/workflow_streams/delta_activities.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/delta_activities.py)
```py
from temporalio import activity
from temporalio.contrib.workflow_streams import WorkflowStreamClient

from workflow_streams.delta_types import Delta, generate_deltas

@activity.defn
async def stream_deltas(order_id: str) -> None:
    client = WorkflowStreamClient.from_within_activity()
    async with client:
        deltas = client.topic("delta", type=Delta)
        for delta in generate_deltas(order_id):
            deltas.publish(delta)
            activity.heartbeat()
        # Buffer is flushed on context manager exit.
```
<!--SNIPEND-->

For a [standalone Activity](/develop/python/activities/standalone-activities) (one started directly via `Client.start_activity` rather than from a Workflow), there is no parent Workflow context to infer, so `from_within_activity()` raises an exception. Fall back to the general pattern with `activity.client()` and the target Workflow Id threaded through the Activity's input.

Two operations give the application explicit control over when batches ship: `force_flush=True` on a publish for latency, and `await client.flush()` for confirmation that prior publications have landed.

Pass `force_flush=True` on a publish to wake the background flusher so the current buffer ships without waiting for the next interval. The flusher only runs while the Workflow Stream client is entered (`async with client`). Otherwise, `force_flush=True` queues the wake event, but nothing ships until you enter the context or call `await client.flush()`. The call returns immediately after appending to the buffer and signaling the flusher. It doesn't wait for delivery to the Workflow or to subscribers:

<!--SNIPSTART python-workflow-streams-force-flush-->
[python-sandbox/workflows/workflow_streams/delta_phases.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/delta_phases.py)
```py
deltas.publish(delta, force_flush=True)
```
<!--SNIPEND-->

Use it for latency-sensitive events like, the first delta of a response so the user sees something fast, or punctuated events like `RETRY` and `STATUS_CHANGE`. See [Tuning](/workflow-streams#tuning) for the trade-off against history pressure.

Use `await client.flush()` when you need a mid-stream barrier. Successful completion of the flush is proof that the Temporal server has received all prior publications, so subsequent work that depends on those events being durable can proceed. The client stays open for further publishing afterward. Exiting `async with client` already flushes on its way out, so the explicit call is only for barriers in the middle:

<!--SNIPSTART python-workflow-streams-flush-barrier-->
[python-sandbox/workflows/workflow_streams/delta_phases.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/delta_phases.py)
```py
async with client:
    deltas = client.topic("delta", type=Delta)
    for delta in first_phase():
        deltas.publish(delta)

    await client.flush()
    checkpoint_id = await record_phase_one_complete()  # only safe once phase-one events are durable in the Workflow log

    for delta in second_phase(checkpoint_id):
        deltas.publish(delta)
```
<!--SNIPEND-->

`publish()` is non-blocking and applies no backpressure. From an Activity or other client, it appends to the client's in-memory buffer and returns. From inside a Workflow, it appends synchronously to the in-memory log. [Subscribers](/workflow-streams#subscribing) pull from the Workflow's log on their own schedule, so a slow subscriber doesn't slow down [publishers](/workflow-streams#publishing). If a publisher emits faster than batches can ship to the server, the buffer grows: the process uses more memory, the stream falls further behind real time, and at the limit Signals can't keep up.

If your application needs to bound this (to cap memory, to keep the stream close to real time, or to apply a policy when the publisher overruns the network), apply that policy upstream of `publish()`. The choice (block, drop, error, sample) is application-specific, and Workflow Streams doesn't pick one for you.

## Subscribe

[Subscribing](/workflow-streams#subscribing) uses the same client construction as publishing: `WorkflowStreamClient.create(client, workflow_id)` from any process that has a Temporal Client or `from_within_activity()` inside an Activity. Subscribing from an Activity is less common in practice, so the general client case is the primary example below.

Once you have a client, iterate a topic handle's `subscribe()`, the counterpart to `publish()`. The handle's bound type drives decoding, so each `item.data` arrives as `T` via the client's payload converter. The codec chain is applied once at the Update envelope, not per item.

<!--SNIPSTART python-workflow-streams-subscribe-->
[python-sandbox/workflows/workflow_streams/subscriber.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/subscriber.py)
```py
async def watch_order(order_id: str) -> None:
    temporal_client = await Client.connect("localhost:7233")
    stream = WorkflowStreamClient.create(temporal_client, workflow_id=order_id)

    status = stream.topic("status", type=StatusEvent)
    async for item in status.subscribe():
        evt = item.data
        print(f"[{evt.progress:3d}%] {evt.state}: {evt.detail}")
        if evt.state == "completed":
            break
```
<!--SNIPEND-->

The iterator handles re-polling, pagination when a poll response hits the ~1 MB cap, and Workflow-side log truncation transparently. Callers don't need to wrap the iterator for the common cases.

Two edge cases are worth knowing:

- An RPC timeout where Continue-As-New cannot be followed ends the iterator silently (no exception raised)
- A validator rejection during a Continue-As-New handoff can surface as a `WorkflowUpdateFailedError`. See the [API reference](https://python.temporal.io/temporalio.contrib.workflow_streams.html) for details.

### Heterogeneous topics

A topic handle binds one name to one type, so it only fits a single-type subscription. To consume multiple topics whose payload types differ, call `client.subscribe()` directly with a list of names (or `subscribe([])` for every topic) and pass `result_type=temporalio.common.RawValue` so each item arrives as the underlying `Payload` wrapped in a `RawValue`. Dispatch on `item.topic` and decode the wrapped payload with the client's payload converter:

<!--SNIPSTART python-workflow-streams-heterogeneous-->
[python-sandbox/workflows/workflow_streams/subscriber.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/subscriber.py)
```py
from temporalio.common import RawValue

converter = temporal_client.data_converter.payload_converter

async for item in stream.subscribe(["status", "progress"], result_type=RawValue):
    if item.topic == "status":
        evt = converter.from_payload(item.data.payload, StatusEvent)
        print(f"[status] {evt.state}: {evt.detail}")
    elif item.topic == "progress":
        evt = converter.from_payload(item.data.payload, ProgressEvent)
        print(f"[progress] {evt.message}")
```
<!--SNIPEND-->

A single iterator over multiple topics also avoids the cancellation race that two concurrent subscribers would create. `RawValue` is also the right shape when you want to forward the bytes through to another system without decoding them.

Omitting `result_type` entirely or passing `result_type=None` decodes each item with the converter's default rules. For the stock JSON converter, that means a Python primitive, `dict`, or `list`. This works for fully homogeneous streams, but not for the dispatch-by-topic pattern above, where each topic has its own concrete dataclass.

### Closing the stream 

A subscriber's `async for` does not know when the publisher is done. How you [close a stream](/workflow-streams#closing-the-stream) depends on what the application needs. As one example, a common pattern combines two pieces:

1. **An in-band terminator.** The Workflow or its Activity publishes a sentinel event the subscriber recognizes and breaks on. In the `watch_order` example above, `StatusEvent(state="completed")` is the minimal form, and the consumer's `if evt.state == "completed": break` is the matching half. Each subscription decides what its own end-of-stream marker is.
2. **A brief overlap before the Workflow returns.** A poll Update that is still in flight when the Workflow returns surfaces to the client as `AcceptedUpdateCompletedWorkflow`, and no new polls can complete after that. If the Workflow returns immediately after publishing the terminator, subscribers may miss it.

There are two ways to provide that overlap.

- [Fixed sleep](/workflow-streams#fixed-sleep). Sleep between the terminator and the return so any in-flight poll has time to fetch the terminator before the Workflow exits:

    <!--SNIPSTART python-workflow-streams-order-workflow {"selectedLines": ["34-35", "45-47"]}-->
    [python-sandbox/workflows/workflow_streams/order_workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/order_workflow.py)
    ```py
    # ...
        @workflow.run
        async def run(self, input: OrderInput) -> str:
    # ...
            self.status.publish(StatusEvent(state="completed", progress=100))
            await workflow.sleep(timedelta(seconds=30))
            return result
    ```
    <!--SNIPEND-->
- [Acknowledgment handshake](/workflow-streams#acknowledgment-handshake). The subscriber sends a Signal once it has the terminator; the Workflow waits up to a timeout, returning as soon as the ack arrives:

    <!--SNIPSTART python-workflow-streams-chat-workflow {"selectedLines": ["23-28", "37-44"]}-->
    [python-sandbox/workflows/workflow_streams/chat/workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/chat/workflow.py)
    ```py
    # ...
        @workflow.signal
        async def subscriber_acknowledged_terminator(self) -> None:
            self.subscriber_done = True

        @workflow.run
        async def run(self, input: ChatInput) -> str:
    # ...
            try:
                await workflow.wait_condition(
                    lambda: self.subscriber_done,
                    timeout=timedelta(seconds=30),
                )
            except TimeoutError:
                pass  # No subscriber; the run still completes cleanly.
            return result
    ```
    <!--SNIPEND-->

The full pattern is wired into the [Stream LLM output](#stream-llm-output) example below.

You can [inspect the terminal status](/workflow-streams#inspecting-terminal-status). `subscribe()` exits cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELED`, `TERMINATED`, or `TIMED_OUT`, but does not distinguish among them. If your application needs to know which (to display success or failure to the user, log the outcome, or decide whether to retry), call `await temporal_client.get_workflow_handle(workflow_id).describe()` after the loop returns to inspect the Workflow's status.

## Stream from long-running Workflows 

Workflows that run for hours or accumulate thousands of events need to periodically roll over via [Continue-As-New](/workflow-streams#stream-from-long-running-workflows) to keep history bounded. Subscribers automatically follow these rollovers, but the client retained from `WorkflowStreamClient.create()` or `from_within_activity()` is required (clients constructed directly with a single handle cannot re-target the new run). To keep a stream running across rollovers without subscribers seeing a gap, carry both your application state and the stream state across the boundary. Add a `WorkflowStreamState | None` field to your Workflow input, pass it to the constructor, and call `WorkflowStream.continue_as_new(build_args)` to invoke the rollover. The helper drains waiting subscribers, waits for in-flight handlers to finish, then calls `workflow.continue_as_new` with the args produced by `build_args(post_drain_state)`:

<!--SNIPSTART python-workflow-streams-continue-as-new-->
[python-sandbox/workflows/workflow_streams/long_running.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/long_running.py)
```py
from dataclasses import dataclass, field

from temporalio import workflow
from temporalio.contrib.workflow_streams import WorkflowStream, WorkflowStreamState

with workflow.unsafe.imports_passed_through():
    from workflow_streams.long_running_work import do_one_iteration

@dataclass
class AppState:
    items_processed: int = 0

@dataclass
class WorkflowInput:
    app_state: AppState = field(default_factory=AppState)
    stream_state: WorkflowStreamState | None = None

@workflow.defn
class LongRunningWorkflow:
    @workflow.init
    def __init__(self, input: WorkflowInput) -> None:
        self.app_state = input.app_state
        self.stream = WorkflowStream(prior_state=input.stream_state)

    @workflow.run
    async def run(self, input: WorkflowInput) -> None:
        while True:
            await do_one_iteration(self)
            if workflow.info().is_continue_as_new_suggested():
                await self.stream.continue_as_new(
                    lambda stream_state: [
                        WorkflowInput(
                            app_state=self.app_state,
                            stream_state=stream_state,
                        )
                    ]
                )
```
<!--SNIPEND-->

The `| None` on the `stream_state` field type is required: `prior_state` is `None` on a fresh start and a `WorkflowStreamState` instance after a rollover.

Always use the concrete type, not `Any`. With `Any`, the data converter rebuilds the field as a plain `dict` and `WorkflowStream(prior_state=...)` raises an `AttributeError` accessing `.log` / `.base_offset` / `.publishers` on the dict.

To pass other Continue-As-New parameters such as `task_queue`, `retry_policy`, `run_timeout`, use the explicit recipe instead:

<!--SNIPSTART python-workflow-streams-explicit-continue-as-new-->
[python-sandbox/workflows/workflow_streams/long_running.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/long_running.py)
```py
self.stream.detach_pollers()
await workflow.wait_condition(workflow.all_handlers_finished)
workflow.continue_as_new(
    args=[WorkflowInput(app_state=self.app_state, stream_state=self.stream.get_state())],
    task_queue="other-tq",
)
```
<!--SNIPEND-->

The carried `WorkflowStreamState` includes the entire in-memory log of the previous run, so streams that carry large items can hit Temporal's per-payload size limit at the rollover. Offload the bytes via [External Storage](/external-storage) so each item is a small reference rather than the full payload, and combine that with `truncate()` to keep the carried log itself small.

## Deduplication window

See [How events are delivered](/workflow-streams#how-events-are-delivered) for more details on subscriber and publisher behavior. See [Tuning](/workflow-streams#tuning) for more details on how to change your settings to meet the requirements for your Workflow Streams.

There are two limits on the [deduplication window](/workflow-streams#deduplication-window) worth highlighting:

- `publisher_ttl`: At each Continue-As-New, deduplicate entries whose `last_seen` is older than this are dropped. `last_seen` is updated on each *successful* publish, so a publisher that retries through a long partition without success can still time out. Tune upward if your publishers can be silent for extended windows:

    ```python
    WorkflowStream.continue_as_new(publisher_ttl=...)
    ```
- `max_retry_duration`: A `WorkflowStreamClient` retries a failed batch for up to this long. If the duration elapses with the batch still pending, the client gives up, the pending batch is dropped, and a `TimeoutError` is raised.

    On timeout, the dropped batch is at-most-once: it may or may not have reached the Workflow. One operational caveat: the `TimeoutError` raises from inside the background flusher task and terminates it. Until you call `await client.flush()` or exit the `async with` block, subsequent publishes accumulate in the buffer with no flusher to ship them.

## Best practices

There are a few details to note if you're writing custom message handlers or testing the library's capabilities:

- **`WorkflowStreamClient` is asyncio-only.** The client buffer is mutated on the publish path and read from the flusher inside a single event loop. Don't call `publish()` from a Worker thread.
- **Custom handlers read stream state on the first activation.** `WorkflowStream` registers its publish-Signal handler dynamically from `__init__`, so on the first activation a publish Signal can be queued before class-level `@workflow.signal` or `@workflow.update` handlers have run.

    A handler that observes state set by stream initialization in that same activation can see pre-publish state. The fix is to make the handler `async def` and `await` once before reading state. `asyncio.sleep(0)` is a no-op yield that suffices and adds no history events. Don't substitute `workflow.sleep(0)`, which records a timer event. Once the first activation completes, the handler is permanent and the race doesn't recur.
- **Type bindings aren't shared across publishers.** Each `WorkflowStream` and each `WorkflowStreamClient` records topic types only for its own instance. If two publishers bind the same topic name to different types, the mismatch is not caught at publish, and the subscriber gets a decode error when it processes events from the mismatched publisher.

## Example: Stream LLM output 

The headline use case fits the publish/subscribe shapes documented above. An Activity calls the model and publishes deltas as they arrive. The Workflow starts the Activity and waits for the consumer to acknowledge end-of-stream. The consumer subscribes, accumulates the deltas, and clears its accumulated state on `RETRY` before continuing. The shape works for a terminal client, a desktop UI, or a Server-Sent Events (SSE) endpoint forwarding to a browser. Anything that holds the displayed state calls `render()` to display it.

If your Activity can retry, the consumer side has to account for it. A retried attempt is a fresh publisher, so its output appears in the stream alongside the output from the previous attempt. In the LLM streaming pattern below, that means the failed attempt's partial deltas and the retried attempt's full output both reach a subscribed UI unless the UI resets on a `RETRY` event. The example wires up that pattern. See [How events are delivered](/workflow-streams#how-events-are-delivered) for the precise guarantees.

**activity.py**

<!--SNIPSTART python-workflow-streams-chat-activity-->
[python-sandbox/workflows/workflow_streams/chat/activity.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/chat/activity.py)
```py
from dataclasses import dataclass
from datetime import timedelta

from openai import AsyncOpenAI
from temporalio import activity
from temporalio.contrib.workflow_streams import WorkflowStreamClient

@dataclass
class TextDelta:
    text: str

@activity.defn
async def stream_completion(prompt: str) -> str:
    stream_client = WorkflowStreamClient.from_within_activity(
        batch_interval=timedelta(milliseconds=200),
    )
    # Disable provider-side retries; let Temporal own retry policy at the Activity layer.
    openai_client = AsyncOpenAI(max_retries=0)

    async with stream_client:
        deltas = stream_client.topic("delta", type=TextDelta)
        retry = stream_client.topic("retry", type=dict)
        close = stream_client.topic("close")

        # Tell consumers an earlier attempt's deltas are stale.
        if activity.info().attempt > 1:
            retry.publish({"attempt": activity.info().attempt}, force_flush=True)

        full: list[str] = []
        first = True
        oai_stream = await openai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            stream=True,
        )
        async for chunk in oai_stream:
            if not chunk.choices:
                continue
            text = chunk.choices[0].delta.content
            if not text:
                continue
            # force_flush only on the first delta so the user sees something
            # immediately; subsequent deltas batch at the 200 ms interval.
            deltas.publish(TextDelta(text=text), force_flush=first)
            first = False
            full.append(text)
        close.publish({})
    return "".join(full)
```
<!--SNIPEND-->

**workflow.py**

<!--SNIPSTART python-workflow-streams-chat-workflow-->
[python-sandbox/workflows/workflow_streams/chat/workflow.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/chat/workflow.py)
```py
from dataclasses import dataclass
from datetime import timedelta

from temporalio import workflow
from temporalio.contrib.workflow_streams import WorkflowStream

with workflow.unsafe.imports_passed_through():
    from workflow_streams.chat.activity import stream_completion

@dataclass
class ChatInput:
    prompt: str

@workflow.defn
class ChatWorkflow:
    @workflow.init
    def __init__(self, input: ChatInput) -> None:
        self.stream = WorkflowStream()
        self.subscriber_done: bool = False

    @workflow.signal
    async def subscriber_acknowledged_terminator(self) -> None:
        self.subscriber_done = True

    @workflow.run
    async def run(self, input: ChatInput) -> str:
        result = await workflow.execute_activity(
            stream_completion,
            input.prompt,
            start_to_close_timeout=timedelta(minutes=5),
        )
        # Wait for the subscriber to ack the terminal `close` event.
        # The timeout is a fallback for when no subscriber is attached;
        # with the ack, the typical case exits as soon as the subscriber confirms.
        try:
            await workflow.wait_condition(
                lambda: self.subscriber_done,
                timeout=timedelta(seconds=30),
            )
        except TimeoutError:
            pass  # No subscriber; the run still completes cleanly.
        return result
```
<!--SNIPEND-->

**consumer.py**

<!--SNIPSTART python-workflow-streams-chat-consumer-->
[python-sandbox/workflows/workflow_streams/chat/consumer.py](https://github.com/temporalio/documentation-sdk-code-examples/blob/main/python-sandbox/workflows/workflow_streams/chat/consumer.py)
```py
from temporalio.client import Client
from temporalio.common import RawValue
from temporalio.contrib.workflow_streams import WorkflowStreamClient

from workflow_streams.chat.activity import TextDelta
from workflow_streams.chat.workflow import ChatWorkflow

async def stream_chat(temporal_client: Client, chat_id: str) -> str:
    # Subscribe-only; no `async with` needed because the flusher only runs for publishers.
    stream = WorkflowStreamClient.create(temporal_client, workflow_id=chat_id)
    converter = temporal_client.data_converter.payload_converter
    output: list[str] = []

    def render() -> None:
        ...  # display the accumulated output (terminal redraw, UI update, etc.)

    async for item in stream.subscribe(
        ["delta", "retry", "close"], result_type=RawValue
    ):
        if item.topic == "retry":
            # Earlier attempt's deltas are stale; drop what we've shown.
            output.clear()
            render()
        elif item.topic == "delta":
            delta = converter.from_payload(item.data.payload, TextDelta)
            output.append(delta.text)
            render()
        elif item.topic == "close":
            # Acknowledge so the Workflow can return without a sleep.
            await temporal_client.get_workflow_handle(chat_id).signal(
                ChatWorkflow.subscriber_acknowledged_terminator
            )
            break

    return "".join(output)
```
<!--SNIPEND-->

A few choices in this shape are deliberate:

- The Activity is the publisher because it owns the non-deterministic LLM call. The Workflow processes the Activity's return value, never reading its own stream. See [Publish from a client](#publish-from-a-client) for why.
- The Activity publishes a `RETRY` event when `activity.info().attempt > 1`. This lets the UI respond appropriately to the failure, typically by clearing accumulated deltas before the next attempt's deltas arrive (see [How events are delivered](/workflow-streams#how-events-are-delivered)).
- Termination uses an *ack handshake*: the consumer signals the Workflow once it has received the `close` event, so the Workflow can return as soon as the subscriber confirms. The `wait_condition` timeout is the fallback when no subscriber is attached (see [Closing the stream](#closing-the-stream) for the simpler fixed-sleep alternative).
- `force_flush=True` is used on the first delta and on the `RETRY` sentinel, where latency matters. Subsequent deltas batch at the 200 ms `batch_interval`; per-delta `force_flush=True` would generate one Signal per token (see [Tuning](/workflow-streams#tuning) for the trade-off).

## See also

- [Workflow Streams samples (samples-python)](https://github.com/temporalio/samples-python/tree/main/workflow_streams): four runnable scenarios covering basic publish/subscribe, reconnecting subscribers, external publishers, and bounded logs.
- [`temporalio.contrib.workflow_streams` API reference](https://python.temporal.io/temporalio.contrib.workflow_streams.html).
- [Workflow message passing](/develop/python/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on.
- [Payload conversion](/develop/python/data-handling/data-conversion): converters and codecs.

---

# Ruby SDK developer guide

Source: https://docs.temporal.io/develop/ruby

<!-- Ruby SDK feature guidance landing page-->

![Ruby SDK Banner](https://docs.temporal.io/img/assets/banner-ruby-temporal.png)

## Install and get started

You can find detailed installation instructions for the Ruby SDK in the [Quickstart](/develop/ruby/set-up-local-ruby).

There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running.

Once your local Temporal Service is set up, continue building with the following resources:

- [Workflow basics](/develop/ruby/workflows/basics)
- [Activity basics](/develop/ruby/activities/basics)
- [Start an Activity Execution](/develop/ruby/activities/execution)
- [Run a Worker](/develop/ruby/workers/run-worker-process)

From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases.

## [Workflows](/develop/ruby/workflows)

- [Workflow basics](/develop/ruby/workflows/basics)
- [Child Workflows](/develop/ruby/workflows/child-workflows)
- [Continue-As-New](/develop/ruby/workflows/continue-as-new)
- [Cancellation](/develop/ruby/workflows/cancellation)
- [Timeouts](/develop/ruby/workflows/timeouts)
- [Message passing](/develop/ruby/workflows/message-passing)
- [Schedules](/develop/ruby/workflows/schedules)
- [Timers](/develop/ruby/workflows/timers)
- [Futures](/develop/ruby/workflows/futures)
- [Dynamic Workflow](/develop/ruby/workflows/dynamic-workflow)
- [Versioning](/develop/ruby/workflows/versioning)

## [Activities](/develop/ruby/activities)

- [Activity basics](/develop/ruby/activities/basics)
- [Activity Execution](/develop/ruby/activities/execution)
- [Standalone Activities](/develop/ruby/activities/standalone-activities-quickstart)
- [Timeouts](/develop/ruby/activities/timeouts)
- [Asynchronous Activity Completion](/develop/ruby/activities/asynchronous-activity)
- [Dynamic Activity](/develop/ruby/activities/dynamic-activity)
- [Benign exceptions](/develop/ruby/activities/benign-exceptions)

## [Workers](/develop/ruby/workers)

- [Run a Worker](/develop/ruby/workers/run-worker-process)
- [Serverless Workers](/develop/ruby/workers/serverless-workers)
- [Observability](/develop/ruby/platform/observability)

## [Temporal Client](/develop/ruby/client)

- [Temporal Client](/develop/ruby/client/temporal-client)

## [Platform](/develop/ruby/platform)

- [Observability](/develop/ruby/platform/observability)
- [Enriching the UI](/develop/ruby/platform/enriching-ui)

## [Integrations](/develop/ruby/integrations)

- [Rails integration](/develop/ruby/integrations/rails-integration)

## [Best practices](/develop/ruby/best-practices)

- [Error handling](/develop/ruby/best-practices/error-handling)
- [Testing](/develop/ruby/best-practices/testing-suite)
- [Debugging](/develop/ruby/best-practices/debugging)
- [Converters and encryption](/develop/ruby/best-practices/data-handling)

## Temporal Ruby technical resources

- [Ruby SDK Quickstart - Setup Guide](/develop/ruby/set-up-local-ruby)
- [Ruby SDK Code Samples](https://github.com/temporalio/samples-ruby)
- [Ruby API Documentation](https://ruby.temporal.io/)
- [Ruby SDK GitHub](https://github.com/temporalio/sdk-ruby)
- [Temporal 101 in Ruby Free Course](https://learn.temporal.io/courses/temporal_101/ruby/)

## Get connected with the Temporal Ruby community

- [Temporal Ruby Community Slack](https://temporalio.slack.com/archives/C052K5QFBNW)
- [Ruby SDK Forum](https://community.temporal.io/tag/ruby-sdk)

---

# Activities - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities

> Activity Definitions, Activity Execution, Timeouts, Standalone Activities, Dynamic Activities, and benign exceptions in the Temporal Ruby SDK.

![Ruby SDK Banner](https://docs.temporal.io/img/assets/banner-ruby-temporal.png)

## Activities

- [Activity basics](/develop/ruby/activities/basics)
- [Activity Execution](/develop/ruby/activities/execution)
- [Standalone Activities quickstart](/develop/ruby/activities/standalone-activities-quickstart)
- [Standalone Activities feature guide](/develop/ruby/activities/standalone-activities)
- [Timeouts](/develop/ruby/activities/timeouts)
- [Asynchronous Activity Completion](/develop/ruby/activities/asynchronous-activity)
- [Dynamic Activity](/develop/ruby/activities/dynamic-activity)
- [Benign exceptions](/develop/ruby/activities/benign-exceptions)

---

# Asynchronous Activity Completion - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/asynchronous-activity

> Asynchronously complete an Activity in Temporal using the Ruby SDK. Follow simple steps to allow an Activity Function to return without the Activity Execution completing.

## Asynchronously complete an Activity 

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The Temporal Client is used to Heartbeat and complete the Activity.

To mark an Activity as completing asynchronously, do the following inside the Activity.

```ruby
# Capture token for later completion
captured_token = Temporalio::Activity::Context.current.info.task_token

# Raise a special exception that says an activity will be completed somewhere else
raise Temporalio::Activity::CompleteAsyncError
```

To update an Activity outside the Activity, use the [async_activity_handle](https://ruby.temporal.io/Temporalio/Client.html#async_activity_handle-instance_method) method on the client to get the handle of the Activity.

```ruby
handle = my_client.async_activity_handle(captured_token)
```

Then, on that handle, you can use `heartbeat`, `complete`, `fail`, or `report_cancellation` methods to update the Activity.

```ruby
handle.complete('completion value')
```

---

# Activity basics - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/basics

> Define an Activity as a class that extends Temporalio::Activity::Definition, name it, and pick the executor that controls Activity concurrency.

## Develop an Activity 

One of the primary things that Workflows do is orchestrate the execution of Activities.
An Activity is a normal method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file.
An Activity can interact with the world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service.
For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition).

Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/ruby/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/ruby/activities/standalone-activities-quickstart).

You can develop an Activity Definition by creating a class that extends `Temporalio::Activity::Definition`.
To register a class as an Activity with a custom name, use the `activity_name` class method in the class definition.
Otherwise, the activity name is the unqualified class name.

```ruby
class MyActivity < Temporalio::Activity::Definition
  def execute(input)
    "#{input['greeting']}, #{input['name']}!"
  end
end
```

Activity implementation code should be _idempotent_. Learn more about [idempotency](/activity-definition#idempotency).

There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support.
However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB.
And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB.

Some SDKs require that you pass context objects, others do not.
When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single hash or object as an argument that wraps the application data passed to Activities.
This is so that you can change what data is passed to the Activity without breaking a method signature.

The `execute` method in your Activity can technically accept multiple parameters of any data type that Temporal can convert.
However, Temporal strongly encourages using a single parameter object to simplify versioning and maintainability.

### Activity concurrency and executors 

> **📝 Note:**
>
> This section covers advanced concurrency and execution options that most users will not need when getting started.
>

By default, activities run in the "thread pool executor" (that is, `Temporalio::Worker::ActivityExecutor::ThreadPool`).
This default is shared across all workers and is a naive thread pool that continually makes threads as needed when none are
idle/available to handle incoming work.
If a thread sits idle long enough, it will be killed.

The maximum number of concurrent activities a worker will run at a time is configured via its `tuner` option.
The default is `Temporalio::Worker::Tuner.create_fixed` which defaults to 100 activities at a time for that worker.
When this value is reached, the worker will stop asking for work from the server until there are slots available again.

In addition to the thread pool executor, there is also a fiber executor in the default executor set.
To use fibers, call `activity_executor :fiber` class method at the top of the activity class (the default of this value is `:default` which is the thread pool executor).
Activities can only choose the fiber executor if the worker has been created and run in a fiber, but thread pool executor is always available.
Currently due to [an issue](https://github.com/temporalio/sdk-ruby/issues/162), workers can only run in a fiber on Ruby versions 3.3 and newer.

---

# Benign exceptions - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/benign-exceptions

> Mark expected or non-severe Activity errors as benign in Ruby to reduce noise in logs, metrics, and OpenTelemetry traces.

When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues.
By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic.

To mark an error as benign, set the `category` parameter to `Temporalio::Error::ApplicationError::Category::BENIGN` when raising an `ApplicationError`.

Benign errors:
- Have Activity failure logs downgraded to DEBUG level
- Do not emit Activity failure metrics
- Do not set the OpenTelemetry failure status to ERROR

```ruby
require 'temporalio/activity'

class MyActivity < Temporalio::Activity::Definition
  def execute
    begin
      call_external_service
    rescue StandardError => e
      # Mark this error as benign since it's expected
      raise Temporalio::Error::ApplicationError.new(
        e.message,
        category: Temporalio::Error::ApplicationError::Category::BENIGN
      )
    end
  end
end
```

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried.

---

# Dynamic Activity - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/dynamic-activity

> Register one Dynamic Activity per Worker with activity_dynamic to handle Activity Types that match no other registered Activity.

## Set a Dynamic Activity 

A Dynamic Activity in Temporal is an Activity that is invoked dynamically at runtime if no other Activity with the same name is registered.
An Activity can be made dynamic by invoking `activity_dynamic` class method at the top of the definition.
You must register the Activity with the Worker before it can be invoked.
Only one Dynamic Activity can be present on a Worker.

Often, dynamic is used in conjunction with `activity_raw_args` which does not convert arguments but instead passes them
through as a splatted array of `Temporalio::Converters::RawValue` instances.

```ruby
class MyDynamicActivity < Temporalio::Activity::Definition
  # Make this the dynamic activity and accept raw args
  activity_dynamic
  activity_raw_args

  def execute(*raw_args)
    raise Temporalio::Error::ApplicationError, 'One arg expected' unless raw_args.size == 1

    # Use payload converter to convert it
    input = Temporalio::Activity::Context.current.payload_converter.from_payload(raw_args.first.payload)
    "#{input['greeting']}, #{input['name']}!"
  end
end
```

---

# Activity Execution - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/execution

> Call an Activity from a Ruby Workflow with Temporalio::Workflow.execute_activity and set the required Schedule-To-Close or Start-To-Close Timeout.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition).
The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command.
This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and `ActivityTask[Closed]`) in your Workflow Execution Event History.

The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history.
The entire Execution history is transferred from the Temporal Service to Workflow Workers when a Workflow state needs to recover.
A large Execution history can thus adversely impact the performance of your Workflow.

Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values.
Otherwise, no additional limitations exist on Activity implementations.

To spawn an Activity Execution, use the `execute_activity` operation from within your Workflow Definition.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  # Customize the name
  workflow_name :MyDifferentWorkflowName

  def execute(name)
    Temporalio::Workflow.execute_activity(
      MyActivity,
      { greeting: 'Hello', name: },
      start_to_close_timeout: 100
    )
  end
end
```

Activity Execution semantics rely on several parameters.
The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).
These values are set as keyword parameters.

The Activity result is returned from the `execute_activity` call.

---

# Standalone Activities feature guide - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/standalone-activities

> Execute Activities independently without a Workflow using the Temporal Ruby SDK.

[Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated
by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a [`Temporalio::Client`](https://ruby.temporal.io/Temporalio/Client.html).

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/ruby/activities/basics). The only difference is that you execute a Standalone
Activity directly from your Temporal Client.

> **💡 Tip:**
>
> New to Standalone Activities? Start with the [Standalone Activities quickstart](/develop/ruby/activities/standalone-activities-quickstart).
>

This page covers the following:

- [Prerequisites](#prerequisites)
- [Start a Standalone Activity without waiting for the result](#start-activity)
- [Get a handle to an existing Standalone Activity](#get-activity-handle)
- [Wait for the result of a Standalone Activity](#get-activity-result)
- [List Standalone Activities](#list-activities)
- [Count Standalone Activities](#count-activities)
- [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from the
> [standalone_activity](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity)
> sample.
>

## Prerequisites 

Standalone Activities require:

- **Ruby** 3.3+
- **Temporal Ruby SDK** v1.9.0 or higher
- **[Temporal CLI](/cli/setup-cli)** v1.9.1 or higher

The [Standalone Activities quickstart](/develop/ruby/activities/standalone-activities-quickstart)
walks through installing these.

## Start a Standalone Activity without waiting for the result 

Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue
your Activity job, without waiting for it to be executed by your Worker.

Use [`Temporalio::Client#start_activity`](https://ruby.temporal.io/Temporalio/Client.html#start_activity-instance_method) to start a
Standalone Activity and get a handle without waiting for the result:

[start_activity.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/start_activity.rb)

```ruby
handle = client.start_activity(
  StandaloneActivity::MyActivities::ComposeGreeting,
  'Hello', 'World',
  id: 'standalone-activity-id',
  task_queue: 'standalone-activity-sample',
  start_to_close_timeout: 10
)
puts "Started Activity with id=#{handle.id} run_id=#{handle.run_id}"

# Wait for the result later
puts "Activity result: #{handle.result}"
```

With the Temporal Server and Worker running, open a new terminal in the `samples-ruby` directory and
run:

```bash
bundle exec ruby standalone_activity/start_activity.rb
```

Or use the Temporal CLI:

```bash
temporal activity start \
  --type ComposeGreeting \
  --activity-id standalone-activity-id \
  --task-queue standalone-activity-sample \
  --start-to-close-timeout 10s \
  --input '"Hello"' \
  --input '"World"'
```

## Get a handle to an existing Standalone Activity 

Use [`Temporalio::Client#activity_handle`](https://ruby.temporal.io/Temporalio/Client.html#activity_handle-instance_method) to create an [`ActivityHandle`](https://ruby.temporal.io/Temporalio/Client/ActivityHandle.html) for a previously started Standalone Activity:

```ruby
handle = client.activity_handle('standalone-activity-id')
```

Pass no run ID (the default) to target the latest run of the given Activity ID, or pass
`activity_run_id:` to target a specific run. You can then use the handle to wait for the result,
describe, cancel, or terminate the Activity:

```ruby
handle.result      # block until the activity completes; returns the result
handle.describe    # fetch metadata (status, timestamps, attempt, last failure, etc.)
handle.cancel      # request cancellation
handle.terminate   # force-close the activity
```

## Wait for the result of a Standalone Activity 

Under the hood, calling `client.execute_activity` is the same as calling `client.start_activity` to
durably enqueue the Standalone Activity, and then calling `handle.result` to block until the
Activity completes and return the result:

```ruby
result = handle.result
```

Or use the Temporal CLI to wait for a result by Activity ID:

```bash
temporal activity result --activity-id standalone-activity-id
```

## List Standalone Activities 

Use [`Temporalio::Client#list_activities`](https://ruby.temporal.io/Temporalio/Client.html#list_activities-instance_method) to list
Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is an
`Enumerator` of [`ActivityExecution`](https://ruby.temporal.io/Temporalio/Client/ActivityExecution.html) values that fetches pages from the server on demand as the
enumerator is consumed.

These APIs return only Standalone Activity Executions. Activities running inside Workflows are not
included.

[list_activities.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/list_activities.rb)

```ruby
client.list_activities("TaskQueue = 'standalone-activity-sample'").each do |execution|
  puts "#{execution.activity_id} #{execution.activity_type} #{execution.status}"
end
```

Run it:

```bash
bundle exec ruby standalone_activity/list_activities.rb
```

Or use the Temporal CLI:

```bash
temporal activity list
```

The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow
Visibility](/visibility). For example, `ActivityType = 'ComposeGreeting' AND ExecutionStatus = 'Running'`.

## Count Standalone Activities 

Use [`Temporalio::Client#count_activities`](https://ruby.temporal.io/Temporalio/Client.html#count_activities-instance_method) to count
Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the
total count of executions (running, completed, failed, etc.) — not the number of queued tasks. It
works the same way as counting Workflow Executions.

[count_activities.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/count_activities.rb)

```ruby
result = client.count_activities("TaskQueue = 'standalone-activity-sample'")
puts "Total: #{result.count}"
result.groups.each do |group|
  puts "  #{group.group_values.join(',')} => #{group.count}"
end
```

Run it:

```bash
bundle exec ruby standalone_activity/count_activities.rb
```

Or use the Temporal CLI:

```bash
temporal activity count
```

## Run Standalone Activities with Temporal Cloud 

The Worker and Client code in the [Standalone Activities quickstart](/develop/ruby/activities/standalone-activities-quickstart)
use [`Temporalio::EnvConfig::ClientConfig.load_client_connect_options`](https://ruby.temporal.io/Temporalio/EnvConfig/ClientConfig.html#load_client_connect_options-class_method),
so the same code works against Temporal Cloud — configure the connection via environment
variables or a TOML profile. No code changes are needed.

For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate
generation, and authentication setup in the Cloud UI, see
[Connect to Temporal Cloud](/develop/ruby/client/temporal-client#connect-to-temporal-cloud).

### Connect with mTLS

Set these environment variables with values from your Temporal Cloud Namespace settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem'
export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key'
```

### Connect with an API key

Set these environment variables with values from your Temporal Cloud API key settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_API_KEY=<your-api-key>
```

Then run the Worker and starter code as shown in the [Standalone Activities quickstart](/develop/ruby/activities/standalone-activities-quickstart).

---

# Standalone Activities Ruby quickstart - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/standalone-activities-quickstart

> Execute a Standalone Activity with the Temporal Ruby SDK without writing a Workflow.

# Quickstart

Standalone Activities are Activities that run independently, without being orchestrated by a
Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a [`Temporalio::Client`](https://ruby.temporal.io/Temporalio/Client.html).

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/ruby/activities/basics). The only difference is that you execute a Standalone
Activity directly from your Temporal Client.

> **📝 Note:**
>
> This documentation uses source code from the
> [standalone_activity](https://github.com/temporalio/samples-ruby/tree/main/standalone_activity)
> sample.
>

## Get started with Standalone Activities 

Prerequisites:

- **Ruby** 3.3+

- **Temporal Ruby SDK** (v1.9.0 or higher). See the [Ruby Quickstart](/develop/ruby/set-up-local-ruby) for
  install instructions.

- **Temporal CLI** v1.9.1 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`.

Start the Temporal development server with `temporal server start-dev`.

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Server will now be available for client connections on `localhost:7233`, and the
Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233).

```bash
brew install temporal
```

```bash
temporal --version
```

```bash
temporal server start-dev
```

## Clone the sample

Clone the [samples-ruby](https://github.com/temporalio/samples-ruby) repository to follow along:

```bash
git clone https://github.com/temporalio/samples-ruby.git
cd samples-ruby
bundle install
```

The sample consists of separate programs in the `standalone_activity` directory:

```
standalone_activity/
├── my_activities.rb      # Activity definition
├── worker.rb             # Worker that processes activity tasks
├── execute_activity.rb   # Starts an activity and waits for the result
├── start_activity.rb     # Starts an activity without blocking
├── list_activities.rb    # Lists activity executions
└── count_activities.rb   # Counts activity executions
```

## Define your Activity 

An Activity in the Temporal Ruby SDK is a subclass of [`Temporalio::Activity::Definition`](https://ruby.temporal.io/Temporalio/Activity/Definition.html) that
implements an `execute` method. The way you define a Standalone Activity is identical to how you
define an Activity orchestrated by a Workflow. In fact, the same Activity can be executed both as a
Standalone Activity and as a Workflow Activity.

[my_activities.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/my_activities.rb)

```ruby
require 'temporalio/activity'

module StandaloneActivity
  module MyActivities
    class ComposeGreeting < Temporalio::Activity::Definition
      def execute(greeting, name)
        "#{greeting}, #{name}!"
      end
    end
  end
end
```

## Run a Worker with the Activity registered 

Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities —
you create a [`Temporalio::Worker`](https://ruby.temporal.io/Temporalio/Worker.html), register the Activity class, and call `worker.run`. The Worker
doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone
Activity. See [Run a Worker](/develop/ruby/workers/run-worker-process) for more details on
Worker setup and configuration options.

[worker.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/worker.rb)

Open a new terminal, navigate to the `samples-ruby` directory, and run the Worker.
Leave this terminal running — the Worker needs to stay up to process activities.

```ruby
args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options
args[0] ||= 'localhost:7233'
args[1] ||= 'default'

client = Temporalio::Client.connect(*args, **kwargs)

worker = Temporalio::Worker.new(
  client:,
  task_queue: 'standalone-activity-sample',
  activities: [StandaloneActivity::MyActivities::ComposeGreeting]
)

puts 'Starting worker (ctrl+c to exit)'
worker.run(shutdown_signals: ['SIGINT'])
```

```bash
bundle exec ruby standalone_activity/worker.rb
```

## Execute a Standalone Activity 

Use [`Temporalio::Client#execute_activity`](https://ruby.temporal.io/Temporalio/Client.html#execute_activity-instance_method) to execute a
Standalone Activity and block until it completes. Call this from your application code, not from
inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal Server,
waits for it to be executed on your Worker, and then returns the result.

[execute_activity.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/execute_activity.rb)

The first argument is the Activity to run. It can be the [`Activity::Definition`](https://ruby.temporal.io/Temporalio/Activity/Definition.html) subclass, an
instance of one, or a string/symbol name. Positional arguments after it are passed to the Activity's
`execute` method. The call requires `id`, `task_queue`, and at least one of `start_to_close_timeout`
or `schedule_to_close_timeout`.

To run it:

1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above).
2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above).
3. Open a new terminal, navigate to the `samples-ruby` directory, and run:
    ```
    bundle exec ruby standalone_activity/execute_activity.rb
    ```

Or use the Temporal CLI.

```ruby
result = client.execute_activity(
  StandaloneActivity::MyActivities::ComposeGreeting,
  'Hello', 'World',
  id: 'standalone-activity-id',
  task_queue: 'standalone-activity-sample',
  start_to_close_timeout: 10
)
puts "Activity result: #{result}"
```

```bash
bundle exec ruby standalone_activity/execute_activity.rb
```

```bash
temporal activity execute \\
  --type ComposeGreeting \\
  --activity-id standalone-activity-id \\
  --task-queue standalone-activity-sample \\
  --start-to-close-timeout 10s \\
  --input '"Hello"' \\
  --input '"World"'
```

## Run with Temporal Cloud

All code samples on this page use
[`Temporalio::EnvConfig::ClientConfig.load_client_connect_options`](https://ruby.temporal.io/Temporalio/EnvConfig/ClientConfig.html#load_client_connect_options-class_method)
to configure the Temporal Client connection. It responds to [environment
variables](/references/client-environment-configuration) and [TOML configuration
files](/references/client-environment-configuration), so the same code works against a local dev
server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal
Cloud](/develop/ruby/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the feature guide
for mTLS and API key setup.

## Next steps

- **[Standalone Activities feature guide](/develop/ruby/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud.
- **[Activity basics](/develop/ruby/activities/basics)**: Write and register Activities with the Ruby SDK.

---

# Activity Timeouts - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/activities/timeouts

> Set the Schedule-To-Close, Start-To-Close, and Heartbeat Timeouts on a Ruby Activity, tune its Retry Policy, and override the next retry delay.

## Activity timeouts 

Each Activity Timeout controls a different aspect of how long an Activity Execution can take:

- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout)**
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout)**
- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout)**

At least one of `start_to_close_timeout` or `schedule_to_close_timeout` is required.

```ruby
Temporalio::Workflow.execute_activity(
  MyActivity,
  { greeting: 'Hello', name: },
  start_to_close_timeout: 5 * 60
)
```

### Activity Retry Policy 

By default, Activities use a system Retry Policy.
You can override it by specifying a custom Retry Policy.

To create an Activity Retry Policy in Ruby, set the `retry_policy` parameter when executing an activity.

```ruby
Temporalio::Workflow.execute_activity(
  MyActivity,
  { greeting: 'Hello', name: },
  start_to_close_timeout: 5 * 60,
  retry_policy: Temporalio::RetryPolicy.new(max_interval: 10)
)
```

### Override the retry interval with `next_retry_delay` 

If you raise an application-level error, you can override the Retry Policy's delay by specifying a new delay.

```ruby
raise Temporalio::Error::ApplicationError.new(
  'Some error',
  type: 'SomeErrorType',
  next_retry_delay: 3 * Temporalio::Activity::Context.current.info.attempt
)
```

## Heartbeat an Activity 

A Heartbeat is a periodic signal from the Worker to the Temporal Service indicating the Activity is still alive and making progress.

- Heartbeats are used to detect Worker failure.
- Cancellations are delivered via Heartbeats.
- Heartbeats may contain custom progress details.

```ruby
class MyActivity < Temporalio::Activity::Definition
  def execute
    # This is a naive loop simulating work, but similar heartbeat logic
    # applies to other scenarios as well
    loop do
      # Send heartbeat
      Temporalio::Activity::Context.current.heartbeat
      # Sleep before heartbeating again
      sleep(3)
    end
  end
end
```

### Heartbeat Timeout 

The Heartbeat Timeout sets the maximum duration between Heartbeats before the Temporal Service considers the Activity failed.

```ruby
Temporalio::Workflow.execute_activity(
  MyActivity,
  { greeting: 'Hello', name: },
  start_to_close_timeout: 5 * 60,
  heartbeat_timeout: 5
)
```

---

# Best practices - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/best-practices

> This section explains how to implement best practices with the Ruby SDK

![Ruby SDK Banner](https://docs.temporal.io/img/assets/banner-ruby-temporal.png)

## Best practices

- [Error handling](/develop/ruby/best-practices/error-handling)
- [Testing](/develop/ruby/best-practices/testing-suite)
- [Debugging](/develop/ruby/best-practices/debugging)
- [Converters and encryption](/develop/ruby/best-practices/data-handling)

---

# Data handling - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/best-practices/data-handling

All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three
layers that handle different concerns:

![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)

Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON
serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when
your application requires non-JSON types, encryption, or payload offloading.

|                           | [PayloadConverter](/develop/ruby/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/ruby/best-practices/data-handling/data-encryption) |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- |
| **Purpose**               | Serialize application data to bytes                               | Transform encoded payloads (encrypt, compress)                |
| **Default**               | JSON serialization                                                | None (passthrough)                                            |

For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage).

---

# Payload conversion - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/best-practices/data-handling/data-conversion

> Customize how Temporal serializes application objects using Payload Converters in the Ruby SDK.

## Payload conversion 

Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back.

### Conversion sequence 

The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter.
You can set multiple encoding Payload Converters to run your conversions.
When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

Payload Converters can be customized independently of a Payload Codec.
Temporal's Converter architecture looks like this:

![Temporal converter architecture](https://docs.temporal.io/img/info/converter-architecture.png)

## Supported Data Types 

Data converters are used to convert raw Temporal payloads to/from actual Ruby types.
A custom data converter can be set via the `data_converter` keyword argument when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters.
Payload converters convert Ruby values to/from serialized bytes. Payload codecs convert bytes to bytes (for example, for compression or encryption). Failure converters convert exceptions to/from serialized failures.

Data converters are in the `Temporalio::Converters` module.
The default data converter uses a default payload converter, which supports the following types:

- `nil`
- "bytes" (that is, `String` with `Encoding::ASCII_8BIT` encoding)
- `Google::Protobuf::MessageExts` instances
- [JSON module](https://docs.ruby-lang.org/en/master/JSON.html) for everything else

This means that normal Ruby objects will use `JSON.generate` when serializing and `JSON.parse` when deserializing (with `create_additions: true` set by default).
So a Ruby object will often appear as a hash when deserialized.
Also, hashes that are passed in with symbol keys end up with string keys when deserialized.
While "JSON Additions" are supported, it is not cross-SDK-language compatible since this is a Ruby-specific construct.

The default payload converter is a collection of "encoding payload converters".
On serialize, each encoding converter will be tried in order until one accepts (default falls through to the JSON one).
The encoding converter sets an `encoding` metadata value which is used to know which converter to use on deserialize.
Custom encoding converters can be created, or even the entire payload converter can be replaced with a different implementation.

**NOTE:** For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models.
Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons.
Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics.
Instead, consider having models specific for workflows/activities and translate to/from existing models as needed.
See the next section on how to do this with ActiveModel objects.

#### ActiveModel 

By default, ActiveModel objects do not natively support the `JSON` module.
A mixin can be created to add this support for ActiveModel, for example:

```ruby
module ActiveModelJSONSupport
  extend ActiveSupport::Concern
  include ActiveModel::Serializers::JSON

  included do
    def as_json(*)
      super.merge(::JSON.create_id => self.class.name)
    end

    def to_json(*args)
      as_json.to_json(*args)
    end

    def self.json_create(object)
      object = object.dup
      object.delete(::JSON.create_id)
      new(**object.symbolize_keys)
    end
  end
end
```

Now if `include ActiveModelJSONSupport` is present on any ActiveModel class, on serialization `to_json` will be used which will use `as_json` which calls the super `as_json` but also includes the fully qualified class name as the JSON
`create_id` key.
On deserialization, Ruby JSON then uses this key to know what class to call `json_create` on.

---

# Payload encryption - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/best-practices/data-handling/data-encryption

> Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Ruby SDK.

Temporal's security model is designed around client-side encryption of Payloads.
A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server.
This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data.
It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures.

A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client.
Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these.

The server itself never adds encryption over Payloads.
Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows.
When working with sensitive data, you should always implement Payload encryption.

## Custom Payload Codec 

Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps.
If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`.

The Payload Codec needs to extend `Temporalio::Converters::PayloadCodec` and implement `encode` and `decode` methods.
These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field.
Do not mutate the existing payloads.
Here is an example of an encryption codec that just uses base64 in each direction:

```ruby
class Base64Codec < Temporalio::Converters::PayloadCodec
  def encode(payloads)
    payloads.map do |p|
      Temporalio::Api::Common::V1::Payload.new(
        # Set our specific encoding. We may also want to add a key ID in here for use by
        # the decode side
        metadata: { 'encoding' => 'binary/my-payload-encoding' },
        data: Base64.strict_encode64(p.to_proto)
      )
    end
  end

  def decode(payloads)
    payloads.map do |p|
      # Ignore if it doesn't have our expected encoding
      next p unless p.metadata['encoding'] == 'binary/my-payload-encoding'

      Temporalio::Api::Common::V1::Payload.decode(
        Base64.strict_decode64(p.data)
      )
    end
  end
end
```

### Set Data Converter to use custom Payload Codec

When creating a client, the default `DataConverter` can be updated with the payload codec like so:

```ruby
my_client = Temporalio::Client.connect(
  'localhost:7233',
  'my-namespace',
  data_converter: Temporalio::Converters::DataConverter.new(payload_codec: Base64Codec.new)
)
```

- Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted.
- Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster.
  Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output.

For reference, see the [Encryption](https://github.com/temporalio/samples-ruby/tree/main/encryption) sample.

## Using a Codec Server

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints.
You create, operate, and manage access to your Codec Server in your own environment.
The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.
Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server.

---

# Debugging - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/best-practices/debugging

> Debug Workflows in development and production environments using Temporal Ruby SDK. Use logging, debugger, CLI, replay, tracing, and more.

## Debugging 

This page shows how to do the following:

- [Debug in a development environment](#debug-in-a-development-environment)
- [Debug in a production environment](#debug-in-a-production-environment)

## Debug in a development environment 

In developing Workflows, you can use the normal development tools of logging and a debugger to see what’s happening in your Workflow.

In addition to the normal development tools of logging and a debugger, you can also see what’s happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli).
The Web UI provides insight into your Workflows, making it easier to identify issues and monitor the state of your Workflows in real time.

## Debug in a production environment 

For production Workflows, debugging options include:

- [Web UI](/web-ui)
- [Temporal CLI](/cli)
- [Replay](/develop/ruby/best-practices/testing-suite#replay-test)
- [Tracing](/develop/ruby/platform/observability#tracing)
- [Logging](/develop/ruby/platform/observability#logging)

You can analyze Worker performance using:

- [Metrics](/develop/ruby/platform/observability#metrics)
- [Worker performance guide](/develop/worker-performance)

To monitor Server performance:

- Use [Cloud metrics](/cloud/metrics/openmetrics) if you're on Temporal Cloud
- Or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics) if running your own deployment

---

# Error handling - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/best-practices/error-handling

> Handle errors with Temporal Ruby SDK

## Raise and handle exceptions 

In each Temporal SDK, error handling is implemented idiomatically, following the conventions of the language.
Temporal uses several different error classes internally — for example, [`CancelledError`](https://ruby.temporal.io/Temporalio/Error/CanceledError.html) in the Ruby SDK, to handle a Workflow cancellation. 
You should not raise or otherwise implement these manually, as they are tied to Temporal platform logic.

The one Temporal error class that you will typically raise deliberately is [`ApplicationError`](https://ruby.temporal.io/Temporalio/Error/ApplicationError.html).
In fact, *any* other exceptions that are raised from your Ruby code in a Temporal Activity will be converted to an `ApplicationError` internally.
This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries.

In other words, these two code samples do the same thing:

```ruby
class MyError < StandardError
end

class SomethingThatFails < Temporalio::Activity::Definition
  def execute(details)
    Temporalio::Activity::Context.current.logger.info(
      "We have a problem."
    )
    raise MyError.new('Simulated failure')
  end
end
```

```ruby
class SomethingThatFails < Temporalio::Activity::Definition
  def execute(details)
    Temporalio::Activity::Context.current.logger.info(
      "We have a problem."
    )
    raise Temporalio::Error::ApplicationError.new('Simulated failure', type: 'MyError')
  end
end
```

Depending on your implementation, you may decide to use either method.
One reason to use the Temporal `ApplicationError` class is because it allows you to set an additional `non_retryable` parameter.
This way, you can decide whether an error should not be retried automatically by Temporal.
This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse:

```ruby
class SomethingThatFails < Temporalio::Activity::Definition
  def execute(details)
    Temporalio::Activity::Context.current.logger.info(
      "We have a problem."
    )
    raise Temporalio::Error::ApplicationError.new('Simulated failure', non_retryable: true)
  end
end
```

You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](/develop/ruby/activities/timeouts#activity-retries).

## Failing Workflows 

One of the core design principles of Temporal is that an Activity Failure will never directly cause a Workflow Failure — a Workflow should never return as Failed unless deliberately.
The default retry policy associated with Temporal Activities is to retry them until reaching a certain timeout threshold.
Activities will not actually *return* a failure to your Workflow until this condition or another non-retryable condition is met.
At this point, you can decide how to handle an error returned by your Activity the way you would in any other program.
For example, you could implement a [Saga Pattern](/design-patterns/saga-pattern) — see this [Ruby sample](https://github.com/temporalio/samples-ruby/tree/main/saga) — that uses `rescue` blocks to "unwind" some of the steps your Workflow has performed up to the point of Activity Failure.

**You will only fail a Workflow by manually raising an `ApplicationError` from the Workflow code.**
You could do this in response to an Activity Failure, if the failure of that Activity means that your Workflow should not continue:

```ruby
class SagaWorkflow < Temporalio::Workflow::Definition
  def execute(details)
    Temporalio::Workflow.execute_activity(Activities::SomethingThatFails, details,start_to_close_timeout: 30)
  rescue StandardError
    raise Temporalio::Error::ApplicationError.new('Fail the Workflow')
```

This works differently in a Workflow than raising exceptions from Activities.
In an Activity, any Ruby exceptions or custom exceptions are converted to a Temporal `ApplicationError`.
By default, in a Workflow, any exceptions that are raised other than an explicit Temporal `ApplicationError` will only fail that particular [Workflow Task](/tasks#workflow-task-execution) and be retried.
This includes any typical Ruby `RuntimeError`s that are raised automatically.
These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly.
To have other exception types fail the Workflow Execution, list them with `workflow_failure_exception_type` in the Workflow Definition, or in the `workflow_failure_exception_types` Worker option.

---

# Testing - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/best-practices/testing-suite

> The Ruby test-suite guide covers Workflow and integration testing for Temporal. It includes end-to-end, integration, and unit testing, emphasizing the use of the test server to optimize test execution.

This page shows how to do the following:

- [Understand types of tests](#types-of-tests)
- [Use compatible test frameworks](#test-frameworks)
- [Test Workflows](#testing-workflows)
- [Test Activities](#test-activities)
- [Replay tests](#replay-test)

The Ruby test-suite feature guide describes the frameworks that facilitate Workflow and integration testing.

## Types of tests 

In the context of Temporal, you can create these types of automated tests:

- **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client.
- **Integration:** Anything between end-to-end and unit testing.
  - Running Activities with mocked Context and other SDK imports (and usually network requests).
  - Running Workers with mock Activities, and using a Client to start Workflows.
  - Running Workflows with mocked SDK imports.
- **Unit:** Running a piece of Workflow or Activity code and mocking any code it calls.

We generally recommend writing the majority of your tests as integration tests.

Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers.

## Test frameworks 

**Compatible testing frameworks**

The Ruby SDK is compatible with any testing framework and does not have a specific recommendation.
Most Ruby SDK samples use [minitest](https://github.com/minitest/minitest).

## Testing Workflows 

Workflow testing can be done in an integration-test fashion against a real server, however it is hard to simulate timeouts and other long time-based code.
Using the time-skipping Workflow test environment can help there.

### Testing Workflows with standard server

A non-time-skipping `Temporalio::Testing::WorkflowEnvironment` can be started via `start_local` which supports all standard Temporal features.
It is actually the real Temporal dev server packaged in the Temporal CLI, lazily downloaded on first use, and run as a sub-process in the background.
Assuming tests properly use separate Task Queues, the same server can and should be reused across tests.

Here's a simple example of a Workflow:

```ruby
class SimpleWorkflow < Temporalio::Workflow::Definition
  def execute(name)
    "Hello, #{name}!"
  end
end
```

Here's how a test of that Workflow may appear in minitest:

```ruby
def test_simple_workflow
  # Start local server that is stopped when block is done
  Temporalio::Testing::WorkflowEnvironment.start_local do |env|
    # Start worker that is stopped when block is done
    worker = Temporalio::Worker.new(
      env.client,
      task_queue: "tq-#{SecureRandom.uuid}",
      workflows: [SimpleWorkflow]
    )
    worker.run do
      # Execute workflow and check result
      result = env.client.execute_workflow(
        SimpleWorkflow, 'some-name',
        id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue
      )
      assert_equal 'Hello, some-name!', result
    end
  end
end
```

While this is just a demonstration, a local server is often used as a fixture across many tests.
In minitest for instance, users often start the environment lazily (with no block), and shut it down inside a block passed to `Minitest.after_run`.

### Testing Workflows with time skipping

Sometimes there is a need to test Workflows that run a long time or to test that timeouts occur.
A time-skipping `Temporalio::Testing::WorkflowEnvironment` can be started via `start_time_skipping` which is a reimplementation of the Temporal server with special time skipping capabilities.
Like `start_local`, this also lazily downloads the process to run when first called.
Note, unlike `start_local`, this class is not thread safe nor safe for use with independent tests.
It can be technically be reused, but only for one test at a time because time skipping is locked/unlocked at the environment level.
Developers are encouraged to run it per test needed.

#### Automatic time skipping

Here's a simple example of a Workflow that waits a day:

```ruby
class WaitADayWorkflow < Temporalio::Workflow::Definition
  def execute
    Temporalio::Workflow.sleep(1 * 24 * 60 * 60)
    'all done'
  end
end
```

A regular integration test of this Workflow on a normal server would be way too slow.
However, the time-skipping server automatically skips to the next event when we wait on the result.
Here's a test for that Workflow in minitest:

```ruby
def test_wait_a_day_workflow
  # Start time-skipping test server that is stopped when block is done
  Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env|
    # Start worker that is stopped when block is done
    worker = Temporalio::Worker.new(
      env.client,
      task_queue: "tq-#{SecureRandom.uuid}",
      workflows: [WaitADayWorkflow]
    )
    worker.run do
      # Execute workflow and check result
      result = env.client.execute_workflow(
        WaitADayWorkflow,
        id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue
      )
      assert_equal 'all done', result
    end
  end
end
```

This test will run almost instantly.
This is because by calling `execute_workflow` on our client, we are actually calling `start_workflow` + `result`, and `result` automatically skips time as much as it can (basically until the end of the workflow or until an activity is run).

To disable automatic time-skipping while waiting for a workflow result, run code in a block passed to `env.auto_time_skipping_disabled`.

#### Manual time skipping

Until a Workflow is waited on, all time skipping in the time-skipping environment is done manually via `WorkflowEnvironment#sleep`.

Here's a Workflow that waits for a Signal or times out:

```ruby
class SignalWorkflow < Temporalio::Workflow::Definition
  def execute
    # Wait for signal or timeout in 45 seconds
    Temporalio::Workflow.timeout(45 * 60) do
      Temporalio::Workflow.wait_condition { @signal_received }
    end
    'got signal'
  rescue Timeout::Error
    'got timeout'
  end

  workflow_signal
  def some_signal
    @signal_received = true
  end
end
```

To test a normal Signal in minitest, you might:

```ruby
def test_signal_workflow
  Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env|
    worker = Temporalio::Worker.new(
      env.client,
      task_queue: "tq-#{SecureRandom.uuid}",
      workflows: [SignalWorkflow]
    )
    worker.run do
      handle = env.client.start_workflow(
        SignalWorkflow,
        id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue
      )
      handle.signal(SignalWorkflow.some_signal)
      assert_equal 'got signal', handle.result
    end
  end
end
```

But how would you test the timeout part? Like so:

```ruby
def test_signal_workflow_timeout
  Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env|
    worker = Temporalio::Worker.new(
      env.client,
      task_queue: "tq-#{SecureRandom.uuid}",
      workflows: [SignalWorkflow]
    )
    worker.run do
      handle = env.client.start_workflow(
        SignalWorkflow,
        id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue
      )
      # Advance 50 seconds
      env.sleep(50)
      assert_equal 'got timeout', handle.result
    end
  end
end
```

### Mocking Activities

When testing Workflows, often you don't want to actually run the Activities.
Activities are just classes that extend `Temporalio::Activity::Definition`.
Simply write different/empty/fake/asserting ones and pass those to the Worker to have different activities called during the test.

## Testing Activities 

Unit testing an Activity or any code that could run in an Activity is done via the `Temporalio::Testing::ActivityEnvironment` class.
Simply instantiate the class, and any code inside the block to `run` will be invoked inside the activity context.
Several things about the activity environment can be customized via parameters when constructing the environment including setting the info, providing a proc to call back on each heartbeat, and setting the cancellation to be used.

## Replay test 

Given a Workflow's history, it can be replayed locally to check for things like non-determinism errors.
For example, assuming the `history_json` parameter below is given a JSON string of history exported from the CLI or web UI for workflow `MyWorkflow`, the following method will replay it:

```ruby
def replay_from_json(history_json)
  # Create a replayer
  replayer = Temporalio::Worker::WorkflowReplayer.new(workflows: [MyWorkflow])
  # Replay the history
  history = Temporalio::WorkflowHistory.from_history_json(history_json)
  replayer.replay_workflow(history)
end
```

If there is a non-determinism, this will raise an exception.

Event history can be loaded from more than just JSON.
It can be fetched individually from a Workflow handle, or even in a list.
For example, the following code will check that all Workflow histories for a certain Workflow type (that is, workflow class) are safe with the current Workflow code.

```ruby
# Create a replayer
replayer = Temporalio::Worker::WorkflowReplayer.new(workflows: [MyWorkflow])
# Replay all workflows from a list
replayer.replay_workflows(client.list_workflows("WorkflowType = 'MyWorkflow'")).each do |result|
  # Raise if any failed (could have just set raise_on_replay_failure: true, but this
  # demonstrates iterating over the results)
  raise result.replay_failure if result.replay_failure
end
```

---

# Client - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/client

![Ruby SDK Banner](https://docs.temporal.io/img/assets/banner-ruby-temporal.png)

## Temporal Client

- [Temporal Client](/develop/ruby/client/temporal-client)

---

# Temporal Client - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service.
Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and
Queries to Workflow Executions, getting Workflow results, and more.

For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow.

This page shows you how to do the following using the Ruby SDK with the Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-development-service)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Start a Workflow Execution](#start-workflow)
- [Get Workflow results](#get-workflow-results)

A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a
Temporal Client inside an Activity to communicate with a Temporal Service.

## Connect to development Temporal Service 

Use [`Client.connect`](https://ruby.temporal.io/Temporalio/Client.html#connect-class_method) to create a client.
Connection options include the Temporal Server address, Namespace, and (optionally) TLS configuration. You can provide
these options directly in code, load them from **environment variables**, or a **TOML configuration file** using the
[`EnvConfig`](https://ruby.temporal.io/Temporalio/EnvConfig.html) helpers. We recommend environment variables or a
configuration file for secure, repeatable configuration.

When you're running a Temporal Service locally (such as with the
[Temporal CLI dev server](/cli/command-reference/server#start-dev)), the required options are minimal. If you
don't specify a host/port, most connections default to `127.0.0.1:7233` and the `default` Namespace.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the
TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK
looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to
[Environment Configuration](/references/client-environment-configuration) for more details about configuration
files and profiles.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own
set of connection options.

```toml title="config.toml"
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Optional: Add custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
# TLS auto-enables when TLS config or an API key is present
# disabled = false
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path  = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment     = "production"
service-version = "v1.2.3"
```

You can create a Temporal Client using a profile from the configuration file using the
`ClientConfig.load_client_connect_options` function as follows. In this example, you load the `default` profile for
local development:

```ruby
require 'temporalio/client'
require 'temporalio/env_config'

def main
  puts '--- Loading default profile from config.toml ---'

  # For this sample to be self-contained, we explicitly provide the path to
  # the config.toml file included in this directory.
  # By default though, the config.toml file will be loaded from
  # ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
  config_file = File.join(__dir__, 'config.toml')

  # load_client_connect_options is a helper that loads a profile and prepares
  # the configuration for Client.connect. By default, it loads the
  # "default" profile.
  args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options(
    config_source: Pathname.new(config_file)
  )

  puts "Loaded 'default' profile from #{config_file}."
  puts "  Address: #{args[0]}"
  puts "  Namespace: #{args[1]}"
  puts "  gRPC Metadata: #{kwargs[:rpc_metadata]}"

  puts "\nAttempting to connect to client..."
  begin
    client = Temporalio::Client.connect(*args, **kwargs)
    puts '✅ Client connected successfully!'
    sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new)
    puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}"
  rescue StandardError => e
    puts "❌ Failed to connect: #{e}"
  end
end

main if $PROGRAM_NAME == __FILE__
```

**Environment Variables**

Use the `EnvConfig` package to set connection options for the Temporal Client using environment variables. For a list of
all available environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

Set the following environment variables before running your application. Replace the placeholder values with your actual
configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and the
`default` Namespace. You may omit these variables entirely since they're the defaults.

```bash
export TEMPORAL_NAMESPACE="default"
export TEMPORAL_ADDRESS="localhost:7233"
```

After setting the environment variables, you can create a Temporal Client as follows:

```ruby {9}
require 'temporalio/client'
require 'temporalio/env_config'

def main
  # load_client_connect_options is a helper that loads a profile and prepares
  # the configuration for Client.connect. By default, it loads the
  # "default" profile and also reads from environment variables. The environment
  # variables take precedence over the config file.
  args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options()

  puts "  Address: #{args[0]}"
  puts "  Namespace: #{args[1]}"
  puts "  gRPC Metadata: #{kwargs[:rpc_metadata]}"

  puts "\nAttempting to connect to client..."
  begin
    client = Temporalio::Client.connect(*args, **kwargs)
    puts '✅ Client connected successfully!'
    sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new)
    puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}"
  rescue StandardError => e
    puts "❌ Failed to connect: #{e}"
  end
end

main if $PROGRAM_NAME == __FILE__
```

**Code**

If you don't want to use environment variables or a configuration file, you can specify connection options directly in
code. This is convenient for local development and testing. You can also load a base configuration from environment
variables or a configuration file, and then override specific options in code.

Use the `connect` class method on the `Temporalio::Client` class to create and connect to a Temporal Client to the
Temporal Service.

```ruby
client = Temporalio::Client.connect('localhost:7233', 'default')
```

## Connect to Temporal Cloud 

You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal
Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured
local development instance:

- Your credentials for authentication.
  - If you are using an API key, provide the API key value.
  - If you are using mTLS, provide the mTLS CA certificate and mTLS private key.
- Your _Namespace and Account ID_ combination, which follows the format `<namespace_id>.<account_id>`.
- The recommended _endpoint_ is the gRPC Namespace endpoint: `<namespace>.<account>.tmprl.cloud:7233`.
  This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability).
  See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options.

You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab.

For more information about managing and generating client certificates for Temporal Cloud, see
[How to manage certificates in Temporal Cloud](/cloud/certificates).

You can provide these connection options using environment variables, a configuration file, or directly in code.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to
[Environment Configuration](/references/client-environment-configuration).

You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path
to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the
default path `~/.config/temporalio/temporal.toml`.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to
connect to Temporal Cloud via an API key:

```toml
# Cloud profile for Temporal Cloud
[profile.staging]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```

If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and
private key:

```toml
# Cloud profile for Temporal Cloud
[profile.staging]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
tls_client_cert_data = "your-tls-client-cert-data"
tls_client_key_path = "your-tls-client-key-path"
```

With the connections options defined in the configuration file, use the
[`Client.connect` method](https://ruby.temporal.io/Temporalio/Client.html#connect-class_method) to create a Temporal
Client using the `staging` profile as follows. After loading the profile, you can also programmatically override
specific connection options before creating the client.

```ruby {8,14-16}
require 'temporalio/client'
require 'temporalio/env_config'

def main
  puts "--- Loading 'staging' profile with programmatic overrides ---"

  config_file = File.join(__dir__, 'config.toml')
  profile_name = 'staging'

  puts "The 'staging' profile in config.toml has an incorrect address (localhost:9999)."
  puts "We'll programmatically override it to the correct address."

  # Load the 'staging' profile.
  args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options(
    profile: profile_name,
    config_source: Pathname.new(config_file)
  )

  # Override the target host to the correct address.
  # This is the recommended way to override configuration values.
  args[0] = 'localhost:7233'

  puts "\nLoaded '#{profile_name}' profile from #{config_file} with overrides."
  puts "  Address: #{args[0]} (overridden from localhost:9999)"
  puts "  Namespace: #{args[1]}"

  puts "\nAttempting to connect to client..."
  begin
    client = Temporalio::Client.connect(*args, **kwargs)
    puts '✅ Client connected successfully!'
    sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new)
    puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}"
  rescue StandardError => e
    puts "❌ Failed to connect: #{e}"
  end
end

main if $PROGRAM_NAME == __FILE__
```

**Environment Variables**

The following environment variables are required to connect to Temporal Cloud:

- `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `<namespace_id>.<account_id>`.
- `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace.
- `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication.
- `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path.
  Required if you are using mTLS authentication.
- `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path.
  Required if you are using mTLS authentication.

Ensure these environment variables exist in your environment before running your application.

Require the `temporalio/env_config` module to set connection options for the Temporal Client using environment variables.
The `Temporalio::EnvConfig::ClientConfig.load_client_connect_options` method will automatically load all environment variables. For a list of all available
environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

After setting the environment variables, use the following code to create the Temporal Client:

```ruby {9, 17}
require 'temporalio/client'
require 'temporalio/env_config'

def main
  # load_client_connect_options is a helper that loads a profile and prepares
  # the configuration for Client.connect. By default, it loads the
  # "default" profile. It also reads from environment variables. The environment
  # variables take precedence over the config file.
  args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options()

  puts "  Address: #{args[0]}"
  puts "  Namespace: #{args[1]}"
  puts "  gRPC Metadata: #{kwargs[:rpc_metadata]}"

  puts "\nAttempting to connect to client..."
  begin
    client = Temporalio::Client.connect(*args, **kwargs)
    puts '✅ Client connected successfully!'
    sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new)
    puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}"
  rescue StandardError => e
    puts "❌ Failed to connect: #{e}"
  end
end

main if $PROGRAM_NAME == __FILE__
```

**Code**

You can also specify connection options directly in code to connect to Temporal Cloud. To create an initial connection,
provide the endpoint, Namespace and Account ID combination, and API key values to the `Client.connect` method.

```ruby
client = Temporalio::Client.connect(
  '<endpoint>', # Endpoint
  '<namespace_id>.<account_id>', # Namespace
  api_key: '<api_key>',
  tls: true
)
```

To connect using mTLS instead of an API key, provide the mTLS certificate and private key as follows:

```ruby
client = Temporalio::Client.connect(
  '<endpoint>', # Endpoint
  '<namespace_id>.<account_id>', # Namespace
  tls: Temporalio::Client::Connection::TLSOptions.new(
    client_cert: File.read('my-client-cert.pem'),
    client_private_key: File.read('my-client-key.pem')
  )
)
```

For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see
[Temporal Customization Samples](https://github.com/temporalio/samples-server).

`TLSOptions` is static: the certificate you pass to `Client.connect` is fixed for the lifetime of that client. To rotate
an mTLS client certificate without restarting your Worker, connect a new client with the new certificate, then assign
it to the running `Worker`'s `client`:

```ruby
new_client = Temporalio::Client.connect(
  '<endpoint>', # Endpoint
  '<namespace_id>.<account_id>', # Namespace
  tls: Temporalio::Client::Connection::TLSOptions.new(
    client_cert: File.read('my-client-cert-new.pem'),
    client_private_key: File.read('my-client-key-new.pem')
  )
)

# my_worker is the Worker instance already running against the old client
my_worker.client = new_client
```

The Worker starts using `new_client` for subsequent calls to the Temporal Service; calls already in flight on the old
client finish normally.

## Start a Workflow Execution 

To start a Workflow Execution, supply:

- A Task Queue
- A Workflow Type
- Input arguments
- Workflow options such as Workflow Id

To start a Workflow Execution in Ruby, use either the `start_workflow` or `execute_workflow` methods in the Client. You
must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) and [Task Queue](/task-queue) in the
parameters given to the method.

```ruby
result = my_client.execute_workflow(
  MyWorkflow, 'some-input',
  id: 'my-workflow-id', task_queue: 'my-task-queue'
)
puts "Result: #{result}"
```

### Set a Workflow's Task Queue 

The only Workflow Option that you must set is the name of the [Task Queue](/task-queue).

For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task
Queue name.

To set a Task Queue in Ruby, specify the `task_queue` keyword argument on `start_workflow` or `execute_workflow`.

```ruby
result = my_client.execute_workflow(
  MyWorkflow, 'some-input',
  id: 'my-workflow-id', task_queue: 'my-task-queue'
)
```

### Set a Workflow Id 

You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id).

When setting a Workflow Id, we recommend mapping it to a business process or business entity identifier, such as an
order identifier or customer identifier.

To set a Workflow Id in Ruby, specify the `id` keyword argument on `start_workflow` or `execute_workflow`.

```ruby
result = my_client.execute_workflow(
  MyWorkflow, 'some-input',
  id: 'my-workflow-id', task_queue: 'my-task-queue'
)
```

## Get Workflow results 

Once a Workflow Execution is started, the Workflow Id and Run Id can be used to uniquely identify it.

You can block until the result is available, or retrieve it later using the handle.

You can also use Queries to access Workflow state and results while the Workflow is running.

Use `start_workflow` or `workflow_handle` on the Client to return a Workflow handle. Then use the `result` method to
await on the result of the Workflow.

```ruby
handle = my_client.workflow_handle('my-workflow-id')
result = handle.result
puts "Result: #{result}"
```

---

# Integrations - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/integrations

> This section covers integrations with the Ruby SDK

The following integrations are available for the Temporal Ruby SDK.

- [Rails](/develop/ruby/integrations/rails-integration) — Integrate Temporal durable workflows into Ruby on Rails applications. _(Ruby · Framework)_

---

# Rails integration - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/integrations/rails-integration

> Use the Temporal Ruby SDK with Ruby on Rails, including conventions for ActiveRecord, ActiveModel, and handling lazy/eager loading.

Temporal Ruby SDK is a generic Ruby library that can work in any Ruby environment.
However, there are some common conventions for Rails users to be aware of.

See the [rails_app sample](https://github.com/temporalio/samples-ruby/tree/main/rails_app) for an example of using Temporal from Rails.

## ActiveRecord

For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models.
Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons.
Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics.
Instead, consider having models specific for Workflows/Activities and translate to/from existing models as needed.
See the [ActiveModel section](/develop/ruby/best-practices/data-handling/data-conversion#active-model) on how to do this with ActiveModel objects.

## Lazy/eager loading

By default, Rails eagerly loads all application code on application start in production, but lazily loads it in non-production environments.
Temporal Workflows by default disallow use of IO during the Workflow run.
With lazy loading enabled in dev/test environments, when an Activity class is referenced in a Workflow before it has been explicitly required, it can give an error like:

```
Cannot access File path from inside a workflow. If this is known to be safe, the code can be run in a Temporalio::Workflow::Unsafe.illegal_call_tracing_disabled block.
```

This comes from bootsnap via zeitwerk because it is lazily loading a class/module at Workflow runtime.
It is not good to lazily load code during a Workflow run because it can be side effecting.
Workflows and the classes they reference should be eagerly loaded.

To resolve this, either always eagerly load (for example, `config.eager_load = true`) or explicitly require what is used by a workflow at the top of the file.

Note, this only affects non-production environments.

---

# Platform - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/platform

> This section explains how to implement platform with the Ruby SDK

![Ruby SDK Banner](https://docs.temporal.io/img/assets/banner-ruby-temporal.png)

## Platform

- [Observability](/develop/ruby/platform/observability)
- [Enriching the UI](/develop/ruby/platform/enriching-ui)

---

# Enriching the user interface - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/platform/enriching-ui

> Add contextual information to workflows and events in the Temporal UI using the Ruby SDK.

Temporal supports adding context to Workflows and Events with metadata. 
This helps users identify and understand Workflows and their operations.

## Adding summary and details to Workflows

### Starting a Workflow

When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI:

```ruby
require 'temporalio/client'

# Create client
client = Temporalio::Client.connect('localhost:7233')

# Start a workflow with static summary and details
handle = client.start_workflow(
  'YourWorkflow',
  'workflow input',
  id: 'your-workflow-id',
  task_queue: 'your-task-queue',
  static_summary: 'Order processing for customer #12345',
  static_details: 'Processing premium order with expedited shipping'
)
```

`static_summary:` is a single-line description that appears in the Workflow list view, limited to 200 bytes.
`static_details:` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes.

The input format is standard Markdown excluding images, HTML, and scripts.

You can also use `execute_workflow` for synchronous execution:

```ruby
# Execute workflow synchronously
result = client.execute_workflow(
  'YourWorkflow',
  'workflow input',
  id: 'your-workflow-id',
  task_queue: 'your-task-queue',
  static_summary: 'Order processing for customer #12345',
  static_details: 'Processing premium order with expedited shipping'
)
```

### Inside the Workflow

Within a Workflow, you can get and set the _current workflow details_. 
Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. 
Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines.

```ruby
require 'temporalio'

class YourWorkflow < Temporalio::Workflow::Definition
  def execute(input)
    # Get the current details
    current_details = Temporalio::Workflow.current_details
    Temporalio::Workflow.logger.info("Current details: #{current_details}")
    
    # Set/update the current details
    Temporalio::Workflow.current_details = 'Updated workflow details with new status'
    
    'Workflow completed'
  end
end
```

### Adding Summary to Activities and Timers

You can attach a `summary:` to activities when starting them from within a Workflow:

```ruby
require 'temporalio'

class YourWorkflow < Temporalio::Workflow::Definition
  def execute(input)
    # Execute an activity with a summary
    result = Temporalio::Workflow.execute_activity(
      'YourActivity',
      input,
      start_to_close_timeout: 10,
      summary: 'Processing user data'
    )
    
    result
  end
end
```

Similarly, you can attach a `summary:` to timers within a Workflow:

```ruby
require 'temporalio'

class YourWorkflow < Temporalio::Workflow::Definition
  def execute(input)
    # Create a timer with a summary
    Temporalio::Workflow.sleep(300, summary: 'Waiting for payment confirmation')
    
    'Timer completed'
  end
end
```

The input format for `summary:` is a string, and limited to 200 bytes.

## Viewing summary and details in the UI

Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. 
Navigate to your Workflow's details page to see the metadata displayed in three key locations:

### User Metadata

On the Workflow Details page, under the **User Metadata** tab, you'll find the workflow-level metadata:

- **Summary & Details** - Displays the static summary and static details set when starting the workflow
- **Current Details** - Displays the dynamic details that can be updated during workflow execution

All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays.

### Timeline

The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance.

Labels longer than 120 characters are truncated with an ellipsis.

Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. 

Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later.

### Event History

Individual events in the Workflow's Event History display their associated summaries when available.

Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. When you do expand an event, the summary is also prominently displayed in the detailed view.

---

# Observability - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/platform/observability

> Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility using the Ruby SDK.

This page covers capabilities related to viewing the state of the application, including:

- [Metrics](#metrics)
- [Tracing](#tracing)
- [Logging](#logging)
- [Visibility](#visibility)

The observability guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application).
This includes viewing [Workflow Executions](/workflow-execution) tracked by the [Temporal Platform](/temporal#temporal-platform), as well as inspecting state at any point during execution.

## Emit metrics 

Each Temporal SDK can optionally emit metrics from either the Client or Worker process.
Metrics can be scraped by systems like Prometheus, and graphs can be created using tools like Grafana.

- For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide.
- For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics).

Metrics in Ruby are configured on the `metrics` argument of the `telemetry` argument when creating a global `Temporalio::Runtime`. That object should be created globally and should be used for all clients; therefore, you should configure this before any other Temporal code.

### Set a Prometheus endpoint

The following example exposes a Prometheus endpoint on port `9000`.

```ruby
Temporalio::Runtime.default = Temporalio::Runtime.new(
  telemetry: Temporalio::Runtime::TelemetryOptions.new(
    metrics: Temporalio::Runtime::MetricsOptions.new(
      prometheus: Temporalio::Runtime::PrometheusMetricsOptions.new(
        bind_address: '0.0.0.0:9000'
      )
    )
  )
)
```

### Custom metric handling

Instead of Prometheus or OpenTelemetry, an instance of `Temporalio::Runtime::MetricBuffer` can be provided as a `buffer` argument to the `MetricsOptions`.
`retrieve_updates` can then be periodically called on the buffer to get metric updates.

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Set [`global_tags`](https://ruby.temporal.io/Temporalio/Runtime/MetricsOptions.html#global_tags-instance_method) on the `MetricsOptions` to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker.

```ruby
Temporalio::Runtime.default = Temporalio::Runtime.new(
  telemetry: Temporalio::Runtime::TelemetryOptions.new(
    metrics: Temporalio::Runtime::MetricsOptions.new(
      opentelemetry: Temporalio::Runtime::OpenTelemetryMetricsOptions.new(
        url: 'http://127.0.0.1:4317',
        durations_as_seconds: true
      ),
      global_tags: {
          'team' => 'content-platform',
          'service' => 'checkout',
          'cost_center' => 'cc-1042',
          'environment' => 'production'
      }
    )
  )
)
```

### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag           | Example            | Question it answers                                       |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team`        | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service`     | `checkout`         | Which application emits these metrics?                    |
| `cost_center` | `cc-1042`          | Which budget does this Worker fleet belong to?            |
| `environment` | `production`       | Is this production traffic, or staging or test?           |
| `region`      | `us-east-2`        | Where does the Worker fleet run?                          |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Set up tracing 

Tracing enables observability into the sequence of calls across your application, including Workflows and Activities.

OpenTelemetry tracing for clients, activities, and workflows can be enabled using the `Temporalio::Contrib::OpenTelemetry::TracingInterceptor`. Specifically, when creating a client, set the interceptor like so:

```ruby
require 'opentelemetry/api'
require 'opentelemetry/sdk'
require 'temporalio/client'
require 'temporalio/contrib/open_telemetry'

# ... assumes my_otel_tracer_provider is a tracer provider created by the user
my_tracer = my_otel_tracer_provider.tracer('my-otel-tracer')

my_client = Temporalio::Client.connect(
  'localhost:7233', 'my-namespace',
  interceptors: [Temporalio::Contrib::OpenTelemetry::TracingInterceptor.new(my_tracer)]
)
```

When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker.
Spans are created and serialized through the server to give one trace for a Workflow Execution.

## Log from a Workflow 

Logging enables you to capture and persist important execution details from your Workflow and Activity code.

Logging uses the Ruby standard logging APIs. You can find the log levels supported in [their official documentation](https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html). The Temporal SDK core normally uses `WARN` as its default logging level.

The `logger` can be set when connecting a client.
The following example shows logging on the console and sets the level to `INFO`.

```ruby
require 'logger'
require 'temporalio/client'

my_client = Temporalio::Client.connect(
  'localhost:7233', 'my-namespace',
  logger: Logger.new($stdout, level: Logger::INFO)
)
```

You can log from a Workflow using `Temporalio::Workflow.logger` which is a special instance of Ruby's `Logger` that
appends workflow details to every log and does not log during replay.

```ruby
Temporalio::Workflow.logger.info("Some log #{some_value}")
```

There's also one for use in activities that appends Activity details to every log:

```ruby
Temporalio::Activity::Context.current.logger.info("Some log #{some_value}")
```

## Use Visibility APIs 

Visibility refers to Temporal features for listing, filtering, and inspecting Workflow Executions.

### Use Search Attributes 

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime`, and `ExecutionStatus` are automatically indexed.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) let you store domain-specific metadata for Workflows.

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service in the CLI or Web UI.
  - For example: `temporal operator search-attribute create --name CustomKeywordField --type Text`
    - Replace `CustomKeywordField` with the name of your Search Attribute.
    - Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList`
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an argument when starting the Execution.
  - In the Workflow by calling `Temporalio::Workflow.upsert_search_attributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `describe` on a `WorkflowHandle`.
  - In the Workflow by looking at `Temporalio::Workflow.search_attributes`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [In the Temporal CLI](/cli/command-reference/operator#list-2)
  - In code by calling `list_workflows`.

### List Workflow Executions 

Use the [list_workflows](https://ruby.temporal.io/Temporalio/Client.html#list_workflows-instance_method) method on the Client and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows.
The result is a lazy enumerator/enumerable.

```ruby
my_client.list_workflows("WorkflowType='GreetingWorkflow'").each do |wf|
  puts "Workflow: #{wf.id}"
end
```

### Set Custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

To set custom Search Attributes, use the `search_attributes` parameter for `start_workflow` or `execute_workflow`.
Keys should be predefined for reuse.

```ruby
# Predefined search attribute key, usually a global somewhere
MY_KEYWORD_KEY = Temporalio::SearchAttributes::Key.new(
  'my-keyword',
  Temporalio::SearchAttributes::IndexedValueType::KEYWORD
)

# ...

# Start workflow with the search attribute set
handle = my_client.start_workflow(
  MyWorkflow, 'some-input',
  id: 'my-workflow-id', task_queue: 'my-task-queue',
  search_attributes: Temporalio::SearchAttributes.new({ MY_KEYWORD_KEY => 'some-value' })
)
```

### Upsert Search Attributes 

You can upsert Search Attributes to add, update, or remove Search Attributes from within Workflow code.

To upsert custom Search Attributes, use the [`upsert_search_attributes`](https://ruby.temporal.io/Temporalio/Workflow.html#upsert_search_attributes-class_method) method with a set of updates.
Keys should be predefined for reuse.

```ruby
# Predefined search attribute key, usually a global somewhere
MY_KEYWORD_KEY = Temporalio::SearchAttributes::Key.new(
  'my-keyword',
  Temporalio::SearchAttributes::IndexedValueType::KEYWORD
)

# ...

class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    # ...

    Temporalio::Workflow.upsert_search_attributes(MY_KEYWORD_KEY.value_set('some-new-value'))

    # ...
  end
end
```

---

# Set up a local development environment with the Ruby SDK

Source: https://docs.temporal.io/develop/ruby/set-up-local-ruby

> Configure your local development environment to get started developing with Temporal

# Quickstart

This guide walks you through setting up the Temporal Ruby SDK and running your first Workflow. In just a few steps,
you'll install the SDK and start a local development server. To validate that your local environment is correctly
installed, we will execute a Workflow that will output "Hello, Temporal".

## Installation

This step sets up a new Ruby project using Bundler and installs the Temporal Ruby SDK.

We recommend using [Bundler](https://bundler.io/) to manage your Ruby project dependencies, including the Temporal SDK.
These tutorials assume Ruby 3.4.3 or higher.

Follow the steps to create a directory, initialize the project with a `Gemfile`, and add the Temporal SDK.

**Note:**

- Only macOS ARM/x64 and Linux ARM/x64 are supported.
- Source gem is published but **cannot be built directly**.
- Windows (MinGW) is not supported.
- `fibers`/`async` are only supported on Ruby **3.3+**.
- See [Platform Support](#) for full details.

**1. Check your Ruby version:**
```bash
ruby -v
```

You should see output like `ruby 3.4.3`. Ruby 3.2+ is required. We recommend Ruby 3.4.3.

**2. Create your project folder:**
```bash
mkdir temporal-project
cd temporal-project
```

**3. Initialize with Bundler:**
```bash
bundle init
```

**4. Add the Temporal Ruby SDK:**
```bash
bundle add temporalio
```

You should see output like:
```bash
Fetching gem metadata from https://rubygems.org/...
Resolving dependencies...
Installing temporalio 0.4.0 (arm64-darwin)
Bundle complete! 1 Gemfile dependency, 6 gems now installed.
```

**5. Install dependencies:**
```bash
bundle install
```

## Install Temporal CLI

The fastest way to get a development version of the Temporal Service running on your local machine is to use
[Temporal CLI](/cli).

Choose your operating system to install Temporal CLI.

**macOS**

Install the Temporal CLI using Homebrew:
```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH, for example:
```bash
sudo mv temporal /usr/local/bin
```

## Start the development server

Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory
database.

The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at
http://localhost:8233.

Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal
Service at any time by pressing CTRL+C.

Once you have everything installed, you're ready to build apps with Temporal on your local machine.

After installing, open a new Terminal window and start the development server:

```bash
temporal server start-dev
```

#### Change the Web UI port
The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server:
```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and
Activity.

This test will confirm that:

- The Temporal Ruby SDK is properly installed
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

### 1. Create the Activity

Create an Activity file (say_hello_activity.rb):

```ruby
require 'temporalio/activity'

# Implementation of a simple activity
class SayHelloActivity < Temporalio::Activity::Definition
  def execute(name)
    "Hello, #{name}!"
  end
end
```

### 2. Create the Workflow

Create a Workflow file (say_hello_workflow.rb):

```ruby
require 'temporalio/workflow'
require_relative 'say_hello_activity'

class SayHelloWorkflow < Temporalio::Workflow::Definition
  def execute(name)
    Temporalio::Workflow.execute_activity(
      SayHelloActivity,
      name,
      schedule_to_close_timeout: 300
    )
  end
end
```

### 3. Create and Run the Worker

With your Activity and Workflow defined, you need a Worker to execute them. Workers are a crucial part of your Temporal
application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on
Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a
[deep dive into Workers](/workers).

Create a Worker file (worker.rb):

```ruby
require 'temporalio/client'
require 'temporalio/worker'
require_relative 'say_hello_activity'
require_relative 'say_hello_workflow'

# Create a client
client = Temporalio::Client.connect('localhost:7233', 'default')

# Create a worker with the client, activities, and workflows
worker = Temporalio::Worker.new(
  client:,
  task_queue: 'my-task-queue',
  workflows: [SayHelloWorkflow],
  # There are various forms an activity can take, see "Activities" section for details
  activities: [SayHelloActivity]
)

# Run the worker until SIGINT. This can be done in many ways, see "Workers" section for details.
worker.run(shutdown_signals: ['SIGINT'])
```

Run the Worker:

```bash
ruby worker.rb
```

### 4. Execute the Workflow

Now that your Worker is running, it's time to start a Workflow Execution.

Create a separate file called starter.rb:

```ruby
require 'temporalio/client'
require_relative 'say_hello_workflow'

# Create a client
client = Temporalio::Client.connect('localhost:7233', 'default')

# Run workflow
result = client.execute_workflow(
  SayHelloWorkflow,
  'Temporal', # This is the input to the workflow
  id: 'my-workflow-id',
  task_queue: 'my-task-queue'
)
puts "Result: #{result}"
```

Then run:

```bash
ruby starter.rb
```

### Verify success

If everything is working correctly, you should see:

- Worker processing the workflow and activity
- Output: `Workflow result: Hello, Temporal!`
- Workflow Execution details in the [Temporal Web UI](http://localhost:8233)

- [Run your first Temporal Application](https://learn.temporal.io/getting_started/ruby/first_program_in_ruby/): Create a basic Workflow and run it with the Temporal Ruby SDK

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Workers - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workers

> This section explains how to implement Workers with the Ruby SDK

![Ruby SDK Banner](https://docs.temporal.io/img/assets/banner-ruby-temporal.png)

## Workers

- [Run a Worker](/develop/ruby/workers/run-worker-process)
- [Serverless Workers](/develop/ruby/workers/serverless-workers)

---

# Run a Worker - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workers/run-worker-process

> Create and run a Temporal Worker using the Ruby SDK.

## Create and run a Worker 

Create a `Temporalio::Worker` with a [Temporal Client](/develop/ruby/client/temporal-client), the Task Queue to poll, and the Workflows and Activities it can execute.
Call `run` to start polling.

<!--SNIPSTART ruby-create-worker-->
[features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb)
```rb
worker = Temporalio::Worker.new(
  client: client,
  task_queue: 'my-task-queue',
  workflows: [GreetingWorkflow],
  activities: [SayHello]
)

worker.run
```
<!--SNIPEND-->

`run` blocks until the Worker shuts down.
To run several Workers in one process, use `Temporalio::Worker.run_all`, which returns once every Worker it was given has stopped.

## Register Workflows and Activities 

All Workers polling the same Task Queue must register the same Workflow Types and Activity Types.
A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue.
A Worker that receives a Task for a type it did not register fails that Task.

Pass Workflow classes in `workflows` and Activities in `activities`.
For Activities you can pass either the class or an instance:

```ruby
worker = Temporalio::Worker.new(
  client:,
  task_queue: 'my-task-queue',
  workflows: [GreetingWorkflow, OrderWorkflow],
  activities: [SayHello, ChargeCard.new(payment_client)]
)
```

Passing a class makes the Worker instantiate it for each Activity Execution.
Passing an instance reuses that object for every execution, which is how Activities share state such as a database client. A shared instance must be thread-safe.

## Connect to Temporal Cloud 

To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials.
See [Connect to Temporal Cloud](/develop/ruby/client/temporal-client#connect-to-temporal-cloud) for setup instructions.

## Configure Worker options 

`Temporalio::Worker.new` takes keyword arguments that control concurrency limits, pollers, timeouts, and caching, including `max_concurrent_activities`, `max_concurrent_workflow_tasks`, and `max_cached_workflows`.
The defaults work for most cases.

To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference).

## Run a versioned Worker 

Set a Worker Deployment Version and enable versioning in `deployment_options`, then set a versioning behavior on each Workflow.

<!--SNIPSTART ruby-versioned-worker-->
[features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb)
```rb
worker = Temporalio::Worker.new(
  client: client,
  task_queue: 'my-task-queue',
  workflows: [VersionedGreetingWorkflow],
  activities: [SayHello],
  deployment_options: Temporalio::Worker::DeploymentOptions.new(
    version: Temporalio::WorkerDeploymentVersion.new(
      deployment_name: 'my-app',
      build_id: '1.0'
    ),
    use_worker_versioning: true
  )
)
```
<!--SNIPEND-->

Declare the behavior in the Workflow class with `workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED` or `AUTO_UPGRADE`, or set a default for the whole Worker with `default_versioning_behavior` on `DeploymentOptions`.

A versioning behavior applies only to a Worker that has versioning enabled.
If a Workflow declares one and its Worker does not enable versioning, the server rejects the Workflow Task and the Task retries instead of failing outright.

See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out.

## Shut down a Worker 

Pass the signals that should stop the Worker to `run`.
The Worker stops polling for new Tasks and waits for in-flight Tasks to finish.

<!--SNIPSTART ruby-worker-graceful-shutdown-->
[features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb)
```rb
worker.run(shutdown_signals: %w[SIGINT SIGTERM])
```
<!--SNIPEND-->

You can also pass a block to `run`, which shuts the Worker down when the block completes, or stop the Worker with a `Temporalio::Cancellation`.
`Temporalio::Worker.run_all` takes the same `shutdown_signals`, `cancellation`, and block, and applies them to every Worker it was given.

See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities.

---

# Serverless Workers - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workers/serverless-workers

> Write Temporal Workers that run on serverless compute using the Ruby SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

Serverless Workers run on compute that Temporal starts and stops for you, rather than on long-lived processes you operate.

For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers).
For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Supported providers

- [**GCP Cloud Run**](/develop/ruby/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in.

---

# Serverless Workers on GCP Cloud Run - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workers/serverless-workers/cloud-run

> Run a Temporal Worker on a GCP Cloud Run worker pool using the Ruby SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Ruby Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific gem.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).

## Create a versioned Worker 

Build the Worker as you would any long-running Ruby Worker, then pass `deployment_options` to `Temporalio::Worker.new` to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

```ruby
require 'temporalio/client'
require 'temporalio/worker'

client = Temporalio::Client.connect(
  ENV.fetch('TEMPORAL_ADDRESS'),
  ENV.fetch('TEMPORAL_NAMESPACE'),
  api_key: ENV.fetch('TEMPORAL_API_KEY'),
  tls: true
)

worker = Temporalio::Worker.new(
  client:,
  task_queue: ENV.fetch('TEMPORAL_TASK_QUEUE'),
  workflows: [GreetingWorkflow],
  activities: [SayHello],
  deployment_options: Temporalio::Worker::DeploymentOptions.new(
    version: Temporalio::WorkerDeploymentVersion.new(
      deployment_name: 'my-app',
      build_id: 'build-1'
    ),
    use_worker_versioning: true,
    default_versioning_behavior: Temporalio::VersioningBehavior::PINNED
  )
)

worker.run
```

`deployment_name` and `build_id` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage.

Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`.
Setting `default_versioning_behavior` as shown above covers every Workflow on the Worker.
To set the behavior per Workflow instead, call `workflow_versioning_behavior` in the Workflow class:

```ruby
class GreetingWorkflow < Temporalio::Workflow::Definition
  workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED

  def execute(name)
    # ...
  end
end
```

If versioning is on and neither is set, the Worker raises an error at startup rather than polling.

For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/ruby/workers/run-worker-process).

## Configure the Temporal connection 

Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext.
The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace.

For the shared configuration format that other Temporal tools read, see [Environment configuration](/develop/environment-configuration).

## Package the Worker image 

The Ruby SDK ships precompiled gems per platform, so install it in the image rather than building the native extension from source:

```dockerfile
FROM ruby:3.3-slim

WORKDIR /app
RUN gem install temporalio --no-document
COPY worker.rb ./

CMD ["ruby", "worker.rb"]
```

Installing through Bundler in a container can select the source gem instead of the precompiled one, which then fails to build without a Rust toolchain. If you use Bundler, add the target platform to the lockfile with `bundle lock --add-platform x86_64-linux`.

## Keep Activities safe across scale-in 

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing.
An instance running a long Activity can be stopped mid-execution.

Use [Activity Heartbeats](/develop/ruby/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over:

```ruby
class Process < Temporalio::Activity::Definition
  def execute(items)
    items.each_with_index do |item, i|
      Temporalio::Activity::Context.current.heartbeat(i)
      # ... process item
    end
    'done'
  end
end
```

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability 

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Ruby SDK](/develop/ruby/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).

---

# Workflows - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows

> This section explains how to implement Workflows with the Ruby SDK

![Ruby SDK Banner](https://docs.temporal.io/img/assets/banner-ruby-temporal.png)

## Workflows

- [Workflow basics](/develop/ruby/workflows/basics)
- [Child Workflows](/develop/ruby/workflows/child-workflows)
- [Continue-As-New](/develop/ruby/workflows/continue-as-new)
- [Cancellation](/develop/ruby/workflows/cancellation)
- [Timeouts](/develop/ruby/workflows/timeouts)
- [Message passing](/develop/ruby/workflows/message-passing)
- [Schedules](/develop/ruby/workflows/schedules)
- [Timers](/develop/ruby/workflows/timers)
- [Futures](/develop/ruby/workflows/futures)
- [Dynamic Workflow](/develop/ruby/workflows/dynamic-workflow)
- [Versioning](/develop/ruby/workflows/versioning)

---

# Workflow basics - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/basics

> This section explains Workflow basics with the Ruby SDK

## Develop a Workflow 

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal Ruby SDK programming model, Workflows are defined as classes.

Have the Workflow class extend `Temporalio::Workflow::Definition` to define a Workflow.

The entrypoint is the `execute` method.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute(name)
    Temporalio::Workflow.execute_activity(
      MyActivity,
      { greeting: 'Hello', name: },
      start_to_close_timeout: 100
    )
  end
end
```

### Define Workflow parameters 

Temporal Workflows may have any number of custom parameters.
However, we strongly recommend that you pass a single hash or object parameter that contains all the input fields, so that you can add fields without breaking the signature of the Workflow.
For more reasons, see [Workflow parameters](/workflow-definition#workflow-parameters).

### Customize Workflow Type 

Workflows have a Type that are referred to as the Workflow name.

The following examples demonstrate how to set a custom name for your Workflow Type.

You can customize the Workflow name with a custom name in a `workflow_name` class method call on the class.
The Workflow name defaults to the unqualified class name.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  # Customize the name
  workflow_name :MyDifferentWorkflowName

  def execute(name)
    Temporalio::Workflow.execute_activity(
      MyActivity,
      { greeting: 'Hello', name: },
      start_to_close_timeout: 100
    )
  end
end
```

### Use Workflow constructors

Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). The `workflow_init` class method above `initialize` gives it access to [Workflow input](/handling-messages#workflow-initializers). When you use the `workflow_init` on your constructor, you give the constructor the same Workflow parameters as your `execute` method.

The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/ruby/client/temporal-client#start-workflow). The Workflow input arguments are also passed to your `execute` method. That always happens, whether or not you use the `workflow_init` class method above `initialize`.

Here's an example.
The constructor and `execute` must have the same parameters with the same types:

```ruby
class WorkflowInitWorkflow < Temporalio::Workflow::Definition
  workflow_init
  def initialize(input)
    @name_with_title = "Knight #{input['name']}"
  end

  def execute(input)
    Temporalio::Workflow.wait_condition { @title_has_been_checked }
    "Hello, #{@name_with_title}"
  end
end
```

## Workflow logic requirements 

Temporal Workflow Definitions [must be deterministic](/workflow-definition#deterministic-constraints), which includes
Ruby Workflows. This means there are several things Workflows cannot do such as:

- Perform IO (network, disk, stdio, etc)
- Access/alter external mutable state
- Do any threading
- Do anything using the system clock (for example, `Time.Now`)
- Make any random calls
- Make any not-guaranteed-deterministic calls

To prevent illegal Workflow calls, a call tracer is put on the Workflow thread that raises an exception if any illegal
calls are made.
Which calls are illegal is configurable in the Worker options.

The SDK provides replay-safe alternatives for common needs.

### Logging

Use [`Temporalio::Workflow.logger`](https://ruby.temporal.io/Temporalio/Workflow.html#logger-class_method) instead of
`puts` or a `Logger` you create yourself. The `Logger` class is on the default illegal call list, and the SDK logger
appends Workflow details to every log and skips logging during replay:

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute(name)
    Temporalio::Workflow.logger.info("Starting workflow for #{name}")
    # ...
  end
end
```

For logger configuration, see [Observability: Log from a Workflow](/develop/ruby/platform/observability#logging).

### Random numbers and UUIDs

Use [`Temporalio::Workflow.random`](https://ruby.temporal.io/Temporalio/Workflow.html#random-class_method) to get a
`Random` instance seeded per Workflow Execution. The SDK requires `random/formatter`, so this instance also has the
standard library's [`Random::Formatter#uuid`](https://rubyapi.org/4.0/o/random/formatter#method-i-uuid) method. Use it
instead of `SecureRandom.uuid`:

```ruby
value = Temporalio::Workflow.random.rand(1..100)
unique_id = Temporalio::Workflow.random.uuid
```

Don't use `SecureRandom`, `Kernel#rand`, `Kernel#srand`, or `Random.new` in Workflow code. They're on the default
illegal call list, and the call tracer raises a `Temporalio::Workflow::NondeterminismError` when it detects them.

Access the instance each time you need it rather than storing it in an instance variable. The SDK may recreate it with a
different seed, such as after a Workflow reset.

### Current time

Use [`Temporalio::Workflow.now`](https://ruby.temporal.io/Temporalio/Workflow.html#now-class_method) instead of
`Time.now`. It returns the UTC time of the last Workflow Task, which is consistent across replays:

```ruby
current_time = Temporalio::Workflow.now
```

To wait, use [`Temporalio::Workflow.sleep`](https://ruby.temporal.io/Temporalio/Workflow.html#sleep-class_method)
instead of `Kernel#sleep`.

### Detecting replay (advanced)

Use [`Temporalio::Workflow::Unsafe.replaying?`](https://ruby.temporal.io/Temporalio/Workflow/Unsafe.html#replaying?-class_method)
to guard code that should only run on the first execution, such as emitting metrics or sending external notifications
from an Interceptor.

> **⚠️ Caution:**
>
> Never use this to affect Workflow business logic. Branching on replay status breaks determinism.
>

```ruby
unless Temporalio::Workflow::Unsafe.replaying?
  emit_metric('workflow_started', 1)
end
```

If your goal is to always take action when something new is happening, check that
[`Temporalio::Workflow::Unsafe.replaying_history_events?`](https://ruby.temporal.io/Temporalio/Workflow/Unsafe.html#replaying_history_events?-class_method)
is false instead. That is false during read-only operations like Queries and Update validators. This is what the SDK's
built-in logger uses internally.

---

# Cancellation - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/cancellation

> Handle Cancellation in Ruby Workflows and Activities, request Cancellation from a Client, and choose between Cancellation, termination, and reset.

This page shows how to interrupt a Workflow Execution.

You can interrupt a Workflow Execution in one of the following ways:

- [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution.
- [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution.

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the
Workflow is stuck and cannot be canceled normally.

## Cancellation 

To give a Workflow and its Activities the ability to be cancelled, do the following:

- Handle a Cancellation request within a Workflow.
- Set Activity Heartbeat Timeouts.
- Listen for and handle a Cancellation request within an Activity.
- Send a Cancellation request from a Temporal Client.

## Handle Cancellation in Workflow 

Workflow Definitions can be written to respond to cancellation requests. It is common for an Activity to be run on
Cancellation to perform cleanup.

Cancellation Requests on Workflows cancel the `Temporalio::Workflow.cancellation` which is a `Temporalio::Cancellation`
that effectively serves as a cancellation token. This is the cancellation that is implicitly used for all calls within
the workflow as well (for example, Timers, Activities, etc) and therefore cancellation is propagated to them to be handled and
bubble out.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    # Whether this workflow waits on the activity to handle the cancellation or not is
    # dependent upon the cancellation_type parameter. We leave the default here which
    # sends the cancellation but does not wait on it to be handled.
    Temporalio::Workflow.execute_activity(MyActivity, start_to_close_timeout: 100)
  rescue Temporalio::Error => e
    # For this sample, we only want to execute cleanup when it's a cancellation
    raise unless Temporalio::Error.canceled?(e)

    # Call a cleanup activity. We have to do this with a new/detached cancellation
    # because the default workflow-level one is already canceled at this point.
    Temporalio::Workflow.execute_activity(
      MyCleanupActivity,
      start_to_close_timeout: 100,
      cancellation: Temporalio::Cancellation.new
    )

    # Re-raise the original exception
    raise
  end
end
```

## Handle Cancellation in an Activity 

Ensure that the Activity is [Heartbeating](/develop/ruby/activities/timeouts#activity-heartbeats) to receive the
Cancellation request and stop execution. Also make sure that the
[Heartbeat Timeout](/develop/ruby/activities/timeouts#heartbeat-timeout) is set on the Activity Options when calling from
the Workflow. An Activity Cancellation Request raises a `Temporalio::Error::CanceledError` in the Activity.

```ruby
class MyActivity < Temporalio::Activity::Definition
  def execute
    # This is a naive loop simulating work, but similar heartbeat/cancellation logic
    # applies to other scenarios as well
    loop do
      # Send heartbeat
      Temporalio::Activity::Context.current.heartbeat
      # Sleep before heartbeating again
      sleep(3)
    end
  rescue Temporalio::Error::CanceledError
    raise 'Canceled!'
  end
end
```

## Request Cancellation 

Use `cancel` on the `WorkflowHandle` to cancel a Workflow Execution.

```ruby
# Get a workflow handle by its workflow ID. This could be made specific to a run by
# passing run ID. This could also just be a handle that is returned from
# start_workflow instead.
handle = my_client.workflow_handle('my-workflow-id')

# Send cancellation. This returns when cancellation is received by the server. Wait on
# the handle's result to wait for cancellation to be applied.
handle.cancel
```

By default, Activities are automatically cancelled when the Workflow is cancelled since the workflow cancellation is
used by activities by default. To issue a cancellation explicitly, a new cancellation token can be created.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    # Create a new cancellation linked to the workflow one, so that it inherits
    # cancellation that comes from the workflow. Users can choose to make it
    # completely detached by not providing a parent.
    cancellation, cancel_proc = Temporalio::Cancellation.new(
      Temporalio::Workflow.cancellation
    )

    # Start the activity in the background. Whether this workflow waits on the activity
    # to handle the cancellation or not is dependent upon the cancellation_type
    # parameter. We leave the default here which sends the cancellation but does not wait
    # on it to be handled.
    future = Temporalio::Future.new do
      Temporalio::Workflow.execute_activity(
        MyActivity,
        start_to_close_timeout: 100,
        cancellation:
      )
    end

    # Wait 5 minutes, then cancel it
    Temporalio::Workflow.sleep(5 * 60)
    cancel_proc.call

    # Wait on the activity which will raise an activity error with a cause of
    # cancellation which will fail the workflow
    future.wait
  end
end
```

## Termination 

To Terminate a Workflow Execution in Ruby, use the `terminate` method on the Workflow handle.

```ruby
# Get a workflow handle by its workflow ID. This could be made specific to a run by
# passing run ID. This could also just be a handle that is returned from
# start_workflow instead.
handle = my_client.workflow_handle('my-workflow-id')

# Terminate
handle.terminate
```

Workflow Executions can also be Terminated directly from the WebUI. In this case, a custom note can be logged from the
UI when that happens.

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Child Workflows - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/child-workflows

> Start a Child Workflow Execution and set a Parent Close Policy using Temporal Ruby SDK.

This page shows how to do the following:

- [Start a Child Workflow Execution](#child-workflows) using the Ruby SDK
- [Set a Parent Close Policy](#parent-close-policy) using the Ruby SDK

## Start a Child Workflow Execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In Ruby, calling `start_child_workflow` or `execute_child_workflow` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call returns.

To spawn a Child Workflow Execution in Ruby, use the `execute_child_workflow` method which starts the Child Workflow and waits for completion or
use the `start_child_workflow` method to start a Child Workflow and return its handle.
This is useful if you want to do something after it has only started, or to get the Workflow/Run ID, or to be able to signal it while running.

> **📝 Note:**
>
> `execute_child_workflow` is a helper method for `start_child_workflow(...).result`.
>

```ruby
Temporalio::Workflow.execute_child_workflow(MyChildWorkflow, 'my-workflow-arg')
```

## Set a Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

Set the `parent_close_policy` parameter for `execute_child_workflow` or `start_child_workflow` to specify the behavior of the Child Workflow when the Parent Workflow closes.

```ruby
Temporalio::Workflow.execute_child_workflow(
  MyChildWorkflow,
  'my-workflow-arg',
  parent_close_policy: Temporalio::Workflow::ParentClosePolicy::ABANDON
)
```

---

# Continue-As-New - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/continue-as-new

> Use Continue-As-New in Ruby to close a Workflow Execution and start a new one with the same Workflow Id and a fresh Event History.

This page describes how to Continue-As-New using the Temporal Ruby SDK.

[Continue-As-New](/workflow-execution/continue-as-new) enables a Workflow Execution to close successfully and create a new Workflow Execution in a single atomic operation if the number of Events in the Event History is becoming too large.
The Workflow Execution spawned from the use of Continue-As-New has the same Workflow Id, a new Run Id, and a fresh Event History and is passed all the appropriate parameters.

> **⚠️ Caution:**
>
> As a precautionary measure, the Workflow Execution's Event History is limited to [51,200 Events](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L441) or [50 MB](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L425) and will warn you after 10,240 Events or 10 MB.
>

To prevent a Workflow Execution Event History from exceeding this limit and failing, use Continue-As-New to start a new Workflow Execution with a fresh Event History.

A very large Event History can adversely affect the performance of a Workflow Execution.
For example, in the case of a Workflow Worker failure, the full Event History must be pulled from the Temporal Service and given to another Worker via a Workflow Task.
If the Event history is very large, it may take some time to load it.

The Continue-As-New feature enables developers to complete the current Workflow Execution and start a new one atomically.

The new Workflow Execution has the same Workflow Id, but a different Run Id, and has its own Event History.

## Continue-As-New in Ruby 

To Continue-As-New in Ruby, raise a `Temporalio::Workflow::ContinueAsNewError` from inside your Workflow, which will stop the Workflow immediately and Continue-As-New.

```ruby
raise Temporalio::Workflow::ContinueAsNewError.new('my-new-arg')
```

> **⚠️ Warning:**
> Using Continue-as-New and Updates
>
> - Temporal _does not_ support Continue-as-New functionality within Update handlers.
> - Complete all handlers _before_ using Continue-as-New.
> - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution.
>

---

# Dynamic Workflow - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/dynamic-workflow

> This section explains Dynamic Workflows with the Ruby SDK

## Set a Dynamic Workflow 

A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered.
A Workflow can be made dynamic by invoking `workflow_dynamic` class method at the top of the definition.
You must register the Workflow with the Worker before it can be invoked.
Only one Dynamic Workflow can be present on a Worker.

Often, dynamic is used in conjunction with `workflow_raw_args` which does not convert arguments but instead passes them
through as a splatted array of `Temporalio::Converters::RawValue` instances.

```ruby
class MyDynamicWorkflow < Temporalio::Workflow::Definition
  # Make this the dynamic workflow and accept raw args
  workflow_dynamic
  workflow_raw_args

  def execute(*raw_args)
    # Require a single arg for our workflow
    raise Temporalio::Error::ApplicationError, 'One arg expected' unless raw_args.size == 1

    # Use payload converter to convert it
    name = Temporalio::Workflow.payload_converter.from_payload(raw_args.first.payload)
    Temporalio::Workflow.execute_activity(
      MyActivity,
      { greeting: 'Hello', name: },
      start_to_close_timeout: 100
    )
  end
end
```

---

# Workflow futures - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/futures

> This section explains Workflow futures with the Ruby SDK

## Workflow Futures 

`Temporalio::Workflow::Future` can be used for running things in the background or concurrently.
Temporal provides Workflow-safe wrappers around some core language features in cases like these.
`Temporalio::Workflow::Future` is a safe wrapper around `Fiber.schedule` for running multiple Activities at once.
The Ruby SDK also provides `Workflow.wait_condition` for awaiting a result.

Futures are never used implicitly, but they work with all Workflow code and constructs.
For instance, to run 3 activities and wait for them all to complete, something like this can be written:

```ruby
# Start 3 activities in background
fut1 = Temporalio::Workflow::Future.new do
  Temporalio::Workflow.execute_activity(MyActivity1, schedule_to_close_timeout: 300)
end
fut2 = Temporalio::Workflow::Future.new do
  Temporalio::Workflow.execute_activity(MyActivity2, schedule_to_close_timeout: 300)
end
fut3 = Temporalio::Workflow::Future.new do
  Temporalio::Workflow.execute_activity(MyActivity3, schedule_to_close_timeout: 300)
end

# Wait for them all to complete
Temporalio::Workflow::Future.all_of(fut1, fut2, fut3).wait

Temporalio::Workflow.logger.info("Got: #{fut1.result}, #{fut2.result}, #{fut3.result}")
```

Or, say, to wait on the first of 5 activities or a timeout to complete:

```ruby
# Start 5 activities
act_futs = 5.times.map do |i|
  Temporalio::Workflow::Future.new do
    Temporalio::Workflow.execute_activity(MyActivity, "my-arg-#{i}", schedule_to_close_timeout: 300)
  end
end
# Start a timer
sleep_fut = Temporalio::Workflow::Future.new { Temporalio::Workflow.sleep(30) }

# Wait for first act result or sleep fut
act_result = Temporalio::Workflow::Future.any_of(sleep_fut, *act_futs).wait
# Fail if timer done first
raise Temporalio::Error::ApplicationError, 'Timer expired' if sleep_fut.done?
# Print act result otherwise
Temporalio::Workflow.logger.info("Act result: #{act_result}")
```

There are several other details not covered here about futures, such as how exceptions are handled and how to use a setter
proc instead of a block. See the [API documentation](https://ruby.temporal.io/Temporalio/Workflow/Future.html) for details.

---

# Workflow message passing - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/message-passing

> Develop with Queries, Signals, and Updates using the Temporal Ruby SDK.

A Workflow can act like a stateful service that receives messages: Queries, Signals, and Updates.
These messages interact with the Workflow via handler methods defined in the Workflow code.
Clients use messages to read Workflow state or change its behavior.

See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview.

## Write message handlers 

> **ℹ️ Info:**
> The code that follows is part of a [working solution](https://github.com/temporalio/samples-ruby/tree/main/message_passing_simple).

Follow these guidelines when writing your message handlers:

- Message handlers are defined as methods on the Workflow class, decorated by calling one of three class methods before defining the handler method: `workflow_query`, `workflow_signal`, and `workflow_update`.
- These also implicitly create class-methods with the same name as the instance methods for use by callers.
- The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion).
- Prefer single hash/object input parameter to multiple input parameters.
  Hash/object parameters allow you to add fields without changing the calling signature.

### Query handlers 

A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution.
Define as a method:

```ruby
class GreetingWorkflow < Temporalio::Workflow::Definition
  # ...

  workflow_query
  def languages(input)
    # A query handler returns a value: it can inspect but must not mutate the Workflow state.
    if input['include_unsupported']
      CallGreetingService.greetings.keys.sort
    else
      @greetings.keys.sort
    end
  end

  # ...
end
```

Or as an attribute reader:

```ruby
class GreetingWorkflow < Temporalio::Workflow::Definition
  # This is the equivalent of:
  #    workflow_query
  #    def language
  #      @language
  #    end
  workflow_query_attr_reader :language

  # ...
end
```

- The `workflow_query` class method can accept arguments.
  See the API reference docs: [`workflow_query`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_query-class_method).
- A Query handler must not modify Workflow state.
- You can't perform async blocking operations such as executing an Activity in a Query handler.

### Signal handlers 

A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow:

```ruby
class GreetingWorkflow < Temporalio::Workflow::Definition
  # ...

  workflow_signal
  def approve(input)
    # A signal handler mutates the workflow state but cannot return a value.
    @approved_for_release = true
    @approver_name = input['name']
  end

  # ...
end
```

- The `workflow_signal` class method can accept arguments.
  Refer to the API docs: [`workflow_signal`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_signal-class_method).

- The handler should not return a value.
  The response is sent immediately from the server, without waiting for the Workflow to process the Signal.

- Signal (and Update) handlers can be asynchronous and blocking.
  This allows you to use Activities, Child Workflows, durable Timers, wait conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Signal and Update handlers.

### Update handlers and validators 

An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution.
It can change the Workflow state, control its flow, and return a result.
The sender must wait until the Worker accepts or rejects the Update.
The sender may wait further to receive a returned value or an exception if something goes wrong:

```ruby
class GreetingWorkflow < Temporalio::Workflow::Definition
  # ...

  workflow_update
  def set_language(new_language) # rubocop:disable Naming/AccessorMethodName
    # An update handler can mutate the workflow state and return a value.
    prev = @language.to_sym
    @language = new_language.to_sym
    prev
  end

  workflow_update_validator(:set_language)
  def validate_set_language(new_language)
    # In an update validator you raise any exception to reject the update.
    raise "#{new_language} is not supported" unless @greetings.include?(new_language.to_sym)
  end

  # ...
end
```

- The `workflow_update` class method can take arguments as described in the API reference docs for [`workflow_update`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_update-class_method).

- About validators:
  - Use validators to reject an Update before it is written to History.
    Validators are always optional.
    If you don't need to reject Updates, you can skip them.
  - Define an Update validator with the [`workflow_update_validator`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_update-class_method) class method invoked before defining the method.
    The first parameter when declaring the validator is the name of the Update handler method.
    The validator must accept the same argument types as the handler and should not return a value.

- Accepting and rejecting Updates with validators:
  - To reject an Update, raise an exception of any type in the validator.
  - Without a validator, Updates are always accepted.
- Validators and Event History:
  - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic.
  - When a Validator raises an error, the Update is rejected, the Update is not run, and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History.
    The caller receives an "Update failed" error.

- Use [`current_update_info`](https://ruby.temporal.io/Temporalio/Workflow.html#current_update_info-class_method) to obtain information about the current Update.
  This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
- Update (and Signal) handlers can be asynchronous and blocking.
  This allows you to use Activities, Child Workflows, durable Timers, wait conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Update and Signal handlers.

## Send messages 

To send Queries, Signals, or Updates you call methods on a [`WorkflowHandle`](https://ruby.temporal.io/Temporalio/Client/WorkflowHandle.html) instance.
To obtain the Workflow handle, you can:

- Use [`Client#start_workflow`](https://ruby.temporal.io/Temporalio/Client.html#start_workflow-instance_method) to start a Workflow and return its handle.
- Use the [`Client#workflow_handle`](https://ruby.temporal.io/Temporalio/Client.html#workflow_handle-instance_method) method to retrieve a Workflow handle by its Workflow Id.

For example:

```ruby
client = Temporalio::Client.connect('localhost:7233', 'default')
handle = client.start_workflow(
  MessagePassingSimple::GreetingWorkflow,
  id: 'message-passing-simple-sample-workflow-id',
  task_queue: 'message-passing-simple-sample'
)
```

To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition.

> **⚠️ Warning:**
> Using Continue-as-New and Updates
>
> - Temporal _does not_ support Continue-as-New functionality within Update handlers.
> - Complete all handlers _before_ using Continue-as-New.
> - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution.
>

### Send a Query 

Call a Query method with [`WorkflowHandle#query`](https://ruby.temporal.io/Temporalio/Client/WorkflowHandle.html#query-instance_method):

```ruby
supported_languages = handle.query(MessagePassingSimple::GreetingWorkflow.languages, { include_unsupported: false })
```

- Sending a Query doesn’t add events to a Workflow's Event History.

- You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period.
  This includes Workflows that have completed, failed, or timed out.
  Querying terminated Workflows is not safe and, therefore, not supported.

- A Worker must be online and polling the Task Queue to process a Query.

### Send a Signal 

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution.
However, you can only send Signals to Workflow Executions that haven’t closed.

#### From a Client 

Use [`WorkflowHandle#signal`](https://ruby.temporal.io/Temporalio/Client/WorkflowHandle.html#signal-instance_method) from Client code to send a Signal:

```ruby
handle.signal(MessagePassingSimple::GreetingWorkflow.approve, { name: 'John Q. Approver' })
```

- The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution.

- The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History.

#### From a Workflow 

A Workflow can send a Signal to another Workflow, known as an _External Signal_.
In this case you need to obtain a Workflow handle for the external Workflow.
Use `Temporalio::Workflow.external_workflow_handle`, passing a running Workflow Id, to retrieve a Workflow handle:

```ruby
class WorkflowB < Temporalio::Workflow::Definition
  def execute
    handle = Temporalio::Workflow.external_workflow_handle('workflow-a-id')
    handle.signal(WorkflowA.some_signal, 'some signal arg')
  end
end
```

When an External Signal is sent:

- A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History.
- A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History.

#### Signal-With-Start 

Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running.
If there's a Workflow running with the given Workflow Id, it will be signaled.
If there isn't, a new Workflow will be started and immediately signaled.

To use Signal-With-Start, call `signal_with_start_workflow` with a `WithStartWorkflowOperation`:

```ruby
client = Temporalio::Client.connect('localhost:7233', 'default')

# Create start-workflow operation for use with signal-with-start
start_workflow_operation = Temporalio::Client::WithStartWorkflowOperation.new(
  MyWorkflow, 'my-workflow-input',
  id: 'my-workflow-id', task_queue: 'my-workflow-task-queue'
)
# Perform signal-with-start
handle = client.signal_with_start_workflow(
  MyWorkflow.my_signal, 'signal-input', start_workflow_operation:
)
```

### Send an Update 

An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result.

A Client sending an Update must wait until the Server delivers the Update to a Worker.
Workers must be available and responsive.
If you need a response as soon as the Server receives the request, use a Signal instead.
You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity.

- `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation.
- `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished.

To send an Update to a Workflow Execution, you can:

- Call the Update method with `execute_update` from the Workflow handle and wait for the Update to complete.
  This code fetches an Update result:

  ```ruby
  prev_language = handle.execute_update(MessagePassingSimple::GreetingWorkflow.set_language, :chinese)
  ```

2. Use `start_update` to receive a handle as soon as the Update is accepted.
     It returns a `WorkflowUpdateHandle`

  - Use this `WorkflowUpdateHandle` later to fetch your results.
  - Asynchronous Update handlers normally perform long-running async Activities.
  - `start_update` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete.

  For example:

  ```ruby
  # Start an update and then wait for it to complete
  update_handle = handle.start_update(
    MessagePassingSimple::GreetingWorkflow.apply_language_with_lookup,
    :arabic,
    wait_for_stage: Temporalio::Client::WorkflowUpdateWaitStage::ACCEPTED
  )
  prev_language = update_handle.result
  ```

  For more details, see the "Async handlers" section.

#### Update-With-Start 

> **💡 Tip:**
> Stability
>
> In [Public Preview](/evaluate/product-release-stages#public-preview) in Temporal Cloud.
>
> Minimum Temporal Server version [Temporal Server version 1.26](https://github.com/temporalio/temporal/releases/tag/v1.26.2)
>

[Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](/develop/ruby/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists:

- If the Workflow exists, the Update is processed.
- If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute.

Use `execute_update_with_start_workflow` to start the Update and wait for the result in one go.

Alternatively, use `start_update_with_start_workflow` to start the Update and receive a `WorkflowUpdateHandle`, and then use `update_handle.result` to retrieve the result from the Update.

These calls return once the requested Update wait stage has been reached, or when the request times out.

- You will need to provide a `WithStartWorkflowOperation` to define the Workflow that will be started if necessary, and its arguments.
- You must specify an [id_conflict_policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) when creating the `WithStartWorkflowOperation`.
  Note that a `WithStartWorkflowOperation` can only be used once.

Here's an example:

```ruby
client = Temporalio::Client.connect('localhost:7233', 'default')

# Create start-workflow operation for use with update-with-start
start_workflow_operation = Temporalio::Client::WithStartWorkflowOperation.new(
  MyWorkflow, 'my-workflow-input',
  id: 'my-workflow-id', task_queue: 'my-workflow-task-queue',
  id_conflict_policy: Temporalio::WorkflowIDConflictPolicy::USE_EXISTING
)
# Perform update-with-start and get update result
update_result = client.execute_with_start_workflow(
  MyWorkflow.my_update, 'update-input', start_workflow_operation:
)
# The workflow handle is on the start operation, here's an example of waiting on
# workflow result
workflow_result = start_workflow_operation.workflow_handle.result
```

## Message handler patterns 

This section covers common write operations, such as Signal and Update handlers.
It doesn't apply to pure read operations, like Queries or Update Validators.

> **💡 Tip:**
>
> For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow) and [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
>

### Add async handlers 

Signal and Update handlers can be asynchronous as well as blocking.
Using asynchronous calls allows you to wait for Activities, Child Workflows, Durable Timers, wait conditions, and so on.
This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls.

To use asynchronous handlers safely, understand what can go wrong.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below.

The following code is an Activity that simulates a network call to a remote service:

```ruby
class CallGreetingService < Temporalio::Activity::Definition
  def execute(to_language)
    # Simulate a network call
    sleep(0.2)
    # This intentionally returns nil on not found
    CallGreetingService.greetings[to_language.to_sym]
  end

  def self.greetings
    @greetings ||= {
      arabic: 'مرحبا بالعالم',
      chinese: '你好，世界',
      english: 'Hello, world',
      french: 'Bonjour, monde',
      hindi: 'नमस्ते दुनिया',
      portuguese: 'Olá mundo',
      spanish: 'Hola mundo'
    }
  end
end
```

The following code is a Workflow Update for asynchronous use of the preceding Activity:

```ruby
class GreetingWorkflow < Temporalio::Workflow::Definition
  # ...

  workflow_update
  def apply_language_with_lookup(new_language)
    # Call an activity if it's not there.
    unless @greetings.include?(new_language.to_sym)
      # We use a mutex so that, if this handler is executed multiple times, each execution
      # can schedule the activity only when the previously scheduled activity has
      # completed. This ensures that multiple calls to apply_language_with_lookup are
      # processed in order.
      @apply_language_mutex ||= Mutex.new
      @apply_language_mutex.synchronize do
        greeting = Temporalio::Workflow.execute_activity(
          CallGreetingService, new_language, start_to_close_timeout: 10
        )
        # The requested language might not be supported by the remote service. If so, we
        # raise ApplicationError, which will fail the update. The
        # WorkflowExecutionUpdateAccepted event will still be added to history. (Update
        # validators can be used to reject updates before any event is written to history,
        # but they cannot be async, and so we cannot use an update validator for this
        # purpose.)
        raise Temporalio::Error::ApplicationError, "Greeting service does not support #{new_language}" unless greeting

        @greetings[new_language.to_sym] = greeting
      end
    end
    set_language(new_language)
  end
end
```

After updating the code for asynchronous calls, your Update handler can schedule an Activity and await the result.
Although an async Signal handler can initiate similar network tasks, using an Update handler allows the Client to receive a result or error once the Activity completes.
This lets your Client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, and so on.

### Use wait conditions 

Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue.
Using a wait condition with [`wait_condition`](https://ruby.temporal.io/Temporalio/Workflow.html#wait_condition-class_method) sets a function that prevents the code from proceeding until the condition is truthy.
This is an important feature that helps you control your handler logic.

Here are two important use cases for `wait_condition`:

- Waiting in a handler until it is appropriate to continue.
- Waiting in the main Workflow until all active handlers have finished.

The condition state you're waiting for can be updated by and reflect any part of the Workflow code.
This includes the main Workflow method, other handlers, or child coroutines spawned by the main Workflow method, and so forth.

#### In handlers 

Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue.
Using a wait condition with [`wait_condition`](https://ruby.temporal.io/Temporalio/Workflow.html#wait_condition-class_method) sets a function that prevents the code from proceeding until the condition is truthy.
This is an important feature that helps you control your handler logic.

Consider a `ready_for_update_to_execute` method that runs before your Update handler executes.
The `wait_condition` call waits until your condition is met:

```ruby
workflow_update
def my_update(my_update_input)
  Temporalio::Workflow.wait_condition { ready_for_update_to_execute(my_update_input) }
  # ...
end
```

Remember: Handlers can execute before the main Workflow method starts.

#### Before finishing the Workflow 

Workflow wait conditions can ensure your handler completes before a Workflow finishes.
When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result.
The Workflow completing may interrupt the handler before it finishes crucial work and cause Client errors when trying retrieve Update results.
Use `Temporalio::Workflow.all_handlers_finished?` to address this problem and allow your Workflow to end smoothly:

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    # ...

    Temporalio::Workflow.wait_condition { Temporalio::Workflow.all_handlers_finished? }
    'workflow-result'
  end
end
```

By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions.
You can silence these warnings on a per-handler basis by passing the `unfinished_policy` argument to the [`workflow_signal`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_signal-class_method) / [`workflow_update`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_update-class_method) class methods:

```ruby
workflow_update unfinished_policy: Temporalio::Workflow::HandlerUnfinishedPolicy::ABANDON
def my_update
  # ...
```

See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information.

### Use workflow_init to access input early

The `workflow_init` class method above `initialize` gives it access to [Workflow input](/handling-messages#workflow-initializers).
When you use the `workflow_init` on your constructor, you give the constructor the same Workflow parameters as your `execute` method.
The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/ruby/client/temporal-client#start-workflow).
The Workflow input arguments are also passed to your `execute` method -- that always happens, whether or not you use the `workflow_init` class method above `initialize`.

Here's an example.
The constructor and `execute` must have the same parameters with the same types:

```ruby
class WorkflowInitWorkflow < Temporalio::Workflow::Definition
  workflow_init
  def initialize(input)
    @name_with_title = "Sir #{input['name']}"
  end

  def execute(input)
    Temporalio::Workflow.wait_condition { @title_has_been_checked }
    "Hello, #{@name_with_title}"
  end

  workflow_update
  def check_title_validity
    # The handler is now guaranteed to see some workflow input since it was
    # processed by the constructor
    valid = Temporalio::Workflow.execute_activity(
      CheckTitleValidityActivity,
      @name_with_title,
      start_to_close_timeout: 100
    )
    @title_has_been_checked = true
    valid
  end
end
```

### Use locks to prevent concurrent handler execution 

Concurrent processes can interact in unpredictable ways.
Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously.
Here's an example of a pathological case:

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  # ...

  workflow_signal
  def bad_handler
    data = Temporalio::Workflow.execute_activity(
      FetchDataActivity,
      start_to_close_timeout: 100
    )
    @x = data['x']
    # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then
    # there may be times when the Workflow has @x from one Activity execution and @y
    # from another.
    Temporalio::Workflow.sleep(1)
    @y = data['y']
  end
end
```

Coordinating access with `Mutex`, a mutual exclusion lock, corrects this code.
Locking makes sure that only one handler instance can execute a specific section of code at any given time:

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  # ...

  workflow_signal
  def safe_handler
    @mutex ||= Mutex.new
    @mutex.synchronize do
      data = Temporalio::Workflow.execute_activity(
        FetchDataActivity,
        start_to_close_timeout: 100
      )
      @x = data['x']
      # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then
      # there may be times when the Workflow has @x from one Activity execution and @y
      # from another.
      Temporalio::Workflow.sleep(1)
      @y = data['y']
    end
  end
end
```

For additional concurrency options, `wait_condition` can be used to do more advanced things such as using an integer
attribute + `wait_condition` as a semaphore.

## Troubleshooting 

When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors:

- **The Client can't contact the server**:
  You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception whose `code` is an `UNAVAILABLE` constant defined in [`Code`](https://ruby.temporal.io/Temporalio/Error/RPCError/Code.html) (after some retries).

- **The Workflow does not exist**:
  You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception whose `code` is a `NOT_FOUND` constant defined in [`Code`](https://ruby.temporal.io/Temporalio/Error/RPCError/Code.html).

See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Ruby-specific discussion of this topic.

### Signal issues 

When using Signal, the only exception that will result from your requests during its execution is `RPCError`.
All handlers may experience additional exceptions during the initial (pre-Worker) part of a handler request lifecycle.

For Queries and Updates, the Client waits for a response from the Worker.
If an issue occurs during the handler Execution by the Worker, the Client may receive an exception.

### Update issues 

When working with Updates, you may encounter these errors:

- **No Workflow Workers are polling the Task Queue**:
  Your request will be retried by the SDK Client indefinitely.
  Use a `Cancellation` in your [RPC options](https://ruby.temporal.io/Temporalio/Client/RPCOptions.html) to cancel the Update.
  This raises a [WorkflowUpdateRPCTimeoutOrCanceledError](https://ruby.temporal.io/Temporalio/Error/WorkflowUpdateRPCTimeoutOrCanceledError.html) exception.

- **Update failed**: You'll receive a [`WorkflowUpdateFailedError`](https://ruby.temporal.io/Temporalio/Error/WorkflowUpdateFailedError.html) exception.
  There are two ways this can happen:

  - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler.

  - The Update failed after having been accepted.

  Update failures are like [Workflow failures](/references/failures).
  Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.
  These might include:

      - A failed Child Workflow
      - A failed Activity (if the Activity retries have been set to a finite number)
      - The Workflow author raising `ApplicationError`
      - Any error listed in `workflow_failure_exception_types` on the Worker or [`workflow_failure_exception_type`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_failure_exception_type-class_method) on the Workflow (empty by default)

- **The handler caused the Workflow Task to fail**:
  A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage:
  - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception.
  - If the request has been accepted, it is durable.
    Once the Workflow is healthy again after a code deploy, use an [`WorkflowUpdateHandle`](https://ruby.temporal.io/Temporalio/Client/WorkflowUpdateHandle.html) to fetch the Update result.

- **The Workflow finished while the Update handler execution was in progress**:
  You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) "workflow execution already completed".

  This will happen if the Workflow finished while the Update handler execution was in progress, for example because

  - The Workflow was canceled or failed.

  - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers).

### Query issues 

When working with Queries, you may encounter these errors:

- **There is no Workflow Worker polling the Task Queue**:
  You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception whose `code` is a `FAILED_PRECONDITION` constant defined in [`Code`](https://ruby.temporal.io/Temporalio/Error/RPCError/Code.html).

- **Query failed**:
  You'll receive a [`WorkflowQueryFailedError`](https://ruby.temporal.io/Temporalio/Error/WorkflowQueryFailedError.html) exception if something goes wrong during a Query.
  Any exception in a Query handler will trigger this error.
  This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead.

- **The handler caused the Workflow Task to fail.**
  This would happen, for example, if the Query handler blocks the thread for too long without yielding.

## Dynamic handlers 

Temporal supports Dynamic Queries, Signals, Updates, Workflows, and Activities.
These are unnamed handlers that are invoked if no other statically defined handler with the given name exists.

Dynamic Handlers provide flexibility to handle cases where the names of Queries, Signals, Updates, Workflows, or Activities, aren't known at run time.

> **⚠️ Caution:**
>
> Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach.
> Overusing them can lead to maintainability and debugging issues down the line.
>
> Instead, Signals, Queries, Workflows, or Activities should be defined statically whenever possible, with clear names that indicate their purpose.
> Use static definitions as the primary way of structuring your Workflows.
>
> Reserve Dynamic Handlers for cases where the handler names are not known at compile time and need to be looked up dynamically at runtime.
> They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic.
>

### Dynamic Query 

A Dynamic Query in Temporal is a Query method that is invoked dynamically at runtime if no other Query with the same name is registered.
A Query can be made dynamic by setting `dynamic` to `true` on the `workflow_query` class method.
Only one Dynamic Query can be present on a Workflow.

The Query Handler parameters must accept a string name as the first parameter. Often users set `raw_args` to `true` and set the second parameter as `*args` which will be an array of `Temporalio::Converters::RawValue`.
The [Temporalio::Workflow.payload_converter](https://ruby.temporal.io/Temporalio/Workflow.html#payload_converter-class_method) property is used to convert the raw value instances to proper types.

```ruby
workflow_query dynamic: true, raw_args: true
def dynamic_query(query_name, *args)
  first_param = Temporalio::Workflow.payload_converter.from_payload(
    args.first || raise 'Missing first parameter'
  )
  "Got parameter #{first_param} for query #{query_name}"
end
```

### Dynamic Signal 

A Dynamic Signal in Temporal is a Signal that is invoked dynamically at runtime if no other Signal with the same input is registered.
A Signal can be made dynamic by setting `dynamic` to `true` on the `workflow_signal` class method.
Only one Dynamic Signal can be present on a Workflow.

The Signal Handler parameters must accept a string name as the first parameter. Often users set `raw_args` to `true` and set the second parameter as `*args` which will be an array of `Temporalio::Converters::RawValue`.
The [Temporalio::Workflow.payload_converter](https://ruby.temporal.io/Temporalio/Workflow.html#payload_converter-class_method) property is used to convert the raw value instances to proper types.

```ruby
workflow_signal dynamic: true, raw_args: true
def dynamic_signal(signal_name, *args)
  first_param = Temporalio::Workflow.payload_converter.from_payload(
    args.first || raise 'Missing first parameter'
  )
  @pending_things << "Got parameter #{first_param} for signal #{signal_name}"
end
```

### Dynamic Update 

A Dynamic Update in Temporal is an Update that is invoked dynamically at runtime if no other Update with the same input is registered.
An Update can be made dynamic by setting `dynamic` to `true` on the `workflow_update` class method.
Only one Dynamic Update can be present on a Workflow.

The Update Handler parameters must accept a string name as the first parameter. Often users set `raw_args` to `true` and set the second parameter as `*args` which will be an array of `Temporalio::Converters::RawValue`.
The [Temporalio::Workflow.payload_converter](https://ruby.temporal.io/Temporalio/Workflow.html#payload_converter-class_method) property is used to convert the raw value instances to proper types.

```ruby
workflow_update dynamic: true, raw_args: true
def dynamic_update(update_name, *args)
  first_param = Temporalio::Workflow.payload_converter.from_payload(
    args.first || raise 'Missing first parameter'
  )
  @pending_things << "Got parameter #{first_param} for update #{update_name}"
end
```

---

# Schedules - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/schedules

> Manage and optimize Scheduled Workflows using the Temporal Ruby SDK; Schedule, Create, Backfill, Update, Delete, Describe, List, Pause, Trigger, and more.

This page shows how to do the following:

- [Schedule a Workflow](#schedule-a-workflow)
  - [Create a Scheduled Workflow](#create-a-workflow)
  - [Backfill a Scheduled Workflow](#backfill-a-scheduled-workflow)
  - [Delete a Scheduled Workflow](#delete-a-scheduled-workflow)
  - [Describe a Scheduled Workflow](#describe-a-scheduled-workflow)
  - [List a Scheduled Workflow](#list-a-scheduled-workflow)
  - [Pause a Scheduled Workflow](#pause-a-scheduled-workflow)
  - [Trigger a Scheduled Workflow](#trigger-a-scheduled-workflow)
  - [Update a Scheduled Workflow](#update-a-scheduled-workflow)
- [Use Start Delay](#start-delay)

## Schedule a Workflow 

Scheduling Workflows is a crucial aspect of automation.
By scheduling a Workflow, you can automate repetitive tasks, reduce manual intervention, and ensure timely execution.

Use the following actions to manage Scheduled Workflows.

Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy).
If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions.

### Create a Scheduled Workflow 

The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands.

To create a Scheduled Workflow Execution in Ruby, use the [create_schedule](https://ruby.temporal.io/Temporalio/Client.html#create_schedule-instance_method)
method on the Client.
Then pass the Schedule ID and the Schedule object to the method to create a Scheduled Workflow Execution.
Set the Schedule's `action` member to an instance of `Temporalio::Client::Schedule::Action::StartWorkflow` to schedule a Workflow Execution.

```ruby
handle = my_client.create_schedule(
  'my_schedule_id',
  Temporalio::Client::Schedule.new(
    action: Temporalio::Client::Schedule::Action::StartWorkflow.new(
      MyWorkflow, 'some-input',
      id: 'my-workflow-id', task_queue: 'my-task-queue'
    ),
    spec: Temporalio::Client::Schedule::Spec.new(
      intervals: [
        Temporalio::Client::Schedule::Spec::Interval.new(
          every: 5 * 24 * 60 * 60.0, # 5 days
        )
      ]
    )
  )
)
```

> **💡 Tip:**
> Schedule Auto-Deletion
>
> Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again.
> The Temporal Service doesn't guarantee when this removal will happen.
>

### Backfill a Scheduled Workflow 

The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time.

To backfill a Scheduled Workflow Execution in Ruby, use the [backfill](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#backfill-instance_method)
method on the Schedule Handle.

```ruby
handle = my_client.schedule_handle('my-schedule-id')
now = Time.now(in: 'UTC')
handle.backfill(
  Temporalio::Client::Schedule::Backfill.new(
    start_at: now - (4 * 60),
    end_at: now - (2 * 60),
    overlap: Temporalio::Client::Schedule::OverlapPolicy::ALLOW_ALL
  )
)
```

### Delete a Scheduled Workflow 

The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule.

To delete a Scheduled Workflow Execution in Ruby, use the [delete](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#delete-instance_method) method on the Schedule Handle.

```ruby
handle = my_client.schedule_handle('my-schedule-id')
handle.delete
```

### Describe a Scheduled Workflow 

The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs.

To describe a Scheduled Workflow Execution in Ruby, use the [describe](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#describe-instance_method) method on the Schedule Handle.

```ruby
handle = my_client.schedule_handle('my-schedule-id')
desc = handle.describe
puts "Schedule info: #{desc.info}"
```

### List a Scheduled Workflow 

The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs.

To list all schedules, use the [list_schedules](https://ruby.temporal.io/Temporalio/Client.html#list_schedules-instance_method) asynchronous method on the Client.
This returns an enumerator/enumerable.
If a schedule is added or deleted, it may not be available in the list immediately.

```ruby
my_client.list_schedules.each do |sched|
  puts "Schedule info: #{sched}"
end
```

### Pause a Scheduled Workflow 

The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason.

To pause a Scheduled Workflow Execution in Ruby, use the [pause](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#pause-instance_method) method on the Schedule Handle.
You can pass a note to the `pause` method to provide a reason for pausing the schedule.

```ruby
handle = my_client.schedule_handle('my-schedule-id')
handle.pause(note: 'Pausing the schedule for now')
```

### Trigger a Scheduled Workflow 

The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time.

To trigger a Scheduled Workflow Execution in Ruby, use the [trigger](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#trigger-instance_method) method on the Schedule Handle.

```ruby
handle = my_client.schedule_handle('my-schedule-id')
handle.trigger
```

### Update a Scheduled Workflow 

The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval.

To update a Scheduled Workflow Execution in Ruby, use the [update](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#update-instance_method) method on the Schedule Handle.
This method accepts a block which itself accepts an update input object and is expected to return an update with a new
schedule to update, or `nil` to not update.

```ruby
handle = my_client.schedule_handle('my-schedule-id')
handle.update do |input|
  # Return a new schedule with the action updated
  Temporalio::Client::Schedule::Update.new(
    schedule: input.description.schedule.with(
      # Update the action
      action: Temporalio::Client::Schedule::Action::StartWorkflow.new(
        MyNewWorkflow, 'some-new-input',
        id: 'my-workflow-id', task_queue: 'my-task-queue'
      )
    )
  )
end
```

## Use Start Delay 

Use the `start_delay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule.

Use the `start_delay` parameter on either the `start_workflow` or `execute_workflow` methods in the Client.

```ruby
handle = my_client.start_workflow(
  MyWorkflow, 'some-input',
  id: 'my-workflow-id', task_queue: 'my-task-queue',
  start_delay: 3 * 60 * 60 # 3 hours
)
```

---

# Workflow Timeouts - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/timeouts

> Optimize Workflow Execution with Temporal Ruby SDK - Set Workflow Timeouts and Retry Policies efficiently.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)**: Limits how long the full Workflow Execution can run.
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout)**: Limits the duration of an individual run of a Workflow Execution.
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout)**: Limits the time allowed for a Worker to process a Workflow Task.

Set these values as keyword parameter options when starting a Workflow.

```ruby
result = my_client.execute_workflow(
  MyWorkflow, 'some-input',
  id: 'my-workflow-id', task_queue: 'my-task-queue',
  execution_timeout: 5 * 60
)
```

## Workflow retries 

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a [Retry Policy](/encyclopedia/retry-policies) to automatically retry Workflow Executions on failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

The `retry_policy` can be set when calling `start_workflow` or `execute_workflow`.

```ruby
result = my_client.execute_workflow(
  MyWorkflow, 'some-input',
  id: 'my-workflow-id', task_queue: 'my-task-queue',
  retry_policy: Temporalio::RetryPolicy.new(max_interval: 10)
)
```

---

# Timers - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/timers

This page describes how to set a Durable Timer using the Temporal Ruby SDK.

A [Durable Timer](/workflow-execution/timers-delays) is used to pause the execution of a Workflow for a specified
duration. A Workflow can sleep for days or even months. Timers are persisted, so even if your Worker or Temporal Service
is down when the time period completes, as soon as your Worker and Temporal Service are back up, the Durable Timer call
will resolve and your code will continue executing.

Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single
Worker.

To add a Timer in a Workflow, use `Temporalio::Workflow.sleep`. _Technically_ `Kernel#sleep` works, but the workflow
form allows one to set a summary to view in the UI.

```ruby
# Sleep for 72 hours
Temporalio::Workflow.sleep(72 * 60 * 60, summary: 'my timer')
```

There is also a `Temporalio::Workflow.timeout` method that accepts a block and works like standard Ruby
`Timeout.timeout` if needing the ability to timeout a set of code.

---

# Versioning - Ruby SDK

Source: https://docs.temporal.io/develop/ruby/workflows/versioning

> Use the Ruby SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks.

Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress.

The Temporal Platform requires that Workflow Definitions are [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably.

With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version.
There are two primary Versioning methods that you can use:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths.
- [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.

## Worker Versioning

Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching.

## Versioning with Patching 

### Adding a patch

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow Executions, create a patch.

Suppose you have an initial Workflow that runs `PrePatchActivity`:

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    result = Temporalio::Workflow.execute_activity(
      PrePatchActivity,
      start_to_close_timeout: 100
    )

    # ...
  end
end
```

Now, you want to update your code to run `PostPatchActivity` instead. This represents your desired end state.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    result = Temporalio::Workflow.execute_activity(
      PostPatchActivity,
      start_to_close_timeout: 100
    )

    # ...
  end
end
```

The problem is that you cannot deploy this new revision directly until you're certain there are no more running Workflows created using the `PrePatchActivity` code, otherwise you are likely to cause a nondeterminism error.
Instead, you'll need to use the [`patched`](https://ruby.temporal.io/Temporalio/Workflow.html#patched-class_method) function to check which version of the code should be executed.

Patching is a three-step process:

1. Patch in any new, updated code using the `patched()` function. Run the new patched code alongside old code.
2. Remove old code and use `deprecate_patch()` to mark a particular patch as deprecated.
3. Once there are no longer any open Workflow Executions of the previous version of the code, remove `deprecate_patch()`.
   Let's walk through this process in sequence.

### Patching in new code

Using `patched` inserts a marker into the Event History.
During Replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker (in this case `my-patch`).
This ensures you can safely deploy new code paths alongside the original branch.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    if Temporalio::Workflow.patched('my-patch')
      result = Temporalio::Workflow.execute_activity(
        PostPatchActivity,
        start_to_close_timeout: 100
      )
    else
      result = Temporalio::Workflow.execute_activity(
        PrePatchActivity,
        start_to_close_timeout: 100
      )
    end

    # ...
  end
end
```

### Deprecating patches 

After ensuring that all Workflows started with `v1` code have left retention, you can [deprecate the patch](https://ruby.temporal.io/Temporalio/Workflow.html#deprecate_patch-class_method).

Once your Workflows are no longer running the pre-patch code paths, you can deploy your code with `deprecate_patch()`.
These Workers will be running the most up-to-date version of the Workflow code, which no longer requires the patch.
The `deprecate_patch()` function works similarly to the `patched()` function by recording a marker in the Event history.
This marker does not fail replay when Workflow code does not emit it.
Deprecated patches serve as a bridge between the pre-patch code paths and the post-patch code paths, and are useful for avoiding errors resulting from patched code paths in your Event history.

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    Temporalio::Workflow.deprecate_patch('my-patch')
    result = Temporalio::Workflow.execute_activity(
      PostPatchActivity,
      start_to_close_timeout: 100
    )

    # ...
  end
end
```

### Removing a patch 

Once the pre-patch Workflows have left retention, you can then safely deploy Workers that no longer use either the `patched()` or `deprecate_patch()` calls:

Patching allows you to make changes to currently running Workflows.
It introduces compatible changes without causing non-determinism errors.

### Workflow cutovers

To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow.

Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes.
To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers.

For example, you would duplicate `MyWorkflow` as `MyWorkflowV2`:

```ruby
class MyWorkflow < Temporalio::Workflow::Definition
  def execute
    # ...
  end
end

class MyWorkflowV2 < Temporalio::Workflow::Definition
  def execute
    # ...
  end
end
```

You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types:

```ruby
client = Temporalio::Client.connect('localhost:7233', 'default')

worker = Temporalio::Worker.new(
  client:,
  task_queue: 'my-task-queue',
  workflows: [MyWorkflow, MyWorkflowV2]
)
```

The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow.
This can become impractical over time.
This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching.

### Testing a Workflow for replay safety

To determine whether your Workflow your needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/ruby/best-practices/testing-suite#replay-test).

---

# Run a development server

Source: https://docs.temporal.io/develop/run-a-development-server

> Shows how to run a development Temporal Service

## How to install the Temporal CLI and run a development server 

This page describes how to install the [Temporal CLI](/cli) and run a development Temporal Service. The local
development Temporal Service comes packaged with the [Temporal Web UI](/web-ui).

For information on deploying and running a self-hosted production Temporal Service, see the
[Self-hosted guide](/self-hosted-guide), or sign up for [Temporal Cloud](/cloud) and let us run your production Temporal
Service for you.

Temporal CLI is a tool for interacting with a Temporal Service from the command line and it includes a distribution of
the Temporal Server and Web UI. This local development Temporal Service runs as a single process with zero runtime
dependencies and it supports persistence to disk and in-memory mode through SQLite.

**Install the Temporal CLI**

The Temporal CLI is available on macOS, Windows, and Linux.

### macOS

Choose one of the following install methods to install the Temporal CLI on macOS:

**Install the Temporal CLI with Homebrew**

```bash
brew install temporal
```

**Install the Temporal CLI from CDN**

1. Select the platform and architecture needed.

- Download for Darwin amd64: https://temporal.download/cli/archive/latest?platform=darwin&arch=amd64
- Download for Darwin arm64: https://temporal.download/cli/archive/latest?platform=darwin&arch=arm64

2. Extract the downloaded archive.

3. Add the `temporal` binary to your PATH.

### Linux

Choose one of the following install methods to install the Temporal CLI on Linux:

**Install the Temporal CLI with Homebrew**

```bash
brew install temporal
```

**Install the Temporal CLI from CDN**

1. Select the platform and architecture needed.

- Download for Linux amd64: https://temporal.download/cli/archive/latest?platform=linux&arch=amd64
- Download for Linux arm64: https://temporal.download/cli/archive/latest?platform=linux&arch=arm64

2. Extract the downloaded archive.

3. Add the `temporal` binary to your PATH.

### Windows

Follow these instructions to install the Temporal CLI on Windows:

**Install the Temporal CLI from CDN**

1. Select the platform and architecture needed and download the binary.

- Download for Windows amd64: https://temporal.download/cli/archive/latest?platform=windows&arch=amd64
- Download for Windows arm64: https://temporal.download/cli/archive/latest?platform=windows&arch=arm64

2. Extract the downloaded archive.

3. Add the `temporal.exe` binary to your PATH.

### Start the Temporal Development Server

Start the Temporal Development Server by using the `server start-dev` command.

```bash
temporal server start-dev
```

This command automatically starts the Web UI, creates the default [Namespace](/namespaces), and uses an in-memory
database.

The Temporal Server should be available on `localhost:7233` and the Temporal Web UI should be accessible at
[`http://localhost:8233`](http://localhost:8233/).

The server's startup configuration can be customized using command line options. For a full list of options, run:

```bash
temporal server start-dev --help
```

---

# Rust SDK developer guide

Source: https://docs.temporal.io/develop/rust

![Rust SDK Banner](https://docs.temporal.io/img/assets/banner-rust-temporal.png)

## Install and get started

You can find detailed installation instructions for the Rust SDK in the [Quickstart](/develop/rust/quickstart).

Once your local Temporal Service is set up, continue building with the following resources:

- [Develop a Workflow](/develop/rust/workflows/basics)
- [Develop an Activity](/develop/rust/activities/basics)
- [Start an Activity Execution](/develop/rust/activities/execution)
- [Run a Worker](/develop/rust/workers/worker-process)

## [Workflows](/develop/rust/workflows)

- [Workflow basics](/develop/rust/workflows/basics)
- [Child Workflows](/develop/rust/workflows/child-workflows)
- [Continue-As-New](/develop/rust/workflows/continue-as-new)
- [Message passing](/develop/rust/workflows/message-passing)
- [Cancellation](/develop/rust/workflows/cancellation)
- [Timers](/develop/rust/workflows/timers)
- [Timeouts](/develop/rust/workflows/timeouts)

## [Activities](/develop/rust/activities)

- [Activity basics](/develop/rust/activities/basics)
- [Activity Execution](/develop/rust/activities/execution)
- [Standalone Activities](/develop/rust/activities/standalone-activities-quickstart)
- [Timeouts](/develop/rust/activities/timeouts)

## [Workers](/develop/rust/workers)

- [Run a Worker](/develop/rust/workers/worker-process)
- [Serverless Workers](/develop/rust/workers/serverless-workers)

## [Temporal Client](/develop/rust/client)

- [Temporal Client](/develop/rust/client/temporal-client)

## [Temporal Nexus](/develop/rust/nexus)

- [Feature guide](/develop/rust/nexus/feature-guide)

## Temporal Rust technical resources

- [Rust SDK Quickstart - Setup Guide](/develop/rust/quickstart)
- [Rust API Documentation](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/)
- [Rust SDK GitHub](https://github.com/temporalio/sdk-rust/tree/main/crates/sdk)

## Get connected with the Temporal Rust community

- [Temporal Rust Community Slack](https://temporalio.slack.com/archives/C08G723SFNZ/p1773935454727179)

---

# Activities - Rust SDK

Source: https://docs.temporal.io/develop/rust/activities

> Activity Definitions, Activity Execution, and Activity Timeouts in the Temporal Rust SDK, including Retry Policies and Heartbeats.

![Rust SDK Banner](https://docs.temporal.io/img/assets/banner-rust-temporal.png)

## Activities

- [Activity basics](/develop/rust/activities/basics)
- [Activity Execution](/develop/rust/activities/execution)
- [Standalone Activities quickstart](/develop/rust/activities/standalone-activities-quickstart)
- [Standalone Activities feature guide](/develop/rust/activities/standalone-activities)
- [Timeouts](/develop/rust/activities/timeouts)

---

# Activity basics - Rust SDK

Source: https://docs.temporal.io/develop/rust/activities/basics

> Define an Activity as a Rust function, set its parameters and serializable return values, and give its Activity Type a custom name.

## Develop a basic Activity 

One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file.

An Activity can interact with the world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, you need to define the [Activity Definition](/activity-definition).

Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of executing an Activity from within a Workflow Definition, you execute a Standalone Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/rust/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/rust/activities/standalone-activities-quickstart).

The `#[activities]` macro marks an `impl` block as containing Activity definitions. Each method decorated with `#[activity]` becomes an Activity that can be invoked from a Workflow.

Here's an example of an Activity:

```rust
use temporalio_sdk::activities::{ActivityContext, ActivityError};
use temporalio_macros::activities;

pub struct GreetingActivities;

#[activities]
impl GreetingActivities {
    #[activity]
    pub async fn greet(_ctx: ActivityContext, name: String) -> Result<String, ActivityError> {
        Ok(format!("Hello, {}!", name))
    }

    #[activity]
    pub async fn send_notification(_ctx: ActivityContext, message: String) -> Result<(), ActivityError> {
        println!("Sending notification: {}", message);
        Ok(())
    }
}
```

### Define Activity parameters 

There is a limit of 6 parameters that an [Activity Definition](/activity-definition) may support. There is also a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB.

Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance.

We recommend that you use a single struct as an argument that wraps all the application data passed to Activities. This way you can change what data is passed to the Activity without breaking the function signature.

Each Activity method must:

- Be `async` (return a future)
- Take `ActivityContext` as the first parameter
- Return `Result<T, ActivityError>` where `T` is the return type
- Be `pub` (public)

The `ActivityContext` parameter provides access to Activity execution information and capabilities like heartbeating. If you don't need it, you can use `_ctx` as a parameter name.

Activities can also take `Arc<Self>` and be registered using an instance. Here's an example using `Arc`:

```rust
struct SleeperActivities {
    acts_started: Arc<Semaphore>,
    acts_done: Arc<Semaphore>,
}

#[activities]
impl SleeperActivities {
    #[activity]
    pub async fn sleeper(
        self: Arc<Self>,
        ctx: ActivityContext,
        _: String,
    ) -> Result<(), ActivityError> {
        self.acts_started.add_permits(1);
        // just wait to be cancelled
        ctx.cancelled().await;
        self.acts_done.add_permits(1);
        Err(ActivityError::cancelled())
    }
}
```

Activity parameters should be serializable and deserializable using serde. Use `#[derive(Serialize, Deserialize)]` on your data types:

```rust
use serde::{Serialize, Deserialize};
use temporalio_macros::activities;
use temporalio_sdk::activities::{ActivityContext, ActivityError};

#[derive(Serialize, Deserialize)]
pub struct GreetingInput {
    pub greeting: String,
    pub name: String,
}

pub struct GreetingActivities;

#[activities]
impl GreetingActivities {
    #[activity]
    pub async fn compose_greeting(
        _ctx: ActivityContext,
        input: GreetingInput,
    ) -> Result<String, ActivityError> {
        Ok(format!("{} {}!", input.greeting, input.name))
    }
}
```

### Define Activity return values 

All data returned from an Activity must be serializable.

Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction. Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history).

The return type of an Activity is `Result<T, ActivityError>`. The `T` type must implement `Serialize`. Use `ApplicationFailure::new` for errors that should be retried, and `ApplicationFailure::non_retryable` for permanent failures:

```rust
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone, PartialEq, Eq, Hash)]
pub struct ProcessedData {
    pub processed: String,
}

#[activities]
impl MyActivities {
    #[activity]
    pub async fn process_data(
        _ctx: ActivityContext,
        input: String,
    ) -> Result<ProcessedData, ActivityError> {
        // If an error should be retried
        if !validate_input(&input) {
            return Err(ApplicationFailure::builder("Invalid input format")
                .next_retry_delay(Duration::from_secs(5))
                .build()
                .into());
        }

        // If an error should not be retried
        if input.len() > 1000000 {
            return Err(
                ApplicationFailure::non_retryable("Input too large").into(),
            );
        }

        let result = ProcessedData {
            processed: input.to_uppercase(),
        };

        Ok(result)
    }
}
```

### Customize your Activity Type 

Activities have a Type that refers to the Activity name. The Activity name is used to identify Activity Types in the Workflow Execution Event History, Visibility Queries, and Metrics.

By default, the Activity name is the method name. You can customize it by providing a `name` parameter to the `#[activity]` macro:

```rust
#[activities]
impl GreetingActivities {
    #[activity(name = "compose_greeting")]
    pub async fn greet(_ctx: ActivityContext, name: String) -> Result<String, ActivityError> {
        Ok(format!("Hello, {}!", name))
    }

    #[activity(name = "send_email")]
    pub async fn send_notification(_ctx: ActivityContext, message: String) -> Result<(), ActivityError> {
        println!("Sending notification: {}", message);
        Ok(())
    }
}
```

---

# Activity Execution - Rust SDK

Source: https://docs.temporal.io/develop/rust/activities/execution

> Call an Activity from a Rust Workflow with ctx.execute_activity, set the required Timeout, and await the Future it returns.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a
[Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the
[ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in a set of three [Activity Task](/tasks#activity-task) related Events in your Workflow Execution Event History:
[ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTaskClosed.

A single instance of the Activity implementation may be used across multiple concurrent Activity invocations. Activity implementation code should be *idempotent*.

Values passed to Activities as input parameters or returned as results are recorded in the Workflow Execution history. This history is replayed to Workflow Workers during recovery. Large payloads can negatively impact Workflow performance.

Be mindful of the size of data passed to and from Activities. Otherwise, there are no strict limitations on Activity implementations.

To spawn an Activity Execution, use the Workflow context’s Activity execution APIs within your Workflow code.

In Rust, Activities are typically executed using `ctx.execute_activity(...)`, which returns a `Future` that can be awaited.

```rust
#[workflow_methods]
impl GreetingWorkflow {
    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());
        // Execute an activity
        let greeting = ctx
            .execute_activity(
                MyActivities::greet,
                name,
                ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
            )
            .await?;

        println!("{}", greeting);

        Ok(greeting)
    }
}
```

### Set the required Activity Timeouts 

Activity Execution semantics rely on several timeout parameters. You need to set at least one of these:

* [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout)
* [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout)

These are configured as part of the Activity options when scheduling the Activity. Available timeouts include:

- `start_to_close_timeout`
- `schedule_to_close_timeout`
- `with_start_to_close_timeout`
- `with_schedule_to_close_timeout`

```rust
#[workflow_methods]
impl GreetingWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, name: String) -> Self {
        Self { name }
    }

    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());
        // Execute an activity
        let greeting = ctx
            .execute_activity(
                MyActivities::greet,
                name,
                ActivityOptions::schedule_to_close_timeout(Duration::from_secs(30)),
            )
            .await?;

        println!("{}", greeting);
        Ok(greeting)
    }
}
```

### Get the results of an Activity Execution 

Spawning an [Activity Execution](/activity-execution) generates a [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and returns a `Future` to the Workflow.

Workflows can either:

* `await` the result immediately (blocking progress), or
* store the `Future` and await it later to allow concurrent execution.

In Rust, calling `.await` on the Activity invocation returns the result. If you need more control (for example, parallel execution), you can create multiple Activity futures and await them selectively.

You must provide either `schedule_to_close_timeout` or `start_to_close_timeout`.

```rust
#[workflow_methods]
impl GreetingWorkflow {
    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());
        // Execute an activity
        let greeting = ctx
            .execute_activity(
                MyActivities::greet,
                name,
                ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
            )
            .await?;

        println!("{}", greeting);
        Ok(greeting)
    }
}
```

For concurrent execution:

```rust
use temporalio_sdk::workflows::join;

#[run]
pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
    let name = ctx.state(|s| s.name.clone());
    // Execute an activity
    let greeting = ctx.execute_activity(
        MyActivities::greet,
        name,
        ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
    );

    let language = ctx.execute_activity(
        MyActivities::call_greeting_service,
        ActivityLanguages::English,
        ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
    );

    // Run in parallel
    let (greeting_res, language_res) = join!(greeting, language);

    Ok(format!("{} ({})", greeting_res?, language_res?))
}
```

Use direct `.await` in most cases. More advanced patterns, like parallel execution or cancellation, can be built using Rust’s async primitives.

---

# Standalone Activities feature guide - Rust SDK

Source: https://docs.temporal.io/develop/rust/activities/standalone-activities

> Execute Activities independently without a Workflow using the Temporal Rust SDK.

[Standalone Activities](/standalone-activity) are Activities that run independently, without being
orchestrated by a Workflow. Instead of executing an Activity from within a Workflow Definition using
`ctx.execute_activity()`, you execute a Standalone Activity directly from a
[`Client`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html).

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/rust/activities/basics). The only difference is that you execute a Standalone
Activity directly from your Temporal Client.

> **💡 Tip:**
>
> New to Standalone Activities? Start with the [Standalone Activities quickstart](/develop/rust/activities/standalone-activities-quickstart).
>

This page covers the following:

- [Prerequisites](#prerequisites)
- [Start a Standalone Activity without waiting for the result](#start-activity)
- [Get a handle to an existing Standalone Activity](#get-activity-handle)
- [Wait for the result of a Standalone Activity](#get-activity-result)
- [List Standalone Activities](#list-activities)
- [Count Standalone Activities](#count-activities)
- [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from the
> [standalone_activities](https://github.com/temporalio/sdk-rust/tree/main/crates/sdk/examples/standalone_activities)
> sample.
>

## Prerequisites 

Standalone Activities require:

- **Rust** 1.92.0+
- **Temporal Rust SDK** v1.0.0 or higher
- **[Temporal CLI](/cli/setup-cli)** v1.9.1 or higher

The [Standalone Activities quickstart](/develop/rust/activities/standalone-activities-quickstart)
walks through installing these.

## Start a Standalone Activity without waiting for the result 

Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue
your Activity job, without waiting for it to be executed by your Worker.

Use [`Client::start_activity`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html#method.start_activity)
to start a Standalone Activity and get a handle without waiting for the result:

[start_activity.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/start_activity.rs)

```rust
let options = ActivityStartOptions::with_start_to_close_timeout(
    "standalone-activities",
    "standalone-activity-id",
    Duration::from_secs(10),
)
.build();

// Returns as soon as the server has durably enqueued the activity.
let handle = client
    .start_activity(
        GreetingActivities::compose_greeting,
        ("Hello".to_string(), "Temporal".to_string()),
        options,
    )
    .await?;

println!(
    "Started activity, id: {} run_id: {:?}",
    handle.activity_id(),
    handle.run_id()
);
```

The first argument identifies the Activity to run. Passing the Activity method itself, such as
`GreetingActivities::compose_greeting`, gives you a typed
[`ActivityHandle`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityHandle.html),
so the input and output types are checked at compile time. The second argument is the Activity's
input; if your Activity takes several values, pass them as a tuple.

[`ActivityStartOptions`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityStartOptions.html)
requires a Task Queue, an Activity ID, and a close timeout. The
`with_start_to_close_timeout` and `with_schedule_to_close_timeout` constructors return a builder
with that timeout already set; call `.build()` to finish, or set additional fields such as
`retry_policy`, `heartbeat_timeout`, `id_reuse_policy`, or `priority` first.

With the Temporal Server and Worker running, open a new terminal in the `crates/sdk` directory and
run:

```bash
cargo run --features examples --example standalone-activities-start
```

Or use the Temporal CLI:

```bash
temporal activity start \
  --type 'GreetingActivities::compose_greeting' \
  --activity-id standalone-activity-id \
  --task-queue standalone-activities \
  --start-to-close-timeout 10s \
  --input '["Hello","Temporal"]'
```

By default, the `#[activities]` macro names each Activity `<ImplType>::<method_name>`, so the
`compose_greeting` method on `GreetingActivities` registers as `GreetingActivities::compose_greeting`. Use that
name when referring to the Activity from the CLI or from a [List Filter](/list-filter) query.

## Get a handle to an existing Standalone Activity 

Use [`Client::get_activity_handle`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html#method.get_activity_handle)
to create an
[`ActivityHandle`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityHandle.html)
for a previously started Standalone Activity:

[get_activity_handle.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/get_activity_handle.rs)

```rust
// Passing `None` for the run ID targets the latest run with this activity ID.
let handle = client.get_activity_handle(
    GreetingActivities::compose_greeting,
    "standalone-activity-id",
    None,
);
```

Pass `None` for the run ID to target the latest run of the given Activity ID, or pass
`Some(run_id)` to target a specific run.

If you don't have the Activity definition on hand, for example in a tool that operates on
Activities it didn't start, use
[`Client::get_untyped_activity_handle`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html#method.get_untyped_activity_handle)
instead. You can still describe, cancel, and terminate through an untyped handle.

You can then use the handle to wait for the result, describe, cancel, or terminate the Activity:

```rust
handle.result().await?;                            // block until the activity completes
handle.describe(Default::default()).await?;        // status, timestamps, attempt, ...
handle.cancel(Default::default()).await?;          // request cancellation
handle.terminate(Default::default()).await?;       // force-close the activity
```

[`describe`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityHandle.html#method.describe)
takes an
[`ActivityDescribeOptions`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityDescribeOptions.html).
Its `include_input`, `include_outcome`, `include_heartbeat_details`, and `include_last_failure`
fields are off by default, because those fields carry Payloads that can be arbitrarily large. Turn
them on only when you need them:

```rust
let description = handle
    .describe(
        ActivityDescribeOptions::builder()
            .include_outcome(true)
            .build(),
    )
    .await?;

println!("Status: {:?}", description.status());
println!("Type: {}", description.activity_type());
println!("Attempt: {}", description.attempt());
```

The accessors on the description (`status()`, `activity_type()`, `schedule_time()`, and so on)
come from the
[`ActivityExecutionInfoLike`](https://docs.rs/temporalio-client/latest/temporalio_client/trait.ActivityExecutionInfoLike.html)
trait, so bring it into scope to use them.

Run it, after executing an Activity with one of the samples above:

```bash
cargo run --features examples --example standalone-activities-get-handle
```

Or use the Temporal CLI to describe an Activity by ID:

```bash
temporal activity describe --activity-id standalone-activity-id
```

## Wait for the result of a Standalone Activity 

The Rust SDK has no single call that both starts an Activity and waits for its result. Call
`start_activity` to durably enqueue the Activity, then
[`ActivityHandle::result`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityHandle.html#method.result)
to block until it completes and return the result:

[execute_activity.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/execute_activity.rs)

```rust
// There is no single "execute" call: start the activity, then await its result.
let handle = client
    .start_activity(
        GreetingActivities::compose_greeting,
        ("Hello".to_string(), "Temporal".to_string()),
        options,
    )
    .await?;

let result = handle.result().await?;
println!("Activity result: {result}");
```

Because the handle is typed, `result` returns the Activity's own output type, `String` in this
case, with no downcasting. It fails with an
[`ActivityResultError`](https://docs.rs/temporalio-client/latest/temporalio_client/errors/enum.ActivityResultError.html)
if the Activity failed, was cancelled, or was terminated.

Splitting start from result also means you don't have to wait in the same process, or even the same
program, that started the Activity: [get a handle](#get-activity-handle) later and call `result` on
it.

Or use the Temporal CLI to wait for a result by Activity ID:

```bash
temporal activity result --activity-id standalone-activity-id
```

## List Standalone Activities 

Use [`Client::list_activities`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html#method.list_activities)
to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is
a [`ListActivitiesStream`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ListActivitiesStream.html),
a `Stream` of
[`ActivityExecutionInfo`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityExecutionInfo.html)
values that fetches pages from the server on demand as the stream is consumed.

These APIs return only Standalone Activity Executions. Activities running inside Workflows are not
included.

[list_activities.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/list_activities.rs)

```rust
use futures::StreamExt;
use temporalio_client::ActivityExecutionInfoLike;

let mut executions =
    client.list_activities("TaskQueue = 'standalone-activities'", Default::default());

while let Some(execution) = executions.next().await {
    let execution = execution?;
    println!(
        "{} {} {:?}",
        execution.activity_id(),
        execution.activity_type(),
        execution.status()
    );
}
```

`list_activities` is not `async`. It returns the stream immediately, and the requests happen as you
poll it. Each item is a `Result`, because a page fetch can fail partway through the
stream.

Run it:

```bash
cargo run --features examples --example standalone-activities-list
```

Or use the Temporal CLI:

```bash
temporal activity list
```

The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow
Visibility](/visibility). For example,
`ActivityType = 'GreetingActivities::compose_greeting' AND ExecutionStatus = 'Running'`.

## Count Standalone Activities 

Use [`Client::count_activities`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html#method.count_activities)
to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns
the total count of executions (running, completed, failed, etc.) — not the number of queued tasks.
It works the same way as counting Workflow Executions.

[count_activities.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/count_activities.rs)

```rust
let count = client
    .count_activities("TaskQueue = 'standalone-activities'", Default::default())
    .await?;

println!("Total: {}", count.count());
// Non-empty only when the query has a GROUP BY clause.
for group in count.groups() {
    println!("  {:?} => {}", group.get::<String>(0), group.count());
}
```

If the query has a `GROUP BY` clause,
[`groups()`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityExecutionCount.html#method.groups)
holds the per-group counts and `count()` is their sum; otherwise `groups()` is empty. Group values
are typed: read them with `get::<T>(index)`, or with `try_get` if you want to handle a
deserialization failure rather than get `None`.

Run it:

```bash
cargo run --features examples --example standalone-activities-count
```

Or use the Temporal CLI:

```bash
temporal activity count
```

## Run Standalone Activities with Temporal Cloud 

The Worker and Client code in the [Standalone Activities quickstart](/develop/rust/activities/standalone-activities-quickstart)
use [`ClientOptions::load_from_config`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ClientOptions.html#method.load_from_config),
so the same code works against Temporal Cloud — configure the connection via environment variables
or a TOML profile. No code changes are needed.

For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate
generation, and authentication setup in the Cloud UI, see
[Connect to Temporal Cloud](/develop/rust/client/temporal-client#connect-to-temporal-cloud).

### Connect with mTLS

Set these environment variables with values from your Temporal Cloud Namespace settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem'
export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key'
```

### Connect with an API key

Set these environment variables with values from your Temporal Cloud API key settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_API_KEY=<your-api-key>
```

Then run the Worker and starter code as shown in the [Standalone Activities quickstart](/develop/rust/activities/standalone-activities-quickstart).

---

# Standalone Activities Rust quickstart - Rust SDK

Source: https://docs.temporal.io/develop/rust/activities/standalone-activities-quickstart

> Execute a Standalone Activity with the Temporal Rust SDK without writing a Workflow.

# Quickstart

Standalone Activities are Activities that run independently, without being orchestrated by a
Workflow. Instead of executing an Activity from within a Workflow Definition using
`ctx.execute_activity()`, you execute a Standalone Activity directly from a
[`Client`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html).

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/rust/activities/basics). The only difference is that you execute a Standalone
Activity directly from your Temporal Client.

> **📝 Note:**
>
> This documentation uses source code from the
> [standalone_activities](https://github.com/temporalio/sdk-rust/tree/main/crates/sdk/examples/standalone_activities)
> sample.
>

## Get started with Standalone Activities 

Prerequisites:

- **Rust** 1.92.0+

- **Temporal Rust SDK** (v1.0.0 or higher). See the [Rust Quickstart](/develop/rust/quickstart) for install instructions.

- **Temporal CLI** v1.9.1 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`.

Start the Temporal development server with `temporal server start-dev`.

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Server will now be available for client connections on `localhost:7233`, and the
Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233).

```bash
brew install temporal
```

```bash
temporal --version
```

```bash
temporal server start-dev
```

## Clone the sample

Clone the [sdk-rust](https://github.com/temporalio/sdk-rust) repository to follow along:

```bash
git clone https://github.com/temporalio/sdk-rust.git
cd sdk-rust/crates/sdk
```

The sample consists of separate programs in the `crates/sdk/examples/standalone_activities` directory:

```
standalone_activities/
├── activities.rs           # Activity definition, shared by the programs below
├── worker.rs               # Worker that processes Activity Tasks
├── execute_activity.rs     # Executes an Activity and waits for the result
├── start_activity.rs       # Starts an Activity without blocking
├── get_activity_handle.rs  # Gets a handle to an existing Activity
├── list_activities.rs      # Lists Activity Executions
└── count_activities.rs     # Counts Activity Executions
```

To write the same code in your own project, add the dependencies shown here to your `Cargo.toml`.
Standalone Activities need `temporalio-client` for the Client, `temporalio-sdk` and
`temporalio-macros` for the Activity and Worker, and `tokio` as the async runtime. `futures` is
needed to consume the stream returned by `list_activities`.

```toml
[dependencies]
futures = "0.3"
temporalio-client = "1.0.0"
temporalio-macros = "1.0.0"
temporalio-sdk = "1.0.0"
tokio = { version = "1", features = ["full"] }

```

## Define your Activity 

An Activity in the Temporal Rust SDK is an `async` method on an `impl` block marked with the
`#[activities]` macro, annotated with `#[activity]`. The way you define a Standalone Activity is
identical to how you define an Activity orchestrated by a Workflow. In fact, the same Activity can
be executed both as a Standalone Activity and as a Workflow Activity.

Each Activity method takes an
[`ActivityContext`](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/activities/struct.ActivityContext.html)
as its first parameter and returns `Result<T, ActivityError>`. Use `_ctx` if you don't need the
context. To pass more than one value to an Activity, take them as a tuple, as
`compose_greeting` does here.

By default, the macro names each Activity `<ImplType>::<method_name>`, so this one registers as
`GreetingActivities::compose_greeting`. That's the name to use from the Temporal CLI and in [List
Filter](/list-filter) queries.

[activities.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/activities.rs)

```rust
use temporalio_macros::activities;
use temporalio_sdk::activities::{ActivityContext, ActivityError};

pub struct GreetingActivities;

#[activities]
impl GreetingActivities {
    #[activity]
    pub async fn compose_greeting(
        _ctx: ActivityContext,
        input: (String, String),
    ) -> Result<String, ActivityError> {
        let (greeting, name) = input;
        Ok(format!("{greeting}, {name}!"))
    }
}
```

## Run a Worker with the Activity registered 

Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities —
you build [`WorkerOptions`](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/struct.WorkerOptions.html)
for a Task Queue, register the Activities with `register_activities`, and call `worker.run()`. The
Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone
Activity, and a Worker that runs Standalone Activities needs no registered Workflows at all. See
[How to run a Worker](/develop/rust/workers/worker-process) for more details on Worker setup and
configuration options.

[worker.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/worker.rs)

Open a new terminal, navigate to the `crates/sdk` directory, and run the Worker.
Leave this terminal running — the Worker needs to stay up to process activities.

```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = Runtime::from_current_tokio(Default::default())?;
    let (conn_opts, client_opts) =
        ClientOptions::load_from_config(LoadClientConfigProfileOptions::default())?;
    let connection = Connection::connect(conn_opts).await?;
    let client = Client::new(connection, client_opts)?;

    // A Worker that only runs Standalone Activities needs no registered workflows.
    let worker_options = WorkerOptions::new("standalone-activities")
        .register_activities(GreetingActivities)
        .build();

    let mut worker = Worker::new(&runtime, client, worker_options)?;
    println!("Worker started on task queue: standalone-activities");
    worker.run().await?;

    Ok(())
}
```

```bash
cargo run --features examples --example standalone-activities-worker
```

## Execute a Standalone Activity 

Use [`Client::start_activity`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.Client.html#method.start_activity)
to start a Standalone Activity, then
[`ActivityHandle::result`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityHandle.html#method.result)
to block until it completes. Call these from your application code, not from inside a Workflow
Definition. `start_activity` durably enqueues your Standalone Activity in the Temporal Server, and
`result` waits for it to be executed on your Worker and returns the result.

[execute_activity.rs](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk/examples/standalone_activities/execute_activity.rs)

The first argument to `start_activity` is the Activity to run. Passing the Activity method itself,
`GreetingActivities::compose_greeting`, gives you a typed
[`ActivityHandle`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityHandle.html),
so the input and output types are checked at compile time and `result` returns the Activity's own
return type. The second argument is the Activity's input.

[`ActivityStartOptions`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ActivityStartOptions.html)
requires a Task Queue, an Activity ID, and a close timeout. The `with_start_to_close_timeout` and
`with_schedule_to_close_timeout` constructors return a builder with that timeout already set; call
`.build()` to finish.

To run it:

1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above).
2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above).
3. Open a new terminal, navigate to the `crates/sdk` directory, and run the execute command.

Or use the Temporal CLI. Because `compose_greeting` takes its two values as a tuple, the CLI input
is a single two-element JSON array.

```rust
let options = ActivityStartOptions::with_start_to_close_timeout(
    "standalone-activities",
    "standalone-activity-id",
    Duration::from_secs(10),
)
.build();

// There is no single "execute" call: start the activity, then await its result.
let handle = client
    .start_activity(
        GreetingActivities::compose_greeting,
        ("Hello".to_string(), "Temporal".to_string()),
        options,
    )
    .await?;

let result = handle.result().await?;
println!("Activity result: {result}");
```

```bash
cargo run --features examples --example standalone-activities-execute
```

```bash
temporal activity execute \\
  --type 'GreetingActivities::compose_greeting' \\
  --activity-id standalone-activity-id \\
  --task-queue standalone-activities \\
  --start-to-close-timeout 10s \\
  --input '["Hello","Temporal"]'
```

## Run with Temporal Cloud

All code samples on this page use
[`ClientOptions::load_from_config`](https://docs.rs/temporalio-client/latest/temporalio_client/struct.ClientOptions.html#method.load_from_config)
to configure the Temporal Client connection. It responds to [environment
variables](/references/client-environment-configuration) and [TOML configuration
files](/references/client-environment-configuration), so the same code works against a local dev
server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal
Cloud](/develop/rust/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the feature guide
for mTLS and API key setup.

## Next steps

- **[Standalone Activities feature guide](/develop/rust/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud.
- **[Activity basics](/develop/rust/activities/basics)**: How to write and register Activities with the Rust SDK.

---

# Activity Timeouts - Rust SDK

Source: https://docs.temporal.io/develop/rust/activities/timeouts

> Set the Schedule-To-Close, Start-To-Close, and Heartbeat Timeouts on a Rust Activity, tune its Retry Policy, and override the next retry delay.

## Set Activity timeouts 

Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution.

The following timeouts are available in Activity options:

- [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout): the maximum amount of time allowed for the overall [Activity Execution](/activity-execution).
- [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout): the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution).
- [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout): the maximum amount of time allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design.

An Activity Execution must have either the Start-To-Close Timeout or the Schedule-To-Close Timeout set. Temporal strongly recommends setting a Start-To-Close Timeout because the service relies on it to detect lost Activity Tasks and trigger retries when appropriate.

In Rust, these values are configured as part of the Activity options when scheduling an Activity from a Workflow.

Available timeout fields include:

- `schedule_to_close_timeout`
- `schedule_to_start_timeout`
- `start_to_close_timeout`

```rust
let greeting = ctx.execute_activity(
    MyActivities::greet,
    name,
    ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
).await?;
```

### Set an Activity Retry Policy 

A Retry Policy works together with timeouts to provide fine-grained control over Activity failure handling. Activities automatically use a default [Retry Policy](/encyclopedia/retry-policies) unless you provide a custom one.

In Rust, configure the Retry Policy as part of the Activity options when scheduling the Activity from Workflow code. Because the Rust SDK API is still evolving, treat the following as representative of the current style rather than a guaranteed stable surface.

```rust
let language = ctx.execute_activity(
    MyActivities::call_greeting_service,
    ActivityLanguages::English,
    ActivityOptions::with_start_to_close_timeout(Duration::from_secs(30))
        .retry_policy(
            RetryPolicy::builder()
                .initial_interval(Duration::from_secs(10))
                .backoff_coefficient(2.0)
                .maximum_interval(Duration::from_secs(100))
                .maximum_attempts(5)
                .non_retryable_error_types(["NonRetryableError"])
                .build(),
        )
        .build(),
).await?;
```

### Override the retry interval with `explicit_delay` 

To override the next retry interval set by the current policy, return a failure from an Activity with a custom next retry delay. That value replaces the interval the Retry Policy would otherwise use for the next retry attempt. This is useful when retry timing depends on runtime state such as the current attempt number.

For example, you can increase the delay linearly with each attempt instead of using the exponential backoff defined by a backoff coefficient:

```rust
use temporalio_macros::activities;
use temporalio_sdk::{
    ApplicationFailure,
    activities::{ActivityContext, ActivityError},
};
use std::sync::atomic::AtomicUsize;

struct TestGreetActivities {
    counter: AtomicUsize,
}

#[activities]
impl TestGreetActivities {
    #[activity]
    pub async fn greet(_ctx: ActivityContext, name: String) -> Result<String, ActivityError> {
        if name == "ziggy" {
            return Err(ApplicationFailure::builder("Ziggy is not a valid name")
                // next retry will be after 5 seconds
                .next_retry_delay(std::time::Duration::from_secs(5))
                .build()
                .into());
        }
        Ok(format!("Hello, {}!", name))
    }
}
```

## Heartbeat an Activity 

An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a signal from the [Worker Process](/workers#worker-process) executing the Activity to the [Temporal Service](/temporal-service). Each heartbeat tells the service that the [Activity Execution](/activity-execution) is still making progress and that the Worker has not crashed. If the service does not receive a heartbeat within the configured [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), the Activity can time out and be retried according to its Retry Policy.

Heartbeats may be throttled by the Worker, so not every heartbeat call is necessarily sent immediately to the Temporal Service. Activity cancellation is also delivered through heartbeat processing, which means Activities that don't heartbeat cannot receive cancellation promptly.

Heartbeats can include `details` that describe current progress. If the Activity fails and is retried, the retried attempt can retrieve the details from the most recently recorded heartbeat. The Rust SDK exposes activity context support for heartbeat details.

To heartbeat an Activity in Rust, call the heartbeat API from inside the Activity with `record_heartbeat`:

```rust
pub async fn greet(ctx: ActivityContext, name: String) -> Result<String, ActivityError> {
    ctx.record_heartbeat("greet activity started".to_string())
        .await?;

    if name == "ziggy" {
        return Err(ApplicationFailure::new("Ziggy is not a valid name").into());
    }

    Ok(format!("Hello, {}!", name))
}
```

### Set a Heartbeat Timeout 

A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works together with Activity heartbeats and sets the maximum time allowed between heartbeats. Configure it as part of the Activity options when scheduling the Activity.

```rust
let language = ctx.execute_activity(
    MyActivities::call_greeting_service,
    ActivityLanguages::English,
    ActivityOptions::with_start_to_close_timeout(Duration::from_secs(30))
        .heartbeat_timeout(Duration::from_secs(5))
        .build(),
);
```

---

# Client - Rust SDK

Source: https://docs.temporal.io/develop/rust/client

![Rust SDK Banner](https://docs.temporal.io/img/assets/banner-rust-temporal.png)

## Temporal Client

- [Temporal Client](/develop/rust/client/temporal-client)

---

# Temporal Client - Rust SDK

Source: https://docs.temporal.io/develop/rust/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) lets your application communicate with the Temporal Service. Use it to start Workflow Executions, send Signals, run Queries, fetch Workflow results, and more.

For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow.

This page shows how to do the following using the Rust SDK and Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-development-service)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Start a Workflow Execution](#start-workflow-execution)
- [Get Workflow results](#get-workflow-results)

A Temporal Client can't be created and used inside Workflow code. However, using a Temporal Client inside an Activity is acceptable when you need to communicate with the Temporal Service.

## Connect to development Temporal Service 

In Rust, create a client by establishing a `Connection` and then constructing a `Client`.
You can provide connection options directly in code or load them from environment variables.

When you are running Temporal locally, the minimal setup is typically a local server address and the `default` Namespace.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client.
The configuration file supports multiple profiles, each with its own connection options.

If you don't specify a configuration file path, the SDK looks in the default OS-specific location.
Environment variables take precedence over values from the configuration file.

For example, the following TOML file defines two profiles:

```toml title="temporal.toml"
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Optional: Add custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path  = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment     = "production"
service-version = "v1.2.3"
```

Load the configuration and connect with the `prod` profile as follows:

```rust
use temporalio_client::{
    Client, ClientOptions, Connection,
    envconfig::LoadClientConfigProfileOptions,
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (conn_opts, client_opts) = ClientOptions::load_from_config(
        LoadClientConfigProfileOptions::builder()
            .config_file_profile("prod".to_owned())
            .build(),
    )?;
    let connection = Connection::connect(conn_opts).await?;
    let client = Client::new(connection, client_opts)?;

    ...

    Ok(())
}
```

**Environment Variables**

You can also configure the Temporal Client with environment variables using `envconfig`. This is useful for local development, CI, and production deployments.

```rust
use temporalio_client::{
    Client, ClientOptions, Connection,
};
use temporalio_sdk::{Runtime, Worker, WorkerOptions};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = Runtime::from_current_tokio(Default::default())?;
    let (conn_opts, client_opts) =
        ClientOptions::load_from_config(Default::default())?;
    let connection = Connection::connect(conn_opts).await?;
    let client = Client::new(connection, client_opts)?;

    let worker_options = WorkerOptions::new("hello-world")
        .register_workflow::<HelloWorldWorkflow>()?
        .register_activities(GreetingActivities)
        .build();

    let mut worker = Worker::new(&runtime, client, worker_options)?;
    println!("Worker started on task queue: hello-world");
    worker.run().await?;

    Ok(())
}
```

**Code**

You can also specify connection options directly in code. This is convenient for local development and testing.

```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (conn_opts, client_opts) =
        ClientOptions::load_from_config(Default::default())?;
    let connection = Connection::connect(conn_opts).await?;
    let client = Client::new(connection, client_opts)?;

    let wf_handle = client
        .start_workflow(
            GreetingsWorkflow::run,
            (),
            WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build(),
        )
        .await?;
}
```

## Connect to Temporal Cloud 

You can connect to Temporal Cloud using either an API key or mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance:

- Your authentication credentials:
    - For API key authentication, provide the API key.
    - If you are using mTLS, provide the mTLS CA certificate and mTLS private key.
- Your _Namespace_ and _Account ID_ combination in the format `<namespace_id>.<account_id>`
- The recommended gRPC endpoint for your Namespace, such as `<namespace>.<account>.tmprl.cloud:7233`

For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates).

You can provide these connection options using environment variables, a configuration file, or directly in code.

**Configuration File**

You can define a Temporal Cloud profile in `temporal.toml`:

```toml
[profile.api]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```

If you want to use mTLS instead of an API key:

```toml
[profile.mtls]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"

[profile.mtls.tls]
client_cert_path = "/path/to/client.pem"
client_key_path = "/path/to/client.key"
```

Then load the profile and connect:

```rust
use temporalio_client::{
    Client, ClientOptions, Connection,
    envconfig::LoadClientConfigProfileOptions,
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (conn_opts, client_opts) = ClientOptions::load_from_config(
        LoadClientConfigProfileOptions::builder()
            .config_file_profile("api".to_owned())
            .build(),
    )?;

    // Client setup
    let connection = Connection::connect(conn_opts).await?;
    let client = Client::new(connection, client_opts)?;

    println!("Connected to Temporal Cloud!");
    Ok(())
}
```

**Environment Variables**

The following environment variables are commonly used to connect to Temporal Cloud:

* `TEMPORAL_NAMESPACE`
* `TEMPORAL_ADDRESS`
* `TEMPORAL_API_KEY`
* `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`
* `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`

After setting the environment variables, load the configuration and connect:

```rust
use temporalio_client::{
    Client, ClientOptions, Connection,
    envconfig::LoadClientConfigProfileOptions,
};
use temporalio_sdk::{Runtime, Worker, WorkerOptions};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = Runtime::from_current_tokio(Default::default())?;
    let (conn_opts, client_opts) =
        ClientOptions::load_from_config(LoadClientConfigProfileOptions::default())?;
    let connection = Connection::connect(conn_opts).await?;
    let client = Client::new(connection, client_opts)?;

    let worker_options = WorkerOptions::new("hello-world")
        .register_workflow::<HelloWorldWorkflow>()?
        .register_activities(GreetingActivities)
        .build();

    let mut worker = Worker::new(&runtime, client, worker_options)?;
    println!("Worker started on task queue: hello-world");
    worker.run().await?;

    Ok(())
}
```

**Code**

You can also specify connection options directly in code for Temporal Cloud.

```rust
use std::str::FromStr;
use temporalio_client::{Client, ClientOptions, Connection, ConnectionOptions, Url};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let connection_options = ConnectionOptions::new(Url::from_str(
        "https://your-namespace.a1b2c.tmprl.cloud:7233",
    )?)
        .api_key("your-api-key")
        .build();

    let connection = Connection::connect(connection_options).await?;
    let _client = Client::new(
        connection,
        ClientOptions::new("your-namespace").build(),
    )?;

    println!("Connected to Temporal Cloud!");
    Ok(())
}
```

With the default features, an mTLS client certificate is read when `Connection::connect` creates the connection.
The `dynamic-tls` feature provides `TlsOptions::client_cert_resolver` for rotating client certificates without
restarting the Worker; see
[Update certificates using Temporal Cloud UI/tcld](/cloud/certificates#manage-certificates) for the zero-downtime
staging sequence.

## Start a Workflow Execution 

To start a Workflow Execution, supply:

- the Workflow Type
- the Workflow input
- a [Task Queue](/task-queue) that a Worker is polling
- a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id)

Starting a Workflow Execution creates the first [WorkflowExecutionStarted](/references/events#workflowexecutionstarted) Event in the Event History, followed by the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event.

In Rust, use `start_workflow()` to start a Workflow and return a handle.

```rust
let handle = client.start_workflow(
    GreetingsWorkflow::run,
    (),
    WorkflowStartOptions::new(
        "my-task-queue",
        "greetings-workflow-10",
    ).build()
).await?;
```

### Set a Workflow's Task Queue 

The only Workflow Option that you must set is the name of the [Task Queue](/task-queue).

For a Workflow to make progress, at least one Worker must be polling the same Task Queue.

In Rust, set the Task Queue in `WorkflowStartOptions`:

```rust
let handle = client
    .start_workflow(
        GreetingsWorkflow::run,
        (),
        WorkflowStartOptions::new(
            "your-task-queue", 
            "your-workflow-id"
        ).build(),
    ).await?;
```

### Set a Workflow Id 

You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id).

A Workflow Id should usually map to a business process or business entity identifier, such as an order identifier or
customer identifier.

In Rust, set the Workflow Id in `WorkflowStartOptions`:

```rust
let handle = client
    .start_workflow(
        GreetingsWorkflow::run,
        (),
        WorkflowStartOptions::new(
            "your-task-queue", 
            "your-workflow-id"
        ).build(),
    ).await?;
```

## Get Workflow results 

If starting a Workflow succeeds, you get a Workflow handle.
You can use that handle to wait for the result, describe the Workflow, or interact with it through Signals, Queries, and Updates.

To get the result of a newly started Workflow:

```rust
let handle = client
    .start_workflow(
        GreetingsWorkflow::run,
        (),
        WorkflowStartOptions::new(
            "your-task-queue", 
            "your-workflow-id"
        ).build(),
    ).await?;

let result = handle
    .get_result(WorkflowGetResultOptions::default())
    .await?;

println!("Result: {:?}", result);
```

---

# Nexus - Rust SDK

Source: https://docs.temporal.io/develop/rust/nexus

![Rust SDK Banner](https://docs.temporal.io/img/assets/banner-rust-temporal.png)

## Temporal Nexus

- [Feature guide](/develop/rust/nexus/feature-guide)

---

# Nexus feature guide - Rust SDK

Source: https://docs.temporal.io/develop/rust/nexus/feature-guide

> Use Temporal Nexus within the Rust SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

[Nexus](/nexus) is a tool for coordinating asynchronous operations between Temporal and external systems. Service handlers allow Workflows to receive inbound requests through Nexus.

## Call a Nexus Operation from a Workflow 

You can start a Nexus operation from a Workflow using `ctx.start_nexus_operation()`:
Enable the `experimental` feature for the `temporalio-sdk` dependency to use this API.

```rust
use std::time::Duration;

use temporalio_common::protos::coresdk::AsJsonPayloadExt;
use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{
    ApplicationFailure, NexusOperationOptions, WorkflowContext, WorkflowContextView,
    WorkflowResult,
};

#[workflow]
pub struct GreetingWorkflow {
    pub name: String,
}

#[workflow_methods]
impl GreetingWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, name: String) -> Self {
        Self { name }
    }

    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());
        let nexus_started = ctx
            .start_nexus_operation(
                NexusOperationOptions::builder()
                    .endpoint("my-endpoint")
                    .service("my-service")
                    .operation("my-operation")
                    .input(name.as_json_payload().map_err(ApplicationFailure::new)?)
                    .start_to_close_timeout(Duration::from_secs(10))
                    .build(),
            )
            .await;

        let nexus_started = match nexus_started {
            Ok(started) => started,
            Err(failure) => return Ok(format!("Nexus start failed: {failure:?}")),
        };
        let nexus_result = nexus_started.result().await;

        println!("Nexus result: {:?}", nexus_result);

        Ok(format!("nexus result: {:?}", nexus_result))
    }
}
```

### Nexus Operation arguments

- `endpoint` - The Nexus endpoint name
- `service` - The service name
- `operation` - The operation name
- `input` - The input payload (optional)
- `start_to_close_timeout` - How long the operation can run
- `schedule_to_close_timeout` - How long the caller waits for the operation to complete

---

# Quickstart - Rust SDK

Source: https://docs.temporal.io/develop/rust/quickstart

> Configure your local development environment to get started developing with Temporal and the Rust SDK

Configure your local development environment to get started developing with Temporal using the Rust SDK.

## Install Rust

Make sure you have Rust installed on your system. You can download and install Rust from [rustup.rs](https://rustup.rs/).

After installation, verify it's working by checking the version. You'll also need Cargo, which is Rust's package manager and comes bundled with Rust.

```bash
rustc --version
```

## Create a Project

Now that you have Rust and Cargo installed, create a new Rust project to manage your dependencies.

```bash
mkdir temporal-rust-project
```

```bash
cd temporal-rust-project
```

```bash
cargo init --name temporal-hello-world
```

## Add Temporal Rust SDK Dependencies

Now update your project's `Cargo.toml` file to match the example and run `cargo build`.

The core dependencies you'll need are:

- `temporalio-sdk` - The Rust SDK for Temporal
- `tokio` - Async runtime required by the SDK
- `serde` - For serialization/deserialization

Next, you'll configure a local Temporal Service for development.

```toml
[package]
name = "temporal-hello-world"
version = "0.1.0"
edition = "2024"

[dependencies]
futures = "0.3"
serde = { version = "1", features = ["derive"] }
temporalio-client = "1.0.0"
temporalio-common = "1.0.0"
temporalio-macros = "1.0.0"
temporalio-sdk = "1.0.0"
temporalio-workflow = "1.0.0"
tokio = { version = "1", features = ["full"] }

```

If you run into issues installing the dependencies, try:
```bash
brew install protobuf
```

## Install Temporal CLI and start the development server

The fastest way to get a development version of the Temporal Service running on your local machine is to use
[Temporal CLI](/cli).

Choose your operating system to install Temporal CLI:

**macOS**

```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH:
```bash
sudo mv temporal /usr/local/bin
```

## Start the development server

Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database.

The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233.

Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C.

After installing, open a new Terminal. Keep this running in the background:
```bash
temporal server start-dev
```

#### Change the Web UI port
The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server:
```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity.

This test will confirm that:

- The Temporal Rust SDK is properly installed
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

### 1. Define your Activity

Create a `/src/activities.rs` file with the Activity definition:

```rust
use temporalio_macros::activities;
use temporalio_sdk::activities::{ActivityContext, ActivityError};

pub struct MyActivities;

#[activities]
impl MyActivities {
    #[activity]
    pub async fn greet(_ctx: ActivityContext, name: String) -> Result<String, ActivityError> {
        Ok(format!("Hello, {}!", name))
    }
}
```

### 2. Define your Workflow

Create a `/src/workflows.rs` file with the Workflow definition:

```rust
use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{ActivityOptions, WorkflowContext, WorkflowContextView, WorkflowResult};
use std::time::Duration;

use crate::activities::MyActivities;

#[workflow]
pub struct GreetingWorkflow {
    name: String,
}

#[workflow_methods]
impl GreetingWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, name: String) -> Self {
        Self { name }
    }

    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());

        // Execute an activity
        let greeting = ctx
            .execute_activity(
                MyActivities::greet,
                name,
                ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
            )
            .await?;

        println!("{}", greeting);
        Ok(greeting)
    }
}
```

### 3. Create and Run a Worker

Update your `/src/main.rs` file with the following:

```rust
use temporalio_client::{
    Client, ClientOptions, Connection,
    envconfig::LoadClientConfigProfileOptions,
};
use temporalio_sdk::{Runtime, Worker, WorkerOptions};

mod workflows;
mod activities;

use crate::workflows::GreetingWorkflow;
use crate::activities::MyActivities;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = Runtime::from_current_tokio(Default::default())?;

    // Set up client connection options, loading from config if available
    let (connection_options, client_options) = ClientOptions::load_from_config(
        LoadClientConfigProfileOptions::default(),
    )?;

    let connection = Connection::connect(connection_options).await?;
    let client = Client::new(connection, client_options)?;

    let worker_options = WorkerOptions::new("my-task-queue")
        .register_activities(MyActivities)
        .register_workflow::<GreetingWorkflow>()?
        .build();

    let mut worker = Worker::new(&runtime, client, worker_options)?;
    worker.run().await?;

    Ok(())
}
```

Open a new terminal and run the Worker with:

```bash
cargo run
```

### 4. Start a Workflow

You can now start a Workflow execution using the client. In a new terminal, run the Workflow with:

```bash
temporal workflow start \
  --type GreetingWorkflow \
  --task-queue my-task-queue \
  --input '"Ziggy"'
```

## Next steps

Now that you have the basics working, explore the following resources to build more sophisticated applications:

- [Develop a Workflow](/develop/rust/workflows/basics) - Learn how to write complex workflow logic
- [Develop an Activity](/develop/rust/activities/basics) - Understand activity patterns and best practices
- [Run a Worker](/develop/rust/workers/worker-process) - Configure and scale workers
- [Using the Temporal Client](/develop/rust/client/temporal-client) - Start workflows and interact with the Temporal Service

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Workers - Rust SDK

Source: https://docs.temporal.io/develop/rust/workers

![Rust SDK Banner](https://docs.temporal.io/img/assets/banner-rust-temporal.png)

## Workers

- [Run a Worker](/develop/rust/workers/worker-process)
- [Serverless Workers](/develop/rust/workers/serverless-workers)

---

# Serverless Workers - Rust SDK

Source: https://docs.temporal.io/develop/rust/workers/serverless-workers

> Write Temporal Workers that run on serverless compute using the Rust SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

Serverless Workers run on compute that Temporal starts and stops for you, rather than on long-lived processes you operate.

For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers).
For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Supported providers

- [**GCP Cloud Run**](/develop/rust/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in.

---

# Serverless Workers on GCP Cloud Run - Rust SDK

Source: https://docs.temporal.io/develop/rust/workers/serverless-workers/cloud-run

> Run a Temporal Worker on a GCP Cloud Run worker pool using the Rust SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

The code on this page is written against `temporalio-sdk` 1.0.0.

On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Rust Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific crate.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).

## Create a versioned Worker 

Build the Worker as you would any long-running Rust Worker, then set `deployment_options` on `WorkerOptions` to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

```rust
use std::str::FromStr;

use temporalio_client::{Client, ClientOptions, Connection, ConnectionOptions, TlsOptions, Url};
use temporalio_common::worker::{
    VersioningBehavior, WorkerDeploymentOptions, WorkerDeploymentVersion,
};
use temporalio_sdk::{Runtime, Worker, WorkerOptions};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let address = std::env::var("TEMPORAL_ADDRESS")?;
    let namespace = std::env::var("TEMPORAL_NAMESPACE")?;
    let api_key = std::env::var("TEMPORAL_API_KEY")?;

    let runtime = Runtime::from_current_tokio(Default::default())?;

    let connection_options = ConnectionOptions::new(Url::from_str(&format!("https://{address}"))?)
        .api_key(api_key)
        .tls_options(TlsOptions::default())
        .build();
    let connection = Connection::connect(connection_options).await?;
    let client = Client::new(connection, ClientOptions::new(namespace).build())?;

    let worker_options = WorkerOptions::new(std::env::var("TEMPORAL_TASK_QUEUE")?)
        .deployment_options(
            WorkerDeploymentOptions::new(
                WorkerDeploymentVersion::builder()
                    .deployment_name("my-app")
                    .build_id("build-1")
                    .build(),
            )
            .use_worker_versioning(true)
            .default_versioning_behavior(VersioningBehavior::Pinned)
            .build(),
        )
        .register_workflow::<MyWorkflow>()?
        .register_activities(MyActivities)
        .build();

    let mut worker = Worker::new(&runtime, client, worker_options)?;
    worker.run().await?;

    Ok(())
}
```

`deployment_name` and `build_id` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage.

The Rust SDK sets the versioning behavior on the Worker rather than per Workflow, so `default_versioning_behavior` covers every Workflow the Worker registers.
Setting it to `VersioningBehavior::Unspecified` is an error at startup.
See [versioning behaviors](/worker-versioning#versioning-behaviors) for what `Pinned` and `AutoUpgrade` mean.

For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/rust/workers/worker-process).

## Configure the Temporal connection 

Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext.
The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace.

Setting `api_key` alone does not enable TLS. `Connection::connect` applies TLS only when `tls_options` is set, so a Temporal Cloud connection needs `.tls_options(TlsOptions::default())` as well, or it fails with `Connecting to HTTPS without TLS enabled`.
To rotate the key without restarting the Worker, call `set_api_key` on the connected Client.

For the shared configuration format that other Temporal tools read, see [Environment configuration](/develop/environment-configuration).

## Package the Worker image 

Compile the Worker in one stage and copy the binary into a runtime image:

```dockerfile
FROM rust:1.92-slim AS build

RUN apt-get update \
    && apt-get install -y --no-install-recommends pkg-config libssl-dev protobuf-compiler libprotobuf-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /src
COPY Cargo.toml ./
COPY src ./src
RUN cargo build --release

FROM debian:bookworm-slim

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY --from=build /src/target/release/my-worker /app/worker
CMD ["/app/worker"]
```

The build stage needs `libprotobuf-dev` as well as `protobuf-compiler`. The compiler package alone installs `protoc` without the well-known type definitions, and the build then fails with `google/protobuf/duration.proto: File not found`.

The runtime stage needs `ca-certificates`. The Worker reads TLS roots from the operating system's certificate store, and `debian:bookworm-slim` ships without one.

## Keep Activities safe across scale-in 

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing.
An instance running a long Activity can be stopped mid-execution.

Use [Activity Heartbeats](/develop/rust/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over.

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability 

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics and telemetry, see the runtime options you pass to `Runtime::from_current_tokio` and the [SDK metrics reference](/references/sdk-metrics).

---

# Run a Worker - Rust SDK

Source: https://docs.temporal.io/develop/rust/workers/worker-process

> Create and run a Temporal Worker using the Rust SDK.

The code on this page is written against `temporalio-sdk` 1.0.0.

## Create and run a Worker 

Start with these dependencies.
The `#[workflow]` and `#[activities]` macros expand to code that refers to the `temporalio-workflow` and `futures` crates, so both must be direct dependencies of your crate even though your own code never names them:

```toml
[dependencies]
temporalio-sdk = "1.0.0"
temporalio-client = "1.0.0"
temporalio-common = "1.0.0"
temporalio-macros = "1.0.0"
temporalio-workflow = "1.0.0"
futures = "0.3"
tokio = { version = "1", features = ["full"] }
```

A Worker needs a `Runtime` and a connected [Temporal Client](/develop/rust/client/temporal-client).
Build `WorkerOptions` with the Task Queue to poll, register the Workflows and Activities the Worker can execute, then call `run()`:

```rust
use std::str::FromStr;

use temporalio_client::{Client, ClientOptions, Connection, ConnectionOptions, Url};
use temporalio_sdk::{Runtime, Worker, WorkerOptions};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = Runtime::from_current_tokio(Default::default())?;

    let connection_options =
        ConnectionOptions::new(Url::from_str("http://localhost:7233")?).build();
    let connection = Connection::connect(connection_options).await?;
    let client = Client::new(connection, ClientOptions::new("default").build())?;

    let worker_options = WorkerOptions::new("my-task-queue")
        .register_workflow::<GreetingWorkflow>()?
        .register_activities(GreetingActivities)
        .build();

    let mut worker = Worker::new(&runtime, client, worker_options)?;
    worker.run().await?;

    Ok(())
}
```

`run()` polls until the Worker shuts down.
`ClientOptions::new()` takes the Namespace, and `Connection::connect()` takes the server address.

## Register Workflows and Activities 

All Workers polling the same Task Queue must register the same Workflow Types and Activity Types.
A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue.
A Worker that receives a Task for a type it did not register fails that Task.

Register each Workflow with `register_workflow::<T>()` and each Activity implementer with `register_activities()`.
`register_workflow` returns a `Result`, so it needs `?` or other error handling:

```rust
let worker_options = WorkerOptions::new("my-task-queue")
    .register_workflow::<GreetingWorkflow>()?
    .register_workflow::<OrderWorkflow>()?
    .register_activities(GreetingActivities)
    .build();
```

`register_activities()` takes an instance, so Activities can share state such as a database client through the fields of that value.

## Connect to Temporal Cloud 

To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials.
See [Connect to Temporal Cloud](/develop/rust/client/temporal-client#connect-to-temporal-cloud) for setup instructions.

## Configure Worker options 

`WorkerOptions` controls the Task Queue, cache size, poller behavior, and slot allocation, including `max_cached_workflows`, `workflow_task_poller_behavior`, and `tuner`.
Set `client_identity_override` to give the Worker an identity that is more useful than the default of `{pid}@{hostname}`.
The defaults work for most cases.

To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference).

## Run a versioned Worker 

Set a Worker Deployment Version and enable versioning in `deployment_options`, then set the default versioning behavior for the Workflows on the Worker.

```rust
use temporalio_common::worker::{
    VersioningBehavior, WorkerDeploymentOptions, WorkerDeploymentVersion,
};

let worker_options = WorkerOptions::new("my-task-queue")
    .deployment_options(
        WorkerDeploymentOptions::new(
            WorkerDeploymentVersion::builder()
                .deployment_name("my-app")
                .build_id("1.0")
                .build(),
        )
        .use_worker_versioning(true)
        .default_versioning_behavior(VersioningBehavior::Pinned)
        .build(),
    )
    .register_workflow::<GreetingWorkflow>()?
    .register_activities(GreetingActivities)
    .build();
```

The Rust SDK sets the versioning behavior on the Worker, not per Workflow, so `default_versioning_behavior` covers every Workflow the Worker registers.
Setting it to `VersioningBehavior::Unspecified` is an error at startup.

See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out.

## Shut down a Worker 

`run()` borrows the Worker mutably, so call `shutdown_handle()` before starting the Worker.
Calling the handle it returns initiates shutdown, which stops polling for new Tasks and lets in-flight Tasks finish.

```rust
let mut worker = Worker::new(&runtime, client, worker_options)?;
let shutdown = worker.shutdown_handle();

tokio::spawn(async move {
    tokio::signal::ctrl_c().await.expect("failed to listen for ctrl-c");
    shutdown();
});

worker.run().await?;
```

See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities.

---

# Workflows - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows

![Rust SDK Banner](https://docs.temporal.io/img/assets/banner-rust-temporal.png)

## Workflows

- [Workflow basics](/develop/rust/workflows/basics)
- [Child Workflows](/develop/rust/workflows/child-workflows)
- [Continue-As-New](/develop/rust/workflows/continue-as-new)
- [Message passing](/develop/rust/workflows/message-passing)
- [Cancellation](/develop/rust/workflows/cancellation)
- [Timers](/develop/rust/workflows/timers)
- [Timeouts](/develop/rust/workflows/timeouts)

---

# Workflow basics - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows/basics

> This section explains how to implement Workflows with the Rust SDK

## Develop a Workflow 

Workflows are the fundamental unit of a Temporal Application and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal Rust SDK programming model, a Workflow Definition is made of a Workflow struct and associated methods decorated with macros.

A Workflow is defined by:

1. A struct that holds the Workflow state
2. A `#[run]` method that contains the main Workflow logic
3. An optional `#[init]` method that initializes the Workflow
4. Optional `#[signal]`, `#[query]`, and `#[update]` methods for external interaction

```rust
use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{WorkflowResult, WorkflowContextView, WorkflowContext};

#[workflow]
pub struct GreetingWorkflow {
    name: String,
}

#[workflow_methods]
impl GreetingWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, name: String) -> Self {
        Self { name }
    }

    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());
        Ok(format!("Hello, {}!", name))
    }
}
```

The `#[workflow]` macro marks the struct as a Workflow. The `#[workflow_methods]` macro is applied to the `impl` block containing the Workflow methods.

### Workflow struct 

The Workflow struct holds the state of your Workflow Execution. This state is persisted and recovered during replays. All fields in a Workflow struct should be serializable.

### Workflow initialization 

The `#[init]` method is optional and is called when the Workflow first starts. It receives the initial Workflow input parameters and initializes the Workflow struct:

```rust
#[init]
fn new(ctx: &WorkflowContextView, name: String, age: u32) -> Self {
    Self {
        name,
        age,
        started_at: ctx.start_time(),
    }
}
```

The `#[init]` method receives a `WorkflowContextView`, which provides read-only access to Workflow execution information.

#### Use Workflow constructors

Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. However, if you use the `#[init]` annotation on your constructor, you can give it the same [Workflow parameters](/develop/rust/workflows/basics#workflow-parameters) as your `#[run]`.

The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/rust/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `#[run]` method. That always happens, whether or not you use the `#[init]` annotation.

Here's an example. Notice that the constructor and `get_greeting` must have the same parameters:

```rust
#[workflow_methods]
impl WorkflowRunSeesWorkflowInitWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, workflow_input: MyWorkflowInput) -> Self {
        Self {
            name_with_title: format!("Knight {}", workflow_input.name),
            title_has_been_checked: false,
        }
    }

    #[run]
    pub async fn get_greeting(
        ctx: &mut WorkflowContext<Self>,
        _workflow_input: MyWorkflowInput,
    ) -> WorkflowResult<String> {
        ctx.wait_condition(|state| state.title_has_been_checked)
            .await?;

        let name_with_title = ctx.state(|state| state.name_with_title.clone());
        Ok(format!("Hello, {}", name_with_title))
    }
}
```

### Run method 

The `#[run]` method is required and contains the main Workflow logic. It:

- Must be `async`
- Must be `public`
- Receives a mutable `WorkflowContext<Self>`
- Returns `WorkflowResult<T>` where T is the Workflow return type
- Executes exactly once per Workflow execution

```rust
#[run]
pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
    // Execute activities, timers, child workflows, etc.
    let result = ctx
        .execute_activity(
            MyActivities::greet,
            name,
            ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
        )
        .await?;

    Ok(result)
}
```

## Define Workflow parameters 

Temporal Workflows may have any number of custom parameters. However, we strongly recommend that structs are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All Workflow Definition parameters must be serializable. For more reasons, see [Workflow parameters](/workflow-definition#workflow-parameters).

A method annotated with `#[init]` can have any number of parameters. We recommend passing a single struct that contains all the input fields:

```rust
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
pub struct ProcessingInput {
    pub data: Vec<String>,
    pub timeout_seconds: u32,
}

#[workflow]
pub struct ProcessingWorkflow {
    data: Vec<String>,
    timeout_seconds: u32,
}

#[workflow_methods]
impl ProcessingWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, input: ProcessingInput) -> Self {
        Self {
            data: input.data,
            timeout_seconds: input.timeout_seconds,
        }
    }

    #[run]
    pub async fn run(_ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        // Use the initialized state
        Ok("Processing complete".to_string())
    }
}
```

All Workflow input should be serializable by `serde`.

## Define Workflow return parameters 

Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error.

The return type of a Workflow is `WorkflowResult<T>` where `T` implements `Serialize`. Success is represented by `Ok(value)` and failure by `Err(...)`:

```rust
#[run]
async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<ProcessingResult> {
    // Can return a complex result type
    let result = ProcessingResult {
        status: "completed".to_string(),
        records_processed: 100,
    };

    Ok(result)
}
```

## Customize your Workflow Type 

Workflows have a Type that is referred to as the Workflow name. By default, the Workflow type is the name of the Workflow struct. You can customize it by providing a `name` parameter to the `#[run]` macro:

```rust
#[workflow]
pub struct GreetingWorkflow {
    name: String,
}

#[workflow_methods]
impl GreetingWorkflow {
    #[run(name = "my-custom-workflow")]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        Ok(ctx.state(|s| format!("Hello, {}!", s.name)))
    }
}
```

The Workflow Type defaults to the struct name if not specified. For example, this Workflow would have the type `GreetingWorkflow`:

```rust
#[workflow]
pub struct GreetingWorkflow {
    // ...
}
```

## Workflow logic requirements 

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). For non-deterministic operations like API calls, and database queries, use [Activities](/develop/rust/activities/basics).

A Workflow Definition must be deterministic because the Temporal Server may replay your Workflow to reconstruct its state. This means:

### Don't use nondeterministic functions

- No direct system time access - use `ctx.workflow_time()` instead of `SystemTime::now()`
- No nondeterministic random number generation - use `ctx.random()` instead
- No external I/O (network, filesystem, etc.) - perform these in Activities instead
- No UUID generation via random means - use `ctx.uuid4()` instead
- Do not use `tokio` or `futures` concurrency primitives directly in Workflow code. Many of them, like `tokio::select!`, `tokio::spawn`, `futures::select!`, introduce non-deterministic behavior that will break Workflow replay.

Instead, use the deterministic wrappers provided in `temporalio_sdk::workflows`:
    - `select!` — deterministic select (polls in declaration order)
    - `join!` — deterministic join for a fixed number of futures
    - `join_all` — deterministic join for a dynamic collection of futures

### Use Workflow-safe primitives

The Rust SDK provides:

- `ctx.timer()` - Wait for a duration
- `ctx.wait_condition(closure)` - Wait until a condition is true
- `workflows::select!` - Deterministic select statement
- `ctx.execute_activity()` - Execute Activities
- `ctx.execute_local_activity()` - Execute local Activities
- `ctx.start_child_workflow()` - Start Child Workflows
- `ctx.cancelled()` - Wait for Workflow cancellation

```rust
use std::time::Duration;

#[run]
pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
    // Good - deterministic timer
    ctx.timer(
        TimerOptions::builder(Duration::from_secs(60))
            .summary("important timer".to_owned())
            .build(),
    )
    .await;

    // Good - deterministic wait for condition
    ctx.wait_condition(|s| s.data.len() >= 3).await?;

    // Bad - nondeterministic sleep
    // tokio::time::sleep(Duration::from_secs(10)).await;

    // Bad - nondeterministic time
    // SystemTime::now()

    Ok("Done".to_string())
}
```

## Access Workflow State 

Use `ctx.state()` for read-only access and `ctx.state_mut()` for mutable access to your Workflow state:

```rust
#[run]
pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
    // Read-only access
    let name = ctx.state(|s| s.name.clone());

    // Mutable access (for signal handlers or update handlers)
    // Available in sync methods

    Ok(name)
}
```

In synchronous [`Signal`](/develop/rust/workflows/message-passing#signals) and [`Update`](/develop/rust/workflows/message-passing#updates) handlers, you can mutate state directly via `&mut self`.

## Workflow return types 

The `#[run]` method must return `WorkflowResult<T>`. This is a type alias for `Result<T, WorkflowTermination>`.

For application failures, construct an `ApplicationFailure` and convert it into `WorkflowTermination`:

```rust
use temporalio_sdk::ApplicationFailure;

#[run]
pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
    if some_validation_fails {
        return Err(ApplicationFailure::new("validation_failed: Input is invalid").into());
    }

    Ok("Success".to_string())
}
```

Workflow errors will cause the Workflow Execution to fail and the error details will be available to clients.

---

# Cancellation - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows/cancellation

You can interrupt a Workflow Execution in one of the following ways:

- [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution.
- [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution.

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the
Workflow is stuck and cannot be canceled normally.

## Cancel a Workflow Execution 

Canceling a Workflow provides a graceful way to stop Workflow Execution. This action resembles sending a `SIGTERM` to a
process.

- The system records a `WorkflowExecutionCancelRequested` event in the Event History.
- A Workflow Task gets scheduled to process the cancelation.
- The Workflow code can handle the cancelation and execute any cleanup logic.
- The system doesn't forcefully stop the Workflow.

To cancel a Workflow Execution in Rust, use the `cancel` method on the Workflow handler:

```rust
let handle = client.start_workflow(
    GreetingsWorkflow::run, 
    (), 
    WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build()
).await?;

handle.cancel(WorkflowCancelOptions::builder().reason("No longer needed").build()).await?;
```

### Cancel an Activity from a Workflow 

Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat
Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate
Activity is executed, the Activity Execution should send Heartbeats and set a
[Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is
still working.

When an Activity is canceled, an error is returned in the Activity at the next available opportunity. If cleanup logic
needs to be performed, it can be done when handling the cancellation error. However, for the Activity to appear canceled
the error must be propagated.

Example of a cancellable Activity in Rust:

```rust
#![allow(unreachable_pub)]
use temporalio_macros::activities;
use temporalio_sdk::activities::{ActivityContext, ActivityError};

pub struct CancellationActivities;

#[activities]
impl CancellationActivities {
    #[activity]
    pub async fn long_running_cancellable_activity(
        ctx: ActivityContext,
        _input: (),
    ) -> Result<String, ActivityError> {
        loop {
            if ctx.is_cancelled() {
                return Err(ActivityError::cancelled());
            }
            ctx.record_heartbeat(()).await?;
            tokio::time::sleep(std::time::Duration::from_millis(200)).await;
        }
    }

    #[activity]
    pub async fn cleanup(_ctx: ActivityContext, _input: ()) -> Result<String, ActivityError> {
        Ok("cleanup done".to_string())
    }
}
```

Canceling the Activity from a Workflow:

```rust
fn activity_opts() -> ActivityOptions {
    ActivityOptions::with_start_to_close_timeout(Duration::from_secs(300))
        .heartbeat_timeout(Duration::from_secs(5))
        .build()
}

#[workflow]
#[derive(Default)]
pub struct CancellationWorkflow;

#[workflow_methods]
impl CancellationWorkflow {
    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>, _input: ()) -> WorkflowResult<String> {
        let result = ctx
            .execute_activity(
                CancellationActivities::long_running_cancellable_activity,
                (),
                activity_opts(),
            )
            .await;

        match result {
            Ok(value) => Ok(value),
            Err(ActivityExecutionError::Cancelled(_)) => {
                let reason = ctx.cancellation_token().reason().unwrap_or_default();
                let cleanup_result = ctx
                    .execute_activity(
                        CancellationActivities::cleanup,
                        (),
                        ActivityOptions::with_start_to_close_timeout(Duration::from_secs(10))
                            .cancellation_token(WorkflowCancellationToken::new())
                            .build(),
                    )
                    .await?;

                Ok(format!("Cancelled (reason={reason}), {cleanup_result}"))
            }
            Err(err) => Err(err.into()),
        }
    }
}
```

## Terminate a Workflow Execution 

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Event History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

To terminate a Workflow Execution in Rust, use the `terminate` method on the client.

```rust
let handle = client.start_workflow(
    GreetingsWorkflow::run, 
    (), 
    WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build()
).await?;

handle.terminate(WorkflowTerminateOptions::builder()
    .reason("Emergency shutdown")
    .build()
).await?;

```

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Child Workflows - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows/child-workflows

> Start a Child Workflow Execution and set a Parent Close Policy using the Rust SDK. Manage Child Workflow Events.

This page shows how to do the following:

- [Start a Child Workflow execution](#start-child-workflow)
- [Set a Parent Close Policy](#parent-close-policy)

## Start a Child Workflow execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In Rust, awaiting `ctx.start_child_workflow()` waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call resolves.

To start a Child Workflow in Rust, use `ctx.start_child_workflow()`:

```rust
use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{ChildWorkflowOptions, WorkflowContext, WorkflowContextView, WorkflowResult};

// child workflow
#[workflow]
pub struct ComposeGreetingWorkflow {
    pub name: String,
}

#[workflow_methods]
impl ComposeGreetingWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, name: String) -> Self {
        Self { name }
    }

    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());
        Ok(format!("Hello from child: {}", name))
    }
}

// parent workflow
#[workflow]
pub struct GreetingWorkflow {
    pub name: String,
}

#[workflow_methods]
impl GreetingWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, name: String) -> Self {
        Self { name }
    }

    #[run]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.name.clone());

        let started = ctx
            .start_child_workflow(
                ComposeGreetingWorkflow::run,
                name,
                ChildWorkflowOptions::workflow_id("greeting-child-en".to_string()),
            )
            .await?;

        let result = started.result().await?;

        Ok(format!("ComposeGreetingWorkflow result: {result}"))
    }
}
```

### Specify Child Workflow options

Use [`ChildWorkflowOptions`](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/struct.ChildWorkflowOptions.html) to customize Child Workflow behavior.

### Execute multiple Child Workflows in parallel

You can start multiple Child Workflows and wait for all of them:

```rust
pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<Vec<Option<String>>> {
    let name = ctx.state(|s| s.name.clone());

    let en_greeting_child = ctx
        .start_child_workflow(
            ComposeEnGreetingWorkflow::run,
            name.clone(),
            ChildWorkflowOptions::workflow_id("greeting-child-en".to_string()),
        )
        .await?;
    
    let es_greeting_child = ctx
        .start_child_workflow(
            ComposeEsGreetingWorkflow::run,
            name,
            ChildWorkflowOptions::workflow_id("greeting-child-es".to_string()),
        )
        .await?;
    
    let en_result = en_greeting_child.result().await;
    let es_result = es_greeting_child.result().await;

    let combined = vec![en_result, es_result];

    print!("Combined greetings: {:?}", combined);

    ...
}
```

Both child Workflows run in parallel and the parent waits for both to complete.

## Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy is set to terminate the Child Workflow Execution.

Set Parent Close Policy using the [`parent_close_policy`](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/enum.ParentClosePolicy.html) field in `ChildWorkflowOptions`:

```rust
use temporalio_sdk::ParentClosePolicy;

let es_greeting_child = ctx
    .start_child_workflow(
        ComposeEsGreetingWorkflow::run,
        name,
        ChildWorkflowOptions::builder()
            .workflow_id("greeting-child-es".to_string())
            .parent_close_policy(ParentClosePolicy::Abandon)
            .build(),
    )
    .await?;
```

### Parent Close Policy options

- `Terminate` (default) - The Child Workflow will be terminated immediately when the parent closes
- `Abandon` - The Child Workflow will continue running even if the parent closes
- `RequestCancel` - The Child Workflow will receive a cancellation request when the parent closes

---

# Continue-As-New - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows/continue-as-new

> Use Temporal's Continue-As-New in Rust to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters.

This page covers the following for Rust developers:

- [What is Continue-As-New?](#what)
- [Use Continue-As-New](#how)
- [When is it right to Continue-As-New?](#when)

## What is Continue-As-New? 

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow execution close successfully and creates a new Workflow execution. You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History.
It also receives your Workflow's usual parameters.

## Use Continue-As-New with the Rust SDK 

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically passed as a parameter or stored in the Workflow struct.

Inside your Workflow, call `ctx.continue_as_new()` and propagate its result:

```rust
use std::time::Duration;

use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{ActivityOptions, ContinueAsNewOptions, WorkflowContext, WorkflowResult};

use crate::activities::MyActivities;

#[workflow]
#[derive(Default)]
pub struct GreetingWorkflow;

#[workflow_methods]
impl GreetingWorkflow {
    #[run(name = "greeting-workflow-1")]
    pub async fn run(ctx: &mut WorkflowContext<Self>, name: String) -> WorkflowResult<String> {
        let greeting = ctx
            .execute_activity(
                MyActivities::greet,
                name.clone(),
                ActivityOptions::start_to_close_timeout(Duration::from_secs(30)),
            )
            .await?;

        println!("{}", greeting);

        if name == "Ziggy" {
            Ok(greeting)
        } else {
            ctx.continue_as_new("New Name".to_string(), ContinueAsNewOptions::default())?;
        }
    }
}
```

The `ctx.continue_as_new()` method accepts the input to pass to the next Workflow Run.

## When is it right to Continue-As-New with the Rust SDK? 

Use Continue-as-New when your Workflow might encounter degraded performance or [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New. Call `ctx.continue_as_new_suggested()` to check if it's time.

## Test Continue-As-New with the Rust SDK 

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, if you have an internal value like `test_continue_as_new == True`, this sample takes a variable called `max_history_length` and that can be set to a small value. A helper method in the Workflow impl checks it each time it considers using Continue-as-New:

```rust
use serde::{Deserialize, Serialize};
use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{WorkflowContext, WorkflowResult};

#[derive(Serialize, Deserialize)]
pub struct GreetingInput {
    pub name: String,
    pub max_history_length: u32,
}

...

#[workflow_methods]
impl GreetingWorkflow {
    #[run]
    pub async fn run(
        ctx: &mut WorkflowContext<Self>,
        input: GreetingInput,
    ) -> WorkflowResult<String> {
        // your Workflow code here

        if Self::should_continue_as_new(ctx, input.max_history_length) {
            // Continue as new
        }
    }

    fn should_continue_as_new(
        ctx: &WorkflowContext<Self>,
        max_history_length: u32,
    ) -> bool {
        if ctx.continue_as_new_suggested() {
            return true;
        }

        // For testing
        if max_history_length > 0 && ctx.history_length() > max_history_length {
            return true;
        }

        false
    }
}
```

## Best practices 

1. Pass all necessary state: When continuing as new, include all state the next run needs.
2. Use meaningful iteration markers: Include iteration numbers or timestamps to track progress.
3. Test your state passing: Ensure parameters serialize and deserialize correctly.
4. Don't continue-as-new too frequently: It's better to have some Event History than to continue-as-new on every execution.
5. Consider batch sizes: Find the right balance between batch size and number of continues as new.

---

# Workflow message passing - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows/message-passing

> Develop with Queries, Signals, and Updates with the Temporal Rust SDK.

A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates.
The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values.

Temporal Clients use messages to read Workflow state and control its execution.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic. This page introduces these features for the Temporal Rust SDK.

## Write message handlers 

Follow these guidelines when writing your message handlers:

- Message handlers are defined as methods on your Workflow struct and registered with the Workflow runtime.
- The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion).
- Prefer structs to multiple input parameters to allow for forward-compatible changes.

### Query handlers 

A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution:

```rust
// workflows.rs
use std::collections::HashMap;

use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{WorkflowContext, WorkflowContextView, WorkflowResult};

#[derive(serde::Serialize, serde::Deserialize, Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Language {
    Chinese,
    English,
    French,
}

#[derive(serde::Serialize, serde::Deserialize)]
pub struct GetLanguagesInput {
    pub include_unsupported: bool,
}

#[workflow]
pub struct GreetingsWorkflow {
    pub greetings: HashMap<Language, String>,
}

#[workflow_methods]
impl GreetingsWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView) -> Self {
        let mut greetings = HashMap::new();
        greetings.insert(Language::Chinese, "你好，世界".to_string());
        greetings.insert(Language::English, "Hello, world".to_string());
        
        Self { greetings }
    }

    #[run(name = "greetings-workflow-10")]
    pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        let name = ctx.state(|s| s.greetings.clone());
        Ok(format!("Hola: {:?}", name))
    }

    #[query]
    pub fn get_languages(
        &self,
        _ctx: &WorkflowContextView,
        input: GetLanguagesInput,
    ) -> Vec<Language> {
        if input.include_unsupported {
            vec![Language::Chinese, Language::English, Language::French]
        } else {
            self.greetings.keys().copied().collect()
        }
    }
}
```

```rust
// main.rs
let wf_handle = client
    .start_workflow(
        GreetingsWorkflow::run,
        (),
        WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build(),
    )
    .await?;

let supported_languages = wf_handle
    .query(
        GreetingsWorkflow::get_languages,
        GetLanguagesInput {
            include_unsupported: true,
        },
        WorkflowQueryOptions::default(),
    )
    .await?;
```

- Query handlers can't mutate Workflow state.
- Query handlers can't perform async operations, like executing Activities.

### Signal handlers 

A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow:

```rust
#[derive(serde::Serialize, serde::Deserialize)]
pub struct ApproveInput {
    pub name: String,
}

// Other structs
...

#[workflow_methods]
impl GreetingsWorkflow {
    // Other Workflow logic
    ...

    #[signal]
    pub fn approve(&mut self, _ctx: &mut SyncWorkflowContext<Self>, input: ApproveInput) {
        self.approved_for_release = true;
        self.approver_name = Some(input.name);
    }
}
```

```rust
// main.rs
wf_handle
    .signal(
        GreetingsWorkflow::approve,
        ApproveInput {
            name: "Ziggy".to_string(),
        },
        WorkflowSignalOptions::default(),
    )
    .await?;
```

* Signal handlers do not return values.
* They can trigger async work (Activities, timers) depending on SDK capabilities.

### Update handlers and validators 

An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution. It can change the Workflow state, control its flow, and return a result. The sender must wait until the Worker accepts or rejects the Update. The sender may wait further to receive a returned value or an exception if something goes wrong:

```rust
// workflows.rs
...

#[derive(serde::Serialize, serde::Deserialize)]
pub struct SetLanguageInput {
    pub language: Language,
}

...

#[workflow_methods]
impl GreetingsWorkflow {
    // Other Workflow logic
    ...

    #[update]
    pub fn set_language(
        &mut self,
        _ctx: &mut SyncWorkflowContext<Self>,
        input: SetLanguageInput,
    ) -> Language {
        let previous_language = self.language;
        self.language = input.language;
        
        previous_language
    }

    #[update_validator(set_language)]
    fn validate_set_language(
        &self,
        _ctx: &WorkflowContextView,
        input: &SetLanguageInput,
    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
        if !self.greetings.contains_key(&input.language) {
            Err("Not a valid language".into())
        } else {
            Ok(())
        }
    }
}
```

- About validators:
    - Use validators to reject an Update before it is written to History. Validators are always optional. If you don't need to reject Updates, you can skip them.
    - If you choose to create validators for your Updates, they will reject Updates before they're applied.
- Accepting and rejecting Updates with validators:
    - To reject an Update, raise an exception of any type in the validator.
    - Without a validator, Updates are always accepted.
- Validators and Event History:
    - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic.
    - When a Validator raises an error, the Update is rejected and `WorkflowExecutionUpdateAccepted` won't be added to the Event History. The caller receives an "Update failed" error.
- Update and Signal handlers can be async, letting you use Activities, Child Workflows, and more. See [Async handlers](#async-handlers) for safe usage guidelines.

## Send messages 

To send Queries, Signals, or Updates, use a Workflow handle from the client:

```rust
let client = Client::new(connection, ClientOptions::new("default").build())?;

let wf_handle = client.start_workflow(
    GreetingsWorkflow::run, 
    (), 
    WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build()
).await?;
```

### Send a Query 

```rust
let supported_languages = wf_handle.query(
    GreetingsWorkflow::get_languages, 
    GetLanguagesInput { include_unsupported: true }, 
    WorkflowQueryOptions::default()
).await?;
```

- Sending a Query doesn’t add events to a Workflow's Event History.
- You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period. This includes Workflows that have completed, failed, or timed out. Querying terminated Workflows is not safe and, therefore, not supported.
- A Worker must be online and polling the Task Queue to process a Query.

### Send a Signal 

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed.

#### From a Client

```rust
wf_handle.signal(
    GreetingsWorkflow::approve, 
    ApproveInput { name: "Ziggy".to_string() }, 
    WorkflowSignalOptions::default()
).await?;
```

- The call returns when the server accepts the Signal; it does not wait for the Signal to be delivered to the Workflow Execution.

#### From a Workflow

```rust
ctx.external_workflow("workflow-id-1", Some("run-id-1".into()))
    .signal(
        GreetingsWorkflow::approve,
        ApproveInput { name: "Ziggy".to_string() },
        SignalWorkflowOptions::default(),
    )
    .await?;
```

### Signal-With-Start 

```rust
let wf_handle = client.signal_with_start_workflow(
    GreetingsWorkflow::run,
    (),
    GreetingsWorkflow::approve,
    ApproveInput {
        name: "Ziggy".to_string(),
    },
    WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build(),
).await?;
```

### Send an Update 

An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result.

A client sending an Update must wait until the Server delivers the Update to a Worker. Workers must be available and responsive. If you need a response as soon as the Server receives the request, use a Signal instead. You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity.

- `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation.
- `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished.

To send an Update to a Workflow Execution, you can call `execute_update` and wait for the Update to complete.

This code fetches an Update result:

```rust
let previous_language = wf_handle.execute_update(
    GreetingsWorkflow::set_language,
    SetLanguageInput { language: Language::French },
    WorkflowExecuteUpdateOptions::default()
).await?;
```

#### Update-With-Start

You can also send `start_update` to receive an `UpdateHandle` as soon as the Update is accepted.

- Use this `UpdateHandle` later to fetch your results.
- Async Update handlers normally perform long-running asynchronous operations, such as executing an Activity.
- `start_update` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete.

For example:

```rust
let update_handle = main_wf_handle.start_update(
    GreetingsWorkflow::set_language, 
    SetLanguageInput { language: Language::French }, 
    WorkflowStartUpdateOptions::default()
).await?;
```

- Updates are synchronous and return results.
- Worker must accept the Update before it proceeds.

## Message handler patterns 

### Async handlers 

Signal and Update handlers can be `async fn` as well as `fn`. Using `async fn` allows you to use await with Activities, Child Workflows, Timers, and so on. This expands the possibilities for what can be done by a handler, but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls.

To use `async fn` handlers safely, understand what can go wrong. See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, the Safe message handlers sample and the sections below.

The following code executes an Activity that makes a network call to a remote service:

```rust
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Language {
    Arabic,
    Chinese,
    English,
    French,
    Hindi,
    Spanish,
}

pub struct GreetingActivities;

#[activities]
impl GreetingActivities {
    #[activity]
    pub async fn call_greeting_service(_ctx: ActivityContext, to_language: Language) -> Result<Option<String>, ActivityError> {
        // Pretend that we are calling a remote service.
        sleep(Duration::from_millis(200)).await;

        let greetings = HashMap::from([
            (Language::Arabic, "مرحبا بالعالم".to_string()),
            (Language::Chinese, "你好，世界".to_string()),
            (Language::English, "Hello, world".to_string()),
            (Language::French, "Bonjour, monde".to_string()),
            (Language::Hindi, "नमस्ते दुनिया".to_string()),
            (Language::Spanish, "Hola mundo".to_string()),
        ]);

        Ok(greetings.get(&to_language).cloned())
    }
}
```

After updating the code to use an `async fn`, your Update handler can schedule an Activity and await the result. Although an `async fn` Signal handler can also execute an Activity, using an Update handler allows the client to receive a result or error once the Activity completes.

This lets your client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, and so on. Here's how you could start that Activity from within your Workflow:

```rust
#[update]
async fn set_language_activity(
    ctx: &mut WorkflowContext<Self>,
    language: Language,
) -> Result<Language, Box<dyn std::error::Error + Send + Sync>> {
    let needs_greeting = ctx.state(|s| !s.greetings.contains_key(&language));

    if needs_greeting {
        ctx.wait_condition(|s| !s.approved_for_release).await?;
        ctx.state_mut(|s| {
            s.approved_for_release = true;
        });

        let greeting = ctx
            .execute_activity(
                GreetingActivities::call_greeting_service,
                Language::French,
                ActivityOptions::start_to_close_timeout(Duration::from_secs(10)),
            )
            .await;

        ctx.state_mut(|s| {
            s.approved_for_release = false;
        });

        let greeting = greeting?;
        if let Some(greeting) = greeting {
            ctx.state_mut(|s| {
                s.greetings.insert(language, greeting);
            });
        }
    }

    let previous_language = ctx.state(|s| s.language);

    ctx.state_mut(|s| {
        s.language = language;
    });

    Ok(previous_language)
}
```

### Add wait conditions to block

Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue. You can use `ctx.wait_condition` to prevent the code from proceeding until a condition is true. You specify the condition by passing a function that returns a boolean and you can optionally set a timeout. This is an important feature that helps you control your handler logic.

Here are three important use cases for `ctx.wait_condition`:

- Wait for a Signal or Update to arrive.
- Wait in a handler until it's appropriate to continue.
- Wait in the main Workflow until all active handlers have finished.

It's common to use `ctx.wait_condition` to wait for a particular Signal or Update to be sent by a client. In your Workflow, you will have something like:

```rust
#[run]
pub async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
    let name = ctx.state(|s| s.greetings.clone());

    ctx.wait_condition(|s| !s.approved_for_release).await?;

    // Other Workflow logic
    ...
}
```

### Ensure handlers finish before completion

Workflow wait conditions can ensure your handler completes before a Workflow finishes. When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result.

The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying retrieve Update results. Use `ctx.wait_condition` to address this problem and allow your Workflow to end smoothly.

---

# Workflow Timeouts - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows/timeouts

> Optimize Workflow Execution with the Temporal Rust SDK by configuring Workflow Timeouts and Retry Policies.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

In most cases, **Workflow Timeouts are not recommended**. Temporal Workflows are designed to be long-running and resilient. Setting timeouts can unnecessarily limit their ability to tolerate delays or extended processing.

If you need to trigger logic after a specific duration, use a **Timer inside the Workflow** instead of a timeout.

You can configure Workflow timeouts when starting a Workflow Execution.

Available timeouts include:

- [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout): Maximum total time a Workflow Execution can run.
- [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout): Maximum time a single Workflow Run can last.
- [Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout): Maximum time a Worker can take to complete a Workflow Task.

In Rust, you set these via `WorkflowStartOptions` when starting or executing a Workflow.

Available fields:

- `execution_timeout`
- `run_timeout`
- `task_timeout`

```rust
let wf_handle = client.start_workflow(
    GreetingsWorkflow::run, 
    (), 
    WorkflowStartOptions::new(
        "my-task-queue", 
        "greetings-workflow-10",
    )
    // Set timeouts
    .execution_timeout(Duration::from_secs(3600))
    .run_timeout(Duration::from_secs(600))
    .task_timeout(Duration::from_secs(10))
    .build()
).await?;
```

## Workflow retries 

A Retry Policy can be used alongside timeouts to control how Workflow Executions are retried after failure.

Workflow Executions do **not retry by default**. Retry Policies should only be applied when restarting the entire Workflow is safe and intentional.

To enable retries, configure a [Retry Policy](/encyclopedia/retry-policies) when starting the Workflow.

```rust
let wf_handle = client.start_workflow(
    GreetingsWorkflow::run, 
    (), 
    WorkflowStartOptions::new(
        "my-task-queue", 
        "greetings-workflow-10",
    )
    .retry_policy(
        RetryPolicy::builder()
            .initial_interval(Duration::from_secs(1))
            .backoff_coefficient(2.0)
            .maximum_interval(Duration::from_secs(100))
            .maximum_attempts(5)
            .non_retryable_error_types(["NonRetryableError"])
            .build(),
    )
    .build()
).await?;
```

Retry Policies define retry behavior such as backoff intervals, maximum attempts, and retry conditions.

---

# Timers - Rust SDK

Source: https://docs.temporal.io/develop/rust/workflows/timers

> Set durable Timers in a Workflow using the Rust SDK. Timers persist through Worker and Temporal Service downtime.

A Workflow can set a Durable Timer for a fixed time period. In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`.

A Workflow can sleep for days, months, or even years. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing.

Sleeping is a resource-light operation: it doesn't tie up the process, and you can run millions of Timers off a single Worker.

To set a Timer in Rust, use the `timer()` function and pass the duration you want to wait before continuing.

```rust
ctx.timer(
    TimerOptions::builder(Duration::from_secs(60))
        .summary("important timer".to_owned())
        .build(),
)
.await;
```

---

# Safely deploying changes to Workflow code

Source: https://docs.temporal.io/develop/safe-deployments

> Safely deploy changes to existing Workflow code by validating first for determinism errors before deploying to production.

Making changes safely to existing Workflow code requires care. Your Workflow Definition--as opposed to your Activity code--must be [deterministic](/workflow-definition#deterministic-constraints). This means your changes to it have to be as well. Changes to your Workflow code that qualify as non-deterministic need to be protected by either using [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) to pin your Workflows to specific code revisions, or by using the [patching APIs](/workflow-definition#workflow-versioning) within your Workflow code.

> **📝 Note:**
> We strongly recommend using Worker Versioning as users see improved error rates when adopting it.

In this article, we’ll provide some advice on how you can safely validate changes to your Workflow code, ensuring that you won’t experience unexpected non-determinism errors in production when rolling them out.

> **⚠️ Caution:**
> Eager start does not respect Worker versioning. An eagerly started Workflow may run on any available local Worker even if that Worker is not the Current or Ramping version of its Worker deployment.

## Use replay testing before and during your deployments

The best way to verify that your code won’t cause non-determinism errors once deployed is to make use of [replay testing](/workflow-execution#replay).

Replay testing takes one or more existing [Event Histories](/workflow-execution/event#event-history) that ran against a previous version of Workflow code and runs them against your _current_ Workflow code, verifying that it is compatible with the provided history.

In the case of Worker Versioning, you may have a [pinned Workflow](/worker-versioning#pinned) that you're switching over to the [current Worker deployment version](/worker-versioning#versioning-definitions) and you want to make sure that the changes don't introduce non-determinism errors. Or you may have an [Auto-Upgrade Workflow](/worker-versioning#auto-upgrade) that you want to run automated tests on to ensure the deployments don't trigger errors.

There are multiple points in your development lifecycle where running replay tests can make sense. They exist on a spectrum, with shortest time to feedback on one end, and most representative of a production deployment on the other.

- During development, replay testing lets you get feedback as early as possible on whether your changes are compatible. For example, you might include some integration tests that run your Workflows against the Temporal Test Server to produce histories which you then check in. You can use those checked-in histories for replay tests to verify you haven’t made breaking changes.
- During pre-deployment validation (such as during some automated deployment validation) you can get feedback in a more representative environment. For example, you might fetch histories from a live Temporal environment (whether production or some kind of pre-production) and use them in replay tests.
- At deployment time, your environment _is_ production, but you are using the new code to replay recent real-world Workflow histories.

When you're writing changes to Workflow code, you can fetch some representative histories from your pre-production or production Temporal environment and verify they work with your changes. You can do the same with the pre-merge CI pipeline. However, if you are using encrypted Payloads, which is a typical and recommended setup in production, you may not be able to decrypt the fetched histories. 

Additionally if your Workflows contain any PII (which should be encrypted), make sure this information is scrubbed for the purposes of your tests, or err on the side of caution and don’t use this method.
With that constraint in mind, we’ll focus on how you can perform replay tests in a production deployment of a Worker with new Workflow code. The core of how replay testing is done is the same regardless of when you choose to do it, so you can apply some of the lessons here to earlier stages in your development process.

## Implement a deployment-time replay test

The key to a successful safe deployment is to break it into two phases: a verification phase, where you’ll run the replay test, followed by the actual deployment of your new Worker code.

You can accomplish this by wrapping your Worker application with some code that can choose whether it will run in verification mode, or in production. This is most easily done if you do not deploy your Workers side-by-side with other application code, which is a recommended best practice. If you do deploy your Workers as part of some other application, you will likely need to separate out a different entry point specifically for verification.

### Run a replay and real Worker with the same code

The following code demonstrates how the same entry point could be used to either verify the new code using replay testing, or to actually run the Worker.

```python
import argparse
import asyncio
from datetime import datetime, timedelta

from temporalio.client import Client
from temporalio.worker import Worker, Replayer

async def main():
    parser = argparse.ArgumentParser(prog='MyTemporalWorker')
    parser.add_argument('mode', choices=['verify', 'run'])
    args = parser.parse_args()

    temporal_url = "localhost:7233"
    task_queue = "your-task-queue"
    my_workflows = [YourWorkflow]
    my_activities = [your_activity]

    client = await Client.connect(temporal_url)
```

Everything up to this point is standard. You import the Workflow and Activity code, instantiate a parser with two modes, and create your Task Queue, Workflow, and Activity.

You can pass in the `args.mode` from any appropriate spot in your code. If the mode is set to `verify`, you conduct the replay testing by specifying the time period to test, and passing in the Workflows corresponding to that time period. Note that the Workflows are consumed as histories, using [the `map_histories()` function](https://python.temporal.io/temporalio.client.WorkflowExecutionAsyncIterator.html#map_histories).

```python
if args.mode == 'verify':
    start_time = (datetime.now() - timedelta(hours=10)).isoformat(timespec='seconds')
    workflows = client.list_workflows(
     f"TaskQueue={task_queue} and StartTime > '{start_time}'",
    limit = 100)
    histories = workflows.map_histories()
    replayer = Replayer(
        workflows=my_workflows,
    )
    await replayer.replay_workflows(histories)
    return
```

If any of the Workflows fail to replay, an error will be thrown. If no errors occur, you can return successfully to indicate success here, or communicate with an endpoint you've defined to indicate success or failure of the verification. You could switch to the `run` mode, and have this Worker transition to a real Worker that will start pulling from the Task Queue and processing Workflows:

```python
    else:
        worker = Worker(
            client,
            task_queue=task_queue,
            workflows=my_workflows,
            activities=my_activities,
        )
        await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

### Use the multi-modal Worker

The most straightforward way to use this bimodal Worker is to deploy one instance of it at the beginning of your deployment process in verify mode, see that it passes, and then proceed to deploy the rest of your new workers in run mode.

---

# Task Queue Priority and Fairness

Source: https://docs.temporal.io/develop/task-queue-priority-fairness

> Control Task dispatch order within a Task Queue using priority levels, and use Fairness to stop tenants from blocking each other.

[Task Queue Priority](#task-queue-priority) and [Task Queue Fairness](#task-queue-fairness) are two ways to manage the
distribution of work within a Task Queue. Priority dispatches [Tasks](/tasks) in Priority order.
Fairness prevents one set of Tasks from blocking others within the same priority level.

You can use Priority and Fairness individually or combine them to express Fairness within a Priority level.

## Task Queue Priority

**Task Queue Priority** lets you control the dispatch order of Workflows, Activities, and Child Workflows based on assigned priority values within a Task Queue. Each priority level acts as a sub-queue that separates Tasks so that high priority Tasks can cut in front of low priority Tasks.

![Flowchart of how Priority dispatches Tasks from highest to lowest priority queue](https://docs.temporal.io/img/develop/task-queue-priority-fairness/priority-details.png)

Priority is enforced within a single Task Queue [partition](/task-queue#task-ordering). A Task Queue by default uses multiple partitions and randomly distributes Tasks across them, so partitions are usually balanced and priority ordering closely approximates the Task Queue as a whole. When partitions become imbalanced, lower-priority Tasks on a lighter partition can dispatch ahead of higher-priority Tasks waiting on a heavier one.

### When to use Priority

If you need a way to specify the order your Tasks dispatch in, you can use Priority to manage that. Priority lets you differentiate between your Tasks, like batch and real-time Tasks, so that you can use a single pool of Workers for efficient resource allocation, while ensuring real-time Tasks dispatch ahead of batch Tasks.

You can also use this as a way to move urgent Tasks ahead of others in the backlog. For example, if you are running an e-commerce platform, you may want payment-related Tasks to dispatch before less time-sensitive Tasks like internal inventory management.

### How to use Priority

Priority is enabled by default in both Temporal Cloud and self-hosted Temporal. To disable Priority in self-hosted Temporal, set the [dynamic config](/temporal-service/configuration#dynamic-configuration) `matching.useNewMatcher` to `false` on a Task Queue, Namespace, or globally.

To use Priority, you need to set a _priority key_ at the Workflow, Activity, or Child Workflow level to a value within the integer range `[1,5]`.
A lower value implies higher priority, so `1` is the highest priority level. If you don't specify a Priority, a Task defaults to a
Priority of `3`. Activities and Child Workflows will inherit their Workflow's priority unless they explicitly specify
their own priority.

When Priority is enabled, all Tasks within a Task Queue dispatch in Priority order. For example, all priority level `1` Tasks dispatch before the first priority level `2` Task, and so on. Lower priority Tasks remain in the backlog until all higher priority Tasks have dispatched. Tasks dispatch by default in first-in-first-out (FIFO) order within each priority level. If you need greater control of task ordering within a priority level, such as preventing large tenants from overwhelming small tenants, check out [the Fairness section](#task-queue-fairness).

You can set a Workflow's priority key via the CLI like so:

```
temporal workflow start \
  --type ChargeCustomer \
  --task-queue my-task-queue \
  --workflow-id my-workflow-id \
  --input '{"customerId":"12345"}' \
  --priority-key 1
```

You can set priority keys for a Workflow within the SDK like so:

**Go**

```go
workflowOptions := client.StartWorkflowOptions{
  ID:        "my-workflow-id",
  TaskQueue: "my-task-queue",
  Priority:  temporal.Priority{PriorityKey: 5},
}
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow)
```

**Java**

```java
WorkflowOptions options = WorkflowOptions.newBuilder()
  .setTaskQueue("my-task-queue")
  .setPriority(Priority.newBuilder().setPriorityKey(5).build()) 
  .build();

WorkflowClient client = WorkflowClient.newInstance(service); MyWorkflow workflow =
client.newWorkflowStub(MyWorkflow.class, options); workflow.run();

````

**Python**

```python
await client.start_workflow(
  MyWorkflow.run,
  args="hello",
  id="my-workflow-id",
  task_queue="my-task-queue",
  priority=Priority(priority_key=1),
)
````

**.NET**

```csharp
var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(1),
  }
);
```

You can set priority keys for an Activity within the SDK like so:

**Go**

```go
ao := workflow.ActivityOptions{
  StartToCloseTimeout: time.Minute,
  Priority:            temporal.Priority{PriorityKey: 3},
}
ctx := workflow.WithActivityOptions(ctx, ao)
err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil)
```

**Java**

```java
ActivityOptions options = ActivityOptions.newBuilder()
  .setStartToCloseTimeout(Duration.ofMinutes(1))
  .setPriority(Priority.newBuilder().setPriorityKey(3).build())
  .build();

MyActivity activity = Workflow.newActivityStub(MyActivity.class, options); activity.perform();

````

**Python**

```python
await workflow.execute_activity(
  say_hello,
  "hi",
  priority=Priority(priority_key=3),
  start_to_close_timeout=timedelta(seconds=5),
)
````

**TypeScript**

**.NET**

```csharp
await Workflow.ExecuteActivityAsync(
  () => SayHello("hi"),
    new()
    {
      StartToCloseTimeout = TimeSpan.FromSeconds(5),
      Priority = new(3),
    }
  );
```

You can set priority keys for a Child Workflow within the SDK like so:

**Go**

```go
cwo := workflow.ChildWorkflowOptions{
  WorkflowID: "child-workflow-id",
  TaskQueue:  "child-task-queue",
  Priority:   temporal.Priority{PriorityKey: 1},
}
ctx := workflow.WithChildOptions(ctx, cwo)
err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil)
```

**Java**

```java
ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder()
  .setTaskQueue("child-task-queue")
  .setWorkflowId("child-workflow-id")
  .setPriority(Priority.newBuilder().setPriorityKey(1).build())
  .build();

MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions); child.run();

````

**Python**

```python
await workflow.execute_child_workflow(
  MyChildWorkflow.run,
  args="hello child",
  priority=Priority(priority_key=1),
)
````

**.NET**

```csharp
await Workflow.ExecuteChildWorkflowAsync(
  (MyChildWorkflow wf) => wf.RunAsync("hello child"),
  new() { Priority = new(1) }
);
```

## Task Queue Fairness

Task Queue Fairness lets you distribute Tasks based on _fairness keys_ and _fairness weights_ within a Task Queue.

Each fairness key creates its own "virtual queue", allowing you to organize Tasks into logical groups like tenants, applications, or workload types. These virtual queues operate using a round-robin dispatch mechanism, meaning the system cycles through each fairness key in turn when selecting the next Task to dispatch. This prevents any single fairness key from dominating dispatch, even if one key has a much larger backlog than the others.

By default, each fairness key is weighted equally in the round-robin, with a _fairness weight_ of 1.0. This behavior can be customized by assigning a different fairness weight to a key. For example, Tasks belonging to a fairness key with a weight of 2.0 will be dispatched twice as often as keys with the default weight.

### When to use Fairness

Fairness is intended to address common situations like:

- Multi-tenant applications with big and small tenants where small tenants shouldn't be blocked by big ones.
- Assigning Tasks to different capacity bands and then, for example, dispatching 80% from one band and 20% from another
  without limiting overall capacity when one band is empty.

It sequences Tasks in the Task Queue probabilistically using a weighted distribution based on:

- Fairness weights you set
- The current backlog of Tasks
- A [data structure](https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch) that tracks how you've distributed Tasks for different fairness keys

As an example, imagine a workload with three tenants, _tenant-big_, _tenant-mid_, _tenant-small_, that have varying
numbers of Tasks at all times. Your _tenant-big_ has a large number of Tasks that can dominate dispatch on your Task Queue and
delay _tenant-mid_ and _tenant-small_ Tasks. With Fairness, you can give each tenant a different
fairness key so that _tenant-big_ doesn't dominate dispatch and block the others. In this case,
_tenant-mid_ and _tenant-small_ Tasks dispatch in between _tenant-big_ Tasks so that dispatch stays fair across tenants.

### How to use Fairness

Fairness is available for both self-hosted Temporal instances and Temporal Cloud.

To enable Fairness for a Namespace in Temporal Cloud, navigate to the Namespace's Overview page in the UI and activate the Fairness toggle. Note that Fairness is a paid feature in Temporal Cloud. For more information, see [Fairness pricing](/cloud/pricing#fairness-pricing).

If you're self-hosting Temporal, set `matching.enableFairness` to `true` in the [dynamic config](/temporal-service/configuration#dynamic-configuration) on the relevant Task Queues or Namespaces.

To use Fairness, you need to set fairness keys and optionally fairness weights at the Workflow, Activity, or Child Workflow level. Tasks with different fairness keys are dispatched in proportion to their fairness weights. For example, if you weight _premium-tier_ at 5.0, _basic-tier_ at 3.0, and _free-tier_ at 2.0, then 50% of dispatched Tasks come from _premium-tier_, 30% from _basic-tier_, and 20% from _free-tier_. If there are Tasks in the Task Queue backlog that have the same fairness key, then they're dispatched in [FIFO order](/task-queue#task-ordering).

![Flowchart of how Fairness selects a virtual queue weighted by fairness weight when the backlog has Tasks](https://docs.temporal.io/img/develop/task-queue-priority-fairness/fairness-details.png)

You can set a Workflow's fairness key and weight via the CLI like so:

```
temporal workflow start \
  --type ChargeCustomer \
  --task-queue my-task-queue \
  --workflow-id my-workflow-id \
  --input '{"customerId":"12345"}' \
  --priority-key 1 \
  --fairness-key a-key \
  --fairness-weight 3.14
```

You can set fairness keys and weights for a Workflow within the SDK like so. Select a concept to highlight the matching lines:

**Go**

  annotations={[
    {
      label: 'Priority key',
      description:
        'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.',
      lines: [5],
    },
    {
      label: 'Fairness key',
      description:
        'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.',
      lines: [6],
    },
    {
      label: 'Fairness weight',
      description:
        'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.',
      lines: [7],
    },
  ]}
>
```go
workflowOptions := client.StartWorkflowOptions{
  ID:        "my-workflow-id",
  TaskQueue: "my-task-queue",
  Priority:  temporal.Priority{
    PriorityKey:    1,
    FairnessKey:    "a-key",
    FairnessWeight: 3.14,
  },
}
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow)
```

**Java**

  annotations={[
    {
      label: 'Priority key',
      description:
        'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.',
      lines: [4],
    },
    {
      label: 'Fairness key',
      description:
        'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.',
      lines: [5],
    },
    {
      label: 'Fairness weight',
      description:
        'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.',
      lines: [6],
    },
  ]}
>
```java
WorkflowOptions options = WorkflowOptions.newBuilder()
  .setTaskQueue("my-task-queue")
  .setPriority(Priority.newBuilder()
      .setPriorityKey(5)
      .setFairnessKey("a-key")
      .setFairnessWeight(3.14)
      .build())
  .build();
WorkflowClient client = WorkflowClient.newInstance(service);
MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options);
workflow.run();
```

**Python**

  annotations={[
    {
      label: 'Priority key',
      description:
        'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.',
      lines: [7],
    },
    {
      label: 'Fairness key',
      description:
        'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.',
      lines: [8],
    },
    {
      label: 'Fairness weight',
      description:
        'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.',
      lines: [9],
    },
  ]}
>
```python
await client.start_workflow(
  MyWorkflow.run,
  args="hello",
  id="my-workflow-id",
  task_queue="my-task-queue",
  priority=Priority(
      priority_key=3,
      fairness_key="a-key",
      fairness_weight=3.14,
  ),
)
```

**Ruby**

  annotations={[
    {
      label: 'Priority key',
      description:
        'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.',
      lines: [6],
    },
    {
      label: 'Fairness key',
      description:
        'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.',
      lines: [7],
    },
    {
      label: 'Fairness weight',
      description:
        'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.',
      lines: [8],
    },
  ]}
>
```ruby
client.start_workflow(
  MyWorkflow, "input-arg",
  id: "my-workflow-id",
  task_queue: "my-task-queue",
  priority: Temporalio::Priority.new(
    priority_key: 3,
    fairness_key: "a-key",
    fairness_weight: 3.14
  )
)
```

**TypeScript**

  annotations={[
    {
      label: 'Priority key',
      description:
        'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.',
      lines: [4],
    },
    {
      label: 'Fairness key',
      description:
        'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.',
      lines: [5],
    },
    {
      label: 'Fairness weight',
      description:
        'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.',
      lines: [6],
    },
  ]}
>
```ts
const handle = await startWorkflow(workflows.priorityWorkflow, {
  args: [false, 1],
  priority: {
    priorityKey: 3,
    fairnessKey: 'a-key',
    fairnessWeight: 3.14,
  },
});
```

**.NET**

  annotations={[
    {
      label: 'Priority key',
      description:
        'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.',
      lines: [8],
    },
    {
      label: 'Fairness key',
      description:
        'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.',
      lines: [9],
    },
    {
      label: 'Fairness weight',
      description:
        'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.',
      lines: [10],
    },
  ]}
>
```csharp
var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(
      priorityKey: 3,
      fairnessKey: "a-key",
      fairnessWeight: 3.14
    )
  }
);
```

You can set fairness keys and weights for an Activity within the SDK like so:

**Go**

```go
ao := workflow.ActivityOptions{
    StartToCloseTimeout: time.Minute,
    Priority: temporal.Priority{
    PriorityKey: 1,
    FairnessKey: "a-key",
    FairnessWeight: 3.14,
  },
}
ctx := workflow.WithActivityOptions(ctx, ao)
err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil)
````

**Java**

```java
ActivityOptions options = ActivityOptions.newBuilder()
  .setStartToCloseTimeout(Duration.ofMinutes(1))
  .setPriority(Priority.newBuilder().setPriorityKey(3).setFairnessKey("a-key").setFairnessWeight(3.14).build())
  .build();
MyActivity activity = Workflow.newActivityStub(MyActivity.class, options);
activity.perform();
````

**Python**

```python
await workflow.execute_activity(
  say_hello,
  "hi",
  priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14),
  start_to_close_timeout=timedelta(seconds=5),
)
```

**Ruby**

```ruby
client.start_activity(
  MyActivity, "input-arg",
  id: "my-workflow-id",
  task_queue: "my-task-queue",
  priority: Temporalio::Priority.new(
    priority_key: 3,
    fairness_key: "a-key",
    fairness_weight: 3.14
  )
)
```

**TypeScript**

```ts
const handle = await startWorkflow(workflows.priorityWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },
});
```

**.NET**

```csharp
var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(
      priorityKey: 3,
      fairnessKey: "a-key",
      fairnessWeight: 3.14
    )
  }
);
```

You can set fairness keys and weights for a Child Workflow within the SDK like so:

**Go**

```go
cwo := workflow.ChildWorkflowOptions{
  WorkflowID: "child-workflow-id",
  TaskQueue:  "child-task-queue",
  Priority:   temporal.Priority{
    PriorityKey:    1,
    FairnessKey:    "a-key",
    FairnessWeight: 3.14,
  },
}
ctx := workflow.WithChildOptions(ctx, cwo)
err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil)
```

**Java**

```java
ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder()
  .setTaskQueue("child-task-queue")
  .setWorkflowId("child-workflow-id")
  .setPriority(Priority.newBuilder().setPriorityKey(1).setFairnessKey("a-key").setFairnessWeight(3.14).build())
  .build();
MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions);
child.run();
```

**Python**

```python
await workflow.execute_child_workflow(
  MyChildWorkflow.run,
  args="hello child",
  priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14),
)
```

**Ruby**

```ruby
client.start_child_workflow(
  MyChildWorkflow, "input-arg",
  id: "my-child-workflow-id",
  task_queue: "my-task-queue",
  priority: Temporalio::Priority.new(
    priority_key: 3,
    fairness_key: "a-key",
    fairness_weight: 3.14
  )
)
```

**TypeScript**

```ts
const handle = await startChildWorkflow(workflows.priorityWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },
});
```

**.NET**

```csharp
var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(
      priorityKey: 3,
      fairnessKey: "a-key",
      fairnessWeight: 3.14
    )
  }
);
```

Tasks that do not have a `fairness_key` set are grouped together under an implicit empty-string key. All unkeyed Tasks share this single default bucket and participate in the same round-robin dispatch alongside named fairness keys, with a default weight of 1.0. This means Fairness adoption can be incremental: you can assign fairness keys to some tenants but not others. Unkeyed Tasks do not bypass Fairness; they compete as one group alongside all explicitly keyed Tasks.

> **ℹ️ Info:**
>
> There should only be one fairness weight assigned to each fairness key within a Task Queue. Having multiple fairness weights on a fairness key will result in unspecific behavior.
>

### Choosing between Priority, Fairness, and both

- **Priority alone** when you need strict priority ordering - for example, separating real-time Tasks from batch Tasks.
- **Fairness alone** when you need tier or tenant isolation so no group is starved, but you don't need to preempt any group ahead of another.
- **Both** when you have a tiered SLA hierarchy - Priority for the broad tier (for example, paid versus free), Fairness for per-tenant equity within a tier.

When you use Priority and Fairness together, the next Task to dispatch is chosen by walking three rules in order:

1. **Priority tier (strict).** Tasks at a higher priority always dispatch before tasks at lower priorities, regardless of fairness keys or weights.
2. **Fairness key within a tier (weighted).** Within a priority tier, each fairness key is a virtual queue. Keys are dispatched proportional to their weights - a key with weight 2.0 is dispatched twice as often as one with weight 1.0.
3. **FIFO within a key.** Tasks that share a priority tier _and_ fairness key dispatch in the order they were enqueued.

These rules apply within a Task Queue partition.

![Flowchart of how Priority and Fairness combine: poller checks priority levels in order, then within each level Fairness selects a virtual queue weighted by fairness weight](https://docs.temporal.io/img/develop/task-queue-priority-fairness/priority-fairness.png)

### Inheritance

Each field of Priority (`priority_key`, `fairness_key`, `fairness_weight`) is resolved independently.

![Flowchart of how a Priority field is resolved: check for a Task Queue weight override (fairness_weight only), then an explicit value, then inherit from the calling Workflow, otherwise use the default](https://docs.temporal.io/img/develop/task-queue-priority-fairness/inheritance.png)

**Activity inheritance order** (highest precedence first):

1. [Fairness weight overrides](#fairness-weight-overrides) on the Task Queue (`fairness_weight` only)
2. Value set explicitly in the Activity options
3. Inherited from the calling Workflow
4. Default value (`priority_key=3`, `fairness_key=""`, `fairness_weight=1.0`)

**Workflow inheritance order** (highest precedence first):

1. [Fairness weight overrides](#fairness-weight-overrides) on the Task Queue (`fairness_weight` only)
2. Value set explicitly in the Workflow start options
3. Inherited from the parent Workflow (Child Workflows only)
4. Default value (`priority_key=3`, `fairness_key=""`, `fairness_weight=1.0`)

Continue-As-New inherits from the current execution unless explicit values are passed.

### Enabling or disabling Fairness with an active backlog

When Fairness is enabled on a Namespace, Task Queues in the Namespace begin honoring fairness keys on Tasks for dispatch ordering. Existing queued Tasks are dispatched first, in their original priority + FIFO order. Fairness keys on Tasks already in the backlog do not retroactively affect their dispatch order.

When Fairness is disabled on a Namespace, Task Queues in the Namespace stop honoring fairness keys for dispatch ordering. The existing fairness-ordered backlog is dispatched first, in its original fairness order. After the backlog drains, Task Queues dispatch in priority + FIFO order.

In both directions, the existing backlog is dispatched before any new Tasks queued under the new mode. New Tasks dispatch only after the backlog fully drains. Tasks are not lost in either transition.

### Set rate limits at the Task Queue level

Within a Task Queue, you can set dispatch rate limits for the whole queue using `queue-rps-limit` and for each fairness key using `fairness-key-rps-limit-default`.

```
temporal task-queue config set \
    --task-queue my-task-queue \
    --task-queue-type activity \
    --namespace my-namespace \
    --queue-rps-limit 500 \
    --queue-rps-limit-reason "overall limit" \
    --fairness-key-rps-limit-default 33.3 \
    --fairness-key-rps-limit-reason "per-key limit"
```

**Whole queue rate limits:** applies to the whole queue regardless of the fairness key. This is the same setting as is exposed through the [Worker Options](/develop/worker-tuning-reference#io-configuration-options) in the SDKs, and when set via the API, takes precedence over the limit set through Worker Options.

**Fairness key rate limits:** The per-fairness-key rate limit works in conjunction with Task Queue Fairness. If you think of Fairness as dividing the queue into one virtual queue for each key, then the per-fairness-key rate limit is a limit on each individual virtual queue. Some important notes on the per-fairness-key limit:

- The whole queue limit and per-fairness-key limit may be set independently: none, one or the other, or both may be set. If both are set, then the more restrictive one applies.
- The per-fairness-key limit for a key is scaled by the fairness weight assigned to that key. So if the per-fairness-key limit for a queue is set to 10, then all keys with the default weight (1.0) will have a limit of 10 tasks/second. But if a particular key is given a weight of 2.5, then the per-key rate limit for that key will be 25 tasks/second.
- Since the dispatch rate for each key should be proportional to its weight, if any key is hitting the per-key limit, then nearly all of them are. The way it works is if the next Task to be dispatched hits the per-key limit, then dispatch will wait until it can go.
- Usually there isn't actually any blocking, but there can be when the fairness weight for a key is changed between when a Task is scheduled and when it's dispatched. If the fairness weight for a key is lowered, for example, the new lower per-key rate limit will be respected. Since those Tasks were originally scheduled with the higher rate, they will block other Tasks as they're dispatched. This limitation will be improved in the future.

### Fairness weight overrides

You can override the weights of up to 1000 keys through the config API. When an override is set for a key, the weight attached to the Task, through Workflow or Activity priority metadata, will be ignored, and the overridden weight will be used instead.

Weight overrides are stored per Task Queue, including type, so they must be set for both Workflow and Activity Task Queues to take effect for both.

Set overrides with `temporal task-queue config set`:

```
temporal task-queue config set \
    --task-queue my-task-queue \
    --task-queue-type activity \
    --namespace my-namespace \
    --fairness-key-weight premium=5.0 \
    --fairness-key-weight basic=1.0
```

To unset a single key's override, pass `key=default`. To clear all overrides on the Task Queue, use `--fairness-key-weight-clear-all`.

### Limitations of Fairness

- There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more.
- Fairness is enforced within a single Task Queue [partition](/task-queue#task-ordering). When a Task Queue's partitions are imbalanced, Fairness may not appear to hold, since it applies only within individual partitions. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition.
- The fairness weight applies at schedule time, not at dispatch time. So it only affects newly-scheduled Tasks, not currently backlogged ones. This means if you need to throttle a single fairness key in the existing backlog of Tasks, you won't be able to.
- When you use Worker Versioning and you're moving Workflows from one version to another, Priority will still apply between versions. Fairness isn't guaranteed between versions. For example, you may have Tasks that were originally queued on Worker version _alpha_, Tasks that were queued on Worker version _beta_, and some Tasks were moved from _alpha_ to _beta_. Fairness is only guaranteed when Tasks are originally queued on the same Worker version. So there might be some discrepancies on the Tasks moved from _alpha_ to _beta_.
- During server restarts, Temporal preserves fairness state for the top 100 keys. Other keys rebuild their fairness state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering reduces this distortion by spreading keys' initial positions according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, [contact Temporal Support](/evaluate/cloud/support#support-ticket).
- Fairness doesn't consider Task executions that have already been dispatched to Workers. As a result, fair dispatch may not be immediately visible in the mix of Tasks currently running on Workers.

---

# TypeScript SDK developer guide

Source: https://docs.temporal.io/develop/typescript

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Install and get started

You can find detailed installation instructions for the TypeScript SDK in the [Quickstart](/develop/typescript/set-up-your-local-typescript).

There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running.

Once your local Temporal Service is set up, continue building with the following resources:

- [Develop a Workflow](/develop/typescript/workflows/basics)
- [Develop an Activity](/develop/typescript/activities/basics)
- [Start an Activity Execution](/develop/typescript/activities/execution)
- [Run a Worker](/develop/typescript/workers/run-worker-process)

## [Workflows](/develop/typescript/workflows)

- [Workflow basics](/develop/typescript/workflows/basics)
- [Child Workflows](/develop/typescript/workflows/child-workflows)
- [Continue-As-New](/develop/typescript/workflows/continue-as-new)
- [Message passing](/develop/typescript/workflows/message-passing)
- [Cancellation](/develop/typescript/workflows/cancellation)
- [Cancellation scopes](/develop/typescript/workflows/cancellation-scopes)
- [Timeouts](/develop/typescript/workflows/timeouts)
- [Schedules](/develop/typescript/workflows/schedules)
- [Timers](/develop/typescript/workflows/timers)
- [Versioning](/develop/typescript/workflows/versioning)
- [Workflow Streams](/develop/typescript/workflows/workflow-streams)

## [Activities](/develop/typescript/activities)

- [Activity basics](/develop/typescript/activities/basics)
- [Activity Execution](/develop/typescript/activities/execution)
- [Timeouts](/develop/typescript/activities/timeouts)
- [Asynchronous Activity Completion](/develop/typescript/activities/asynchronous-activity)
- [Benign exceptions](/develop/typescript/activities/benign-exceptions)

## [Workers](/develop/typescript/workers)

- [Run a Worker](/develop/typescript/workers/run-worker-process)
- [Interceptors](/develop/typescript/workers/interceptors)
- [Serverless Workers](/develop/typescript/workers/serverless-workers)

## [Temporal Client](/develop/typescript/client)

- [Temporal Client](/develop/typescript/client/temporal-client)
- [Namespaces](/develop/typescript/client/namespaces)

## [Temporal Nexus](/develop/typescript/nexus)

- [Quickstart](/develop/typescript/nexus/quickstart)
- [Feature guide](/develop/typescript/nexus/feature-guide)
- [Standalone Operations](/develop/typescript/nexus/standalone-operations)

## [Platform](/develop/typescript/platform)

- [Observability](/develop/typescript/platform/observability)
- [Enriching the UI](/develop/typescript/platform/enriching-ui)

## [Best practices](/develop/typescript/best-practices)

- [Testing](/develop/typescript/best-practices/testing-suite)
- [Debugging](/develop/typescript/best-practices/debugging)
- [Converters and encryption](/develop/typescript/best-practices/data-handling)
- [Entity pattern](/develop/typescript/best-practices/entity-pattern)

## [Integrations](/develop/typescript/integrations)

- [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#typescript)
- [Google ADK integration](/develop/typescript/integrations/google-adk-agents)
- [Laminar integration](https://laminar.sh/docs/integrations/temporal)
- [LangSmith integration](/develop/typescript/integrations/langsmith)
- [Mastra integration](https://mastra.ai/integrations/deploy/temporal)
- [OpenAI Agents SDK integration](/develop/typescript/integrations/openai-agents)
- [Parseable integration](https://github.com/parseablehq/temporal-plugin/blob/main/INTEGRATION.md)
- [Strands Agents integration](/develop/typescript/integrations/strands-agents)
- [Vercel AI SDK integration](/develop/typescript/integrations/ai-sdk)

## Temporal TypeScript technical resources

- [TypeScript SDK Quickstart - Setup Guide](/develop/typescript/set-up-your-local-typescript)
- [TypeScript API Documentation](https://typescript.temporal.io)
- [TypeScript SDK Code Samples](https://github.com/temporalio/samples-typescript)
- [TypeScript SDK GitHub](https://github.com/temporalio/sdk-typescript)
- [Temporal 101 in TypeScript Free Course](https://learn.temporal.io/courses/temporal_101/typescript/)
- [Build a Job Queue with Standalone Activities Tutorial](https://learn.temporal.io/tutorials/typescript/standalone-activities/)

## Get connected with the Temporal TypeScript community

- [Temporal TypeScript Community Slack](https://temporalio.slack.com/archives/C01DKSMU94L)
- [TypeScript SDK Forum](https://community.temporal.io/tag/typescript-sdk)

## Linting and types in TypeScript 

If you started your project with `@temporalio/create`, you already have our recommended TypeScript and ESLint
configurations.

If you incrementally added Temporal to an existing app, we do recommend setting up linting and types because they help catch bugs well before you ship them to production, and they improve your development feedback loop. Take a look at our recommended [.eslintrc](https://github.com/temporalio/samples-typescript/blob/main/.shared/.eslintrc.js) file and tweak to suit your needs.

---

# Activities - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities

> Activity Definitions, Activity Execution, Timeouts, Standalone Activities, and benign exceptions in the Temporal TypeScript SDK.

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Activities

- [Activity basics](/develop/typescript/activities/basics)
- [Activity Execution](/develop/typescript/activities/execution)
- [Standalone Activities quickstart](/develop/typescript/activities/standalone-activities-quickstart)
- [Standalone Activities feature guide](/develop/typescript/activities/standalone-activities)
- [Timeouts](/develop/typescript/activities/timeouts)
- [Asynchronous Activity Completion](/develop/typescript/activities/asynchronous-activity)
- [Benign exceptions](/develop/typescript/activities/benign-exceptions)

---

# Asynchronous Activity Completion - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities/asynchronous-activity

> Asynchronously complete an Activity in Temporal by enabling the Activity Function to return before the Activity Execution finishes, using AsyncCompletionClient.

## Asynchronously complete an Activity 

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The Temporal Client is used to Heartbeat and complete the Activity.

To asynchronously complete an Activity, call [`AsyncCompletionClient.complete`](https://typescript.temporal.io/api/classes/client.AsyncCompletionClient#complete).

<!--SNIPSTART typescript-activity-complete-async -->
[activities-examples/src/activities/async-completion.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/activities/async-completion.ts)
```ts
import { CompleteAsyncError, activityInfo } from '@temporalio/activity';
import { Client } from '@temporalio/client';

export async function doSomethingAsync(): Promise<string> {
  const taskToken = activityInfo().taskToken;
  setTimeout(() => doSomeWork(taskToken), 1000);
  throw new CompleteAsyncError();
}

// this work could be done in a different process or on a different machine
async function doSomeWork(taskToken: Uint8Array): Promise<void> {
  const client = new Client();
  // does some work...
  await client.activity.complete(taskToken, "Job's done!");
}
```
<!--SNIPEND-->

## Local Activities 

To call [Local Activities](/local-activity) in TypeScript, use [`proxyLocalActivities`](https://typescript.temporal.io/api/namespaces/workflow/#proxylocalactivities).

```ts
import * as workflow from '@temporalio/workflow';

const { getEnvVar } = workflow.proxyLocalActivities({
  startToCloseTimeout: '2 seconds',
});

export async function yourWorkflow(): Promise<void> {
  const someSetting = await getEnvVar('SOME_SETTING');
  // ...
}
```

Local Activities must be registered with the Worker the same way non-local Activities are.

---

# Activity basics - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities/basics

> Export an Activity as an async TypeScript function, type its parameters and return value, and share dependencies through dependency injection.

## Develop an Activity 

One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function
or method execution that's intended to execute a single, well-defined action (either short or long-running), such as
querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with the world outside
the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to
execute the Activity, we must define the [Activity Definition](/activity-definition).

Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client.

The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/typescript/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/typescript/activities/standalone-activities-quickstart).

- Activities execute in the standard Node.js environment.
- Activities cannot be in the same file as Workflows and must be separately registered.
- Activities may be retried repeatedly, so you may need to use idempotency keys for critical side effects.

Activities are _just functions_. The following is an Activity that accepts a string parameter and returns a string.

<!--SNIPSTART typescript-activity-fn -->
[snippets/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/activities.ts)
```ts
export async function greet(name: string): Promise<string> {
  return `👋 Hello, ${name}!`;
}
```
<!--SNIPEND-->

## Define Activity parameters 

There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may
support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload.

A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the
arguments, is limited to a maximum of 4 MB.

Also, keep in mind that all Payload data is recorded in the
[Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker
performance. This is because the entire Event History could be transferred to a Worker Process with a
[Workflow Task](/tasks#workflow-task).

Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data
that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the
application data passed to Activities. This is so that you can change what data is passed to the Activity without
breaking a function or method signature.

This Activity takes a single `name` parameter of type `string`.

<!--SNIPSTART typescript-activity-fn -->
[snippets/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/activities.ts)
```ts
export async function greet(name: string): Promise<string> {
  return `👋 Hello, ${name}!`;
}
```
<!--SNIPEND-->

## Define Activity return values 

All data returned from an Activity must be serializable.

Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there
is a hard limit of 4MB for any gRPC message size in the Event History transaction
([see Cloud limits here](/evaluate/cloud/limits#per-message-grpc-limit)). Keep in mind that all return
values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history).

In TypeScript, the return value is always a Promise.

In the following example, `Promise<string>` is the return value.

```typescript
export async function greet(name: string): Promise<string> {
  return `👋 Hello, ${name}!`;
}
```

## Customize your Activity Type 

Activities have a Type that are referred to as the Activity name. The following examples set a custom name for
your Activity Type.

You can customize the name of the Activity when you register it with the Worker. In the following example, the Activity
Name is `activityFoo`.

<!--SNIPSTART typescript-custom-activity-type -->
[snippets/src/worker-activity-type-custom.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/worker-activity-type-custom.ts)
```ts
import { Worker } from '@temporalio/worker';
import { greet } from './activities';

async function run() {
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows'),
    taskQueue: 'snippets',
    activities: {
      activityFoo: greet,
    },
  });

  await worker.run();
}
```
<!--SNIPEND-->

## Important design patterns for Activities 

The following are some important (and frequently requested) patterns for using our Activities APIs. These patterns
address common needs and use cases.

### Share dependencies in Activity functions (dependency injection)

Because Activities are "just functions," you can also create functions that create Activities. This is a helpful pattern
for using closures to do the following:

- Store expensive dependencies for sharing, such as database connections.
- Inject secret keys (such as environment variables) from the Worker to the Activity.

<!--SNIPSTART typescript-activity-with-deps-->
[activities-dependency-injection/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-dependency-injection/src/activities.ts)
```ts
export interface DB {
  get(key: string): Promise<string>;
}

export const createActivities = (db: DB) => ({
  async greet(msg: string): Promise<string> {
    const name = await db.get('name'); // simulate read from db
    return `${msg}: ${name}`;
  },
  async greet_es(mensaje: string): Promise<string> {
    const name = await db.get('name'); // simulate read from db
    return `${mensaje}: ${name}`;
  },
});
```
<!--SNIPEND-->

#### See full example

When you register these in the Worker, pass your shared dependencies accordingly:

<!--SNIPSTART typescript-activity-deps-worker {"enable_source_link": false}-->
```ts
import { createActivities } from './activities';

async function run() {
  // Mock DB connection initialization in Worker
  const db = {
    async get(_key: string) {
      return 'Temporal';
    },
  };

  const worker = await Worker.create({
    taskQueue: 'dependency-injection',
    workflowsPath: require.resolve('./workflows'),
    activities: createActivities(db),
  });

  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```
<!--SNIPEND-->

Because Activities are always referenced by name, inside the Workflow they can be proxied as normal, although the types
need some adjustment:

<!--SNIPSTART typescript-activity-deps-workflow-->
[activities-dependency-injection/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-dependency-injection/src/workflows.ts)
```ts
import type { createActivities } from './activities';

// Note usage of ReturnType<> generic since createActivities is a factory function
const { greet, greet_es } = proxyActivities<ReturnType<typeof createActivities>>({
  startToCloseTimeout: '30 seconds',
});
```
<!--SNIPEND-->

### Import multiple Activities simultaneously

You can proxy multiple Activities from the same `proxyActivities` call if you want them to share the same timeouts,
retries, and options:

```ts
export async function Workflow(name: string): Promise<string> {
  // destructuring multiple activities with the same options
  const { act1, act2, act3 } = proxyActivities<typeof activities>();
  /* activityOptions */
  await act1();
  await Promise.all([act2, act3]);
}
```

### Dynamically reference Activities

Because Activities are referenced only by their string names, you can reference them dynamically if needed:

```js
export async function DynamicWorkflow(activityName, ...args) {
  const acts = proxyActivities(/* activityOptions */);

  // these are equivalent
  await acts.activity1();
  await acts['activity1']();

  // dynamic reference to activities using activityName
  let result = await acts[activityName](...args);
}
```

Type safety is still supported here, but we encourage you to validate and handle mismatches in Activity names. An
invalid Activity name leads to a `NotFoundError` with a message that looks like this:

```
ApplicationFailure: Activity function actC is not registered on this Worker, available activities: ["actA", "actB"]
```

---

# Benign exceptions - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities/benign-exceptions

> Mark expected or non-severe Activity errors as benign in TypeScript to reduce noise in logs, metrics, and OpenTelemetry traces.

When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues.
By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic.

To mark an error as benign, set the `category` field to `ApplicationFailureCategory.BENIGN` when creating an [`ApplicationFailure`](https://typescript.temporal.io/api/classes/common.ApplicationFailure).

Benign errors:
- Have Activity failure logs downgraded to DEBUG level
- Do not emit Activity failure metrics
- Do not set the OpenTelemetry failure status to ERROR

```typescript
import {
  ApplicationFailure,
  ApplicationFailureCategory,
} from '@temporalio/common';

export async function myActivity(): Promise<string> {
  try {
    return await callExternalService();
  } catch (err) {
    const message = err instanceof Error ? err.message : String(err);
    throw ApplicationFailure.create({
      message,
      // Mark this error as benign since it's expected
      category: ApplicationFailureCategory.BENIGN,
    });
  }
}
```

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried.

---

# Activity Execution - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities/execution

> Call an Activity from a TypeScript Workflow through proxyActivities, set the required Timeout, and keep Activity inputs and results small.

## Start an Activity Execution 

Calls to spawn [Activity Executions](/activity-execution) are written within a
[Workflow Definition](/workflow-definition). In TypeScript, you never call an Activity function directly. Instead, you
pass in the _types_ of your Activities and Activity options to the `proxyActivities` function. This will give you an
_Activity Handle_, a type-safe proxy object with the same function names and signatures as your real activities. From
the Activity Handle, you can call your Activities as if they were normal async functions.

```typescript
import { proxyActivities } from '@temporalio/workflow';
// Only import the activity types, not the functions themselves
import type * as activities from './activities';

// Retrieve the Activity Handle by passing in the Activity types and options
const activityHandle = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

// Deconstruct the individual Activity functions from the Activity Handle
const { greet } = activityHandle;

// A workflow that calls an activity
export async function example(name: string): Promise<string> {
  return await greet(name);
}
```

When you call a proxied function, the Workflow does not execute the Activity code directly. Instead, it schedules an
Activity Task. After the Activity Task is scheduled, it becomes available for a Worker to pick up and execute. This
results in the set of three [Activity Task](/tasks#activity-task) related Events:
[ActivityTaskScheduled](/references/events#activitytaskscheduled),
[ActivityTaskStarted](/references/events#activitytaskstarted), and
[ActivityTaskCompleted](/references/events#activitytaskcompleted) in your Workflow Execution Event History.

The Worker may run many Activity executions at the same time, all using the same Activity function code. Temporal can
also retry an Activity if it fails or times out. For this reason, you should write Activities to be
[idempotent](/encyclopedia/activities/activity-definition.mdx#idempotency): calling them multiple times with the
same input should have the same effect as calling them once.

> **💡 Tip:**
>
> Every Activity call you make is recorded in the Workflow’s execution history, including the parameters you pass
> in and the value that comes back. This history is what allows Temporal to recover a Workflow after a failure. Because
> the entire history must be stored and replayed, avoid passing large objects as Activity inputs or return values. Keeping
> payloads small will help your Workflows replay and recover efficiently.
>

## Set the required Activity Timeouts 

Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a
[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a
[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the
Activity Options.

## Get the results of an Activity Execution 

The call to spawn an [Activity Execution](/activity-execution) generates the
[ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable.
Workflow Executions can either block progress until the result is available through the Awaitable or continue
progressing, making use of the result when it becomes available.

Since Activities are referenced by their string name, you can reference them dynamically to get the result of an
Activity Execution.

```typescript
export async function DynamicWorkflow(activityName, ...args) {
  const acts = proxyActivities(/* activityOptions */);

  // these are equivalent
  await acts.activity1();
  await acts['activity1']();

  let result = await acts[activityName](...args);
  return result;
}
```

The `proxyActivities()` returns an object that calls the Activities in the function. `acts[activityName]()` references
the Activity using the Activity name, then it returns the results.

---

# Standalone Activities feature guide - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities/standalone-activities

> Execute Activities independently without a Workflow using the Temporal TypeScript SDK.

[Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated
by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a [Temporal Client](/develop/typescript/client/temporal-client).

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/typescript/activities/basics). The only difference is that you execute a
Standalone Activity directly from your Temporal Client.

> **💡 Tip:**
>
> New to Standalone Activities? Start with the [Standalone Activities quickstart](/develop/typescript/activities/standalone-activities-quickstart).
>

This page covers the following:

- [Prerequisites](#prerequisites)
- [Start a Standalone Activity without waiting for the result](#start-activity)
- [Get a handle to an existing Standalone Activity](#get-activity-handle)
- [Wait for the result of a Standalone Activity](#get-activity-result)
- [List Standalone Activities](#list-activities)
- [Count Standalone Activities](#count-activities)
- [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code from the [standalone-activity](https://github.com/temporalio/samples-typescript/tree/main/standalone-activity) sample.
>

## Prerequisites 

Standalone Activities require:

- **Node.js** 20+
- **Temporal TypeScript SDK** v1.24.0 or higher
- **[Temporal CLI](/cli/setup-cli)** v1.9.1 or higher

The [Standalone Activities quickstart](/develop/typescript/activities/standalone-activities-quickstart)
walks through installing these.

## Start a Standalone Activity without waiting for the result 

Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue
your Activity job, without waiting for it to be executed by your Worker.

Use
[`start`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#start) method of
the client or the typed interface to start your Standalone Activity and get a handle:

<!--SNIPSTART typescript-standalone-activity-start-->
[standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts)
```ts
const handle = await activitiesClient.start('greet', {
  ...activityOptions,
  id: activityId,
  args: ['Temporal'],
});
```
<!--SNIPEND-->

Or use the Temporal CLI:

```bash
temporal activity start \
  --type greet \
  --activity-id my-standalone-activity-id \
  --task-queue hello-standalone-activities \
  --start-to-close-timeout 10s \
  --input '"World"'
```

## Get a handle to an existing Standalone Activity 

You can also use [`getHandle`](https://typescript.temporal.io/api/classes/client.ActivityClient#gethandle)
to create a handle to a previously started Standalone Activity. Because the client doesn't know
how the Activity was started, this method is not available in the typed interface. The method
takes an optional type argument to constrain the Activity result type, but correctness of this argument
is not verified.

<!--SNIPSTART typescript-standalone-activity-get-handle-->
[standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts)
```ts
const newHandle = client.activity.getHandle<string>(activityId);
```
<!--SNIPEND-->

You can now use the handle to wait for the result, describe, cancel, or terminate the Activity.

## Wait for the result of a Standalone Activity 

Under the hood, calling [`execute`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#execute)
is the same as calling [`start`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#start)
to durably enqueue the Standalone Activity, and then calling `await handle.result()` to
wait for the Activity to be executed and fetch the result:

<!--SNIPSTART typescript-standalone-activity-result-->
[standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts)
```ts
console.log(await handle.result()); // Hello, Temporal!
```
<!--SNIPEND-->

Or use the Temporal CLI to wait for a result by Activity ID:

```bash
temporal activity result --activity-id my-standalone-activity-id
```

## List Standalone Activities 

Use
[`list`](https://typescript.temporal.io/api/classes/client.ActivityClient#list) method of the client to list
Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is an `AsyncIterable`
that yields [`ActivityExecutionInfo`](https://typescript.temporal.io/api/interfaces/client.ActivityExecutionInfo)
entries.

These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included.

```typescript
const query = 'TaskQueue="hello-standalone-activities"';

for await (const a of client.activity.list(query)) {
  console.log(
    `${a.activityId} | ${a.activityRunId} | ${a.activityType} | ${a.status} | ${a.closeTime?.toISOString()}`,
  );
}
```

The sample file 
[standalone-activity/src/list.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/list.ts)
lists and counts activities.

Run it:

```bash
npm run list
```

Or use the Temporal CLI:

```bash
temporal activity list
```

The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow
Visibility](/visibility). For example, "ActivityType = 'MyActivity' AND ExecutionStatus = 'Running'".

## Count Standalone Activities 

Use
[`count`](https://typescript.temporal.io/api/classes/client.ActivityClient#count) method of the client
to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns
the total count of executions (running, completed, failed, etc.) - not the number of queued tasks. It
works the same way as counting Workflow Executions. The same query will work for both listing and
counting.

<!--SNIPSTART typescript-standalone-activity-count-->
[standalone-activity/src/list.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/list.ts)
```ts
const { count } = await client.activity.count(query);
console.log(`Total activities: ${count}`);
```
<!--SNIPEND-->

The sample file 
[standalone-activity/src/list.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/list.ts)
lists and counts activities.

Run it:

```bash
npm run list
```

Or use the Temporal CLI:

```bash
temporal activity count
```

## Run Standalone Activities with Temporal Cloud 

The Worker and Client code in the [Standalone Activities quickstart](/develop/typescript/activities/standalone-activities-quickstart)
use [`loadClientConnectConfig`](https://typescript.temporal.io/api/namespaces/envconfig#loadclientconnectconfig),
so the same code works against Temporal Cloud - configure the connection via environment variables or a TOML
profile. No code changes are needed.

For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate
generation, and authentication setup in the Cloud UI, see
[Connect to Temporal Cloud](/develop/typescript/client/temporal-client#connect-to-temporal-cloud).

### Connect with mTLS

Set these environment variables with values from your Temporal Cloud Namespace settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem'
export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key'
```

### Connect with an API key

Set these environment variables with values from your Temporal Cloud API key settings:

```
export TEMPORAL_ADDRESS=<your-namespace>.<your-account-id>.tmprl.cloud:7233
export TEMPORAL_NAMESPACE=<your-namespace>.<your-account-id>
export TEMPORAL_API_KEY=<your-api-key>
```

Then run the Worker and starter code as shown in the [Standalone Activities quickstart](/develop/typescript/activities/standalone-activities-quickstart).

---

# Standalone Activities TypeScript quickstart - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities/standalone-activities-quickstart

> Execute a Standalone Activity with the Temporal TypeScript SDK without writing a Workflow.

# Quickstart

Standalone Activities are Activities that run independently, without being orchestrated by a
Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone
Activity directly from a [Temporal Client](/develop/typescript/client/temporal-client).

The way you write the Activity and register it with a Worker is identical to [Workflow
Activities](/develop/typescript/activities/basics). The only difference is that you execute a
Standalone Activity directly from your Temporal Client.

> **📝 Note:**
>
> This documentation uses source code from the [standalone-activity](https://github.com/temporalio/samples-typescript/tree/main/standalone-activity) sample.
>

## Get started with Standalone Activities 

Prerequisites:

- **[Node.js](https://nodejs.org/en/download/)** 20+

- **Temporal TypeScript SDK** (v1.24.0 or higher). See the [TypeScript Quickstart](/develop/typescript/set-up-your-local-typescript) for install instructions.

- **Temporal CLI** v1.9.1 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`.

Start the Temporal development server with `temporal server start-dev`.

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace.
It uses an in-memory database, so do not use it for real use cases.

The Temporal Server will now be available for client connections on `localhost:7233`, and the
Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233).

```bash
brew install temporal
```

```bash
temporal --version
```

```bash
temporal server start-dev
```

## Clone the sample

Clone the [samples-typescript](https://github.com/temporalio/samples-typescript) repository to follow along:

```
git clone https://github.com/temporalio/samples-typescript.git
cd samples-typescript
```

The sample project is structured as follows:

```
standalone-activity/src/
├── activities.ts
├── execute.ts
├── list.ts
└── worker.ts
```

## Write an Activity function 

The way you write a Standalone Activity is identical to how you write an Activity to be orchestrated
by a Workflow. In fact, an Activity can be executed both as a Standalone Activity and as a Workflow
Activity.

[standalone-activity/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/activities.ts)

```typescript
import { ApplicationFailure } from '@temporalio/activity';

export async function greet(name: string): Promise<string> {
  if (typeof name !== 'string') {
    throw ApplicationFailure.create({ message: 'name must be a string', nonRetryable: true });
  }
  return \`Hello, \${name}!\`;
}
```

## Run a Worker with the Activity registered 

Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities —
you create a Worker, register the Activity, and run the Worker. The Worker doesn't need to know
whether the Activity will be invoked from a Workflow or as a Standalone Activity. See
[Run a Worker](/develop/typescript/workers/run-worker-process#run-a-dev-worker) for more details on Worker setup
and configuration options.

[standalone-activity/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/worker.ts)

Open a new terminal, navigate to the `samples-typescript/standalone-activity` directory, and run the Worker.
Leave this terminal running - the Worker needs to stay up to process activities.

```typescript
import { NativeConnection, Worker } from '@temporalio/worker';
import * as activities from './activities';
import { loadClientConnectConfig } from '@temporalio/envconfig';

async function run() {
  const config = loadClientConnectConfig();
  const connection = await NativeConnection.connect(config.connectionOptions);
  try {
    const worker = await Worker.create({
      connection,
      namespace: config.namespace ?? 'default',
      taskQueue: 'hello-standalone-activities',
      activities,
    });
    await worker.run();
  } finally {
    await connection.close();
  }
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

```bash
npm run start
```

## Run the sample client 

The sample file [standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts)
contains a program demonstrating various ways to execute Standalone Activities and fetch results. The code in
the following sections is copied from this file.

To run it:

1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above).
2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above).
3. Open a new terminal, navigate to the `samples-typescript/standalone-activity` directory, and run:
    ```
    npm run execute
    ```

```bash
npm run execute
```

## Execute a Standalone Activity with type checking 

Start by [creating a Temporal Client](/develop/typescript/client/temporal-client).
Then call [`client.activity.typed()`](https://typescript.temporal.io/api/classes/client.ActivityClient#typed)
to get a typed Activity Client interface. Any TypeScript type can be used as type argument as long
as it has Activity functions as its methods. An easy way to provide such a type is to use the `typeof`
operator on imported activities. Note that calling `typed` does not create a new Client object - it
only adjusts the type annotation of the existing Client. `typed` can be called multiple times with
different type arguments to use the same Client for multiple Activity interfaces.

Afterwards, call the [`execute`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#execute)
method of the typed client to execute a Standalone Activity. Call this from your application code, not from
inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal Server, waits
for it to be executed on your Worker, and then fetches the result. An unknown or mistyped Activity name, or
wrong argument types will cause compilation to fail.

```typescript
import { Connection, Client, ActivityExecutionFailedError } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import * as activities from './activities';
import { nanoid } from 'nanoid';

const config = loadClientConnectConfig();
const connection = await Connection.connect(config.connectionOptions);
const client = new Client({ connection, namespace: config.namespace });

const activitiesClient = client.activity.typed<typeof activities>();
```

```typescript
const taskQueue = 'hello-standalone-activities';
const activityOptions = {
  taskQueue,
  startToCloseTimeout: '10s',
};

// In practice, use a meaningful business identifier, like customer or transaction identifier
const activityId = nanoid();

const result = await activitiesClient.execute('greet', {
  ...activityOptions,
  id: activityId,
  args: ['World'],
});
```

## Execute a Standalone Activity without type checking 

Since Activity types are not always available, the [`start`](https://typescript.temporal.io/api/classes/client.ActivityClient#start)
and [`execute`](https://typescript.temporal.io/api/classes/client.ActivityClient#execute) methods can be
called on [`ActivityClient`](https://typescript.temporal.io/api/classes/client.ActivityClient) directly
without using the typed interface. When called that way, neither the Activity name nor argument types are
checked client-side.

Or use the Temporal CLI.

```typescript
await client.activity.execute('greet', {
  ...activityOptions,
  id: activityId,
  args: [1],
});
```

```bash
temporal activity execute \\
  --type greet \\
  --activity-id my-standalone-activity-id \\
  --task-queue hello-standalone-activities \\
  --start-to-close-timeout 10s \\
  --input '"World"'
```

## Run with Temporal Cloud

All code samples on this page use
[`loadClientConnectConfig()`](https://typescript.temporal.io/api/namespaces/envconfig#loadclientconnectconfig)
to configure the Temporal Client connection. It responds to [environment
variables](/references/client-environment-configuration) and [TOML configuration
files](/references/client-environment-configuration), so the same code works against a local dev
server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal
Cloud](/develop/typescript/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the feature guide
for mTLS and API key setup.

## Next steps

- **[Standalone Activities feature guide](/develop/typescript/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud.
- **[Build a Job Queue with Standalone Activities](https://learn.temporal.io/tutorials/typescript/standalone-activities/)**: Tutorial that builds a durable job queue, with idempotency keys, server-side deduplication, throughput limits, [Priority and Fairness](/develop/task-queue-priority-fairness), and heartbeat checkpoints.
- **[Activity basics](/develop/typescript/activities/basics)**: Write and register Activities with the TypeScript SDK.

---

# Activity Timeouts - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/activities/timeouts

> Set the Schedule-To-Close, Start-To-Close, and Heartbeat Timeouts on a TypeScript Activity, tune its Retry Policy, and override the next retry delay.

## Activity Timeouts 

Each Activity Timeout controls the maximum duration of a different aspect of an Activity Execution.

The following Timeouts are available in the Activity Options:

- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the entire [Activity Execution](/activity-execution), from when the [Activity Task](/tasks#activity-task) is initially scheduled by the Workflow to when the server receives a successful completion for that Activity Task.
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution), from when the Activity Task Execution gets polled by a [Worker](/workers#worker) to when the server receives a successful completion for that Activity Task.
- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is initially scheduled by the Workflow to when a [Worker](/workers#worker) polls the Activity Task Execution. This timeout is non-retryable by design.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

The following properties can be set on the [`ActivityOptions`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions) when creating Activity proxy functions using the [`proxyActivities()`](https://typescript.temporal.io/api/namespaces/workflow#proxyactivities) API:

- [`scheduleToCloseTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions/#scheduletoclosetimeout)
- [`startToCloseTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions/#starttoclosetimeout)
- [`scheduleToStartTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions/#scheduletostarttimeout)

```typescript
const { myActivity } = proxyActivities<typeof activities>({
  scheduleToCloseTimeout: '5m',
  // startToCloseTimeout: "30s", // recommended
  // scheduleToStartTimeout: "60s",
});
```

### Activity Retry Policy 

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided.

To set an Activity's Retry Policy in TypeScript, assign the [`ActivityOptions.retry`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions#retry) property when creating the corresponding Activity proxy function using the [`proxyActivities()`](https://typescript.temporal.io/api/namespaces/workflow#proxyactivities) API.

```typescript
const { myActivity } = proxyActivities<typeof activities>({
  // ...
  retry: {
    initialInterval: '10s',
    maximumAttempts: 5,
  },
});
```

### Activity next retry delay 

The time to wait after a retryable Activity failure until the next retry is attempted is normally determined by that Activity's Retry Policy.
However, an Activity may override that duration when explicitly failing with an [`ApplicationFailure`](https://typescript.temporal.io/api/classes/common.ApplicationFailure) by setting a next Retry delay.

To override the next Retry delay for an `ApplicationFailure` thrown by an Activity in TypeScript, provide the [`nextRetryDelay`](https://typescript.temporal.io/api/interfaces/common.ApplicationFailureOptions#nextretrydelay) property on the object argument of the [`ApplicationFailure.create()`](https://typescript.temporal.io/api/classes/common.ApplicationFailure#create) factory method.

```typescript
throw ApplicationFailure.create({
  // ...
  nextRetryDelay: '15s',
});
```

## Heartbeat an Activity 

An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service).
Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed.
If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered as timed out and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat.
Activities that don't Heartbeat can't get notified of Cancellation requests.

> **📝 Note:**
> Handling Activity Cancellation
> In the TypeScript SDK, Activity implementations must opt in to observe cancellation.
>
> Use one of the following in your Activity to be notified of cancellation:
> - `await Context.current().cancelled` — rejects with `CancelledFailure`.
> - `await Context.current().sleep(ms)` — rejects on cancellation.
> - Pass `Context.current().cancellationSignal` (an `AbortSignal`) to libraries that support abort.
>
> > **Important:** Activities receive cancellation notifications when they heartbeat; if an Activity doesn’t heartbeat, delivery of the cancellation notification can be delayed.
>
> See [Activity namespace reference](https://typescript.temporal.io/api/namespaces/activity#cancellation) and [Context API](https://typescript.temporal.io/api/classes/activity.Context) for more information.

Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker.
Heartbeat throttling may lead to Cancellation getting delivered later than expected.

To Heartbeat an Activity Execution in TypeScript, call the [`heartbeat()`](https://typescript.temporal.io/api/namespaces/activity#heartbeat) function from the Activity implementation.

```typescript
export async function myActivity(): Promise<void> {
  for (let progress = 1; progress <= 1000; ++progress) {
    // Do something that takes time
    await sleep('1s');

    heartbeat();
  }
}
```

An Activity may optionally checkpoint its progression, by providing a `details` argument to the [`heartbeat()`](https://typescript.temporal.io/api/namespaces/activity#heartbeat) function.
Should the Activity Execution times out and gets retried, then the Temporal Server will provide the `details` from the last Heartbeat it received to the next Activity Execution.
This can be used to allow the Activity to efficiently resume its work.

```typescript
export async function myActivity(): Promise<void> {
  // Resume work from latest heartbeat, if there's one, or start from 1 otherwise
  const startingPoint = activityInfo().heartbeatDetails?.progress ?? 1;

  for (let progress = startingPoint; progress <= 1000; ++progress) {
    // Do something that takes time
    await sleep('1s');

    heartbeat({ progress });
  }
}
```

### Activity Heartbeat Timeout 

A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat).
If the Temporal Server doesn't receive a Heartbeat before expiration of the Heartbeat Timeout, the Activity is considered as timed out and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy.

To set an Activity's Heartbeat Timeout in TypeScript, set the [`ActivityOptions.heartbeatTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions#heartbeattimeout) property when creating the corresponding Activity proxy functions using the [`proxyActivities()`](https://typescript.temporal.io/api/namespaces/workflow#proxyactivities) API.

```typescript
const { myLongRunningActivity } = proxyActivities<typeof activities>({
  // ...
  heartbeatTimeout: '30s',
});
```

---

# Best practices - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Best practices

- [Testing](/develop/typescript/best-practices/testing-suite)
- [Debugging](/develop/typescript/best-practices/debugging)
- [Converters and encryption](/develop/typescript/best-practices/data-handling)
- [Entity pattern](/develop/typescript/best-practices/entity-pattern)

---

# Data handling - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling

All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three
layers that handle different concerns:

![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)

Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON
serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when
your application requires non-JSON types, encryption, or payload offloading.

|                           | [PayloadConverter](/develop/typescript/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/typescript/best-practices/data-handling/data-encryption) | [ExternalStorage](/develop/typescript/best-practices/data-handling/external-storage) |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------- |
| **Purpose**               | Serialize application data to bytes                               | Transform encoded payloads (encrypt, compress)                | Offload large payloads to external store                               |
| **Default**               | JSON serialization                                                | None (passthrough)                                            | None (all payloads are stored in Event History)                      |

For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage).

---

# Payload conversion - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling/data-conversion

> Customize how Temporal serializes application objects using Payload Converters in the TypeScript SDK.

Payload Converters serialize your application objects into a `Payload` and deserialize them back.
A `Payload` is a binary form with metadata that Temporal uses to transport data.

By default, Temporal uses a Payload Converter that handles `null`, byte arrays, protobuf messages, and anything JSON-serializable.
You only need a custom Payload Converter when your application uses types that aren't natively supported.

## Default supported types

The default Data Converter supports converting multiple types including:

- `undefined`
- `null`
- `Uint8Array`
- JSON serializables

You can work with [protobufs](#protobufs) like `proto3-json-serializer` or `protobufjs` by using an instance of `DefaultPayloadConverterWithProtobufs`.

## Payload Codec

> API documentation: [PayloadCodec](https://typescript.temporal.io/api/interfaces/common.PayloadCodec)

The default `PayloadCodec` does nothing. To create a custom one, you can implement the following interface:

```ts
interface PayloadCodec {
  /**
   * Encode an array of {@link Payload}s for sending over the wire.
   * @param payloads May have length 0.
   */
  encode(payloads: Payload[]): Promise<Payload[]>;

  /**
   * Decode an array of {@link Payload}s received from the wire.
   */
  decode(payloads: Payload[]): Promise<Payload[]>;
}
```

## Use custom payload conversion

Temporal SDKs provide a [Payload Converter](/payload-converter) that can be customized to convert a custom data type to a [Payload](/dataconversion#payload) and back.

The order in which your encoding Payload Converters are applied depends on the order given to the Data Converter.
You can set multiple encoding Payload Converters to run your conversions.
When the Data Converter receives a value for conversion, the value gets passed through each Payload Converter in sequence until the converter that handles the data type does the conversion.

## Composite Data Converters

Use a [Composite Data Converter](https://typescript.temporal.io/api/classes/common.CompositePayloadConverter) to apply custom, type-specific Payload Converters in a specified order. Defining a new Composite Data Converter is not always necessary to implement custom data handling. You can override the default Converter with a custom Codec, but a Composite Data Converter may be necessary for complex Workflow logic.

A Composite Data Converter can include custom rules created, and it can also
leverage the default Data Converters built into Temporal. In fact, the default
Data Converter logic is implemented internally in the Temporal source as a
Composite Data Converter. It defines these rules in this order:

```typescript
export class DefaultPayloadConverter extends CompositePayloadConverter {
  constructor() {
    super(
      new UndefinedPayloadConverter(),
      new BinaryPayloadConverter(),
      new JsonPayloadConverter(),
    );
  }
}
```

The order of applying the Payload Converters is important. During
serialization, the Data Converter tries the Payload Converters in that specific
order until a Payload Converter returns a non-null Payload.

To replace the default Data Converter with a custom `CompositeDataConverter`, use the following:

```typescript
export const payloadConverter = new CompositePayloadConverter(
  new UndefinedPayloadConverter(),
  new EjsonPayloadConverter(),
);
```

You can do this in its own `payload-converter.ts` file for example.

In the code snippet above, a converter is created that first attempts to handle `null` and `undefined` values. If the value isn't `null` or `undefined`, the EJSON serialization logic written in the `EjsonPayloadConverter` is then used. The Payload Converter is then provided to the Worker and Client.

Here is the Worker code:

```typescript
const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  taskQueue: 'ejson',
  dataConverter: {
    payloadConverterPath: require.resolve('./payload-converter'),
  },
});
```

With this code, you now ensure that the Worker serializes and deserializes Workflow and Activity inputs and outputs using your EJSON-based logic, along with handling undefined values appropriately.

Here is the Client:

```typescript
const client = new Client({
  dataConverter: {
    payloadConverterPath: require.resolve('./payload-converter'),
  },
});
```

You can now use a variety of data types in arguments.

## How to use a custom payload converter in TypeScript 

To support custom Payload conversion, create a [custom Payload Converter](/payload-converter#composite-data-converters) and configure the Data Converter to use it in your Client options.
You can use Custom Payload Converters to change how application objects get serialized to binary Payload. To handle custom data types that are not natively JSON-serializable (for example, `BigInt`, `Date`, or binary data), you can create a custom Payload Converter. A Custom Payload Converter is responsible for converting your custom data types to a payload format that Temporal can manage.

To implement a Custom Payload Converter in TypeScript, you need to do the following steps:

1. **Implement `PayloadConverter` Interface**: Start by creating a class that implements Temporal's [`PayloadConverter`](https://typescript.temporal.io/api/interfaces/common.PayloadConverter) interface.

```typescript
interface PayloadConverter {
  /**
   * Converts a value to a {@link Payload}.
   * @param value The value to convert. Example values include the Workflow args sent by the client and the values returned by a Workflow or Activity.
   */
  toPayload<T>(value: T): Payload;

  /**
   * Converts a {@link Payload} back to a value.
   */
  fromPayload<T>(payload: Payload): T;
}
```

This custom converter should include logic for both serialization (`toPayload`) and deserialization (`fromPayload`), handling your specific data types or serialization format. The method `toPayload` returns a Payload object, which is used to manage and transport serialized data. The method `fromPayload` returns the deserialized data. This ensures that the data returned is in the same format as it was before serialization, allowing it to be used directly in the application.

2. Configure the Data Converter. To send values that are not JSON-serializable like a `BigInt` or `Date`, provide the custom Data Converter to the Client and Worker as described in the [Composite Data Converters](#composite-data-converters) section.

#### Custom implementation

Some example implementations are in the SDK itself:

- [common/src/converter/payload-converter.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/common/src/converter/payload-converter.ts)
- [common/src/converter/protobuf-payload-converters.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/common/src/converter/protobuf-payload-converters.ts)

The sample project [samples-typescript/ejson](https://github.com/temporalio/samples-typescript/tree/main/ejson) creates an EJSON custom `PayloadConverter`.
It implements `PayloadConverterWithEncoding` instead of `PayloadConverter` so that it could be used with [CompositePayloadConverter](https://typescript.temporal.io/api/classes/common.CompositePayloadConverter/):

<!--SNIPSTART typescript-ejson-converter-impl -->
[ejson/src/ejson-payload-converter.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/ejson-payload-converter.ts)
```ts
import {
  EncodingType,
  METADATA_ENCODING_KEY,
  Payload,
  PayloadConverterWithEncoding,
  PayloadConverterError,
} from '@temporalio/common';
import EJSON from 'ejson';
import { decode, encode } from '@temporalio/common/lib/encoding';

/**
 * Converts between values and [EJSON](https://docs.meteor.com/api/ejson.html) Payloads.
 */
export class EjsonPayloadConverter implements PayloadConverterWithEncoding {
  // Use 'json/plain' so that Payloads are displayed in the UI
  public encodingType = 'json/plain' as EncodingType;

  public toPayload(value: unknown): Payload | undefined {
    if (value === undefined) return undefined;
    let ejson;
    try {
      ejson = EJSON.stringify(value);
    } catch (e) {
      throw new UnsupportedEjsonTypeError(
        `Can't run EJSON.stringify on this value: ${value}. Either convert it (or its properties) to EJSON-serializable values (see https://docs.meteor.com/api/ejson.html ), or create a custom data converter. EJSON.stringify error message: ${errorMessage(
          e,
        )}`,
        e as Error,
      );
    }

    return {
      metadata: {
        [METADATA_ENCODING_KEY]: encode('json/plain'),
        // Include an additional metadata field to indicate that this is an EJSON payload
        format: encode('extended'),
      },
      data: encode(ejson),
    };
  }

  public fromPayload<T>(content: Payload): T {
    return content.data ? EJSON.parse(decode(content.data)) : content.data;
  }
}

export class UnsupportedEjsonTypeError extends PayloadConverterError {
  public readonly name: string = 'UnsupportedJsonTypeError';

  constructor(
    message: string | undefined,
    public readonly cause?: Error,
  ) {
    super(message ?? undefined);
  }
}
```
<!--SNIPEND-->

Then we instantiate one and export it:

<!--SNIPSTART typescript-ejson-converter -->
[ejson/src/payload-converter.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/payload-converter.ts)
```ts
import { CompositePayloadConverter, UndefinedPayloadConverter } from '@temporalio/common';
import { EjsonPayloadConverter } from './ejson-payload-converter';

export const payloadConverter = new CompositePayloadConverter(
  new UndefinedPayloadConverter(),
  new EjsonPayloadConverter(),
);
```
<!--SNIPEND-->

We provide it to the Worker and Client:

<!--SNIPSTART typescript-ejson-worker -->
[ejson/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/worker.ts)
```ts
const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  taskQueue: 'ejson',
  dataConverter: { payloadConverterPath: require.resolve('./payload-converter') },
});
```
<!--SNIPEND-->

<!--SNIPSTART typescript-ejson-client-setup -->
[ejson/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/client.ts)
```ts
const client = new Client({
  connection,
  dataConverter: { payloadConverterPath: require.resolve('./payload-converter') },
});
```
<!--SNIPEND-->

Then we can use supported data types in arguments:

<!--SNIPSTART typescript-ejson-client -->
[ejson/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/client.ts)
```ts
const user: User = {
  id: uuid(),
  // age: 1000n, BigInt isn't supported
  hp: Infinity,
  matcher: /.*Stormblessed/,
  token: Uint8Array.from([1, 2, 3]),
  createdAt: new Date(),
};

const handle = await client.workflow.start(example, {
  args: [user],
  taskQueue: 'ejson',
  workflowId: `example-user-${user.id}`,
});
```
<!--SNIPEND-->

And they get parsed correctly for the Workflow:

<!--SNIPSTART typescript-ejson-workflow -->
[ejson/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/workflows.ts)
```ts
import type { Result, User } from './types';

export async function example(user: User): Promise<Result> {
  const success =
    user.createdAt.getTime() < Date.now() &&
    user.hp > 50 &&
    user.matcher.test('Kaladin Stormblessed') &&
    user.token instanceof Uint8Array;
  return { success, at: new Date() };
}
```
<!--SNIPEND-->

#### Protobufs

To serialize values as [Protocol Buffers](https://protobuf.dev/) (protobufs):

- Use [protobufjs](https://protobufjs.github.io/protobuf.js/).
- Use runtime-loaded messages (not generated classes) and `MessageClass.create` (not `new MessageClass()`).
- Generate `json-module.js` with a command like the following:

  ```sh
  pbjs -t json-module --workflow-id commonjs -o protos/json-module.js protos/*.proto
  ```

- Patch `json-module.js`:

<!--SNIPSTART typescript-protobuf-root -->
[protobufs/protos/root.js](https://github.com/temporalio/samples-typescript/blob/main/protobufs/protos/root.js)
```js
const { patchProtobufRoot } = require('@temporalio/common/lib/protobufs');
const unpatchedRoot = require('./json-module');
module.exports = patchProtobufRoot(unpatchedRoot);
```
<!--SNIPEND-->

- Generate `root.d.ts` with the following command:

  ```sh
  pbjs -t static-module protos/*.proto | pbts -o protos/root.d.ts -
  ```

- Create a [`DefaultPayloadConverterWithProtobufs`](https://typescript.temporal.io/api/classes/protobufs.DefaultPayloadConverterWithProtobufs/):

<!--SNIPSTART typescript-protobuf-converter -->
[protobufs/src/payload-converter.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/payload-converter.ts)
```ts
import { DefaultPayloadConverterWithProtobufs } from '@temporalio/common/lib/protobufs';
import root from '../protos/root';

export const payloadConverter = new DefaultPayloadConverterWithProtobufs({ protobufRoot: root });
```
<!--SNIPEND-->

Alternatively, we can use Protobuf Payload Converters directly, or with other converters.
If we know that we only use Protobuf objects, and we want them binary encoded (which saves space over proto3 JSON, but can't be viewed in the Web UI), we could do the following:

```ts
import { ProtobufBinaryPayloadConverter } from '@temporalio/common/lib/protobufs';
import root from '../protos/root';

export const payloadConverter = new ProtobufBinaryPayloadConverter(root);
```

Similarly, if we wanted binary-encoded Protobufs in addition to the other default types, we could do the following:

```ts
import {
  BinaryPayloadConverter,
  CompositePayloadConverter,
  JsonPayloadConverter,
  UndefinedPayloadConverter,
} from '@temporalio/common';
import { ProtobufBinaryPayloadConverter } from '@temporalio/common/lib/protobufs';
import root from '../protos/root';

export const payloadConverter = new CompositePayloadConverter(
  new UndefinedPayloadConverter(),
  new BinaryPayloadConverter(),
  new ProtobufBinaryPayloadConverter(root),
  new JsonPayloadConverter(),
);
```

- Provide it to the Worker:

<!--SNIPSTART typescript-protobuf-worker -->
[protobufs/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/worker.ts)
```ts
const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  activities,
  taskQueue: 'protobufs',
  dataConverter: { payloadConverterPath: require.resolve('./payload-converter') },
});
```
<!--SNIPEND-->

[WorkerOptions.dataConverter](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#dataconverter)

- Provide it to the Client:

<!--SNIPSTART typescript-protobuf-client -->
[protobufs/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/client.ts)
```ts
import { Client, Connection } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { v4 as uuid } from 'uuid';
import { foo, ProtoResult } from '../protos/root';
import { example } from './workflows';

async function run() {
  const config = loadClientConnectConfig();
  const connection = await Connection.connect(config.connectionOptions);
  const client = new Client({
    connection,
    dataConverter: { payloadConverterPath: require.resolve('./payload-converter') },
  });

  const handle = await client.workflow.start(example, {
    args: [foo.bar.ProtoInput.create({ name: 'Proto', age: 2 })],
    // can't do:
    // args: [new foo.bar.ProtoInput({ name: 'Proto', age: 2 })],
    taskQueue: 'protobufs',
    workflowId: 'my-business-id-' + uuid(),
  });

  console.log(`Started workflow ${handle.workflowId}`);

  const result: ProtoResult = await handle.result();
  console.log(result.toJSON());
}
```
<!--SNIPEND-->

- Use protobufs in your Workflows and Activities:

<!--SNIPSTART typescript-protobuf-workflow -->
[protobufs/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/workflows.ts)
```ts
import { proxyActivities } from '@temporalio/workflow';
import { foo, ProtoResult } from '../protos/root';
import type * as activities from './activities';

const { protoActivity } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

export async function example(input: foo.bar.ProtoInput): Promise<ProtoResult> {
  const result = await protoActivity(input);
  return result;
}
```
<!--SNIPEND-->

<!--SNIPSTART typescript-protobuf-activity -->
[protobufs/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/activities.ts)
```ts
import { foo, ProtoResult } from '../protos/root';

export async function protoActivity(input: foo.bar.ProtoInput): Promise<ProtoResult> {
  return ProtoResult.create({ sentence: `${input.name} is ${input.age} years old.` });
}
```
<!--SNIPEND-->

---

# Payload encryption - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling/data-encryption

> Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the TypeScript SDK.

Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures.

A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these.

The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption.

## Custom Payload Codec

> Background: [Encryption](/payload-codec#encryption)

The following is an example class that implements the `PayloadCodec` interface:

<!--SNIPSTART typescript-encryption-codec -->
[encryption/src/encryption-codec.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/encryption-codec.ts)
```ts
import { webcrypto as crypto } from 'node:crypto';
import { METADATA_ENCODING_KEY, Payload, PayloadCodec, ValueError } from '@temporalio/common';
import { temporal } from '@temporalio/proto';
import { decode, encode } from '@temporalio/common/lib/encoding';
import { decrypt, encrypt } from './crypto';

const ENCODING = 'binary/encrypted';
const METADATA_ENCRYPTION_KEY_ID = 'encryption-key-id';

export class EncryptionCodec implements PayloadCodec {
  constructor(
    protected readonly keys: Map<string, crypto.CryptoKey>,
    protected readonly defaultKeyId: string,
  ) {}

  static async create(keyId: string): Promise<EncryptionCodec> {
    const keys = new Map<string, crypto.CryptoKey>();
    keys.set(keyId, await fetchKey(keyId));
    return new this(keys, keyId);
  }

  async encode(payloads: Payload[]): Promise<Payload[]> {
    return Promise.all(
      payloads.map(async (payload) => ({
        metadata: {
          [METADATA_ENCODING_KEY]: encode(ENCODING),
          [METADATA_ENCRYPTION_KEY_ID]: encode(this.defaultKeyId),
        },
        // Encrypt entire payload, preserving metadata
        data: await encrypt(
          temporal.api.common.v1.Payload.encode(payload).finish(),
          this.keys.get(this.defaultKeyId)!, // eslint-disable-line @typescript-eslint/no-non-null-assertion
        ),
      })),
    );
  }

  async decode(payloads: Payload[]): Promise<Payload[]> {
    return Promise.all(
      payloads.map(async (payload) => {
        if (!payload.metadata || decode(payload.metadata[METADATA_ENCODING_KEY]) !== ENCODING) {
          return payload;
        }
        if (!payload.data) {
          throw new ValueError('Payload data is missing');
        }

        const keyIdBytes = payload.metadata[METADATA_ENCRYPTION_KEY_ID];
        if (!keyIdBytes) {
          throw new ValueError('Unable to decrypt Payload without encryption key id');
        }

        const keyId = decode(keyIdBytes);
        let key = this.keys.get(keyId);
        if (!key) {
          key = await fetchKey(keyId);
          this.keys.set(keyId, key);
        }
        const decryptedPayloadBytes = await decrypt(payload.data, key);
        console.log('Decrypting payload.data:', payload.data);
        return temporal.api.common.v1.Payload.decode(decryptedPayloadBytes);
      }),
    );
  }
}

async function fetchKey(_keyId: string): Promise<crypto.CryptoKey> {
  // In production, fetch key from a key management system (KMS). You may want to memoize requests if you'll be decoding
  // Payloads that were encrypted using keys other than defaultKeyId.
  const key = Buffer.from('test-key-test-key-test-key-test!');
  const cryptoKey = await crypto.subtle.importKey(
    'raw',
    key,
    {
      name: 'AES-GCM',
    },
    true,
    ['encrypt', 'decrypt'],
  );

  return cryptoKey;
}
```
<!--SNIPEND-->

The encryption and decryption code is in [src/crypto.ts](https://github.com/temporalio/samples-typescript/tree/main/encryption/src/crypto.ts).
Because encryption is CPU intensive, and doing AES with the crypto module built into Node.js blocks the main thread, we use `@ronomon/crypto-async`, which uses the Node.js thread pool.

As before, we provide a custom Data Converter to the Client and Worker:

<!--SNIPSTART typescript-encryption-client -->
[encryption/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/client.ts)
```ts
const client = new Client({
  connection,
  dataConverter: await getDataConverter(),
});

const handle = await client.workflow.start(example, {
  args: ['Alice: Private message for Bob.'],
  taskQueue: 'encryption',
  workflowId: `my-business-id-${uuid()}`,
});

console.log(`Started workflow ${handle.workflowId}`);
console.log(await handle.result());
```
<!--SNIPEND-->

<!--SNIPSTART typescript-encryption-worker -->
[encryption/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/worker.ts)
```ts
const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  taskQueue: 'encryption',
  dataConverter: await getDataConverter(),
});
```
<!--SNIPEND-->

When the Client sends `'Alice: Private message for Bob.'` to the Workflow, it gets encrypted on the Client and decrypted in the Worker.
The Workflow receives the decrypted message and appends another message.
When it returns that longer string, the string gets encrypted by the Worker and decrypted by the Client.

<!--SNIPSTART typescript-encryption-workflow -->
[encryption/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/workflows.ts)
```ts
export async function example(message: string): Promise<string> {
  return `${message}\nBob: Hi Alice, I'm Workflow Bob.`;
}
```
<!--SNIPEND-->

---

# External Storage - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling/external-storage

> Offload large payloads to external storage using the claim check pattern in the TypeScript SDK.

> **Public Preview**
> APIs and configuration may change before General Availability. Join the
> [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for
> help.

The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted
deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external
storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how
to set up External Storage with Amazon S3 or Google Cloud Storage, and how to implement a custom storage driver.

For a conceptual overview of External Storage and its use cases, see [External Storage](/external-storage).

## Store and retrieve large payloads with Amazon S3 or Google Cloud Storage 

The TypeScript SDK includes storage drivers for Amazon S3 and Google Cloud Storage. Select your storage backend in the
tabs that follow. Only the driver setup differs between the two. Everything after that is the same.

### Prerequisites

- A bucket that you have read and write access to. Refer to
  [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire
  lifetime of the Workflow. For multi-region durability, see
  [Durable External Storage](/external-storage#durable-external-storage).
- Credentials with permission to write objects on components that store payloads, and to read objects on components
  that retrieve them. Components that only retrieve payloads do not need write access, and the reverse is also true.
- Install the driver, the client adapter for your cloud provider's SDK, and that SDK:

**Amazon S3**

  ```sh
  npm install @temporalio/external-storage-s3 \
    @temporalio/external-storage-s3-aws-sdk \
    @aws-sdk/client-s3
  ```

**Google Cloud Storage**

  ```sh
  npm install @temporalio/external-storage-gcs \
    @temporalio/external-storage-gcs-google-sdk \
    @google-cloud/storage
  ```

### Procedure

1. Create a storage client, wrap it in the matching driver client, and pass the result to the driver. Each cloud SDK
   picks up your standard credentials from the environment:

**Amazon S3**

   The AWS SDK reads environment variables, an IAM role, or your AWS config file.

   <!--SNIPSTART typescript-s3-driver-create-->
   [features/snippets/external_storage/s3_setup/s3_driver_create.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_driver_create.ts)
   ```ts
   import { S3Client } from '@aws-sdk/client-s3';
   import { S3StorageDriver } from '@temporalio/external-storage-s3';
   import { AwsSdkS3StorageDriverClient } from '@temporalio/external-storage-s3-aws-sdk';

   const s3Client = new S3Client({ region: 'us-east-2' });

   const driver = new S3StorageDriver({
     client: new AwsSdkS3StorageDriverClient(s3Client),
     bucket: 'my-temporal-payloads',
   });
   ```
   <!--SNIPEND-->

**Google Cloud Storage**

   The Google Cloud SDK reads Application Default Credentials.

   <!--SNIPSTART typescript-gcs-driver-create-->
   [features/snippets/external_storage/gcs_setup/gcs_driver_create.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/gcs_setup/gcs_driver_create.ts)
   ```ts
   import { Storage } from '@google-cloud/storage';
   import { GcsStorageDriver } from '@temporalio/external-storage-gcs';
   import { GoogleCloudGcsStorageDriverClient } from '@temporalio/external-storage-gcs-google-sdk';

   const storage = new Storage();

   const driver = new GcsStorageDriver({
     client: new GoogleCloudGcsStorageDriverClient(storage),
     bucket: 'my-temporal-payloads',
   });
   ```
   <!--SNIPEND-->

   To route payloads to different buckets at runtime, pass a function as `bucket` instead of a string. The function
   receives the store context and the payload, and returns a bucket name.

2. Build an `ExternalStorage` instance from the driver, set it on your Data Converter, and pass the converter to your
   Client and Worker. This step is the same for every driver. External Storage runs outside the Workflow sandbox, so
   you can pass the driver object directly rather than referencing it by path the way a custom Payload Converter
   requires:

   <!--SNIPSTART typescript-s3-external-storage-setup-->
   [features/snippets/external_storage/s3_setup/s3_external_storage_setup.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_external_storage_setup.ts)
   ```ts
   import { Client, Connection } from '@temporalio/client';
   import { ExternalStorage } from '@temporalio/common';
   import { Worker } from '@temporalio/worker';

   async function createClientAndWorker() {
     const dataConverter = {
       externalStorage: new ExternalStorage({ drivers: [driver] }),
     };

     const connection = await Connection.connect();
     const client = new Client({ connection, dataConverter });

     const worker = await Worker.create({
       workflowsPath: require.resolve('./workflows'),
       taskQueue: 'my-task-queue',
       dataConverter,
     });
   }
   ```
   <!--SNIPEND-->

By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the
`payloadSizeThreshold` option, even setting it to `0` to externalize all payloads regardless of size. Refer to
[Configure payload size threshold](#configure-payload-size-threshold) for more information.

All Workflows and Activities running on the Worker use the storage driver automatically without changes to your business
logic. Both drivers upload and download payloads concurrently, address objects by a SHA-256 hash of their contents so
identical payloads are stored once, and verify that hash on retrieve. Each rejects any single payload larger than
`maxPayloadSize`, which defaults to 50 MiB, and includes diagnostic metadata in error messages to help troubleshoot
storage failures.

## Implement a custom storage driver

If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver.
Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and
[Lifecycle management](/external-storage#lifecycle) for retention requirements.

A shared filesystem works for local development and for Workers that mount the same volume. For anything else, use a
storage system that every Client and Worker can reach.

The following sections walk through the key parts of a custom driver.

### 1. Implement the StorageDriver interface

A custom driver implements the `StorageDriver` interface, which has two readonly properties and two methods:

- `name` is a unique string that identifies the driver instance. The SDK stores this name in the claim check reference
  so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks
  retrieval. For example, two S3 drivers could be named `"s3-primary"` and `"s3-archive"`.
- `type` is a string that identifies the driver implementation, and the Worker reports it in its heartbeat. Unlike
  `name`, `type` must be the same across all instances of the same driver type regardless of configuration. Two S3
  drivers named `"s3-primary"` and `"s3-archive"` would both report `"aws.s3driver"` as their type, while the built-in
  GCS driver reports `"gcp.gcsdriver"`.
- `store()` receives an array of payloads and returns one `StorageDriverClaim` per payload. A claim wraps a set of
  string key-value pairs that the driver uses to locate the payload later.
- `retrieve()` receives the claims that `store()` produced and returns the original payloads.

### 2. Store payloads

In `store()`, serialize each Payload protobuf message to bytes and write the bytes to your storage system. The
application data has already been serialized by the
[Payload Converter](/develop/typescript/best-practices/data-handling/data-conversion) and
[Payload Codec](/develop/typescript/best-practices/data-handling/data-encryption) before it reaches the driver. See the
[data conversion pipeline](/external-storage#data-pipeline) for more details.

Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `context.target`
provides identity information and is a discriminated union: check the `kind` property to distinguish
`"workflow"` from `"activity"`, then read `namespace`, `id`, `runId`, and `type`. Consider structuring your storage keys
to include this information so that you can identify which Workflow owns each payload. Within that scope,
content-addressable keys, such as a SHA-256 hash of the payload bytes, can help deduplicate identical payloads. The
built-in S3 and GCS drivers use this approach.

### 3. Retrieve payloads

In `retrieve()`, download the bytes using the claim data, then reconstruct the Payload protobuf message. The Payload
Converter handles deserializing the application data after the driver returns the payload.

### 4. Configure the Data Converter

Pass your driver to an `ExternalStorage` instance on the Data Converter, and use the same converter when creating your
Client and Worker. Both sides need it: a Client without External Storage configured cannot read an offloaded result. You
can also package your driver as a [plugin](/develop/plugins-guide) for easier reuse across services:

<!--SNIPSTART typescript-custom-driver-data-converter -->
```ts
export function createDataConverter(rootDir: string = STORAGE_ROOT): DataConverter {
  return {
    externalStorage: new ExternalStorage({
      drivers: [new FileSystemStorageDriver({ rootDir })],
      payloadSizeThreshold: PAYLOAD_SIZE_THRESHOLD,

      // With one driver registered, every offloaded payload goes to it. Register more
      // than one and a `driverSelector` becomes required, letting you route per
      // payload: a cheap archive tier for a known-bulky Workflow type, a driver per
      // tenant, or a per-region bucket. Returning `null` from the selector keeps that
      // payload inline, which is how you exempt specific payloads from offloading.
      //
      // driverSelector: (context, _payload) =>
      //   context.target?.type === 'processDocument' ? coldDriver : hotDriver,
    }),
  };
}
```
<!--SNIPEND-->

## Configure payload size threshold

You can configure the payload size threshold that triggers external storage. By default, payloads of 256 KiB or larger
are offloaded to external storage. You can adjust this with the `payloadSizeThreshold` option, or set it to `0` to
externalize all payloads regardless of size. Payloads smaller than the threshold stay inline in Event History. The size
compared against the threshold is that of the serialized Payload, which includes its metadata, not just your data.

<!--SNIPSTART typescript-external-storage-threshold-->
[features/snippets/external_storage/threshold/threshold_config.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/threshold/threshold_config.ts)
```ts
const dataConverter = {
  externalStorage: new ExternalStorage({
    drivers: [driver],
    payloadSizeThreshold: 0,
  }),
};
```
<!--SNIPEND-->

## Use multiple storage drivers

When you register multiple drivers, you must provide a `driverSelector` function that chooses which driver stores each
payload. `ExternalStorage` throws if you register more than one driver without a selector. Any driver in the list that
is not selected for storing is still available for retrieval, which is useful when migrating between storage backends.
Return `null` from the selector to keep a specific payload inline in Event History.

Multiple drivers are useful in scenarios such as:

- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one
  you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old
  driver remains available for retrieving existing claims.
- Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3
  for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver
  based on the runtime environment.

Every registered driver needs a distinct `name`. Because `S3StorageDriver` defaults its `name` to `"aws.s3driver"`,
registering two S3 drivers requires setting the `driverName` option on at least one of them.

The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver`
is only registered so the Worker can retrieve payloads that were previously stored with it:

<!--SNIPSTART typescript-external-storage-multiple-drivers-->
[features/snippets/external_storage/multiple_drivers/multiple_drivers.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/multiple_drivers/multiple_drivers.ts)
```ts
const preferredDriver = new S3StorageDriver({
  client: new AwsSdkS3StorageDriverClient(s3Client),
  bucket: 'my-bucket',
});
const legacyDriver = new LegacyStorageDriver();

const externalStorage = new ExternalStorage({
  drivers: [preferredDriver, legacyDriver],
  driverSelector: () => preferredDriver,
});
```
<!--SNIPEND-->

## Multi-region durability with Amazon S3

To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with
[Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) and an
[S3 Multi-Region Access Point (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/), then point the
driver at the MRAP ARN instead of a bucket name. See
[Durable External Storage](/external-storage#durable-external-storage) for the full pattern and trade-offs.

MRAP requests are signed with [SigV4A](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html).
The AWS SDK for JavaScript does not bundle a SigV4A signer, so install one alongside your existing dependencies:

```sh
npm install @aws-sdk/signature-v4a
```

Import the signer once during application startup. The import registers the signer with the AWS SDK, so you do not
reference it directly:

```ts
import '@aws-sdk/signature-v4a';
```

`@aws-sdk/signature-v4-crt` is an alternative implementation backed by the AWS Common Runtime. When both are installed,
the AWS SDK prefers the CRT implementation.

With the signer registered, the only remaining change is the value you pass as `bucket`:

```ts
const driver = new S3StorageDriver({
  client: new AwsSdkS3StorageDriverClient(s3Client),
  bucket: 'arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap',
});
```

---

# Debugging - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices/debugging

> The Temporal TypeScript SDK Debugging guide provides tools and tips for debugging Workflows and Workers in development and production environments. Troubleshoot common issues using the Web UI, Temporal CLI, and more.

The Debugging section of the Temporal TypeScript SDK developer's guide covers tools for debugging and how to troubleshoot common issues.

## How to debug in a development environment 

In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli).

## How to debug in a production environment 

You can debug production Workflows using:

- [Web UI](/web-ui)
- [Temporal CLI](/cli)
- [Replay](/develop/typescript/best-practices/testing-suite#replay)
- [Tracing](/develop/typescript/platform/observability#tracing)
- [Logging](/develop/typescript/platform/observability#logging)

You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance).
For information on setting up SDK metrics, see [Metrics](/develop/typescript/platform/observability#metrics) in the Observability section of the TypeScript SDK developer's guide.

Debug Server performance with [Cloud metrics](/cloud/metrics/openmetrics) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics).

## How to troubleshoot common issues in the TypeScript SDK 

### Two locations to watch

- Workflow Errors are reflected in Temporal Web.
- Worker errors and logs are reflected in the terminal.

If something isn't behaving the way you expect, make sure to check both locations for helpful error messages.

### Stale Workflows

If you are developing Workflows and finding that code isn't executing as expected, the first place to look is whether old Workflows are still running.

If those old Workflows have the same name and are on the same Task Queue, Temporal will try to continue executing them on your new code by design. You may get errors that make no sense to you because:

- Temporal is trying to execute old Workflow code that no longer exists in your codebase.
- Your new Client code is expecting Temporal to execute old Workflow/Activity code it doesn't yet know about.

Stale Workflows are usually a non-issue because the errors generated are just noise from code you no longer want to run. If you need to terminate old stale Workflows, you can do so with Temporal Web or the Temporal CLI.

### Workflow/Activity registration errors

**If your Workflows or Activities are not imported or spelled correctly**, here are some errors we've seen:

- `ApplicationFailure: 'MyFunction' is not a function`
- `Workflow did not register a handler for MyQuery`

Double check that your Workers are registering the right Workflow and Activity Definitions (function names) on the right Task Queues.

**If you are running Temporal in a monorepo**, then your `node_modules` may be in a different location than where Temporal expects to find it by default, which results in errors like:

```bash
[ERROR] Module not found: Error: Can't resolve '@temporalio/workflow/lib/worker-interface.js' in '/src'
```

Our [Next.js tutorial](https://learn.temporal.io/tutorials/typescript/nextjs) is written for people setting up Temporal **within an existing monorepo**, which may be of use here.

When you pass a `workflowsPath`, our Webpack config expects to find `node_modules` in the same or a parent/ancestor directory.

**If you are custom bundling your own Workflows** you may get errors like these:

```bash
[ERROR] Failed to activate workflow {
  runId: 'aaf84a83-51ce-462a-9ab7-6a641a703bff',
  error: ReferenceError: exports is not defined,
  workflowExists: false
}
```

Temporal Workflow Bundles need to [export a set of methods that fit the compiled `worker-interface.ts` from `@temporalio/workflow`](https://github.com/temporalio/sdk-typescript/blob/803d95ada755736c17c5af719fa6c447e4cd75ac/packages/worker/src/workflow/bundler.ts#L180) as an entry point.
We do offer a `bundleWorkflowCode` method to assist you with this, though it uses our Webpack settings.
For more information, see the [Register types](/develop/typescript/workers/run-worker-process#register-types) section.

### Webpack errors

The TypeScript SDK's Worker bundles Workflows based on `workflowsPath` with [Webpack](https://webpack.js.org/) and run them inside v8 isolates.

If Webpack fails to create the bundle, the SDK will throw an error and emit webpack logs using the SDK's [logger](/develop/typescript/platform/observability#logging).

If you do not see Webpack output in your terminal make sure that you have not disabled SDK logging (see reference to `Runtime.install()` in the link above).

**A common mistake for newcomers to the TypeScript SDK is trying to use Node.js built-ins and modules in their Workflow code.** Usually, the best thing to do is move that code to an Activity.

Some common examples that will **not** work in the Workflow isolate:

    Importing node built-in modules

> **🚨 Danger:**
> Antipattern
>
> ```ts
> import fs from 'fs';
>
> const config = fs.readFileSync('config.json', 'utf8');
> ```
>

This is invalid because reading from the filesystem is a non-deterministic operation: the file may change from the time of the original Workflow Execution to when the Workflow is replayed.

You'll typically see an error in this form in the Webpack output:

```
2021-10-14T19:22:00.606Z [INFO] Module not found: Error: Can't resolve 'fs' in '/Users/you/your-project/src'
2021-10-14T19:22:00.606Z [INFO] resolve 'fs' in '/Users/you/your-project/src'
2021-10-14T19:22:00.606Z [INFO]   Parsed request is a module
2021-10-14T19:22:00.606Z [INFO]   using description file: /Users/you/your-project/package.json (relative path: ./src)
2021-10-14T19:22:00.606Z [INFO]     Field 'browser' doesn't contain a valid alias configuration
```

    Importing and calling Activities directly from Workflow code

> **🚨 Danger:**
> Antipattern
>
> ```ts
> import { makeHTTPRequest } from './activities';
>
> export async function yourWorkflow(): Promise<string> {
>   return await makeHTTPRequest('https://temporal.io');
> }
> ```
>

This is invalid because activity implementations should not be directly referenced by Workflow code.
Activities are used by Workflows in order to make network calls and read from the filesystem, operations which are non-deterministic by nature because they rely on external state.
Temporal records Activity results in the Event history and in case your Workflow is replayed, completed Activities will not be rerun, instead their recorded result will be delivered to the Workflow.

You'll typically see an error in this form in the Webpack output:

```
2021-10-14T19:46:52.731Z [INFO] ERROR in ./src/activities.ts 8:31-46
2021-10-14T19:46:52.731Z [INFO] Module not found: Error: Can't resolve 'http' in '/Users/you/your-project/src'
2021-10-14T19:46:52.731Z [INFO]
2021-10-14T19:46:52.731Z [INFO] BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
2021-10-14T19:46:52.731Z [INFO] This is no longer the case. Verify if you need this module and configure a polyfill for it.
2021-10-14T19:46:52.731Z [INFO]
2021-10-14T19:46:52.731Z [INFO] If you want to include a polyfill, you need to:
2021-10-14T19:46:52.731Z [INFO]         - add a fallback 'resolve.fallback: { "http": require.resolve("stream-http") }'
2021-10-14T19:46:52.731Z [INFO]         - install 'stream-http'
2021-10-14T19:46:52.731Z [INFO] If you don't want to include a polyfill, you can use an empty module like this:
2021-10-14T19:46:52.731Z [INFO]         resolve.fallback: { "http": false }
```

To properly call your Activities from Workflow code use `proxyActivities` and make sure to only import the Activity types.

```ts
import { proxyActivities } from '@temporalio/workflow';

import type * as activities from './activities';
const { makeHTTPRequest } = proxyActivities<typeof activities>();

export async function yourWorkflow(): Promise<string> {
  return await makeHTTPRequest('https://temporal.io');
}
```

### Works in dev but not in prod

The two main sources of dev-prod discrepancies are in bundling and connecting.

#### Production bundling

You may experience your Client sending stripped names as the Workflow "Type" when scheduling a Workflow.
Webpack can change the Workflow's function name to something shorter.
Temporal won't know how to handle the mismatch between the shorter name and the expected Workflow type.

You may experience errors like this:

```
Error: 3 INVALID_ARGUMENT: WorkflowType is not set on request.
```

Or you may see shorter names in the Temporal Service's Web UI when Webpack changed the Workflow's function name to something shorter, in this case the single letter 's':

![Temporal Web UI showing stripped ](https://docs.temporal.io/img/webui/stripped_workflow_types_in_webui.png)

This issue can happen when your bundler strips out Workflow function names.
Temporal relies on those names to set the "Workflow Type" in the Service Web UI.
To prevent the build process from shortening Workflow function names, modify the webpack configuration file ( `webpack.config.js`) to set the Boolean that retains the original names in the `TerserPlugin` configuration section.
Setting the option (`keep_fnames`) to `true` prevents name stripping.

**webpackterser**

```js
// webpack.config.js
module.exports = {
  optimization: {
    minimize: true,
    minimizer: [
      new TerserPlugin({
        terserOptions: {
          keep_fnames: true, // don't strip function names in production
        },
      }),
    ],
  },
};
```

**esbuild**

```js
require('esbuild').buildSync({
  entryPoints: ['app.js'],
  minify: true,
  keepNames: true,
  outfile: 'out.js',
});
```

See the [esbuild docs](https://esbuild.github.io/api/#keep-names) for more information.

#### Connecting to Temporal Server

If you are trying to connect in production and getting this:

```bash
[TransportError: transport error]
```

It is a sign that something is wrong with your Cert/Key pair.
Log it out and make sure it is an exact match with what is expected (often, the issue can be whitespace when injecting from your production secrets management environment).

### Resetting Workflows to deal with logical bugs

You can "rewind time" using the Temporal CLI, resetting Event History to some previous point in time. You can read the Temporal CLI docs on:

- [Restarting and resetting Workflows by ID](/cli)
- [Resetting all Workflows by binary checksum identifier](/cli)

If you need to reset programmatically, the TS SDK does not have any high level APIs for this, but you can make raw gRPC calls to [resetWorkflowExecution](https://typescript.temporal.io/api/classes/proto.temporal.api.workflowservice.v1.WorkflowService-1/#resetworkflowexecution).

Resetting should only be used to deal with serious logical bugs in your code: it's not for handling transient failures, like a downstream service being unreachable. It should not be used in the course of normal application flows.

### gRPC call timeouts (context deadline exceeded)

The opaque `context deadline exceeded` error comes from `gRPC`:

```
Error: 4 DEADLINE_EXCEEDED: context deadline exceeded
    at Object.callErrorFromStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client.js:179:52)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 4,
  details: 'context deadline exceeded',
  metadata: Metadata {
    internalRepr: Map(1) { 'content-type' => [Array] },
    options: {}
  },
  page: '/api/getBuyState'
}
```

Several conditions can cause this error, including network hiccups, timeouts that are too short, and an overloaded server.
Querying a Workflow Execution whose query handler causes an error can result in the query call timing out.

Some troubleshooting actions you can take:

- Verify the connection from your Worker to the Temporal Server is working and doesn't have unusually high latency.
- If you are running Temporal Server yourself, check your [server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics) to ensure it's not overloaded.
- If what's timing out is a query, check the logs of your Workers to see if they are having issues handling the query.

If none of the preceding actions help you discover why timeouts are occurring, please try to produce a minimal repro and we'll be glad to help.

---

# Entity pattern - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices/entity-pattern

> Implement the Single-Entity Design Pattern in TypeScript to manage Workflow iterations and handle Signals, ensuring efficient Workflow Execution with updates.

### Single-entity design pattern in TypeScript 

The following is a simple pattern that represents a single entity.
It tracks the number of iterations regardless of frequency, and calls `continueAsNew` while properly handling pending updates from Signals.

```ts
interface Input {
  /* Define your Workflow input type here */
}
interface Update {
  /* Define your Workflow update type here */
}

const MAX_ITERATIONS = 1;

export async function entityWorkflow(
  input: Input,
  isNew = true,
): Promise<void> {
  try {
    const pendingUpdates = Array<Update>();
    setHandler(updateSignal, (updateCommand) => {
      pendingUpdates.push(updateCommand);
    });

    if (isNew) {
      await setup(input);
    }

    for (let iteration = 1; iteration <= MAX_ITERATIONS; ++iteration) {
      // Ensure that we don't block the Workflow Execution forever waiting
      // for updates, which means that it will eventually Continue-As-New
      // even if it does not receive updates.
      await condition(() => pendingUpdates.length > 0, '1 day');

      while (pendingUpdates.length) {
        const update = pendingUpdates.shift();
        await runAnActivityOrChildWorkflow(update);
      }
    }
  } catch (err) {
    if (isCancellation(err)) {
      await CancellationScope.nonCancellable(async () => {
        await cleanup();
      });
    }
    throw err;
  }
  await continueAsNew<typeof entityWorkflow>(input, false);
}
```

---

# Testing - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/best-practices/testing-suite

> The Testing section of the Temporal Application development guide covers frameworks for Workflow and integration testing, including end-to-end, integration, unit testing, and time-skipping functionalities.

The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing.

In the context of Temporal, you can create these types of automated tests:

- **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client.
- **Integration:** Anything between end-to-end and unit testing.
  - Running Activities with mocked Context and other SDK imports (and usually network requests).
  - Running Workers with mock Activities, and using a Client to start Workflows.
  - Running Workflows with mocked SDK imports.
- **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls.

We generally recommend writing the majority of your tests as integration tests.

Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers.

## Test frameworks 

Some SDKs have support or examples for popular test frameworks, runners, or libraries.

TypeScript has sample tests for [Jest](https://jestjs.io/) and [Mocha](https://mochajs.org/).

**Jest**

- Minimum Jest version: `27.0.0`
- [Sample test file](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/workflows.test.ts)
- [`jest.config.js`](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/jest.config.js) (must use [`testEnvironment: 'node'`](https://jestjs.io/docs/configuration#testenvironment-string); `testEnvironment: 'jsdom'` is not supported)

**Mocha**

- [Sample test file](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/mocha/workflows.test.ts)
- Test coverage library: [`@temporalio/nyc-test-coverage`](https://github.com/temporalio/sdk-typescript/tree/main/packages/nyc-test-coverage)

## Testing Activities 

An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity.
This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity.

### Run an Activity 

If an Activity references its context, you need to mock that context when testing in isolation.

First, create a [`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment).
The constructor accepts an optional partial Activity [`Info`](https://typescript.temporal.io/api/interfaces/activity.Info) object in case any info fields are needed for the test.

Then use [`MockActivityEnvironment.run()`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment#run) to run a function in an Activity [Context](https://typescript.temporal.io/api/classes/activity.Context).

```ts
import { activityInfo } from '@temporalio/activity';
import { MockActivityEnvironment } from '@temporalio/testing';
import assert from 'assert';

// A function that takes two numbers and returns a promise that resolves to the sum of the two numbers
// and the current attempt.
async function activityFoo(a: number, b: number): Promise<number> {
  return a + b + activityInfo().attempt;
}

// Create a MockActivityEnvironment with attempt set to 2. Run the activityFoo
// function with parameters 5 and 35. Assert that the result is 42.
const env = new MockActivityEnvironment({ attempt: 2 });
const result = await env.run(activityFoo, 5, 35);
assert.equal(result, 42);
```

### Listen to Heartbeats 

When an Activity sends a Heartbeat, be sure that you can see the Heartbeats in your test code so that you can verify them.

[`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment) is an [`EventEmitter`](https://nodejs.org/api/events.html#class-eventemitter) that emits a `heartbeat` event that you can use to listen for Heartbeats emitted by the Activity.

When an Activity is run by a Worker, Heartbeats are throttled to avoid overloading the server.
`MockActivityEnvironment`, however, does not throttle Heartbeats.

```ts
import { heartbeat } from '@temporalio/activity';
import assert from 'assert';

async function activityFoo(): Promise<void> {
  heartbeat(6);
}

const env = new MockActivityEnvironment();

env.on('heartbeat', (d: unknown) => {
  assert(d === 6);
});

await env.run(activityFoo);
```

### Cancel an Activity 

If an Activity is supposed to react to a Cancellation, you can test whether it reacts correctly by canceling it.

[`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment) exposes a [`.cancel()`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment#cancel) method that cancels the Activity Context.

```ts
import { CancelledFailure, heartbeat, sleep } from '@temporalio/activity';
import { MockActivityEnvironment } from '@temporalio/testing';
import assert from 'assert';

async function activityFoo(): Promise<void> {
  heartbeat(6);
  // @temporalio/activity's sleep() is Cancellation-aware, which means that on Cancellation,
  // CancelledFailure will be thrown from it.
  await sleep(100);
}

const env = new MockActivityEnvironment();

env.on('heartbeat', (d: unknown) => {
  assert(d === 6);
});

await assert.rejects(env.run(activityFoo), (err) => {
  assert.ok(err instanceof CancelledFailure);
});
```

## Testing Workflows 

### How to mock Activities 

Mock the Activity invocation when unit testing your Workflows.

When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker.

Implement only the relevant Activities for the Workflow being tested.

```ts
import type * as activities from './activities';

// Creating a mock object of the activities.
const mockActivities: Partial<typeof activities> = {
  makeHTTPRequest: async () => '99',
};

// Creating a worker with the mocked activities.
const worker = await Worker.create({
  activities: mockActivities,
  // ...
});
```

### How to skip time 

Some long-running Workflows can persist for months or even years.
Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount.

For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works.
Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner.

The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time.
Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests.
If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server.
For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel.

#### Set up time skipping 

Make sure you have the `testing` package installed for the TypeScript SDK:

```bash
npm install @temporalio/testing
```

The `@temporalio/testing` package downloads the test server and exports [`TestWorkflowEnvironment`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment), which you use to connect the Client and Worker to the test server and interact with the test server.

[`TestWorkflowEnvironment.createTimeSkipping`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#createtimeskipping) starts the test server.
A typical test suite should set up a single instance of the test environment to be reused in all tests (for example, in a [Jest](https://jestjs.io/) `beforeAll` hook or a [Mocha](https://mochajs.org/) `before()` hook).

```typescript
import { TestWorkflowEnvironment } from '@temporalio/testing';

let testEnv: TestWorkflowEnvironment;

// beforeAll and afterAll are injected by Jest
beforeAll(async () => {
  testEnv = await TestWorkflowEnvironment.createTimeSkipping();
});

afterAll(async () => {
  await testEnv?.teardown();
});
```

`TestWorkflowEnvironment` has [`client`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#client) and [`nativeConnection`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#nativeconnection) for creating Workers:

```typescript
import { Worker } from '@temporalio/worker';
import { v4 as uuid4 } from 'uuid';
import { workflowFoo } from './workflows';

test('workflowFoo', async () => {
  const worker = await Worker.create({
    connection: testEnv.nativeConnection,
    taskQueue: 'test',
    ...
  });
  const result = await worker.runUntil(
    testEnv.client.workflow.execute(workflowFoo, {
      workflowId: uuid4(),
      taskQueue: 'test',
    })
  );
  expect(result).toEqual('foo');
});
```

This test uses the test connection to create a Worker, runs the Worker until the Workflow is complete, and then makes an assertion about the Workflow's result.
The Workflow is executed using `testEnv.client.workflow`, which is connected to the test server.

#### Skip time automatically 

Start a test server process that skips time as needed.
For example, in the time-skipping mode, Timers, which include sleeps and conditional timeouts, are fast-forwarded except when Activities are running.

The test server starts in "normal" time.
When you use `TestWorkflowEnvironment.client.workflow.execute()` or `.result()`, the test server switches to "skipped" time mode until the Workflow completes.
In "skipped" mode, timers (`sleep()` calls and `condition()` timeouts) are fast-forwarded except when Activities are running.

`workflows.ts`

```ts
import { sleep } from '@temporalio/workflow';

export async function sleeperWorkflow() {
  await sleep('1 day');
}
```

`test.ts`

```ts
import { sleeperWorkflow } from './workflows';

test('sleep completes almost immediately', async () => {
  const worker = await Worker.create({
    connection: testEnv.nativeConnection,
    taskQueue: 'test',
    workflowsPath: require.resolve('./workflows'),
  });
  // Does not wait an entire day
  await worker.runUntil(
    testEnv.client.workflow.execute(sleeperWorkflow, {
      workflowId: uuid(),
      taskQueue: 'test',
    }),
  );
});
```

#### Skip time manually 

You can call `testEnv.sleep()` from your test code to advance the test server's time.
This is useful for testing intermediate states or indefinitely long-running Workflows.
However, to use `testEnv.sleep()`, you need to avoid automatic time skipping by starting the Workflow with `.start()` instead of `.execute()` (and not calling `.result()`).

`workflow.ts`

```ts
import { sleep } from '@temporalio/workflow';
import { defineQuery, setHandler } from '@temporalio/workflow';

export const daysQuery = defineQuery('days');

export async function sleeperWorkflow() {
  let numDays = 0;

  setHandler(daysQuery, () => numDays);

  for (let i = 0; i < 100; i++) {
    await sleep('1 day');
    numDays++;
  }
}
```

`test.ts`

```ts
test('sleeperWorkflow counts days correctly', async () => {
  const worker = await Worker.create({
    connection: testEnv.nativeConnection,
    taskQueue: 'test',
    workflowsPath: require.resolve('./workflows'),
  });

  // `start()` starts the test server in "normal" mode, not skipped time mode.
  // If you don't advance time using `testEnv.sleep()`, then `sleeperWorkflow()`
  // will run for days.
  handle = await testEnv.client.workflow.start(sleeperWorkflow, {
    workflowId: uuid4(),
    taskQueue,
  });

  worker.run();

  let numDays = await handle.query(daysQuery);
  assert.equal(numDays, 0);

  // Advance the test server's time by 25 hours
  await testEnv.sleep('25 hours');
  numDays = await handle.query(daysQuery);
  assert.equal(numDays, 1);

  await testEnv.sleep('25 hours');
  numDays = await handle.query(daysQuery);
  assert.equal(numDays, 2);
});
```

#### Skip time in Activities 

Call [`TestWorkflowEnvironment.sleep`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#sleep) from the mock Activity.

In the following test, `processOrderWorkflow` sends a notification to the user after one day.
The `processOrder` mocked Activity calls `testEnv.sleep(‘2 days')`, during which the Workflow sends email (by calling the `sendNotificationEmail` Activity).

Then, after the Workflow completes, we assert that `sendNotificationEmail` was called.

Workflow implementation

<!--SNIPSTART typescript-timer-reminder-workflow-->
[timer-examples/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/timer-examples/src/workflows.ts)
```ts
export async function processOrderWorkflow({
  orderProcessingMS,
  sendDelayedEmailTimeoutMS,
}: ProcessOrderOptions): Promise<string> {
  let processing = true;
  // Dynamically define the timeout based on given input
  const { processOrder } = proxyActivities<ReturnType<typeof createActivities>>({
    startToCloseTimeout: orderProcessingMS,
  });

  const processOrderPromise = processOrder().then(() => {
    processing = false;
  });

  await Promise.race([processOrderPromise, sleep(sendDelayedEmailTimeoutMS)]);

  if (processing) {
    await sendNotificationEmail();

    await processOrderPromise;
  }

  return 'Order completed!';
}
```
<!--SNIPEND-->

<!--SNIPSTART typescript-timer-reminder-test-->
[timer-examples/src/test/workflows.test.ts](https://github.com/temporalio/samples-typescript/blob/main/timer-examples/src/test/workflows.test.ts)
```ts
it('sends reminder email if processOrder does not complete in time', async () => {
  // This test doesn't actually take days to complete: the TestWorkflowEnvironment starts the
  // Test Server, which automatically skips time when there are no running Activities.
  let emailSent = false;
  const mockActivities: ReturnType<typeof createActivities> = {
    async processOrder() {
      // Test server switches to "normal" time while an Activity is executing.
      // Call `env.sleep` to skip ahead 2 days, by which time sendNotificationEmail
      // should have been called.
      await env.sleep('2 days');
    },
    async sendNotificationEmail() {
      emailSent = true;
    },
  };
  const worker = await Worker.create({
    connection: env.nativeConnection,
    taskQueue: 'test',
    workflowsPath: require.resolve('../workflows'),
    activities: mockActivities,
  });
  await worker.runUntil(
    env.client.workflow.execute(processOrderWorkflow, {
      workflowId: uuid(),
      taskQueue: 'test',
      args: [{ orderProcessingMS: ms('3 days'), sendDelayedEmailTimeoutMS: ms('1 day') }],
    }),
  );
  assert.ok(emailSent);
});
```
<!--SNIPEND-->

### Test functions in Workflow context 

For a function or method to run in the Workflow context (where it's possible to get the current Workflow info, or running inside the sandbox in the case of TypeScript or Python), it needs to be run by the Worker as if it were a Workflow.

> **📝 Note:**
>
> This section is applicable in Python and TypeScript.
> In Python, we allow testing of Workflows only and not generic Workflow-related code.
>

To test a function in your Workflow code that isn't a Workflow, put the file it's exported from in [WorkerOptions.workflowsPath](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#workflowspath).
Then execute the function as if it were a Workflow:

`workflows/file-with-workflow-function-to-test.ts`

```ts
import { sleep } from '@temporalio/workflow';

export async function functionToTest(): Promise<number> {
  await sleep('1 day');
  return 42;
}
```

`test.ts`

```ts
const worker = await Worker.create({
  connection: testEnv.nativeConnection,
  workflowsPath: require.resolve(
    './workflows/file-with-workflow-function-to-test',
  ),
});

const result = await worker.runUntil(
  testEnv.client.workflow.execute(functionToTest, workflowOptions),
);

assert.equal(result, 42);
```

If `functionToTest` starts a Child Workflow, that Workflow must be exported from the same file (so that the Worker knows about it):

```ts
import { sleep } from '@temporalio/workflow';
import { someWorkflowToRunAsChild } from './some-workflow';

export { someWorkflowToRunAsChild };

export async function functionToTest(): Promise<number> {
  const result = await wf.executeChild(someWorkflowToRunAsChild);
  return result + 42;
}
```

### Assert in Workflow 

The `assert` statement is a convenient way to insert debugging assertions into the Workflow context.

The `assert` method is available in Python and TypeScript.

The Node.js [`assert`](https://nodejs.org/api/assert.html) module is included in Workflow bundles.

By default, a failed `assert` statement throws `AssertionError`, which causes a [Workflow Task](/tasks#workflow-task) to fail and be indefinitely retried.

To prevent this behavior, use [`workflowInterceptorModules`](https://typescript.temporal.io/api/namespaces/testing/#workflowinterceptormodules) from `@temporalio/testing`.
These interceptors catch an `AssertionError` and turn it into an `ApplicationFailure` that fails the entire Workflow Execution (not just the Workflow Task).

`workflows/file-with-workflow-function-to-test.ts`

```ts
import assert from 'assert';

export async function functionToTest() {
  assert.ok(false);
}
```

`test.ts`

```ts
import {
  TestWorkflowEnvironment,
  workflowInterceptorModules,
} from '@temporalio/testing';

const worker = await Worker.create({
  connection: testEnv.nativeConnection,
  interceptors: {
    workflowModules: workflowInterceptorModules,
  },
  workflowsPath: require.resolve(
    './workflows/file-with-workflow-function-to-test',
  ),
});

await worker.runUntil(
  testEnv.client.workflow.execute(functionToTest, workflowOptions), // throws WorkflowFailedError
);
```

## How to Replay a Workflow Execution 

Replay recreates the exact state of a Workflow Execution.
You can replay a Workflow from the beginning of its Event History.

Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view.

When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks:

1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test.
2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI.
3. Run the Event Histories through replay.
4. Fail CI if any error is encountered during replay.

The following are examples of fetching and replaying Event Histories:

To replay a single Event History, use [worker.runReplayHistory](https://typescript.temporal.io/api/classes/worker.Worker#runreplayhistory).

When an Event History is replayed and non-determinism is detected (that is, the Workflow code is incompatible with the History), [DeterminismViolationError](https://typescript.temporal.io/api/classes/workflow.DeterminismViolationError) is thrown.
If replay fails for any other reason, [ReplayError](https://typescript.temporal.io/api/classes/worker.ReplayError) is thrown.

In the following example, a single Event History is loaded from a JSON file on disk (as obtained from the [Web UI](/web-ui) or the [Temporal CLI](/cli/command-reference/workflow#show)):

```ts
const filePath = './history_file.json';
const history = await JSON.parse(fs.promises.readFile(filePath, 'utf8'));
await Worker.runReplayHistory(
  {
    workflowsPath: require.resolve('./your/workflows'),
  },
  history,
);
```

Alternatively, we can download the Event History programmatically using a Client:

```ts
const connection = await Connection.connect({ address });
const client = new Client({ connection, namespace: 'your-namespace' });
const handle = client.workflow.getHandle('your-workflow-id');
const history = await handle.fetchHistory();
await Worker.runReplayHistory(
  {
    workflowsPath: require.resolve('./your/workflows'),
  },
  history,
);
```

To gain confidence that changes to a Workflow are safe to deploy, we recommend that you obtain Event Histories from the relevant Task Queue and replay them in bulk.
You can do so by combining the [Client.workflow.list()](https://typescript.temporal.io/api/classes/client.WorkflowClient#list) and [worker.runReplayHistories()](https://typescript.temporal.io/api/classes/worker.Worker#runreplayhistories) APIs.

In the following example (which, as of server 1.18, requires [Advanced Visibility](/visibility#advanced-visibility) to be enabled), Event Histories are downloaded from the server and then replayed by passing in a client and a set of Workflows Executions.
The [results](https://typescript.temporal.io/api/interfaces/worker.ReplayResult) returned by the async iterator contain information about the Workflow Execution and whether an error occurred during replay.

```ts
const executions = client.workflow.list({
  query: 'TaskQueue=foo and StartTime > "2022-01-01T12:00:00"',
});
const histories = executions.intoHistories();
const results = Worker.runReplayHistories(
  {
    workflowsPath: require.resolve('./your/workflows'),
  },
  histories,
);
for await (const result of results) {
  if (result.error) {
    console.error('Replay failed', result);
  }
}
```

---

# Client - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/client

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Temporal Client

- [Temporal Client](/develop/typescript/client/temporal-client)
- [Namespaces](/develop/typescript/client/namespaces)

---

# Namespaces - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/client/namespaces

> Register and manage Namespaces with the Temporal CLI or SDK APIs. Isolate Workflow Executions, match development lifecycles, and restrict access with a custom Authorizer.

This page shows how to do the following:

- [Register a Namespace](#register-namespace)
- [Manage Namespaces](#manage-namespaces)

You can create, update, deprecate or delete your [Namespaces](/namespaces) using either the Temporal CLI or SDK APIs.

Use Namespaces to isolate your Workflow Executions according to your needs.
For example, you can use Namespaces to match the development lifecycle by having separate `dev` and `prod` Namespaces.
You could also use them to ensure Workflow Executions between different teams never communicate - such as ensuring that the `teamA` Namespace never impacts the `teamB` Namespace.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) to create and manage a Namespace from the UI, or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces from the command line.

On self-hosted Temporal Service, you can register and manage your Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

You must register a Namespace with the Temporal Service before setting it in the Temporal Client.

## Register a Namespace 

Registering a Namespace creates a Namespace on the Temporal Service or Temporal Cloud.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to create Namespaces.

On self-hosted Temporal Service, you can register your Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

## Manage Namespaces 

You can get details for your Namespaces, update Namespace configuration, and deprecate or delete your Namespaces.

On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces.

On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs.
Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud).

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces.

You must register a Namespace with the Temporal Service before setting it in the Temporal Client.

---

# Temporal Client - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/client/temporal-client

A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service.
Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and
Queries to Workflow Executions, getting Workflow results, and more. You cannot initialize a Temporal Client inside a
Workflow. However, they're commonly initialized inside an Activity to communicate with a Temporal Service.

For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow.

This page shows you how to do the following using the TypeScript SDK with the Temporal Client:

- [Connect to a local development Temporal Service](#connect-to-development-service)
- [Connect to Temporal Cloud](#connect-to-temporal-cloud)
- [Connect to Temporal Service from a Worker](#connect-to-temporal-service-from-a-worker)
- [Start a Workflow Execution](#start-workflow-execution)
- [Get Workflow results](#get-workflow-results)

In the TypeScript SDK, connecting to Temporal Service from a Temporal Application and from within an Activity rely on a
different type of connection than connecting from a Worker. The sections
[Connect to a local development Temporal Service](#connect-to-development-service) and
[Connect to Temporal Cloud](#connect-to-temporal-cloud) apply to connecting from a Temporal Application or from within
an Activity. See [Connect to Temporal Service from a Worker](#connect-to-temporal-service-from-a-worker) for details on
connecting from a Worker.

## Connect to development Temporal Service 

To connect to a development Temporal service from a Temporal Application or from within an Activity, import the
[`Connection` class](https://typescript.temporal.io/api/classes/client.Connection) from `@temporalio/client` and use
[`Connection.connect`](https://typescript.temporal.io/api/classes/client.Connection#connect) to create a Connection
object to connect to the Temporal Service. Then pass in that connection when you create a new `Client` instance. If you
leave the connection options empty, the SDK defaults to connecting to `127.0.0.1:7233` in the `default` Namespace.

```ts
import { Connection, Client } from '@temporalio/client';

async function run() {
  const connection = await Connection.connect();

  // your code goes here

  const client = new Client({ connection });
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

If you need to connect to a Temporal Service with custom options, you can provide connection options directly in code,
load them from **environment variables**, or a **TOML configuration file** using the `@temporalio/envconfig` helpers. We
recommend environment variables or a configuration file for secure, repeatable configuration.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client.

You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path
to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path
`~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to
[Environment Configuration](/develop/environment-configuration) for more details about configuration files and profiles.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own
set of connection options.

```toml title="config.toml"
# Default profile for local development
[profile.default]
address = "localhost:7233"
namespace = "default"

# Optional: Add custom gRPC headers
[profile.default.grpc_meta]
my-custom-header = "development-value"
trace-id = "dev-trace-123"

# Production profile for Temporal Cloud
[profile.prod]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"

# TLS configuration for production
[profile.prod.tls]
# TLS auto-enables when TLS config or an API key is present
# disabled = false
client_cert_path = "/etc/temporal/certs/client.pem"
client_key_path  = "/etc/temporal/certs/client.key"

# Custom headers for production
[profile.prod.grpc_meta]
environment     = "production"
service-version = "v1.2.3"
```

You can create a Temporal Client using a profile from the configuration file as follows. In this example, you load the
`default` profile for local development:

[env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts)
```ts {17-19,28-29}
import { Connection, Client } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { resolve } from 'path';

async function main() {
  console.log('--- Loading default profile from config.toml ---');

  // For this sample to be self-contained, we explicitly provide the path to
  // the config.toml file included in this directory.
  // By default though, the config.toml file will be loaded from
  // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS).
  const configFile = resolve(__dirname, '../config.toml');

  // loadClientConnectConfig is a helper that loads a profile and prepares
  // the configuration for Connection.connect and Client. By default, it loads the
  // "default" profile.
  const config = loadClientConnectConfig({
    configSource: { path: configFile },
  });

  console.log(`Loaded 'default' profile from ${configFile}.`);
  console.log(`  Address: ${config.connectionOptions.address}`);
  console.log(`  Namespace: ${config.namespace}`);
  console.log(`  gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`);

  console.log('\nAttempting to connect to client...');
  try {
    const connection = await Connection.connect(config.connectionOptions);
    const client = new Client({ connection, namespace: config.namespace });
    console.log('✅ Client connected successfully!');
    await connection.close();
  } catch (err) {
    console.log(`❌ Failed to connect: ${err}`);
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

**Environment Variables**

Use the `@temporalio/envconfig` module to set connection options for the Temporal Client using environment variables.
For a list of all available environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

Set the following environment variables before running your application. Replace the placeholder values with your actual
configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and the
`default` Namespace. You may omit these variables entirely since they're the defaults.

```bash
export TEMPORAL_NAMESPACE="default"
export TEMPORAL_ADDRESS="localhost:7233"
```

After setting the environment variables, use the following code to create the Temporal Client. Since the environment
variables take precedence, they will override any values set in the configuration file. Therefore, you may leave
`loadClientConnectConfig`'s arguments empty:

[env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts)
```ts {7,17-18}
import { Connection, Client } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { resolve } from 'path';

async function main() {
// ...
  const config = loadClientConnectConfig({
// ...
  });
// ...
  console.log(`  Address: ${config.connectionOptions.address}`);
  console.log(`  Namespace: ${config.namespace}`);
  console.log(`  gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`);

  console.log('\nAttempting to connect to client...');
  try {
    const connection = await Connection.connect(config.connectionOptions);
    const client = new Client({ connection, namespace: config.namespace });
    console.log('✅ Client connected successfully!');
    await connection.close();
  } catch (err) {
    console.log(`❌ Failed to connect: ${err}`);
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

**Code**

If you don't want to use environment variables or a configuration file, you can specify connection options directly in
code. This is convenient for local development and testing. You can also load a base configuration from environment
variables or a configuration file, and then override specific options in code.

```ts
const connection = await Connection.connect({
    address: <endpoint>,
    tls: true,
    apiKey: <APIKey>,
});
const client = new Client({
    connection,
    namespace: <namespace_id>.<account_id>,
});
```

## Connect to Temporal Cloud 

You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal
Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured
local development instance:

- Your credentials for authentication.
  - If you are using an API key, provide the API key value.
  - If you are using mTLS, provide the mTLS CA certificate and mTLS private key.
- Your [_Namespace Id_](/cloud/namespaces#temporal-cloud-namespace-id) and _Account Id_ combination, which follows the
  format `<namespace_id>.<account_id>`.
- The recommended _endpoint_ is the gRPC Namespace endpoint: `<namespace>.<account>.tmprl.cloud:7233`. This endpoint
  works for all Namespaces and automatically directs traffic to the active region for Namespaces with
  [High Availability](/cloud/high-availability). See [accessing Namespaces](/cloud/namespaces#access-namespaces) for
  more information on endpoint options.

You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab.

For more information about managing and generating client certificates for Temporal Cloud, see
[How to manage certificates in Temporal Cloud](/cloud/certificates).

You can provide these connection options using environment variables, a configuration file, or directly in code.

**Configuration File**

You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you
configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when
creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to
[Environment Configuration](/references/client-environment-configuration).

You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path
to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the
default path `~/.config/temporalio/temporal.toml`.

> **ℹ️ Info:**
>
> The connection options set in configuration files have lower precedence than environment variables. This means that if
> you set the same option in both the configuration file and as an environment variable, the environment variable value
> overrides the option set in the configuration file.
>

For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to
connect to Temporal Cloud via an API key:

```toml
# Cloud profile for Temporal Cloud
[profile.staging]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```

If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and
private key:

```toml
# Cloud profile for Temporal Cloud
[profile.staging]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
tls_client_cert_data = "your-tls-client-cert-data"
tls_client_key_path = "your-tls-client-key-path"
```

With the connections options defined in the configuration file, use the `loadClientConnectConfig` helper from
`@temporalio/envconfig` to load the `staging` profile from the configuration file. You can then pass the resulting
configuration to the `Connection.connect` method. After that, you then pass the `connection` object and the Namespace to
the `Client` constructor to create a Temporal Client using the `staging` profile as follows. After loading the profile,
you can also programmatically override specific connection options before creating the client.

[env-config/src/load-profile.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-profile.ts)
```ts {15-18,30-31}
import { Connection, Client } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { resolve } from 'path';

async function main() {
  console.log("--- Loading 'staging' profile with programmatic overrides ---");

  const configFile = resolve(__dirname, '../config.toml');
  const profileName = 'staging';

  // The 'staging' profile in config.toml has an incorrect address (localhost:9999)
  // We'll programmatically override it to the correct address

  // Load the 'staging' profile.
  const config = loadClientConnectConfig({
    profile: profileName,
    configSource: { path: configFile },
  });

  // Override the target host to the correct address.
  // This is the recommended way to override configuration values.
  config.connectionOptions.address = 'localhost:7233';

  console.log(`\nLoaded '${profileName}' profile from ${configFile} with overrides.`);
  console.log(`  Address: ${config.connectionOptions.address} (overridden from localhost:9999)`);
  console.log(`  Namespace: ${config.namespace}`);

  console.log('\nAttempting to connect to client...');
  try {
    const connection = await Connection.connect(config.connectionOptions);
    const client = new Client({ connection, namespace: config.namespace });
    console.log('✅ Client connected successfully!');
    await connection.close();
  } catch (err) {
    console.log(`❌ Failed to connect: ${err}`);
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

**Environment Variables**

The following environment variables are required to connect to Temporal Cloud:

- `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `<namespace_id>.<account_id>`.
- `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace.
- `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication.
- `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path.
  Required if you are using mTLS authentication.
- `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path.
  Required if you are using mTLS authentication.

Ensure these environment variables exist in your environment before running your application.

Import the `@temporalio/envconfig` package to set connection options for the Temporal Client using environment
variables. The `loadClientConnectConfig` function will automatically load all environment variables. For a list of all
available environment variables and their default values, refer to
[Environment Configuration](/references/client-environment-configuration).

For example, the following code snippet loads all environment variables and creates a Temporal Client with the options
specified in those variables. If you have defined a configuration file at either the default location
(`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment
variable, this will also load the default profile in the configuration file. However, any options set via environment
variables will take precedence.

[env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts)
```ts {17-19,28-29}
import { Connection, Client } from '@temporalio/client';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { resolve } from 'path';

async function main() {
// ...
  const config = loadClientConnectConfig({
// ...
  });
// ...
  console.log(`  Address: ${config.connectionOptions.address}`);
  console.log(`  Namespace: ${config.namespace}`);
  console.log(`  gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`);

  console.log('\nAttempting to connect to client...');
  try {
    const connection = await Connection.connect(config.connectionOptions);
    const client = new Client({ connection, namespace: config.namespace });
    console.log('✅ Client connected successfully!');
    await connection.close();
  } catch (err) {
    console.log(`❌ Failed to connect: ${err}`);
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

**Code**

You can also provide connections options in your code directly. To create an initial connection, provide the Namespace
and API key values to the `Connection.connect` method.

```ts
const connection = await Connection.connect({
    address: <endpoint>,
    tls: true,
    apiKey: <APIKey>,
});
const client = new Client({
    connection,
    namespace: <namespace_id>.<account_id>,
});
```

To update an API key, use the `setApiKey` method on the Connection object:

```ts
connection.setApiKey(<APIKey>);
```

## Connect to Temporal Service from a Worker 

Connecting to Temporal Service from a Worker requires the same set of connections options as connecting from a Temporal
Application or from within an Activity, but the connection type is different. When connecting from a Worker, you create
a `NativeConnection` object instead of a `Connection` object. The
[`NativeConnection` class](https://typescript.temporal.io/api/classes/worker.NativeConnection) is imported from
`@temporalio/worker` instead of `@temporalio/client`. After you create the `NativeConnection` object, you pass it to
`Worker.create()` when creating the Worker.

To provide connection options to the `NativeConnection`, you can use environment variables, a configuration file, or
directly in code. The following code snippets show how to create a `NativeConnection` object using each method. Refer to
[Connect to a local development Temporal Service](#connect-to-development-service) and
[Connect to Temporal Cloud](#connect-to-temporal-cloud) for details on how to provide connection options using each
method.

**Configuration File**

Ensure you have a TOML configuration file with the necessary connection options defined. For example, the following TOML
configuration file defines a `staging` profile with the necessary connection options to connect to Temporal Cloud via an
API key:

```toml
# Cloud profile for Temporal Cloud
[profile.staging]
address = "your-namespace.a1b2c.tmprl.cloud:7233"
namespace = "your-namespace"
api_key = "your-api-key-here"
```

Use the `loadClientConnectConfig` helper from `@temporalio/envconfig` to load the `staging` profile from the
configuration file and create a `NativeConnection` object as follows:

```ts {1,15,17}
import { NativeConnection } from '@temporalio/worker';
import { loadClientConnectConfig } from '@temporalio/envconfig';
import { resolve } from 'path';

async function main() {
  const configFile = resolve(__dirname, '../config.toml');
  const profileName = 'staging'

  // Load the 'staging' profile.
  const config = loadClientConnectConfig({
    profile: profileName,
    configSource: { path: configFile },
  });

  const connection = await NativeConnection.connect(config.connectionOptions);

  const worker = await Worker.create({
    connection,
    namespace: <namespace_id>.<account_id>,
    // ...
});
}
```

**Environment Variables**

Ensure you have set the necessary environment variables to connect to Temporal Cloud. For example:

```bash
export TEMPORAL_NAMESPACE="your-namespace.your-account-id"
export TEMPORAL_ADDRESS="your-namespace.a1b2c.tmprl.cloud:7233"
export TEMPORAL_TLS_CLIENT_CERT_PATH="/path/to/your/client/cert.pem"
export TEMPORAL_TLS_CLIENT_KEY_PATH="/path/to/your/client/key.pem"
```

After setting the environment variables, use the following code to create a `NativeConnection` object using the
`loadClientConnectConfig` helper from `@temporalio/envconfig`:

```ts {1,5}
import { NativeConnection } from '@temporalio/worker';
import { loadClientConnectConfig } from '@temporalio/envconfig';

async function main() {
  const config = loadClientConnectConfig();

  const connection = await NativeConnection.connect(config.connectionOptions);

  const worker = await Worker.create({
    connection,
    namespace: process.env.TEMPORAL_NAMESPACE,
    // ...
  });
}
```

**Code**

You can also provide connections options in your TypeScript code directly. To create an initial connection, provide the
connections to the ` NativeConnection.connect` method, and then pass the resulting `NativeConnection` object to
`Worker.create()` when creating the Worker:

```ts {1,4,9}
import { NativeConnection } from '@temporalio/worker';
import { loadClientConnectConfig } from '@temporalio/envconfig';

const connection = await NativeConnection.connect({
    address: <endpoint>,
    tls: true,
    apiKey: <APIKey>,
});
const worker = await Worker.create({
    connection,
    namespace: <namespace_id>.<account_id>,
    // ...
});
```

`@temporalio/worker` v1.15.0 and later support replacing a running Worker's connection, which lets you rotate an mTLS
client certificate without restarting the Worker. gRPC's TLS credentials don't support dynamic certs, so instead of
updating the existing `NativeConnection` you create a new one with the new certificate and assign it to `worker.connection`:

```ts
import { readFileSync } from 'fs';

const newConnection = await NativeConnection.connect({
    address: <endpoint>,
    tls: {
        clientCertPair: {
            crt: readFileSync('client-cert-new.pem'),
            key: readFileSync('client-key-new.pem'),
        },
    },
});

worker.connection = newConnection;
```

The Worker starts using `newConnection` for subsequent calls to the Temporal Service; calls already in flight on the old
connection finish normally.

## `NativeConnection`, `Connection`, and `Client`

`NativeConnection`, `Connection`, and `Client` are all classes provided by the TypeScript SDK to facilitate
communication with the Temporal Service. This section explains the differences between these classes and their
respective use cases. For detailed information about each class, refer to the
[Temporal TypeScript API documentation](https://typescript.temporal.io/api/namespaces/client).

### `NativeConnection` versus `Connection` 

The TypeScript SDK provides two types of connection classes to connect to the Temporal Service: `NativeConnection` and
`Connection`. The `NativeConnection` class is used to connect from a Worker, while the `Connection` class is used to
connect from a Temporal Application or from within an Activity, typically through a `Client` object. Both connection
classes accept the same set of connection options.

### `Connection` versus `Client` 

A `Client` object is a high-level, lightweight abstraction that simplifies interaction with the Temporal Service. It
internally manages a `Connection` object to handle the low-level communication details. The `Client` class provides
convenient methods for common operations such as starting Workflow Executions, sending Signals and Queries, and
retrieving Workflow results.

A `Connection` object is a lower-level and expensive object that represents a direct connection to the Temporal Service.
You pass in a `Connection` object to the `Client` constructor to create a `Client` instance. Since a `Connection` is
expensive to create, create a single `Connection` object and reuse it across your application whenever possible.

When instantiating a `Connection`, you specify most connection options except for the Namespace, such as the Temporal
Service endpoint, TLS settings, and authentication credentials. When instantiating a `Client`, you provide the
`Connection` object and the Namespace you want to connect to, along with other client options.

## Start a Workflow Execution 

[Workflow Execution](/workflow-execution) semantics rely on several parameters. To start a Workflow Execution, you must
supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific
contextual data, and Workflow Function parameters.

In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from
within another Workflow Execution, use either the Child Workflow or External Workflow APIs.

See the [Customize Workflow Type](/develop/typescript/workflows/basics#workflow-type) section to see how to customize
the name of the Workflow Type.

A request to spawn a Workflow Execution causes the Temporal Service to create the first Event
([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The
Temporal Service then creates the first Workflow Task, resulting in the first
[WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event.

When you have a Client, you can schedule the start of a Workflow with `client.workflow.start()`, specifying
`workflowId`, `taskQueue`, and `args` and returning a Workflow handle immediately after the Server acknowledges the
receipt.

```typescript
const handle = await client.workflow.start(example, {
  workflowId: 'your-workflow-id',
  taskQueue: 'your-task-queue',
  args: ['argument01', 'argument02', 'argument03'], // this is typechecked against workflowFn's args
});
const result = await handle.result();
```

Calling `client.workflow.start()` and `client.workflow.execute()` send a command to Temporal Server to schedule a new
Workflow Execution on the specified Task Queue. It does not actually start until a Worker that has a matching Workflow
Type, polling that Task Queue, picks it up.

You can test this by executing a Client command without a matching Worker. Temporal Server records the command in Event
History, but does not make progress with the Workflow Execution until a Worker starts polling with a matching Task Queue
and Workflow Definition.

Workflow Executions run in a separate V8 isolate context to provide a
[deterministic runtime](/workflow-definition#deterministic-constraints).

### Set a Workflow's Task Queue 

The only Workflow Option that you must set is the name of the [Task Queue](/task-queue).

For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task
Queue name.

A Task Queue is a dynamic queue in Temporal polled by one or more Workers.

Workers bundle Workflow code and node modules using Webpack v5 and execute them inside V8 isolates. Activities are
directly required and run by Workers in the Node.js environment.

Workers are flexible. You can host any or all of your Workflows and Activities on a Worker, and you can host multiple
Workers on a single machine.

The Worker needs three main things:

- `taskQueue`: The Task Queue to poll. This is the only required argument.
- `activities`: Optional. Imported and supplied directly to the Worker.
- Workflow bundle. Choose one of the following options:
  - Specify `workflowsPath` pointing to your `workflows.ts` file to pass to Webpack; for example,
    `require.resolve('./workflows')`. Workflows are bundled with their dependencies.
  - If you prefer to handle the bundling yourself, pass a prebuilt bundle to `workflowBundle`.

```ts
import { Worker } from '@temporalio/worker';
import * as activities from './activities';

async function run() {
  // Step 1: Register Workflows and Activities with the Worker and connect to
  // the Temporal server.
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows'),
    activities,
    taskQueue: 'hello-world',
  });
  // Worker connects to localhost by default and uses console.error for logging.
  // Customize the Worker by passing more options to create():
  // https://typescript.temporal.io/api/classes/worker.Worker
  // If you need to configure server connection parameters, see docs:
  // /develop/typescript/client/temporal-client#connect-to-temporal-service-from-a-worker

  // Step 2: Start accepting tasks on the `hello-world` queue
  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

`taskQueue` is the only required option; however, use `workflowsPath` and `activities` to register Workflows and
Activities with the Worker.

When scheduling a Workflow, you must specify `taskQueue`.

```ts
import { Client, Connection } from '@temporalio/client';
// This is the code that is used to start a Workflow.
const connection = await Connection.connect();
const client = new Client({ connection });
const result = await client.workflow.execute(yourWorkflow, {
  // required
  taskQueue: 'your-task-queue',
  // required
  workflowId: 'your-workflow-id',
});
```

When creating a Worker, you must pass the `taskQueue` option to the `Worker.create()` function.

```ts
const worker = await Worker.create({
  // imported elsewhere
  activities,
  taskQueue: 'your-task-queue',
});
```

Optionally, in Workflow code, when calling an Activity, you can specify the Task Queue by passing the `taskQueue` option
to `proxyActivities()`, `startChild()`, or `executeChild()`. If you do not specify `taskQueue`, the TypeScript SDK
places Activity and Child Workflow Tasks in the same Task Queue as the Workflow Task Queue.

### Set a Workflow Id 

Although it is not required, we recommend providing your own
[Workflow Id](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity
identifier, such as an order identifier or customer identifier.

Call `client.workflow.start()` with your Workflow and any arguments. Then specify your `taskQueue` and set your
`workflowId` to a meaningful business identifier.

```typescript
const handle = await client.workflow.start(example, {
  workflowId: 'yourWorkflowId',
  taskQueue: 'yourTaskQueue',
  args: ['your', 'arg', 'uments'],
});
```

This starts a Workflow Execution with the given Workflow ID, Task Queue name, and arguments.

## Get Workflow results 

If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id.

The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its
result.

It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time
(asynchronous execution).

In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and
results of Workflow Executions.

To return the results of a Workflow Execution:

```typescript
return 'Completed ' + wf.workflowInfo().workflowId + ', Total Charged: ' + totalCharged;
```

`totalCharged` is just a function declared in your code. For a full example, see
[subscription-workflow-project-template-typescript/src/workflows.ts](https://github.com/temporalio/subscription-workflow-project-template-typescript/blob/main/src/workflows.ts).

A Workflow function may return a result. If it doesn't (in which case the return type is `Promise<void>`), the result
will be `undefined`.

If you started a Workflow with `client.workflow.start()`, you can choose to wait for the result anytime with
`handle.result()`.

```typescript
const handle = client.workflow.getHandle(workflowId);
const result = await handle.result();
```

Using a Workflow Handle isn't necessary with `client.workflow.execute()`.

If you call `result()` on a Workflow that prematurely ended for some reason, it throws a
[`WorkflowFailedError` error](https://typescript.temporal.io/api/classes/client.WorkflowFailedError/) that reflects the
reason. For that reason, it is recommended to catch that error.

```typescript
const handle = client.workflow.getHandle(workflowId);
try {
  const result = await handle.result();
} catch (err) {
  if (err instanceof WorkflowFailedError) {
    throw new Error('Temporal workflow failed: ' + workflowId, {
      cause: err,
    });
  } else {
    throw new Error('error from Temporal workflow ' + workflowId, {
      cause: err,
    });
  }
}
```

---

# Install the TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/install-typescript-sdk

> Scaffold a new TypeScript project or add Temporal packages to an existing one with npm, plus where to find the API reference.

## How to install a Temporal SDK 

A [Temporal SDK](/encyclopedia/architecture/temporal-sdks) provides a framework for
[Temporal Application](/temporal#temporal-application) development.

An SDK provides you with the following:

- A [Temporal Client](/encyclopedia/temporal-client) to communicate with a
  [Temporal Service](/temporal-service).
- APIs to develop [Workflows](/workflows).
- APIs to create and manage [Worker Processes](/workers#worker).
- APIs to author [Activities](/activity-definition).

[![NPM](https://img.shields.io/npm/v/temporalio.svg?style=for-the-badge)](https://www.npmjs.com/search?q=author%3Atemporal-sdk-team)

This project requires Node.js 18 or later.

**Create a project**

```bash
npx @temporalio/create@latest ./your-app
```

**Add to an existing project**

```bash
npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity @temporalio/common
```

> **📝 Note:**
>
> The TypeScript SDK is designed with TypeScript-first developer experience in mind, but it works equally well with
> JavaScript.
>

### How to find the TypeScript SDK API reference 

The Temporal TypeScript SDK API reference is published to [typescript.temporal.io](https://typescript.temporal.io).

### Where are SDK-specific code examples? 

You can find a complete list of executable code samples in
[Temporal's GitHub repository](https://github.com/temporalio?q=samples-&type=all&language=&sort=).

Additionally, several of the [Tutorials](https://learn.temporal.io) are backed by a fully executable template
application.

Use the [TypeScript samples library](https://github.com/temporalio/samples-typescript) stored on GitHub to demonstrate
various capabilities of Temporal.

**Where can I find video demos?**

[Temporal TypeScript YouTube playlist](https://www.youtube.com/playlist?list=PLl9kRkvFJrlTavecydpk9r6cF7qBmQJvb).

### How to import an ECMAScript module 

The JavaScript ecosystem is quickly moving toward publishing ECMAScript modules (ESM) instead of CommonJS modules. For
example, `node-fetch@3` is ESM, but `node-fetch@2` is CommonJS.

For more information about importing a pure ESM dependency, see our
[Fetch ESM](https://github.com/temporalio/samples-typescript/tree/main/fetch-esm) sample for the necessary configuration
changes:

- `package.json` must have include the `"type": "module"` attribute.
- `tsconfig.json` should output in `esnext` format.
- Imports must include the `.js` file extension.

## Linting and types in TypeScript 

If you started your project with `@temporalio/create`, you already have our recommended TypeScript and ESLint
configurations.

If you incrementally added Temporal to an existing app, we do recommend setting up linting and types because they help
catch bugs well before you ship them to production, and they improve your development feedback loop. Take a look at our
recommended [.eslintrc](https://github.com/temporalio/samples-typescript/blob/main/.shared/.eslintrc.js) file and tweak
to suit your needs.

---

# Integrations - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/integrations

> Integrations with other tools and services.

The following integrations are available for the Temporal TypeScript SDK.

- [AI SDK by Vercel](/develop/typescript/integrations/ai-sdk) — Build AI-powered applications with Durable Execution using the Vercel AI SDK. _(TypeScript · Agent framework)_
- [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#typescript) — Monitor and evaluate AI application performance with Braintrust observability. _(TypeScript · Agent observability)_
- [Google ADK](/develop/typescript/integrations/google-adk-agents) — Run Google ADK agents as durable Temporal Workflows. _(TypeScript · Agent framework)_
- [Laminar](https://laminar.sh/docs/integrations/temporal) — Trace Temporal Workflows, Activities, and the LLM calls inside them with Laminar. _(TypeScript · Agent observability)_
- [LangSmith](/develop/typescript/integrations/langsmith) — Trace and debug LLM calls in Temporal Workflows with LangSmith. _(TypeScript · Agent observability)_
- [Mastra](https://mastra.ai/integrations/deploy/temporal) — Build durable AI agents and workflows with the Mastra TypeScript framework. _(TypeScript · Agent framework)_
- [OpenAI Agents SDK](/develop/typescript/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(TypeScript · Agent framework)_
- [OpenRouter](https://github.com/temporalio/samples-typescript/tree/main/openrouter) — Call hundreds of models through OpenRouter from Temporal Activities, with durable retries and cost tracking. _(TypeScript · Model gateway)_
- [Parseable](https://github.com/parseablehq/temporal-plugin/blob/main/INTEGRATION.md) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(TypeScript · Agent observability)_
- [Strands Agents](/develop/typescript/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(TypeScript · Agent framework)_
- [xMemory](https://xmemory.ai/temporal/#quickstart--typescript) — Store and recall agent memory across Workflow Executions with xMemory. _(TypeScript · Agent memory)_

---

# AI SDK by Vercel integration - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/integrations/ai-sdk

> Run Vercel AI SDK agents in Temporal Workflows with durable model calls, tools, MCP requests, and streamed responses in TypeScript.

Temporal's integration with [Vercel's AI SDK](https://ai-sdk.dev/) lets you use the AI SDK's API directly in Workflow
code while Temporal handles Durable Execution.

Like all API calls, LLM API calls are non-deterministic. In a [Temporal Application](/glossary#temporal-application),
that means you cannot make LLM calls directly from a [Workflow](/glossary#workflow); they must run as
[Activities](/glossary#activity). The AI SDK plugin handles this automatically: when you call methods in the AI SDK such
as `generateText()` or `streamText()`, the plugin wraps those calls in Activities behind the scenes.
This preserves the Vercel AI SDK's developer experience that you are already familiar with while Temporal handles
Durable Execution for you.

> **📝 Note:**
>
> Import the plugin (`AiSdkPlugin`) from `@temporalio/vercel-ai-sdk` in your Worker setup. Import Workflow-side helpers
> such as `temporalProvider` and `TemporalMCPClient` from `@temporalio/vercel-ai-sdk/workflow`. The `/workflow` subpath is
> safe to use in the deterministic Workflow sandbox.
>

All code snippets in this guide are taken from the TypeScript SDK
[Vercel AI SDK samples](https://github.com/temporalio/samples-typescript/tree/main/vercel-ai-sdk). Refer to the samples for the
complete code and run them locally.

The standalone integration is maintained in
[`temporalio/ai-integrations`](https://github.com/temporalio/ai-integrations/tree/main/typescript/vercel-ai-sdk).

## Prerequisites

- This guide assumes you are already familiar with the Vercel AI SDK. If you aren't, refer to the
  [Vercel AI SDK documentation](https://ai-sdk.dev/) for more details.
- If you are new to Temporal, we also recommend you read the [Understanding Temporal](/evaluate/understanding-temporal)
  document or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course to understand the basics
  of Temporal.
- Ensure you have set up your local development environment by following the
  [Set up a local development environment with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript) guide. When you are
  done, leave the Temporal Development Server running if you want to test your code locally.
- Use Node.js, Temporal SDK, and Vercel AI SDK versions that satisfy the integration's
  [package requirements](https://github.com/temporalio/ai-integrations/blob/main/typescript/vercel-ai-sdk/package.json).

## Configure Workers to use the AI SDK

Workers are the compute layer of a Temporal Application. They are responsible for executing the code that defines your
[Workflows](/glossary#workflow) and [Activities](/glossary#activity). Before you can execute a Workflow or Activity with
the Vercel AI SDK, you need to create a Worker and configure it to use the AI SDK plugin.

Follow the steps below to configure your Worker.

1. Install the standalone [`@temporalio/vercel-ai-sdk`](https://www.npmjs.com/package/@temporalio/vercel-ai-sdk) package.
   The integration is versioned separately from the Temporal TypeScript SDK.

   ```bash
   npm install @temporalio/vercel-ai-sdk
   ```

2. Create a `worker.ts` file and configure the Worker to use the AI SDK plugin.

   ```ts {9-11}
   import { openai } from '@ai-sdk/openai';
   import { AiSdkPlugin } from '@temporalio/vercel-ai-sdk';

   //... other import statements, initializing a connection
   //  to the Temporal Service to be used by the Worker

   const worker = await Worker.create({
     plugins: [
       new AiSdkPlugin({
         modelProvider: openai,
       }),
     ],
     connection,
     namespace: 'default',
     taskQueue: 'vercel-ai-sdk',
     workflowsPath: require.resolve('./workflows'),
     activities,
   });

   // ... code that runs the worker
   ```

   The `modelProvider` specifies which AI provider to use when creating models. Choose the provider that best suits your
   needs. In the Worker options, you are also specifying that the Worker polls the `vercel-ai-sdk` Task Queue for work in the
   `default` Namespace. Make sure that you configure your Client application to use the same Task Queue and Namespace.

3. Run the Worker. This Worker will now poll the Temporal Service for work on the `vercel-ai-sdk` Task Queue in the `default`
   Namespace until you stop it.

   ```bash
   nodemon worker.ts
   ```

   You must ensure the Worker process has access to your API credentials. Most provider SDKs read credentials from
   environment variables. Refer to the [Vercel AI SDK documentation](https://ai-sdk.dev/providers/ai-sdk-providers) for
   instructions on how to set up your environment variables for the provider you chose.

> **💡 Tip:**
>
>    You only need to give provider credentials to the Worker process. The client application, meaning the application
>    that sends requests to the Temporal Service to start Workflow Executions, doesn't need to know about the credentials.
>

See the full example at [Vercel AI SDK samples](https://github.com/temporalio/samples-typescript/tree/main/vercel-ai-sdk).

## Develop a simple haiku agent

To help you get started, you can develop a simple Haiku Agent that generates haikus based on a prompt.

If you weren't using Temporal, you would write code like this to generate a haiku:

```ts
import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';

async function haikuAgent(prompt: string): Promise<string> {
  const result = await generateText({
    model: openai('gpt-4o-mini'),
    prompt,
    system: 'You only respond in haikus.',
  });
  return result.text;
}
```

To add Durable Execution to your agent, implement the agent as a Temporal Workflow. Use the AI SDK as you normally
would, but pass `temporalProvider.languageModel()` as the model. The string you provide (like `'gpt-4o-mini'`) is passed
to your configured `modelProvider` to create the model.

```ts {2,6}
import { generateText } from 'ai';
import { temporalProvider } from '@temporalio/vercel-ai-sdk/workflow';

export async function haikuAgent(prompt: string): Promise<string> {
  const result = await generateText({
    model: temporalProvider.languageModel('gpt-4o-mini'),
    prompt,
    system: 'You only respond in haikus.',
  });
  return result.text;
}
```

With only two line changes, you have added Durable Execution to your agent. Your agent now gets automatic retries,
timeouts, and the ability to run for extended periods without losing state if the process crashes.

## Provide your durable agent with tools

The Vercel AI SDK lets you provide tools to your agents, and when the model calls them, they execute in the Workflow.
Since tool functions run in Workflow context, they must follow Workflow rules. That means they must call Activities or
Child Workflows to perform non-deterministic operations like API calls.

For example, if you want to call an external API to get the weather, you would implement it as an Activity and call it
from the tool function. The following is an example of an Activity that gets the weather for a given location:

<!--SNIPSTART typescript-vercel-ai-sdk-weather-activity -->
[vercel-ai-sdk/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/vercel-ai-sdk/src/activities.ts)
```ts
export async function getWeather(input: {
  location: string;
}): Promise<{ city: string; temperatureRange: string; conditions: string }> {
  return {
    city: input.location,
    temperatureRange: '14-20C',
    conditions: 'Sunny with wind.',
  };
}
```
<!--SNIPEND-->

Then in your agent implementation, provide the tool to the model using the `tools` option and instruct the model to use
the tool when needed.

```ts {15-23}
import { proxyActivities } from '@temporalio/workflow';
import { generateText, tool } from 'ai';
import { temporalProvider } from '@temporalio/vercel-ai-sdk/workflow';
import { z } from 'zod';

const { getWeather } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

export async function toolsAgent(question: string): Promise<string> {
  const result = await generateText({
    model: temporalProvider.languageModel('gpt-4o-mini'),
    prompt: question,
    system: 'You are a helpful agent.',
    tools: {
      getWeather: tool({
        description: 'Get the weather for a given city',
        inputSchema: z.object({
          location: z.string().describe('The location to get the weather for'),
        }),
        execute: getWeather,
      }),
    },
    stopWhen: stepCountIs(5),
  });
  return result.text;
}
```

## Integrate with Model Context Protocol (MCP) servers

[Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is an open standard that lets AI applications connect
to external tools and data sources. Calls to MCP servers, being calls to external APIs, are non-deterministic and would
usually need to be implemented as Activities. The Temporal AI SDK integration handles this for you and provides a
built-in implementation of a stateless MCP client that you can use inside Workflows.

Follow the steps below to integrate your agent with an MCP server.

1. Create a connection to the MCP servers using the `createMCPClient` function from the `@ai-sdk/mcp`
   package. You can register multiple MCP servers by providing multiple factory functions in `mcpClientFactories`.

   ```ts
   import { createMCPClient } from '@ai-sdk/mcp';
   import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

   const mcpClientFactories = {
     testServer: () =>
       createMCPClient({
         transport: new StdioClientTransport({
           command: 'node',
           args: ['lib/mcp-server.js'],
         }),
       }),
   };
   ```

   The example uses `StdioClientTransport` as the transport mechanisms for client-server communication. Each time the
   Worker processes a Task that requires communication with the MCP server, it will start the server process and connect
   to it as required by the Task.

2. Configure the Worker to use the MCP client factories.

   ```ts {6}
   const worker = await Worker.create({
     // ... other Worker options
     plugins: [
       new AiSdkPlugin({
         modelProvider: openai,
         mcpClientFactories,
       }),
     ],
   });
   ```

3. In your agent Workflow, use `TemporalMCPClient` to get tools from the MCP server by referencing it by name:

   ```ts {4-5,9}
   import { TemporalMCPClient, temporalProvider } from '@temporalio/vercel-ai-sdk/workflow';

   export async function mcpAgent(prompt: string): Promise<string> {
     const mcpClient = new TemporalMCPClient({ name: 'testServer' });
     const tools = await mcpClient.tools();
     const result = await generateText({
       model: temporalProvider.languageModel('gpt-4o-mini'),
       prompt,
       tools,
       system: 'You are a helpful agent, You always use your tools when needed.',
       stopWhen: stepCountIs(5),
     });
     return result.text;
   }
   ```

   Both listing tools and calling them run as Activities behind the scenes, giving you automatic retries, timeouts, and
   full observability.

## Stream model output from a Workflow

The AI SDK's `streamText()` function streams a response incrementally instead of returning it all at once.
The plugin runs the model call in an Activity and publishes each delta onto a
[Workflow Stream](/develop/typescript/workflows/workflow-streams) topic. External consumers can subscribe to that topic
by Workflow ID to render tokens live as they arrive, while the Workflow durably reassembles the final result.

> **📝 Note:**
>
> Inside the Workflow, the streamed result is reassembled after the Activity completes, so the Workflow's own `textStream`
> / `partialOutputStream` is not incremental. Live, token-by-token deltas are delivered to **external** consumers through
> a `WorkflowStreamClient`. This is inherent to Durable Execution: the Activity must run to completion before the Workflow
> can observe a deterministic, replayable result.
>

Streaming uses the [`@temporalio/workflow-streams`](/develop/typescript/workflows/workflow-streams) package. Install it
alongside the AI SDK plugin:

```bash
npm install @temporalio/workflow-streams
```

To stream text, configure a provider with a `streamingTopic` (this enables streaming and names the topic that deltas are
published to), host a `WorkflowStream` in your Workflow, and consume the AI SDK stream as usual:

```ts
import { streamText } from 'ai';
import { TemporalProvider } from '@temporalio/vercel-ai-sdk/workflow';
import { WorkflowStream } from '@temporalio/workflow-streams/workflow';

export const STREAM_TOPIC = 'text-stream';

const streamingProvider = new TemporalProvider({
  languageModel: { streamingTopic: STREAM_TOPIC },
});
```

<!--SNIPSTART typescript-vercel-ai-sdk-streaming-agent -->
[vercel-ai-sdk/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/vercel-ai-sdk/src/workflows.ts)
```ts
export async function streamingAgent(prompt: string): Promise<string> {
  // Host the WorkflowStream as the first statement of the workflow so its
  // publish-signal handler is registered before the streaming activity starts
  // publishing deltas to it.
  new WorkflowStream();

  // A subscriber flips this once it has consumed the final delta (see below).
  let consumerDone = false;
  setHandler(consumerDoneSignal, () => {
    consumerDone = true;
  });

  const result = streamText({
    model: streamingProvider.languageModel('gpt-4o-mini'),
    prompt,
    system: 'You only respond in haikus.',
  });

  // The model call runs in an activity that publishes each delta to
  // STREAM_TOPIC for external subscribers. Inside the workflow the deltas are
  // replayed after the activity completes, so this loop durably reassembles
  // the full text.
  let text = '';
  for await (const delta of result.textStream) {
    text += delta;
  }

  // The workflow's stream log lives in memory and is discarded once the run
  // completes, which can race a subscriber's final poll. Rather than guess at a
  // fixed delay, wait for the subscriber to signal that it received the last
  // delta. The timeout is a fallback for when nothing is subscribed, so the run
  // can't hang forever.
  await condition(() => consumerDone, '10 seconds');
  return text;
}
```
<!--SNIPEND-->

The consumer subscribes to the topic by Workflow ID and renders each delta as it arrives. Each item's payload is the
JSON-encoded AI SDK stream part:

<!--SNIPSTART typescript-vercel-ai-sdk-streaming-consumer -->
[vercel-ai-sdk/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/vercel-ai-sdk/src/client.ts)
```ts
// Subscribe to a Workflow's stream topic and render each text delta live as it
// is published by the streaming activity. Each item's payload is the
// JSON-encoded AI SDK stream part; `resultType: true` decodes it to raw bytes.
async function renderStream(client: Client, workflowId: string, topic: string): Promise<void> {
  const streamClient = WorkflowStreamClient.create(client, workflowId);
  for await (const item of streamClient.subscribe<Uint8Array>(topic, 0, { resultType: true })) {
    const part = JSON.parse(new TextDecoder().decode(item.data));
    if (part.type === 'text-delta') process.stdout.write(part.delta);
    if (part.type === 'finish') break;
  }
  process.stdout.write('\n');
  // Acknowledge receipt so the Workflow can complete without racing this final
  // poll against its in-memory stream log being discarded.
  await client.workflow.getHandle(workflowId).signal(consumerDoneSignal);
}
```
<!--SNIPEND-->

### Stream structured output

Use `streamText()` with `output: Output.object({ schema })` to stream structured output. Choose a distinct
`streamingTopic` for each concurrent stream. Inside the Workflow, consume `partialOutputStream` and await `result.output`
for the validated result. External subscribers receive the model deltas through Workflow Streams.

See the [Vercel AI SDK structured output guide](https://ai-sdk.dev/docs/ai-sdk-core/generating-structured-data#stream-structured-outputs)
for schemas and output options.

See the full example at [Vercel AI SDK samples](https://github.com/temporalio/samples-typescript/tree/main/vercel-ai-sdk).

---

# Google ADK integration - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/integrations/google-adk-agents

> Run Google ADK agents as durable Temporal Workflows in TypeScript, with model and MCP calls running as retryable Activities.

The Temporal [Google Agent Development Kit (ADK)](https://google.github.io/adk-docs/) integration runs an ADK agent
graph as a durable Temporal Workflow. The agent loop, tool selection, and state run in the Workflow, while model
inference and Model Context Protocol (MCP) operations run as Activities. Completed calls are recorded in Event History
and aren't repeated during replay.

`GoogleAdkPlugin` configures the Worker and Workflow bundle. In Workflow code, `TemporalModel` replaces a standard ADK
model and routes each model call to an Activity. The integration also provides Workflow-safe APIs for Activity-backed
tools, MCP servers, and streamed model responses.

Code snippets on this page come from the
[Google ADK samples directory](https://github.com/temporalio/samples-typescript/tree/main/google-adk).

> **Public Preview**

## Prerequisites

- This guide assumes you are familiar with Google ADK. If you aren't, refer to the
  [Google ADK documentation](https://google.github.io/adk-docs/) for an introduction to agents, runners, and tools.
- If you are new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the
  [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following
  [Set up a local development environment with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript). Leave the Temporal
  development server running to run the samples locally.
- Use Node.js, Temporal SDK, and Google ADK versions that satisfy the integration's
  [package requirements](https://github.com/temporalio/ai-integrations/blob/main/typescript/google-adk/package.json).

## Install the Google ADK integration

Install the standalone [`@temporalio/google-adk`](https://www.npmjs.com/package/@temporalio/google-adk) package. It is
versioned separately from the Temporal TypeScript SDK.

```bash
npm install @temporalio/google-adk
```

Keep `@google/adk` within the package's supported peer dependency range. The Workflow bundle uses compatibility shims
for that range. Install `@modelcontextprotocol/sdk` if you use `TemporalMCPToolset`.

When migrating from `@temporalio/google-adk-agents`, follow the
[upgrade guidance](https://github.com/temporalio/ai-integrations/tree/main/typescript/google-adk#upgrading-from-adk-15).
Complete existing Workflow Executions before switching, or keep them on separate Task Queues. The standalone integration
cannot replay Workflow histories created with `@temporalio/google-adk-agents`.

The Worker reads Gemini credentials from `GOOGLE_GENAI_API_KEY` or `GEMINI_API_KEY`. Credentials remain in the Worker
process. Model requests and responses are Activity inputs and results, so they are stored in Event History. Use a
[Payload Codec to encrypt sensitive data](/develop/typescript/best-practices/data-handling/data-encryption), and account
for the [programming model limits](/evaluate/cloud/limits#programming-model-level), including the 2 MB limit on a single payload.

## Run an ADK agent in a Workflow

The agent chat sample contains a complete Workflow, Worker, and Client. It uses the standard ADK `LlmAgent` and
`InMemoryRunner` APIs in the Workflow, with `TemporalModel` as the agent's model. The Client sends messages through
Workflow Updates and can query the conversation history.

Create the agent and runner in Workflow code:

<!--SNIPSTART typescript-google-adk-agent-chat-workflow -->
[google-adk/src/agent-chat/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/google-adk/src/agent-chat/workflows.ts)
```ts
const agent = new LlmAgent({
  name: 'assistant',
  model: new TemporalModel('gemini-2.5-flash'),
  instruction: 'Continue the conversation using its prior context. Respond in one sentence.',
});
const runner = new InMemoryRunner({ agent, appName: 'agent-chat' });
```
<!--SNIPEND-->

Register `GoogleAdkPlugin` on the Worker to install the model Activities and Workflow bundler configuration:

<!--SNIPSTART typescript-google-adk-agent-chat-worker -->
[google-adk/src/agent-chat/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/google-adk/src/agent-chat/worker.ts)
```ts
const worker = await Worker.create({
  connection,
  taskQueue: 'google-adk-agent-chat',
  workflowsPath: require.resolve('./workflows'),
  plugins: [
    new GoogleAdkPlugin(process.env.MODEL_PROVIDER === 'fake' ? { modelProvider: offlineModelProvider() } : {}),
  ],
});
await worker.run();
```
<!--SNIPEND-->

The Client starts the Workflow normally and doesn't need the plugin.

Use each API from its package entry point:

| Entry point | APIs |
| --- | --- |
| `@temporalio/google-adk` | `GoogleAdkPlugin` for Worker code |
| `@temporalio/google-adk/workflow` | `TemporalModel`, `TemporalMCPToolset`, and `activityAsTool` for Workflow code |
| `@temporalio/google-adk/testing` | `fakeModelProvider` and `mockMCPToolset` for tests |

The [agent chat sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/agent-chat)
shows this Workflow, Worker, and Client together, including its Update, Query, and Continue-As-New behavior.

An agent graph can delegate work among multiple `LlmAgent` instances. The agents and their transfers remain in the
Workflow, while each `TemporalModel` call runs as an Activity. The
[multi-agent sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/multi-agent)
shows a coordinator transferring a request to a researcher and then a writer.

## Configure model calls

Pass Activity options to `TemporalModel` to set timeouts, retries, a Task Queue, or an Activity summary. Set these under
the model's `activity` option. The default `startToCloseTimeout` is one minute.

The plugin disables retries in the underlying model SDK so that the Activity retry policy controls retries and backoff.
Set `heartbeatTimeout` to detect a dead Worker and deliver cancellation to a long model call. The Activity heartbeats on
a timer, so a Heartbeat Timeout doesn't detect a stalled call; `startToCloseTimeout` bounds a stalled call.

Use ADK's `outputSchema` to constrain a model response, then validate the result in the Workflow before returning it.
The [structured-output sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/structured-output)
shows this pattern for an incident summary.

## Add tools and MCP servers

Google ADK function tools run as part of the agent graph inside the Workflow. Use them for deterministic operations,
such as transforming values or updating agent state. A tool that reads a file, calls an API, queries a database, or
performs other I/O must run outside the Workflow.

Use `activityAsTool` to expose an existing Activity to an agent. Its `name` must match an Activity registered in the
Worker's `activities` option. The model's arguments object becomes the Activity's single argument, and the `activity`
option controls its timeouts and retries. The [tools sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/tools)
shows the Workflow and Worker configuration together.

An ADK `LongRunningFunctionTool` can wait in Workflow code for external input. A Signal can release the tool
asynchronously, while an Update can release it and return an accepted result to the caller. The
[human-in-the-loop sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/human-in-the-loop)
shows both forms of approval.

For MCP, register a named factory with `mcpToolsets: { <name>: factory }` in the Worker plugin, then create a
`TemporalMCPToolset` with the same name in Workflow code. Listing tools and calling them execute as Activities. Each
operation opens a new MCP session, so session state doesn't carry between operations unless the factory returns a
long-lived toolset. MCP failures often have no status and are retryable, so set `activity.retry.maximumAttempts` on the
`TemporalMCPToolset` when retries must be bounded. See the complete Worker and Workflow pair in the
[MCP sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/mcp).

## Stream model responses

Install [`@temporalio/workflow-streams`](https://www.npmjs.com/package/@temporalio/workflow-streams), host a
`WorkflowStream` at the top of the Workflow, and set `streamingTopic` in the `TemporalModel` options. The model call must
also request streaming, either through ADK runner configuration with `StreamingMode.SSE` or by calling
`generateContentAsync(request, true)`. A streaming call without a topic fails.

The model Activity publishes chunks to the topic and returns the complete response to the Workflow. The Activity result
is the deterministic value used during replay. Stream delivery is at-least-once, and a retried Activity publishes its
chunks again from the beginning.

The [streaming sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/streaming)
shows how a Workflow publishes chunks and waits for a stream consumer to finish.

## Test and observe your agents

The testing entry point provides `fakeModelProvider` and `mockMCPToolset`. Pass them to `GoogleAdkPlugin` to test without
model credentials or a live MCP server while exercising the Worker plugin, Workflow bundle, and Activities.

Use [replay testing](/develop/typescript/best-practices/testing-suite#replay) for Workflow changes. Pass
`plugins: [new GoogleAdkPlugin()]` to `Worker.runReplayHistory` because the plugin's bundler configuration is required
to load Google ADK in the replay sandbox.

Compose `GoogleAdkPlugin` after `OpenTelemetryPlugin` from `@temporalio/interceptors-opentelemetry` to export ADK's
agent, model, and tool spans from the Workflow sandbox.

The OpenTelemetry plugin's Worker sink suppresses span export during replay. Export is at-least-once because a failed or
timed-out Workflow Task can execute again without being a replay.

ADK span attributes can contain prompts and model responses. Send them only to an approved destination or remove
sensitive attributes in the span processor. `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS` doesn't control this behavior in a
Workflow because the Workflow sandbox doesn't expose Worker environment variables.

The [observability sample](https://github.com/temporalio/samples-typescript/tree/main/google-adk/src/observability)
shows the plugin order and an OpenTelemetry span processor that records model usage.

## Reference and troubleshooting

### Feature support

| Feature | Support | Notes |
| --- | --- | --- |
| ADK agent graphs, including sequential and delegated multi-agent patterns | Supported | Compose agents normally; delegated agents remain part of the durable Workflow graph. |
| Function tools and Activities as tools | Supported | Regular function tools run in the Workflow and must be deterministic; `activityAsTool` moves I/O to Activities. |
| MCP toolsets | Supported | `toolFilter` accepts only string arrays, not ADK `ToolPredicate`, and filters advertised names after prefixing. Connection-parameter factories open one session per Activity; factory-returned `BaseToolset`s may remain long-lived. |
| Human approval through Signals or Updates | Supported | Use Signals for asynchronous input or Updates when the caller needs an accepted result. |
| Structured model output | Supported | ADK output schemas work with durable agent execution. |
| SSE response streaming | Supported | Requires `streamingTopic`; delivery is at-least-once, and retries may republish from the beginning. |
| OpenTelemetry tracing | Supported | Register `OpenTelemetryPlugin` before `GoogleAdkPlugin`. |
| Live bidirectional streaming through `BaseLlm.connect` | Not supported in Workflows | Use SSE response streaming for supported streaming behavior. |

### Composing with other plugins

Register observability and governance plugins before `GoogleAdkPlugin`. In particular, place `OpenTelemetryPlugin`
first so that ADK's Workflow-side spans bind to its tracer provider. Register `GoogleAdkPlugin` only on the Worker; a
Client plugin isn't required.

Custom payload and failure converter modules load before the plugin's polyfills. If either module imports `@google/adk`
or `@google/genai`, import `@temporalio/google-adk/workflow` first in that module.

### Replay safety

The ADK runner, agent graph, regular function tools, and callbacks execute inside the Workflow and must remain
deterministic. `TemporalModel`, `TemporalMCPToolset`, and `activityAsTool` move model calls, MCP operations, and other I/O
into Activities. Completed Activity results are read from Event History during replay.

The full model request and response cross the Activity boundary and are recorded in Event History. For long-running
conversations, use [Continue-As-New](/develop/typescript/workflows/continue-as-new) before Event History approaches its
limits.

### Configuration

Options under `activity` accept the standard TypeScript SDK `ActivityOptions` fields.

| API | Option | Default | Behavior |
| --- | --- | --- | --- |
| `GoogleAdkPlugin` | `modelProvider` | ADK `LLMRegistry` | Resolves model names in model Activities. Use it for another provider, proxy, or test double. |
| `GoogleAdkPlugin` | `mcpToolsets` | `{}` | Maps names to MCP factories and registers `<name>-listTools` and `<name>-callTool` Activities. |
| `TemporalModel` | `activity` | `startToCloseTimeout: '1 minute'` | Configures every model Activity. |
| `TemporalModel` | `summary` | ADK agent name, then `adk.invokeModel <model>` | Sets the Activity summary. A function receives the request and must be deterministic. This takes precedence over `activity.summary`. |
| `TemporalModel` | `streamingTopic` | None | Publishes SSE response chunks to this Workflow streams topic when streaming is requested. |
| `TemporalModel` | `streamingBatchInterval` | `'100 milliseconds'` | Sets how frequently response chunks are batched for publication. |

| API | Option | Default | Behavior |
| --- | --- | --- | --- |
| `TemporalMCPToolset` | `name` | Required | Selects the Worker-registered factory and names its Activity pair. |
| `TemporalMCPToolset` | `toolFilter` | All tools | Advertises only listed tool names, matched after applying `prefix`. ADK `ToolPredicate` filters aren't supported. |
| `TemporalMCPToolset` | `prefix` | None | Advertises each tool as `<prefix>_<name>` without changing its MCP server name. |
| `TemporalMCPToolset` | `activity` | `startToCloseTimeout: '1 minute'` | Configures tool discovery and tool-call Activities. |
| `TemporalMCPToolset` | `connectionParams` | None | Creates a real MCP toolset only outside a Workflow. Worker-side configuration belongs in `mcpToolsets`. |
| `activityAsTool` | `name` | Required | Names the tool and the registered Activity it calls. |
| `activityAsTool` | `description` | Required | Describes the tool to the model. |
| `activityAsTool` | `parameters` | Empty object schema | Defines the arguments passed to the Activity as its single input. |
| `activityAsTool` | `activity` | `startToCloseTimeout: '1 minute'` | Configures the Activity call. |
| `FakeLlm` | `model` | `'fake-model'` | Sets the test double's model name. |
| `FakeLlm` | `responses` | One canned text response | Sets the responses yielded in order. |
| `fakeModelProvider` | `responses` | One canned text response | Returns a `FakeLlm` for every model name. |
| `mockMCPToolset` | `definitions` | Required | Creates an MCP factory from tool declarations and handlers. |

An MCP factory can return connection parameters or a `BaseToolset`. Connection parameters create and close one MCP
session per Activity. A `BaseToolset` remains owned by the factory, isn't closed by the plugin, and can maintain state.

### Failure behavior

The plugin exports constants for its public `ApplicationFailure.type` values. Model and MCP failures originate in
Activities, so catch the surrounding `ActivityFailure` and inspect its cause chain for these types.

| Failure type | Meaning |
| --- | --- |
| `GoogleAdkModelError[.<status>]` | A model call failed. Statuses 408, 409, 429, and 5xx are retryable; other HTTP statuses are non-retryable. A failure without a status is retryable. `x-should-retry` overrides this classification, and `retry-after` or `retry-after-ms` sets the next retry delay. |
| `GoogleAdkMCPError[.<status>]` | MCP discovery or a tool call failed. It uses the same status classification as model errors. Failures without a status are retryable, so set `activity.retry.maximumAttempts` to bound retries. |
| `GoogleAdkMCPToolNotFound` | A factory-provided `BaseToolset` didn't contain the requested tool. This failure is non-retryable. |
| `GoogleAdkStreamingTopicRequired` | SSE streaming was requested without `streamingTopic`. This failure is non-retryable and is thrown directly in the Workflow. |
| `GoogleAdkUnsupported` | `BaseLlm.connect` was called in a Workflow. This failure is non-retryable and is thrown directly in the Workflow. |

ADK converts an error from an agent's model call into an event. The integration records that failure and re-raises it
after the Workflow or handler frame returns. To recover in an ADK `onModelErrorCallback`, pass the error to
`markModelFailureHandled` and return a substitute event created with ADK's `createEvent`. Cancellation can't be handled
this way.

If a model call fails with a sandbox error such as `fetch is not defined`, use `new TemporalModel(...)` for a custom
model instance that performs I/O. The plugin routes built-in Gemini and Apigee model strings through `TemporalModel`
automatically; keep `autoRouteModels` enabled to use that behavior.

## Resources

- [Google ADK integration package](https://www.npmjs.com/package/@temporalio/google-adk)
- [Google ADK integration source](https://github.com/temporalio/ai-integrations/tree/main/typescript/google-adk)
- [Google ADK samples directory](https://github.com/temporalio/samples-typescript/tree/main/google-adk)
- [Google ADK source](https://github.com/google/adk-js)
- [Google ADK documentation](https://google.github.io/adk-docs/)
- [Temporal TypeScript SDK documentation](/develop/typescript)

---

# LangSmith integration - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/integrations/langsmith

> Add LangSmith tracing to TypeScript Workflows using the Temporal TypeScript SDK.

> **Public Preview**

Temporal's LangSmith integration lets you trace AI agent Workflows in [LangSmith](https://smith.langchain.com/)
alongside every LLM call, tool execution, and Temporal operation.

Temporal gives your agent code [durable execution](/temporal#durable-execution). LangSmith adds
the observability side, so you can inspect LLM inputs and outputs, follow a request from the Client through to the
model, and compare runs over time.

The `LangSmithPlugin` is what connects the two. It propagates trace context across Temporal boundaries
(Workflow → Activity → Child Workflow) so that runs started on the Client nest correctly under Workflow and Activity
runs on the Worker. It can also emit LangSmith runs for the Temporal operations themselves: Workflow executions, Activity
executions, Signals, and Updates.

All code snippets in this guide are taken from the
[LangSmith samples](https://github.com/temporalio/samples-typescript/tree/main/langsmith). Refer to the samples for
complete code.

## Prerequisites

- This guide assumes you are already familiar with LangSmith. If you aren't, refer to the
  [LangSmith documentation](https://docs.smith.langchain.com/) for more details.
- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking
  the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Ensure you have set up your local development environment by following the
  [Set up a local development environment with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript) guide. When you are
  done, leave the Temporal Development Server running if you want to test your code locally.

## Configure Workers to use LangSmith

Workers execute the code that defines your Workflows and Activities. To trace Workflow and Activity execution in
LangSmith, add the `LangSmithPlugin` to your Worker.

Follow the steps below to configure your Worker.

1. Install the `@temporalio/langsmith` package along with `langsmith`, which is a peer dependency.

   ```bash
   npm install @temporalio/langsmith langsmith
   ```

2. Construct a LangSmith Client and a single shared `LangSmithPlugin`, then pass the plugin to your Worker.

   ```ts {16}
   import { NativeConnection, Worker } from '@temporalio/worker';
   import { Client as LangSmithClient } from 'langsmith';
   import { LangSmithPlugin } from '@temporalio/langsmith';
   import * as activities from './activities';

   const connection = await NativeConnection.connect({ address: 'localhost:7233' });

   const langsmith = new LangSmithClient(); // reads LANGSMITH_API_KEY from the env
   const plugin = new LangSmithPlugin({ client: langsmith, addTemporalRuns: true });

   const worker = await Worker.create({
     connection,
     taskQueue: 'langsmith',
     workflowsPath: require.resolve('./workflows'),
     activities,
     plugins: [plugin],
   });

   await worker.run();
   ```

   Reuse this one instance on both the Worker and Client — see [Compose with other plugins](#compose-with-other-plugins).

3. Run the Worker with LangSmith tracing enabled. Ensure the Worker process has access to your LangSmith API key.

   ```bash
   export LANGSMITH_TRACING=true
   export LANGSMITH_API_KEY="your-api-key"
   ```

   Tracing is **off by default**, matching the `langsmith` library. With no tracing flag set to `true`, the plugin
   emits nothing.

   The plugin reads the same flags `langsmith` itself uses — `LANGSMITH_TRACING` (or `LANGSMITH_TRACING_V2`) and their
   `LANGCHAIN_` aliases.

## Configure Clients to use LangSmith

In TypeScript the Client and the Worker are configured independently, so add a `LangSmithPlugin` to your
`Client` too. This links client-side operations, like starting a Workflow, to the Workflows they trigger.

```ts {4}
import { Connection, Client } from '@temporalio/client';

const connection = await Connection.connect();
const client = new Client({ connection, plugins: [plugin] });
```

Wrap the call that starts your Workflow in a `traceable` so the rest of the trace nests under your own run.

```ts {3-11}
import { traceable } from 'langsmith/traceable';

const pipeline = traceable(
  async () => {
    return client.workflow.execute(GreetingWorkflow, {
      taskQueue: 'langsmith',
      workflowId: 'greeting-1',
      args: ['hello'],
    });
  },
  { name: 'user_pipeline' }
);

await pipeline();
```

## Trace Activities

Any non-deterministic work in a Temporal Workflow (LLM calls, tool executions, database queries, external API calls,
and so on) must run inside an Activity. That makes Activities an important place to add LangSmith runs. A `traceable`
from `langsmith/traceable` works unchanged inside an Activity body: the run shows up in LangSmith nested under
`RunActivity:` for the Activity that scheduled it.

<!--SNIPSTART typescript-langsmith-activity -->
[langsmith/activity-tracing/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/activity-tracing/src/activities.ts)
```ts
import { traceable } from 'langsmith/traceable';

const callModel = traceable(
  async (prompt: string): Promise<string> => {
    return `answer to: ${prompt}`;
  },
  { name: 'inner_llm_call' },
);

export async function answer(prompt: string): Promise<string> {
  return callModel(prompt);
}
```
<!--SNIPEND-->

## Trace Workflows

A `traceable` also works inside a Workflow body, and the plugin keeps it replay-safe: each run is emitted exactly
once and is never duplicated when the Workflow replays its history.

<!--SNIPSTART typescript-langsmith-workflow -->
[langsmith/workflow-tracing/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/workflow-tracing/src/workflows.ts)
```ts
import { traceable } from 'langsmith/traceable';

const extractKeyPoints = traceable(async (text: string): Promise<string> => `points:${text}`, {
  name: 'extract_key_points',
});

const summarize = traceable(async (points: string): Promise<string> => `summary:${points}`, {
  name: 'summarize',
});

export async function SummarizeWorkflow(text: string): Promise<string> {
  const points = await extractKeyPoints(text);
  return summarize(points);
}
```
<!--SNIPEND-->

> **📝 Note:**
>
> Inside a Workflow body, sequential `await inner(...)` nesting is exact. Under `Promise.all(...)` fan-out, or for
> `traceable` calls made after an `await` in the same scope, parenting falls back to the Workflow run. This affects only
> the visual shape of the trace, never Workflow history or control flow. Activity-side and client-side `traceable` are
> unaffected.
>

## Trace Signal and Update handlers

A `traceable` inside a Signal or Update handler nests under that handler's run, following the same Workflow-body
semantics. Temporal-internal Queries (`__temporal*`, `__stack_trace`) are never traced.

<!--SNIPSTART typescript-langsmith-handlers -->
[langsmith/message-handlers/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/message-handlers/src/workflows.ts)
```ts
import { traceable } from 'langsmith/traceable';
import { allHandlersFinished, condition, defineSignal, defineUpdate, setHandler } from '@temporalio/workflow';

const classifyMessage = traceable(async (text: string): Promise<string> => `intent:${text}`, {
  name: 'classify_intent',
});

const draftReply = traceable(async (text: string): Promise<string> => `reply:${text}`, {
  name: 'draft_reply',
});

export const handleMessage = defineSignal<[string]>('handle_message');
export const composeReply = defineUpdate<string, [string]>('compose_reply');
export const complete = defineSignal('complete');

export async function ConversationWorkflow(): Promise<string[]> {
  const log: string[] = [];
  let done = false;

  setHandler(handleMessage, async (text: string) => {
    log.push(await classifyMessage(text));
  });

  setHandler(composeReply, async (text: string) => {
    const reply = await draftReply(text);
    log.push(reply);
    return reply;
  });

  setHandler(complete, () => {
    done = true;
  });

  await condition(() => done && allHandlersFinished());
  return log;
}
```
<!--SNIPEND-->

## Trace multi-step agents across Activities and Child Workflows

A single trace threads through a multi-step agent that calls Activities and a Child Workflow. The parent Workflow
gathers facts and writes a report in Activities, then delegates review to a Child Workflow, and every run nests under
the same trace.

<!--SNIPSTART typescript-langsmith-agent-pipeline -->
[langsmith/agent-pipeline/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/agent-pipeline/src/workflows.ts)
```ts
import { executeChild, proxyActivities, workflowInfo } from '@temporalio/workflow';
import type * as activities from './activities';

const { gatherFacts, writeReport, reviewReport } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

export async function ReviewWorkflow(report: string): Promise<string> {
  return reviewReport(report);
}

export async function ResearchWorkflow(topic: string): Promise<string> {
  const facts = await gatherFacts(topic);
  const report = await writeReport(facts);
  return executeChild(ReviewWorkflow, {
    args: [report],
    workflowId: `${workflowInfo().workflowId}-review`,
  });
}
```
<!--SNIPEND-->

## Include Temporal operations as runs

By default (`addTemporalRuns: false`), the plugin only propagates LangSmith context so that your `traceable` runs nest
correctly. It does not create its own runs.

Set `addTemporalRuns: true` if you want first-class runs for the Temporal operations themselves: Workflow executions,
Activity executions, Signals, Updates, and so on (`StartWorkflow:`, `RunWorkflow:`, `StartActivity:`, `RunActivity:`,
`HandleSignal:`, `HandleUpdate:`). `Start*` and `Run*` pairs appear as siblings: the `Start*` run is emitted by the
side scheduling the operation (for example, the Client), and the `Run*` run is emitted by the side executing it (for
example, the Worker).

With the plugin configured on both the Client and the Worker, and `addTemporalRuns: true`, a trace for a simple LLM
call looks like this:

```
user_pipeline
  StartWorkflow:GreetingWorkflow
  RunWorkflow:GreetingWorkflow
    StartActivity:answer
    RunActivity:answer
      inner_llm_call
```

Without `addTemporalRuns` (the default), only your `traceable` runs appear. Context still propagates, so they nest
correctly under the client-side run:

```
user_pipeline
  inner_llm_call
```

## Plugin options

| Option            | Default           | Meaning                                                                                                                                        |
| ----------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `client`          | `new Client()`    | The LangSmith `Client` that runs are emitted to.                                                                                               |
| `addTemporalRuns` | `false`           | Emit first-class runs for Temporal operations (`StartWorkflow:`, `RunActivity:`, `HandleSignal:`, …) in addition to your `traceable` runs.     |
| `projectName`     | LangSmith default | Target LangSmith project for emitted runs.                                                                                                     |
| `tags`            | —                 | Tags attached to every run the plugin emits.                                                                                                   |
| `metadata`        | —                 | Metadata merged into every run the plugin emits. Credential-looking keys are scrubbed before emission.                                         |

The LangSmith API key is never accepted as a plugin option and never crosses a Temporal boundary. Supply a
pre-constructed `Client` (which reads `LANGSMITH_API_KEY` from the process environment) or let the plugin build a
default client from the environment.

## Compose with other plugins

Register observability **first** (outermost) so it observes everything beneath it, then governance, then
agent-framework plugins.

```ts
const worker = await Worker.create({
  connection,
  taskQueue: 'langsmith',
  workflowsPath: require.resolve('./workflows'),
  activities,
  plugins: [
    new LangSmithPlugin({ client: langsmith }), // observability — first
    // new GovernancePlugin(...),
    // new AgentFrameworkPlugin(...),
  ],
});
```

The plugin de-duplicates its own instrumentation, so a Worker built from a plugin-configured Client will not
double-instrument.

## Samples

| Sample                                                                                  | Demonstrates                                                                                  |
| --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| [activity-tracing](https://github.com/temporalio/samples-typescript/tree/main/langsmith/activity-tracing) | A `traceable` model call inside an Activity, nested under the Workflow and Activity runs.      |
| [workflow-tracing](https://github.com/temporalio/samples-typescript/tree/main/langsmith/workflow-tracing) | Replay-safe `traceable` calls in a Workflow body — emitted once, never duplicated on replay.   |
| [agent-pipeline](https://github.com/temporalio/samples-typescript/tree/main/langsmith/agent-pipeline)     | A multi-step agent whose trace threads through Activities and a Child Workflow.                |
| [message-handlers](https://github.com/temporalio/samples-typescript/tree/main/langsmith/message-handlers) | `traceable` calls inside Signal and Update handlers, nested under each handler's run.          |

---

# OpenAI Agents SDK integration - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/integrations/openai-agents

> Run OpenAI Agents SDK agents as durable TypeScript Workflows, with model calls, MCP operations, and sandbox operations running as retryable Activities.

The Temporal integration for the [OpenAI Agents SDK](https://openai.github.io/openai-agents-js/) runs its agents as
durable Temporal Workflows in TypeScript. The integration keeps the agent loop, tool selection, and handoffs in the
Workflow and runs each model call, MCP operation, and sandbox operation as an [Activity](/glossary#activity), because a
[Workflow Definition](/workflow-definition) must be deterministic and can't call a model, an MCP server, or a sandbox
itself.

Activities retry on failure. Completed ones are recorded in Event History and aren't repeated when the Workflow replays,
so an agent survives Worker restarts and can run for as long as it needs.

Code snippets on this page come from the
[OpenAI Agents samples directory](https://github.com/temporalio/samples-typescript/tree/main/openai-agents).

> **Pre-release**

## Install the integration

Before you start:

- This guide assumes you're familiar with the OpenAI Agents SDK. If you aren't, start with the
  [OpenAI Agents SDK documentation](https://openai.github.io/openai-agents-js/).
- If you're new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the
  [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local environment by following
  [Set up a local development environment with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript),
  and leave the Temporal development server running to try your code locally.

Install the [`@temporalio/openai-agents`](https://www.npmjs.com/package/@temporalio/openai-agents) package along with
the OpenAI Agents SDK packages it uses as peer dependencies,
[`@openai/agents-core`](https://www.npmjs.com/package/@openai/agents-core),
[`@openai/agents-openai`](https://www.npmjs.com/package/@openai/agents-openai), and
[`openai`](https://www.npmjs.com/package/openai):

```bash
npm install @temporalio/openai-agents @openai/agents-core @openai/agents-openai openai
```

The integration supports a single minor version of the OpenAI Agents SDK packages, so install them in the same command
and npm picks versions that match. Keep every `@temporalio/*` package in your application on the same version, since the
integration depends on exact versions of them.

Import Worker and Client APIs from `@temporalio/openai-agents`, and Workflow APIs from
`@temporalio/openai-agents/workflow`, which is safe to load in the Workflow sandbox. [Configuration](#configuration)
lists every entry point.

## Run an agent in a Workflow

A Temporal-backed agent has three parts: a Workflow that runs the agent, a Worker with the integration's plugin, and a
Client that starts the Workflow.

In the Workflow, run the agent with `TemporalOpenAIRunner` instead of the OpenAI Agents SDK `Runner`:

<!--SNIPSTART typescript-openai-agents-hello-world-workflow-->
[openai-agents/src/basic/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/workflows.ts)
```ts
export async function helloWorld(prompt: string): Promise<string> {
  const agent = new Agent({ name: 'HelloAgent', instructions: 'You are a helpful assistant.' });
  const result = await new TemporalOpenAIRunner().run(agent, prompt);
  return result.finalOutput ?? '';
}
```
<!--SNIPEND-->

Register `OpenAIAgentsPlugin` on the Worker with a model provider:

<!--SNIPSTART typescript-openai-agents-hello-world-worker-->
[openai-agents/src/basic/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/worker.ts)
```ts
const worker = await Worker.create({
  connection,
  taskQueue: 'openai-agents-basic',
  workflowsPath: require.resolve('./workflows'),
  activities,
  plugins: [
    new OpenAIAgentsPlugin({
      modelProvider: new OpenAIProvider({ apiKey }),
      modelParams: { useLocalActivity: true },
    }),
  ],
  bundlerOptions: {
    webpackConfigHook: (config) => ({
      ...config,
      resolve: {
        ...config.resolve,
        conditionNames: ['require', 'browser', 'default'],
      },
    }),
  },
});
await worker.run();
```
<!--SNIPEND-->

The plugin registers the Activity that calls the model, along with the interceptors and Workflow bundle polyfills the
OpenAI Agents SDK needs. The model provider runs inside that Activity, so only Workers need your model provider's
credentials.

Register the plugin on the Client too:

<!--SNIPSTART typescript-openai-agents-hello-world-client-->
[openai-agents/src/basic/client.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/client.ts)
```ts
const connection = await Connection.connect();
const client = new Client({
  connection,
  plugins: [new OpenAIAgentsPlugin({ modelProvider: new OpenAIProvider({ apiKey }) })],
});

const taskQueue = 'openai-agents-basic';
const workflowId = 'openai-agents-' + nanoid();
```
<!--SNIPEND-->

The Client's plugin requires a `modelProvider` but never calls it, so that provider doesn't need credentials. The
plugin sends its `modelParams` and `interceptorOptions` to each Workflow the Client starts. `modelParams` holds the
model Activity options, such as timeouts, a retry policy, or `useLocalActivity: true`, and `interceptorOptions` holds
the tracing options.

> **📝 Note:**
>
> A Worker plugin's `modelParams` doesn't reach model calls, even on the Worker that runs them. Set model Activity options
> on the Client's plugin or in the runner's `defaultModelParams`.
>

Agents name their model with a string, or leave it out to use the provider's default, and the Worker's model provider
resolves it inside the Activity. A model object can't cross into an Activity, so an agent that has one fails the run. To
run every agent in one run on the same model, set `runConfig.model` to that model's name. Structured output through an
agent's `outputType` works as in the OpenAI Agents SDK, and the
[basic sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/basic) shows it along with
run context and a per-run model override.

Any OpenAI Agents SDK `ModelProvider` works on the Worker. The
[model providers sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/model-providers)
points `OpenAIProvider` at an OpenAI-compatible endpoint by setting its `baseURL`.

To get a reasoning model's summary, set `modelSettings.reasoning.summary` on the agent and read the `reasoning_item`
entries in `result.newItems`. The
[reasoning content sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/reasoning-content)
takes a different route and calls the `openai` package directly from its own Activity, outside the agent loop.

The rest of the agent loop runs in the Workflow, including handoffs, guardrails, and `instructions` functions, so that
code must be deterministic. The `context` you pass to `run()` reaches that code and `tool()` functions, but not
`activityAsTool` Activities or `agentAsTool` nested runs.

Each model call is its own Activity, so agents you run together with `Promise.all` make their model calls in parallel.
The [handoffs](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/handoffs),
[agent patterns](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/agent-patterns), and
[multi-agent](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/multi-agent) samples show
triage handoffs, chained and parallel agents, an LLM judge, and guardrails.

In a run without `stream: true`, an error the OpenAI Agents SDK throws during the run, such as a tripped guardrail or
`MaxTurnsExceededError`, reaches the Workflow as a non-retryable `ApplicationFailure` of type `AgentsWorkflowError`, with
the SDK's error as its `cause`. Check `cause` when you catch it, as the agent patterns sample does for guardrails.

## Add tools and MCP servers

An OpenAI Agents SDK `tool()` runs its `execute` function in the Workflow, so a tool that calls an API, reads a file,
queries a database, or does other I/O has to run in an Activity or a Nexus Operation instead.

Wrap an Activity as a tool with `activityAsTool`:

<!--SNIPSTART typescript-openai-agents-activity-tool-workflow-->
[openai-agents/src/basic/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/workflows.ts)
```ts
export async function tools(prompt: string): Promise<string> {
  const weatherTool = activityAsTool<typeof activities.getWeather>(
    {
      name: 'getWeather',
      description: 'Get the current weather for a city.',
      parameters: {
        type: 'object',
        properties: { city: { type: 'string', description: 'The city name' } },
        required: ['city'],
        additionalProperties: false,
      },
    },
    { startToCloseTimeout: '1 minute' },
  );

  const agent = new Agent({
    name: 'WeatherAgent',
    instructions: 'You are a helpful weather assistant. Always use the getWeather tool to answer weather questions.',
    tools: [weatherTool],
  });
  const result = await new TemporalOpenAIRunner().run(agent, prompt);
  return result.finalOutput ?? '';
}
```
<!--SNIPEND-->

The tool's `name` must match an Activity registered on the Worker. The model's arguments become the Activity's single
argument, and the Activity's result goes back to the model as a string, or as JSON if it isn't one. `activityAsTool`'s
second argument sets the Activity's timeouts, retry policy, and Task Queue.

Use `tool()` for deterministic work. It can also call an Activity, as the basic sample's local Activity tool does
through `proxyLocalActivities`.

`nexusOperationAsTool` exposes a [Nexus](/nexus) Operation as a tool. Pass it the Operation from your service
definition, a tool definition shaped like `activityAsTool`'s, and options with the `service` and `endpoint`. The
Workflow runs the Operation and returns its result to the agent, as in the
[Nexus tools sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/nexus-tools).

To call another agent as a tool, use the integration's `agentAsTool` rather than the SDK's `agent.asTool()`, so the
nested agent's model calls run as Activities too. The nested run gets only the parent run's `runConfig.sandbox`, so the
parent's other run settings, such as `traceIncludeSensitiveData` and its `runConfig` guardrails, don't apply inside it,
and `agentAsTool` has no option to set them. The agent patterns sample uses `agentAsTool` for an orchestrator that asks
a specialist.

MCP servers run on the Worker, and Workflow code refers to them by name. Register a provider for each server in the
plugin's `mcpServerProviders`, then add a handle with the same name to the agent's `mcpServers`:

- **Stateless servers:** `StatelessMCPServerProvider` takes a name and a factory that returns an OpenAI Agents SDK MCP
  server, such as `MCPServerStdio` or `MCPServerStreamableHttp`. In the Workflow, use `statelessMcpServer(name)`. Each
  tool listing and tool call is an Activity that connects to the server, does that one operation, and disconnects, so no
  state carries between calls.
- **Stateful servers:** `StatefulMCPServerProvider` also takes a `NativeConnection`, usually the Worker's. In the Workflow, call
  `connect()` on the `statefulMcpServer(name)` handle before the run and `cleanup()` in a `finally` block. `connect()`
  starts a long-running Activity that opens the server and runs a dedicated Worker for that Workflow run, so the server
  keeps its state until `cleanup()`.

The handles expose a server's tools, not its prompts, so fetch an MCP prompt in your own Activity. The
[MCP sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/mcp) does this and shows
stdio, Streamable HTTP, SSE, and stateful servers.

Hosted tools from `@openai/agents-openai`, such as `webSearchTool()`, `imageGenerationTool()`, and
`codeInterpreterTool()`, run inside the model provider during the model Activity, as the
[hosted tools sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/tools) shows. A
`hostedMcpTool()` works the same way, so the model provider calls the remote MCP server itself. Its `onApproval`
callback runs in the Workflow and can wait for a Signal, as in the
[hosted MCP sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/hosted-mcp).

Tool declarations are part of the model Activity's input, so a credential written into one, such as a hosted MCP tool's
`authorization`, is recorded in Event History. Supply it from the Worker with the plugin's `hostedToolCredentials`
instead, which keeps it out of that input, as described in [Configuration](#configuration).

## Keep conversation history and pause for approval

Agent state lives in the Workflow, so a conversation or a pending approval survives Worker restarts for as long as the
Workflow runs.

To keep conversation history across `run()` calls, pass a `WorkflowSafeMemorySession` as the `session` option, since
the runner rejects the SDK's `MemorySession`. The history is held in the Workflow and rebuilt by replay, but a
Continue-As-New starts with an empty session. To carry it over, pass `await session.getItems()` to `continueAsNew()`
and construct the new run's session with `initialItems`. The
[sessions sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/sessions) shows a
multi-turn chat and the carry-over.

Instead of a session, a long-running Workflow can keep the conversation itself and pass the accumulated
`result.history` to each `run()` call, as the
[stateful conversation sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/stateful-conversation)
does.

A tool with `needsApproval` pauses the run, and `run()` returns with `result.interruptions` set. Wait for a Signal or
Update, call `result.state.approve(interruption)` or `result.state.reject(interruption)` for each entry in
`result.interruptions`, then resume by passing `result.state` back to `run()`.

To resume in a fresh Event History instead, serialize the state with `result.state.toString()` and pass it to
`continueAsNew()` once the approval arrives. In the new run, rehydrate it with `RunState.fromString()`, passing an
agent with the same tools, handoffs, and MCP servers as the run that paused, then approve or reject each entry in its
`getInterruptions()` and pass the state to `run()`. The
[human approval sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/human-approval)
takes this route and includes the Client that sends the approval Signal. For how the SDK serializes `RunState`, see
[Dealing with longer approval times](https://openai.github.io/openai-agents-js/guides/human-in-the-loop/#dealing-with-longer-approval-times)
in OpenAI's human-in-the-loop guide.

## Run sandbox agents

A `SandboxAgent` works in a sandbox, a workspace where it runs shell commands and reads and edits files. The
integration runs each sandbox operation as an Activity, including creating the session, every command and file
operation, and the teardown. Sandbox support is experimental. For manifests, capabilities, and backends, see OpenAI's
[sandbox agents concepts guide](https://openai.github.io/openai-agents-js/guides/sandbox-agents/concepts/).

On the Worker, wrap a sandbox backend's client in a `SandboxClientProvider` under a name, and pass it to the plugin's
`sandboxClientProviders`:

<!--SNIPSTART typescript-openai-agents-sandboxes-worker-configuration-->
[openai-agents/src/sandboxes/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/sandboxes/worker.ts)
```ts
plugins: [
  new OpenAIAgentsPlugin({
    modelProvider: new OpenAIProvider({ apiKey }),
    sandboxClientProviders: [new SandboxClientProvider('local', new UnixLocalSandboxClient())],
    resolvableWorkerEnvVars: ['SANDBOX_SERVICE_TOKEN'],
  }),
],
```
<!--SNIPEND-->

> **⚠️ Warning:**
>
> `UnixLocalSandboxClient`, from `@openai/agents-core/sandbox/local`, runs the agent's commands directly on the Worker
> host as the Worker's OS user, with no container or chroot around them. Use it only for local development. To isolate
> the agent, register `DockerSandboxClient`, from the same entry point, or a hosted backend from OpenAI's
> [sandbox clients guide](https://openai.github.io/openai-agents-js/guides/sandbox-agents/clients/) under the same name.
> Only the Worker changes.
>

In the Workflow, set `runConfig.sandbox.client` to `temporalSandboxClient()` with the provider's name:

<!--SNIPSTART typescript-openai-agents-sandboxes-workflow-->
[openai-agents/src/sandboxes/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/sandboxes/workflows.ts)
```ts
import { Manifest, SandboxAgent } from '@openai/agents-core/sandbox';
import { TemporalOpenAIRunner, temporalSandboxClient, workerEnvValue } from '@temporalio/openai-agents/workflow';
import { ApplicationFailure } from '@temporalio/workflow';

export async function sandboxWorkflow(prompt: string): Promise<string> {
  const agent = new SandboxAgent({
    name: 'SandboxAgent',
    instructions:
      'Use the sandbox to do the work rather than answering from memory, then report what the commands returned. ' +
      'Run each command to completion in a single exec_command call instead of leaving a process running to check ' +
      'on later. SERVICE_TOKEN holds a credential: you may use it, but never print its value.',
    defaultManifest: new Manifest({
      environment: { SERVICE_TOKEN: workerEnvValue('SANDBOX_SERVICE_TOKEN') },
    }),
  });
  const result = await new TemporalOpenAIRunner().run(agent, prompt, {
    runConfig: { sandbox: { client: temporalSandboxClient('local') } },
  });
  if (typeof result.finalOutput !== 'string') {
    throw ApplicationFailure.nonRetryable('SandboxAgent returned no final output');
  }
  return result.finalOutput;
}
```
<!--SNIPEND-->

Event History records every sandbox Activity's input and result, including the session's manifest and each command's
output, so a literal manifest value is recorded even when it's marked `ephemeral: true`. To keep a secret's value out
of the manifest, set the manifest value to `workerEnvValue()` with the name of a Worker environment variable, as the
Workflow above does, and list that name in the plugin's `resolvableWorkerEnvVars`. The manifest then records only the
variable's name. Unix-local and Docker also leave the resolved value out of the session state they report, so with
those backends the value doesn't enter Event History. With other backends, keeping it out of that state is the
backend's responsibility. Set the variable on every Worker that runs sandbox Activities, because a listed variable that
isn't set reads as an empty string.

`workerEnvValue()` doesn't stop a secret from reaching history another way. A command that prints the secret still
records it in Event History, so tell the agent not to print it, and encrypt what reaches history with a
[Payload Codec](/develop/typescript/best-practices/data-handling/data-encryption).

The runner tears down the session when the run completes or fails, unless you passed it in `runConfig.sandbox.session`.
Close a session you pass that way yourself when you're done with it, by calling its `stop()`, `shutdown()`, and
`delete()` methods in the Workflow. A run that pauses for approval keeps its session, so resume it as described in
[Keep conversation history and pause for approval](#keep-conversation-history-and-pause-for-approval) and pass the
same `runConfig.sandbox` to that `run()` call.

Canceling, terminating, or timing out the Workflow skips the teardown, as does never resuming a paused run. Remove
those sessions through your backend, for example by deleting the Docker container or the hosted sandbox.

Unix-local and Docker sessions stay on the machine that created them, so a Worker on another machine can't reach them.
With those backends, send sandbox Activities to a Task Queue that only one host's Workers poll, by passing Activity
options with that `taskQueue` as the second argument to `temporalSandboxClient()`. The options replace the default
five-minute Start-To-Close Timeout instead of merging with it, so include a `startToCloseTimeout`.

With Unix-local, let a single Worker process handle sandbox Activities and don't restart it during a run. A Unix-local
session that another Worker process reconnects to, including the same Worker after a restart, goes back to the snapshot
taken when the session was created or last paused and loses later changes. Neither Unix-local nor Docker can reconnect
a process that's still running, which is why the Workflow above tells the agent to finish each command in one call.
[Failure behavior](#failure-behavior) covers reconnects in detail, and the
[sandboxes sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/sandboxes) shows this
Worker and Workflow with the Client that starts them.

## Stream model output

Pass `{ stream: true }` to `run()` to stream model events to a client while the run stays durable. Streaming is
experimental. It publishes over [Workflow Streams](/develop/typescript/workflows/workflow-streams), so install
[`@temporalio/workflow-streams`](https://www.npmjs.com/package/@temporalio/workflow-streams) and construct a
`WorkflowStream` at the top of the Workflow. Set the topic in `modelParams.streamingTopic` on the Client's plugin, or in
the runner's `defaultModelParams`, and leave `useLocalActivity` off, since a streaming run fails with it.

Each model call runs as a streaming Activity that publishes the model's events to the topic as they arrive. In the
Workflow, iterate the result to drive the run to completion. That iteration yields each call's events only after the
call finishes, so subscribe from outside the Workflow to see them live. A retried model call publishes its events again
from the start, so a subscriber can see a call's events more than once.

Catch errors from iterating the result and from awaiting `result.completed`, since the OpenAI Agents SDK's errors
arrive there as its own errors rather than as `AgentsWorkflowError`. An uncaught one doesn't fail the Workflow but fails
the Workflow Task, which retries.

Subscribe with `WorkflowStreamClient`, as described in
[Subscribe](/develop/typescript/workflows/workflow-streams#subscribe), and keep the Workflow open until the subscriber
has the last event, as described in
[Closing the stream](/develop/typescript/workflows/workflow-streams#closing-the-stream). The
[streaming sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/streaming) shows the
Workflow, the Client's topic setting, and a subscriber that signals the Workflow when it's done.

## Trace agent runs

OpenAI Agents SDK traces follow an agent run across the Client, the Workflow, Activities, Nexus Operations, and MCP
calls. Spans from Workflow code go to the trace processors in the Worker process, so register exporters and processors
there, not in Workflow code. Those spans aren't emitted again when the Worker replays history.

To send traces to OpenAI's dashboard, register OpenAI's exporter with
`addTraceProcessor(new BatchTraceProcessor(new OpenAITracingExporter()))`. Register your own `TracingProcessor` with
`addTraceProcessor` too. Don't call `setDefaultOpenAITracingExporter()` after constructing the plugin, because it
replaces every trace processor, including the one the plugin uses for OpenTelemetry.

By default, OpenAI Agents SDK spans include model and tool inputs and outputs, and
`runConfig.traceIncludeSensitiveData: false` leaves them out, as described in
[Sensitive data](https://openai.github.io/openai-agents-js/guides/tracing/#sensitive-data) in OpenAI's tracing guide.
The SDK's `sandbox.exec` span records each sandbox command's text even then, so a command that contains a secret
reaches every trace processor.

To export OpenAI Agents SDK spans through OpenTelemetry, install the optional peer dependency
[`@opentelemetry/sdk-trace-base`](https://www.npmjs.com/package/@opentelemetry/sdk-trace-base), then do the following
in both the Worker and the Client process:

1. Create a tracer provider with `createTracerProvider()` from `@temporalio/openai-agents/otel`, and add your
   exporter's span processor to it with `addSpanProcessor()`. The provider starts with no span processor, so it exports
   nothing until you add one.
2. Before constructing the plugin, register the provider with `trace.setGlobalTracerProvider()`.
3. Set `useOtelInstrumentation: true` in the plugin's `interceptorOptions`.

If your application already has its own tracer provider, see
[Composing with other plugins](#composing-with-other-plugins).

Set `addTemporalSpans: true` in `interceptorOptions` to add `temporal:*` spans for Temporal operations, such as Workflow
starts, Signals, and Activities, and `sandbox:*` spans for sandbox operations. They're OpenAI Agents SDK spans, so they
reach every trace processor, and OpenTelemetry when it's on. A span for a Client call, and for the Workflow start,
Signal, Query, or Update it makes, appears only when that call runs inside an agent trace, such as one started with
`withTrace()`.

Workflow code reads `interceptorOptions` from the Client that started the Workflow, the same way it reads `modelParams`,
so set them on the Client's plugin for spans from Workflow code. On the Worker's plugin, `addTemporalSpans` adds spans
for Activities. The
[tracing sample](https://github.com/temporalio/samples-typescript/tree/main/openai-agents/src/tracing) shows a custom
processor, OpenAI's exporter, and OpenTelemetry.

## Reference and troubleshooting

### Feature support

| Feature | Support | Notes |
| --- | --- | --- |
| Multi-turn agent runs | Supported | The agent loop runs in the Workflow. |
| Handoffs | Supported | `Agent[]` and `handoff()` forms, including input filters. |
| Function tools | Supported | `tool()` runs in the Workflow and must be deterministic. |
| Activities as tools | Supported | `activityAsTool()`. The Activity doesn't receive the run's `context`. |
| Nexus Operations as tools | Supported | `nexusOperationAsTool()`. |
| Agents as tools | Supported | `agentAsTool()`. |
| Hosted tools | Supported | The model provider runs them during the model Activity. |
| Hosted MCP tools | Supported | `onApproval` runs in the Workflow. |
| Stateless MCP servers | Supported | One connection per tool listing or tool call. |
| Stateful MCP servers | Supported | A dedicated Worker holds the server for each Workflow run. |
| MCP prompts | Not supported through MCP handles | Fetch prompts in your own Activity. |
| Sandbox agents | Supported (experimental) | Unix-local and Docker sessions are tied to one host. |
| Reconnecting to a sandbox process that's still running | Not supported on Unix-local or Docker | Have the agent finish each command in one `exec_command` call. See [Failure behavior](#failure-behavior). |
| Sandbox runs paused for approval | Supported on backends with `resume()` | Every paused-run resume calls the backend's `resume()`. See [Failure behavior](#failure-behavior). |
| Sessions | Supported | `WorkflowSafeMemorySession`. The SDK's `MemorySession` is rejected. |
| Run state and approvals | Supported | Serialize `RunState` to resume across Continue-As-New. |
| Guardrails | Supported | Run in the Workflow. Without `stream: true`, a tripwire arrives as the `cause` of an `AgentsWorkflowError`. |
| Structured output | Supported | `outputType` on the agent. |
| Run context | Supported | Visible to code in the Workflow, not to Activity tools or `agentAsTool` nested runs. |
| Per-run model override | Supported | `runConfig.model` takes a model name. |
| Model objects on agents | Not supported | Name the model with a string. |
| Custom model providers | Supported | Any `ModelProvider`, called in the model Activity. |
| Local Activities for model calls | Supported | `useLocalActivity: true` in the Client's `modelParams` or the runner's `defaultModelParams`. |
| Streaming | Supported (experimental) | Requires a `streamingTopic` and a `WorkflowStream` constructed in the Workflow. Not with Local Activities. |
| Tracing | Supported | OpenAI's exporter, custom processors, OpenTelemetry, and `temporal:*` spans. |
| Cancellation through `signal` | Not supported | Cancel the Workflow or use a [cancellation scope](/develop/typescript/workflows/cancellation-scopes). Both skip a sandbox run's teardown. See [Failure behavior](#failure-behavior). |
| Child Workflows and Continue-As-New | Supported | Plugin settings from the Client carry over. |
| Voice and realtime agents | Not supported | |

### Composing with other plugins

`OpenAIAgentsPlugin` adds its Activities and interceptors to the ones already on the Worker and Client, so it works
alongside other plugins. Register one instance per Client. When several are registered, the last one's `modelParams` and
`interceptorOptions` reach the Workflow.

The plugin's bundler hook puts its polyfills first in the Workflow bundle, ahead of every other module, so plugin order
doesn't affect them. When you prebuild the bundle with `bundleWorkflowCode`, pass the plugin to `bundleWorkflowCode` as
well as to the Worker, so the bundle includes the polyfills and the Workflow interceptor. To replay histories, as in
[How to Replay a Workflow Execution](/develop/typescript/best-practices/testing-suite#replay), pass the plugin to
`Worker.runReplayHistory` too.

The plugin registers Activities named `invokeModelActivity` and `invokeModelStreamActivity`, plus Activities whose names
start with each MCP and sandbox provider's name. Don't register your own Activities under those names.

OpenTelemetry export needs a replay-safe global tracer provider. If your application already registers its own global
tracer provider, configure it with `TemporalIdGenerator` and mark it with `markReplaySafeTracerProvider()`. Mark only
providers that use `TemporalIdGenerator`.

### Replay safety

These run in the Workflow and must be deterministic: the agent loop, agent `instructions` functions, `tool()` `execute`
functions, guardrails, handoff input filters, `callModelInputFilter`, hosted MCP `onApproval` callbacks,
`WorkflowSafeMemorySession`, a `ModelSummaryProvider`, and `agentAsTool`'s `customOutputExtractor`. They can use
Workflow APIs such as Signals, `condition()`, and `proxyActivities`.

These run in Activities on the Worker: model calls through the model provider, `hostedToolCredentials`, Activities
wrapped with `activityAsTool`, MCP server operations, and sandbox operations. `nexusOperationAsTool` runs a Nexus
Operation.

Model requests and responses, tool arguments and results, MCP results, and sandbox inputs and results, including
manifests, command output, and the session state a backend reports, are all Activity or Nexus inputs and results. Event
History stores them, and replay reads them back instead of calling again. Encrypt sensitive data with a
[Payload Codec](/develop/typescript/best-practices/data-handling/data-encryption), and account for the payload size
limit listed under [Programming model level](/evaluate/cloud/limits#programming-model-level) in the Temporal Cloud
limits. For long conversations, use
[Continue-As-New](/develop/typescript/workflows/continue-as-new) before Event History approaches its limits.

Replay reproduces what the original run saw. It rebuilds `WorkflowSafeMemorySession` history, a streaming run iterates
the same events its streaming Activity returned, and trace and span identifiers come out the same because they come
from a replay-seeded random source. A Continue-As-New starts the session empty, but the plugin settings from the Client
carry over to the new run and to child Workflows.

### Configuration

| Entry point | Use from | APIs |
| --- | --- | --- |
| `@temporalio/openai-agents` | Worker and Client code | `OpenAIAgentsPlugin`, `StatelessMCPServerProvider`, `StatefulMCPServerProvider`, `SandboxClientProvider`, `OpenAIAgentsTraceClientInterceptor` for wiring a Client by hand, `DEDICATED_WORKER_FAILURE_TYPE`, and option types |
| `@temporalio/openai-agents/workflow` | Workflow code | `TemporalOpenAIRunner`, `activityAsTool`, `nexusOperationAsTool`, `agentAsTool`, `statelessMcpServer`, `statefulMcpServer`, `WorkflowSafeMemorySession`, `temporalSandboxClient`, `workerEnvValue`, `ToolSerializationError`, and `DEDICATED_WORKER_FAILURE_TYPE` |
| `@temporalio/openai-agents/otel` | Worker and Client code | `createTracerProvider`, `TemporalIdGenerator`, `markReplaySafeTracerProvider`, and `isReplaySafeTracerProvider`. Requires `@opentelemetry/sdk-trace-base`. |
| `@temporalio/openai-agents/workflow-interceptor` | Worker bundling | The Workflow interceptor module, for listing by hand in `interceptors.workflowModules` on `Worker.create`, or in `workflowInterceptorModules` for `bundleWorkflowCode` |

`OpenAIAgentsPlugin` options:

| Option | Set on | Default | Behavior |
| --- | --- | --- | --- |
| `modelProvider` | Worker, and required on the Client | None | Resolves model names in the model Activity. The Client's plugin requires one but never calls it. |
| `modelParams` | Client | None | Model Activity options sent to each Workflow the Client starts. See the next table. |
| `interceptorOptions.addTemporalSpans` | Client and Worker | `false` | Adds `temporal:*` spans and `sandbox:*` spans. The Client covers Client calls and Workflow code, and the Worker covers Activities. See the notes below this table. |
| `interceptorOptions.useOtelInstrumentation` | Client and Worker | `false` | On the Client, forwards spans from Workflow code to OpenTelemetry. On the Worker, it only makes the constructor check for a replay-safe tracer provider, since OpenAI Agents SDK spans from Activities reach the global tracer provider either way. Requires a replay-safe global tracer provider in that process before the plugin is constructed. |
| `mcpServerProviders` | Worker | `[]` | Registers each MCP provider's Activities. |
| `sandboxClientProviders` | Worker | `[]` | Registers each sandbox provider's Activities. Experimental. |
| `resolvableWorkerEnvVars` | Worker | `[]` | Worker environment variable names that a `workerEnvValue()` may read. `'*'` allows every name. A listed variable that isn't set reads as an empty string. Experimental. |
| `hostedToolCredentials` | Worker | None | A function that takes a hosted tool's identity and returns, or resolves to, its credentials. Returning `undefined` leaves the tool as the Workflow declared it. Experimental. See the notes below this table. |

Notes on `addTemporalSpans`:

- On the Client, it adds `temporal:*` spans for Client calls and, in Workflow code, for Workflow starts, Signals, Queries,
  Updates, Activities, Local Activities, child Workflows, Nexus Operations, and Continue-As-New, plus `sandbox:*` spans
  for sandbox operations. On the Worker, it adds `temporal:executeActivity:*` spans and `sandbox:*:result` spans.
- A span for a Client call, and for the Workflow start, Signal, Query, or Update it makes, appears only when that call
  runs inside an agent trace.
- The `sandbox:*` spans carry only the session identifier and, depending on the operation, an exit code, a port, or a
  result size. Session creation gets no `sandbox:*` span and no `:result` span, and shows up only as the generic
  `temporal:startActivity:*` and `temporal:executeActivity:*` spans for the provider's `-sandbox-client-create` Activity.
- The SDK's own `sandbox.exec` span is separate, isn't controlled by this option, and records the command text even with
  `traceIncludeSensitiveData: false`.

Notes on `hostedToolCredentials`:

- The identity is `{ tool: 'hostedMcp', name, serverLabel, serverUrl?, connectorId? }`,
  `{ tool: 'shell', name, allowedDomains }`, or `{ tool: 'codeInterpreter', name, allowedDomains }`. The credentials are
  `{ authorization?, headers?, domainSecrets? }`, with each domain secret as `{ domain, name, value }`.
- The model Activity calls the function on every model call for each hosted MCP tool with a server label, shell tool
  with a `container_auto` domain allowlist, and code interpreter with an inline container allowlist, so cache secrets it
  fetches.
- It fills only fields the Workflow left out: `authorization` and `headers` for hosted MCP, and `domainSecrets` for shell
  and code interpreter.
- The model Activity's result records the model provider's full response, so a credential the provider echoes back is
  recorded in Event History. The plugin doesn't strip it, since redacting the response is the model provider's
  responsibility.
- A throw fails the model Activity, and its message is recorded in Event History, so keep credentials out of it.

Model Activity options, in the Client's `modelParams` or the runner's `defaultModelParams`:

| Option | Default | Behavior |
| --- | --- | --- |
| `ActivityOptions` fields, such as `startToCloseTimeout`, `scheduleToCloseTimeout`, `retry`, `taskQueue`, and `heartbeatTimeout` | `startToCloseTimeout: '60s'` | Apply to every model Activity. With a `heartbeatTimeout`, the model Activity heartbeats on a timer at half that interval, which detects a lost Worker but not a stalled call. |
| `useLocalActivity` | `false` | Runs model calls as [Local Activities](/local-activity). Only `startToCloseTimeout`, `scheduleToCloseTimeout`, `retry`, `cancellationType`, and `summary` apply then. |
| `summary` | None | The Activity summary shown in the Temporal UI. A string works anywhere. A `ModelSummaryProvider`, whose `provide(agent, instructions, input)` runs in the Workflow and must be deterministic, works only in `defaultModelParams`. |
| `streamingTopic` | None | The Workflow Stream topic for streaming runs. Required with `{ stream: true }`. Experimental. |
| `streamingBatchInterval` | The Workflow Streams default | How often the streaming Activity flushes published events. Experimental. |

`TemporalOpenAIRunner` options:

| API | Option | Default | Behavior |
| --- | --- | --- | --- |
| Constructor | `defaultModelParams` | None | Model Activity options set in Workflow code, which also reach Workflows started without the plugin, such as by a Schedule or the Temporal CLI. The Client's `modelParams` take precedence for each field they set. They don't reach `agentAsTool` nested runs, which use only the Client's `modelParams` and the plugin's defaults. |
| `run()` | `context`, `maxTurns`, `previousResponseId`, `conversationId`, `session`, `sessionInputCallback`, `callModelInputFilter`, `tracing` | SDK defaults | As in the SDK's run options. `session` can't be the SDK's `MemorySession`. Other SDK run options, such as `signal`, `errorHandlers`, `toolErrorFormatter`, and `toolExecution`, aren't accepted. |
| `run()` | `stream` | `false` | Streams model events over a Workflow Stream. Experimental. |
| `run()` | `runConfig.model` | Each agent's model | A model name used by every agent in the run. |
| `run()` | `runConfig.modelSettings`, `handoffInputFilter`, `inputGuardrails`, `outputGuardrails`, `tracingDisabled`, `traceIncludeSensitiveData`, `workflowName`, `traceId`, `groupId`, `traceMetadata` | SDK defaults | As in the SDK's run config. Guardrails and filters run in the Workflow. |
| `run()` | `runConfig.sandbox` | None | Required when a `SandboxAgent` is reachable from the agent through handoffs or an `agentAsTool` tool. `client` must come from `temporalSandboxClient()`. Experimental. See the notes below this table. |

Notes on `runConfig.sandbox`:

- It's passed on to `agentAsTool` nested runs.
- With a `SandboxAgent` reached only through an `agentAsTool` tool, a missing `runConfig.sandbox` on the parent run isn't
  caught up front. The nested run raises `SandboxConfigurationError` when the tool is called, which ends the parent run
  with an `AgentsWorkflowError` that carries it on `cause.error`.
- `client` is required even with `session`.
- The other fields, `options`, `manifest`, `snapshot`, `session`, `sessionState`, `concurrencyLimits`, and
  `archiveLimits`, go to the OpenAI Agents SDK unchanged, as described in
  [sandbox run options](https://openai.github.io/openai-agents-js/guides/sandbox-agents/concepts/#sandbox-run-options)
  in OpenAI's sandbox agents concepts guide.
- `options`, `manifest`, and `snapshot` go into the create Activity's input, so Event History records them.

Tool, MCP, session, and sandbox APIs:

| API | Option | Default | Behavior |
| --- | --- | --- | --- |
| `activityAsTool(definition, options)` | `name`, `description`, `parameters` | Required | `name` must match a registered Activity. `parameters` is a JSON schema object, and the type parameter doesn't check it against the Activity's input. |
| `activityAsTool` | `startToCloseTimeout`, `heartbeatTimeout`, `taskQueue`, `retryPolicy` | `startToCloseTimeout: '1 minute'` | Activity options for the tool call. |
| `activityAsTool`, `nexusOperationAsTool` | `strict` | `true` | Strict schema mode for the tool's arguments. The schema is sent as written, so with `strict: true`, list every property in `required` and set `additionalProperties: false`, or OpenAI's API rejects the request and the model Activity fails with `ModelInvocationError.BadRequest`. |
| `nexusOperationAsTool(operation, definition, options)` | `service`, `endpoint` | Required | The Nexus service definition and the Nexus Endpoint name. `definition` takes the same fields as `activityAsTool`'s. |
| `nexusOperationAsTool` | `scheduleToCloseTimeout` | None | The Operation's Schedule-To-Close Timeout. |
| `agentAsTool(agent, options)` | `toolName` | Required | The tool's name. The tool takes one string argument, `input`. The nested run gets only the parent's `runConfig.sandbox`. See the notes below this table. |
| `agentAsTool` | `toolDescription` | `''` | The tool's description. |
| `agentAsTool` | `customOutputExtractor` | The final output, as JSON if it isn't a string | Turns the nested run's result into the tool's output. |
| `statelessMcpServer(name, options)` | `startToCloseTimeout`, `heartbeatTimeout`, `taskQueue`, `retryPolicy` | `startToCloseTimeout: '1 minute'` | Activity options for listing and calling tools. |
| `statelessMcpServer`, `statefulMcpServer` | `cacheToolsList` | `true` | Reuses the server's tool list after the first listing. Set it to `false` if the tools can change during a run. |
| `statelessMcpServer`, `statefulMcpServer` | `factoryArgument` | None | Passed to the provider's factory on the Worker. |
| `statefulMcpServer` | `config` | `startToCloseTimeout: '1 minute'`, `scheduleToStartTimeout: '30 seconds'` | Activity options for tool operations, which also take `heartbeatTimeout` and `retryPolicy`. Operations always go to the run's dedicated Task Queue. |
| `statefulMcpServer` | `serverSessionConfig` | `startToCloseTimeout: '1 hour'` | Start-To-Close and Heartbeat Timeouts for the Activity that holds the server open. Raise the timeout for runs that keep the server longer. |
| `StatelessMCPServerProvider(name, factory)` | Arguments | Required | `factory` returns a new SDK MCP server for each operation. |
| `StatefulMCPServerProvider(name, factory, connection)` | Arguments | Required | `factory` returns a new server for each Workflow run. The dedicated Worker connects through `connection`, runs in the Worker process that ran the server session Activity, and handles one tool operation at a time, so parallel tool calls to one stateful server run one after another. |
| `WorkflowSafeMemorySession(options)` | `sessionId` | The Workflow ID | The session identifier. |
| `WorkflowSafeMemorySession` | `initialItems` | `[]` | History to start from, such as the previous run's items. |
| `SandboxClientProvider(name, client)` | Arguments | Required | Wraps an SDK sandbox client. Its Activities are named after `name`. |
| `temporalSandboxClient(name, options)` | `options` | `{ startToCloseTimeout: '5 minutes' }` | Activity options for every sandbox operation. They replace the default instead of merging with it. |
| `workerEnvValue(name)` | Arguments | Required | A manifest environment value that the Worker reads from its environment variable `name`. |

Notes on `agentAsTool` nested runs:

- The nested run is started with the parent's `runConfig.sandbox` and nothing else from the parent's `run()` call.
- It doesn't get the parent's `context`, the parent's other run options (`maxTurns`, `previousResponseId`,
  `conversationId`, `session`, `sessionInputCallback`, `callModelInputFilter`, and `tracing`), or any other `runConfig`
  field (`model`, `modelSettings`, `handoffInputFilter`, `inputGuardrails`, `outputGuardrails`, `tracingDisabled`,
  `traceIncludeSensitiveData`, `workflowName`, `traceId`, `groupId`, and `traceMetadata`). These all take their SDK
  defaults, and `agentAsTool` has no option to pass them.
- Its model Activities use the Client's `modelParams` but not the parent runner's `defaultModelParams`, so a field the
  Client didn't set takes the plugin's default, a 60-second Start-To-Close Timeout on a regular Activity.

Providers are matched to Workflow handles by name, and each provider on a plugin needs a unique name. The plugin adds a
`-stateful` suffix to a stateful MCP provider's name, so a stateless and a stateful MCP provider can share a name.

### Failure behavior

Which error reaches your Workflow depends on where the failure starts. A failed model call, MCP tool listing, or sandbox
session operation, such as creating or resuming the session, reaches the Workflow as an `ActivityFailure` with the
Activity's error as its cause, once its Activity stops retrying. A failed `activityAsTool`, `nexusOperationAsTool`, or
`agentAsTool` call ends the run with an `AgentsWorkflowError`, as described next. Failures that the table below lists as
raised in the Workflow reach it as they are when they occur outside a tool call.

A failed `activityAsTool` or `agentAsTool` call ends the run, and so does a failed `nexusOperationAsTool` call, which
builds its tool the same way. In a run without `stream: true`, the failure reaches the Workflow as a non-retryable
`AgentsWorkflowError` whose `cause` is the OpenAI Agents SDK's `ToolCallError`. The original failure is on that
`ToolCallError`'s `error` property, not on the cause chain, so read it from `cause.error`. It's the `ActivityFailure`
once an `activityAsTool` Activity stops retrying, the `NexusOperationFailure` from `nexusOperationAsTool`, or, from
`agentAsTool`, the nested run's own failure, such as a `NestedAgentInterruption`, `SandboxConfigurationError`, or
`AgentsWorkflowError`. The Workflow failure recorded in Event History keeps only the message text, so an
`activityAsTool` Activity's own error appears only in that Activity's failure event.

A failure inside an MCP tool call or a sandbox tool, such as `exec_command`, `write_stdin`, `view_image`, or
`apply_patch`, doesn't reach the Workflow. The OpenAI Agents SDK returns it to the model as the tool's output, such as
`An error occurred while running the tool. Please try again. Error: ActivityFailure: Activity task failed`, without the
underlying error, and the run continues. That includes the `DedicatedWorkerFailure`, `StatefulMCPServerNotFound`, and
`SandboxOperationUnsupported` errors below when an MCP or sandbox tool call raises them.

| Failure type | Raised in | Meaning |
| --- | --- | --- |
| `ModelInvocationError`, and `ModelInvocationError.RateLimit`, `.Authentication`, `.BadRequest`, `.Timeout`, `.Conflict`, or `.ServerError` | Model Activity | The model call failed. HTTP 408, 409, 429, and 5xx responses are retryable, and other statuses aren't. An `x-should-retry` header overrides that, and `retry-after-ms` or `retry-after` sets the next retry delay. An `openai` `APIError` without an HTTP status, such as `APIConnectionError`, is non-retryable. Other errors are retryable. |
| `AgentsWorkflowError` | Workflow | Non-retryable. An agent names its model with a model object instead of a string, an agent's `tools` holds a raw function, or the OpenAI Agents SDK threw during a run without `stream: true`, with its error as the `cause`. A failed `activityAsTool`, `nexusOperationAsTool`, or `agentAsTool` call in such a run also arrives this way, with the SDK's `ToolCallError` as the `cause` and the original failure on `cause.error`. In a streaming run, an error the SDK throws after `run()` returns isn't wrapped, and iterating the result or awaiting `result.completed` throws it as it is, so an uncaught one fails the Workflow Task instead of the Workflow. |
| `UnsafeSessionError` | Workflow | Non-retryable. `run()` received the SDK's `MemorySession`. |
| `StreamingTopicNotConfigured` | Workflow | Non-retryable. A streaming run has no `streamingTopic`. |
| `StreamingLocalActivityUnsupported` | Workflow | Non-retryable. A streaming run has `useLocalActivity: true`. |
| `NestedAgentInterruption` | Workflow | Non-retryable. An `agentAsTool` nested run paused for approval. It ends the parent run and reaches the Workflow on `cause.error` of an `AgentsWorkflowError`. |
| `DedicatedWorkerFailure`, exported as `DEDICATED_WORKER_FAILURE_TYPE` | Workflow | Retryable. A stateful MCP operation wasn't picked up within its Schedule-To-Start Timeout or missed a heartbeat, so the run's dedicated Worker isn't polling. |
| `StatefulMCPServerAlreadyRunning` | Server session Activity | Non-retryable. Two stateful handles with the same name connected in one Workflow run. |
| `StatefulMCPServerNotFound` | Dedicated Worker | Non-retryable. A tool operation reached a Worker that doesn't hold the server. |
| `SandboxConfigurationError` | Workflow | Non-retryable. A `SandboxAgent` is reachable but `runConfig.sandbox` is missing, or its `client` didn't come from `temporalSandboxClient()`. It's also raised when a manifest environment value uses a resolver function, directly or through `resolve`, or is an `EnvValueReference` that can't be serialized, since neither can travel in a sandbox Activity's input. Use `workerEnvValue()` for a value the Worker supplies. |
| `SandboxOperationUnsupported` | Sandbox Activity | Non-retryable. The backend lacks the operation. A Worker that doesn't hold the session calls `resume()` before handling one of its operations, and every paused-run resume calls it too, so on a backend without `resume()`, the `run()` call that resumes a paused sandbox run fails. |
| `WorkerEnvValueError` | Sandbox Activity | Non-retryable. A `workerEnvValue()` names a variable that isn't in `resolvableWorkerEnvVars`. |
| `SandboxSessionStateInvalid` | Workflow | Non-retryable. Sandbox session state being resumed wasn't produced by a Temporal sandbox client. |
| The sandbox error's `code` | Sandbox Activity | Non-retryable. The OpenAI Agents SDK marked the sandbox error as not retryable. Other sandbox errors are retried. |

The OpenAI Agents SDK parses a tool call's arguments before the tool runs, so `ToolSerializationError` isn't thrown
during a run. When the arguments for an `activityAsTool`, `nexusOperationAsTool`, or `agentAsTool` tool aren't valid
JSON, the run pauses as if the tool needed approval: `run()` returns with that call in `result.interruptions` and no
`finalOutput`. Approving it sends the model a parse error instead of running the tool.

Calling a stateful MCP handle's tools before `connect()` throws an `ApplicationFailure` that fails the Workflow. The
plugin constructor throws if two of its providers share a name, if `useOtelInstrumentation` is set without a
replay-safe global tracer provider, or if one `SandboxClientProvider` is passed to plugins with different
`resolvableWorkerEnvVars`.

**Sandbox session lifecycle.** Creating a session isn't idempotent, so a create Activity retried after the backend made
the session leaves that session behind. A session is also left running when a
[cancellation scope](/develop/typescript/workflows/cancellation-scopes) stops `run()`, since the runner skips the
teardown even if the Workflow then completes normally. Remove those sessions through your backend. The runner's teardown
calls the session's `stop()`, `shutdown()`, and `delete()`, each as an Activity, but only for a session it created or
resumed, so a session passed in `runConfig.sandbox.session` stays open after the run completes or fails.

**Sandbox reconnects.** Any Worker polling the Task Queue can receive a sandbox operation. A Worker that doesn't hold
the session, such as a restarted Worker, reconnects to it with the backend's `resume()`, and resuming a paused run
reconnects the same way, even on the Worker that ran it before the pause. What `resume()` restores depends on the
backend, as described in
[Resume and snapshots](https://openai.github.io/openai-agents-js/guides/sandbox-agents/clients/#resume-and-snapshots) in
OpenAI's sandbox clients guide.

With Unix-local and Docker, an operation that reaches a Worker on another machine fails, and Temporal retries it. A
Docker session keeps its workspace across a reconnect while its container is running. On both backends, resuming a
paused run keeps the workspace as it was at the pause.

**Sandbox processes.** If a command is still running when `exec_command` returns, the agent gets a process identifier
to pass to `write_stdin` later. On Unix-local and Docker, process identifiers start over after a reconnect, including
when a paused run resumes. A `write_stdin` with an earlier identifier either reports `session not found` or reaches a
different process that the agent started after the reconnect. A process started before a pause keeps running, on
Unix-local even after the session is torn down. `exec_command` returns when the command finishes or when
`yield_time_ms` passes, so tell the agent to raise it for slower commands, up to a limit below the sandbox Activity's
Start-To-Close Timeout.

## Resources

- [`@temporalio/openai-agents` package](https://www.npmjs.com/package/@temporalio/openai-agents)
- [OpenAI Agents integration source](https://github.com/temporalio/ai-integrations/tree/main/typescript/openai-agents)
- [OpenAI Agents samples directory](https://github.com/temporalio/samples-typescript/tree/main/openai-agents)
- [OpenAI Agents SDK documentation](https://openai.github.io/openai-agents-js/)
- [OpenAI Agents SDK source](https://github.com/openai/openai-agents-js)
- [Workflow Streams in the TypeScript SDK](/develop/typescript/workflows/workflow-streams)
- [Temporal Plugins guide](/develop/plugins-guide)

---

# Strands Agents integration - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/integrations/strands-agents

> Run Strands Agents AI Workflows with Durable Execution using the Temporal TypeScript SDK and Strands plugin.

Temporal's integration with [Strands Agents](https://strandsagents.com/) is an [SDK Plugin](/develop/plugins-guide) that
gives your Strands agents [Durable Execution](/temporal#durable-execution) via the Temporal platform. The plugin routes
model invocations, tool calls, MCP tool calls, and hooks through Temporal Activities, so every step your agent takes is
recorded in Workflow history and can survive crashes, restarts, and infrastructure failures.

> **ℹ️ Info:**
>
> The Temporal TypeScript SDK integration with Strands Agents is currently at an experimental release stage. The API may
> change in future versions.
>

Code snippets in this guide are taken from the
[Strands Agents plugin samples](https://github.com/temporalio/samples-typescript/tree/main/strands-agents). Refer to the
samples for the complete code.

## Get started

Install the plugin, then run a minimal Strands agent inside a Temporal Workflow.

### Prerequisites

- This guide assumes you are already familiar with Strands Agents. If you are not, refer to the
  [Strands Agents documentation](https://strandsagents.com/) for more details.
- If you are new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the
  [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
- Set up your local development environment by following the
  [Set up a local development environment with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript) guide. Leave the
  Temporal development server running if you want to test your code locally.

### Install the plugin

Install the Strands Agents plugin alongside the Strands Agents SDK:

```bash
npm install @temporalio/strands-agents @strands-agents/sdk
```

### Run a Strands agent with Durable Execution

The following example runs a Strands agent inside a Temporal Workflow. Model calls execute as Temporal Activities, which
means they get automatic retries, timeouts, and Durable Execution. If the Worker process crashes mid-conversation,
Temporal replays the Workflow and resumes from the last completed Activity.

**1. Define the Workflow**

Create a Workflow that constructs a `TemporalAgent` and invokes it with a prompt. The `startToCloseTimeout` in
`activityOptions` sets the maximum time each model call Activity can run:

<!--SNIPSTART typescript-strands-hello-world-workflow -->
[strands-agents/src/workflows/hello-world.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/hello-world.ts)
```ts
import { TemporalAgent } from '@temporalio/strands-agents';

export async function helloWorld(prompt: string): Promise<string> {
  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
  });
  const result = await agent.invoke(prompt);
  return result.toString();
}
```
<!--SNIPEND-->

**2. Start a Worker**

Create a Worker that registers your Workflows and the `StrandsPlugin`. The plugin automatically registers the Activities
that handle model calls. The same Worker serves every example in this guide; the `mcpClients` wiring is explained in
[Connect to MCP servers](#connect-to-mcp-servers):

<!--SNIPSTART typescript-strands-worker -->
[strands-agents/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/worker.ts)
```ts
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
import { McpClient } from '@strands-agents/sdk';
import { StrandsPlugin } from '@temporalio/strands-agents';
import { NativeConnection, Worker } from '@temporalio/worker';
import * as activities from './activities';

const ext = path.extname(fileURLToPath(import.meta.url));
const ECHO_SERVER = fileURLToPath(new URL(`./mcp-server${ext}`, import.meta.url));

function makeEchoClient(): McpClient {
  return new McpClient({
    transport: new StdioClientTransport({
      command: 'npx',
      args: ['tsx', ECHO_SERVER],
    }),
  });
}

async function run() {
  const connection = await NativeConnection.connect({
    address: process.env.TEMPORAL_ADDRESS ?? 'localhost:7233',
  });
  try {
    const worker = await Worker.create({
      connection,
      taskQueue: 'strands-agents',
      workflowsPath: fileURLToPath(new URL(`./workflows${ext}`, import.meta.url)),
      activities,
      // Omit `models:` so the plugin registers its default `BedrockModel` under
      // the name `bedrock`. To use a different provider or pin a model ID,
      // pass e.g. `models: { bedrock: () => new BedrockModel({ modelId: '...' }) }`.
      plugins: [new StrandsPlugin({ mcpClients: { echo: makeEchoClient } })],
    });
    console.log('Worker started. Ctrl+C to exit.');
    await worker.run();
  } finally {
    await connection.close();
  }
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```
<!--SNIPEND-->

**3. Run the Workflow**

Start the Workflow from a separate client script. This example sends the prompt "Write a haiku about durable execution"
and prints the agent's response:

<!--SNIPSTART typescript-strands-hello-world-client -->
[strands-agents/src/hello-world.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/hello-world.ts)
```ts
import { Client, Connection } from '@temporalio/client';
import { helloWorld } from './workflows';

async function run() {
  const connection = await Connection.connect({
    address: process.env.TEMPORAL_ADDRESS ?? 'localhost:7233',
  });
  const client = new Client({ connection });

  const result = await client.workflow.execute(helloWorld, {
    args: ['Write a haiku about durable execution.'],
    taskQueue: 'strands-agents',
    workflowId: 'strands-hello-world',
  });
  console.log(`Result: ${result}`);
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```
<!--SNIPEND-->

## Build the agent

Customize which model provider your agent uses, add tools that run as Activities, subscribe to lifecycle events with
hooks, and connect to MCP servers.

### Choose and configure models

`new StrandsPlugin({ models })` takes a mapping of `name` to factory function. Each factory is called lazily on first
use (on the Worker, outside the Workflow sandbox) and the constructed model is cached for the Worker's lifetime. If you
omit `models`, the plugin registers a single `BedrockModel` factory under the name `"bedrock"`, matching Strands' own
implicit default.

When you provide a custom `models` mapping, each `TemporalAgent` selects which factory to invoke by name with the
`model` option:

```ts
import { BedrockModel } from '@strands-agents/sdk/models/bedrock';
import { AnthropicModel } from '@strands-agents/sdk/models/anthropic';
import { TemporalAgent, StrandsPlugin } from '@temporalio/strands-agents';

// workflow
export async function multiModelWorkflow(prompt: string): Promise<string> {
  const a = new TemporalAgent({
    model: 'claude',
    activityOptions: { startToCloseTimeout: '60 seconds' },
  });
  const b = new TemporalAgent({
    model: 'bedrock',
    activityOptions: { startToCloseTimeout: '60 seconds' },
  });
  // ...
}

// worker
new StrandsPlugin({
  models: {
    claude: () => new AnthropicModel({ apiKey: '...' }),
    bedrock: () => new BedrockModel({}),
  },
});
```

Each `TemporalAgent` carries its own Activity options (timeouts, Retry Policy, Task Queue, streaming topic) and
dispatches to a shared model Activity, which resolves the model name against the registered factories at runtime. A
model name not present in the `models` mapping throws inside the Activity.

### Run non-deterministic tools as Activities

Strands tools that perform I/O, access external services, or produce non-deterministic results need to run as Temporal
Activities rather than inline in the Workflow. Register the tool as an Activity on the Worker, and pass it to the agent
using `workflow.activityAsTool`. Deterministic tools can run directly in the Workflow as a plain Strands `tool()`.

Define an Activity for the tool:

<!--SNIPSTART typescript-strands-tools-activity -->
[strands-agents/src/activities/tools.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/activities/tools.ts)
```ts
export async function fetchWeather(input: {
  city: string;
}): Promise<{ city: string; temperatureF: number; conditions: string }> {
  return {
    city: input.city,
    temperatureF: 72,
    conditions: 'sunny',
  };
}
```
<!--SNIPEND-->

Pass the Activity to the agent in the Workflow using `workflow.activityAsTool` (imported here as `strandsWorkflow`). The
`inputSchema` is a JSON Schema (or a Zod schema) that tells the model how to call the tool:

<!--SNIPSTART typescript-strands-tools-workflow -->
[strands-agents/src/workflows/tools.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/tools.ts)
```ts
import { tool } from '@strands-agents/sdk';
import { TemporalAgent, workflow as strandsWorkflow } from '@temporalio/strands-agents';
import { z } from 'zod';

const letterCounter = tool({
  name: 'letterCounter',
  description: 'Count how many times `letter` appears in `word` (case-insensitive).',
  inputSchema: z.object({
    word: z.string(),
    letter: z.string(),
  }),
  callback: ({ word, letter }) => word.toLowerCase().split(letter.toLowerCase()).length - 1,
});

export async function toolsWorkflow(prompt: string): Promise<string> {
  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    tools: [
      letterCounter,
      strandsWorkflow.activityAsTool('fetchWeather', {
        description: 'Fetch the current weather for a city.',
        inputSchema: {
          type: 'object',
          properties: { city: { type: 'string' } },
          required: ['city'],
        },
        activityOptions: { startToCloseTimeout: '30 seconds', retry: { maximumAttempts: 3 } },
      }),
    ],
  });
  const result = await agent.invoke(prompt);
  return result.toString();
}
```
<!--SNIPEND-->

Register the Activity functions on the Worker by passing them in the `activities` option, as shown in
[Start a Worker](#run-a-strands-agent-with-durable-execution). The `activityName` passed to `activityAsTool` must match
the name the Activity is registered under.

### React to agent lifecycle events

Strands' [hook system](https://strandsagents.com/) lets you subscribe callbacks to events in the agent lifecycle, such
as invocation start/end, model call before/after, tool call before/after, and message added. Use hooks to add logging,
metrics, or custom logic at each stage.

Register callbacks with `agent.addHook(EventClass, callback)`. Hook callbacks fire in Workflow context, so deterministic
callbacks work without any extra setup. For callbacks that need I/O (writing audit logs, metrics, alerting), use
`workflow.activityAsHook` to dispatch the work as a Temporal Activity. The following example shows both patterns. The
first callback mutates Workflow state (deterministic), while `persistToolCall` runs as an Activity (I/O-safe):

<!--SNIPSTART typescript-strands-hooks-workflow -->
[strands-agents/src/workflows/hooks.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/hooks.ts)
```ts
import { AfterToolCallEvent, tool } from '@strands-agents/sdk';
import { TemporalAgent, workflow as strandsWorkflow } from '@temporalio/strands-agents';
import { z } from 'zod';

const echo = tool({
  name: 'echo',
  description: 'Echo back the input text.',
  inputSchema: z.object({ text: z.string() }),
  callback: ({ text }) => text,
});

export async function hooksWorkflow(prompt: string): Promise<string[]> {
  const fired: string[] = [];

  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    tools: [echo],
  });

  // Callback 1: in-workflow, deterministic state mutation.
  agent.addHook(AfterToolCallEvent, (event) => {
    fired.push(event.toolUse.name);
  });

  // Callback 2: dispatch to a Temporal activity for I/O.
  agent.addHook(
    AfterToolCallEvent,
    strandsWorkflow.activityAsHook('persistToolCall', {
      activityInput: (event) => event.toolUse.name,
      activityOptions: { startToCloseTimeout: '15 seconds', retry: { maximumAttempts: 3 } },
    }),
  );

  await agent.invoke(prompt);
  return fired;
}
```
<!--SNIPEND-->

The Activity dispatched by the hook is a normal Temporal Activity registered on the Worker:

<!--SNIPSTART typescript-strands-hooks-activity -->
[strands-agents/src/activities/hooks.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/activities/hooks.ts)
```ts
import { log } from '@temporalio/activity';

export async function persistToolCall(toolName: string): Promise<void> {
  // In production, write to a database / S3 / your audit pipeline.
  log.info(`audit: tool ${toolName} completed`);
}
```
<!--SNIPEND-->

> **⚠️ Caution:**
>
> Hook callbacks run in Workflow context, so they must be
> [deterministic](/develop/typescript/workflows/basics#workflow-logic-requirements). Do not use `Date.now()`,
> `randomUUID()`, or I/O inside hook callbacks. Use `workflow.activityAsHook` for anything that requires I/O.
>

The `activityInput` function extracts serializable values from the event to pass as the Activity's input. This is needed
because hook events hold references to the `Agent`, `Tool` instances, and other objects that cannot cross the Activity
boundary.

### Connect to MCP servers

If your agent needs access to tools provided by an [MCP](https://modelcontextprotocol.io/) server, configure the MCP
clients on the Worker and reference them by name in the Workflow.

`new StrandsPlugin({ mcpClients })` takes a mapping of `name` to `McpClient` factory, mirroring the `models` pattern. The
plugin registers per-server `{name}-listTools` and `{name}-callTool` Activities. In the Workflow,
`new TemporalMCPClient({ server: 'name' })` is a thin handle that references the server by name and carries the per-call
Activity options.

Define the Workflow with a `TemporalMCPClient`:

<!--SNIPSTART typescript-strands-mcp-workflow -->
[strands-agents/src/workflows/mcp.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/mcp.ts)
```ts
import { TemporalAgent, TemporalMCPClient } from '@temporalio/strands-agents';

export async function mcpWorkflow(prompt: string): Promise<string> {
  const echo = new TemporalMCPClient({
    server: 'echo',
    activityOptions: { startToCloseTimeout: '30 seconds', retry: { maximumAttempts: 3 } },
  });
  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    tools: [echo],
  });
  const result = await agent.invoke(prompt);
  return result.toString();
}
```
<!--SNIPEND-->

Register the MCP client factory on the Worker via `mcpClients`, as shown in the
[Worker](#run-a-strands-agent-with-durable-execution) above. Each factory returns a fully configured `McpClient`, so you
can pass any options the `McpClient` constructor accepts (transport, URL, headers, and so on).

By default, `TemporalMCPClient` re-lists the server's tools on every agent turn, so an MCP server that is restarted or
redeployed mid-Workflow — with tools added, removed, or renamed — is picked up. To list the tools just once at the
beginning of the Workflow and reuse that schema for the Workflow's lifetime (one fewer Activity per turn), set
`cacheTools: true`:

```ts
const echo = new TemporalMCPClient({
  server: 'echo',
  cacheTools: true,
  activityOptions: { startToCloseTimeout: '30 seconds' },
});
```

To amortize connection setup, the `{name}-listTools` and `{name}-callTool` Activities share one Worker-process MCP
connection and reuse it across calls. The connection is disconnected after it sits idle for `mcpConnectionIdleTimeout`
(default 5 minutes); the Timer resets on every reuse. `mcpConnectionIdleTimeout` accepts a millisecond number or a
duration string (such as `'30 seconds'`), like `startToCloseTimeout`:

```ts
new StrandsPlugin({
  mcpClients: { echo: () => new McpClient({ url: 'http://localhost:8765/mcp' }) },
  mcpConnectionIdleTimeout: '30 seconds',
});
```

## Interact with the agent

Control the shape of agent responses, stream output in real time, and pause the agent for human approval.

### Add human approval gates

Some agent actions, such as deleting resources or sending messages, may require human approval before proceeding.
Strands offers two ways to interrupt an agent and wait for a response. Both work with the plugin.

In each case, `agent.invoke()` returns an `AgentResult` with `stopReason: 'interrupt'` and an `interrupts` array instead
of throwing. Pair this with a Signal handler that supplies responses, then resume by calling `agent.invoke(responses)`.

#### Interrupt from a hook

A hook on an interruptible event such as `BeforeToolCallEvent` can pause the agent by calling `event.interrupt(...)`.
The hook runs in Workflow context, so it must be deterministic. The Workflow waits for a Signal carrying the approval
response, then resumes the agent:

<!--SNIPSTART typescript-strands-human-in-the-loop-workflow -->
[strands-agents/src/workflows/human-in-the-loop.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/human-in-the-loop.ts)
```ts
import {
  BeforeToolCallEvent,
  tool,
  type InterruptResponseContent,
  type InterruptResponseContentData,
} from '@strands-agents/sdk';
import { TemporalAgent } from '@temporalio/strands-agents';
import { condition, defineQuery, defineSignal, setHandler } from '@temporalio/workflow';
import { z } from 'zod';

export const hitlApproveSignal = defineSignal<[string]>('hitlApprove');
export const hitlPendingApprovalQuery = defineQuery<string | null>('hitlPendingApproval');

const deleteFile = tool({
  name: 'deleteFile',
  description: 'Delete a file at the given path.',
  inputSchema: z.object({ path: z.string() }),
  callback: ({ path }) => `deleted ${path}`,
});

export async function humanInTheLoop(prompt: string): Promise<string> {
  let approval: string | null = null;
  let pendingReason: string | null = null;

  setHandler(hitlApproveSignal, (response) => {
    approval = response;
  });
  setHandler(hitlPendingApprovalQuery, () => pendingReason);

  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    tools: [deleteFile],
  });

  agent.addHook(BeforeToolCallEvent, (event) => {
    if (event.toolUse.name !== 'deleteFile') return;
    const path = (event.toolUse.input as { path?: string }).path;
    const response = event.interrupt<string>({
      name: 'approval',
      reason: `approve delete of ${path}?`,
    });
    if (response !== 'approve') {
      event.cancel = 'denied';
    }
  });

  let result = await agent.invoke(prompt);
  while (result.stopReason === 'interrupt') {
    const interrupts = result.interrupts ?? [];
    pendingReason = (interrupts[0]?.reason as string | undefined) ?? null;
    await condition(() => approval !== null);
    const response = approval!;
    approval = null;
    pendingReason = null;
    const responses: InterruptResponseContentData[] = interrupts.map((i) => ({
      type: 'interruptResponse',
      interruptResponse: { interruptId: i.id, response },
    }));
    result = await agent.invoke(responses as InterruptResponseContent[]);
  }
  return result.toString();
}
```
<!--SNIPEND-->

#### Interrupt from an Activity tool

An `activityAsTool`-wrapped Activity can interrupt the agent by throwing an interrupt-shaped `ApplicationFailure`. The
plugin's Failure Converter preserves the interrupt payload across the Activity boundary, so `AgentResult.interrupts` is
populated the same way as for hooks.

Define the Activity that raises the interrupt with the `STRANDS_INTERRUPT_TYPE` failure type:

<!--SNIPSTART typescript-strands-activity-interrupt-activity -->
[strands-agents/src/activities/activity-interrupt.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/activities/activity-interrupt.ts)
```ts
import { ApplicationFailure } from '@temporalio/common';
import { STRANDS_INTERRUPT_TYPE } from '@temporalio/strands-agents';

const APPROVED = new Set<string>();

export async function deleteThing(input: { name: string }): Promise<string> {
  if (!APPROVED.has(input.name)) {
    // First attempt: mark the name as approved on the way out (simulating the
    // human flipping a flag during the interrupt pause) and stop the agent by
    // raising an interrupt-shaped failure. The plugin's `StrandsFailureConverter`
    // would also recognize a thrown `{ interrupts: [{ toJSON: () => ... }] }`,
    // but throwing `ApplicationFailure` directly avoids any chance of the
    // converter being skipped (and keeps `nonRetryable: true` so the workflow
    // sees the interrupt instead of a retry-then-success).
    APPROVED.add(input.name);
    throw ApplicationFailure.create({
      message: 'interrupt:approval',
      type: STRANDS_INTERRUPT_TYPE,
      nonRetryable: true,
      details: [
        {
          id: `delete:${input.name}`,
          name: 'approval',
          reason: `approve delete of protected resource '${input.name}'?`,
          source: 'tool',
        },
      ],
    });
  }
  return `deleted ${input.name}`;
}
```
<!--SNIPEND-->

The Workflow resumes the agent the same way as for a hook interrupt:

<!--SNIPSTART typescript-strands-activity-interrupt-workflow -->
[strands-agents/src/workflows/activity-interrupt.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/activity-interrupt.ts)
```ts
import type { InterruptResponseContent, InterruptResponseContentData } from '@strands-agents/sdk';
import { TemporalAgent, workflow as strandsWorkflow } from '@temporalio/strands-agents';
import { condition, defineQuery, defineSignal, setHandler } from '@temporalio/workflow';

export const activityInterruptApproveSignal = defineSignal<[string]>('activityInterruptApprove');
export const activityInterruptPendingApprovalQuery = defineQuery<string | null>('activityInterruptPendingApproval');

export async function activityInterrupt(prompt: string): Promise<string> {
  let approval: string | null = null;
  let pendingReason: string | null = null;

  setHandler(activityInterruptApproveSignal, (response) => {
    approval = response;
  });
  setHandler(activityInterruptPendingApprovalQuery, () => pendingReason);

  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    tools: [
      strandsWorkflow.activityAsTool('deleteThing', {
        description: 'Delete a thing by name.',
        inputSchema: {
          type: 'object',
          properties: { name: { type: 'string' } },
          required: ['name'],
        },
        activityOptions: { startToCloseTimeout: '30 seconds', retry: { maximumAttempts: 3 } },
      }),
    ],
  });

  let result = await agent.invoke(prompt);
  while (result.stopReason === 'interrupt') {
    const interrupts = result.interrupts ?? [];
    pendingReason = (interrupts[0]?.reason as string | undefined) ?? null;
    await condition(() => approval !== null);
    const response = approval!;
    approval = null;
    pendingReason = null;
    const responses: InterruptResponseContentData[] = interrupts.map((i) => ({
      type: 'interruptResponse',
      interruptResponse: { interruptId: i.id, response },
    }));
    result = await agent.invoke(responses as InterruptResponseContent[]);
  }
  return result.toString();
}
```
<!--SNIPEND-->

> **⚠️ Caution:**
>
> Activity-tool interrupts rely on the plugin's Failure Converter, which is installed via the client's Data Converter.
> Attach `StrandsPlugin` to the **client** (not just the Worker) for Activity-tool interrupts to work. Workers built from
> that client pick up the plugin automatically.
>
> ```ts
> const client = new Client({ connection, plugins: [new StrandsPlugin({ models })] });
> ```
>

### Return structured data from an agent

To have the agent return a typed object instead of free-form text, pass a `structuredOutputSchema` (any Zod schema) to
`TemporalAgent`. The values flow through the model Activity unchanged, and the parsed object is available on
`result.structuredOutput`:

<!--SNIPSTART typescript-strands-structured-output-workflow -->
[strands-agents/src/workflows/structured-output.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/structured-output.ts)
```ts
import { TemporalAgent } from '@temporalio/strands-agents';
import { z } from 'zod';

export const PersonInfo = z.object({
  name: z.string().describe('Name of the person'),
  age: z.number().describe('Age of the person'),
  occupation: z.string().describe('Occupation of the person'),
});

export type PersonInfo = z.infer<typeof PersonInfo>;

export async function structuredOutputWorkflow(prompt: string): Promise<PersonInfo> {
  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    structuredOutputSchema: PersonInfo,
  });
  const result = await agent.invoke(prompt);
  return result.structuredOutput as PersonInfo;
}
```
<!--SNIPEND-->

### Stream agent output to clients

For long-running agent calls, you may want to forward model output chunks to an external consumer as they arrive rather
than waiting for the full response.

Pass `streamingTopic: '...'` to `TemporalAgent` and host a `WorkflowStream` on the Workflow via
[`@temporalio/workflow-streams`](https://github.com/temporalio/sdk-typescript/tree/main/packages/workflow-streams). Each
model stream event is published on the named topic from inside the model Activity. Subscribers read events through
`WorkflowStreamClient`. Chunks are batched on `streamingBatchInterval` (default `'100 milliseconds'`).

Define the Workflow with a `WorkflowStream` and a streaming topic:

<!--SNIPSTART typescript-strands-streaming-workflow -->
[strands-agents/src/workflows/streaming.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/streaming.ts)
```ts
import { TemporalAgent } from '@temporalio/strands-agents';
import { WorkflowStream } from '@temporalio/workflow-streams/workflow';

export async function streamingWorkflow(prompt: string): Promise<string> {
  // Constructing the stream installs the publish/poll handlers that
  // WorkflowStreamClient calls. Nothing in the workflow body reads from it.
  void new WorkflowStream();

  const agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    streamingTopic: 'events',
  });
  const result = await agent.invoke(prompt);
  return result.toString();
}
```
<!--SNIPEND-->

Subscribe to the stream from a client:

<!--SNIPSTART typescript-strands-streaming-client -->
[strands-agents/src/streaming.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/streaming.ts)
```ts
import { Client, Connection } from '@temporalio/client';
import { WorkflowStreamClient } from '@temporalio/workflow-streams/client';
import { streamingWorkflow } from './workflows';

interface StreamEvent {
  type?: string;
  delta?: { type?: string; text?: string };
}

async function run() {
  const connection = await Connection.connect({
    address: process.env.TEMPORAL_ADDRESS ?? 'localhost:7233',
  });
  const client = new Client({ connection });
  const workflowId = 'strands-streaming';

  const handle = await client.workflow.start(streamingWorkflow, {
    args: ['Count from 1 to 5, one number per sentence.'],
    taskQueue: 'strands-agents',
    workflowId,
  });

  const stream = WorkflowStreamClient.create(client, workflowId);
  const consume = (async () => {
    for await (const item of stream.subscribe<StreamEvent>(['events'], 0, {
      pollCooldown: '50 milliseconds',
      resultType: true,
    })) {
      const event = item.data;
      if (event.type === 'modelContentBlockDeltaEvent' && event.delta?.type === 'textDelta' && event.delta.text) {
        process.stdout.write(event.delta.text);
      } else if (event.type === 'modelMessageStopEvent') {
        process.stdout.write('\n');
        return;
      }
    }
  })();

  const result = await handle.result();
  await consume;
  console.log(`Final result: ${result}`);
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```
<!--SNIPEND-->

## Run in production

Configure retry policies, handle long-running chat sessions, and add distributed tracing.

### Configure retries

`TemporalAgent` disables Strands' built-in `ModelRetryStrategy` so that retries are handled exclusively by Temporal.
Configure retries with `activityOptions.retry` on `TemporalAgent` for model calls, and on the Activity options accepted
by `workflow.activityAsTool`, `workflow.activityAsHook`, and `TemporalMCPClient` for their respective calls:

```ts
new TemporalAgent({
  activityOptions: {
    startToCloseTimeout: '60 seconds',
    retry: { maximumAttempts: 3 },
  },
});
```

Passing `retryStrategy` to `new TemporalAgent(...)` throws. Remove the argument (or pass `retryStrategy: null`) and use
`activityOptions.retry` instead.

### Handle long-running chat sessions

A chat-style Workflow accumulates message history with every turn. Over a long session, the Workflow's Event History can
grow large enough to hit Temporal's per-Workflow history limit. To avoid this, use
[Continue-as-New](/develop/typescript/workflows/continue-as-new) to start a fresh Workflow execution while carrying the agent's
message history forward as input.

In this example, each user turn arrives as a Workflow [Update](/develop/typescript/workflows/message-passing#updates), so the
caller gets the agent's reply back from the same call. `workflowInfo().continueAsNewSuggested` flips to `true` once the
server decides history has grown large enough; the Workflow checks it after each turn and hands off to a fresh run,
carrying `agent.messages` as input:

<!--SNIPSTART typescript-strands-continue-as-new-workflow -->
[strands-agents/src/workflows/continue-as-new.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/continue-as-new.ts)
```ts
import type { Message } from '@strands-agents/sdk';
import { TemporalAgent } from '@temporalio/strands-agents';
import {
  allHandlersFinished,
  condition,
  continueAsNew,
  defineQuery,
  defineSignal,
  defineUpdate,
  setHandler,
  workflowInfo,
} from '@temporalio/workflow';

export interface ChatInput {
  messages?: Message[];
}

export const chatTurn = defineUpdate<string, [string]>('turn');
export const chatEnd = defineSignal('endChat');
export const chatMessages = defineQuery<Message[]>('messages');

export async function chatWorkflow(input: ChatInput = {}): Promise<void> {
  let done = false;
  let agent: TemporalAgent | null = null;
  // Serialize concurrent `turn` updates so they can't interleave on `agent.messages`.
  let pending: Promise<unknown> = Promise.resolve();

  setHandler(chatTurn, async (prompt) => {
    await condition(() => agent !== null);
    const prev = pending;
    let release!: () => void;
    pending = new Promise<void>((resolve) => {
      release = resolve;
    });
    try {
      await prev;
      const result = await agent!.invoke(prompt);
      return result.toString().trim();
    } finally {
      release();
    }
  });
  setHandler(chatEnd, () => {
    done = true;
  });
  setHandler(chatMessages, () => (agent ? [...agent.messages] : []));

  agent = new TemporalAgent({
    activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } },
    messages: input.messages ?? [],
  });

  await condition(() => done || workflowInfo().continueAsNewSuggested);
  // Drain in-flight `turn` updates before exiting or handing off.
  await condition(allHandlersFinished);

  if (!done) {
    await continueAsNew<typeof chatWorkflow>({ messages: agent.messages });
  }
}
```
<!--SNIPEND-->

### Add tracing with OpenTelemetry

To get distributed traces across model, tool, and MCP Activities, combine `StrandsPlugin` with the
[OpenTelemetry plugin](https://github.com/temporalio/sdk-typescript/tree/main/packages/interceptors-opentelemetry).
Register `OpenTelemetryPlugin` on both the client and the Worker. You get OpenTelemetry spans around the model, tool,
and MCP Activities the plugin schedules, plus any spans Strands itself emits inside `invoke`:

```ts
import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry';
import { Resource } from '@opentelemetry/resources';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { StrandsPlugin } from '@temporalio/strands-agents';

const otel = new OpenTelemetryPlugin({
  resource: new Resource({ 'service.name': 'strands-worker' }),
  spanProcessor: new SimpleSpanProcessor(new OTLPTraceExporter()),
});

// client
const client = new Client({ connection, plugins: [otel] });

// worker
const worker = await Worker.create({
  connection,
  taskQueue: 'strands-agents',
  workflowsPath: require.resolve('./workflows'),
  plugins: [otel, new StrandsPlugin({ models })],
});
```

### Snapshots are not supported

`TemporalAgent.takeSnapshot()` and `TemporalAgent.loadSnapshot()` throw. Temporal's Event History already persists
Workflow state durably at a finer granularity than Strands snapshots, so snapshots are redundant inside a Workflow.

### Samples

The [Strands Agents plugin samples](https://github.com/temporalio/samples-typescript/tree/main/strands-agents)
demonstrate all supported patterns end-to-end.

---

# Nexus - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/nexus

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Temporal Nexus

- [Quickstart](/develop/typescript/nexus/quickstart)
- [Feature guide](/develop/typescript/nexus/feature-guide)
- [Standalone Operations](/develop/typescript/nexus/standalone-operations)

---

# Nexus feature guide - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/nexus/feature-guide

> Use Temporal Nexus within the TypeScript SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **⚠️ Currently in: Pre-release:**
> This feature guide includes the new Nexus developer experience: pre-release APIs for the [Temporal Operation Handler](/nexus/temporal-operation-handler) and [Nexus Standalone Activity](/nexus/standalone-activity). These APIs are experimental and subject to change.

Use [Temporal Nexus](/evaluate/features/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations.

> **💡 Tip:**
>
> New to Nexus? Start with the [Nexus TypeScript quickstart](/develop/typescript/nexus/quickstart).
>

This page shows how to do the following:

- [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server)
- [Create caller and handler Namespaces](#create-caller-handler-namespaces)
- [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint)
- [Define the Nexus Service contract](#define-nexus-service-contract)
- [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers)
- [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service)
- [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server)
- [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud)

> **📝 Note:**
>
> This documentation uses source code derived from the [TypeScript Nexus sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-hello) and the [nexus-messaging-temporal-operation sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-messaging-temporal-operation).
>

## Run the Temporal Development Server with Nexus enabled 

Prerequisites:

- [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/typescript/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli)
  (`v1.9.1` or higher recommended)
- [Install the latest Temporal TypeScript SDK](https://learn.temporal.io/getting_started/typescript/dev_environment/#add-temporal-typescript-sdk-dependencies)
  (`v1.24.0` or higher recommended)

The first step in working with Temporal Nexus involves starting a Temporal Server with Nexus enabled.

```
temporal server start-dev \
  --dynamic-config-value history.enableUpdateCallbacks=true \
  --dynamic-config-value history.enableCHASMSignalBacklinks=true \
  --dynamic-config-value history.enableSignalWithStartFromWorkflow=true
```

This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases.

The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`.

## Create caller and handler Namespaces 

Before setting up Nexus endpoints, create separate Namespaces for the caller and handler.

```
temporal operator namespace create --namespace my-target-namespace
temporal operator namespace create --namespace my-caller-namespace
```

`my-target-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `my-caller-namespace` to
call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls.

## Create a Nexus Endpoint to route requests from caller to handler 

After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests.

```
temporal operator nexus endpoint create \
  --name my-nexus-endpoint-name \
  --target-namespace my-target-namespace \
  --target-task-queue my-handler-task-queue
```

You can also use the Web UI to create the Namespaces and Nexus endpoint.

## Define the Nexus Service contract 

Defining a clear contract for the Nexus Service is crucial for smooth communication.

In this example, there is a service module that describes the Service and Operation names along with input/output types
for caller Workflows to use the Nexus Endpoint.

You can hand-write that module, but the preferred way is to generate it with [NexGen](/nexus/nexgen).
You write the contract once as a JSON or YAML definition file and run `nexgen` against it, and it emits the typed models,
runtime validators, and the Service definition itself.

This is what makes a Nexus Service polyglot. Both sides generate from the same definition file: the handler implements
the Service, the caller invokes its Operations, and neither side hand-writes request or response types. A TypeScript
handler and a Go caller share no code, but they both run off that same service contract, so they interoperate with no
coordination between the teams beyond the contract itself.

The generated validators check every payload against the contract when a value is parsed off the wire and again when
it is serialized onto it, so bad data is rejected at the boundary rather than reaching your Workflow or Activity. A value validates
identically in every language, which is what lets a caller and a handler written in different languages trust the same
contract. See the [`chat.nexusrpc.yaml`](https://github.com/temporalio/nexgen/blob/main/samples/schemas/chat.nexusrpc.yaml)
sample contract and the [Definition files](https://github.com/temporalio/nexgen#definition-files) section of the
`nexgen` README for the file format.

## Develop a Nexus Service and Operation handlers 

The `nexus-rpc` [`serviceHandler`](https://nexus-rpc.github.io/sdk-typescript/functions/serviceHandler.html) function
defines a Nexus Service handler. It must provide an Operation handler for each Operation declared by the Service.
Nexus Service handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract.
Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous.
They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code.
Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database.
Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive
retryable errors, blocking all Operations from the caller to that Endpoint.

Every Operation is written with [`TemporalOperationHandler`](/nexus/temporal-operation-handler). Its `start` function
receives three things: a context, a Client, and the Operation input. What you do with the Client determines what backs
the Operation:

- **Synchronous.** Return `TemporalOperationResult.sync(...)` and the Operation completes during the handler call. The
  caller has its result as soon as the call returns.
- **Asynchronous.** Call `startWorkflow` or `startActivity` on the Client, or `update` on a handle from
  `getWorkflowHandle`. The handler returns as soon as that Execution has started, and the Operation stays open until the
  Execution finishes, which may be days later. Its result is delivered to the caller through the Nexus completion
  callback. This is what lets an Operation outlive the
  [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

A handler can perform any number of synchronous side effects, such as sending a Signal, but at most one asynchronous
backing per invocation.

### Develop a Synchronous Nexus Operation handler

Return a synchronous result when the Operation can answer immediately. The handler computes the answer and returns it,
and the Operation completes during the call.

Handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking), and the whole
call has to finish inside the [Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout).

```ts
import * as nexus from 'nexus-rpc';
import * as temporalNexus from '@temporalio/nexus';
import { helloService, EchoInput, EchoOutput } from '../api';

export const helloServiceHandler = nexus.serviceHandler(helloService, {
  echo: new temporalNexus.TemporalOperationHandler<EchoInput, EchoOutput>({
    start: async (ctx, client, input) => {
      return temporalNexus.TemporalOperationResult.sync({ message: input.message });
    },
  }),
});
```

### Use the Temporal Client for Signals, Queries, and Updates

A common pattern is to reach a Workflow that is already running. Query it or Signal it from a synchronous Operation, or
use `signalWithStartWorkflow` to make sure the Workflow exists before the Signal arrives. Those calls complete during the
handler call, so they have to finish inside the
[Nexus request timeout](/evaluate/cloud/limits#nexus-operation-request-timeout). The handler receives an `AbortSignal` on
`ctx.abortSignal` that fires when the deadline is exceeded. Pass it to Temporal Client calls so they are canceled if
the timeout is reached.

Updates are the exception. Do not wait for one inside the handler. Start it with `update` on a handle from
`getWorkflowHandle` and it backs the Operation. The handler returns immediately, and the Operation completes when the
Update does, however long it takes.

The [nexus-messaging](https://github.com/temporalio/samples-typescript/tree/main/nexus-messaging)
sample shows a Nexus Service that Queries and Signals a running Workflow from synchronous Operations, and backs an
Operation with a Workflow Update.

The Client your handler receives is not an ordinary Temporal Client. It propagates
[bidirectional links](/nexus/execution-debugging#bi-directional-linking) and request IDs on every call, so the
caller-side and handler-side Executions are connected in the UI without wiring anything. Reach the Workflow Client
through `client.client` rather than constructing your own.

In this example the Workflow ID is derived from an identifier carried in the Operation input, so the caller only needs
the identifier it cares about:

```ts
function workflowIdForUser(userId: string): string {
  return `GreetingWorkflow_for_${userId}`;
}

export const nexusGreetingServiceHandler = nexus.serviceHandler(nexusGreetingService, {
  getLanguages: new temporalNexus.TemporalOperationHandler({
    async start(_ctx, client, input: GetLanguagesInput) {
      const handle = client.client.workflow.getHandle(workflowIdForUser(input.userId));
      const result = await handle.query(getLanguagesQuery);
      return temporalNexus.TemporalOperationResult.sync(result);
    },
  }),
});
```

There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-typescript/tree/main/nexus-messaging/src/callerpattern) and [on-demand pattern](https://github.com/temporalio/samples-typescript/tree/main/nexus-messaging/src/ondemandpattern).
The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it.

### Develop an Asynchronous Nexus Operation handler to start a Workflow

Call `startWorkflow` on the Client. The Operation completes when the Workflow returns, and the Workflow's return value
is delivered to the caller as the Operation's result.

```ts
export const helloServiceHandler = nexus.serviceHandler(helloService, {
  hello: new temporalNexus.TemporalOperationHandler<HelloInput, HelloOutput>({
    start: async (ctx, client, input) =>
      client.startWorkflow(helloWorkflow, {
        args: [input],

        // Workflow IDs should typically be business-meaningful IDs and are used to dedupe workflow starts.
        workflowId: `hello-${input.name}-${input.language}`,

        // Task queue defaults to the task queue this Operation is handled on.
      }),
  }),
});
```

Workflow IDs should typically be business-meaningful identifiers and are used to dedupe Workflow starts.
In general, the Workflow ID should be passed in the Operation input as part of the Nexus Service contract.

> **💡 Tip:**
> RESOURCES
>
> [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing.
>

#### Map a Nexus Operation input to multiple Workflow arguments

A Nexus Operation can only take one input parameter. To start a Workflow that takes several, spread the pieces of the
input across the `args` array:

```ts
client.startWorkflow(helloWorkflow, {
  args: [input.name, input.language],
  workflowId: `hello-${input.name}-${input.language}`,
});
```

### Register a Nexus Service in a Worker

After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register your Nexus Service handler in a Worker.

<!--SNIPSTART typescript-nexus-hello-service-worker {"selectedLines": ["1-3","9-17"]}-->
[nexus-hello/src/service/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/service/worker.ts)
```ts
import { Worker, NativeConnection } from '@temporalio/worker';
import { helloServiceHandler } from './handler';

// ...
    const namespace = 'my-target-namespace';
    const serviceTaskQueue = 'my-handler-task-queue';
    const worker = await Worker.create({
      connection,
      namespace,
      taskQueue: serviceTaskQueue,
      workflowsPath: require.resolve('./workflows'),
      nexusServices: [helloServiceHandler],
    });
```
<!--SNIPEND-->

## Develop a caller Workflow that uses the Nexus Service 

To execute a Nexus Operation from a Workflow, import the necessary service definition types, then use `@temporalio/workflow`'s `createNexusServiceClient` to create a Nexus client for that service.
You will need to provide the Nexus Endpoint name, which you registered previously in [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint).

<!--SNIPSTART typescript-nexus-hello-service-caller-workflow {"selectedLines": ["1-5","21-34"]}-->

[nexus-hello/src/caller/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/caller/workflows.ts)

```ts
import * as wf from "@temporalio/workflow";
import { helloService, LanguageCode } from "../service/api";

const HELLO_SERVICE_ENDPOINT = "hello-service-endpoint-name";

export async function helloCallerWorkflow(name: string, language: LanguageCode): Promise<string> {
  const nexusClient = wf.createNexusServiceClient({
    service: helloService,
    endpoint: HELLO_SERVICE_ENDPOINT,
  });

  const helloResult = await nexusClient.executeOperation(
    "hello",
    { name, language },
    { scheduleToCloseTimeout: "10s" }
  );

  return helloResult.message;
}
```

<!--SNIPEND-->

### Register the caller Workflow in a Worker and start the caller Workflow

This Workflow can be registered with a Worker and started using `client.startWorkflow()` or `client.executeWorkflow()`, as usual.
Refer to the [complete TypeScript sample](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello) for reference.

- [nexus-hello/src/caller/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/caller/worker.ts) shows how to register the caller Workflow in a Worker and run the Worker.
- [nexus-hello/src/starter.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/starter.ts) shows how to use a Temporal Client to execute the sample caller Workflow.

## Make Nexus calls across Namespaces with a development Server 

Follow the steps below to run the Nexus handler Worker, the Nexus caller Worker, and the starter app.

1. Run `npm run start.service` to start the Worker that will be serving the Nexus Operation handlers and its associated
Workflows. That Worker connects to the `my-target-namespace` namespace.

2. In another shell, run `npm run start.caller` to start the Worker that will be serving the Caller Workflows. That
Worker connects to the `my-caller-namespace` namespace.

3. In a third shell, `npm run workflow` to start an instance of the caller Workflows.

Example output:

```bash
Echo message: This message is from the client
Hello message: Hello, Temporal!
```

### Canceling a Nexus Operation 

Nexus Operations, just like other cancellable APIs provided by the `@temporalio/workflow` package, execute within Cancellation Scopes.
Requesting cancellation of a Cancellation Scope results in requesting cancellation for all cancellable operations owned by that scope.
The Workflow itself defines the root Cancellation Scope.
Requesting cancellation of the Workflow therefore propagates the cancellation request to all cancellable operations started by that workflow, including Nexus Operations.

To provide more granular control over cancellation of a specific Nexus Operation, you may explicitly create a new Cancellation Scope, and start the Nexus Operation from within that scope.
An example demonstrating this can be found at our [nexus cancellation sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-cancellation).

Only asynchronous operations can be canceled in Nexus, since cancellation is sent using an operation token.
The Workflow or other resources backing the operation may choose to ignore the cancellation request.

Once the caller Workflow completes, the caller's Nexus Machinery stops attempting to cancel operations that have not yet
been canceled, letting them run to completion.

It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending
operations to finish before exiting the Workflow.

## Make Nexus calls across Namespaces in Temporal Cloud 

This section assumes you are already familiar with how to connect a Worker to Temporal Cloud.
The Temporal Cloud CLI is used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces.

### Install `tcld` and generate certificates

Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command
(on macOS):

```
brew install temporalio/brew/tcld
```

If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below:

```
tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key
```

These certificates will be valid for one year.

### Create caller and handler Namespaces

Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler.
If you already have these Namespaces, you don't need to do this.

**Temporal CLI**

```
temporal cloud login

temporal cloud namespace create \
  --name <your-caller-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

temporal cloud namespace create \
  --name <your-target-namespace> \
  --region aws-us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

**tcld**

```
tcld login

tcld namespace create \
  --namespace <your-caller-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1

tcld namespace create \
  --namespace <your-target-namespace> \
  --cloud-provider aws \
  --region us-west-2 \
  --ca-certificate-file 'path/to/your/ca.pem' \
  --retention-days 1
```

Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/namespaces](https://cloud.temporal.io/namespaces).

### Create a Nexus Endpoint to route requests from caller to handler

To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`.

**Temporal CLI**

```
temporal cloud nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file description.md
```

**tcld**

```
tcld nexus endpoint create \
  --name <my-nexus-endpoint-name> \
  --target-task-queue my-handler-task-queue \
  --target-namespace <my-target-namespace.account> \
  --allow-namespace <my-caller-namespace.account> \
  --description-file description.md
```

The `--allow-namespace` flag builds an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in [Runtime Access Controls](/nexus/security#runtime-access-controls).

Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus).

## Observability

### Web UI

A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history:

![Observability Sync](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-sync.png)

An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history:

![Observability Async](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-async.png)

### Temporal CLI

Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow:

```
temporal workflow describe -w <ID>
```

Nexus events are included in the caller's Event history:

```
temporal workflow show -w <ID>
```

For **asynchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationStarted`
- `NexusOperationCompleted`

For **synchronous Nexus Operations** the following are reported in the caller's history:

- `NexusOperationScheduled`
- `NexusOperationCompleted`

> **📝 Note:**
>
> `NexusOperationStarted` isn't reported in the caller's history for synchronous operations.
>

## Learn more

- Read the high-level description of the [Temporal Nexus feature](/evaluate/features/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082).
- Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus).
- Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus).

---

# Nexus TypeScript quickstart - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/nexus/quickstart

> Build a Nexus Service that wraps an existing Temporal Workflow using the TypeScript SDK

[Temporal Nexus](/evaluate/features/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow.

> **ℹ️ Info:**
> To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/features/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus).

**Prerequisites:** Complete the [TypeScript SDK Quickstart](/develop/typescript/set-up-your-local-typescript) first.
You should have `activities.ts`, `workflows.ts`, `worker.ts`, and `client.ts` from that guide.

## What you'll build

You have `example` running in the `default` Namespace.
By the end of this guide:

1. A Nexus Service will expose `example` as an Operation.
2. A second Namespace will contain a Workflow that calls that Operation.
3. The caller Workflow will get back `"Hello, Temporal!"` — the same result, but across Namespaces.

## 1. Define the Nexus Service

Create a file called `service.ts` that defines the Nexus Service contract.

Creating a Nexus Service establishes the contract between your implementation and any callers.
It provides type safety when invoking Nexus Operations and ensures that Operation Handlers fulfill the contract.

`nexus.service()` declares a named service, and `nexus.operation<I, O>()` defines a typed operation.
The `example` Workflow returns `string`, so the operation output type is `string`.

```typescript
import * as nexus from 'nexus-rpc';

export interface MyInput {
    name: string;
}

export const sayHelloService = nexus.service('say-hello', {
    sayHello: nexus.operation<MyInput, string>(),
});
```

## 2. Define the Nexus Operation handlers

Create a file called `handler.ts` that implements the Nexus Operation handler.

Operation handlers contain the logic that runs when a caller invokes a Nexus Operation.

`WorkflowRunOperationHandler` creates an asynchronous Nexus Operation that starts a Workflow.
The handler bridges the Nexus `MyInput` interface to the `example` Workflow's `string` parameter by extracting `input.name`.

```typescript
import { randomUUID } from 'crypto';
import * as nexus from 'nexus-rpc';
import * as temporalNexus from '@temporalio/nexus';
import { sayHelloService, MyInput } from './service';
import { example } from './workflows';

export const sayHelloHandler = nexus.serviceHandler(sayHelloService, {
    sayHello: new temporalNexus.WorkflowRunOperationHandler<MyInput, string>(
      async (ctx, input: MyInput) => {
        return await temporalNexus.startWorkflow(ctx, example, {
          args: [input.name],
          workflowId: "say-hello-nexus-" + randomUUID(),

          // Task queue defaults to the task queue this Operation is handled on.
        });
      },
    ),
});
```

## 3. Register the Nexus Service handler in a Worker

Update your existing `worker.ts` to register the Nexus Service Handler.

A Worker will only poll for and process incoming Nexus requests if the Nexus Service Handlers are registered.
This is the same Worker concept used for Workflows and Activities.

The `nexusServices` parameter registers the handler so it can receive Nexus Operation requests.

```typescript
import { NativeConnection, Worker } from '@temporalio/worker';
import * as activities from './activities';
import { sayHelloHandler } from './handler';

async function run() {
    const connection = await NativeConnection.connect({
      address: 'localhost:7233',
    });
    try {
      const worker = await Worker.create({
        connection,
        namespace: 'default',
        taskQueue: 'hello-world',
        workflowsPath: require.resolve('./workflows'),
        activities,
        nexusServices: [sayHelloHandler],
      });
      await worker.run();
    } finally {
      await connection.close();
    }
}

run().catch((err) => {
    console.error(err);
    process.exit(1);
});
```

## 4. Develop the caller Workflow

Update your existing `workflows.ts` file with a Workflow which invokes the Nexus Operation.

The caller Workflow demonstrates the consumer side of Nexus.
Instead of importing handler code directly, the caller only depends on the Service contract.
This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams.

The `wf.createNexusServiceClient()` method creates a client bound to your Nexus Service and Endpoint.
`executeOperation` starts the operation and waits for the result.

```typescript
import { proxyActivities, createNexusServiceClient } from '@temporalio/workflow';
// Only import the activity types
import type * as activities from './activities';
import { sayHelloService } from './service';

const { greet } = proxyActivities<typeof activities>({
    startToCloseTimeout: '1 minute',
});

/** A workflow that simply calls an activity */
export async function example(name: string): Promise<string> {
    return await greet(name);
}

const NEXUS_ENDPOINT = 'my-nexus-endpoint-name';

export async function callerWorkflow(name: string): Promise<string> {
    const nexusClient = createNexusServiceClient({
      service: sayHelloService,
      endpoint: NEXUS_ENDPOINT,
    });
    return await nexusClient.executeOperation('sayHello', { name }, { scheduleToCloseTimeout: '10s' });
}

```

## 5. Create the caller Namespace and Nexus Endpoint

Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler.
The handler uses the `default` Namespace that was created when you started the dev server.

Namespaces provide isolation between the caller and handler sides.
The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue.
The endpoint name must match the variable defined in `caller.ts` from step 4.

Make sure your local Temporal dev server is running (`temporal server start-dev`).

```bash
temporal operator namespace create --namespace my-caller-namespace
```

```bash
temporal operator nexus endpoint create \\
  --name my-nexus-endpoint-name \\
  --target-namespace default \\
  --target-task-queue hello-world
```

## 6. Run and verify

Create a file called `caller-starter.ts` to start the caller Worker and execute the Workflow.

This step brings everything together: the caller Worker hosts `callerWorkflow`, which uses the Nexus client to invoke `sayHello` on the handler side.
The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running the `example` Workflow, and back to the caller.

**Run the application:**

1. Start the handler Worker in one terminal:

```bash
npx ts-node src/worker.ts
```

2. Run the caller in another terminal:

```bash
npx ts-node src/caller-starter.ts
```

You should see:

```
Workflow result: Hello, Temporal!
```

Open the [Temporal Web UI](http://localhost:8233) and find the `callerWorkflow` execution.
You should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in the Event history.

```typescript
import { randomUUID } from 'crypto';
import { Connection, Client } from '@temporalio/client';
import { NativeConnection, Worker } from '@temporalio/worker';
import { callerWorkflow } from './workflows';

const CALLER_TASK_QUEUE = 'my-caller-task-queue';
const NAMESPACE = 'my-caller-namespace';

async function main() {
    const clientConnection = await Connection.connect({
      address: 'localhost:7233',
    });
    const client = new Client({
      connection: clientConnection,
      namespace: NAMESPACE,
    });

    const workerConnection = await NativeConnection.connect({
      address: 'localhost:7233',
    });
    try {
      const worker = await Worker.create({
        connection: workerConnection,
        namespace: NAMESPACE,
        taskQueue: CALLER_TASK_QUEUE,
        workflowsPath: require.resolve('./workflows'),
      });

      await worker.runUntil(async () => {
        const result = await client.workflow.execute(callerWorkflow, {
          args: ['Temporal'],
          workflowId: \`caller-workflow-\${randomUUID()}\`,
          taskQueue: CALLER_TASK_QUEUE,
        });
        console.log('Workflow result:', result);
      });
    } finally {
      await workerConnection.close();
    }
}

main().catch((err) => {
    console.error(err);
    process.exit(1);
});
```

## Next steps

Now that you have a working Nexus Service, here are some resources to deepen your understanding:

- **[TypeScript Nexus feature guide](/develop/typescript/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls.
- **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics.
- **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker.
- **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns.
- **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries.
- **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls.
- **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests.
- **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity.

---

# Standalone Nexus Operations - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/nexus/standalone-operations

> Execute Nexus Operations independently without a Workflow using the Temporal TypeScript SDK.

> **Public Preview**
> Requires TypeScript SDK `v1.20.2` or above. API breaking changes are kept to a minimum before General Availability.

[Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without
being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using
`@temporalio/workflow`'s `createNexusServiceClient()`, you execute a Standalone Nexus Operation directly from a Nexus
service client created on the Temporal Client using `client.nexus.createServiceClient()`.

Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as
Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/typescript/nexus/feature-guide) for details on
[defining a Service contract](/develop/typescript/nexus/feature-guide#define-nexus-service-contract) and
[developing Operation handlers and registering a Service in a Worker](/develop/typescript/nexus/feature-guide#develop-nexus-service-operation-handlers).

This page focuses on the client-side APIs that are unique to Standalone Nexus Operations:

- [Execute a Standalone Nexus Operation](#execute-operation)
- [Start a Standalone Nexus Operation and Wait for the Result](#get-operation-result)
- [List Standalone Nexus Operations](#list-operations)
- [Count Standalone Nexus Operations](#count-operations)
- [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud)

> **📝 Note:**
> This documentation uses source code from the
> [TypeScript Nexus Standalone sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-standalone-operations).
>

## Prerequisites 

Standalone Nexus Operations are a Public Preview feature and require Temporal CLI v1.9.0 or higher.

### 1. Install and verify the Temporal CLI 

The `temporal nexus operation` commands require Temporal CLI v1.9.0 or higher. See
[Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for install options,
then verify:

```bash
temporal --version
```

Which should output v1.9.0 or higher.

### 2. Start a local dev server 

The dev server enables Standalone Nexus Operations by default — no dynamic config is
required. Start it with the caller and handler Namespaces pre-created:

```bash
temporal server start-dev \
  --namespace my-caller-namespace \
  --namespace my-handler-namespace
```

The starter and Worker connect to two different Namespaces (a caller Namespace and a handler
Namespace), mirroring how Nexus crosses Namespace boundaries.

To run the examples on this page against the [TypeScript sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-standalone-operations),
create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue:

```bash
temporal operator nexus endpoint create \
  --name my-nexus-endpoint \
  --target-namespace my-handler-namespace \
  --target-task-queue nexus-handler-queue
```

Start the sample Worker in the handler Namespace:

```bash
TEMPORAL_NAMESPACE=my-handler-namespace npm run worker
```

Run the starter in the caller Namespace (from a separate terminal):

```bash
TEMPORAL_NAMESPACE=my-caller-namespace npm run starter
```

## Execute a Standalone Nexus Operation 

To execute a Standalone Nexus Operation, first create a [`NexusServiceClient`](https://typescript.temporal.io/api/interfaces/client.NexusServiceClient) using [`client.nexus.createServiceClient()`](https://typescript.temporal.io/api/classes/client.NexusClient#createserviceclient),
bound to a specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call
`startOperation()` or `executeOperation()` from application code (for example, a starter program), not from inside a
Workflow Definition.

`executeOperation` waits for the Operation to complete and returns the result.
Both methods require `id`. `scheduleToCloseTimeout` is optional and defaults to the maximum allowed by the Temporal server.

```ts
const nexusClient = client.nexus.createServiceClient({
  endpoint: ENDPOINT_NAME,
  service: myNexusService,
});

// Await the result of the operation immediately.
const echoResult = await nexusClient.executeOperation(
  myNexusService.operations.echo,
  { message: 'hello' },
  {
    id: `echo-${nanoid()}`,
    scheduleToCloseTimeout: '10s',
  },
);
```

See the full
[starter sample](https://github.com/temporalio/samples-typescript/blob/main/nexus-standalone-operations/src/starter.ts)
for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and
counts Operations.

Or use the Temporal CLI to execute a Standalone Nexus Operation:

```bash
temporal nexus operation execute \
  --namespace my-caller-namespace \
  --endpoint my-nexus-endpoint \
  --service myNexusService \
  --operation echo \
  --operation-id my-echo-op \
  --input '{"message":"hello"}'
```

## Start a Standalone Nexus Operation and Wait for the Result 

`startOperation` returns a [`NexusOperationHandle`](https://typescript.temporal.io/api/interfaces/client.NexusOperationHandle).
Use `NexusOperationHandle.result()` to wait until the Operation completes and retrieve its result. This works for both
synchronous and asynchronous Operations.

```ts
// Start an operation and get a NexusOperationHandle
const handle = await nexusClient.startOperation(
  myNexusService.operations.hello,
  { name: 'World' },
  {
    id: `hello-${nanoid()}`,
    scheduleToCloseTimeout: '10s',
  },
);

// Await the result
const helloResult = await handle.result();
console.log(helloResult.greeting);
```

If the Operation completed successfully, the result is returned. If the Operation failed, the failure is thrown as an error.

Or use the Temporal CLI to wait for a result by Operation ID:

```bash
temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op
```

## List Standalone Nexus Operations 

Use [`client.nexus.list()`](https://typescript.temporal.io/api/classes/client.NexusClient#list) to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query.
The result is an async iterator that yields operation metadata entries.

Note that `client.nexus.list()` is called on the base `Client`, not on the `NexusServiceClient`.

```ts
const query = `Endpoint = "${ENDPOINT_NAME}"`;
for await (const op of client.nexus.list({ query })) {
  console.log(
    `OperationId: ${op.operationId},`,
    `Operation: ${op.operation},`,
    `Status: ${op.status}`,
  );
}
```

The `query` parameter accepts [List Filter](/list-filter) syntax. For example,
`"Endpoint = 'my-endpoint' AND ExecutionStatus = 'Running'"`.

Or use the Temporal CLI:

```bash
temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Count Standalone Nexus Operations 

Use [`client.nexus.count()`](https://typescript.temporal.io/api/classes/client.NexusClient#count) to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query.

Note that `client.nexus.count()` is called on the base `Client`, not on the Nexus service client.

```ts
const query = `Endpoint = "${ENDPOINT_NAME}"`;
const count = await client.nexus.count(query);
console.log(`Total Nexus operations: ${count.count}`);
```

Or use the Temporal CLI:

```bash
temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"'
```

## Run Standalone Nexus Operations with Temporal Cloud 

The code samples referenced on this page use [`loadClientConnectConfig()`](https://typescript.temporal.io/api/namespaces/envconfig#loadclientconfig) from `@temporalio/envconfig`, so the same code
works against Temporal Cloud — just configure the connection via environment variables or a TOML
profile. No code changes are needed.

For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint
setup, certificate generation, and authentication options, see
[Make Nexus calls across Namespaces in Temporal Cloud](/develop/typescript/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud)
and [Connect to Temporal Cloud](/develop/typescript/client/temporal-client#connect-to-temporal-cloud).

---

# Platform - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/platform

> This section explains how to implement platform with the TypeScript SDK

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Platform

- [Observability](/develop/typescript/platform/observability)
- [Enriching the UI](/develop/typescript/platform/enriching-ui)

---

# Enriching the user interface - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/platform/enriching-ui

> Add contextual information to workflows and events in the Temporal UI using the TypeScript SDK.

Temporal supports adding context to Workflows and Events with metadata. 
This helps users identify and understand Workflows and their operations.

## Adding summary and details to Workflows

### Starting a Workflow

When starting a Workflow, you can provide a static summary and details to help identify the workflow in the UI:

```typescript
import { Client } from '@temporalio/client';

const client = new Client();

// Start a workflow with static summary and details
const handle = await client.workflow.start(yourWorkflow, {
  args: ['workflow input'],
  taskQueue: 'your-task-queue',
  workflowId: 'your-workflow-id',
  staticSummary: 'Order processing for customer #12345',
  staticDetails: 'Processing premium order with expedited shipping'
});
```

`staticSummary` is a single-line description that appears in the workflow list view, limited to 200 bytes.
`staticDetails` can be multi-line and provides more comprehensive information that appears in the workflow details view, with a larger limit of 20K bytes.

The input format is standard Markdown excluding images, HTML, and scripts.

You can also use the `execute` method with the same parameters:

```typescript
const result = await client.workflow.execute(yourWorkflow, {
  args: ['workflow input'],
  taskQueue: 'your-task-queue',
  workflowId: 'your-workflow-id',
  staticSummary: 'Order processing for customer #12345',
  staticDetails: 'Processing premium order with expedited shipping'
});
```

### Inside the Workflow

Within a Workflow, you can get and set the _current workflow details_. 
Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. 
Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines.

```typescript
import { getCurrentDetails, setCurrentDetails } from '@temporalio/workflow';

export async function yourWorkflow(input: string): Promise<string> {
  // Get the current details
  const currentDetails = getCurrentDetails();
  console.log(`Current details: ${currentDetails}`);
  
  // Set/update the current details
  setCurrentDetails('Updated workflow details with new status');
  
  return 'Workflow completed';
}
```

### Adding Summary to Activities and Timers

You can attach a `summary` to activities by using `executeWithOptions` when calling them:

```typescript
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { yourActivity } = proxyActivities<typeof activities>({
  startToCloseTimeout: '10 seconds'
});

export async function yourWorkflow(input: string): Promise<string> {
  // Execute an activity with a summary using executeWithOptions
  const result = await yourActivity.executeWithOptions(
    {
      staticSummary: 'Processing user data'
    },
    [input] // Note: arguments must be passed as an array
  );
  
  return result;
}
```

Similarly, you can attach a `summary` to timers within a workflow:

```typescript
import { sleep } from '@temporalio/workflow';

export async function yourWorkflow(input: string): Promise<string> {
  // Create a timer with a summary
  await sleep('5 minutes', { summary: 'Waiting for payment confirmation' });
  
  return 'Timer completed';
}
```

The input format for `summary` is a string, and limited to 200 bytes.

## Viewing summary and details in the UI

Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI.
Navigate to your Workflow's details page to see the metadata displayed in three key locations:

### User Metadata

On the Workflow Details page, under the **User Metadata** tab, you'll find the workflow-level metadata:

- **Summary & Details** - Displays the static summary and static details set when starting the workflow
- **Current Details** - Displays the dynamic details that can be updated during workflow execution

All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays.

### Timeline

The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance.

Labels longer than 120 characters are truncated with an ellipsis.

Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. 

Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later.

### Event History

Individual events in the Workflow's Event History display their associated summaries when available. 

Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. 
When you do expand an Event, the summary is also prominently displayed in the detailed view.

---

# Observability - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/platform/observability

> Enhance the observability of your Temporal Application with metrics, tracing, logging, and visibility features. View Workflow state, set up OpenTelemetry, and customize logging to monitor your Workflows and Activities.

The observability section of the TypeScript developer guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

This section covers features related to viewing the state of the application, including:

- [Emit metrics](#metrics)
- [Set up tracing](#tracing)
- [Log from a Workflow](#logging)
- [Visibility APIs](#visibility)

## Emit metrics 

Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process.
For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics).

- For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide.
- For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics).
- For an end-to-end example that exposes metrics with the TypeScript SDK, refer to the [samples-typescript](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) repo.

Workers can emit metrics and traces. There are a few [telemetry options](https://typescript.temporal.io/api/interfaces/worker.TelemetryOptions) that can be provided to [`Runtime.install`](https://typescript.temporal.io/api/classes/worker.Runtime/#install). The common options are:

- `metrics: { otel: { url } }`: The URL of a gRPC [OpenTelemetry collector](https://opentelemetry.io/docs/collector/).
- `metrics: { prometheus: { bindAddress } }`: Address on the Worker host that will have metrics for [Prometheus](https://prometheus.io/) to scrape.

To set up tracing of Workflows and Activities, use our `opentelemetry-interceptors` package.
(For details, see the next section.)

```typescript
telemetryOptions: {
  metrics: {
    prometheus: { bindAddress: '0.0.0.0:9464' },
  },
  logging: { forward: { level: 'DEBUG' } },
},
```

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Set [`globalTags`](https://typescript.temporal.io/api/namespaces/worker#metricsexporterconfig) on the metrics telemetry options passed to [`Runtime.install`](https://typescript.temporal.io/api/classes/worker.Runtime/#install).

```typescript
Runtime.install({
  telemetryOptions: {
    metrics: {
      prometheus: { bindAddress: '0.0.0.0:9464' },
      globalTags: {
        team: 'content-platform',
        service: 'checkout',
        cost_center: 'cc-1042',
        environment: 'production',
      },
    },
  },
});
```

Global tags reach the Prometheus and OpenTelemetry exporters.
A buffered metrics exporter does not receive them.

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag           | Example            | Question it answers                                       |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team`        | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service`     | `checkout`         | Which application emits these metrics?                    |
| `cost_center` | `cc-1042`          | Which budget does this Worker fleet belong to?            |
| `environment` | `production`       | Is this production traffic, or staging or test?           |
| `region`      | `us-east-2`        | Where does the Worker fleet run?                          |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Set up tracing 

Tracing allows you to view the call graph of a Workflow along with its Activities and any Child Workflows.

Temporal Web's tracing capabilities mainly track Activity Execution within a Temporal context. If you need custom tracing specific for your use case, you should make use of context propagation to add tracing logic accordingly.

The [`interceptors-opentelemetry`](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) sample shows how to use the SDK's built-in OpenTelemetry tracing to trace everything from starting a Workflow to Workflow Execution to running an Activity from that Workflow.

The built-in tracing uses protobuf message headers (like [this one](https://github.com/temporalio/api/blob/70cec27214ec2173974fb045667df324b45e705c/temporal/api/workflowservice/v1/request_response.proto#L180) when starting a Workflow) to propagate the tracing information from the client to the Workflow and from the Workflow to its successors (when Continued As New), children, and Activities.
All of these executions are linked with a single trace identifier and have the proper `parent -> child` span relation.

Tracing is compatible between different Temporal SDKs as long as compatible [context propagators](https://opentelemetry.io/docs/concepts/context-propagation/) are used.

**Context propagation**

The TypeScript SDK uses the global OpenTelemetry propagator.

To extend the default ([Trace Context](https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-core/README.md#w3ctracecontextpropagator-propagator) and [Baggage](https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-core/README.md#baggage-propagator) propagators) to also include the [Jaeger propagator](https://www.npmjs.com/package/@opentelemetry/propagator-jaeger), follow these steps:

- `npm i @opentelemetry/propagator-jaeger`

- At the top level of your Workflow code, add the following lines:

  ```js
  import { propagation } from '@opentelemetry/api';
  import {
    CompositePropagator,
    W3CBaggagePropagator,
    W3CTraceContextPropagator,
  } from '@opentelemetry/core';
  import { JaegerPropagator } from '@opentelemetry/propagator-jaeger';

  propagation.setGlobalPropagator(
    new CompositePropagator({
      propagators: [
        new W3CTraceContextPropagator(),
        new W3CBaggagePropagator(),
        new JaegerPropagator(),
      ],
    }),
  );
  ```

Similarly, you can customize the OpenTelemetry `NodeSDK` propagators by following the instructions in the [Initialize the SDK](https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-sdk-node#initialize-the-sdk) section of the `README.md` file.

## Set up logging 

Logging enables you to record critical information during code execution.
Loggers create an audit trail and capture information about your Workflow's operation.
An appropriate logging level depends on your specific needs.
During development or troubleshooting, you might use debug or even trace.
In production, you might use info or warn to avoid excessive log volume.

You can find the log levels supported by the `winston` package used in the examples below in [their official documentation](https://github.com/winstonjs/winston#logging-levels). The Temporal SDK core normally uses `WARN` as its default logging level.

### Logging from Activities

Activities run in the standard Node.js environment and may therefore use any Node.js logger directly.

The Temporal SDK however provides a convenient Activity Context logger, which funnels log messages to the [Runtime's logger](/develop/typescript/platform/observability#customizing-the-default-logger). Attributes from the current Activity context are automatically included as metadata on every log entry emitted using the Activity context logger, and some key events of the Activity's lifecycle are automatically logged (at DEBUG level for most messages; WARN for failures).

Using the Activity Context logger

```ts
import { log } from '@temporalio/activity';

export async function greet(name: string): Promise<string> {
  log.info('Log from activity', { name });
  return `Hello, ${name}!`;
}
```

### Logging from Workflows

Workflows may not use regular Node.js loggers because:

1. Workflows run in a sandboxed environment and cannot do any I/O.
1. Workflow code might get replayed at any time, which would result in duplicated log messages.

The Temporal SDK however provides a Workflow Context logger, which funnels log messages to the [Runtime's logger](/develop/typescript/platform/observability#customizing-the-default-logger). Attributes from the current Workflow context are automatically included as metadata on every log entry emitted using the Workflow context logger, and some key events of the Workflow's lifecycle are automatically logged (at DEBUG level for most messages; WARN for failures).

Using the Workflow Context logger

```ts
import { log } from '@temporalio/workflow';

export async function myWorkflow(name: string): Promise<string> {
  log.info('Log from workflow', { name });
  return `Hello, ${name}!`;
}
```

The Workflow Context Logger tries to avoid reemitting log messages on Workflow Replays.

#### Limitations of Workflow logs

Internally, Workflow logging uses Sinks, and is consequently subject to the same limitations as Sinks.
Notably, logged objects must be serializable using the V8 serialization.

### What is the Runtime's Logger

A Temporal Worker may emit logs in various ways, including:

- Messages emitted using the [Workflow Context Logger](#logging);
- Messages emitted using the [Activity Context Logger](#logging-from-activities);
- Messages emitted by the TypeScript SDK Worker itself;
- Messages emitted by the underlying Temporal Core SDK (native code).

All of these messages are internally routed to a single logger object, called the Runtime's Logger.
By default, the Runtime's Logger simply writes messages to the console (that is, the process's `STDOUT`).

#### How to customize the Runtime's Logger

A custom Runtime Logger may be registered when the SDK `Runtime` is instantiated. This is done only once per process.

To register a custom Runtime Logger, you must explicitly instantiate the Runtime, using the [`Runtime.install()`](https://typescript.temporal.io/api/classes/worker.Runtime/#install) function.
For example:

```typescript
import {
  DefaultLogger,
  makeTelemetryFilterString,
  Runtime,
} from '@temporalio/worker';

// This is your custom Logger.
const logger = new DefaultLogger('WARN', ({ level, message }) => {
  console.log(`Custom logger: ${level} — ${message}`);
});

Runtime.install({
  logger,
  // The following block is optional, but generally desired.
  // It allows capturing log messages emitted by the underlying Temporal Core SDK (native code).
  // The Telemetry Filter String determine the desired verboseness of messages emitted by the
  // Temporal Core SDK itself ("core"), and by other native libraries ("other").
  telemetryOptions: {
    logging: {
      filter: makeTelemetryFilterString({ core: 'INFO', other: 'INFO' }),
      forward: {},
    },
  },
});
```

A common use case for this is to write log messages to a file to be picked up by a collector service, such as the [Datadog Agent](https://docs.datadoghq.com/logs/log_collection/nodejs/?tab=winston30).
For example:

```typescript
import {
  DefaultLogger,
  makeTelemetryFilterString,
  Runtime,
} from '@temporalio/worker';
import winston, { transports } from 'winston';

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  transports: [new transports.File({ filename: '/path/to/worker.log' })],
});

Runtime.install({
  logger: new DefaultLogger('INFO', (entry) => {
    logger.log({
      label: entry.meta?.activityId ? 'activity' : entry.meta?.workflowId ? 'workflow' : 'worker',
      level: entry.level.toLowerCase(),
      message: entry.message,
      timestamp: Number(entry.timestampNanos / 1_000_000n),
      ...entry.meta,
    });
  }),
  // The following block is optional, but generally desired.
  // It allows capturing log messages emitted by the underlying Temporal Core SDK (native code).
  // The Telemetry Filter String determine the desired verboseness of messages emitted by the
  // Temporal Core SDK itself ("core"), and by other native libraries ("other").
  telemetryOptions: {
    logging: {
      filter: makeTelemetryFilterString({ core: 'INFO', other: 'INFO' }),
      forward: {},
    },
  },
});
```

### How to provide a custom logger 

#### Logging in Workers and Clients

The Worker comes with a default logger, which defaults to log any messages with level `INFO` and higher to `STDERR` using `console.error`.
The following [log levels](https://typescript.temporal.io/api/namespaces/worker#loglevel) are listed in increasing order of severity.

#### Customizing the default logger

Temporal uses a [`DefaultLogger`](https://typescript.temporal.io/api/classes/worker.DefaultLogger/) that implements the basic interface:

```ts
import { DefaultLogger, Runtime } from '@temporalio/worker';

const logger = new DefaultLogger('WARN', ({ level, message }) => {
  console.log(`Custom logger: ${level} — ${message}`);
});
Runtime.install({ logger });
```

The previous code example sets the default logger to log only messages with level `WARN` and higher.

#### Accumulate logs for testing and reporting

```ts
import { DefaultLogger, LogEntry, LogLevel } from '@temporalio/worker';

const logs: LogEntry[] = [];
const logger = new DefaultLogger(LogLevel.TRACE, (entry) => logs.push(entry));

logger.debug('hey', { a: 1 });
logger.info('ho');
logger.warn('lets', { a: 1 });
logger.error('go');
```

A common logging use case is logging to a file to be picked up by a collector like the [Datadog Agent](https://docs.datadoghq.com/logs/log_collection/nodejs/?tab=winston30).

```ts
import { 
  DefaultLogger,
  Runtime
} from '@temporalio/worker';
import winston from 'winston';

const winstonLogger = winston.createLogger({
  level: 'debug',
});

const logger = new DefaultLogger('DEBUG', (entry) => {
  winstonLogger.log({
    label: 'worker',
    level: entry.level.toLowerCase(),
    message: entry.message,
    timestamp: Number(entry.timestampNanos / 1_000_000n),
    ...entry.meta,
  });
});

Runtime.install({ logger });
```

## Visibility APIs 

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### How to use Search Attributes 

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI.
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `UpsertSearchAttributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `DescribeWorkflow`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [With the Temporal CLI](/cli/command-reference/workflow#list).
  - In code by calling `ListWorkflowExecutions`.

Here is how to query Workflow Executions:

Use [`WorkflowService.listWorkflowExecutions`](https://typescript.temporal.io/api/classes/proto.temporal.api.workflowservice.v1.WorkflowService-1#listworkflowexecutions):

```typescript
import { Connection } from '@temporalio/client';

const connection = await Connection.connect();
const response = await connection.workflowService.listWorkflowExecutions({
  query: `ExecutionStatus = "Running"`,
});
```

where `query` is a [List Filter](/list-filter).

### How to set custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

Use [`WorkflowOptions.searchAttributes`](https://typescript.temporal.io/api/interfaces/client.WorkflowOptions#searchattributes).

<!--SNIPSTART typescript-search-attributes-client-->
[search-attributes/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/search-attributes/src/client.ts)
```ts
const handle = await client.workflow.start(example, {
  taskQueue: 'search-attributes',
  workflowId: 'search-attributes-example-0',
  searchAttributes: {
    CustomIntField: [2],
    CustomKeywordListField: ['keywordA', 'keywordB'],
    CustomBoolField: [true],
    CustomDatetimeField: [new Date()],
    CustomTextField: [
      'String field is for text. When queried, it will be tokenized for partial match. StringTypeField cannot be used in Order By',
    ],
  },
});

const { searchAttributes } = await handle.describe();
```
<!--SNIPEND-->

The type of `searchAttributes` is `Record<string, string[] | number[] | boolean[] | Date[]>`.

### How to upsert Search Attributes 

You can upsert Search Attributes to add or update Search Attributes from within Workflow code.

Inside a Workflow, we can read from [`WorkflowInfo.searchAttributes`](https://typescript.temporal.io/api/interfaces/workflow.WorkflowInfo#searchattributes) and call [`upsertSearchAttributes`](https://typescript.temporal.io/api/namespaces/workflow#upsertsearchattributes):

<!--SNIPSTART typescript-search-attributes-workflow -->
[search-attributes/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/search-attributes/src/workflows.ts)
```ts
export async function example(): Promise<SearchAttributes> {
  const customInt = (workflowInfo().searchAttributes.CustomIntField?.[0] as number) || 0;
  upsertSearchAttributes({
    // overwrite the existing CustomIntField: [2]
    CustomIntField: [customInt + 1],

    // delete the existing CustomBoolField: [true]
    CustomBoolField: [],

    // add a new value
    CustomDoubleField: [3.14],
  });
  return workflowInfo().searchAttributes;
}
```
<!--SNIPEND-->

### How to remove a Search Attribute from a Workflow 

To remove a Search Attribute that was previously set, set it to an empty array: `[]`.

```typescript
import { upsertSearchAttributes } from '@temporalio/workflow';

async function yourWorkflow() {
  upsertSearchAttributes({ CustomIntField: [1] });

// ... later, to remove:
upsertSearchAttributes({ CustomIntField: [] });
}
```

## Workflow sinks

Sinks enable one-way export of logs, metrics, and traces from the Workflow isolate to the Node.js environment.

Sinks are written as objects with methods.
Similar to Activities, they are declared in the Worker and then proxied in Workflow code, and it helps to share types between both.

### Comparing sinks and Activities

Sinks are similar to Activities in that they are both registered on the Worker and proxied into the Workflow.
However, they differ from Activities in important ways:

- A sink function doesn't return any value back to the Workflow and cannot be awaited.
- A sink call isn't recorded in the Event History of a Workflow Execution (no timeouts or retries).
- A sink function _always_ runs on the same Worker that runs the Workflow Execution it's called from.

### Declare the sink interface

Explicitly declaring a sink's interface is optional but is useful for ensuring type safety in subsequent steps:

<!--SNIPSTART typescript-logger-sink-interface-->
[packages/test/src/workflows/log-sink-tester.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/log-sink-tester.ts)
```ts
import type { Sinks } from '@temporalio/workflow';

export interface CustomLoggerSinks extends Sinks {
  customLogger: {
    info(message: string): void;
  };
}
```
<!--SNIPEND-->

### Implement sinks

Implementing sinks is a two-step process.

Implement and inject the Sink function into a Worker

<!--SNIPSTART typescript-logger-sink-worker-->
[sinks/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/sinks/src/worker.ts)
```ts
import { InjectedSinks, Worker } from '@temporalio/worker';
import { MySinks } from './workflows';

async function main() {
  const sinks: InjectedSinks<MySinks> = {
    alerter: {
      alert: {
        fn(workflowInfo, message) {
          console.log('sending SMS alert!', {
            workflowId: workflowInfo.workflowId,
            workflowRunId: workflowInfo.runId,
            message,
          });
        },
        callDuringReplay: false, // The default
      },
    },
  };
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows'),
    taskQueue: 'sinks',
    sinks,
  });
  await worker.run();
  console.log('Worker gracefully shutdown');
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```
<!--SNIPEND-->

- Sink function implementations are passed as an object into [WorkerOptions](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions/#sinks).
- You can specify whether you want the injected function to be called during Workflow replay by setting the `callDuringReplay` option.

### Proxy and call a sink function from a Workflow

<!--SNIPSTART typescript-logger-sink-workflow-->
[packages/test/src/workflows/log-sample.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/log-sample.ts)
```ts
import * as wf from '@temporalio/workflow';

export async function logSampleWorkflow(): Promise<void> {
  wf.log.info('Workflow execution started');
}
```
<!--SNIPEND-->

Some important features of the [InjectedSinkFunction](https://typescript.temporal.io/api/interfaces/worker.InjectedSinkFunction) interface:

- **Injected WorkflowInfo argument:** The first argument of a Sink function implementation is a [`workflowInfo` object](https://typescript.temporal.io/api/interfaces/workflow.WorkflowInfo/) that contains useful metadata.
- **Limited arguments types:** The remaining Sink function arguments are copied between the sandbox and the Node.js environment using the [structured clone algorithm](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm).
- **No return value:** To prevent breaking determinism, Sink functions cannot return values to the Workflow.

**Advanced: Performance considerations and non-blocking Sinks**

The injected sink function contributes to the overall Workflow Task processing duration.

- If you have a long-running sink function, such as one that tries to communicate with external services, you might start seeing Workflow Task timeouts.
- The effect is multiplied when using `callDuringReplay: true` and replaying long Workflow histories because the Workflow Task timer starts when the first history page is delivered to the Worker.

---

# Set up a local development environment with the TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/set-up-your-local-typescript

> Configure your local development environment to get started developing with Temporal

# Quickstart

Configure your local development environment to get started developing with Temporal.

## Install Node.js

The TypeScript SDK requires Node.js 20 or later. Install Node.js via your package manager by following the [official Node.js instructions](https://nodejs.org/en/download/).

## Install the Temporal TypeScript SDK

You can create a new project with the Temporal SDK:

If you're creating a new project using `npx @temporalio/create`, the required SDK packages will be installed
automatically.

To add Temporal to an existing project, install the required packages manually with
`npm install @temporalio/client @temporalio/worker @temporalio/workflow`.

Next, you'll configure a local Temporal Service for development.

```bash
npx @temporalio/create@latest ./my-app
```

When prompted to select a sample, choose the **hello-world** sample.

## Install Temporal CLI

The fastest way to get a development version of the Temporal Service running on your local machine is to use
[Temporal CLI](/cli).

Choose your operating system to install Temporal CLI.

**macOS**

Install the Temporal CLI using Homebrew:
```bash
brew install temporal
```

**Windows**

Download the Temporal CLI archive for your architecture:

- [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64)
- [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64)

Extract it and add `temporal.exe` to your PATH.

**Linux**

Download the Temporal CLI for your architecture:

- [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64)
- [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64)

Extract the archive and move the `temporal` binary into your PATH, for example:
```bash
sudo mv temporal /usr/local/bin
```

## Start the development server

Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command.

This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory
database.

The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at
http://localhost:8233.

Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal
Service at any time by pressing CTRL+C.

Once you have everything installed, you're ready to build apps with Temporal on your local machine.

After installing, open a new Terminal window and start the development server:

```bash
temporal server start-dev
```

#### Change the Web UI port
The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server:
```bash
temporal server start-dev --ui-port 8080
```

The Temporal Web UI will now be available at http://localhost:8080.

## Run Hello World: Test your installation

Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and
Activity.

This test will confirm that:

- The Temporal TypeScript SDK is properly installed
- Your local Temporal Service is running
- You can successfully create and execute Workflows and Activities
- The communication between components is functioning correctly

### 1. Create the Activity

Create an Activity file (activities.ts):

<!--SNIPSTART typescript-hello-activity-->
[hello-world/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/hello-world/src/activities.ts)
```ts
export async function greet(name: string): Promise<string> {
  return `Hello, ${name}!`;
}
```
<!--SNIPEND-->

An Activity is a normal function or method that executes a single, well-defined action (either short or long running),
which often involve interacting with the outside world, such as sending emails, making network requests, writing to a
database, or calling an API, which are prone to failure. If an Activity fails, Temporal automatically retries it based
on your configuration.

### 2. Create the Workflow

Create a Workflow file (workflows.ts):

<!--SNIPSTART typescript-hello-workflow-->
[hello-world/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/hello-world/src/workflows.ts)
```ts
import { proxyActivities } from '@temporalio/workflow';
// Only import the activity types
import type * as activities from './activities';

const { greet } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

/** A workflow that simply calls an activity */
export async function example(name: string): Promise<string> {
  return await greet(name);
}
```
<!--SNIPEND-->

Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run and
keep running for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will
automatically recreate its pre-failure state so it can continue right where it left off.

### 3. Create and Run the Worker

Create a Worker file (worker.ts):

```ts
import { NativeConnection, Worker } from '@temporalio/worker';
import * as activities from './activities';

async function run() {
  // Step 1: Establish a connection with Temporal server.
  //
  // Worker code uses `@temporalio/worker.NativeConnection`.
  // (But in your application code it's `@temporalio/client.Connection`.)
  const connection = await NativeConnection.connect({
    address: 'localhost:7233',
    // TLS and gRPC metadata configuration goes here.
  });
  try {
    // Step 2: Register Workflows and Activities with the Worker.
    const worker = await Worker.create({
      connection,
      namespace: 'default',
      taskQueue: 'hello-world',
      // Workflows are registered using a path as they run in a separate JS context.
      workflowsPath: require.resolve('./workflows'),
      activities,
    });

    // Step 3: Start accepting tasks on the `hello-world` queue
    //
    // The worker runs until it encounters an unexpected error or the process receives a shutdown signal registered on
    // the SDK Runtime object.
    //
    // By default, worker logs are written via the Runtime logger to STDERR at INFO level.
    //
    // See https://typescript.temporal.io/api/classes/worker.Runtime#install to customize these defaults.
    await worker.run();
  } finally {
    // Close the connection once the worker has stopped
    await connection.close();
  }
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Run the Worker and keep this terminal running:

```bash
npm run start
```

With your Activity and Workflow defined, you need a Worker to execute them. A Worker polls a Task Queue, that you
configure it to poll, looking for work to do. Once the Worker dequeues a Workflow or Activity task from the Task Queue,
it then executes that task.

Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your
Workflows and Activities. For more information on Workers, see
[Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers).

### 4. Execute the Workflow

Now that your Worker is running, it's time to start a Workflow Execution from a [Temporal Client](/develop/typescript/client/temporal-client).

This final step will validate that everything is working correctly with your file labeled `client.ts`.

Create a separate file called `client.ts`.

```ts
import { Client, Connection } from '@temporalio/client';
import { nanoid } from 'nanoid';
import { example } from './workflows';

async function run() {
  // Connect to the default Server location
  const connection = await Connection.connect({ address: 'localhost:7233' });
  // In production, pass options to configure TLS and other settings:
  // {
  //   address: 'foo.bar.tmprl.cloud',
  //   tls: {}
  // }

  const client = new Client({
    connection,
    // namespace: 'foo.bar', // connects to 'default' namespace if not specified
  });

  const handle = await client.workflow.start(example, {
    taskQueue: 'hello-world',
    // type inference works! args: [name: string]
    args: ['Temporal'],
    // in practice, use a meaningful business ID, like customerId or transactionId
    workflowId: 'workflow-' + nanoid(),
  });
  console.log(`Started workflow ${handle.workflowId}`);

  // optional: wait for client result
  console.log(await handle.result()); // Hello, Temporal!
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Then run:

```bash
npm run workflow
```

### Verify success

If everything is working correctly, you should see:

- Worker processing the workflow and activity
- Output: `Workflow result: Hello, Temporal!`
- Workflow Execution details in the [Temporal Web UI](http://localhost:8233)

#### <strong>Additional details about Workflow Execution</strong>

- Temporal clients are not explicitly closed.
- To enable TLS, set the `tls` option to `true` for default settings or pass a [`TLSConfig`](https://typescript.temporal.io/api/interfaces/worker.TLSConfig) for custom configuration.
- Calling `client.workflow.start()` and `client.workflow.execute()` send a command to Temporal Server to schedule a new
  Workflow Execution on the specified Task Queue.
- If you started a Workflow with `client.workflow.start()`, you can choose to wait for the result anytime with
  handle.result().
- Using a Workflow Handle isn't necessary with `client.workflow.execute()`.

- [Run your first Temporal Application](https://learn.temporal.io/getting_started/typescript/first_program_in_typescript/): Create a basic Workflow and run it with the Temporal TypeScript SDK

- [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course

---

# Worker Versioning (legacy) - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/worker-versioning-legacy

> Learn the TypeScript SDK's outdated Worker Versioning APIs.

## How to use Worker Versioning in TypeScript (Deprecated) 

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>
> See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information.
>

A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp.
To use Worker Versioning, you need to pass a Build ID to your TypeScript Worker and opt in to Worker Versioning.

### Assign a Build ID to your Worker and opt in to Worker Versioning

You should understand assignment rules before completing this step.
See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information.

To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on.

```typescript
// ...
const worker = await Worker.create({
  taskQueue: 'your_task_queue_name',
  buildId: buildId,
  useVersioning: true,
  // ...
});
// ...
```

> **⚠️ Warning:**
>
> Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules.
>

### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the Workflow that created them if they are also configured to run on the same Task Queue.
When configured to run on a separate task queue, they will default to using the current assignment rules.

If you want to override this behavior, you can specify your intent via the `versioningIntent`
field available on the options object for each of these commands.

For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent:

```typescript
// ...
const { echo } = proxyActivities<typeof activities>({
  startToCloseTimeout: '20s',
  versioningIntent: 'USE_ASSIGNMENT_RULES',
});
// ...
```

### Tell the Task Queue about your Worker's Build ID (Deprecated)

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID.
You might want to do this as part of your CI deployment process.

```typescript
// ...
await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', {
  operation: 'addNewIdInNewDefaultSet',
  buildId: 'deadbeef',
});
```

This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue.
New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers.

If, instead, you want to add the Build ID to an existing compatible set, you can do this:

```typescript
// ...
await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', {
  operation: 'addNewCompatibleVersion',
  buildId: 'deadbeef',
  existingCompatibleBuildId: 'some-existing-build-id',
});
```

This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set.

You can promote an existing Build ID in a set to be the default for that set:

```typescript
// ...
await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', {
  operation: 'promoteBuildIdWithinSet',
  buildId: 'deadbeef',
});
```

You can promote an entire set to become the default set for the queue. New Workflows will start using that set's default build.

```typescript
// ...
await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', {
  operation: 'promoteSetByBuildId',
  buildId: 'deadbeef',
});
```

You can merge two sets into one, preserving the primary set's default Build ID as the default
for the merged set.

```typescript
// ...
await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', {
  operation: 'mergeSets',
  primaryBuildId: 'deadbeef',
  secondaryBuildId: 'some-existing-build-id',
});
```

---

# Workers - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workers

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Workers

- [Run a Worker](/develop/typescript/workers/run-worker-process)
- [Interceptors](/develop/typescript/workers/interceptors)
- [Serverless Workers](/develop/typescript/workers/serverless-workers)

---

# Interceptors - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workers/interceptors

Interceptors are SDK hooks that let you intercept inbound and outbound Temporal calls. You use them to apply shared
behavior across many calls, such as tracing and authorization, before calls reach the application code and after they return.
This is similar to middleware in other frameworks.

There are two main types of interceptors--inbound and outbound.

* Outbound interceptors wrap network calls, running before they reach the network and after they return.
* Inbound interceptors run after the network hop, wrapping application code and running before it starts and after it returns.

Those further break down into concrete Interceptor types--see below.

## How to implement interceptors in TypeScript 

Interceptors run as a chain.  Each interceptor wraps the entire inner call: your code runs before the call, invokes `next` to execute the rest of the chain, and then runs after the call completes. This means you can inspect or modify both the `input` and the result, handle errors, and perform side effects at either stage.

The TypeScript SDK comes with an optional interceptor package that adds tracing with
[OpenTelemetry](https://www.npmjs.com/package/@temporalio/interceptors-opentelemetry). See how to use it in the
[interceptors-opentelemetry](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) code
sample.

- [WorkflowInboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/workflow.WorkflowInboundCallsInterceptor/):
  Intercept Workflow inbound calls like execution, Signals, and Queries.
- [WorkflowOutboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/workflow.WorkflowOutboundCallsInterceptor/):
  Intercept Workflow outbound calls to Temporal APIs like scheduling Activities and starting Timers.
- [ActivityInboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/worker.ActivityInboundCallsInterceptor):
  Intercept inbound calls to an Activity (such as `execute`).
- [WorkflowClientInterceptor](https://typescript.temporal.io/api/interfaces/client.WorkflowClientInterceptor/):
  Intercept workflow-related methods of [`Client`](https://typescript.temporal.io/api/classes/client.Client/) and
  [`WorkflowHandle`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle) like starting or signaling a
  Workflow.
- [NexusInboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/worker.NexusInboundCallsInterceptor):
  Intercept inbound Nexus Operation calls like `startOperation` and `cancelOperation`.
- [NexusOutboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/worker.NexusOutboundCallsInterceptor):
  Intercept outbound calls from Nexus Operations, such as enriching log attributes and metric tags.

All interceptor methods are optional—it's up to the implementor to choose which methods to intercept.

## Interceptor examples

**Log start and completion of Activities**

```ts
import { ActivityInput, Next, WorkflowOutboundCallsInterceptor } from '@temporalio/workflow';

export class ActivityLogInterceptor implements WorkflowOutboundCallsInterceptor {
  constructor(public readonly workflowType: string) {}

  async scheduleActivity(
    input: ActivityInput,
    next: Next<WorkflowOutboundCallsInterceptor, 'scheduleActivity'>
  ): Promise<unknown> {
    console.log('Starting activity', { activityType: input.activityType });
    try {
      return await next(input);
    } finally {
      console.log('Completed activity', {
        workflow: this.workflowType,
        activityType: input.activityType,
      });
    }
  }
}
```

**Log Nexus Operations**

```ts
import type {
  NexusInboundCallsInterceptor,
  NexusStartOperationInput,
  NexusStartOperationOutput,
  Next,
} from '@temporalio/worker';

export class NexusOperationLogInterceptor implements NexusInboundCallsInterceptor {
  async startOperation(
    input: NexusStartOperationInput,
    next: Next<NexusInboundCallsInterceptor, 'startOperation'>
  ): Promise<NexusStartOperationOutput> {
    console.log('Starting Nexus operation', {
      service: input.ctx.service,
      operation: input.ctx.operation,
    });
    const output = await next(input);
    console.log('Nexus operation started', {
      service: input.ctx.service,
      operation: input.ctx.operation,
      async: output.result.isAsync,
    });
    return output;
  }
}
```

## Register an Interceptor 

Registering an interceptor means providing it to the SDK so Temporal can invoke it when matching Client or Worker calls occur. Once registered, it runs in the call path and can observe or modify request and response data.

### Register via a Plugin

If you're building a reusable library or want to bundle interceptors with other primitives, you can register them
through a [Plugin](/develop/plugins-guide#interceptors).

### Activity and client interceptors registration

- Activity interceptors are registered on Worker creation by passing an array of
  [ActivityInboundCallsInterceptor factory functions](https://typescript.temporal.io/api/interfaces/worker.ActivityInboundCallsInterceptorFactory)
  through [WorkerOptions](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#interceptors).

- Client interceptors are registered on `Client` construction by passing an array of
  [WorkflowClientInterceptor](https://typescript.temporal.io/api/interfaces/client.WorkflowClientInterceptor) via
  [ClientOptions.interceptors](https://typescript.temporal.io/api/interfaces/client.ClientOptions#interceptors).

### Workflow interceptors registration

Workflow interceptor registration is different from the other interceptors because they run in the Workflow isolate. To
register Workflow interceptors, export an `interceptors` function from a file located in the `workflows` directory and
provide the name of that file to the Worker on creation via
[WorkerOptions](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#interceptors).

At the time of construction, the Workflow context is already initialized for the current Workflow. You may call the
[`workflowInfo()`](https://typescript.temporal.io/api/namespaces/workflow#workflowinfo) function to access
Workflow-specific information from an interceptor.

`src/workflows/your-interceptors.ts`

```ts
import { workflowInfo } from '@temporalio/workflow';

export const interceptors = () => ({
  outbound: [new ActivityLogInterceptor(workflowInfo().workflowType)],
  inbound: [],
});
```

`src/worker/index.ts`

```ts
const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  interceptors: {
    workflowModules: [require.resolve('./workflows/your-interceptors')],
  },
});
```

### Nexus interceptor registration

Nexus interceptors are registered on Worker creation via
[WorkerOptions](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#interceptors).
Pass an array of factory functions to `interceptors.nexus`.
Each factory receives an [`OperationContext`](https://typescript.temporal.io/api/classes/nexus.OperationContext) and returns an object with optional `inbound` and `outbound` interceptors.

`src/worker/index.ts`

```ts
import { NexusOperationLogInterceptor } from './nexus-interceptors';

const worker = await Worker.create({
  // ...
  nexusServices: [/* your Nexus services */],
  interceptors: {
    nexus: [
      (_ctx) => ({
        inbound: new NexusOperationLogInterceptor(),
      }),
    ],
  },
});
```

---

# Run a Worker - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workers/run-worker-process

> Create and run a Temporal Worker using the TypeScript SDK.

This page covers long-lived Workers that you host and run as persistent processes.
For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/typescript/workers/serverless-workers).

## Create and run a Worker 

Create a Worker with `Worker.create()`, passing a connection, the Task Queue to poll, and the Workflows and Activities it can execute.
Call `run()` to start polling.

<!--SNIPSTART typescript-hello-worker-->
[hello-world/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/hello-world/src/worker.ts)
```ts
import { NativeConnection, Worker } from '@temporalio/worker';
import * as activities from './activities';

async function run() {
  // Step 1: Establish a connection with Temporal server.
  //
  // Worker code uses `@temporalio/worker.NativeConnection`.
  // (But in your application code it's `@temporalio/client.Connection`.)
  const connection = await NativeConnection.connect({
    address: 'localhost:7233',
    // TLS and gRPC metadata configuration goes here.
  });
  try {
    // Step 2: Register Workflows and Activities with the Worker.
    const worker = await Worker.create({
      connection,
      namespace: 'default',
      taskQueue: 'hello-world',
      // Workflows are registered using a path as they run in a separate JS context.
      workflowsPath: require.resolve('./workflows'),
      activities,
    });

    // Step 3: Start accepting tasks on the `hello-world` queue
    //
    // The worker runs until it encounters an unexpected error or the process receives a shutdown signal registered on
    // the SDK Runtime object.
    //
    // By default, worker logs are written via the Runtime logger to STDERR at INFO level.
    //
    // See https://typescript.temporal.io/api/classes/worker.Runtime#install to customize these defaults.
    await worker.run();
  } finally {
    // Close the connection once the worker has stopped
    await connection.close();
  }
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```
<!--SNIPEND-->

Workflows are registered by path rather than by value, because they run in a separate JavaScript context.

## Register Workflows and Activities 

All Workers polling the same Task Queue must register the same Workflow Types and Activity Types.
A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue.
A Worker that receives a Task for a type it did not register fails that Task.

In development, use [`workflowsPath`](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions/#workflowspath):

<!--SNIPSTART typescript-worker-create -->
[snippets/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/worker.ts)
```ts
import { Worker } from '@temporalio/worker';
import * as activities from './activities';

async function run() {
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows'),
    taskQueue: 'snippets',
    activities,
  });

  await worker.run();
}
```
<!--SNIPEND-->

In this snippet, the Worker bundles the Workflow code at runtime.

In production, you can improve your Worker's startup time by bundling in advance. As part of your production build, call `bundleWorkflowCode`:

<!--SNIPSTART typescript-bundle-workflow -->
[production/src/scripts/build-workflow-bundle.ts](https://github.com/temporalio/samples-typescript/blob/main/production/src/scripts/build-workflow-bundle.ts)
```ts
import { bundleWorkflowCode } from '@temporalio/worker';
import { writeFile } from 'fs/promises';
import path from 'path';

async function bundle() {
  const { code } = await bundleWorkflowCode({
    workflowsPath: require.resolve('../workflows'),
  });
  const codePath = path.join(__dirname, '../../workflow-bundle.js');

  await writeFile(codePath, code);
  console.log(`Bundle written to ${codePath}`);
}
```
<!--SNIPEND-->

Then pass the bundle to the Worker:

<!--SNIPSTART typescript-production-worker-->
[production/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/production/src/worker.ts)
```ts
const workflowOption = () =>
  process.env.NODE_ENV === 'production'
    ? {
        workflowBundle: {
          codePath: require.resolve('../workflow-bundle.js'),
        },
      }
    : { workflowsPath: require.resolve('./workflows') };

async function run() {
  const worker = await Worker.create({
    ...workflowOption(),
    activities,
    taskQueue: 'production-sample',
  });

  await worker.run();
}
```
<!--SNIPEND-->

## Connect to Temporal Cloud 

To run a Worker against Temporal Cloud, configure the connection with your Namespace address and authentication credentials.
See [Connect to Temporal Cloud](/develop/typescript/client/temporal-client#connect-to-temporal-cloud) for setup instructions.

## Configure Worker options 

[`Worker.create()`](https://typescript.temporal.io/api/classes/worker.Worker#create) accepts options that control concurrency limits, pollers, timeouts, and caching.
The defaults work for most cases.

To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference).

## Run a versioned Worker 

Set a Worker Deployment Version and enable versioning in `workerDeploymentOptions`.
When `useWorkerVersioning` is `true`, `defaultVersioningBehavior` is required.

<!--SNIPSTART typescript-versioned-worker-->
[features/snippets/worker/worker.ts](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.ts)
```ts
const worker = await Worker.create({
  connection,
  taskQueue: 'my-task-queue',
  workflowsPath: require.resolve('./workflows'),
  workerDeploymentOptions: {
    version: { deploymentName: 'my-app', buildId: '1.0' },
    useWorkerVersioning: true,
    defaultVersioningBehavior: 'PINNED',
  },
});
```
<!--SNIPEND-->

`defaultVersioningBehavior` covers every Workflow on the Worker.
To set the behavior for a single Workflow instead, pass `versioningBehavior` through [`setWorkflowOptions()`](https://typescript.temporal.io/api/namespaces/workflow#setworkflowoptions) where you define that Workflow.

See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out.

## Shut down a Worker 

Workers shut down when the process receives any of the operating system signals listed in
[shutdownSignals](https://typescript.temporal.io/api/interfaces/worker.RuntimeOptions#shutdownsignals): `'SIGINT'`,
`'SIGTERM'`, `'SIGQUIT'`, and `'SIGUSR2'`.

In development, shut down Workers with `Ctrl+C` (`SIGINT`) or
[nodemon](https://github.com/temporalio/samples-typescript/blob/277ce0e858cd74a582c83991aaf4430a53653aab/hello-world/package.json#L12)
(`SIGUSR2`). In production, give Workers time to finish in-progress Activities by setting
[shutdownGraceTime](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#shutdowngracetime):

<!--SNIPSTART typescript-worker-graceful-shutdown-->
[features/snippets/worker/worker.ts](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.ts)
```ts
const worker = await Worker.create({
  connection,
  taskQueue: 'my-task-queue',
  workflowsPath: require.resolve('./workflows'),
  shutdownGraceTime: '30s',
});

await worker.run();
```
<!--SNIPEND-->

As soon as a Worker receives one of those signals or a shutdown request, the Worker stops polling for new Tasks and allows in-flight
Tasks to complete until `shutdownGraceTime` is reached. Any Activities still running at that time stop running and are
rescheduled by the Temporal Service when an Activity timeout occurs.

If you must guarantee that the Worker eventually shuts down, set
[shutdownForceTime](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#shutdownforcetime).

In integration tests or when automating a fleet of Workers, shut a Worker down programmatically with
[Worker.shutdown()](https://typescript.temporal.io/api/classes/worker.Worker#shutdown).

See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities.

### Worker states

At any time, you can Query Worker state with
[Worker.getState()](https://typescript.temporal.io/api/classes/worker.Worker#getstate). A Worker is always in one of
seven states:

- `INITIALIZED`: The initial state of the Worker after calling
  [Worker.create()](https://typescript.temporal.io/api/classes/worker.Worker#create) and successfully connecting to the
  server.
- `RUNNING`: [Worker.run()](https://typescript.temporal.io/api/classes/worker.Worker#run) was called and the Worker is
  polling Task Queues.
- `FAILED`: The Worker encountered an unrecoverable error; `Worker.run()` should reject with the error.
- The last four states are related to the Worker shutdown process:
  - `STOPPING`: The Worker received a shutdown signal or `Worker.shutdown()` was called. The Worker forcefully shuts
    down after `shutdownGraceTime` expires.
  - `DRAINING`: All Workflow Tasks have been drained; waiting for Activities and cached Workflows eviction.
  - `DRAINED`: All Activities and Workflows have completed; ready to shut down.
  - `STOPPED`: Shutdown complete; `worker.run()` resolves.

If you need more visibility into internal Worker state, see the
[Worker class](https://typescript.temporal.io/api/classes/worker.Worker) in the API reference.

## Run a Worker on Docker 

> **📝 Note:**
>
> To improve Worker startup time, prepare Workflow bundles ahead of time. See the
> [production sample](https://github.com/temporalio/samples-typescript/tree/main/production) for details.
>

Workers based on the TypeScript SDK can be deployed and run as Docker containers.

Use an LTS Node.js release such as 18, 20, 22, or 24. Both `amd64` and `arm64` architectures are supported. A
glibc-based image is required; musl-based images are _not_ supported (see below).

The most direct way to deploy a TypeScript SDK Worker on Docker is to start with the `node:20-bookworm` image. For example:

```dockerfile
FROM node:20-bookworm

# For better cache utilization, copy package.json and lock file first and install the dependencies before copying the
# rest of the application and building.
COPY . /app
WORKDIR /app

# Alternatively, run npm ci, which installs only dependencies specified in the lock file and is generally faster.
RUN npm install --only=production \
    && npm run build

CMD ["npm", "start"]
```

For smaller images and/or more secure deployments, it is also possible to use `-slim` Docker image variants (like
`node:20-bookworm-slim`) or `distroless/nodejs` Docker images (like `gcr.io/distroless/nodejs20-debian12`) with the
following caveats.

### Using `node:slim` images

`node:slim` images leave out common packages found in regular images, which makes them much smaller.

One of the packages they leave out is `ca-certificates`, and the TypeScript SDK requires those root TLS certificates.
The requirement holds even when you connect to a local Temporal Service, and when the connection config does not use
TLS. Install the package when you build the image:

```dockerfile
FROM node:20-bookworm-slim

RUN apt-get update \
    && apt-get install -y ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# ... same as with regular image
```

Failure to install this dependency results in a `[TransportError: transport error]` runtime error, because the
certificates cannot be verified.

### Using `distroless/nodejs` images

`distroless/nodejs` images include only the files required to execute `node`. They are about half the size of
`node:slim` images and carry a smaller attack surface.

TypeScript SDK Workers run on `distroless/nodejs` images, as long as your own code does not need dependencies that those
images leave out.

Build tools, including the `npm` command, are _not_ in the `distroless/nodejs` image, so building inside it fails.
Use a multi-step Dockerfile instead:

```dockerfile
# -- BUILD STEP --

FROM node:20-bookworm AS builder

COPY . /app
WORKDIR /app

RUN npm install --only=production \
    && npm run build

# -- RESULTING IMAGE --

FROM gcr.io/distroless/nodejs20-debian12

COPY --from=builder /app /app
WORKDIR /app

CMD ["node", "build/worker.js"]
```

### Properly configure Node.js memory in Docker

By default, `node` sets its maximum old-gen memory to 25% of the machine's _physical memory_, up to 4 GB. In Docker,
that reads the host's memory rather than the container's limit. The container then either underuses the memory it was
given, or `node` tries to exceed that limit and the operating system kills the process.

Set the `--max-old-space-size` `node` argument explicitly, to roughly 80% of the memory in megabytes you want the `node`
process to use. Tune the value against your own application.

Pass this argument through the [`NODE_OPTIONS` environment variable](https://nodejs.org/api/cli.html#node_optionsoptions).

### Do not use Alpine

Alpine replaces glibc with musl, which is incompatible with the Rust core of the TypeScript SDK. Errors like these point
to a musl-based image:

```sh
Error: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /opt/app/node_modules/@temporalio/core-bridge/index.node)
```

Or like this:

```sh
Error: Error relocating /opt/app/node_modules/@temporalio/core-bridge/index.node: __register_atfork: symbol not found
```

---

# Serverless Workers - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workers/serverless-workers

> Write Temporal Workers that run on serverless compute using the TypeScript SDK.

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview.

Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes.
Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done.

For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers).
For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Supported providers

- [**AWS Lambda**](/develop/typescript/workers/serverless-workers/aws-lambda) - Use the `@temporalio/lambda-worker` package to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, and observability.
- [**GCP Cloud Run**](/develop/typescript/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, container packaging, and handling scale-in.

---

# Serverless Workers on AWS Lambda - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workers/serverless-workers/aws-lambda

> Write a Temporal Worker that runs on AWS Lambda using the TypeScript SDK @temporalio/lambda-worker package.

> **Public Preview**

The `@temporalio/lambda-worker` package lets you run a Temporal Serverless Worker on AWS Lambda.
Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive.
Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline.
You register Workflows and Activities the same way you would with a standard Worker.

For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda).

## Create and run a Worker in Lambda 

Use the `runWorker` function to create a Lambda handler that runs a Temporal Worker.
Pass a deployment version and a configure callback that sets up your Workflows and Activities.

<!--SNIPSTART typescript-lambda-worker {"selectedLines": ["1", "3-11", "13"], "highlightedLines": "6"}-->
[lambda-worker/src/index.ts](https://github.com/temporalio/samples-typescript/blob/main/lambda-worker/src/index.ts)
```ts {6}
import { runWorker } from '@temporalio/lambda-worker';
// ...
import * as activities from './activities';
import { TASK_QUEUE } from './workflows';

export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => {
  config.workerOptions.taskQueue = TASK_QUEUE;
  config.workerOptions.workflowBundle = {
    codePath: require.resolve('./workflow-bundle.js'),
  };
  config.workerOptions.activities = activities;
// ...
});
```
<!--SNIPEND-->

The deployment version is required.
Worker Deployment Versioning is always enabled for Serverless Workers.
Each Workflow must declare a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or `Pinned`.
The default versioning behavior is `PINNED`. To change it, set `workerDeploymentOptions.defaultVersioningBehavior` in the configure callback.

### Pre-bundle Workflow code 

Use `workflowBundle` with pre-bundled code instead of `workflowsPath`.
Pre-bundling avoids webpack bundling overhead on every Lambda cold start.

Build the bundle as a separate build step:

```typescript
import { bundleWorkflowCode } from '@temporalio/worker';
import { writeFile } from 'fs/promises';

const { code } = await bundleWorkflowCode({
  workflowsPath: require.resolve('./workflows'),
});
await writeFile('./workflow-bundle.js', code);
```

Then reference the bundle in your handler with `workflowBundle: { codePath: require.resolve('./workflow-bundle.js') }`.

## Configure the Temporal connection 

The `@temporalio/lambda-worker` package automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details.

Compared with long-lived Workers, the location of the config file is resolved differently, in the following order:

1. `TEMPORAL_CONFIG_FILE` environment variable, if set.
2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`).
3. `temporal.toml` in the current working directory.

The file is optional. If absent, only environment variables are used.

Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options.

## Adjust Worker defaults for Lambda 

The `@temporalio/lambda-worker` package applies conservative defaults suited to short-lived Lambda invocations.
These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment.

| Setting | Lambda default |
|---|---|
| `maxConcurrentActivityTaskExecutions` | 2 |
| `maxConcurrentWorkflowTaskExecutions` | 10 |
| `maxConcurrentLocalActivityExecutions` | 2 |
| `maxConcurrentNexusTaskExecutions` | 5 |
| `workflowTaskPollerBehavior` | `SimpleMaximum(2)` |
| `activityTaskPollerBehavior` | `SimpleMaximum(1)` |
| `nexusTaskPollerBehavior` | `SimpleMaximum(1)` |
| `shutdownGraceTime` | 5 seconds |
| `maxCachedWorkflows` | 30 |
| `shutdownDeadlineBufferMs` | 7000 |

Eager Activities are not supported. Lambda invocations don't maintain persistent connections.

`shutdownDeadlineBufferMs` is specific to the `@temporalio/lambda-worker` package.
It controls how much time before the Lambda deadline the Worker begins its graceful shutdown.
The default is `shutdownGraceTime` (5s) + 2s.

If your Worker handles long-running Activities, increase `shutdownGraceTime`, `shutdownDeadlineBufferMs`, and the Lambda invocation deadline (`--timeout`) together.
For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities).

## Add observability with OpenTelemetry 

The `@temporalio/lambda-worker/otel` module provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layers.
With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions.

The underlying metrics and traces are the same ones the TypeScript SDK emits in any environment.
For general observability concepts and the full list of available metrics, see [Observability - TypeScript SDK](/develop/typescript/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).

<!--SNIPSTART typescript-lambda-worker {"highlightedLines": "2, 12"}-->
[lambda-worker/src/index.ts](https://github.com/temporalio/samples-typescript/blob/main/lambda-worker/src/index.ts)
```ts {2, 12}
import { runWorker } from '@temporalio/lambda-worker';
import { applyDefaults } from '@temporalio/lambda-worker/otel';
import * as activities from './activities';
import { TASK_QUEUE } from './workflows';

export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => {
  config.workerOptions.taskQueue = TASK_QUEUE;
  config.workerOptions.workflowBundle = {
    codePath: require.resolve('./workflow-bundle.js'),
  };
  config.workerOptions.activities = activities;
  applyDefaults(config);
});
```
<!--SNIPEND-->

`applyDefaults` registers Temporal SDK interceptors for tracing and configures the Core SDK to export metrics via OpenTelemetry Protocol (OTLP).
By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint.

To collect this telemetry, attach two ADOT Lambda layers:

1. The [ADOT JavaScript layer](https://aws-otel.github.io/docs/getting-started/lambda/lambda-js) for Node.js-side auto-instrumentation and trace export.
2. The [ADOT Collector layer](https://aws-otel.github.io/docs/getting-started/lambda) (`aws-otel-collector-amd64`) to run the OTel Collector as a Lambda extension, receiving telemetry via OTLP on `localhost:4317` and forwarding traces to X-Ray and metrics to CloudWatch.

The default Collector configuration does not route OTLP data to the traces pipeline.
You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines.
Bundle the following `otel-collector-config.yaml` in your Lambda deployment package:

<!--SNIPSTART typescript-lambda-worker-otel-collector-config-->
[lambda-worker/otel-collector-config.yaml](https://github.com/temporalio/samples-typescript/blob/main/lambda-worker/otel-collector-config.yaml)
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 'localhost:4317'
      http:
        endpoint: 'localhost:4318'

exporters:
  debug:
  awsxray:
    region: us-west-2
  awsemf:
    # AWS EMF exporter for metrics
    # These are example configurations
    namespace: TemporalWorkerMetrics
    log_group_name: /aws/lambda/<your-function-name>
    region: us-west-2
    dimension_rollup_option: NoDimensionRollup
    resource_to_telemetry_conversion:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray, debug]
    metrics:
      receivers: [otlp]
      exporters: [awsemf]
  telemetry:
    logs:
      level: info
    metrics:
      address: localhost:8888
```
<!--SNIPEND-->

Set the following environment variable on the Lambda function:

- `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml`

Enable X-Ray active tracing on the Lambda function:

```bash
aws lambda update-function-configuration \
  --function-name <your-function-name> \
  --tracing-config Mode=Active
```

The Lambda execution role must have permissions to write to X-Ray and CloudWatch.
Add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions to the execution role.
Without these permissions, the Collector fails silently and no telemetry appears.

When pre-bundling Workflow code, pass the plugin from `makeOtelPlugin()` so that Workflow interceptor modules are included in the bundle:

```typescript
import { bundleWorkflowCode } from '@temporalio/worker';
import { makeOtelPlugin } from '@temporalio/lambda-worker/otel';

const { plugin } = makeOtelPlugin();
const { code } = await bundleWorkflowCode({
  workflowsPath: require.resolve('./workflows'),
  plugins: [plugin],
});
```

---

# Serverless Workers on GCP Cloud Run - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workers/serverless-workers/cloud-run

> Run a Temporal Worker on a GCP Cloud Run worker pool using the TypeScript SDK.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other TypeScript Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific package.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).

## Create a versioned Worker 

Build the Worker as you would any long-running TypeScript Worker, then pass `workerDeploymentOptions` to `Worker.create()` to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

```ts
import { NativeConnection, Worker } from '@temporalio/worker';

const connection = await NativeConnection.connect({
  address: process.env.TEMPORAL_ADDRESS,
  apiKey: process.env.TEMPORAL_API_KEY,
  tls: true,
});

const worker = await Worker.create({
  connection,
  namespace: process.env.TEMPORAL_NAMESPACE!,
  taskQueue: process.env.TEMPORAL_TASK_QUEUE!,
  workflowsPath: require.resolve('./workflows'),
  workerDeploymentOptions: {
    version: { deploymentName: 'my-app', buildId: 'build-1' },
    useWorkerVersioning: true,
    defaultVersioningBehavior: 'PINNED',
  },
});

await worker.run();
```

`deploymentName` and `buildId` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage.

Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`.
Setting `defaultVersioningBehavior` as shown above covers every Workflow on the Worker.
To set the behavior per Workflow instead, pass the Workflow function to `setWorkflowOptions()` from `@temporalio/workflow`:

```ts
import { setWorkflowOptions } from '@temporalio/workflow';

setWorkflowOptions({ versioningBehavior: 'PINNED' }, myWorkflow);
export async function myWorkflow(): Promise<string> {
  // ...
}
```

For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/typescript/workers/run-worker-process).

## Configure the Temporal connection 

Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext.

To load the connection settings from a TOML config file and profiles instead of reading each variable by hand, use `loadClientConnectConfig()` from `@temporalio/envconfig` and pass its `connectionOptions` and `namespace` to `NativeConnection.connect()` and `Worker.create()`.
For the supported variables and the config file format, see [Environment configuration](/develop/environment-configuration).

## Package the Worker image 

The Worker reads TLS roots from the operating system's certificate store, and the slim Node.js images ship without one.
On `node:22-slim`, connecting with TLS fails at startup:

```
TransportError: tonic::transport::Error(Transport, NativeCertsNotFound)
```

Install the certificates in the runtime stage of your Dockerfile:

```dockerfile
RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*
```

Use a glibc-based image such as `node:22-slim` rather than an Alpine image. Alpine replaces glibc with musl, which the
Rust core does not support. See [Do not use Alpine](/develop/typescript/workers/run-worker-process#do-not-use-alpine).

Node.js also sizes its heap from the host's memory rather than the container limit, so set
`NODE_OPTIONS=--max-old-space-size=<MB>` on the Worker Pool to about 80% of the instance's memory limit. A Cloud Run
Worker Pool defaults to 512 MiB per instance, so raise `--memory` if your Worker needs more. See
[Run a Worker on Docker](/develop/typescript/workers/run-worker-process#run-a-worker-on-docker).

## Keep Activities safe across scale-in 

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing.
An instance running a long Activity can be stopped mid-execution.

Use [Activity Heartbeats](/develop/typescript/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over:

```ts
import { heartbeat } from '@temporalio/activity';

export async function myActivity(items: string[]): Promise<string> {
  for (let i = 0; i < items.length; i++) {
    heartbeat(i);
    // ... process items[i]
  }
  return 'done';
}
```

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability 

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - TypeScript SDK](/develop/typescript/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).

---

# Workflows - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows

> This section explains how to implement Workflows with the TypeScript SDK

![TypeScript SDK Banner](https://docs.temporal.io/img/assets/banner-typescript-temporal.png)

## Workflows

- [Workflow basics](/develop/typescript/workflows/basics)
- [Child Workflows](/develop/typescript/workflows/child-workflows)
- [Continue-As-New](/develop/typescript/workflows/continue-as-new)
- [Message passing](/develop/typescript/workflows/message-passing)
- [Cancellation](/develop/typescript/workflows/cancellation)
- [Cancellation scopes](/develop/typescript/workflows/cancellation-scopes)
- [Timeouts](/develop/typescript/workflows/timeouts)
- [Schedules](/develop/typescript/workflows/schedules)
- [Timers](/develop/typescript/workflows/timers)
- [Versioning](/develop/typescript/workflows/versioning)
- [Workflow Streams](/develop/typescript/workflows/workflow-streams)

---

# Workflow basics - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/basics

> Shows how to create a Workflow with the TypeScript SDK

## Develop a Workflow 

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal TypeScript SDK programming model, Workflow Definitions are _just functions_, which can store state and orchestrate Activity Functions. The following code snippet uses `example` to schedule a `greet` Activity in the system to say hello.

A Workflow Definition can have multiple parameters; however, we recommend using a single object parameter.

```typescript
type ExampleArgs = {
  name: string;
};

export async function example(args: ExampleArgs): Promise<{ greeting: string }> {
  const greeting = await greet(args.name);
  return { greeting };
}
```

## Define Workflow parameters 

Temporal Workflows may have any number of custom parameters. However, we strongly recommend that you pass a single object
parameter that contains all the input fields, so that you can add fields without breaking the signature of the Workflow.
All Workflow Definition parameters must be serializable. For more reasons, see
[Workflow parameters](/workflow-definition#workflow-parameters).

You can define and pass parameters in your Workflow. In this example, you define your arguments in your `client.ts` file
and pass those parameters to `workflow.ts` through your Workflow function.

Start a Workflow with the parameters that are in the `client.ts` file. In this example we set the `name` parameter to
`Temporal` and `born` to `2019`. Then set the Task Queue and Workflow Id.

`client.ts`

```typescript
import { example } from './workflows';

...
await client.workflow.start(example, {
  args: [{ name: 'Temporal', born: 2019 }],
  taskQueue: 'your-queue',
  workflowId: 'business-meaningful-id',
});
```

In `workflows.ts` define the type of the parameter that the Workflow function takes in. The interface `ExampleParam` is
a name we can now use to describe the requirement in the previous example. It still represents having the two properties
called `name` and `born` that is of the type `string`. Then define a function that takes in a parameter of the type
`ExampleParam` and return a `Promise<string>`. The `Promise` object represents the eventual completion, or failure, of
`await client.workflow.start()` and its resulting value.

```ts
interface ExampleParam {
  name: string;
  born: number;
}
export async function example({ name, born }: ExampleParam): Promise<string> {
  return `Hello ${name}, you were born in ${born}.`;
}
```

## Define Workflow return parameters 

Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly
idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow
Execution will only ever receive one of either the result or the error.

To return a value of the Workflow function, use `Promise<something>`. The `Promise` is used to make asynchronous calls
and comes with guarantees.

The following example uses a `Promise<string>` to eventually return a `name` and `born` parameter.

```typescript
interface ExampleParam {
  name: string;
  born: number;
}
export async function example({ name, born }: ExampleParam): Promise<string> {
  return `Hello ${name}, you were born in ${born}.`;
}
```

## Customize your Workflow Type 

Workflows have a Type that are referred to as the Workflow name.

The following examples demonstrate how to set a custom name for your Workflow Type.

In TypeScript, the Workflow Type is the Workflow function name and there isn't a mechanism to customize the Workflow
Type.

In the following example, the Workflow Type is the name of the function, `helloWorld`.

<!--SNIPSTART typescript-workflow-type -->
[snippets/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/workflows.ts)
```ts
export async function helloWorld(): Promise<string> {
  return '👋 Hello World!';
}
```
<!--SNIPEND-->

## Develop Workflow logic 

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints).
Each Temporal SDK provides a
set of APIs that can be used inside your Workflow to interact with external (to the Workflow) application code.

In the Temporal TypeScript SDK, Workflows run in a deterministic sandboxed environment. The code is bundled on Worker
creation using Webpack, and can import any package as long as it does not reference Node.js or DOM APIs.

> **📝 Note:**
>
> If you **must** use a library that references a Node.js or DOM API and you are certain that those APIs are not used at
> runtime, add that module to the
> [ignoreModules](https://typescript.temporal.io/api/interfaces/worker.BundleOptions#ignoremodules) list.
>

The Workflow sandbox can run only deterministic code, so side effects and access to external state must be done through Activities because Activity outputs are recorded in the Event History and can read be deterministically by the Workflow.

This limitation also means that Workflow code cannot directly import the [Activity Definition](/activity-definition).
[Activity Types](/activity-definition#activity-type) can be imported, so they can be invoked in a type-safe manner.

To make the Workflow runtime deterministic, functions like `Math.random()`, `Date`, and `setTimeout()` are replaced by deterministic versions.

[FinalizationRegistry](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry)
and [WeakRef](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakRef) are removed because v8's garbage collector is not deterministic.

The following sections describe the replay-safe APIs available in the sandbox.

#### Logging

Use [`log`](https://typescript.temporal.io/api/namespaces/workflow#log) from `@temporalio/workflow` instead of `console.log`. The SDK logger automatically suppresses messages during replay to avoid duplicates:

```ts
import { log } from '@temporalio/workflow';

export async function myWorkflow(name: string): Promise<string> {
  log.info('Starting workflow', { name });
  // ...
}
```

For logger configuration, see [Observability: Log from a Workflow](/develop/typescript/platform/observability#logging).

#### Random numbers and UUIDs

`Math.random()` is replaced by a deterministic version in the sandbox, so you can use it directly. It produces the same sequence of values on replay.

For UUIDs, use [`uuid4()`](https://typescript.temporal.io/api/namespaces/workflow#uuid4) from `@temporalio/workflow`. It draws from the sandbox's deterministic random source, so it needs no extra dependency. Avoid `crypto.randomUUID()`, which is not available in the sandbox:

```ts
import { uuid4 } from '@temporalio/workflow';

const value = Math.random();
const id = uuid4();
```

Third-party UUID libraries that rely on `Math.random()` (such as the `uuid` package) are also safe, but the built-in avoids the dependency.

#### Current time

`Date.now()` and `new Date()` are replaced by deterministic versions that return the time of the last Workflow Task completion. The value only advances when you `await` something (like `sleep()`):

```ts
import { sleep } from '@temporalio/workflow';

// Prints the *exact* same timestamp on every iteration
for (let x = 0; x < 10; ++x) {
  console.log(Date.now());
}

// Prints timestamps increasing roughly 1s each iteration
for (let x = 0; x < 10; ++x) {
  await sleep('1 second');
  console.log(Date.now());
}
```

#### Detecting replay (advanced)

Use [`workflowInfo().unsafe.isReplaying`](https://typescript.temporal.io/api/interfaces/workflow.UnsafeWorkflowInfo#isreplaying) to guard code that should only run on the first execution, such as emitting metrics or sending external notifications from an [Interceptor](/develop/typescript/workers/interceptors).
> **⚠️ Caution:**
>
> Never use this to affect Workflow business logic — branching on replay status breaks determinism.
>

```ts
import { workflowInfo } from '@temporalio/workflow';

if (!workflowInfo().unsafe.isReplaying) {
  metrics.emit('workflow_started', 1);
}
```

If your goal is to always take action when something new is happening, check that `workflowInfo().unsafe.isReplayingHistoryEvents` is false instead. This will be false during read-only operations like queries and update validators. This is what the SDK's built-in logger uses internally.

---

# Cancel a Workflow - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/cancellation

> Cancel a TypeScript Activity from a Workflow using Heartbeats and a Heartbeat Timeout, and reset a Workflow Execution from its Event History.

## Cancel an Activity from a Workflow 

Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat
Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate
Activity is executed, the Activity Execution should send Heartbeats and set a
[Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is
still working.

When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic
needs to be performed, it can be done in a `finally` clause or inside a caught cancel error. However, for the Activity
to appear canceled the exception needs to be re-thrown.

> **📝 Note:**
>
> Unlike regular Activities, [Local Activities](/local-activity) can be canceled if they don't send Heartbeats. Local
> Activities are handled locally, and all the information needed to handle the cancellation logic is available in the same
> Worker process.
>

### Set the Activity cancellation type 

The `cancellationType` Activity option decides what the Workflow does when an Activity it is waiting on is cancelled,
whether by a cancellation request against the Workflow Execution or by the
[Cancellation Scope](/develop/typescript/workflows/cancellation-scopes) the Activity was started in.

- `ActivityCancellationType.TRY_CANCEL` is the default. The Workflow asks the Temporal Service to cancel the Activity and
  the Activity call fails right away with an `ActivityFailure` whose `cause` is a `CancelledFailure`. The Activity
  Execution keeps running until its next Heartbeat delivers the cancellation, so Workflow code that continues, for
  example to run compensation or to retry, can overlap with an Activity that is still executing.
- `ActivityCancellationType.WAIT_CANCELLATION_COMPLETED` asks for the cancellation and keeps the Activity call pending
  until the Activity Execution has acknowledged it or completed. Use this when the Workflow must not move on while the
  Activity might still be running. The Activity has to Heartbeat, or the wait lasts until one of its timeouts fires.
- `ActivityCancellationType.ABANDON` fails the Activity call right away and does not ask for the cancellation at all.
  The Activity Execution runs to completion on its own.

```typescript
import { ActivityCancellationType, proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { processOrder } = proxyActivities<typeof activities>({
  startToCloseTimeout: '10 minutes',
  heartbeatTimeout: '30 seconds',
  cancellationType: ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,
});
```

Child Workflows and Nexus Operations have their own cancellation type options, and both of those default to waiting
for the cancellation to complete. See
[`ActivityOptions.cancellationType`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions#cancellationtype)
in the API reference for the full definitions.

## Reset a Workflow Execution 

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

**Web UI**

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

**Temporal CLI**

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

---

# Cancellation scopes - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/cancellation-scopes

> Control cancellation in TypeScript Workflows with the CancellationScope tree, including nonCancellable, withTimeout, nesting, and shared promises.

## Cancellation scopes in TypeScript 

In the TypeScript SDK, Workflows are represented internally by a tree of cancellation scopes, each with cancellation
behaviors you can specify. By default, everything runs in the "root" scope.

Scopes are created using the [CancellationScope](https://typescript.temporal.io/api/classes/workflow.CancellationScope)
constructor or one of three static helpers:

- [cancellable(fn)](https://typescript.temporal.io/api/classes/workflow.CancellationScope#cancellable-1): Children are
  automatically cancelled when their containing scope is cancelled.
  - Equivalent to `new CancellationScope().run(fn)`.
- [nonCancellable(fn)](https://typescript.temporal.io/api/classes/workflow.CancellationScope#noncancellable):
  Cancellation does not propagate to children.
  - Equivalent to `new CancellationScope({ cancellable: false }).run(fn)`.
- [withTimeout(timeoutMs, fn)](https://typescript.temporal.io/api/classes/workflow.CancellationScope#withtimeout): If a
  timeout triggers before `fn` resolves, the scope is cancelled, triggering cancellation of any enclosed operations,
  such as Activities and Timers.
  - Equivalent to `new CancellationScope({ cancellable: true, timeout: timeoutMs }).run(fn)`.

Cancellations are applied to cancellation scopes, which can encompass an entire Workflow or just part of one. Scopes can
be nested, and cancellation propagates from outer scopes to inner ones. A Workflow's `main` function runs in the
outermost scope. Cancellations are handled by catching `CancelledFailure`s thrown by cancelable operations.

`CancellationScope.run()` and the static helpers mentioned earlier return native JavaScript promises, so you can use the
familiar Promise APIs like `Promise.all` and `Promise.race` to model your asynchronous logic. You can also use the
following APIs:

- `CancellationScope.current()`: Get the current scope.
- `scope.cancel()`: Cancel all operations inside a `scope`.
- `scope.run(fn)`: Run an async function within a `scope` and return the result of `fn`.
- `scope.cancelRequested`: A promise that resolves when a scope cancellation is requested, such as when Workflow code
  calls `cancel()` or the entire Workflow is cancelled by an external client.

When a `CancellationScope` is cancelled, it propagates cancellation in any child scopes and of any cancelable operations
created within it, such as the following:

- Activities
- Timers (created with the [sleep](https://typescript.temporal.io/api/namespaces/workflow#sleep) function)
- [Triggers](https://typescript.temporal.io/api/classes/workflow.Trigger)

### CancelledFailure

Timers and triggers throw [CancelledFailure](https://typescript.temporal.io/api/classes/common.CancelledFailure) when
cancelled; Activities and Child Workflows throw `ActivityFailure` and `ChildWorkflowFailure` with cause set to
`CancelledFailure`. One exception is when an Activity or Child Workflow is scheduled in an already cancelled scope (or
Workflow). In this case, they propagate the `CancelledFailure` that was thrown to cancel the scope.

To simplify checking for cancellation, use the
[isCancellation(err)](https://typescript.temporal.io/api/namespaces/workflow#iscancellation) function.

### Internal cancellation example

<!--SNIPSTART typescript-cancel-a-timer-from-workflow-->
[packages/test/src/workflows/cancel-timer-immediately.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/cancel-timer-immediately.ts)
```ts
import { CancelledFailure, CancellationScope, sleep } from '@temporalio/workflow';

export async function cancelTimer(): Promise<void> {
  // Timers and Activities are automatically cancelled when their containing scope is cancelled.
  try {
    await CancellationScope.cancellable(async () => {
      const promise = sleep(1); // <-- Will be cancelled because it is attached to this closure's scope
      CancellationScope.current().cancel();
      await promise; // <-- Promise must be awaited in order for `cancellable` to throw
    });
  } catch (e) {
    if (e instanceof CancelledFailure) {
      console.log('Timer cancelled 👍');
    } else {
      throw e; // <-- Fail the workflow
    }
  }
}
```
<!--SNIPEND-->

Alternatively, the preceding can be written as the following.

<!--SNIPSTART typescript-cancel-a-timer-from-workflow-alternative-impl-->
[packages/test/src/workflows/cancel-timer-immediately-alternative-impl.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/cancel-timer-immediately-alternative-impl.ts)
```ts
import { CancelledFailure, CancellationScope, sleep } from '@temporalio/workflow';

export async function cancelTimerAltImpl(): Promise<void> {
  try {
    const scope = new CancellationScope();
    const promise = scope.run(() => sleep(1));
    scope.cancel(); // <-- Cancel the timer created in scope
    await promise; // <-- Throws CancelledFailure
  } catch (e) {
    if (e instanceof CancelledFailure) {
      console.log('Timer cancelled 👍');
    } else {
      throw e; // <-- Fail the workflow
    }
  }
}
```
<!--SNIPEND-->

### External cancellation example

The following code shows how to handle Workflow cancellation by an external client while an Activity is running.

<!--SNIPSTART typescript-handle-external-workflow-cancellation-while-activity-running-->
[packages/test/src/workflows/handle-external-workflow-cancellation-while-activity-running.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/handle-external-workflow-cancellation-while-activity-running.ts)
```ts
import { CancellationScope, proxyActivities, isCancellation } from '@temporalio/workflow';
import type * as activities from '../activities';

const { httpPostJSON, cleanup } = proxyActivities<typeof activities>({
  startToCloseTimeout: '10m',
});

export async function handleExternalWorkflowCancellationWhileActivityRunning(url: string, data: any): Promise<void> {
  try {
    await httpPostJSON(url, data);
  } catch (err) {
    if (isCancellation(err)) {
      console.log('Workflow cancelled');
      // Cleanup logic must be in a nonCancellable scope
      // If we'd run cleanup outside of a nonCancellable scope it would've been cancelled
      // before being started because the Workflow's root scope is cancelled.
      await CancellationScope.nonCancellable(() => cleanup(url));
    }
    throw err; // <-- Fail the Workflow
  }
}
```
<!--SNIPEND-->

### `nonCancellable` example

`CancellationScope.nonCancellable` prevents cancellation from propagating to children.

<!--SNIPSTART typescript-non-cancellable-shields-children-->
[activities-cancellation-heartbeating/src/cancellation-scopes.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-cancellation-heartbeating/src/cancellation-scopes.ts)
```ts
export async function nonCancellable(url: string): Promise<any> {
  // Prevent Activity from being cancelled and await completion.
  // Note that the Workflow is completely oblivious and impervious to cancellation in this example.
  return CancellationScope.nonCancellable(() => httpGetJSON(url));
}
```
<!--SNIPEND-->

### withTimeout example

A common operation is to cancel one or more Activities if a deadline elapses. `withTimeout` creates a
`CancellationScope` that is automatically cancelled after a timeout.

<!--SNIPSTART typescript-multiple-activities-single-timeout-workflow-->
[packages/test/src/workflows/multiple-activities-single-timeout.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/multiple-activities-single-timeout.ts)
```ts
import { CancellationScope, proxyActivities } from '@temporalio/workflow';
import type * as activities from '../activities';

export function multipleActivitiesSingleTimeout(urls: string[], timeoutMs: number): Promise<any> {
  const { httpGetJSON } = proxyActivities<typeof activities>({
    startToCloseTimeout: timeoutMs,
  });

  // If timeout triggers before all activities complete
  // the Workflow will fail with a CancelledError.
  return CancellationScope.withTimeout(timeoutMs, () => Promise.all(urls.map((url) => httpGetJSON(url))));
}
```
<!--SNIPEND-->

### `scope.cancelRequested`

You can await `cancelRequested` to make a Workflow aware of cancellation while waiting on `nonCancellable` scopes.

<!--SNIPSTART typescript-cancel-requested-with-non-cancellable-->
[packages/test/src/workflows/cancel-requested-with-non-cancellable.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/cancel-requested-with-non-cancellable.ts)
```ts
import { CancellationScope, CancelledFailure, proxyActivities } from '@temporalio/workflow';
import type * as activities from '../activities';

const { httpGetJSON } = proxyActivities<typeof activities>({
  startToCloseTimeout: '10m',
});

export async function resumeAfterCancellation(url: string): Promise<any> {
  let result: any = undefined;
  const scope = new CancellationScope({ cancellable: false });
  const promise = scope.run(() => httpGetJSON(url));
  try {
    result = await Promise.race([scope.cancelRequested, promise]);
  } catch (err) {
    if (!(err instanceof CancelledFailure)) {
      throw err;
    }
    // Prevent Workflow from completing so Activity can complete
    result = await promise;
  }
  return result;
}
```
<!--SNIPEND-->

### Cancellation scopes and callbacks

Callbacks are not particularly useful in Workflows because all meaningful asynchronous operations return promises. In
the rare case that code uses callbacks and needs to handle cancellation, a callback can consume the
`CancellationScope.cancelRequested` promise.

<!--SNIPSTART typescript-cancellation-scopes-with-callbacks-->
[packages/test/src/workflows/cancellation-scopes-with-callbacks.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/cancellation-scopes-with-callbacks.ts)
```ts
import { CancellationScope } from '@temporalio/workflow';

function doSomething(callback: () => any) {
  setTimeout(callback, 10);
}

export async function cancellationScopesWithCallbacks(): Promise<void> {
  await new Promise<void>((resolve, reject) => {
    doSomething(resolve);
    CancellationScope.current().cancelRequested.catch(reject);
  });
}
```
<!--SNIPEND-->

### Nesting cancellation scopes

You can achieve complex flows by nesting cancellation scopes.

<!--SNIPSTART typescript-nested-cancellation-scopes-->
[packages/test/src/workflows/nested-cancellation.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/test/src/workflows/nested-cancellation.ts)
```ts
import { CancellationScope, proxyActivities, isCancellation } from '@temporalio/workflow';

import type * as activities from '../activities';

const { setup, httpPostJSON, cleanup } = proxyActivities<typeof activities>({
  startToCloseTimeout: '10m',
});

export async function nestedCancellation(url: string): Promise<void> {
  await CancellationScope.cancellable(async () => {
    await CancellationScope.nonCancellable(() => setup());
    try {
      await CancellationScope.withTimeout(1000, () => httpPostJSON(url, { some: 'data' }));
    } catch (err) {
      if (isCancellation(err)) {
        await CancellationScope.nonCancellable(() => cleanup(url));
      }
      throw err;
    }
  });
}
```
<!--SNIPEND-->

### Sharing promises between scopes

Operations like Timers and Activities are cancelled by the cancellation scope they were created in. Promises returned by
these operations can be awaited in different scopes.

<!--SNIPSTART typescript-shared-promise-scopes-->
[activities-cancellation-heartbeating/src/cancellation-scopes.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-cancellation-heartbeating/src/cancellation-scopes.ts)
```ts
export async function sharedScopes(): Promise<any> {
  // Start activities in the root scope
  const p1 = httpGetJSON('http://url1.ninja');
  const p2 = httpGetJSON('http://url2.ninja');

  const scopePromise = CancellationScope.cancellable(async () => {
    const first = await Promise.race([p1, p2]);
    // Does not cancel activity1 or activity2 as they're linked to the root scope
    CancellationScope.current().cancel();
    return first;
  });
  return await scopePromise;
  // The Activity that did not complete will effectively be cancelled when
  // Workflow completes unless the Activity is awaited:
  // await Promise.all([p1, p2]);
}
```
<!--SNIPEND-->

<!--SNIPSTART typescript-shield-awaited-in-root-scope-->
[activities-cancellation-heartbeating/src/cancellation-scopes.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-cancellation-heartbeating/src/cancellation-scopes.ts)
```ts
export async function shieldAwaitedInRootScope(): Promise<any> {
  let p: Promise<any> | undefined = undefined;

  await CancellationScope.nonCancellable(async () => {
    p = httpGetJSON('http://example.com'); // <-- Start activity in nonCancellable scope without awaiting completion
  });
  // Activity is shielded from cancellation even though it is awaited in the cancellable root scope
  return p;
}
```
<!--SNIPEND-->

---

# Child Workflows - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/child-workflows

> Start and manage Child Workflow Executions using Temporal's Child Workflow API, including setting Parent Close Policy, handling Events, and advanced Child Workflow options.

## Start a Child Workflow Execution 

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History.

The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
In TypeScript, awaiting `startChild()` or `executeChild()` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call resolves.
If you start a Child Workflow from a non-main context (for example, a Signal or Update handler), make sure the Parent Workflow doesn't complete before that call resolves.

To start a Child Workflow Execution and return a [handle](https://typescript.temporal.io/api/interfaces/workflow.ChildWorkflowHandle/) to it, use [startChild](https://typescript.temporal.io/api/namespaces/workflow/#startchild).

```ts
import { startChild } from '@temporalio/workflow';

export async function parentWorkflow(names: string[]) {
  const childHandle = await startChild(childWorkflow, {
    args: [name],
    // workflowId, // add business-meaningful workflow id here
    // // regular workflow options apply here, with two additions (defaults shown):
    // cancellationType: ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED,
    // parentClosePolicy: ParentClosePolicy.PARENT_CLOSE_POLICY_TERMINATE
  });
  // you can use childHandle to signal or get result here
  await childHandle.signal('anySignal');
  const result = childHandle.result();
  // you can use childHandle to signal, query, cancel, terminate, or get result here
}
```

To start a Child Workflow Execution and await its completion, use [executeChild](https://typescript.temporal.io/api/namespaces/workflow/#executechild).

By default, a child is scheduled on the same Task Queue as the parent.

<!--SNIPSTART typescript-child-workflow -->
[child-workflows/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/child-workflows/src/workflows.ts)
```ts
import { executeChild } from '@temporalio/workflow';

export async function parentWorkflow(...names: string[]): Promise<string> {
  const responseArray = await Promise.all(
    names.map((name) =>
      executeChild(childWorkflow, {
        args: [name],
        // workflowId, // add business-meaningful workflow id here
        // // regular workflow options apply here, with two additions (defaults shown):
        // cancellationType: ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED,
        // parentClosePolicy: ParentClosePolicy.PARENT_CLOSE_POLICY_TERMINATE
      }),
    ),
  );
  return responseArray.join('\n');
}
```
<!--SNIPEND-->

To control any running Workflow from inside a Workflow, use [getExternalWorkflowHandle(workflowId)](https://typescript.temporal.io/api/namespaces/workflow/#getexternalworkflowhandle).

```ts
import { getExternalWorkflowHandle, workflowInfo } from '@temporalio/workflow';

export async function terminateWorkflow() {
  const { workflowId } = workflowInfo(); // no await needed
  const handle = getExternalWorkflowHandle(workflowId); // sync function, not async
  await handle.cancel();
}
```

If the Child Workflow options aren't explicitly set, they inherit their values from the Parent Workflow options.
Two advanced options are unique to Child Workflows:

- [cancellationType](https://typescript.temporal.io/api/enums/proto.coresdk.child_workflow.ChildWorkflowCancellationType): Controls when to throw the `CanceledFailure` exception when a Child Workflow is canceled.
- `parentClosePolicy`: Explained in the next section.

If you need to cancel a Child Workflow Execution, use [cancellation scopes](/develop/typescript/workflows/cancellation-scopes).
A Child Workflow Execution is automatically cancelled when its containing scope is cancelled.

### Set a Parent Close Policy 

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

To specify how a Child Workflow reacts to a Parent Workflow reaching a Closed state, use the [`parentClosePolicy`](https://typescript.temporal.io/api/interfaces/workflow.ChildWorkflowOptions#parentclosepolicy) option.

<!--SNIPSTART typescript-child-workflow -->
[child-workflows/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/child-workflows/src/workflows.ts)
```ts
import { executeChild } from '@temporalio/workflow';

export async function parentWorkflow(...names: string[]): Promise<string> {
  const responseArray = await Promise.all(
    names.map((name) =>
      executeChild(childWorkflow, {
        args: [name],
        // workflowId, // add business-meaningful workflow id here
        // // regular workflow options apply here, with two additions (defaults shown):
        // cancellationType: ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED,
        // parentClosePolicy: ParentClosePolicy.PARENT_CLOSE_POLICY_TERMINATE
      }),
    ),
  );
  return responseArray.join('\n');
}
```
<!--SNIPEND-->

---

# Continue-As-New - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/continue-as-new

> Use Temporal's Continue-As-New in TypeScript to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters.

This page covers the following for TypeScript developers:

- [What is Continue-As-New?](#what)
- [Use Continue-As-New](#how)
- [When is it right to Continue-As-New?](#when)
- [Test Continue-As-New](#how-to-test)

## What is Continue-As-New? 

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution.
You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History.
It also receives your Workflow's usual parameters.

## Use Continue-As-New with the TypeScript SDK 

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically set to `None` for the original caller of the Workflow.

[View the source code](https://github.com/temporalio/samples-typescript/blob/main/message-passing/safe-message-handlers/src/workflows.ts) in the context of the rest of the application code.

```typescript
export interface ClusterManagerInput {
  state?: ClusterManagerState;
}

export async function clusterManagerWorkflow(input: ClusterManagerInput = {}): Promise<ClusterManagerStateSummary> {

````
The test hook in the above snippet is covered [below](#how-to-test).

Inside your Workflow, call the [`continueAsNew()`](https://typescript.temporal.io/api/namespaces/workflow#continueasnew) function with the same type.
This stops the Workflow right away and starts a new one.

[View the source code](https://github.com/temporalio/samples-typescript/blob/main/message-passing/safe-message-handlers/src/workflows.ts) in the context of the rest of the application code.

```typescript
return await wf.continueAsNew<typeof clusterManagerWorkflow>({ 
  state: manager.getState(),
  testContinueAsNew: input.testContinueAsNew 
});
````

### Considerations for Workflows with message handlers 

If you use Updates or Signals, don't call Continue-as-New from the handlers.
Instead, wait for your handlers to finish in your main Workflow before you run `ContinueAsNew`.
See the [`allHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance.

## When is it right to Continue-As-New with the TypeScript SDK? 

Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New.
Call `wf.workflowInfo().continueAsNewSuggested` to check if it's time.

## Test Continue-As-New with the TypeScript SDK 

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive.
Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, when `testContinueAsNew == true`, this sample creates a test-only variable called `this.maxHistoryLength` and sets it to a small value.
A helper method in the Workflow checks it each time it considers using Continue-as-New:

[View the source code](https://github.com/temporalio/samples-typescript/blob/main/message-passing/safe-message-handlers/src/client.ts) in the context of the rest of the application code.

```typescript
shouldContinueAsNew(): boolean {
  if (wf.workflowInfo().continueAsNewSuggested) {
    return true;
  }

  // This is just for ease-of-testing. In production, we trust temporal to tell us when to continue-as-new.
  if (this.maxHistoryLength !== undefined && wf.workflowInfo().historyLength > this.maxHistoryLength) {
    return true;
  }

  return false;
}
```

---

# Workflow message passing - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/message-passing

> Develop with Queries, Signals, and Updates with the Temporal TypeScript SDK.

A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates.
The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values.
Temporal Clients use messages to read Workflow state and control its execution.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic.
This page introduces these features for the Temporal TypeScript SDK.

## Write message handlers 

> **ℹ️ Info:**
> The code that follows is part of a working [message-passing sample](https://github.com/temporalio/samples-typescript/tree/main/message-passing/introduction).

Follow these guidelines when writing your message handlers:

- Define a message type as a global variable using [`defineQuery`](https://typescript.temporal.io/api/namespaces/workflow#definequery), [`defineSignal`](https://typescript.temporal.io/api/namespaces/workflow#definesignal), or [`defineUpdate`](https://typescript.temporal.io/api/namespaces/workflow#defineupdate).
  This is what your client code will use to send a message to the workflow.
- Message handlers are defined by calling [`workflow.setHandler`](https://typescript.temporal.io/api/namespaces/workflow#sethandler) in your Workflow function.
- The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion).
- Prefer using a single object over multiple input parameters.
  A single object allows you to add fields without changing the signature.

### Query handlers 

A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution:

```typescript
export enum Language {
  ARABIC = 'ARABIC',
  CHINESE = 'CHINESE',
  ENGLISH = 'ENGLISH',
  FRENCH = 'FRENCH',
  HINDI = 'HINDI',
  PORTUGUESE = 'PORTUGUESE',
  SPANISH = 'SPANISH',
}

interface GetLanguagesInput {
  includeUnsupported: boolean;
}

// 👉 Use the object returned by defineQuery to set the query handler in
// Workflow code, and when sending the Query in Client code.
export const getLanguages = wf.defineQuery<Language[], [GetLanguagesInput]>('getLanguages');

export async function greetingWorkflow(): Promise<string> {
  const greetings: Partial<Record<Language, string>> = {
    [Language.CHINESE]: '你好，世界',
    [Language.ENGLISH]: 'Hello, world',
  };

  wf.setHandler(getLanguages, (input: GetLanguagesInput): Language[] => {
    // 👉 A Query handler returns a value: it must not mutate the Workflow state
    // and can't perform async operations.
    if (input.includeUnsupported) {
      return Object.values(Language);
    } else {
      return Object.keys(greetings) as Language[];
    }
  });

  ...
}
```

- A Query handler cannot be `async`.
  You can't perform async operations like executing an Activity in a Query handler.

- `setHandler` can take `QueryHandlerOptions` (such as `description`) as described in the API reference docs for [`workflow.setHandler`](https://typescript.temporal.io/api/namespaces/workflow#sethandler).

### Signal handlers 

A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow:

```typescript
// 👉 Use the object returned by defineSignal to set the Signal handler in
// Workflow code, and to send the Signal from Client code.
export const approve = wf.defineSignal<[ApproveInput]>('approve');

export async function greetingWorkflow(): Promise<string> {
  let approvedForRelease = false;
  let approverName: string | undefined;

  wf.setHandler(approve, (input) => {
    // 👉 A Signal handler mutates the Workflow state but cannot return a value.
    approvedForRelease = true;
    approverName = input.name;
  });

  ...
}
...
```

- The handler cannot return a value.
  The response is sent immediately from the server, without waiting for the Workflow to process the Signal.

- Signal and Update handlers can be `async`.
  This allows you to use Activities, Child Workflows, durable [`workflow.sleep`](https://typescript.temporal.io/api/namespaces/workflow#sleep) Timers, [`workflow.condition`](https://typescript.temporal.io/api/namespaces/workflow#condition) conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Signal and Update handlers.

- If your Workflow needs to do some async initialization before handling a Signal or Update, use [`workflow.condition`](https://typescript.temporal.io/api/namespaces/workflow#condition) inside your handler to wait until initialization has completed.

- `setHandler` can take `SignalHandlerOptions` (such as `description` and `unfinishedPolicy`) as described in the API reference docs for [`workflow.setHandler`](https://typescript.temporal.io/api/namespaces/workflow#sethandler).

### Update handlers and validators 

An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution.
It can change the Workflow state, control its flow, and return a result.
The sender must wait until the Worker accepts or rejects the Update.
The sender may wait further to receive a returned value or an exception if something goes wrong:

```typescript
// 👉 Use the object returned by defineUpdate to set the Update handler in
// Workflow code, and to send Updates from Client code.
export const setLanguage = wf.defineUpdate<Language, [Language]>('setLanguage');

export async function greetingWorkflow(): Promise<string> {
  const greetings: Partial<Record<Language, string>> = {
    [Language.CHINESE]: '你好，世界',
    [Language.ENGLISH]: 'Hello, world',
  };

  let language = Language.ENGLISH;

wf.setHandler(
    setLanguage,
    (newLanguage: Language) => {
      // 👉 An Update handler can mutate the Workflow state and return a value.
      const previousLanguage = language;
      language = newLanguage;
      return previousLanguage;
    },
    {
      validator: (newLanguage: Language) => {
        // 👉 Update validators are optional
        if (!(newLanguage in greetings)) {
          throw new Error(`${newLanguage} is not supported`);
        }
      },
    }
  );

  ...
}
```

- `setHandler` can take `UpdateHandlerOptions` (such as `validator`, `description` and `unfinishedPolicy`) as described in the API reference docs for [`workflow.setHandler`](https://typescript.temporal.io/api/namespaces/workflow#sethandler).

- About validators:
  - Use validators to reject an Update before it is written to History.
    Validators are always optional.
    If you don't need to reject Updates, you don't need a validator.
  - To set a validator, pass the validator function in `UpdateHandlerOptions` when calling [`workflow.setHandler`](https://typescript.temporal.io/api/namespaces/workflow#sethandler).
    The validator must be a non-async function that accepts the same argument types as the handler and returns `void`.

- Accepting and rejecting Updates with validators:
  - To reject an Update, throw an error of any type in the validator.
  - Without a validator, Updates are always accepted.
- Validators and Event History:
  - The `WorkflowExecutionUpdateAccepted` event is written into History whether the acceptance was automatic or due to a validator function not throwing an error.
  - When a Validator throws an error, the Update is rejected and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History.
    The caller receives an "Update failed" error.

- Use [`workflow.currentUpdateInfo`](https://typescript.temporal.io/api/namespaces/workflow#current_update_info) to obtain information about the current Update.
  This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing).
- Update and Signal handlers can be `async`, letting them use Activities, Child Workflows, durable [`workflow.sleep`](https://typescript.temporal.io/api/namespaces/workflow#sleep) Timers, [`workflow.condition`](https://typescript.temporal.io/api/namespaces/workflow#condition) conditions, and more.
  See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for safe usage guidelines.
- If your Workflow needs to do some async initialization before handling an Update or Signal, use [`workflow.condition`](https://typescript.temporal.io/api/namespaces/workflow#condition) inside your handler to wait until initialization has completed.

## Send messages 

To send Queries, Signals, or Updates, you call methods on a [WorkflowHandle](https://typescript.temporal.io/api/namespaces/client#workflowhandle) object:

- Use [`client.workflow.start`](https://typescript.temporal.io/api/classes/client.WorkflowClient#start) and return its handle.

- Use [`client.workflow.getHandle`](https://typescript.temporal.io/api/classes/client.WorkflowClient#gethandle) to retrieve a Workflow handle by its Workflow Id.

For example:

```typescript
const handle = await client.workflow.start(greetingWorkflow, {
  taskQueue: 'my-task-queue',
  args: [myArg],
  workflowId: 'my-workflow-id',
});
```

To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition.

> **⚠️ Warning:**
> Using Continue-as-New and Updates
>
> - Temporal _does not_ support Continue-as-New functionality within Update handlers.
> - Complete all handlers _before_ using Continue-as-New.
> - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution.
>

### Send a Query 

Use [`WorkflowHandle.query`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#query) to send a Query to a Workflow Execution:

```typescript
const supportedLanguages = await handle.query(getLanguages, {
  includeUnsupported: false,
});
```

- Sending a Query doesn’t add events to a Workflow's Event History.

- You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period.
  This includes Workflows that have completed, failed, or timed out.
  Querying terminated Workflows is not safe and, therefore, not supported.

- A Worker must be online and polling the Task Queue to process a Query.

### Send a Signal 

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution.
However, you can only send Signals to Workflow Executions that haven’t closed.

#### Send a Signal from a Client 

Use [WorkflowHandle.signal](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle#signal) to send a Signal:

```typescript
await handle.signal(greetingWorkflow.approve, { name: 'me' });
```

- The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution.

- The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History.

### Send a Signal from a Workflow 

A Workflow can send a Signal to another Workflow, in which case it's called an _External Signal_.
Use [`getExternalWorkflowHandle`](https://typescript.temporal.io/api/namespaces/workflow#getExternalWorkflowHandle):

```typescript
import { getExternalWorkflowHandle } from '@temporalio/workflow';
import { joinSignal } from './other-workflow';

export async function yourWorkflowThatSignals() {
  const handle = getExternalWorkflowHandle('workflow-id-123');
  await handle.signal(joinSignal, { userId: 'user-1', groupId: 'group-1' });
}
```

When an External Signal is sent:

- A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History.
- A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History.

The `getExternalWorkflowHandle` method helps ensure that the Workflow Definition remains deterministic.
Recall that one aspect of a deterministic Workflow Definition means not directly making network calls from the Workflow.
This means that developers cannot use a Temporal Client directly within the Workflow code to send Signals or start other Workflows.
Instead, to communicate between Workflows, we use `getExternalWorkflowHandle` to both ensure that the Workflow Definition remains deterministic and also that these interactions are recorded as Events in the Workflow's Event History.

### Signal-With-Start 

Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running.
Use [`Client.workflow.signalWithStart`](https://typescript.temporal.io/api/classes/client.WorkflowClient#signalwithstart):

```typescript
import { Client } from '@temporalio/client';
import { joinSignal, yourWorkflow } from './workflows';

const client = new Client();

await client.workflow.signalWithStart(yourWorkflow, {
  workflowId: 'workflow-id-123',
  taskQueue: 'my-taskqueue',
  args: [{ foo: 1 }],
  signal: joinSignal,
  signalArgs: [{ userId: 'user-1', groupId: 'group-1' }],
});
```

Signal-With-Start is limited to Client use.
It cannot be called from a Workflow.

### Send an Update 

An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result.

A client sending an Update must wait until the Server delivers the Update to a Worker.
Workers must be available and responsive.
If you need a response as soon as the Server receives the request, use a Signal instead.
You can't send Updates directly from one Workflow to another or perform an Update equivalent of Signal-With-Start. If you need to send Updates across Workflows, like to Child Workflows, use an Activity.

- `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation.
- `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished.

To send an Update to a Workflow Execution, you can:

- Call [`WorkflowHandle.executeUpdate`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#executeUpdate) and wait for the Update to complete.
  This code fetches an Update result:

  ```typescript
  let previousLanguage = await handle.executeUpdate(setLanguage, {
    args: [Language.CHINESE],
  });
  ```

- Send [`WorkflowHandle.startUpdate`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#startUpdate) to receive an [`WorkflowUpdateHandle`](https://typescript.temporal.io/api/interfaces/client.WorkflowUpdateHandle) as soon as the Update is accepted or rejected.

  - Use this `UpdateHandle` later to fetch your results.
  - `async` Update handlers normally perform long-running asynchronous operations, such as calling an Activity.
  - `startUpdate` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete.

  For example:

  ```typescript
  const updateHandle = await handle.startUpdate(setLanguage, {
    args: [Language.ENGLISH],
    waitForStage: WorkflowUpdateStage.ACCEPTED,
  });
  previousLanguage = await updateHandle.result();
  ```

  For more details, see the "Async handlers" section.

To obtain an Update handle, you can:

- Use [`WorkflowHandle.startUpdate`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#startUpdate) to start an Update and return the handle, as shown in the preceding example.
- Use [`getUpdateHandle`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#getupdatehandle) to fetch a handle for an in-progress Update using the Update ID.

#### Update-With-Start 

> **💡 Tip:**
>
> For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended.
>

[Update-with-Start](/sending-messages#update-with-start) lets you
[send an Update](/develop/typescript/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists:

- If the Workflow exists, the Update is processed.
- If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute.

Use [`executeUpdateWithStart`](https://typescript.temporal.io/api/classes/client.WorkflowClient#executeUpdateWithStart) to start an Update and wait for the result in one go.

Alternatively, use [`startUpdateWithStart`](https://typescript.temporal.io/api/classes/client.WorkflowClient#startUpdateWithStart) to start an Update and receive a [`WorkflowUpdateHandle`](https://typescript.temporal.io/api/interfaces/client.WorkflowUpdateHandle), and then use `await updateHandle.result()` to retrieve the result from the Update.

These calls return once the requested Update wait stage has been reached, or when the request times out.

You will need to provide a [`WithStartWorkflowOperation`](https://typescript.temporal.io/api/classes/client.WithStartWorkflowOperation) to define the Workflow that will be started if necessary, and its arguments.
You must specify a [WorkflowIdConflictPolicy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) when creating the `WithStartWorkflowOperation`.
Note that a `WithStartWorkflowOperation` can only be used once.

Here's an example taken from the [early-return](https://github.com/temporalio/samples-typescript/tree/main/early-return) sample:

```typescript
const startWorkflowOperation = new WithStartWorkflowOperation.create(
  transactionWorkflow,
  {
    workflowId,
    args: [transactionID],
    taskQueue: 'early-return',
    workflowIdConflictPolicy: 'FAIL',
  },
);

const earlyConfirmation = await client.workflow.executeUpdateWithStart(
  getTransactionConfirmation,
  {
    startWorkflowOperation,
  },
);

const wfHandle = await startWorkflowOperation.workflowHandle();
const finalReport = await wfHandle.result();
```

> **ℹ️ Info:**
> SEND MESSAGES WITHOUT TYPE SAFETY
>
> In real-world development, sometimes you may be unable to import message type objects defined by `defineQuery`, `defineSignal`, or `defineUpdate`.
> When you don't have access to the Workflow Definition or it isn't written in TypeScript, you can still use APIs that aren't type-safe, and dynamic method invocation.
> Pass message type names instead of message type objects to:
>
> - [`client.workflow.start`](https://typescript.temporal.io/api/classes/client.WorkflowClient#start)
> - [`WorkflowHandle.query`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#query)
> - [WorkflowHandle.signal](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle#signal)
> - [`WorkflowHandle.executeUpdate`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#executeUpdate)
> - [`WorkflowHandle.startUpdate`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle/#startUpdate)
>
> Pass Workflow IDs to these APIs to get Workflow handles:
>
> - [`client.workflow.getHandle`](https://typescript.temporal.io/api/classes/client.WorkflowClient#gethandle)
> - [`getExternalWorkflowHandle`](https://typescript.temporal.io/api/namespaces/workflow#getExternalWorkflowHandle).
>

## Message handler patterns 

This section covers common write operations, such as Signal and Update handlers.
It doesn't apply to pure read operations, like Queries or Update Validators.

> **💡 Tip:**
>
> For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing), and [this sample](https://github.com/temporalio/samples-typescript/blob/main/message-passing/safe-message-handlers/README.md) demonstrating safe `async` message handling.
>

### Use async handlers 

Signal and Update handlers can be `async` functions.
Using `async` allows you to use `await` with Activities, Child Workflows, durable [`workflow.sleep`](https://typescript.temporal.io/api/namespaces/workflow#sleep) Timers, [`workflow.condition`](https://typescript.temporal.io/api/namespaces/workflow#condition) conditions, and so on.
This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at `await` calls.
To use `async` handlers safely, understand what can go wrong.
See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, the [Safe message handlers](https://github.com/temporalio/samples-typescript/blob/main/message-passing/safe-message-handlers/README.md) sample, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below.

The following code executes an Activity that makes a network call to a remote service.
It modifies the Update handler from earlier on this page, turning it into an `async` function:

```typescript
// 👉 Use the objects returned by defineUpdate to set the Update handler in
// Workflow code, and to send Updates from Client code.
export const setLanguageUsingActivity = wf.defineUpdate<Language, [Language]>('setLanguageUsingActivity');

export async function greetingWorkflow(): Promise<string> {
  const greetings: Partial<Record<Language, string>> = {
    [Language.CHINESE]: '你好，世界',
    [Language.ENGLISH]: 'Hello, world',
  };

  let language = Language.ENGLISH;

  const lock = new Mutex();
  wf.setHandler(setLanguageUsingActivity, async (newLanguage) => {
    // 👉 An Update handler can mutate the Workflow state and return a value.
    // 👉 Since this update handler is async, it can execute an activity.
    if (!(newLanguage in greetings)) {
      // 👉 Do the following with the lock held to ensure that multiple calls to set_language are processed in order.
      await lock.runExclusive(async () => {
        if (!(newLanguage in greetings)) {
          const greeting = await callGreetingService(newLanguage);
          if (!greeting) {
            // 👉 An update validator cannot be async, so cannot be used to check that the remote
            // call_greeting_service supports the requested language. Raising ApplicationError
            // will fail the Update, but the WorkflowExecutionUpdateAccepted event will still be
            // added to history.
            throw new wf.ApplicationFailure(`${newLanguage} is not supported by the greeting service`);
          }
          greetings[newLanguage] = greeting;
        }
      });
    }
    const previousLanguage = language;
    language = newLanguage;
    return previousLanguage;
  });
  ...
}
```

After updating the code to use `async`, your Update handler can schedule an Activity and await the result.
Although an `async` Signal handler can also execute an Activity, using an Update handler allows the client to receive a result or error once the Activity completes.
This lets your client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, and so on.

### Add wait conditions to block

Sometimes, `async` Signal or Update handlers need to meet certain conditions before they should continue.
You can use [`workflow.condition`](https://typescript.temporal.io/api/namespaces/workflow#condition) to prevent the code from proceeding until a condition is true.
You specify the condition by passing a function that returns `true` or `false`.
This is an important feature that helps you control your handler logic.

Here are three important use cases for `workflow.condition`:

- Waiting for a Signal or Update to arrive
- Waiting in a handler until it is appropriate to continue.
- Waiting in the main Workflow until all active handlers have finished.

#### Wait for a Signal or Update to arrive

It's common to use `workflow.condition` to wait for a particular Signal or Update to be sent by a Client:

```typescript
export async function greetingWorkflow(): Promise<string> {
  let approvedForRelease = false;
  let approverName: string | undefined;

  wf.setHandler(approve, (input) => {
    approvedForRelease = true;
    approverName = input.name;
  });
  ...

  await wf.condition(() => approvedForRelease);
  ...
}
```

#### Use wait conditions in handlers

It's common to use a Workflow wait condition in a handler.
For example, suppose your Workflow has a mutable variable `readyForUpdateToExecute` that indicates whether your Update handler should be allowed to start executing.
You can use `workflow.condition` in the handler to make the handler pause until the condition is met:

```typescript
let readyForUpdateToExecute = false;

wf.setHandler(myUpdate, async (input: MyUpdateInput): Promise<MyUpdateOutput> => {
  await wf.condition(() => readyForUpdateToExecute);
  ...
});
```

Remember: handlers can execute before the main Workflow method starts.

You can also use wait conditions anywhere else in the handler to wait for a specific condition to become true.
This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become true.

#### Ensure your handlers finish before the Workflow completes 

Workflow wait conditions can ensure your handler completes before a Workflow finishes.
When your Workflow uses `async` Signal or Update handlers, your main Workflow method can return or Continue-as-New while a handler is still waiting on an async task, such as an Activity result.
The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying retrieve Update results.
Use [`workflow.condition`](https://typescript.temporal.io/api/namespaces/workflow#condition) and [`allHandlersFinished`](https://typescript.temporal.io/api/namespaces/workflow#condition#allhandlersfinished) to address this problem and allow your Workflow to end smoothly:

```typescript
export async function myWorkflow(): Promise<MyWorkflowOutput> {
  await wf.condition(wf.allHandlersFinished);
  return workflowOutput;
}
```

By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions.
You can silence these warnings on a per-handler basis by setting the `unfinishedPolicy` in `SignalHandlerOptions` or `UpdateHandlerOptions` when calling [`workflow.setHandler`](https://typescript.temporal.io/api/namespaces/workflow#sethandler)

See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information.

### Use a lock to prevent concurrent handler execution 

Concurrent processes can interact in unpredictable ways.
Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously.
Here's an example of a pathological case:

```typescript
export async function myWorkflow(): Promise<MyWorkflowOutput> {
  let x = 0;
  let y = 0;
  wf.setHandler(mySignal, async () => {
    const data = await myActivity();
    x = data.x;

    // 🐛🐛 Bug!! If multiple instances of this handler are executing
    // concurrently, then there may be times when the Workflow has x from one
    // Activity execution and y from another.
    await wf.sleep(500); // or await anything else

    y = data.y;
  });
  ...
}
```

Coordinating access using a lock (also known as a mutex) corrects this code.
Locking makes sure that only one handler instance can execute a specific section of code at any given time:

```typescript
import { Mutex } from 'async-mutex';  // https://github.com/DirtyHairy/async-mutex

...

export async function myWorkflow(): Promise<MyWorkflowOutput> {
  let x = 0;
  let y = 0;
  const lock = new Mutex();

  wf.setHandler(mySignal, async () => {
    await lock.runExclusive(async () => {
      const data = await myActivity();
      x = data.x;

      // ✅ OK: node's event loop may switch now to a different handler
      // execution, or to the main workflow function, but no other execution of
      // this handler can run until this execution finishes.
      await wf.sleep(500); // or await anything else

      y = data.y;
    });
  });
  return {
    name: 'hello',
  };
}
```

## Message handler troubleshooting 

When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors:

- **The client can't contact the server**:
  You'll receive a [`client.ServiceError`](https://typescript.temporal.io/api/classes/client.ServiceError) on which the `cause.code` attribute is [gRPC status code](https://grpc.io/docs/guides/status-codes/) 14 `UNAVAILABLE` (after some retries).

- **The workflow does not exist**:
  You'll receive an [`common.WorkflowNotFoundError`](https://typescript.temporal.io/api/classes/common.WorkflowNotFoundError) error.

### Problems when sending a Signal 

When using Signal, the two errors described above are the only errors that will result from your requests.

For Queries and Updates, the client waits for a response from the Worker and therefore additional errors may occur during the handler Execution by the Worker.

### Problems when sending an Update 

When working with Updates, you may encounter these problems:

- **No Workflow Workers are polling the Task Queue**:
  Your request will be retried by the SDK Client indefinitely.

- **Update failed**: You'll receive a [`client.WorkflowUpdateFailedError`](https://typescript.temporal.io/api/classes/client.WorkflowUpdateFailedError) exception.
  There are two ways this can happen:

  - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler.

  - The Update failed after having been accepted.

  Update failures are like [Workflow failures](/references/failures#errors-in-workflows).
  Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.
  These might include:

      - A failed Child Workflow
      - A failed Activity (if the Activity retries have been set to a finite number)
      - The Workflow author raising `ApplicationFailure`

- **The handler caused the Workflow Task to fail**:
  A [Workflow Task Failure](/references/failures#errors-in-workflows) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage:
  - If the request hasn't been accepted by the server, you receive a [`client.ServiceError`](https://typescript.temporal.io/api/classes/client.ServiceError) on which the `cause.code` attribute is [gRPC status code](https://grpc.io/docs/guides/status-codes/) 9 `FAILED_PRECONDITION` (after some retries).
  - If the request has been accepted, it is durable.
    Once the Workflow is healthy again after a code deploy, use an [`WorkflowUpdateHandle`](https://typescript.temporal.io/api/interfaces/client.WorkflowUpdateHandle) to fetch the Update result.

- **The Workflow finished while the Update handler execution was in progress**:
  You'll receive a [`client.ServiceError`](https://typescript.temporal.io/api/classes/client.ServiceError) on which the `cause.code` attribute is [gRPC status code](https://grpc.io/docs/guides/status-codes/) 5 `NOT_FOUND`.
  This happens if the Workflow finished while the Update handler execution was in progress, for example because

  - The Workflow was canceled or failed.

  - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers).

### Problems when sending a Query 

When working with Queries, you may encounter these errors:

- **There is no Workflow Worker polling the Task Queue**:
  You'll receive a [`client.ServiceError`](https://typescript.temporal.io/api/classes/client.ServiceError) on which the `cause.code` attribute is [gRPC status code](https://grpc.io/docs/guides/status-codes/) 9 `FAILED_PRECONDITION`.

- **Query failed**:
  You'll receive a [`client.QueryNotRegisteredError`](https://typescript.temporal.io/api/classes/client.QueryNotRegisteredError) exception if something goes wrong during a Query.
  Any error in a Query handler will trigger this error.
  This differs from Signal and Update requests, where errors can lead to Workflow Task Failure instead.

- **The handler caused the Workflow Task to fail.**
  This would happen, for example, if the Query handler blocks the thread for too long without yielding.

## Define Signals and Queries statically or dynamically 

- Handlers for both Signals and Queries can take arguments, which can be used inside `setHandler` logic.
- Only Signal Handlers can mutate state, and only Query Handlers can return values.

* [Define Signals and Queries statically](#static-signals-and-queries)
* [Define Signals and Queries dynamically](#dynamic-signals-and-queries)

### Define Signals and Queries statically 

If you know the name of your Signals and Queries upfront, we recommend declaring them outside the Workflow Definition.

<!--SNIPSTART typescript-blocked-workflow-->
[signals-queries/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/signals-queries/src/workflows.ts)
```ts
import * as wf from '@temporalio/workflow';

export const unblockSignal = wf.defineSignal('unblock');
export const isBlockedQuery = wf.defineQuery<boolean>('isBlocked');

export async function unblockOrCancel(): Promise<void> {
  let isBlocked = true;
  wf.setHandler(unblockSignal, () => void (isBlocked = false));
  wf.setHandler(isBlockedQuery, () => isBlocked);
  wf.log.info('Blocked');
  try {
    await wf.condition(() => !isBlocked);
    wf.log.info('Unblocked');
  } catch (err) {
    if (err instanceof wf.CancelledFailure) {
      wf.log.info('Cancelled');
    }
    throw err;
  }
}
```
<!--SNIPEND-->

This technique helps provide type safety because you can export the type signature of the Signal or Query to be called by the Client.

### Define Signals and Queries dynamically 

For more flexible use cases, you might want a dynamic Signal (such as a generated ID).
You can handle it in two ways:

- Avoid making it dynamic by collapsing all Signals into one handler and move the ID to the payload.
- Actually make the Signal name dynamic by inlining the Signal definition per handler.

```ts
import * as wf from '@temporalio/workflow';

// "fat handler" solution
wf.setHandler(`genericSignal`, (payload) => {
  switch (payload.taskId) {
    case taskAId:
      // do task A things
      break;
    case taskBId:
      // do task B things
      break;
    default:
      throw new Error('Unexpected task.');
  }
});

// "inline definition" solution
wf.setHandler(wf.defineSignal(`task-${taskAId}`), (payload) => {
  /* do task A things */
});
wf.setHandler(wf.defineSignal(`task-${taskBId}`), (payload) => {
  /* do task B things */
});

// utility "inline definition" helper
const inlineSignal = (signalName, handler) =>
  wf.setHandler(wf.defineSignal(signalName), handler);
inlineSignal(`task-${taskBId}`, (payload) => {
  /* do task B things */
});
```

    API Design FAQs

**Why not "new Signal" and "new Query"?**

The semantic of `defineSignal` and `defineQuery` is intentional.
They return Signal and Query **definitions**, not unique instances of Signals and Queries themselves
The following is their [entire source code](https://github.com/temporalio/sdk-typescript/blob/803d95ada755736c17c5af719fa6c447e4cd75ac/packages/workflow/src/workflow.ts#L1268-L1296):

```ts
/**
 * Define a signal method for a Workflow.
 */
export function defineSignal<Args extends any[] = []>(
  name: string,
): SignalDefinition<Args> {
  return {
    type: 'signal',
    name,
  };
}

/**
 * Define a query method for a Workflow.
 */
export function defineQuery<Ret, Args extends any[] = []>(
  name: string,
): QueryDefinition<Ret, Args> {
  return {
    type: 'query',
    name,
  };
}
```

Signals and Queries are instantiated only in `setHandler` and are specific to particular Workflow Executions.

These distinctions might seem minor, but they model how Temporal works under the hood, because Signals and Queries are messages identified by "just strings" and don't have meaning independent of the Workflow having a listener to handle them.
This will be clearer if you refer to the Client-side APIs.

**Why setHandler and not OTHER_API?**

We named it `setHandler` instead of `subscribe` because a Signal or Query can have only one "handler" at a time, whereas `subscribe` could imply an Observable with multiple consumers and is a higher-level construct.

```ts
wf.setHandler(MySignal, handlerFn1);
wf.setHandler(MySignal, handlerFn2); // replaces handlerFn1
```

If you are familiar with [RxJS](https://rxjs.dev/), you are free to wrap your Signals and Queries into Observables if you want, or you could dynamically reassign the listener based on your business logic or Workflow state.

---

# Schedules - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/schedules

> Create, backfill, pause, trigger, and update Schedules with the TypeScript SDK, plus Temporal Cron Jobs and Start Delay.

This page shows how to do the following:

- [Schedule a Workflow](#schedule-a-workflow)
  - [Create a Schedule](#create-schedule)
  - [Backfill a Schedule](#backfill-schedule)
  - [Delete a Schedule](#delete-schedule)
  - [Describe a Schedule](#describe-schedule)
  - [List a Schedule](#list-schedule)
  - [Pause a Schedule](#pause-schedule)
  - [Trigger a Schedule](#trigger-schedule)
  - [Update a Schedule](#update-schedule)
- [Temporal Cron Jobs](#temporal-cron-jobs)
- [Start Delay](#start-delay)

## Schedule a Workflow 

Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes.

Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process.

Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy).
If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions.

### Create a Schedule 

The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands.

> **💡 Tip:**
> Schedule Auto-Deletion
>
> Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again.
> The Temporal Service doesn't guarantee when this removal will happen.
>

<!--SNIPSTART typescript-create-a-scheduled-workflow-->
[schedules/src/start-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/start-schedule.ts)
```ts
async function run() {
  const config = loadClientConnectConfig();
  const connection = await Connection.connect(config.connectionOptions);
  const client = new Client({ connection });

  // https://typescript.temporal.io/api/classes/client.ScheduleClient#create
  const schedule = await client.schedule.create({
    action: {
      type: 'startWorkflow',
      workflowType: reminder,
      args: ['♻️ Dear future self, please take out the recycling tonight. Sincerely, past you ❤️'],
      taskQueue: 'schedules',
    },
    scheduleId: 'sample-schedule',
    policies: {
      catchupWindow: '1 day',
      overlap: ScheduleOverlapPolicy.ALLOW_ALL,
    },
    spec: {
      intervals: [{ every: '10s' }],
      // or periodic calendar times:
      // calendars: [
      //   {
      //     comment: 'every wednesday at 8:30pm',
      //     dayOfWeek: 'WEDNESDAY',
      //     hour: 20,
      //     minute: 30,
      //   },
      // ],
      // or a single datetime:
      // calendars: [
      //   {
      //     comment: '1/1/23 at 9am',
      //     year: 2023,
      //     month: 1,
      //     dayOfMonth: 1,
      //     hour: 9,
      //   },
      // ],
    },
  });
```
<!--SNIPEND-->

### Backfill a Schedule 

The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time.

<!--SNIPSTART typescript-backfill-a-scheduled-workflow-->
[schedules/src/backfill-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/backfill-schedule.ts)
```ts
function subtractMinutes(minutes: number): Date {
  const now = new Date();
  return new Date(now.getTime() - minutes * 60 * 1000);
}

async function run() {
  const client = new Client({
    connection: await Connection.connect(),
  });

  const backfillOptions: Backfill = {
    start: subtractMinutes(10),
    end: subtractMinutes(9),
    overlap: ScheduleOverlapPolicy.ALLOW_ALL,
  };

  const handle = client.schedule.getHandle('sample-schedule');
  await handle.backfill(backfillOptions);

  console.log(`Schedule is now backfilled.`);
}
```
<!--SNIPEND-->

### Delete a Schedule 

The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule.

<!--SNIPSTART typescript-delete-a-scheduled-workflow-->
[schedules/src/delete-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/delete-schedule.ts)
```ts
async function run() {
  const client = new Client({
    connection: await Connection.connect(),
  });

  const handle = client.schedule.getHandle('sample-schedule');
  await handle.delete();

  console.log(`Schedule is now deleted.`);
}
```
<!--SNIPEND-->

### Describe a Schedule 

The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs.

<!--SNIPSTART typescript-describe-a-scheduled-workflow-->
[schedules/src/describe-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/describe-schedule.ts)
```ts
async function run() {
  const client = new Client({
    connection: await Connection.connect(),
  });

  const handle = client.schedule.getHandle('sample-schedule');

  const result = await handle.describe();

  console.log(`Schedule description: ${JSON.stringify(result)}`);
}
```
<!--SNIPEND-->

### List a Schedule 

The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs.

<!--SNIPSTART typescript-list-a-scheduled-workflow-->
[schedules/src/list-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/list-schedule.ts)
```ts
async function run() {
  const client = new Client({
    connection: await Connection.connect(),
  });

  const schedules = [];

  const scheduleList = client.schedule.list();

  for await (const schedule of scheduleList) {
    schedules.push(schedule);
  }

  console.log(`Schedules are now listed: ${JSON.stringify(schedules)}`);
}
```
<!--SNIPEND-->

### Pause a Schedule 

The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason.

<!--SNIPSTART typescript-pause-a-scheduled-workflow-->
[schedules/src/pause-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/pause-schedule.ts)
```ts
async function run() {
  const client = new Client({
    connection: await Connection.connect(),
  });

  const handle = client.schedule.getHandle('sample-schedule');
  await handle.pause();

  console.log(`Schedule is now paused.`);
}
```
<!--SNIPEND-->

### Trigger a Schedule 

The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time.

<!--SNIPSTART typescript-trigger-a-scheduled-workflow-->
[schedules/src/trigger-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/trigger-schedule.ts)
```ts
async function run() {
  const client = new Client({
    connection: await Connection.connect(),
  });

  const handle = client.schedule.getHandle('sample-schedule');

  await handle.trigger();

  console.log(`Schedule is now triggered.`);
}
```
<!--SNIPEND-->

### Update a Schedule 

The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval.

<!--SNIPSTART typescript-update-a-scheduled-workflow-->
[schedules/src/update-schedule.ts](https://github.com/temporalio/samples-typescript/blob/main/schedules/src/update-schedule.ts)
```ts
const updateSchedule = (
  input: ScheduleDescription,
): ScheduleUpdateOptions<ScheduleOptionsStartWorkflowAction<Workflow>> => {
  const scheduleAction = input.action;

  scheduleAction.args = ['my updated schedule arg'];

  return { ...input, ...scheduleAction };
};

async function run() {
  const client = new Client({
    connection: await Connection.connect(),
  });

  const handle = client.schedule.getHandle('sample-schedule');

  await handle.update(updateSchedule);

  console.log(`Schedule is now updated.`);
}
```
<!--SNIPEND-->

## Temporal Cron Jobs 

> **⚠️ Caution:**
> Cron support is not recommended
>
> We recommend using [Schedules](/schedule) instead of Cron Jobs.
> Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules.
>

A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution.

A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made.

You can set each Workflow to repeat on a schedule with the `cronSchedule` option:

```typescript
const handle = await client.workflow.start(scheduledWorkflow, {
  // ...
  cronSchedule: '* * * * *', // start every minute
});
```

Temporal Workflow Schedule Cron strings follow this format:

```
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
* * * * *
```

## Start Delay 

Use the `startDelay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule.

You may specify the `startDelay` option on either the [`client.workflow.start()`](https://typescript.temporal.io/api/classes/client.WorkflowClient#start) or [`client.workflow.execute()`](https://typescript.temporal.io/api/classes/client.WorkflowClient#execute) methods of a Workflow Client.
For example:

```typescript
const handle = await client.workflow.start(someWorkflow, {
  // ...
  startDelay: '2 hours',
});
```

---

# Workflow Timeouts - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/timeouts

> Optimize Workflow Execution with Temporal TypeScript SDK - Set Workflow Timeouts and Retry Policies efficiently.

This page shows how to do the following:

- [Raise and handle exceptions](#exception-handling)
- [Fail Workflows deliberately](#workflow-failure)
- [Workflow timeouts](#workflow-timeouts)
- [Workflow retries](#workflow-retries)

## Raise and handle exceptions 

In each Temporal SDK, error handling is implemented idiomatically, following the conventions of the language.
Temporal uses several different error classes internally — for example, [`CancelledFailure`](https://typescript.temporal.io/api/classes/common.CancelledFailure) in the TypeScript SDK, to handle a Workflow cancellation. 
You should not raise or otherwise implement these manually, as they are tied to Temporal platform logic.

The one Temporal error class that you will typically raise deliberately is [`ApplicationFailure`](https://typescript.temporal.io/api/classes/common.ApplicationFailure).
In fact, *any* other exceptions that are raised from your TypeScript code in a Temporal Activity will be converted to an `ApplicationFailure` internally.
This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries.

In other words, these two code samples do the same thing:

```typescript 
class InvalidChargeError extends Error {
    constructor(message: string) {
        super(message);
        this.name = "InvalidChargeError";
        Object.setPrototypeOf(this, InvalidChargeError.prototype);
    }
}

if (chargeAmount < 0) {
  throw new InvalidChargeError(`Invalid charge amount: ${chargeAmount} (must be above zero)`);
}
```

```typescript 
if (chargeAmount < 0) {
  throw ApplicationFailure.create({
    message: `Invalid charge amount: ${chargeAmount} (must be above zero)`,
    type: 'InvalidChargeError',
  });
}
```

Depending on your implementation, you may decide to use either method.
One reason to use the Temporal `ApplicationFailure` class is because it allows you to set an additional `nonRetryable` parameter.
This way, you can decide whether an error should not be retried automatically by Temporal.
This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse:

```typescript
if (chargeAmount < 0) {
  throw ApplicationFailure.create({
    message: `Invalid charge amount: ${chargeAmount} (must be above zero)`,
    nonRetryable: true
  });
}
```

You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](/develop/typescript/activities/timeouts#activity-retries).

## Fail Workflows deliberately 

One of the core design principles of Temporal is that an Activity Failure will never directly cause a Workflow Failure — a Workflow should never return as Failed unless deliberately.
The default retry policy associated with Temporal Activities is to retry them until reaching a certain timeout threshold.
Activities will not actually *return* a failure to your Workflow until this condition or another non-retryable condition is met.
At this point, you can decide how to handle an error returned by your Activity the way you would in any other program.
For example, you could implement a [Saga Pattern](https://github.com/temporalio/samples-typescript/tree/main/saga) that uses `try` and `catch` blocks to "unwind" some of the steps your Workflow has performed up to the point of Activity Failure.

**You will only fail a Workflow by manually raising an `ApplicationFailure` from the Workflow code.**
You could do this in response to an Activity Failure, if the failure of that Activity means that your Workflow should not continue:

```typescript
try {
  await addAddress();
} catch (err) {
  if (err instanceof ActivityFailure && err.cause instanceof ApplicationFailure) {
    log.error(err.cause.message);
    throw err;
  }
}
```

This works differently in a Workflow than raising exceptions from Activities.
In an Activity, any TypeScript exceptions or custom exceptions are converted to a Temporal `ApplicationFailure`.
By default, in a Workflow, any exceptions that are raised other than an explicit Temporal `ApplicationFailure` will only fail that particular [Workflow Task](/tasks#workflow-task-execution) and be retried.
This includes any typical TypeScript runtime errors like an `undefined` error that are raised automatically.
These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly.
To have other error types fail the Workflow Execution, list them in the Workflow's `failureExceptionTypes` option with `setWorkflowOptions()`, or in the `workflowFailureErrorTypes` Worker option.
Both options are experimental.

## Workflow timeouts 

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient.
Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes.
If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer.

Workflow Timeouts are set when starting a Workflow using either the Client or Workflow API.

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task

The following properties can be set on the [`WorkflowOptions`](https://typescript.temporal.io/api/interfaces/client.WorkflowOptions/) when starting a Workflow using either the Client or Workflow API:

- [`workflowExecutionTimeout​`](https://typescript.temporal.io/api/interfaces/client.WorkflowOptions/#workflowexecutiontimeout)
- [`workflowRunTimeout`](https://typescript.temporal.io/api/interfaces/client.WorkflowOptions/#workflowruntimeout)
- [`workflowTaskTimeout`](https://typescript.temporal.io/api/interfaces/client.WorkflowOptions/#workflowtasktimeout)

```typescript
await client.workflow.start(example, {
  taskQueue,
  workflowId,
  // Set Workflow Timeout duration
  workflowExecutionTimeout: '1 day',
  // workflowRunTimeout: '1 minute',
  // workflowTaskTimeout: '30 seconds',
});
```

## Workflow retries 

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

The Retry Policy can be set through the [`WorkflowOptions.retry`](https://typescript.temporal.io/api/interfaces/client.WorkflowOptions/#retry) property when starting a Workflow using either the Client or Workflow API.

```typescript
const handle = await client.workflow.start(example, {
  taskQueue,
  workflowId,
  retry: {
    maximumAttempts: 3,
    maximumInterval: '30 seconds',
  },
});
```

---

# Timers - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/timers

> A Workflow sets durable Timers for fixed periods using sleep() or timer(). Timers are persisted, ensuring execution continues after downtime, using minimal resources.

## What is a Timer? 

A Workflow can set a durable Timer for a fixed time period.
In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`.

A Workflow can sleep for months.
Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing.

Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker.

## Asynchronous design patterns in TypeScript 

The real value of `sleep` and `condition` is in knowing how to use them to model asynchronous business logic.
Here are some examples we use the most; we welcome more if you can think of them!

Racing Timers

Use `Promise.race` with Timers to dynamically adjust delays.

```ts
export async function processOrderWorkflow({
  orderProcessingMS,
  sendDelayedEmailTimeoutMS,
}: ProcessOrderOptions): Promise<void> {
  let processing = true;
  const processOrderPromise = processOrder(orderProcessingMS).then(() => {
    processing = false;
  });

  await Promise.race([processOrderPromise, sleep(sendDelayedEmailTimeoutMS)]);

  if (processing) {
    await sendNotificationEmail();
    await processOrderPromise;
  }
}
```

Racing Signals

Use `Promise.race` with Signals and Triggers to have a promise resolve at the earlier of either system time or human intervention.

```ts
import { defineSignal, sleep, Trigger } from '@temporalio/workflow';

const userInteraction = new Trigger<boolean>();
const completeUserInteraction = defineSignal('completeUserInteraction');

export async function yourWorkflow(userId: string) {
  setHandler(completeUserInteraction, () => userInteraction.resolve(true)); // programmatic resolve
  const userInteracted = await Promise.race([
    userInteraction,
    sleep('30 days'),
  ]);
  if (!userInteracted) {
    await sendReminderEmail(userId);
  }
}
```

You can invert this to create a reminder pattern where the promise resolves _if_ no Signal is received.

> **⚠️ Caution:**
> Antipattern: Racing sleep.then
>
> Be careful when racing a chained `sleep`.
> This might cause bugs because the chained `.then` will still continue to execute.
>
> ```ts
> await Promise.race([
>   sleep('5s').then(() => (status = 'timed_out')),
>   somethingElse.then(() => (status = 'processed')),
> ]);
>
> if (status === 'processed') await complete(); // takes more than 5 seconds
> // status = timed_out
> ```
>

Updatable Timer

Here is how you can build an updatable Timer with `condition`:

```ts
import * as wf from '@temporalio/workflow';

// usage
export async function countdownWorkflow(): Promise<void> {
  const target = Date.now() + 24 * 60 * 60 * 1000; // 1 day!!!
  const timer = new UpdatableTimer(target);
  console.log('timer set for: ' + new Date(target).toString());
  wf.setHandler(setDeadlineSignal, (deadline) => {
    // send in new deadlines via Signal
    timer.deadline = deadline;
    console.log('timer now set for: ' + new Date(deadline).toString());
  });
  wf.setHandler(timeLeftQuery, () => timer.deadline - Date.now());
  await timer; // if you send in a signal with a new time, this timer will resolve earlier!
  console.log('countdown done!');
}
```

This is available in the third-party package [`temporal-time-utils`](https://www.npmjs.com/package/temporal-time-utils#user-content-updatabletimer), where you can also see the implementation:

```ts
// implementation
export class UpdatableTimer implements PromiseLike<void> {
  deadlineUpdated = false;
  #deadline: number;

  constructor(deadline: number) {
    this.#deadline = deadline;
  }

  private async run(): Promise<void> {
    /* eslint-disable no-constant-condition */
    while (true) {
      this.deadlineUpdated = false;
      if (
        !(await wf.condition(
          () => this.deadlineUpdated,
          this.#deadline - Date.now(),
        ))
      ) {
        break;
      }
    }
  }

  then<TResult1 = void, TResult2 = never>(
    onfulfilled?: (value: void) => TResult1 | PromiseLike<TResult1>,
    onrejected?: (reason: any) => TResult2 | PromiseLike<TResult2>,
  ): PromiseLike<TResult1 | TResult2> {
    return this.run().then(onfulfilled, onrejected);
  }

  set deadline(value: number) {
    this.#deadline = value;
    this.deadlineUpdated = true;
  }

  get deadline(): number {
    return this.#deadline;
  }
}
```

---

# Versioning - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/versioning

> Update running Workflows without breaking determinism using the TypeScript SDK's Workflow Patching APIs, Worker Build IDs, and migration strategies.

Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress.

The Temporal Platform requires that Workflow Definitions are [deterministic](/workflow-definition#deterministic-constraints).
If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows.

Common causes include adding, removing, or reordering `await` calls on Command-producing APIs such as Activities and timers. Each `await` is a yield point that affects the Command sequence seen during Replay.

With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version.
There are two primary Versioning methods that you can use:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths.
- [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.

> **🚨 Danger:**
> Support for the experimental Worker Versioning method before 2025 was scheduled for removal from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance. You can still refer to the [Worker Versioning Legacy](/develop/typescript/worker-versioning-legacy) docs if needed.

## Versioning with Patching 

### Adding a patch

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow Executions, create a patch.

Suppose you have an initial Workflow that runs `activityA`:

<!--SNIPSTART typescript-patching-1-->
[patching-api/src/workflows-v1.ts](https://github.com/temporalio/samples-typescript/blob/main/patching-api/src/workflows-v1.ts)
```ts
// v1
export async function myWorkflow(): Promise<void> {
  await activityA();
  await sleep('1 days'); // arbitrary long sleep to simulate a long running workflow we need to patch
  await activityThatMustRunAfterA();
}
```
<!--SNIPEND-->

Now, you want to update your code to run `activityB` instead. This represents your desired end state.

<!--SNIPSTART typescript-patching-final-->
[patching-api/src/workflows-vFinal.ts](https://github.com/temporalio/samples-typescript/blob/main/patching-api/src/workflows-vFinal.ts)
```ts
// vFinal
export async function myWorkflow(): Promise<void> {
  await activityB();
  await sleep('1 days');
}
```
<!--SNIPEND-->

The problem is that you cannot deploy this `vFinal` revision directly until you're certain there are no more running Workflows created using the `v1` code, otherwise you are likely to cause a nondeterminism error.
Instead, you'll need to use the [`patched`](https://typescript.temporal.io/api/namespaces/workflow#patched) function to check which version of the code should be executed.

Patching is a three-step process:

1. Patch in any new, updated code using the `patched()` function. Run the new patched code alongside old code.
2. Remove old code and use `deprecatePatch()` to mark a particular patch as deprecated.
3. Once there are no longer any open Workflow Executions of the previous version of the code, remove `deprecatePatch()`.
   Let's walk through this process in sequence.

### Patching in new code

Using `patched` inserts a marker into the Event History.
During Replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker (in this case `your-change-id`).
This ensures you can safely deploy code from `v2` as a "feature flag" alongside the original version (`v1`).

<!--SNIPSTART typescript-patching-2-->
[patching-api/src/workflows-v2.ts](https://github.com/temporalio/samples-typescript/blob/main/patching-api/src/workflows-v2.ts)
```ts
// v2
import { patched } from '@temporalio/workflow';
export async function myWorkflow(): Promise<void> {
  if (patched('my-change-id')) {
    await activityB();
    await sleep('1 days');
  } else {
    await activityA();
    await sleep('1 days');
    await activityThatMustRunAfterA();
  }
}
```
<!--SNIPEND-->

### Deprecating patches 

After ensuring that all Workflows started with `v1` code have left retention, you can [deprecate the patch](https://typescript.temporal.io/api/namespaces/workflow#deprecatepatch).

Once your Workflows are no longer running the pre-patch code paths, you can deploy your code with `deprecatePatch()`.
These Workers will be running the most up-to-date version of the Workflow code, which no longer requires the patch.
Deprecated patches serve as a bridge between the final stage of the patching process and the final state that no longer has patches. They function similarly to regular patches by adding a marker to the Event History. However, this marker won't cause a replay failure when the Workflow code doesn't produce it.

<!--SNIPSTART typescript-patching-3-->
[patching-api/src/workflows-v3.ts](https://github.com/temporalio/samples-typescript/blob/main/patching-api/src/workflows-v3.ts)
```ts
// v3
import { deprecatePatch } from '@temporalio/workflow';

export async function myWorkflow(): Promise<void> {
  deprecatePatch('my-change-id');
  await activityB();
  await sleep('1 days');
}
```
<!--SNIPEND-->

### Removing a patch 

Once your pre-patch Workflows have left retention, you can then safely deploy Workers that no longer use either the `patched()` or `deprecatePatch()` calls:

Patching allows you to make changes to currently running Workflows.
It introduces compatible changes without causing non-determinism errors.

### Workflow cutovers

To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow.

Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes.
To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers.

For example, you would duplicate `PizzaWorkflow` as `PizzaWorkflowV2`:

```typescript
function pizzaWorkflow(order: PizzaOrder): Promise<OrderConfirmation> {
  // this function contains the original code
}

function pizzaWorkflowV2(order: PizzaOrder): Promise<OrderConfirmation> {
  // this function contains the updated code
}
```

You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types:

```typescript
const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  // other configurations
});
```

The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow.
This can become impractical over time.
This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching.

### Testing a Workflow for replay safety

To determine whether your Workflow your needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/typescript/best-practices/testing-suite#replay).

## Worker Versioning

Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching.

---

# Workflow Streams - TypeScript SDK

Source: https://docs.temporal.io/develop/typescript/workflows/workflow-streams

> Stream events from a Workflow to subscribers using the Temporal TypeScript SDK Workflow Streams contrib module.

> **Public Preview**

[Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its progress in real time. This page walks through enabling a stream, publishing events from Workflows and Activities, subscribing to a stream, and keeping a stream running across long-lived Workflows.

## Enable streaming on a Workflow

The library ships as `@temporalio/workflow-streams` and you can import the Workflow-side interface from `@temporalio/workflow-streams/workflow`. Enable streaming by constructing a `WorkflowStream` at the top of your Workflow function, before any `await`. Construction must happen there because the stream's handlers have to be registered before the first publish Signal arrives. Doing it after an `await` would miss any publishes that arrived before the run body resumed.

```typescript
import { WorkflowStream } from '@temporalio/workflow-streams/workflow';

export interface OrderInput {
  orderId: string;
}

export async function orderWorkflow(input: OrderInput): Promise<void> {
  const stream = new WorkflowStream();
  // ... rest of the workflow
}
```

Construct exactly one `WorkflowStream` at the top of the Workflow function. Constructing `WorkflowStream` creates the in-memory event log and registers the publish Signal, subscribe Update, and offset Query handlers on the current Workflow. If you have more than one `WorkflowStream` on the same Workflow silently replaces the handlers. The TypeScript Workflow runtime doesn't expose an inspection API for existing handlers, so the library can't raise an exception on a duplicate the way the Python SDK does.

If your Workflow is long-running, see [Stream from long-running Workflows](#continue-as-new) for how to carry stream state across rollovers so subscribers don't see gaps.

## Publish from a Workflow

Bind a topic name to its event type once via `stream.topic<T>(name)`, then call `publish()` on the returned handle to append events. The handle carries the topic name and the type `T` so call sites don't have to repeat them on every publish, and so subscribers reading the same handle decode to the matching type. Repeated calls with the same name return the same handle instance.

```typescript
import { WorkflowStream } from '@temporalio/workflow-streams/workflow';

export interface StatusEvent {
  state: string;
  progress?: number;
  detail?: string;
}

export interface OrderInput {
  orderId: string;
}

export async function orderWorkflow(input: OrderInput): Promise<void> {
  const stream = new WorkflowStream();
  const status = stream.topic<StatusEvent>('status');

  status.publish({ state: 'validating', detail: 'checking inventory' });
  await validateOrder(input.orderId);

  status.publish({ state: 'charging', progress: 33, detail: 'authorizing payment' });
  await chargePayment(input.orderId);

  status.publish({ state: 'shipping', progress: 66, detail: 'dispatching to warehouse' });
  await dispatchOrder(input.orderId);

  status.publish({ state: 'completed', progress: 100 });
}
```

`publish()` runs the default payload converter to encode each value. The codec chain (encryption, compression, etc.) runs once on the Signal or Update envelope that carries the batch, never per item, so encryption and compression are applied exactly once each direction.

Unlike the Python SDK, `T` here is a compile-time annotation only: TypeScript has no runtime type representation, so the library can't enforce per-topic type uniformity at the publish site. If two publishers bind the same topic name to different types, the mismatch is not caught at publish. The subscriber gets a decode error when it processes events from the mismatched publisher. A pre-built `Payload` may be passed to `publish()` regardless of the handle's type `T`, taking the zero-copy fast path.

## Publish from a client

Any process that has a Temporal Client and the target Workflow Id can publish to that Workflow's stream by constructing a `WorkflowStreamClient`. This is the general pattern and covers HTTP backends, starters, one-off scripts, Activities from other Workflows, and standalone Activities.

Construct one with:

```typescript
WorkflowStreamClient.create(client, workflowId)
```

Then use it the same way you would the Workflow-side handle: bind a topic, publish through it, and let `await using` flush on scope exit.

When events originate in an Activity, publish from the Activity directly rather than returning them for the Workflow to forward. The Workflow hosts the stream but doesn't read its own stream; it processes the Activity's return value and emits its own lifecycle events. Keeping Workflow state independent of streamed output is what lets retried Activity attempts surface to subscribers without polluting the Workflow's durable state. See [How events are delivered](/workflow-streams#how-events-are-delivered).

```typescript
import { Client } from '@temporalio/client';
import { WorkflowStreamClient } from '@temporalio/workflow-streams/client';

export async function publishStatus(workflowId: string): Promise<void> {
  const temporalClient = new Client();
  await using streamClient = WorkflowStreamClient.create(temporalClient, workflowId, {
    batchInterval: '200 milliseconds',
  });

  const status = streamClient.topic<StatusEvent>('status');
  status.publish({ state: 'started' });
  // ...
  // Buffer is flushed automatically on `await using` scope exit.
}
```

The `await using` declaration relies on TypeScript `5.2+` and Node `20.11+`. On older runtimes, call `await streamClient[Symbol.asyncDispose]()` explicitly at the end of the publishing scope.

Inside an Activity scheduled by a Workflow, `WorkflowStreamClient.fromWithinActivity()` is used to infer the Temporal Client and the parent Workflow Id from the Activity context, so you don't have to thread them through the Activity's input:

```typescript
import { Context } from '@temporalio/activity';
import { WorkflowStreamClient } from '@temporalio/workflow-streams/client';

export interface Delta {
  text: string;
}

export async function streamDeltas(orderId: string): Promise<void> {
  await using client = WorkflowStreamClient.fromWithinActivity();
  const deltas = client.topic<Delta>('delta');

  for await (const delta of generateDeltas(orderId)) {
    deltas.publish(delta);
    Context.current().heartbeat();
  }
  // Buffer is flushed automatically on scope exit.
}
```

For a standalone Activity (one started directly via `Client.activity.start` rather than from a Workflow), there is no parent Workflow context to infer, so `fromWithinActivity()` throws. Fall back to the general pattern with `Context.current().client` and the target Workflow Id threaded through the Activity's input.

Two operations give the application explicit control over when batches ship: `forceFlush: true` on a publish for latency, and `await client.flush()` for confirmation that prior publications have landed.

Pass `{ forceFlush: true }` on a publish to wake the background flusher so the current buffer ships without waiting for the next interval. The flusher only runs while the client is alive (between first `publish()` and `Symbol.asyncDispose`). Otherwise, `forceFlush: true` queues the wake event but nothing ships until a flush or dispose occurs. The call returns immediately after appending to the buffer and signaling the flusher. It doesn't wait for delivery to the Workflow or to subscribers:

```typescript
deltas.publish(delta, { forceFlush: true });
```

Use it for latency-sensitive events: the first delta of a response so the user sees something fast, or punctuated events like `RETRY` and `STATUS_CHANGE`. See [Tuning](/workflow-streams#tuning) for the trade-off against history pressure.

Use `await client.flush()` when you need a mid-stream barrier. Successful completion of the flush is proof that the Temporal server has received all prior publications, so subsequent work that depends on those events being durable can proceed. The client stays open for further publishing afterward. Exiting `await using` already flushes on its way out, so the explicit call is only for barriers in the middle:

```typescript
await using client = WorkflowStreamClient.fromWithinActivity();
const deltas = client.topic<Delta>('delta');

for (const delta of firstPhase()) {
  deltas.publish(delta);
}

await client.flush();
const checkpointId = await recordPhaseOneComplete(); // only safe once phase-one events are durable

for (const delta of secondPhase(checkpointId)) {
  deltas.publish(delta);
}
```

`publish()` is non-blocking and applies no backpressure. From an Activity or other client, it appends to the client's in-memory buffer and returns. From inside a Workflow, it appends synchronously to the in-memory log (no buffer, nothing to flush). [Subscribers](/workflow-streams#subscribing) pull from the Workflow's log on their own schedule, so a slow subscriber doesn't slow down [publishers](/workflow-streams#publishing). If a publisher emits faster than batches can ship to the server, the buffer grows: the process uses more memory, the stream falls further behind real time, and at the limit Signals can't keep up.

If your application needs to bound this (to cap memory, to keep the stream close to real time, or to apply a policy when the publisher overruns the network), apply that policy upstream of `publish()`. The choice (block, drop, error, sample) is application-specific, and Workflow Streams doesn't pick one for you.

## Subscribe

[Subscribing](/workflow-streams#subscribing) uses the same client construction as publishing: `WorkflowStreamClient.create(client, workflowId)` from any process that has a Temporal Client or `fromWithinActivity()` inside an Activity. Subscribing from an Activity is less common in practice, so the general client case is the primary example below.

Once you have a client, iterate a topic handle's `subscribe()`, the counterpart to `publish()`. The handle's bound type drives decoding, so each `item.data` arrives as `T` via the client's payload converter. The codec chain is applied once at the Update envelope, not per item.

```typescript
import { Client } from '@temporalio/client';
import { WorkflowStreamClient } from '@temporalio/workflow-streams/client';

export async function watchOrder(orderId: string): Promise<void> {
  const temporalClient = new Client();
  const stream = WorkflowStreamClient.create(temporalClient, orderId);

  const status = stream.topic<StatusEvent>('status');
  for await (const item of status.subscribe()) {
    const evt = item.data;
    console.log(`[${(evt.progress ?? 0).toString().padStart(3)}%] ${evt.state}: ${evt.detail ?? ''}`);
    if (evt.state === 'completed') break;
  }
}
```

The iterator handles re-polling, pagination when a poll response hits the ~1 MB cap, and Workflow-side log truncation transparently. Callers don't need to wrap the iterator for the common cases.

Two edge cases are worth knowing:

- An RPC timeout where Continue-As-New can't be followed ends the iterator silently
- A validator rejection during a Continue-As-New handoff can surface as a `WorkflowUpdateFailedError`.

A subscriber that doesn't need flushing can skip `await using` because the background flusher only runs for publishers.

### Heterogeneous topics

A topic handle binds one name to one type, so it only fits a single-type subscription. To consume multiple topics whose payload types differ, call `client.subscribe()` directly with a list of names (or `subscribe()` with no arguments for every topic). The default overload yields `WorkflowStreamItem<Payload>`, so each item arrives as the raw `Payload` carrying encoding metadata. Dispatch on `item.topic` and decode the payload with `defaultPayloadConverter.fromPayload<T>(item.data)`:

```typescript
import { defaultPayloadConverter } from '@temporalio/common';

for await (const item of stream.subscribe(['status', 'progress'])) {
  if (item.topic === 'status') {
    const evt = defaultPayloadConverter.fromPayload<StatusEvent>(item.data);
    console.log(`[status] ${evt.state}: ${evt.detail ?? ''}`);
  } else if (item.topic === 'progress') {
    const evt = defaultPayloadConverter.fromPayload<ProgressEvent>(item.data);
    console.log(`[progress] ${evt.message}`);
  }
}
```

A single iterator over multiple topics also avoids the cancellation race that two concurrent subscribers would create. The raw `Payload` is also the right shape when you want to forward the bytes through to another system without decoding them.

### Closing the stream

A subscriber's `for await` doesn't know when the publisher is done. How you [close a stream](/workflow-streams#closing-the-stream) depends on what the application needs. As one example, a common pattern combines two pieces:

1. **An in-band terminator.** The Workflow or its Activity publishes a sentinel event the subscriber recognizes and breaks on. In the `watchOrder` example above, `{ state: 'completed' }` is the minimal form, and the consumer's `if (evt.state === 'completed') break` is the matching half. Each subscription decides what its own end-of-stream marker is.
2. **A brief overlap before the Workflow returns.** A poll Update that is still in flight when the Workflow returns is surfaced to the iterator and consumed silently (the iterator either follows Continue-As-New or exits cleanly), and no new polls can complete after that. If the Workflow returns immediately after publishing the terminator, subscribers may miss it.

There are two ways to provide that overlap.

- [Fixed sleep](/workflow-streams#fixed-sleep). Sleep between the terminator and the return so any in-flight poll has time to fetch the terminator before the Workflow exits:

  ```typescript
  import { sleep } from '@temporalio/workflow';

  // at the end of the workflow function
  status.publish({ state: 'completed', progress: 100 });
  await sleep('30 seconds');
  return result;
  ```

- [Acknowledgment handshake](/workflow-streams#acknowledgment-handshake). The subscriber sends a Signal once it has the terminator; the Workflow waits up to a timeout, returning as soon as the ack arrives:

  ```typescript
  import { condition, defineSignal, setHandler } from '@temporalio/workflow';
  import { WorkflowStream } from '@temporalio/workflow-streams/workflow';

  export const subscriberAcknowledgedTerminator = defineSignal('subscriberAcknowledgedTerminator');

  export async function chatWorkflow(input: ChatInput): Promise<string> {
    const stream = new WorkflowStream();
    let subscriberDone = false;
    setHandler(subscriberAcknowledgedTerminator, () => {
      subscriberDone = true;
    });

    // ... do work and publish events ...

    await condition(() => subscriberDone, '30 seconds');
    // Returns true if the ack arrived, false on timeout. Either way, fall through.
    return result;
  }
  ```

The full pattern is wired into the [Stream LLM output](#stream-llm-output) example below.

You can [inspect the terminal status](/workflow-streams#inspecting-terminal-status). `subscribe()` exits cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELLED`, `TERMINATED`, or `TIMED_OUT`, but doesn't distinguish among them. If your application needs to know which (to display success or failure to the user, log the outcome, or decide whether to retry), call `await temporalClient.workflow.getHandle(workflowId).describe()` after the loop returns to inspect the Workflow's status.

## Stream from long-running Workflows 

Workflows that run for hours or accumulate thousands of events need to periodically roll over via [Continue-As-New](/workflow-streams#stream-from-long-running-workflows) to keep history bounded. Subscribers automatically follow these rollovers, but the client retained from `WorkflowStreamClient.create()` or `fromWithinActivity()` is required (clients constructed directly from a single `WorkflowHandle` can't re-target the new run). To keep a stream running across rollovers without subscribers seeing a gap, carry both your application state and the stream state across the boundary. Add an optional `streamState?: WorkflowStreamState` field to your Workflow input, pass it to the constructor, and call `stream.continueAsNew(buildArgs)` to invoke the rollover. The helper drains waiting subscribers, waits for in-flight handlers to finish, then calls `continueAsNew` with the args produced by `buildArgs(postDrainState)`:

```typescript
import { workflowInfo } from '@temporalio/workflow';
import { WorkflowStream, type WorkflowStreamState } from '@temporalio/workflow-streams/workflow';

export interface WorkflowInput {
  itemsProcessed: number;
  streamState?: WorkflowStreamState;
}

export async function longRunningWorkflow(input: WorkflowInput): Promise<void> {
  const stream = new WorkflowStream(input.streamState);
  let itemsProcessed = input.itemsProcessed;

  while (true) {
    await doOneIteration(stream);
    itemsProcessed++;

    if (workflowInfo().continueAsNewSuggested) {
      await stream.continueAsNew<typeof longRunningWorkflow>((state) => [
        {
          itemsProcessed,
          streamState: state,
        },
      ]);
    }
  }
}
```

The optional `streamState?` on the input field is required: `priorState` is `undefined` on a fresh start and a `WorkflowStreamState` after a rollover. The `buildArgs` lambda receives the post-detach `WorkflowStreamState` as its only argument so the snapshot is guaranteed to happen *after* pollers detach.

To pass other Continue-As-New parameters such as `taskQueue`, `searchAttributes`, or `workflowRunTimeout`, use the explicit recipe with `makeContinueAsNewFunc` instead:

```typescript
import { allHandlersFinished, condition, makeContinueAsNewFunc } from '@temporalio/workflow';

stream.detachPollers();
await condition(allHandlersFinished);
const continueWithOptions = makeContinueAsNewFunc<typeof longRunningWorkflow>({
  taskQueue: 'other-tq',
});
await continueWithOptions({
  itemsProcessed,
  streamState: stream.getState(),
});
```

The carried `WorkflowStreamState` includes the entire in-memory log of the previous run, so streams that carry large items can hit Temporal's per-payload size limit at the rollover. Offload the bytes via [External Storage](/external-storage) so each item is a small reference rather than the full payload, and combine that with `truncate()` to keep the carried log itself small.

## Deduplication window

See [How events are delivered](/workflow-streams#how-events-are-delivered) for more details on subscriber and publisher behavior. See [Tuning](/workflow-streams#tuning) for more details on how to change your settings to meet the requirements for your Workflow Streams.

There are two limits on the [deduplication window](/workflow-streams#deduplication-window) worth highlighting:

- `publisherTtl`: At each Continue-As-New, deduplicate entries whose `lastSeen` is older than this are dropped. `lastSeen` is updated on each *successful* publish (not on each retry attempt), so a publisher that retries through a long partition without success can still age out. A publisher that returns after a longer pause may produce a duplicate. Tune upward if your publishers can be silent for extended windows:

  ```typescript
   stream.continueAsNew(buildArgs, { publisherTtl: '...' })
  ```
- `maxRetryDuration`: A `WorkflowStreamClient` retries a failed batch for up to this long. If the duration elapses with the batch still pending, the client gives up, the pending batch is dropped, and a `FlushTimeoutError` is raised.

    On timeout, the dropped batch is at-most-once: it may or may not have reached the Workflow. Subsequent publishes resume cleanly with the next sequence. One operational caveat: the `FlushTimeoutError` is raised from inside the background flusher task and terminates it. Until you call `await client.flush()` or exit the `await using` scope, subsequent publishes accumulate in the buffer with no flusher to ship them.

## Best practices

There are a few details to note if you're writing custom message handlers or testing the library's capabilities:

- **The package has no root entrypoint.** You can import from one of two subpaths:
  - `@temporalio/workflow-streams/workflow`: The Workflow-safe interface (`WorkflowStream`, `WorkflowStreamState`, etc). Bundles cleanly into Workflow code.
  - `@temporalio/workflow-streams/client`: the client interface (`WorkflowStreamClient`, etc). Pulls in `crypto`, `@temporalio/activity`, and `@temporalio/client`, none of which resolve inside the Workflow sandbox. Don't import from a Workflow file.
- **`WorkflowStreamClient` is single-event-loop.** The client buffer is mutated on the publish path and read from the background flusher inside one Node event loop. Don't call `publish()` from a Worker thread. Route events back to the loop that owns the client.
- **Constructing two `WorkflowStreams` silently replaces handlers.** The TypeScript Workflow runtime doesn't expose an inspection API for existing handlers, so the library can't raise on a duplicate the way the Python SDK does. Construct exactly one `WorkflowStream` per Workflow at the top of the function.
- **Type bindings aren't shared across publishers.** Each `WorkflowStream` and each `WorkflowStreamClient` records topic types only for its own instance, and the type parameter `T` is erased at compile time, so no runtime check enforces uniformity. If two publishers bind the same topic name to different types, the mismatch is not caught at publish, and the subscriber gets a decode error when it processes events from the mismatched publisher.
- **Custom payload converters.** A `WorkflowStreamClient` created via `WorkflowStreamClient.create(client, ...)` picks up the client's configured payload converter. Subscribers decode through the same converter. The Workflow-side always uses `defaultPayloadConverter`. If you ship a custom converter, make sure both sides agree or use types the default converter handles.
- **Cross-realm `Uint8Array` for binary publishes.** Hand-publishing a `Uint8Array` from Workflow code uses a dedicated code path that constructs a `binary/plain` Payload directly, because the sandbox's `TextEncoder` returns a host-realm `Uint8Array` that fails `instanceof` checks against the sandbox's own globals. You generally don't need to think about this, but if you bypass the workflow-side handle and construct payloads manually, use the Workflow-side `WorkflowStream` API rather than building payloads by hand.

## Example: Stream LLM output 

The headline use case fits the publish/subscribe shapes documented above. An Activity calls the model and publishes deltas as they arrive. The Workflow kicks off the Activity and waits for the consumer to acknowledge end-of-stream. The consumer subscribes, accumulates the deltas, and clears its accumulated state on `RETRY` before continuing. The shape works for a terminal client, a desktop UI, or a Server-Sent Events (SSE) endpoint forwarding to a browser. Anything that holds the displayed state calls `render()` to display it.

If your Activity can retry, the consumer side has to account for it. A retried attempt is a fresh publisher, so its output appears in the stream alongside the output from the previous attempt. In the LLM streaming pattern below, that means the failed attempt's partial deltas and the retried attempt's full output both reach a subscribed UI unless the UI resets on a `RETRY` event. The example wires up that pattern. See [How events are delivered](/workflow-streams#how-events-are-delivered) for the precise guarantees.

**activities.ts**

```typescript
import { Context } from '@temporalio/activity';
import { WorkflowStreamClient } from '@temporalio/workflow-streams/client';
import OpenAI from 'openai';

export interface TextDelta {
  text: string;
}

export interface RetryEvent {
  attempt: number;
}

export async function streamCompletion(prompt: string): Promise<string> {
  const attempt = Context.current().info.attempt;
  await using streamClient = WorkflowStreamClient.fromWithinActivity({
    batchInterval: '200 milliseconds',
  });
  // Disable provider-side retries; let Temporal own retry policy at the Activity layer.
  const openai = new OpenAI({ maxRetries: 0 });

  const deltas = streamClient.topic<TextDelta>('delta');
  const retry = streamClient.topic<RetryEvent>('retry');
  const close = streamClient.topic<Record<string, never>>('close');

  // Tell consumers an earlier attempt's deltas are stale.
  if (attempt > 1) {
    retry.publish({ attempt }, { forceFlush: true });
  }

  const oaiStream = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: prompt }],
    stream: true,
  });

  const full: string[] = [];
  let first = true;
  for await (const chunk of oaiStream) {
    const text = chunk.choices[0]?.delta?.content;
    if (!text) continue;
    // forceFlush only on the first delta so the user sees something
    // immediately; subsequent deltas batch at the 200 ms interval.
    deltas.publish({ text }, first ? { forceFlush: true } : undefined);
    first = false;
    full.push(text);
  }
  close.publish({});
  return full.join('');
}
```

**workflows.ts**

```typescript
import { condition, defineSignal, proxyActivities, setHandler } from '@temporalio/workflow';
import { WorkflowStream } from '@temporalio/workflow-streams/workflow';
import type * as activities from './activities';

const { streamCompletion } = proxyActivities<typeof activities>({
  startToCloseTimeout: '5 minutes',
});

export const subscriberAcknowledgedTerminator = defineSignal('subscriberAcknowledgedTerminator');

export interface ChatInput {
  prompt: string;
}

export async function chatWorkflow(input: ChatInput): Promise<string> {
  const stream = new WorkflowStream();
  let subscriberDone = false;
  setHandler(subscriberAcknowledgedTerminator, () => {
    subscriberDone = true;
  });

  const result = await streamCompletion(input.prompt);

  // Wait for the subscriber to ack the terminal `close` event. The timeout
  // is a fallback for when no subscriber is attached; with the ack, the
  // typical case exits as soon as the subscriber confirms.
  await condition(() => subscriberDone, '30 seconds');
  return result;
}
```

**consumer.ts**

```typescript
import { Client } from '@temporalio/client';
import { defaultPayloadConverter } from '@temporalio/common';
import { WorkflowStreamClient } from '@temporalio/workflow-streams/client';
import { subscriberAcknowledgedTerminator } from './workflows';

export async function streamChat(chatId: string): Promise<string> {
  const temporalClient = new Client();
  // Subscribe-only; no `await using` needed because the flusher only runs for publishers.
  const stream = WorkflowStreamClient.create(temporalClient, chatId);
  const output: string[] = [];

  function render(): void {
    // ... display the accumulated output (terminal redraw, UI update, etc.)
  }

  for await (const item of stream.subscribe(['delta', 'retry', 'close'])) {
    if (item.topic === 'retry') {
      // Earlier attempt's deltas are stale; drop what we've shown.
      output.length = 0;
      render();
    } else if (item.topic === 'delta') {
      const delta = defaultPayloadConverter.fromPayload<TextDelta>(item.data);
      output.push(delta.text);
      render();
    } else if (item.topic === 'close') {
      // Acknowledge so the Workflow can return without waiting on the fallback timeout.
      await temporalClient.workflow.getHandle(chatId).signal(subscriberAcknowledgedTerminator);
      break;
    }
  }

  return output.join('');
}
```

A few choices in this shape are deliberate:

- The Activity is the publisher because it owns the non-deterministic LLM call. The Workflow processes only the Activity's return value, never reading its own stream. See [Publish from a client](#publish-from-a-client) for why.
- The Activity publishes a `RETRY` event when `Context.current().info.attempt > 1`. This lets the UI respond appropriately to the failure, typically by clearing accumulated deltas before the next attempt's deltas arrive (see [How events are delivered](/workflow-streams#how-events-are-delivered)).
- Termination uses an *ack handshake*: the consumer signals the Workflow once it has received the `close` event, so the Workflow can return as soon as the subscriber confirms. The `condition` timeout is the fallback when no subscriber is attached (see [Closing the stream](#closing-the-stream) for the simpler fixed-sleep alternative).
- `{ forceFlush: true }` is used only on the first delta and on the `RETRY` sentinel, where latency matters. Subsequent deltas batch at the 200 ms `batchInterval`; per-delta `forceFlush: true` would generate one Signal per token (see [Tuning](/workflow-streams#tuning) for the trade-off).

## See also

- [Workflow Streams samples (samples-typescript)](https://github.com/temporalio/samples-typescript/tree/main/workflow-streams): runnable scenarios covering basic publish/subscribe, reconnecting subscribers, external publishers, bounded logs, and LLM streaming.
- [`@temporalio/workflow-streams/workflow` API reference](https://typescript.temporal.io/api/namespaces/workflowStreams).
- [`@temporalio/workflow-streams/client` API reference](https://typescript.temporal.io/api/namespaces/workflowStreamsClient).
- [Workflow message passing](/develop/typescript/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on.

---

# Worker performance

Source: https://docs.temporal.io/develop/worker-performance

> Optimize Temporal SDK performance by fine-tuning maxConcurrentWorkflowTaskExecutionSize, Worker Cache options, and Poll Success Rate. Ensure balanced Worker resources and monitor metrics for best results.

This guide covers metrics and configurations that drive the efficiency of your Worker fleet.
It includes performance metric families, Worker configuration options, Task Queue information, backlog counts, Task rates, and how to evaluate Worker availability.
Use the pages in this guide for practical methods for querying Task Queue information, and strategies for tuning Workers and Task Queue processing so you manage your resources effectively.

## Worker performance concepts 

A Worker's performance characteristics are affected by, but not limited to, the following elements.

### Task slots 

A **Worker Task Slot**, represents the capacity of a Temporal Worker to execute a single concurrent Task.
Slots are crucial for managing the workload and performance of Workers in a Temporal application.
They're used for both Workflow and Activity Tasks.
When a Worker starts processing a Task, it occupies one slot.
The number of available slots directly affects how many tasks a Worker can handle simultaneously.

### Slot suppliers 

A **Slot Supplier** defines a strategy to provide slots for a Worker, increasing or decreasing the Worker's slot count.
The supplier determines when it's acceptable to begin a new Task.
Each supplier manages one slot type.
There are slot types for Activity, Workflow, Nexus, or Local Activity Tasks.
An available slot determines whether or not a Worker is willing to poll for, and execute, a new Task of that type.

Slot supplier strategies include manual assignment of fixed slot counts and resource-balanced "auto-tuner" assignment.
Resource-based suppliers adjust slot counts based on CPU and memory resources.
Available slot suppliers include:

- **Fixed Size Slot Suppliers**:
  Hands out slots up to a preset limit.
  This is useful if you have a concrete idea of how many resources your tasks are going to consume, and can determine an upper bound on how many should run at once.
  When you need the absolute best performance, review your hardware and environment characteristics.
  This information lets you calculate an appropriate fixed-size limit.
  Evaluate the maximum number of slots you can support without oversubscribing or hitting out-of-memory conditions ("OOMing").
  Using that value with a fixed-size supplier provides optimal results with the least overhead.

- **Resource-Based Slot Suppliers**:
  Hands out slots based on real-time CPU and memory usage.
  You set target utilization for both CPU and memory and the Slot Supplier tries to reach those values without exceeding them under load.
  A resource-based supplier will account for memory limits imposed in containerized environments.
  It dynamically adjusts the number of available slots for different task types with respect to current system resources.

> **ℹ️ Info:**
>
> When running in a containerized environment, all SDKs use cgroups for both CPU and memory. CPU is accounted for at the container level.
>

- **Custom Slot Suppliers**:
  Hands out slots based on the custom logic that you define.
  Use this approach when you need complete control over when Workers accept and execute Tasks.
  For implementation details, see [Implement Custom Slot Suppliers](/develop/worker-performance/runtime-tuning#custom-slot-implementation).

> **⚠️ Caution:**
>
> - You cannot guarantee that the targets for resource-based suppliers won't ever be exceeded.
>   Resources consumed during a task can't be known ahead of time.
>
> - Read about [choosing an appropriate slot supplier type](/develop/worker-performance/runtime-tuning#choosing-slot-supplier-types) before picking one.
>
> - Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options.
>   Using both styles will cause an error at Worker initialization time.
>

### Worker tuning 

Worker tuning is the process of defining customized slot suppliers for the different task slots of a Worker to fine-tune its performance.
You use special types called **Worker tuners** that assign slot suppliers to various Task Types, including Worker, Activity, Nexus, and Local Activity Tasks.

For more on how to configure and use Worker tuners, refer to [Worker runtime performance tuning](/develop/worker-performance/runtime-tuning).

> **⚠️ Caution:**
>
> Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options.
>   Using both styles will cause an error at Worker initialization time.
>

### Task pollers 

A Worker's **Task Pollers** play a crucial role in the Temporal architecture by efficiently ingesting work to Workers to support scalable, resilient Workflow Execution. 
Pollers create long-polling connections to the Temporal Service and actively poll a Task Queue for Tasks to process. When a Task Poller receives a Task, it delivers the Task to the appropriate Executor Slot for processing.

Temporal SDKs implement support for *Poller Autoscaling*, which dynamically adjusts the number of pollers in use to maximize throughput for a given number of workers and the size of the task backlog. 
Temporal recommends using Poller Autoscaling for the majority of use cases, as manually setting the number of pollers too high or too low for your workload will result in decreased performance. 
To configure Poller Autoscaling, see [Configuring Poller Options](/develop/worker-performance/configuration#configuring-poller-options) and samples for each Temporal SDK.

### Eager task execution

> **⚠️ Caution:**
> Eager start does not respect Worker versioning. An eagerly started Workflow may run on any available local Worker even if that Worker is not the Current or Ramping version of its Worker deployment.

As a latency optimization, Activity and Workflow Tasks may be started eagerly in a local Worker under the right circumstances.

#### Eager Activity Start

Eager Activity Start may happen automatically if the Worker processing a Workflow Task has also registered the Activity Definition being called.
If it does, it may try to reserve an Activity Slot for the execution of the Activity, and the server may respond to the Workflow Task completion with the Activity Task for the worker to execute immediately.

#### Eager Workflow Start

> **Public Preview** — Go, Java, Python, .NET
> Eager Workflow Start is enabled for all Temporal Cloud users and self-hosted Temporal Server 1.29.0+. No additional configuration or access request is needed. However, you must set `Request-Eager-Start` to true when starting each Workflow for Eager Workflow Start to be used.

Eager Workflow Start reduces the latency required to initiate a Workflow execution.
It is recommended for short-lived Workflows that use Local Activities to interact with external services, especially when these interactions are initiated in the first Workflow Task and the Workflow is deployed near the Temporal Server to minimize network delay.

This feature is particularly beneficial for Workflows with a “happy path” that must begin external interactions within tens of milliseconds, while still relying on Temporal’s server-driven retries and compensation mechanisms to ensure reliability in failure scenarios.

**Quick Start**

Eager Workflow Start requires the Starter and the Worker to share a Client located in the same process and setting the `request_eager_start` (or similar name) to true in the Start Workflow call.
When set, and the Worker has a Workflow Task slot available and the Workflow Definition registered, the Worker can execute the first task of the Workflow locally without first making a round-trip to the Temporal Server.
This is typically most useful in combination with a Local Activity executing in the first Workflow Task, since other Workflow API calls that require waiting on something will force a round-trip.

> **💡 Tip:**
> RESOURCES
>
> - [Go SDK - Code sample](https://github.com/temporalio/samples-go/tree/main/eager-workflow-start)
> - [Java SDK - Code sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloEagerWorkflowStart.java)
> - Python SDK - use `request_eager_start` when calling `start_workflow` or `execute_workflow`
> - .NET SDK - use `RequestEagerStart` in your `WorkflowOptions` when starting a workflow
> - [Blog: Improving Latency with Eager Workflow Start](https://temporal.io/blog/improving-latency-with-eager-workflow-start)
>

**How it works**

The traditional way to start a Workflow decouples the starter program from the worker by sharing a Task Queue name between them, similar to a publish/subscribe pattern.
This has many advantages: for example, we can reliably schedule a Workflow Execution without a running Worker, or separate the Worker and Workflow implementation from the Starter application and host them independently.

But decoupling also makes it harder to optimize for latency.
Instead, when the **Starter and Worker are collocated in the same process** and aware of each other, they can interact while bypassing the server, saving a few time-intensive operations.

![Eager Workflow Start](https://docs.temporal.io/img/develop/worker-performance/eager-workflow-start-flow.png)

The above figure shows Eager Workflow Start in action:

1. The process begins with the Starter setting `request_eager_start` (or similar name) to true in the Start Workflow Options.
1. The SDK will try to locate a local Worker that is willing to execute the first Workflow Task, and reserve an execution slot for it.
1. If successful, the SDK will provide a hint to the server that eager mode is preferred for the new Workflow.
1. The server not only registers the start of the Workflow in history, it also assigns the first Workflow Task to the Starter, all in the same DB update.
1. The first task is included in the server response, no matching step required.
1. The SDK extracts the task from the response, and dispatches it to the local worker.

To recover from errors, Eager Workflow Start falls back to the non-eager path. For example, when the first Task is returned eagerly, but the local Worker fails or times out while processing the task, the server retries this task non-eagerly after WorkflowTaskTimeout.

## Visualize Workers in the UI 

The Temporal SDK includes a heartbeat for Worker processes that is sent to the Server, carrying information such as available task slots, CPU usage, and Worker configuration.
The Server exposes this data through APIs that surface Worker details in the Temporal UI. See a list of all running Workers by selecting Workers in the left navigation menu.
View the list of Workers assigned to the Workflow Task Queue and inspect Worker details to troubleshoot Workflows with delayed processing by selecting the Workers tab in the Workflow details page.

This feature requires Temporal Server 1.30 or higher with API version 1.62 or higher, and is also available in Temporal Cloud.
Worker Heartbeating is required; see [Manage Worker Heartbeating](/cloud/worker-health#manage-worker-heartbeating) for the minimum SDK versions.

## In this guide

- [Performance metrics for tuning](/develop/worker-performance/metrics)
- [Worker performance options](/develop/worker-performance/configuration) - executor slots, pollers, and the Workflow cache
- [Worker runtime performance tuning](/develop/worker-performance/runtime-tuning)
- [Task Queue performance](/develop/worker-performance/task-queues)
- [Worker tuning quick reference](/develop/worker-tuning-reference)

## Related reading

- [Worker tuning quick reference](/develop/worker-tuning-reference) - SDK defaults and metrics by resource type
- [Worker tuning simulator](/demos/worker-tuning-simulator) - Interactive demo of Pollers, Slots, and cache limits
- [Workers in production operation guide](https://temporal.io/blog/workers-in-production)
- [Full set of SDK Metrics reference](/references/sdk-metrics)

---

# Worker performance options

Source: https://docs.temporal.io/develop/worker-performance/configuration

> Configure Worker executor slots, poller options, and Workflow cache options for Temporal SDK performance.

Each Worker can be configured by providing custom Worker options (`WorkerOptions`) at instantiation.
Options are specific to individual Workers and do not affect other members of your fleet.

## Executor slot options

The `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` options define the number of total available Workflow Task and Activity Task slots for a Worker.

> **⚠️ Caution:**
>
> - Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options.
>   Using both styles will cause an error at Worker initialization time.
>

## Configuring poller options 

### Recommended approach

The Temporal SDKs support Poller Autoscaling, which automatically selects an appropriate number of pollers based on need. Using this feature results in more efficient poller usage, better throughput, and schedule-to-start latency improvements. You can enable this feature by setting the `*_task_poller_behavior` options to `PollerBehaviorAutoscaling`. Names may vary slightly depending on the SDK. For specific examples of enabling Poller Autoscaling, see the SDK examples section below.

> **💡 Tip:**
>
> `PollerBehaviorAutoscaling` is only enabled in Temporal Server v1.28.0 and later.
>

### Manual configuration

There are options available to manually configure minimum, maximum, and initial poller counts, but it is not recommended to set these values manually for production use cases. To set these values manually, the following options are available: 

- `maxConcurrentWorkflowTaskPollers` (in the JavaSDK: `workflowPollThreadCount`)
- `maxConcurrentActivityTaskPollers` (in the JavaSDK: `activityPollThreadCount`)

These options define the maximum count of pollers performing poll requests on Workflow and Activity Task Queues, respectively. 

### SDK examples

Select a concept to highlight the matching poller option:

**Go**

[Go SDK docs](https://pkg.go.dev/go.temporal.io/sdk/worker#PollerBehaviorAutoscalingOptions)
  annotations={[
    {
      label: 'Workflow Task poller',
      description: 'Autoscales the number of pollers for Workflow Tasks based on load.',
      lines: [2],
    },
    {
      label: 'Activity Task poller',
      description: 'Autoscales the number of pollers for Activity Tasks based on load.',
      lines: [3],
    },
    {
      label: 'Nexus Task poller',
      description: 'Autoscales the number of pollers for Nexus Tasks based on load.',
      lines: [4],
    },
  ]}
>
```go
w := worker.New(c, "my-task-queue", worker.Options{
  WorkflowTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}),
  ActivityTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}),
  NexusTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}),
})
```

**Java**

[Java SDK docs](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/tuning/PollerBehaviorAutoscaling.html)
  annotations={[
    {
      label: 'Workflow Task poller',
      description: 'Autoscales the number of pollers for Workflow Tasks based on load.',
      lines: [7],
    },
    {
      label: 'Activity Task poller',
      description: 'Autoscales the number of pollers for Activity Tasks based on load.',
      lines: [8],
    },
    {
      label: 'Nexus Task poller',
      description: 'Autoscales the number of pollers for Nexus Tasks based on load.',
      lines: [9],
    },
  ]}
>
```java
public class WorkerExample {
    public static void main(String[] args) {
        WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
        WorkflowClient client = WorkflowClient.newInstance(service);
        WorkerFactory factory = WorkerFactory.newInstance(client);
        WorkerOptions workerOptions = WorkerOptions.newBuilder()
            .setWorkflowTaskPollersBehavior(new PollerBehaviorAutoscaling())
            .setActivityTaskPollersBehavior(new PollerBehaviorAutoscaling())
            .setNexusTaskPollersBehavior(new PollerBehaviorAutoscaling())
            .build();

        Worker worker = factory.newWorker("my-task-queue", workerOptions);
    }
}
```

**Python**

[Python SDK docs](https://python.temporal.io/temporalio.worker.PollerBehaviorAutoscaling.html)
  annotations={[
    {
      label: 'Workflow Task poller',
      description: 'Autoscales the number of pollers for Workflow Tasks based on load.',
      lines: [7],
    },
    {
      label: 'Activity Task poller',
      description: 'Autoscales the number of pollers for Activity Tasks based on load.',
      lines: [8],
    },
    {
      label: 'Nexus Task poller',
      description: 'Autoscales the number of pollers for Nexus Tasks based on load.',
      lines: [9],
    },
  ]}
>
```python
worker = Worker(
    client,
    task_queue="my-task-queue",
    workflows=[MyWorkflow],
    activities=[my_activity],

    workflow_task_poller_behavior=PollerBehaviorAutoscaling(),
    activity_task_poller_behavior=PollerBehaviorAutoscaling(),
    nexus_task_poller_behavior=PollerBehaviorAutoscaling(),
)
```

**TypeScript**

[TypeScript SDK docs](https://typescript.temporal.io/api/interfaces/proto.temporal.api.sdk.v1.WorkerConfig.IAutoscalingPollerBehavior)
  annotations={[
    {
      label: 'Workflow Task poller',
      description: 'Autoscales the number of pollers for Workflow Tasks based on load.',
      lines: [7],
    },
    {
      label: 'Activity Task poller',
      description: 'Autoscales the number of pollers for Activity Tasks based on load.',
      lines: [8],
    },
    {
      label: 'Nexus Task poller',
      description: 'Autoscales the number of pollers for Nexus Tasks based on load.',
      lines: [9],
    },
  ]}
>
```ts
const worker = await Worker.create({
  connection,
  taskQueue: 'my-task-queue',
  workflowsPath: require.resolve('./workflows'),
  activities,

  workflowTaskPollerBehavior: PollerBehavior.autoscaling(),
  activityTaskPollerBehavior: PollerBehavior.autoscaling(),
  nexusTaskPollerBehavior: PollerBehavior.autoscaling(),
});
```

**.NET**

[DotNet SDK docs](https://dotnet.temporal.io/api/Temporalio.Worker.Tuning.PollerBehavior.Autoscaling.html)
  annotations={[
    {
      label: 'Workflow Task poller',
      description: 'Autoscales the number of pollers for Workflow Tasks based on load.',
      lines: [5],
    },
    {
      label: 'Activity Task poller',
      description: 'Autoscales the number of pollers for Activity Tasks based on load.',
      lines: [6],
    },
    {
      label: 'Nexus Task poller',
      description: 'Autoscales the number of pollers for Nexus Tasks based on load.',
      lines: [7],
    },
  ]}
>
```csharp
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
    {
        WorkflowTaskPollerBehavior = new PollerBehavior.Autoscaling(),
        ActivityTaskPollerBehavior = new PollerBehavior.Autoscaling(),
        NexusTaskPollerBehavior = new PollerBehavior.Autoscaling(),
    }
    .AddWorkflow<MyWorkflow>()
    .AddActivity(MyActivities.MyActivity)
);
```

**Ruby**

[Ruby SDK docs](https://ruby.temporal.io/Temporalio/Worker/PollerBehavior/Autoscaling.html)
  annotations={[
    {
      label: 'Workflow Task poller',
      description: 'Autoscales the number of pollers for Workflow Tasks based on load.',
      lines: [7],
    },
    {
      label: 'Activity Task poller',
      description: 'Autoscales the number of pollers for Activity Tasks based on load.',
      lines: [8],
    },
    {
      label: 'Nexus Task poller',
      description: 'Autoscales the number of pollers for Nexus Tasks based on load.',
      lines: [9],
    },
  ]}
>
```ruby
worker = Temporalio::Worker.new(
  client,
  'my-task-queue',
  workflows: [MyWorkflow],
  activities: [MyActivity],

  workflow_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new,
  activity_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new,
  nexus_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new,
)
```

## Cache options (Java SDK) 

A [Workflow Cache](/workflow-execution#workflow-cache) is created and shared between all Workers on a single host.
It's designed to limit the resources used by the cache for each host/process.
These options are defined on `WorkerFactoryOptions`:

- `WorkerFactoryOptions#workflowCacheSize` defines the maximum number of cached Workflow Executions.
  Each cached Workflow contains at least one Workflow thread and its resources (memory, etc.).
- `maxWorkflowThreadCount` defines the maximum number of Workflow threads that may exist concurrently at any time.

These cache options limit the resource consumption of the in-memory Workflow cache.
Workflow cache options are shared between all Workers because the Workflow cache is tightly integrated with the resource consumption of the entire host.
This includes memory and the total thread count, which should be limited per host/JVM.

For Go, use [`SetStickyWorkflowCacheSize`](https://pkg.go.dev/go.temporal.io/sdk/worker#SetStickyWorkflowCacheSize). For Python, use the `max_cached_workflows` Worker option.

## "Large value" drawbacks

There are drawbacks when you use "large values everywhere."
As with any multithreading system, specifying excessively large values without monitoring with the SDK and system metrics leads to constant resource contention/stealing
This decreases the total throughput and increases latency jitter of the system.

## Invariants (JavaSDK only) 

These properties should always be true for a Worker's configuration.

Perform this sanity check after the adjustments to Worker settings.

1. `workflowCacheSize` should be ≤ `maxWorkflowThreadCount`. Each Workflow has at least one Workflow thread.
2. `maxConcurrentWorkflowTaskExecutionSize` should be ≤ `maxWorkflowThreadCount`. It's recommended that `maxWorkflowThreadCount` be at least 2x of `maxConcurrentWorkflowTaskExecutionSize`. This is because having more Worker slots than the Workflow cache size will lead to resource allocation issues between executors and cause unpredictable delays.
3. `maxConcurrentWorkflowTaskPollers` should be significantly ≤ `maxConcurrentWorkflowTaskExecutionSize`. And `maxConcurrentActivityTaskPollers` should be significantly ≤ `maxConcurrentActivityExecutionSize`. The number of pollers should always be lower than the number of executors.

---

# Performance metrics for tuning

Source: https://docs.temporal.io/develop/worker-performance/metrics

> Use Temporal SDK slot availability, latency, and cache metrics to tune Worker performance.

> **ℹ️ Info:**
>
> All metrics on this page are prepended with the `temporal_` prefix.
> For example, `worker_task_slots_available` is actually `temporal_worker_task_slots_available` when used.
> The omitted prefix makes the names more readable and descriptive.
>

The Temporal SDKs emit metrics from Temporal Client usage and Worker Processes.
Performance tuning uses three important SDK metric groups:

## Slot availability metrics 

Temporal's [`worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available) and `worker_task_slots_used` gauges can report the number of available executor “slots” that are currently available and unoccupied for a Worker type.
Tag these with `worker_type=WorkflowWorker` for Workflow Task Workers or `worker_type=ActivityWorker` for Activity Workers.

> **💡 Tip:**
>
> Unlike `worker_task_slots_used`, `worker_task_slots_available` can only be used with fixed size slot suppliers and can't be used with resource-based slot suppliers.
>

## Latency metrics

Temporal provides two latency timers: [`workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency) for Workflow Tasks and [`activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency) for Activities.
A Schedule-To-Start latency is the time from when a Task is scheduled (that is, placed in a Queue) to when a Worker starts (that is, picks up from the Task Queue) that Task.
These metrics help ensure that Tasks are being processed from the queue in a timely manner.
For more information about `schedule_to_start` timeout and latency, see [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout).

## Cache metrics

The [`sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) and [`workflow_active_thread_count`](/references/sdk-metrics#workflow_active_thread_count) metrics report the size of the Workflow cache and the number of cached Workflow threads.

---

# Worker runtime performance tuning

Source: https://docs.temporal.io/develop/worker-performance/runtime-tuning

> Choose and implement fixed-size, resource-based, and custom slot suppliers, with SDK examples for Worker tuners.

Worker tuning manages the assignment of slot suppliers.
A **Worker Tuner** instance exists per-Worker, providing slot suppliers for different slot types (Activity, Workflow, Nexus, or Local Activity Tasks).
A tuner assigns different suppliers to each slot type.
For example, it might provide a fixed assignment slot supplier for Workflows and use a resource-based supplier for Activities.

## Choosing slot supplier types 

Temporal offers three types of slot suppliers: fixed assignment, resource-based, and custom.
**For most workloads, Temporal recommends fixed-size slot suppliers.**
A fixed-size tuner with appropriately chosen values delivers better performance and more predictable behavior than a resource-based tuner.

Each SDK provides [default slot counts](/develop/worker-tuning-reference#compute-defaults-by-sdk), but Temporal recommends [actively tuning these values](/best-practices/worker#actively-tune-worker-options-instead-of-relying-on-defaults) for your workload.

### When to use resource-based slot suppliers

Resource-based slot suppliers are suited to specific use cases.

- **Fluctuating workloads with low per-Task consumption**:
  The resource-based supplier works well when each Task consumes few resources but may run for a (relatively) long time.
  For example: HTTP calls or other blocking I/Os that spend most of their time waiting on external events.
- **Protection from out-of-memory & over-subscription in the face of unpredictable per-task consumption:**
  Do your Tasks often consume an unpredictable number of resources?
  Do you want to avoid crashes without setting an overly-conservative fixed limit?
  In these cases, the resource-based supplier is a good match.
  Keep in mind that auto-tuning can never do a _perfect_ job and may sometimes exceed your requested system limits for CPU and memory.

Scenarios with tasks that have variable, or very high, per-task resource needs should rely on fixed-size suppliers and manual tuning rather than resource-based suppliers.

### When to use custom slot suppliers

For the highest level of control over slot allocation, consider custom slot suppliers.
Custom suppliers let you tailor the logic of how slots are allocated based on your system requirements, providing flexibility to optimize for specific use cases that fixed assignment and resource-based suppliers do not fully address.

## Implement custom slot suppliers 

Implement your own Slot Supplier to control how Workers are allocated Tasks and manage the processing of Workflows, Activities, and Nexus Operations.
Custom Slot Suppliers let you fine-tune task processing based on your application's needs.

Each SDK's reference documentation explains the specifics of the interface, but the core concepts are consistent across SDKs:

| Language                                               | Slot Supplier Reference                                                                                                  |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| <img src="/img/sdks/png/golang.png" height="24" />     | [`SlotSupplier`](https://pkg.go.dev/go.temporal.io/sdk/worker#SlotSupplier)                                              |
| <img src="/img/sdks/png/java.png" height="24" />       | [`SlotSupplier`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/tuning/SlotSupplier.html) |
| <img src="/img/sdks/png/python.png" height="24" />     | [`CustomSlotSupplier`](https://python.temporal.io/temporalio.worker.CustomSlotSupplier.html)                             |
| <img src="/img/sdks/png/typescript.png" height="24" /> | [`CustomSlotSupplier`](https://typescript.temporal.io/api/interfaces/worker.CustomSlotSupplier)                          |
| <img src="/img/sdks/png/dotnet.png" height="24" />     | [`CustomSlotSupplier`](https://dotnet.temporal.io/api/Temporalio.Worker.Tuning.CustomSlotSupplier.html)                  |

Slot Suppliers issue `SlotPermit`s.
These represent the right to use a slot of a specific type, namely Workflow, Activity, Local Activity, or Nexus.
You control whether a Worker can perform certain tasks by issuing or withholding permits.

Custom Slot Suppliers must implement these functions:

- `reserveSlot` - Called before polling for new tasks. Your implementation can block and must return a Slot Permit once it decides to accept new work.
- `tryReserveSlot` - Called for slot reservations in cases like eager activity processing. This must not block.
- `markSlotUsed` - Called when a slot is about to be used for a task (not while it’s held during polling). It provides information about the task.
- `releaseSlot` - Called when a slot is no longer needed, whether or not it was used.

Custom policies require more effort, but provide finer control over Task processing.
By implementing your own Slot Supplier, you can tailor how Workflows, Activities, and Nexus Operations are handled, optimizing performance for your specific needs.

## Slot supplier throttles

Auto-tuned suppliers may diverge from requested thresholds.
The resources a given Task will use can't be known ahead of time.
There is a fundamental tradeoff between how quickly a slot supplier is willing to accept Tasks and how well it can respect the defined thresholds.

Slot throttling is a mechanism to control the rate at which new slots for concurrent tasks are made available for processing.
This concept is part of the resource-based auto-tuning feature for Workers.
By waiting a brief period between making slots available, the Worker can assess how resource usage has changed since the last task began processing.

This throttle is called `rampThrottle` in the SDK options for resource-based slot suppliers.
It defines the minimum time the Worker will wait between handing out new slots after passing the minimum slots number.

**A higher `rampThrottle` trades off performance for safety.**

For example:

If a just-started worker were to have no throttle, and there was a backlog of Tasks, it might immediately accept 100 Tasks at once.
If each Task allocated 1GB of RAM, the Worker would likely run out of memory and crash.
The throttle enforces a wait before handing out new slots (after a minimum number of slots have been occupied) so you can measure newly consumed resources.

## Performance tuning examples 

The following examples show how to create and provision composite Worker tuners and set other
performance related options.
Each tuner provides slot suppliers for various Task types.
These examples focus on Activities and Local Activities, since Workflow Tasks normally do not need resource-based tuning.

### Go SDK

**Resource-based tuner:**

<!--SNIPSTART go-resource-based-tuner-->
[features/snippets/worker_tuner/worker_tuner.go](https://github.com/temporalio/features/blob/main/features/snippets/worker_tuner/worker_tuner.go)
```go
func resourceBasedTuner() (worker.Options, error) {
	tuner, err := worker.NewResourceBasedTuner(worker.ResourceBasedTunerOptions{
		TargetMem:    0.8,
		TargetCpu:    0.9,
		InfoSupplier: sysinfo.SysInfoProvider(),
	})
	if err != nil {
		return worker.Options{}, err
	}
	return worker.Options{
		Tuner: tuner,
	}, nil
}

```
<!--SNIPEND-->

**Composite tuner:**

A composite tuner lets you mix different slot supplier strategies for each Task type.
For example, you can use fixed-size slot suppliers for Workflow and Nexus Tasks while using resource-based slot suppliers for Activity and Local Activity Tasks.

<!--SNIPSTART go-composite-tuner-->
[features/snippets/worker_tuner/worker_tuner.go](https://github.com/temporalio/features/blob/main/features/snippets/worker_tuner/worker_tuner.go)
```go
func compositeTuner() (worker.Options, error) {
	options := worker.DefaultResourceControllerOptions()
	options.MemTargetPercent = 0.8
	options.CpuTargetPercent = 0.9
	options.InfoSupplier = sysinfo.SysInfoProvider()
	controller := worker.NewResourceController(options)
	wfSS, err := worker.NewFixedSizeSlotSupplier(10)
	if err != nil {
		return worker.Options{}, err
	}

	actSS, err := worker.NewResourceBasedSlotSupplier(controller, worker.DefaultActivityResourceBasedSlotSupplierOptions())
	if err != nil {
		return worker.Options{}, err
	}
	laSS, err := worker.NewResourceBasedSlotSupplier(controller, worker.DefaultActivityResourceBasedSlotSupplierOptions())
	if err != nil {
		return worker.Options{}, err
	}
	nexusSS, err := worker.NewFixedSizeSlotSupplier(10)
	if err != nil {
		return worker.Options{}, err
	}

	compositeTuner, err := worker.NewCompositeTuner(worker.CompositeTunerOptions{
		WorkflowSlotSupplier:      wfSS,
		ActivitySlotSupplier:      actSS,
		LocalActivitySlotSupplier: laSS,
		NexusSlotSupplier:         nexusSS,
	})
	if err != nil {
		return worker.Options{}, err
	}
	return worker.Options{
		Tuner: compositeTuner,
	}, nil
}

```
<!--SNIPEND-->

### Java SDK

```java
// Just resource based
WorkerOptions.newBuilder()
    .setWorkerTuner(
        ResourceBasedTuner.newBuilder()
            .setControllerOptions(
                ResourceBasedControllerOptions.newBuilder(0.8, 0.9).build())
            .build())
    .build())
// Combining different types
SlotSupplier<WorkflowSlotInfo> workflowTaskSlotSupplier = new FixedSizeSlotSupplier<>(10);
SlotSupplier<ActivitySlotInfo> activityTaskSlotSupplier =
    ResourceBasedSlotSupplier.createForActivity(
        resourceController, ResourceBasedTuner.DEFAULT_ACTIVITY_SLOT_OPTIONS);
SlotSupplier<LocalActivitySlotInfo> localActivitySlotSupplier =
    ResourceBasedSlotSupplier.createForLocalActivity(
        resourceController, ResourceBasedTuner.DEFAULT_ACTIVITY_SLOT_OPTIONS);
SlotSupplier<NexusSlotInfo> nexusSlotSupplier = new FixedSizeSlotSupplier<>(10);

WorkerOptions.newBuilder()
    .setWorkerTuner(
        new CompositeTuner(
            workflowTaskSlotSupplier,
            activityTaskSlotSupplier,
            localActivitySlotSupplier,
            nexusSlotSupplier))
    .build();
```

### TypeScript SDK

```tsx
// Just resource based
const resourceBasedTunerOptions: ResourceBasedTunerOptions = {
  targetMemoryUsage: 0.8,
  targetCpuUsage: 0.9,
};
const workerOptions = {
  tuner: {
    tunerOptions: resourceBasedTunerOptions,
  },
};
// Combining different types
const resourceBasedTunerOptions: ResourceBasedTunerOptions = {
  targetMemoryUsage: 0.8,
  targetCpuUsage: 0.9,
};
const workerOptions = {
  tuner: {
    activityTaskSlotSupplier: {
      type: 'resource-based',
      tunerOptions: resourceBasedTunerOptions,
    },
    workflowTaskSlotSupplier: {
      type: 'fixed-size',
      numSlots: 10,
    },
    localActivityTaskSlotSupplier: {
      type: 'resource-based',
      tunerOptions: resourceBasedTunerOptions,
    },
  },
};
```

### Python SDK

```python
# Just a resource based tuner, with poller autoscaling
tuner = WorkerTuner.create_resource_based(
    target_memory_usage=0.5,
    target_cpu_usage=0.5,
)
worker = Worker(
    client,
    task_queue="foo",
    tuner=tuner,
    workflow_task_poller_behavior=PollerBehaviorAutoscaling(),
    activity_task_poller_behavior=PollerBehaviorAutoscaling()
)
# Combining different types, with poller autoscaling
resource_based_options = ResourceBasedTunerConfig(0.8, 0.9)
tuner = WorkerTuner.create_composite(
    workflow_supplier=FixedSizeSlotSupplier(10),
    activity_supplier=ResourceBasedSlotSupplier(
        ResourceBasedSlotConfig(),
        resource_based_options,
    ),
    local_activity_supplier=ResourceBasedSlotSupplier(
        ResourceBasedSlotConfig(),
        resource_based_options,
    ),
)
worker = Worker(
    client,
    task_queue="foo",
    tuner=tuner,
    workflow_task_poller_behavior=PollerBehaviorAutoscaling(),
    activity_task_poller_behavior=PollerBehaviorAutoscaling()
)
```

### .NET C# SDK

```csharp
// Just resource based
var worker = new TemporalWorker(
    Client,
    new TemporalWorkerOptions("my-task-queue")
    {
        Tuner = WorkerTuner.CreateResourceBased(0.8, 0.9),
    });
// Combining different types
var resourceTunerOptions = new ResourceBasedTunerOptions(0.8, 0.9);
var worker = new TemporalWorker(
    Client,
    new TemporalWorkerOptions("my-task-queue")
    {
        Tuner = new WorkerTuner(
             new FixedSizeSlotSupplier(10),
             new ResourceBasedSlotSupplier(
                 new ResourceBasedSlotSupplierOptions(),
                 resourceTunerOptions),
             new ResourceBasedSlotSupplier(
                 new ResourceBasedSlotSupplierOptions(),
                 resourceTunerOptions)),
    });
```

---

# Task Queue performance

Source: https://docs.temporal.io/develop/worker-performance/task-queues

> Use Task Queue backlog metrics and CLI or SDK queries to evaluate Worker load and tune Task Queue processing.

This page covers Available Task Queue information, Evaluate Task Queue performance, and Task Queue processing tuning.

## Available Task Queue information 

> **ℹ️ Info:**
>
> The information listed in this section is readable using the `DescribeTaskQueueEnhanced` method in the [Go SDK](https://github.com/temporalio/sdk-go/blob/main/client/client.go), with the [Temporal CLI](https://github.com/temporalio/cli/releases/tag/v1.1.0) `task-queue describe` command, and using `DescribeTaskQueue` through RPC.
>

The Temporal Service reports information separately for each Task Queue type (not aggregated).
Use the following Task Queue properties to retrieve and evaluate information about Task Queue health and performance.
Available data include:

- [`ApproximateBacklogCount`](#ApproximateBacklogCountAndAge) and [`ApproximateBacklogAge`](#ApproximateBacklogCountAndAge)
- [`TasksAddRate`](#TasksAddRate-and-TasksDispatchRate) and [`TasksDispatchRate`](#TasksAddRate-and-TasksDispatchRate)
- [`BacklogIncreaseRate`](#BacklogIncreaseRate) (derived from [`TasksAddRate`](#TasksAddRate-and-TasksDispatchRate) and [`TasksDispatchRate`](#TasksAddRate-and-TasksDispatchRate))

### `ApproximateBacklogCount` and `ApproximateBacklogAge` 

`ApproximateBacklogCount` represents the approximate count of Tasks currently backlogged in this Task Queue.
The number may include expired Tasks as well as active Tasks, but it will eventually converge to the correct count over time.

`ApproximateBacklogAge` returns the approximate age of the oldest Task in the backlog.
The age is based on the creation time of the Task at the head of the queue.

You can rely on both these counts when making scaling decisions.

#### Known accuracy limitations 

These values are approximate.
The most common sources:

- **Overcount from invalid or expired Tasks**: Tasks belonging to cancelled, terminated, completed, or timed out Workflows and Activities stay in the count until they reach the head of the queue and are processed and discarded.
  An invalid or expired Task at the head is removed quickly, so it rarely holds up the count for long.
  A valid Task at the head can stay there longer when there aren't enough Workers to dispatch it. While it sits there, invalid or expired Tasks queued behind it cannot be removed.
  Invalid and expired Tasks are eventually accounted for, but the count may not return to a fully accurate value because of other sources of discrepancy like infrequent metadata updates and database row expirations.
- **Reset to zero on idle Task Queue unload**: If a Task Queue sees no activity for approximately 5 minutes - no Worker polls, no new Tasks added, and no other Task Queue calls (`DescribeTaskQueue`, `UpdateTaskQueueConfig`, etc.) - the Temporal Service unloads it from memory.
  When this happens, `ApproximateBacklogCount` reports zero until the Task Queue is reloaded by the next Worker poll, new Task, or Task Queue API call.
  An idle Task Queue with a backlog but no active Workers can therefore temporarily report zero even though there are Tasks waiting to be processed.
- **Sticky queue exclusion**: [Sticky queues](/sticky-execution) are not included in these values.
  Because Sticky queue Tasks only remain valid for a few seconds, this inaccuracy diminishes as the backlog grows.

### `TasksAddRate` and `TasksDispatchRate` 

Reports the approximate Tasks-per-second added to or dispatched from a Task Queue.
This rate is averaged over the most recent 30-second time interval.
The calculations include Tasks that were added to or dispatched from the backlog as well as Tasks that were immediately dispatched and bypassed the backlog (sync-matched).

The actual Task delivery count may be significantly higher than the number reported by these two values:

- Eager dispatch refers to a Temporal feature where Activities can be requested by an SDK using one Workflow Task completion response.
  Tasks using Eager dispatch do not pass through Task Queues.
- Tasks passed to Sticky Task Queues not included in the returned values for `TasksAddRate` and `TasksDispatchRate`.

### `BacklogIncreaseRate` 

Approximates the _net_ Tasks per second added to the backlog, averaged over the most recent 30 seconds.
This is calculated as:

```
TasksAddRate - TasksDispatchRate
```

- Positive values of `X` indicate the backlog is growing by about `X` Tasks per second.
- Negative values of `X` indicate the backlog is shrinking by about `X` Tasks per second.

While individual `add` and `dispatch` rates may be inaccurate due to Eager and Sticky Task Queues, the `BacklogIncreaseRate` reliably reflects the rate at which the backlog is shrinking or growing for backlogs older than a few seconds.

## Evaluate Task Queue performance 

A [Task Queue](/task-queue) is a lightweight, dynamically allocated queue.
[Worker Entities](/workers#worker-entity) poll the queue for [Tasks](/tasks#task) and retrieve Tasks to work on.
Tasks are contexts that a Worker progresses using a specific Workflow Execution, Activity Execution, or a Nexus Task Execution.
Each Task Queue type offers its Tasks to compatible Workers for Task completion.
The Temporal Service dynamically creates different [Task Queue types](/task-queue) including Activity Task Queues, Workflow Task Queues, and Nexus Task Queues.

With an accurate estimate of backlog Tasks, you can determine the optimal number of Workers to deploy.
Balance your Worker count with the number of Tasks to achieve the best performance.
This approach minimizes Task backlog saturation and reduces idle Workers.

Task Queue data provide numerical insights into your Task Queue activity and backlog characteristics.
Use these numbers to tune your production deployments.
Evaluate your Worker loads and assess whether you need to scale up or reduce your Worker deployment.

> **📝 Note:**
> RATE LIMITS
>
> [Visibility API rate limits](/evaluate/cloud/limits#visibility-api-rate-limit) apply to Task Queue performance data requests.
>

### Query Task Queue info with Temporal CLI 

The Temporal CLI helps you monitor and evaluate Worker performance.
Issue the following command to display a list of active Workers that have recently polled a Task Queue:

```
temporal task-queue describe \
    --task-queue YourTaskQueueName \
    [additional options]
```

This command retrieves poller information, backlog statistics, and task reachability for Task types (available in Temporal Server v1.25.0, Temporal CLI 1.1 and later).

> **⚠️ Warning:**
>
> Task reachability status is experimental.
> Determining Task reachability incurs a non-trivial computing cost.
> This feature may significantly change or be removed in a future release.
>

### Query Task Queue info with the Go SDK 

Retrieve Task Queue data using the Go SDK by calling `DescribeTaskQueueEnhanced`.
Specify the Task Queue name and set `ReportStats` to `true`, as in the following example:

```go
for _, taskQueueName := range taskQueueNames {
        resp, err := s.client.DescribeTaskQueueEnhanced(ctx, client.DescribeTaskQueueEnhancedOptions{
            TaskQueue:   taskQueueName,
            ReportStats: true,
        })
        if err != nil {
            log.Printf("Error describing task queue %s: %v", taskQueueName, err)
        }

        // Get the backlog count from the enhanced response
        backlogCount += getBacklogCount(resp)
    }
```

### Evaluate Worker availability and capacity issues 

Each Temporal [Server](/temporal-service/temporal-server) records the last time of each poll request.
This time is displayed in the `temporal task-queue describe` output.

- A `LastAccessTime` value exceeding one minute may indicate that the Worker fleet is at capacity or that Workers have shut down or been removed.

- Values under 5 minutes typically suggest the Worker fleet is at capacity.
  "At capacity" means that all Workflow and Activity slots are full.

- Values over 5 minutes since the last poll request usually suggest that Workers have shut down or been removed.
  Workers are removed if 5 minutes have passed since the last poll request.

### Manage your Worker fleet 

You can adjust the number of Workers to enhance Workflow Execution performance and manage your fleet size.
For instance, a large backlog of Tasks with too few Workers will slow down Workflow Execution completions and decrease processing efficiency.
Adding more Workers boosts speeds up completion rates and improves throughput.
An empty backlog indicates low Worker utilization, allowing you to reduce your fleet and associated costs.
Keep at least two Workers polling each Task Queue when you reduce your fleet. See [Run at least two Workers per Task Queue](/best-practices/worker#run-at-least-two-workers-per-task-queue).

The values provided by `temporal task-queue describe` can help you manage your Worker fleet deployment:

- `ApproximateBacklogAge` shows how long Tasks have been waiting to be dispatched.
  If this time grows too long, more Workers can boost Workflow efficiency.

- Calculate the demand per Worker by dividing the number of backlogged Tasks (`ApproximateBacklogCount`) by the number of Workers.
  Determine if your task processing rate is within an acceptable range for your needs using the per-Worker demand (how many Tasks each Worker has yet to process), the backlog consumption rate (`TasksDispatchRate`, the rate at which Workers are processing Tasks), and the dispatch latency (`ApproximateBacklogAge`, the time the oldest Task has been waiting to be assigned to a Worker).

- The backlog increase rate (`BacklogIncreaseRate`) shows the changing demand on your Workers over time.
  As this rate increases, you may need to add more Workers until demand and capacity are balanced.
  As it decreases, you may be able to reduce your Worker fleet.

## Task Queue processing tuning 

The following steps limit delays in Task Queue processing due to insufficient or unbalanced Workers.
Review these steps if you notice high `schedule_to_start` metrics.

The steps are arranged in the recommended order of execution.

### Hosts and resources provisioning

If currently provisioned Worker hosts are fully utilized (near full CPU usage, high load average, etc), additional Workers hosts have to be provisioned to increase the capacity of the Workers pool.

**It's possible to have too many Workers**

Monitor the poll success (`poll_success`/`poll_success_sync`) and poll timeout `poll_timeouts` Server metric counters.

Poll Success Rate = (`poll_success` + `poll_success_sync`) / (`poll_success` + `poll_success_sync` + `poll_timeouts`)

Poll Success Rate should be >90% in most cases of systems with a steady load. For high volume and low latency, try to target >95%.

If you see

1. low Poll Success Rate, and
2. low `schedule_to_start_latency`, and
3. low Worker hosts resource utilization at the same time,

then you might have too many workers, consider sizing down.

### Worker executor slots sizing

The main area to focus on when tuning is the number of Worker Executor Slots.
Increase the maximum number of working slots by adjusting `maxConcurrentWorkflowTaskExecutionSize` or `maxConcurrentActivityExecutionSize` if both of the following conditions are met:

1. The Worker hosts are underutilized (for example, no bottlenecks on CPU or load average).
2. The `worker_task_slots_available` metric from the corresponding Worker type frequently shows a depleted number of available Worker slots.

Alternatively, consider using a resource-based slot supplier as described [here](/develop/worker-performance#slot-suppliers).

### Poller count 

Sometimes, it can be appropriate to increase the number of task pollers.
This is usually more common in situations where your Workers have somewhat high latency when communicating with the server.
You can use automated poller tuning to handle this.

Consider manual adjustment if:

1. The Worker hosts are underutilized, for example, there are no bottlenecks on CPU or load average.
2. `worker_task_slots_available` metric from the corresponding Worker type shows that a significant percentage of Worker slots are available on a regular basis.
3. The `schedule_to_start` metric is abnormally long.

Then consider increasing the number of pollers by adjusting `maxConcurrentWorkflowTaskPollers` or `maxConcurrentActivityTaskPollers`, depending on which type of `schedule_to_start` metric is elevated.

### Rate limiting

If, after adjusting the poller and executors count as specified earlier, you still observe an elevated `schedule_to_start`, underutilized Worker hosts, or high `worker_task_slots_available`, you might want to check the following:

- If server-side rate limiting per Task Queue is set by `WorkerOptions#maxTaskQueueActivitiesPerSecond`, remove the limit or adjust the value up. (See [Go](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) and [Java](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.Builder.html).)
- If Worker-side rate limiting per Worker is set by `WorkerOptions#maxWorkerActivitiesPerSecond`, remove the limit. (See [Go](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions), [TypeScript](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#maxconcurrentactivitytaskexecutions), and [Java](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.Builder.html).)

---

# Workflow cache tuning

Source: https://docs.temporal.io/develop/worker-performance/workflow-cache

> Tune Workflow cache size and thread limits to balance Worker memory use against replay latency.

When the number of cached Workflow Executions reported by `sticky_cache_size` hits `workflowCacheSize` _or_ the number of threads reported by the `workflow_active_thread_count` metrics gauge hits `maxWorkflowThreadCount`, Workflow Executions will start to be evicted from the cache.
An evicted Workflow Execution will need to be replayed when it gets any action that may advance it.

If the Workflow Cache limits described above are hit, and Worker hosts have enough free RAM and are not close to reasonable thread limits, then you may choose to increase `workflowCacheSize` and `maxWorkflowThreadCount` limits to decrease the overall latency and cost of the Replays in the system.
If the opposite occurs, consider decreasing the limits.

> **📝 Note:**
>
> In CoreSDK based SDKs, like TypeScript, this metric works differently and should be monitored and adjusted on a per Worker and Task Queue basis.
>
> The `maxWorkflowThreadCount` and `workflow_active_thread_count` parameters are for the Java SDK only.
>

---

# Worker tuning quick reference

Source: https://docs.temporal.io/develop/worker-tuning-reference

> A quick reference guide for Temporal Worker configuration defaults across SDKs, organized by resource type (compute, memory, IO) with key metrics for each.

This page provides a quick reference for Worker configuration options and their default values across Temporal SDKs.
Use this guide alongside the comprehensive [Worker performance](/develop/worker-performance) documentation for detailed tuning guidance.

Worker performance is constrained by three primary resources:

| Resource | Description |
|----------|-------------|
| **Compute** | CPU-bound operations, concurrent Task execution |
| **Memory** | Workflow cache, thread pools |
| **IO** | Network calls to Temporal Service, polling |

## How a Worker works

Workers poll a [Task Queue](/task-queue) in Temporal Cloud or a self-hosted Temporal Service, execute Tasks, and respond with the result.

```
┌─────────────────┐     Poll for Tasks       ┌──────────────────┐
│   - Worker      │ ◄─────────────────────── │ Temporal Service │
│   - Workflows   │                          │                  │
│   - Activities  │ ───────────────────────► │                  │
└─────────────────┘   Respond with results   └──────────────────┘
```

Multiple Workers can poll the same Task Queue, providing horizontal scalability.

### How Worker failure recovery works

When a Worker crashes or experiences a host outage:

1. The Workflow Task times out
2. Another available Worker picks up the Task
3. The new Worker replays the Event History to reconstruct state
4. Execution continues from where it left off

Step 2 needs another Worker polling the same Task Queue. With only one Worker, the Task waits until that Worker restarts.
Run at least two Workers per Task Queue, as described in
[Run at least two Workers per Task Queue](/best-practices/worker#run-at-least-two-workers-per-task-queue).

For more details on Worker architecture, see [What is a Temporal Worker?](/workers)

## Compute settings

Compute settings control how many Tasks a Worker can execute concurrently.

### Compute configuration options

| Setting | Description |
|---------|-------------|
| `MaxConcurrentWorkflowTaskExecutionSize` | Maximum concurrent Workflow Tasks |
| `MaxConcurrentActivityExecutionSize` | Maximum concurrent Activity Tasks |
| `MaxConcurrentLocalActivityExecutionSize` | Maximum concurrent Local Activities |
| `MaxWorkflowThreadCount` / `workflowThreadPoolSize` | Thread pool for Workflow execution |

### Compute defaults by SDK

| SDK | MaxConcurrentWorkflowTaskExecutionSize | MaxConcurrentActivityExecutionSize | MaxConcurrentLocalActivityExecutionSize | MaxWorkflowThreadCount |
|-----|----------------------------------------|----------------------------------------|---------------------------------------------|------------------------|
| **Go** | 1,000 | 1,000 | 1,000 | - |
| **Java** | 200 | 200 | 200 | 600 |
| **TypeScript** | 40 | 100 | 100 | 1 (reuseV8Context) |
| **Python** | 100 | 100 | 100 | - |
| **.NET** | 100 | 100 | 100 | - |

### Resource-based slot suppliers

Instead of fixed slot counts, you can use resource-based slot suppliers that automatically adjust available Task slots based on CPU and memory utilization.
For implementation details, see [Slot suppliers](/develop/worker-performance#slot-suppliers).

## Memory settings

Memory settings control the Workflow cache size and thread pool allocation.

### Memory configuration options

| Setting | Description |
|---------|-------------|
| `MaxCachedWorkflows` / `StickyWorkflowCacheSize` | Number of Workflows to keep in cache |
| `MaxWorkflowThreadCount` | Thread pool size |
| `reuseV8Context` (TypeScript) | Reuse V8 context for Workflows |

### Memory defaults by SDK

| SDK | MaxCachedWorkflows / StickyWorkflowCacheSize |
|-----|----------------------------------------------|
| **Go** | 10,000 |
| **Java** | 600 |
| **TypeScript** | Dynamic (for example, 2000 for 4 GiB RAM) |
| **Python** | 1,000 |
| **.NET** | 10,000 |

For cache tuning guidance, see [Workflow cache tuning](/develop/worker-performance/workflow-cache).

## IO settings

IO settings control the number of pollers and rate limits for Task Queue interactions.

### IO configuration options

| Setting | Description |
|---------|-------------|
| `MaxConcurrentWorkflowTaskPollers` | Number of concurrent Workflow pollers |
| `MaxConcurrentActivityTaskPollers` | Number of concurrent Activity pollers |
| `Namespace APS` | Actions per second limit for Namespace |
| `TaskQueueActivitiesPerSecond` | Activity rate limit per Task Queue |

### IO defaults by SDK

| SDK | MaxConcurrentWorkflowTaskPollers | MaxConcurrentActivityTaskPollers | Namespace APS | TaskQueueActivitiesPerSecond |
|-----|----------------------------------|----------------------------------|---------------|------------------------------|
| **Go** | 2 | 2 | 400 | Unlimited |
| **Java** | 5 | 5 | - | - |
| **TypeScript** | 10 | 10 | - | - |
| **Python** | 5 | 5 | - | - |
| **.NET** | 5 | 5 | - | - |

### Poller autoscaling

Use poller autoscaling to automatically adjust the number of concurrent polls based on workload.
For configuration details, see [Configuring poller options](/develop/worker-performance/configuration#configuring-poller-options).

## Metrics reference by resource type

Use these metrics to identify bottlenecks and guide tuning decisions.
For the complete metrics reference, see [SDK metrics](/references/sdk-metrics).

### Compute-related metrics

| Worker configuration option | SDK metric |
|-----------------------------|------------|
| `MaxConcurrentWorkflowTaskExecutionSize` | [`worker_task_slots_available {worker_type = WorkflowWorker}`](/references/sdk-metrics#worker_task_slots_available) |
| `MaxConcurrentActivityExecutionSize` | [`worker_task_slots_available {worker_type = ActivityWorker}`](/references/sdk-metrics#worker_task_slots_available) |
| `MaxWorkflowThreadCount` | [`workflow_active_thread_count`](/references/sdk-metrics#workflow_active_thread_count) (Java only) |
| CPU-intensive logic | [`workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) |

Also monitor your machine's CPU consumption (for example, `container_cpu_usage_seconds_total` in Kubernetes).

### Memory-related metrics

| Worker configuration option | SDK metric |
|-----------------------------|------------|
| `StickyWorkflowCacheSize` | [`sticky_cache_total_forced_eviction`](/references/sdk-metrics#sticky_cache_total_forced_eviction), [`sticky_cache_size`](/references/sdk-metrics#sticky_cache_size), [`sticky_cache_hit`](/references/sdk-metrics#sticky_cache_hit), [`sticky_cache_miss`](/references/sdk-metrics#sticky_cache_miss) |

Also monitor your machine's memory consumption (for example, `container_memory_usage_bytes` in Kubernetes).

### IO-related metrics

| Worker configuration option | SDK metric |
|-----------------------------|------------|
| `MaxConcurrentWorkflowTaskPollers` | [`num_pollers {poller_type = workflow_task}`](/references/sdk-metrics#num_pollers) |
| `MaxConcurrentActivityTaskPollers` | [`num_pollers {poller_type = activity_task}`](/references/sdk-metrics#num_pollers) |
| Network latency | [`request_latency {namespace, operation}`](/references/sdk-metrics#request_latency) |

### Task Queue metrics

| Metric | Description |
|--------|-------------|
| [`poll_success_sync_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_sync_count) | Sync match rate (Tasks immediately assigned to Workers) |
| [`approximate_backlog_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count) | Approximate number of Tasks in a Task Queue |
| [`approximate_backlog_age`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_age) | Approximate age, in seconds, of the oldest Task in a Task Queue backlog |

Task Queue statistics are also available via the `DescribeTaskQueue` API:
- `ApproximateBacklogCount`
- `ApproximateBacklogAge`
- `TasksAddRate`
- `TasksDispatchRate`
- `BacklogIncreaseRate`

For more on Task Queue metrics, see [Available Task Queue information](/develop/worker-performance/task-queues#task-queue-metrics).

### Failure metrics

| Metric | Description |
|--------|-------------|
| [`long_request_failure`](/references/sdk-metrics#long_request_failure) | Failures for long-running operations (polling, history retrieval) |
| [`request_failure`](/references/sdk-metrics#request_failure) | Failures for standard operations (Task completion responses) |

Common failure codes:
- `RESOURCE_EXHAUSTED` - Rate limits exceeded
- `DEADLINE_EXCEEDED` - Operation timeout
- `NOT_FOUND` - Resource not found

## Worker tuning tips

1. **Scale test before production**: Validate your configuration under realistic load.
2. **Infrastructure matters**: Workers don't operate in a vacuum. Consider network latency, database performance, and external service dependencies.
3. **Tune and observe**: Make incremental changes and monitor metrics before making additional adjustments.
4. **Identify the bottleneck**: Use the [theory of constraints](https://en.wikipedia.org/wiki/Theory_of_constraints). Improving non-bottleneck resources won't improve overall throughput.

For detailed tuning guidance, see:
- [Worker performance](/develop/worker-performance)
- [Worker deployment and performance best practices](/best-practices/worker)
- [Performance bottlenecks troubleshooting](/troubleshooting/performance-bottlenecks)

## Related resources

- [What is a Temporal Worker?](/workers) - Conceptual overview
- [Worker performance](/develop/worker-performance) - Comprehensive tuning guide
- [Worker tuning simulator](/demos/worker-tuning-simulator) - Interactive demo of Pollers, Slots, and cache limits
- [Worker deployment and performance](/best-practices/worker) - Best practices
- [SDK metrics reference](/references/sdk-metrics) - Complete metrics documentation
- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) - Safe deployments
- [Workers in production](https://temporal.io/blog/workers-in-production) - Blog post
- [Introduction to Worker Tuning](https://temporal.io/blog/an-introduction-to-worker-tuning) - Blog post

---

# Dual Visibility

Source: https://docs.temporal.io/dual-visibility

> This guide on Temporal Dual Visibility explains how to set up, configure, and use Dual Visibility in Temporal Server versions. Learn about configuring primary and secondary Visibility stores, migrating databases, and ensuring a smooth transition for Visibility data.

This page discusses [Dual Visibility](#dual-visibility).

## What is Dual Visibility? 

Dual Visibility is a feature that lets you set a secondary Visibility store in addition to a primary store in your Temporal Service.
Setting up Dual Visibility is optional and can be used to [migrate your Visibility database](/self-hosted-guide/visibility/migrate) or create a backup Visibility store.

For example, if you have Cassandra configured as your Visibility database, you can set up a supported SQL database as your secondary Visibility store and gradually migrate your data to the secondary store before deprecating your primary one.

A Dual Visibility setup requires two Visibility store configurations:

- **Primary Visibility:** The primary Visibility store where Visibility data is written to and read from by default. The primary Visibility store is set with the `visibilityStore` configuration key in your Temporal Service.
- **Secondary Visibility:** A secondary storage for your Visibility data. The secondary Visibility store is set with the `secondaryVisibilityStore` configuration key in your Temporal Service.

For configuration details, see [Dual Visibility setup](/self-hosted-guide/visibility/dual-visibility).

The following combinations are allowed in a Dual Visibility setting.

| Primary                     | Secondary                       |
| --------------------------- | ------------------------------- |
| Standard (Cassandra or SQL) | Advanced (SQL or Elasticsearch) |
| Advanced (SQL)              | Advanced (SQL)                  |
| Advanced (Elasticsearch)    | Advanced (Elasticsearch)        |

With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to write to primary only, secondary only, or to both primary and secondary Visibility stores.
When migrating from one Visibility store database to another, set up the database you want to migrate to as your secondary Visibility store.

You can plan your migration using specific dynamic configuration keys that help you transition your read and write operations from the primary to the secondary Visibility store.
For details on migrating your Visibility store databases, see [Dual Visibility](/self-hosted-guide/visibility/dual-visibility).

---

# Dynamic handler

Source: https://docs.temporal.io/dynamic-handler

> Dynamic Handlers can serve as fallback mechanisms for handling Workflows, Activities, Signals, Queries, or Updates that aren't registered by name.

This page discusses [Dynamic Handler](#dynamic-handler).

## What is a Dynamic Handler? 

Temporal supports Dynamic Workflows, Activities, Signals, and Queries.

> **📝 Note:**
>
> Currently, the Temporal SDKs that support Dynamic Handlers are:
>
> - [Java](/develop/java/workflows/message-passing#dynamic-handler)
> - [Python](/develop/python/workflows/message-passing#dynamic-handler)
> - [.NET](/develop/dotnet/workflows/message-passing#dynamic-handler)
> - [Go](/develop/go/workflows/dynamic-workflow)
> - [Ruby](/develop/ruby/workflows/message-passing#dynamic-handler)
>

These are unnamed handlers that are invoked if no other statically defined handler with the given name exists.

Dynamic Handlers provide flexibility to handle cases where the names of Workflows, Activities, Signals, or Queries aren't known at run time.

> **⚠️ Caution:**
>
> Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach.
> Overusing them can lead to maintainability and debugging issues down the line.
>
> Instead, Workflows, Activities, Signals, and Queries should be defined statically whenever possible, with clear names that indicate their purpose.
> Use static definitions as the primary way of structuring your Workflows.
>
> Reserve Dynamic Handlers for cases where the handler names are not known at compile time and need to be looked up dynamically at runtime.
> They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic.
>

---

# Temporal Encyclopedia

Source: https://docs.temporal.io/encyclopedia

> Discover the key concepts, components, and features of Temporal for building scalable and reliable applications. Learn about Temporal SDKs, Workflows, Activities, Workers, and more.

[Temporal](/evaluate/why-temporal) provides developers a suite of effective tools for building reliable applications at scale.

The following Encyclopedia pages describe the concepts, components, and features of Temporal in detail:

- [Temporal](/temporal)
- [Temporal architecture](/encyclopedia/architecture/temporal-architecture)
- [Temporal SDKs](/encyclopedia/architecture/temporal-sdks)
- [Temporal Client](/encyclopedia/temporal-client)
- [Workflows](/workflows)
- [Activities](/activities)
- [Failures and error handling](/encyclopedia/failures-and-error-handling)
- [Workers](/workers)
- [Event History](/encyclopedia/event-history)
- [Workflow Message Passing](/encyclopedia/workflow-message-passing)
- [Child Workflows](/child-workflows)
- [Visibility](/visibility)
- [Temporal Service](/temporal-service)
- [Namespaces](/namespaces)
- [Temporal Nexus](/nexus)
- [Extensibility](/encyclopedia/extensibility)

For a complete list of Temporal terms, see the [Glossary](/glossary).

For information on how to implement the developer-facing features see the [Develop](/develop) section.

For information on how to use Temporal Cloud see the [Temporal Cloud production deployment](/cloud) section.

For information on how to self-host a Temporal Service see the [Self-hosted production deployment](/self-hosted-guide) section.

---

# Application failures

Source: https://docs.temporal.io/encyclopedia/application-failures

> Learn what application failures are in Temporal, how they differ from platform failures, and how errors propagate between Activities and Workflows.

Temporal handles many types of failures automatically through Durable Execution.
Worker crashes, network interruptions, and infrastructure outages are all recovered from without any intervention.
But some failures require your application to detect and respond to them.
Understanding which failures Temporal handles and which ones your application must handle is fundamental to building reliable Temporal applications.

## Platform failures vs application failures 

Failures fall into two categories based on where they are detected and mitigated: platform failures and application failures.

### Platform failures

Platform failures occur due to issues with the infrastructure: server outages, network interruptions, Worker crashes, or other environmental factors outside of your application's control.
Temporal's Durable Execution handles these failures transparently.
When a Worker crashes mid-execution, another Worker picks up the work and continues from where it left off.
Your application code does not need to account for these failures.

Platform failures are resolved through **forward recovery**: the system retries the failed operation, and if the retry succeeds, the application continues from the point of failure without undoing any previous work.

### Application failures

Application failures are generated by your code.
They indicate an issue with your application logic, such as invalid input data, a business rule violation, or a failed call to an external service.

Application failures do not resolve on their own through retries alone.
Recovering from an application failure may require fixing a bug, passing different input data, or performing some external mitigation.

Application failures often involve **backward recovery**: the system undoes some of the work that has already been performed to return to a previous state.
For example, if a payment step fails after inventory has already been reserved, the application may need to release that inventory.

For guidance on categorizing failures and deciding how to handle them, see [Error handling](/best-practices/error-handling).

## How Temporal represents failures 

When a failure surfaces to your application code, the SDK represents it as a typed error object.
Each SDK uses the conventions of its language: what is called a Failure in one SDK might be called an Error or Exception in another.

Most SDKs have a base class that other failure types extend.
This provides a common interface and shared behavior across different failure types.
For example:

- TypeScript: [TemporalFailure](https://typescript.temporal.io/api/classes/common.TemporalFailure)
- Java: [TemporalFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TemporalFailure.html)
- Python: [FailureError](https://python.temporal.io/temporalio.exceptions.FailureError.html)
- PHP: [TemporalFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-TemporalFailure.html)
- Go: Uses specific error types rather than a base class

For the complete list of failure types and their SDK-specific classes, see [Failures reference](/references/failures).

Errors that extend this base class are referred to as **Temporal failures**.
These are the SDK's typed error classes for failures that surface to application code, whether generated by the system (such as ActivityFailure or TimeoutFailure) or by your code (ApplicationFailure).
Platform failures like Worker crashes and network interruptions do not produce Temporal failure objects. The platform handles those transparently through retries.

The SDK uses whether an error is a Temporal failure to determine how to handle it.
By default, in Workflow code, throwing a Temporal failure fails the Workflow Execution, while throwing any other error fails the Workflow Task and is retried automatically.
Most SDKs let you list other exception types that fail the Workflow Execution. See [Errors in Workflows](/references/failures#errors-in-workflows).

The Temporal failure types are:

| Failure type | Description |
| :--- | :--- |
| **Application Failure** | Raised by your code to indicate application-specific errors. This is the only failure type you create directly. |
| **Activity Failure** | Wraps an error from an Activity Execution. The `cause` field contains the underlying error. |
| **Child Workflow Failure** | Wraps an error from a Child Workflow Execution. |
| **Timeout Failure** | Occurs when an Activity or Workflow exceeds its configured timeout. |
| **Cancelled Failure** | Results from cancellation of a Workflow, Activity, or Timer. |
| **Terminated Failure** | Occurs when a Workflow Execution is forcefully terminated. |
| **Server Failure** | Originates from the Temporal Service itself. |

Do not extend the base failure class or any of its children in your code.
The provided classes are designed to work with Temporal's serialization mechanism, which converts failures to Protocol Buffer messages for communication across process and language boundaries.
Custom subclasses can break this serialization and lead to unexpected behavior.

### Application Failure

Application Failure is the failure type you use to communicate application-specific errors.
It is the only failure type designed to be created and thrown directly by your code.

When you throw an Application Failure, you can set these fields:

- **message**: A human-readable description of the error.
- **type**: A string that categorizes the failure (for example, `"InvalidInput"` or `"InsufficientFunds"`).
- **non_retryable**: A flag that prevents the operation from being retried, regardless of the Retry Policy.
- **details**: Additional data about the failure.

Any non-Temporal error thrown from an Activity is automatically converted to an Application Failure.
During this conversion, the error's type name, message, and call stack are preserved, and `non_retryable` is set to `false`.

### Failure Converters

When Temporal returns a failure, the default Failure Converter copies error messages and stack traces as plain text.
This text is accessible in the Web UI and through the CLI.

If your errors might contain sensitive information, you can encrypt the message and stack trace by configuring a custom Failure Converter with a codec.
See [Failure Converter](/failure-converter) for details.

## Workflow Task failures vs Workflow Execution failures 

When an error occurs in Workflow code, it produces one of two outcomes depending on the error type: a Workflow Task failure or a Workflow Execution failure.
Understanding the difference is important because they have very different implications.

|  | Workflow Task failure | Workflow Execution failure |
| :--- | :--- | :--- |
| **Caused by** | Non-Temporal errors (null reference, division by zero, type errors, non-determinism errors) | Temporal failures thrown by your code, such as Application Failure |
| **Retried?** | Yes, automatically | No |
| **Workflow state** | Preserved. You can fix the bug and redeploy without losing progress. | "Failed" state permanently. No more attempts are made. |
| **Typical cause** | A bug in the Workflow code | A permanent business logic failure where retrying with the same input will not help |

When a Workflow Task failure is retried:

1. The Worker removes the Workflow Execution from its cache.
2. The Temporal Service schedules a new Workflow Task on the original Task Queue.
3. A Worker picks up the Task and replays the Workflow Execution from Event History to restore the correct state before continuing.

## How errors propagate 

When an Activity fails, Temporal wraps the error in an Activity Failure before delivering it to the Workflow.
The Activity Failure provides context about the failure, including the Activity Type, the number of retry attempts, and the original cause.

The original error is in the `cause` field.
For example, if an Activity throws an Application Failure with `type: "InvalidInput"`, the Workflow receives an Activity Failure whose `cause` is that Application Failure.
If an Activity times out instead, the `cause` is a Timeout Failure.

This wrapping pattern applies to other execution types as well.
A failed Child Workflow delivers a Child Workflow Failure to the parent Workflow, with the original error in the `cause` field.

If a Temporal failure propagates unhandled through Workflow code, it fails the Workflow Execution.
The exception is Cancelled Failure, which puts the Workflow in "Cancelled" state instead of "Failed".

### The outermost error type determines retryability 

When an Activity returns an error, the SDK inspects the **outermost** error to decide how to represent the failure to the Temporal Service.
The SDK performs a type check on the outermost error and converts it to a Protocol Buffer [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) message.
If the outermost error is an Application Failure, the SDK preserves its `non_retryable` flag and `type` field in the resulting `ApplicationFailureInfo` proto.
If the outermost error is any other type, the SDK falls back to creating a default, **retryable** `ApplicationFailureInfo`.

The Temporal Service only inspects the **top-level** `failure_info` on the Failure proto when making retry decisions.
The original error is preserved in the `cause` chain, but the Service does not look at `cause` to determine retryability.

This means that wrapping an Application Failure in a generic language error silently loses the `non_retryable` flag.
If an Activity throws a non-retryable Application Failure, but your code catches it and re-throws it wrapped in a standard error, the Activity will be retried despite the original intent.

> **⚠️ Caution:**
> Wrapping errors can lose retryability flags
> If you need to add context to an error, wrap it in another Application Failure that preserves the `non_retryable` flag.
> Do not wrap Application Failures in generic language errors (such as `Error` in TypeScript, `Exception` in Python, or `fmt.Errorf` in Go), as this causes the SDK to treat the error as a new, retryable failure.

This behavior is consistent across all Temporal SDKs.
For language-specific examples and correct wrapping patterns, see the error handling guide for your SDK.

## Failures in Event History 

Failures are recorded in Event History, which provides a detailed record for debugging.

### Activity failures

An Activity Execution that completes results in three Events: `ActivityTaskScheduled`, `ActivityTaskStarted`, and `ActivityTaskCompleted`.

If an Activity fails and the Retry Policy does not cause it to retry, the Temporal Service adds an `ActivityTaskFailed` Event that contains the error details.
If an Activity times out, an `ActivityTaskTimedOut` Event is added instead.

While an Activity is running, `ActivityTaskScheduled` is the most recent Event visible for that Activity.
The `ActivityTaskStarted` Event is not written until the Activity Task closes, because the final retry attempt number (an attribute of `ActivityTaskStarted`) is not known until then.

You can view pending Activity Executions in the Web UI's Pending Activities section, which shows the Activity Type, current retry attempt, remaining attempts, and heartbeat information.

### Workflow Execution failures

An Activity failure does not directly cause a Workflow Execution failure.
If an Activity fails and the error propagates out of the Workflow function without being caught (or is caught and intentionally re-raised as an Application Failure), the Workflow Execution fails.

When a Workflow Execution fails, the Temporal Service adds a `WorkflowExecutionFailed` Event.
If the failure was caused by an unhandled Activity error, the `activityFailureInfo` is attached to that Event.

---

# How Temporal works

Source: https://docs.temporal.io/encyclopedia/architecture/how-temporal-works

> How the Temporal Client, Server, and Worker interact during a Workflow and Activity lifecycle, with a step-by-step walkthrough and interactive demo.

In a Temporal application, the [Temporal Client](/encyclopedia/architecture/temporal-architecture#client-application), [Temporal Server](/encyclopedia/architecture/temporal-architecture#the-temporal-server), and [Worker](/workers#worker) are three separate parts that communicate over the network. The Client sends control requests such as starting a Workflow, sending a Signal, or requesting a result. The Server records [Event History](/workflow-execution/event), schedules [Tasks](/tasks#task) on [Task Queues](/task-queue), and persists state to a database. The Worker polls Task Queues and runs Workflow and Activity code.
The Client does not execute Workflow or Activity code. The Server does not run your application code directly. 

Understanding how these three parts exchange requests, Tasks, Commands, and Events is the basis for understanding Temporal's internal mechanics.

## Relationship between the Client, Service, and Worker

### Service

The Temporal Service is a group of backend services that together form the core Temporal system. It exposes a [Frontend Service](/encyclopedia/architecture/temporal-architecture#frontend-service), which is the network API that everything else connects to, and [backend services](/encyclopedia/architecture/temporal-architecture#temporal-service-backend) like the [History Service](/encyclopedia/architecture/temporal-architecture#history-service), which keeps the full Event History and state for each Workflow Execution, and the [Matching Service](/encyclopedia/architecture/temporal-architecture#matching-service), which manages [Task Queues](/task-queue) that hold pending work items for [Workers](/workers#worker).

### Server

The [Server](/encyclopedia/architecture/temporal-architecture#the-temporal-server) is a part of the Temporal Service that handles all the Temporal functionality. All of this state is written to a database, so the Server can remember what has happened even if processes or machines fail. You can run a Server yourself (self‑hosted cluster) or use [Temporal Cloud](/cloud), which is the same kind of service but operated for you.

### Client

A [Temporal Client](/encyclopedia/architecture/temporal-architecture#client-application) is a library object from a [Temporal SDK](/develop) that your application code uses to talk to the [Temporal Server's Frontend](/encyclopedia/architecture/temporal-architecture#frontend-service). It opens a network connection (using gRPC) to the Server and sends requests like "start this Workflow with these inputs," "cancel this Workflow," "send this Signal to a running Workflow," "run this Query on a Workflow," or "get the result of this Workflow."

The Client does not run Workflow or Activity code; it only sends and receives these control requests. This is important because it gives your existing services, CLIs, or UIs a simple and consistent way to interact with Temporal without knowing anything about how Tasks, Task Queues, or Event histories are implemented inside the Server.

## Worker

A [Worker](/workers#worker) is the application component responsible for running your Workflow and Activity code, built with a Temporal SDK. Workers have historically been long-running processes deployed on bare metal, virtual machines, or in containers.

They can now also run as [Serverless Workers](/serverless-workers) on serverless cloud infrastructure, such as AWS Lambda. Regardless of how and where Workers are deployed, they are always separate and distinct from the Temporal Service.

The Worker connects to the Server and polls Task Queues managed by the Matching Service to ask "do you have work for me?"

When it receives a Workflow Task, it replays the Workflow's Event History and runs your Workflow code until it either completes or reaches a point where it must wait (for example, waiting for an Activity or a Timer). It then sends Commands back to the Server that describe what should happen next, such as "schedule this Activity" or "start this Timer."

When it receives an Activity Task, it calls your Activity function or method and then sends the outcome (success or failure and any additional result data) back to the Server, which records this as Events in the Event History. Temporal explicitly does not run your Workflow or Activity code inside the Server; that code only runs in Workers, which are under your control.

The Server records all state and creates Workflow and Activity Tasks in Task Queues. Workers poll those queues, run your Workflow and Activity code, and send back Commands and results. The Server uses this information to update the [Event History](/workflow-execution/event) so that any Client can later request the current status or the final result, and execution can safely continue (or replay) even if individual Worker processes or machines fail.

This is the heart of how durable execution works.

## End-to-end lifecycle of a Workflow and Activity

The following fifteen steps describe one Workflow Execution from start to finish, including a single Activity. The interactive demo below shows the same sequence and highlights which components communicate at each point.

### Interactive demo

### Workflow lifecycle

**Step 1: The Temporal Client asks Temporal to start a Workflow**

Your application calls `StartWorkflowExecution` on the Temporal API (gRPC) exposed by the Frontend Service. The request includes the Workflow type (which Workflow function/class to run), input arguments, and the Task Queue name. Frontend forwards the request to the History Service, which owns this Workflow Execution.

**Step 2: The History Service creates the Workflow Execution**

History creates a new Workflow Execution in the database and initializes its Event History with: `WorkflowExecutionStarted` and `WorkflowTaskScheduled` (to tell a Worker to run Workflow code). It also creates an internal Transfer Task telling the system to put a Workflow Task on the specified Task Queue.

**Step 3: The Matching Service adds a Workflow Task to the Task Queue**

A background queue processor in the History Service reads the Transfer Task and calls the Matching Service to `AddWorkflowTask` for that Task Queue. The Task Queue now has one pending Workflow Task for this Workflow Execution.

**Step 4: A Worker polls for a Workflow Task**

A Worker process (your code + SDK) is already polling that Task Queue using `PollWorkflowTask` via Frontend Service. The Frontend Service asks Matching for a Task. The Matching Service picks this Workflow Task and tells the History Service that it was started. The History Service appends `WorkflowTaskStarted` to the Event History and returns the Workflow Task (plus history) to the Worker through Frontend.

**Step 5: The Worker runs your Workflow code**

The SDK replays the Event History to reconstruct logical Workflow state, then starts your Workflow function/method. The Workflow runs until either returns a result or until it needs to wait (for an Activity, Timer, Signal, etc.).

**Step 6: Your Workflow asks for work via Commands**

When your Workflow calls Temporal APIs (for example, "execute Activity"), the SDK records Commands like `ScheduleActivityTask` or `StartTimer` instead of executing them directly. When the Workflow cannot make more progress, the Worker sends `RespondWorkflowTaskCompleted` to Frontend, carrying the list of Commands. Then the Frontend forwards this list to History.

**Step 7: The History Service turns Commands into Events and new Tasks**

The History appends Events based on the Commands and updates state, for example: For `ScheduleActivityTask`: `WorkflowTaskCompleted`, then `ActivityTaskScheduled`. It then creates new internal Tasks (Transfer/Timer Tasks) and, through the queue processors, calls Matching to add Activity Tasks to Activity Task Queues or later add more Workflow Tasks when needed.

At this point, the Workflow is waiting on whatever it asked for (Activities, Timers, etc.). When those complete, the Workflow will resume.

### Activity lifecycle 

**Step 8: The Matching Service puts an Activity Task on the Activity Task Queue**

For each `ActivityTaskScheduled` Event, a queue processor in History calls Matching to `AddActivityTask` on the corresponding Activity Task Queue.

**Step 9: An Activity Worker polls and starts the Activity**

A (possibly different) Worker is polling that Activity Task Queue using `PollActivityTask` via Frontend. Frontend asks Matching; Matching selects the Activity Task and notifies History that it has started. History records the started attempt and sets up any Activity timeout timers (for example, start-to-close). History doesn't append `ActivityTaskStarted` to the Event History yet; it writes that Event when the Activity Execution completes or fails for the last time. The Activity Task is then returned to the Worker via Frontend.

**Step 10: Worker runs your Activity code**

The Worker calls your Activity function/method with the inputs from the Task. This code can do I/O and call external services because it is not replayed the same way as Workflow code. 

**Step 11: Activity reports success or failure**

On success, the Worker sends `RespondActivityTaskCompleted` (with the result) to Frontend; Frontend forwards it to History. History appends `ActivityTaskStarted`, `ActivityTaskCompleted` (including the result), and `WorkflowTaskScheduled` (to wake up the Workflow), and adds a Transfer Task to create the next Workflow Task.

On failure, the Worker sends `RespondActivityTaskFailed`. If the [Retry Policy](/encyclopedia/retry-policies) allows another attempt, History schedules the retry without appending any Events, and the Activity Task returns to the Task Queue after the retry interval. When retries are exhausted or the failure is non-retryable, History appends `ActivityTaskStarted`, `ActivityTaskFailed`, and `WorkflowTaskScheduled`, and the failure propagates to the Workflow.

### Workflow resumes and finishes

**Step 12: A new Workflow Task is scheduled**

Because of the Activity completion (or other Events like Timers), History has appended Events and scheduled a new Workflow Task. A queue processor in History calls Matching to add that Workflow Task to the Workflow Task Queue.

**Step 13: Worker picks up the next Workflow Task and continues**

A Worker polls the Workflow Task Queue again (`PollWorkflowTask`). The Matching Service selects the Task; the History Service appends `WorkflowTaskStarted`. The Worker receives the updated Event History via Frontend. The SDK replays the Event History, unblocks the waiting Activity or Timer, and your Workflow code continues from this new state.

**Step 14: Workflow eventually closes**

This cycle (Workflow Task → Commands → Events → new Tasks) repeats as many times as needed. When the Workflow function/method returns a result, the Worker sends `RespondWorkflowTaskCompleted` with a `CompleteWorkflowExecution` command. The History Service appends `WorkflowTaskCompleted` and `WorkflowExecutionCompleted` and marks the Workflow Execution as closed. Workflows can also close via failure, cancellation, termination, or "continue-as-new," but in all cases they move from open to closed and do not reopen.

**Step 15: The Client reads the result or status from the Frontend Service**

While the Workflow is open, Clients can query it (read-only inspection of state) and send Signals to it. After it is closed, a Client can request the final result or failure by calling the SDK, which talks to Frontend; the Frontend Service reads the necessary data from the History Service/Persistence Layer and returns it.

---

# Temporal architecture

Source: https://docs.temporal.io/encyclopedia/architecture/temporal-architecture

> A comprehensive guide to Temporal's architecture, covering how the platform orchestrates durable Workflow Executions through its Frontend, backend services, and Worker coordination.

Before reading this, it would be helpful to read [Understanding Temporal](/evaluate/understanding-temporal) as it assumes knowledge of the following introductory concepts: Workflow, Activity, Worker, Task Queues, and Event History.

Temporal is a tool that abstracts away a lot of difficulty for anyone managing applications that need to be resilient to failures. This guide provides you with a high level overview of how Temporal works under the hood.

## Application structure recap

Before diving into the architecture, let's recap what you're responsible for as a developer: the Temporal Application code which includes writing the Activity Definition, Workflow Definition, and the code to configure and start the Workers that coordinate with a Temporal Service to execute your Workflow and Activity code. 

As part of the Temporal Platform, the Temporal Service runs the Temporal Server with its database and optional components, and is responsible for orchestrating execution. A Temporal application gains its durability, scalability, and reliability from the support provided by the Temporal Service.

In the wider platform, there are three kinds of clients that talk to the Temporal Server:

- Temporal's [command-line interface (CLI)](/cli)
- Temporal's [web-based user interface (Web UI)](/web-ui)
- A [Temporal Client](#client-application) embedded into the applications you run

## Client application

Temporal Clients offer a set of APIs to communicate with a Temporal Service. They are one aspect of your Temporal application and provided by a Temporal SDK. You instantiate and use it in your own application code to start and manage Workflow Executions.

Consider the example of an order processing system. 

A Temporal Client lets you:

- **Start a Workflow Execution** when a customer places an order.
- **Signal a Workflow Execution** to update the order if the customer changes their shipping address.
- **Query a Workflow Execution** to retrieve the current status of the order.
- **List Workflow Executions** to view all orders being processed.
- **Get the result of a Workflow Execution** to retrieve the final outcome of the order processing. 

Clients are responsible for starting Workflows by sending requests to initiate new Workflow Executions. They can query Workflows to retrieve current state or synchronous data from running Workflows and signal Workflows to send asynchronous messages that influence their behavior. Clients also manage Workflows by canceling or terminating Workflow Executions.

## The Temporal Server 

The Temporal Server is the heart of the platform. It's responsible for orchestrating Workflow Execution, maintaining state, and ensuring reliability.

The Temporal Server consists of a Frontend and multiple backend services, plus a database as a required external component. A Temporal Server may also include some optional components, such as Elasticsearch for advanced search visibility or Grafana for creating operational dashboards for observability.

### Frontend Service

From your application's perspective, the Frontend Service is the Temporal endpoint your Temporal Client talks to. Your client sends requests (start Workflow, Signal, Query, etc.) to the Frontend, and the Frontend forwards them to the appropriate backend services.

It handles rate limiting, authorization, validation, and routing requests internally to the right subsystem. Clients never communicate directly with backend services or Workers so all requests goes through this unified entry point.

### Temporal Service backend

The backend consists of several specialized services including the History, Matching, and Worker services, and a Persistence Layer.

#### History Service

The History Service is the part of Temporal that keeps track of everything that happens in each Workflow. It writes an ordered [Event History](/encyclopedia/event-history) to the database so Temporal always knows what has happened. This includes when a Workflow started, Activities ran, Signals were received, timers fired, and when it completed or failed. 

The Event History (every event/step of each Workflow) is the key to making your application reliable and crash-proof. When an error or failure happens in your app, Temporal will recreate the state by parsing the Event History and replaying each step.

That's why [determinism](/workflow-definition#deterministic-constraints) and [idempotency](/activity-definition#idempotency) are important when you're creating your Workflows.

The History Service is a central part of how Temporal provides Durable Execution. It persists all Workflow Execution state, including the Event History, any mutable state, and internal task queues like timers, transfers, replication, and visibility/indexing.

#### Matching Service

The Matching Service manages most of the coordination with the other services, especially the Task Queues or Task Queue partitions. This is where Tasks are dispatched to their respective queues before being picked up by the corresponding Workers. To determine how many more Tasks should be sent to a Worker from the Task Queue, Worker polling also takes place here. This is how Tasks are dispatched to different queues before being sent to a Worker.

When a Workflow Task or Activity needs to be done, the Matching Service finds an appropriate Worker polling the queue and hands off the Task.

#### Worker Service

The Worker Service is where all of the background functionality that keeps the Temporal Service running smoothly is handled. It takes care of all the internal/system Workflows that Temporal needs like maintenance jobs, cleanup, replication, archival, and visibility indexing. You won't need to directly interact with this layer because it's different from your application Workers. The Worker Service handles Temporal's own operational tasks, like archiving old Event Histories or running scheduled maintenance.

#### Persistence layer

You can configure the database that you want to store Workflow state, Event History, and Task Queues. You can choose between Cassandra, MySQL, PostgreSQL, and SQLite. This is also where you store metadata about your Workflows and other data that needs to be recorded in order to ensure Durable Execution. This persistence enables recovery and replay - if anything fails, Temporal can reconstruct the exact state of a Workflow from its Event History.

Application [Workers](/workers#worker) run your Workflow and Activity code on your own infrastructure. For how they coordinate with the Server, see [How Temporal works](/encyclopedia/architecture/how-temporal-works).

---

# About Temporal SDKs

Source: https://docs.temporal.io/encyclopedia/architecture/temporal-sdks

> Temporal SDKs are open-source tools enabling scalable and reliable application development. They feature APIs for Workflow and Activity execution, automatic retries, and resilience mechanisms, making it easier to build fault-tolerant applications.

A Temporal SDK (software development kit) is the open-source library you add to your application to use Temporal.

It provides everything needed to build Workflows, Activities, and other Temporal features in a specific programming language including APIs to configure a Temporal Client, develop your Temporal Application, and run [Workers](/workers#worker) you can scale out as your load grows.

## Temporal Applications 

Developers create Temporal Applications using an [official Temporal SDK](#official-sdks).

A Temporal Application is the code you write, comprised of [Workflow Definitions](/workflow-definition), [Activity Definitions](/activity-definition), code used to configure [Temporal Clients](#temporal-client), and code used to configure and start [Workers](/workers#worker).

Consider that the Workflow Definition code can be executed repeatedly.
The Temporal Platform can concurrently support millions to billions of Workflow Executions, each of which representing an invoked Workflow Definition.
A Temporal Application can run for seconds or years in the presence of arbitrary load and failures.

## Official SDKs 

**What are the officially supported SDKs?**

Each Temporal SDK targets a specific programming language.

- [Go SDK developer guide](/develop/go)
- [Java SDK developer guide](/develop/java)
- [Python SDK developer guide](/develop/python)
- [TypeScript SDK developer guide](/develop/typescript)
- [.NET SDK developer guide](/develop/dotnet)
- [Ruby SDK developer guide](/develop/ruby)
- [PHP SDK developer guide](/develop/php)
- [Rust SDK developer guide](/develop/rust)

### Third-party SDKs

The following third-party SDKs exist but are not officially supported by Temporal:

- [Swift](https://github.com/apple/swift-temporal-sdk) from [@Swift Community](https://github.com/apple)
- [Haskell](https://github.com/MercuryTechnologies/hs-temporal-sdk) from [@MercuryTechnologies](https://github.com/MercuryTechnologies) 
- [Clojure](https://github.com/manetu/temporal-clojure-sdk) from [@Manetu](https://github.com/manetu)
- [Scala](https://github.com/vitaliihonta/zio-temporal) from [@vitaliihonta](https://github.com/vitaliihonta)

## Why use a Temporal SDK? 

Temporal SDKs empower developers to concentrate on creating dependable and scalable business logic, alleviating the need to build home-grown supervisor systems to ensure reliability and fault-tolerance. This is possible because the Temporal SDK provides a unified library that abstracts the intricacies of how Temporal handles distributed systems.

SDKs are more than a development toolkit. The APIs encourage a code pattern that mirrors real-world processes, and the SDK's internal implementation, working with the [Temporal Service](/temporal-service), steps through that code at runtime to guarantee execution progression.

### Development pattern

By abstracting complexities and streamlining boilerplate code, developers can craft straightforward code that directly aligns with their business logic, enhancing code readability and bolstering developer productivity.

Consider a bank loan application.
Developers can design the business logic of a bank loan using the Temporal SDK.
The Workflow defines the overarching business logic, encompassing tasks such as validating applicant information, credit checks, loan approval, and applicant notifications, as Activities.

> **⚠️ Caution:**
> Do not copy and use code
>
> The following is pseudocode. For tested samples see your language SDK's developer's guide.
>

```
func LoanApplicationWorkflow {

    sdk.ExecuteActivity(CreditCheck)

    sdk.ExecuteActivity(AutomatedApproval)

    sdk.ExecuteActivity(NotifyApplicant)

    // ...
}
```

For instance, Temporal SDKs have built-in support for handling failures, timeouts, and retries.
In the event of an Activity failure, the SDK automatically initiates retries according to configurable policies established by the developer within the SDK. This streamlined process simplifies the integration of fault-tolerance mechanisms into applications.

### Replays

Another quality of the SDKs lies in their ability to replay Workflow Executions, a complex operation that contributes significantly to the Platform's promised reliability.

![The SDKs Replay code execution to continue from the last step](https://docs.temporal.io/diagrams/replay-basic.svg)

We will delve into this idea more later, but for now, it signifies that the SDKs can automatically continue a process from the point of interruption, should a failure occur.
This capability stems from the SDK's ability to persist each step the program takes.

## Temporal SDKs major components 

**What are the major components of Temporal SDKs?**

Temporal SDKs offer developers the following:

- A Temporal Client to communicate with a Temporal Service
- APIs to develop application code (Workflows & Activities)
- APIs to configure and run Workers

![Temporal SDK components create a runtime across your environment and a Temporal Service](https://docs.temporal.io/diagrams/temporal-sdk-components.svg)

Let's break down each one.

### Temporal Client

A Temporal Client acts as the bridge for communication between your applications and the Temporal Service.
The Client performs key functions that facilitate the execution of, management of, and communication with Workflows.

The most common operations that a Temporal Client enables you to perform are the following:

- Get the result of Workflow Execution.
- List Workflow Executions.
- Query a Workflow Execution.
- Signal a Workflow Execution.
- Start a Workflow Execution.

Developers use the Client as the main entry point for interacting with the application through Temporal, for example, to start or Signal Workflows, Query a Workflow's state, or get a Workflow result.

- [Temporal Client - Go](/develop/go/client/temporal-client)
- [Temporal Client - Java](/develop/java/client/temporal-client)
- [Temporal Client - PHP](/develop/php/client/temporal-client)
- [Temporal Client - Python](/develop/python/client/temporal-client)
- [Temporal Client - Ruby](/develop/ruby/client/temporal-client)
- [Temporal Client - Rust](/develop/rust/client/temporal-client)
- [Temporal Client - TypeScript](/develop/typescript/client/temporal-client)
- [Temporal Client - .NET](/develop/dotnet/client/temporal-client)

### APIs to develop Workflows

Workflows are defined as code: either a function or an object method, depending on the language.

The Workflow code uses Temporal SDK APIs to orchestrate the steps of the application.

A Workflow executes Activities (other functions that interact with external systems), handles and sends messages (Queries, Signals, Updates), and interacts with other Workflows.

This Workflow code, while executing, can be paused, resumed, and migrated across physical machines without losing state.

When a Workflow calls the API to execute an Activity, the Worker sends a [Command](/references/commands) back to the Temporal Service. The Temporal Service creates Activity Tasks in response which the same or a different Worker can then pick up and begin executing. In this way, the Worker and Temporal Service work together to incrementally execute Workflow code in a reliable way.
We discuss this more in the expandable [section below](#sdk-and-service-relationship).

The SDK APIs also enable developers to write code that more genuinely maps to their process. This is because without a specialized SDK, developers might have to write a lot of boilerplate code. This can lead to code that's hard to maintain, difficult to understand, or that doesn't directly correspond to the underlying business process.

The level of abstraction that APIs offer enables the developer to focus on business logic without having to worry about the intricacies of distributed computing such as retries, or having to explicitly maintain a state machine and the intermediate state for each step of the process.

Additionally, the state of the Workflow is automatically persisted so if a failure does occur, it resumes right where it left off.

- [Workflows - Go](/develop/go/workflows/basics)
- [Workflows - Java](/develop/java/workflows/basics)
- [Workflows - PHP](/develop/php/workflows/basics)
- [Workflows - Python](/develop/python/workflows/basics)
- [Workflows - Ruby](/develop/ruby/workflows/basics)
- [Workflows - Rust](/develop/rust/workflows/basics)
- [Workflows - TypeScript](/develop/typescript/workflows/basics)
- [Workflows - .NET](/develop/dotnet/workflows/basics)

### APIs to create and manage Worker Processes

Workers are responsible for executing Workflow and Activity code (application code). The SDK provides APIs for configuring and starting Workers, enabling developers to control how the code is executed.
Workers are horizontally scalable, often run with systems like Kubernetes, and configured according to the application's needs.

The Worker polls on the specified Task Queue, processing those Tasks, and reporting the results back to the Temporal Service. They execute both the Workflows and Activities, and the SDK ensures that they perform these tasks efficiently and reliably.

- [Workers - Go](/develop/go/workers)
- [Workers - Java](/develop/java/workers)
- [Workers - PHP](/develop/php/workers)
- [Workers - Python](/develop/python/workers)
- [Workers - Ruby](/develop/ruby/workers)
- [Workers - Rust](/develop/rust/workers)
- [Workers - TypeScript](/develop/typescript/workers)
- [Workers - .NET](/develop/dotnet/workers)

### APIs to customize Activity Execution behavior

Activities in Temporal are individual units of work that often represent non-deterministic parts of the code logic, such as querying a database or calling an external service. The SDK provides APIs to customize the behavior of an Activity Execution.

By default, if an Activity attempts to communicate with another system and encounters a transient failure like a network issue, Temporal ensures the Activity is tried again automatically.

However, Temporal enables developers to control a variety of timeouts, a Retry Policy, Heartbeat monitoring, and asynchronous completion.

- [Activities - Go](/develop/go/activities/basics)
- [Activities - Java](/develop/java/activities/basics)
- [Activities - PHP](/develop/php/activities/basics)
- [Activities - Python](/develop/python/activities/basics)
- [Activities - Ruby](/develop/ruby/activities/basics)
- [Activities - Rust](/develop/rust/activities/basics)
- [Activities - TypeScript](/develop/typescript/activities/basics)
- [Activities - .NET](/develop/dotnet/activities/basics)

## How SDKs work with the Temporal Service 

The section below explains how your Client, Workers, and the Temporal Service work together when you run Workflow and Activity code.

How do the Temporal SDKs work with the Temporal Service?

The Temporal Service functions more as a choreographer than a conductor. Rather than directly assigning tasks to Workers, the Temporal Service arranges the Tasks into a Task Queue while Workers poll the Task Queue. Developers may create a fleet of Workers and tune them so that a Task is picked up as soon as it is available. If a Worker goes down, Tasks can wait until the next Worker is available.

A Workflow might request to execute an Activity, start a Timer, or start a Child Workflow, each of which translates into a Command, dispatched to the Temporal Service.
In addition to acting on these Commands, the Temporal Service documents that interaction by appending their corresponding Events into the Workflow Execution's Event History.

Take for instance the call to execute an Activity. When a Workflow invokes it, the Worker doesn't immediately execute that Activity code. Instead, it generates a ScheduleActivityTask Command, dispatching it to the Temporal Service. In response, the Temporal Service queues up a new Activity Task. Only when a Worker finds itself free, it collects the task and begins executing the Activity code.

The Temporal Service persists Workflow Execution Event History, so that if there is a failure, the SDK Worker is able to Replay the execution and resume where it left off.

This is where the deterministic constraints of the Workflow Definition come into play, requiring the use of Activities to create side effects and interact with the outside world.

Activities perform work outside the Workflow, for example, calling an external API. Since the call can fail due to transient issues, you define it as an Activity and configure retry options in your SDK.

When you create a new Worker process, the Worker creates a long-lasting connection to the Temporal Service, polling a Task Queue for Tasks that are related to the code it is capable of executing.

![A Worker long polls for Tasks](https://docs.temporal.io/diagrams/how-sdk-works-1.svg)

Although the Worker is now running, unless a Workflow is explicitly started, the Task Queue doesn't have any Tasks on it and so, no code executes.
We can use a Temporal Client (available in Temporal SDKs and the Temporal CLI) to start a new Workflow.

![Start a Workflow using a Temporal Client](https://docs.temporal.io/diagrams/how-sdk-works-2.svg)

Starting a Workflow Execution creates a new Event, WorkflowExecutionStarted, and adds it to the Workflow Execution's Event History.

The Temporal Service then schedules a Workflow Task by adding it to the Task Queue.
When the Worker has capacity, it picks up this Task, and begin executing code.

Each step of the Task (for example, Scheduled, Started, and Completed), gets recorded into the Event History.

- Scheduled means that the Temporal Service has added a Task to the Task Queue.
- Started means that the Worker has dequeued the Task.
- Completed means that the Worker finished executing the Task by responding to the Temporal Service.

When the call to invoke the Activity is evaluated, the Worker suspends executing the code and sends a Command to the Temporal Service to schedule an Activity Task.

![Worker suspends code execution and sends a Command to the Temporal Service](https://docs.temporal.io/diagrams/how-sdk-works-3.svg)

When the Worker process can perform more work, it picks up the Activity Task and begins executing the Activity code, which includes the call to the external API.

If the Activity fails, say the API goes down, Temporal will automatically retry the Activity with one second between intervals, as the configurations have defined, an infinite number of times until the Activity succeeds or is canceled.

In the case where the calls succeeds, and the code completes, the Worker tells the Temporal Service the Activity Task completed.

![The Worker reports that the Activity Execution completed](https://docs.temporal.io/diagrams/how-sdk-works-activity.svg)

Included is any data that was returned from the Activity (results of the API call), which is then persisted in the Workflow Execution Event History, and is now accessible to the Workflow code.

The Temporal Service creates a new Workflow Task which the Worker picks up.

![The Worker picks up the new Task](https://docs.temporal.io/diagrams/how-sdk-works-1.svg)

This is when the SDK Worker Replays the Workflow code, using the Event History as guidance on what to expect. If the Replay encounters an Event that doesn't match up with what is expected from the code, a [non-determinism](/references/workflow-task-errors#non-deterministic-error) error gets thrown.

If there is alignment, the Worker continues evaluating code.

Assuming the Activity Execution is successful, the Workflow now has the result of the Activity and the Worker is able to finish evaluating and executing the Workflow code, responding to the Temporal Service when complete.

The result of the Workflow can now be retrieved using a Temporal Client.

![The Temporal Client can now access the result of the Workflow](https://docs.temporal.io/diagrams/how-sdk-works-4.svg)

And that’s how a Temporal Worker and Temporal Service work together.

---

# Context Propagation

Source: https://docs.temporal.io/encyclopedia/context-propagation

> Pass custom key-value data across Workflow, Activity, and Child Workflow boundaries using Temporal headers.

Context propagation lets you pass custom key-value data from a Client to Workflows, and from Workflows to Activities and Child Workflows, without threading values through every function signature.

Common use cases:

- Propagating distributed tracing IDs (for example, OpenTelemetry trace context)
- Passing tenant IDs for multi-tenant applications
- Forwarding auth tokens or request-scoped metadata

Each SDK provides a **context propagator** interface you implement to control which values are injected and extracted. You register propagators on the Client, and the SDK calls them automatically at every boundary.

## Implementing context propagation

Here are SDK-specific guides:

- [Go](/develop/go/best-practices/context-propagation)

---

# Detecting Activity failures

Source: https://docs.temporal.io/encyclopedia/detecting-activity-failures

> Understand Activity Execution timeouts in Temporal; Schedule-To-Start, Start-To-Close, Schedule-To-Close, and Activity Heartbeats, for effective Workflow management.

Temporal detects different kinds of Activity Execution failures — for both Workflow Activities and
[Standalone Activities](/standalone-activity) — through the following timeouts:

- [Schedule-To-Start Timeout](#schedule-to-start-timeout)
- [Start-To-Close Timeout](#start-to-close-timeout)
- [Schedule-To-Close Timeout](#schedule-to-close-timeout)
- [Activity Heartbeats](#activity-heartbeat)

## Schedule-To-Start Timeout 

**What is a Schedule-To-Start Timeout in Temporal?**

A Schedule-To-Start Timeout is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled (that is, placed in a Task Queue) to when a [Worker](/workers#worker) starts (that is, picks up from the Task Queue) that Activity Task.
In other words, it's a limit for how long an Activity Task can be enqueued.

**Related:**

- [Set a Schedule-To-Start Timeout using the .NET SDK](/develop/dotnet/activities/timeouts)
- [Set a Schedule-To-Start Timeout using the Go SDK](/develop/go/activities/timeouts)
- [Set a Schedule-To-Start Timeout using the Java SDK](/develop/java/activities/timeouts)
- [Set a Schedule-To-Start Timeout using the PHP SDK](/develop/php/activities/timeouts#activity-timeouts)
- [Set a Schedule-To-Start Timeout using the Python SDK](/develop/python/activities/timeouts)
- [Set a Schedule-To-Start Timeout using the Ruby SDK](/develop/ruby/activities/timeouts)
- [Set a Schedule-To-Start Timeout using the Rust SDK](/develop/rust/activities/timeouts)
- [Set a Schedule-To-Start Timeout using the TypeScript SDK](/develop/typescript/activities/timeouts)

The moment that the Task is picked by the Worker, from the Task Queue, is considered to be the start of the Activity Task Execution for the purposes of the Schedule-To-Start Timeout and associated metrics.
This definition of "Start" avoids issues that a clock difference between the Temporal Service and a Worker might create.

![Schedule-To-Start Timeout period](https://docs.temporal.io/diagrams/schedule-to-start-timeout.svg)

"Schedule" in Schedule-To-Start and Schedule-To-Close have different frequency guarantees.

The Schedule-To-Start Timeout is enforced for each Activity Task, whereas the Schedule-To-Close Timeout is enforced once per Activity Execution.
Thus, "Schedule" in Schedule-To-Start refers to the scheduling moment of _every_ Activity Task in the sequence of Activity Tasks that make up the Activity Execution, while
"Schedule" in Schedule-To-Close refers to the _first_ Activity Task in that sequence.

An Activity Execution can be composed of multiple Activity Task Executions, with each Task representing a single attempt or retry. Each time you start an Activity — as a step in a Workflow or as a Standalone Activity — that call represents a single logical Activity Execution.

If that Activity fails (for example, due to a timeout or error) and a Retry Policy is in place, Temporal will schedule another attempt to execute the same Activity. Each attempt is called an Activity Task Execution.

A [Retry Policy](/encyclopedia/retry-policies) attached to an Activity Execution retries an Activity Task.

![Start-To-Close Timeout period with retries](https://docs.temporal.io/diagrams/schedule-to-start-timeout-with-retry.svg)

This timeout has two primary use cases:

1. Detect whether an individual Worker has crashed.
2. Detect whether the fleet of Workers polling the Task Queue is not able to keep up with the rate of Activity Tasks.

**The default Schedule-To-Start Timeout is ∞ (infinity).**

If this timeout is used, we recommend setting this timeout to the maximum time a Workflow Execution is willing to wait for an Activity Execution in the presence of all possible Worker outages, and have a concrete plan in place to reroute Activity Tasks to a different Task Queue.
This timeout is non-retryable by design. It **does not** trigger any retries regardless of the Retry Policy, as a retry would place the Activity Task back into the same Task Queue.

In most cases, we recommend monitoring the `temporal_activity_schedule_to_start_latency` metric to know when Workers slow down picking up Activity Tasks, instead of setting this timeout.

## Start-To-Close Timeout 

**What is a Start-To-Close Timeout in Temporal?**

A Start-To-Close Timeout is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution).

**Related:**

- [Set a Start-To-Close Timeout using the .NET SDK](/develop/dotnet/activities/timeouts)
- [Set a Start-To-Close Timeout using the Go SDK](/develop/go/activities/timeouts)
- [Set a Start-To-Close Timeout using the Java SDK](/develop/java/activities/timeouts)
- [Set a Start-To-Close Timeout using the PHP SDK](/develop/php/activities/timeouts#activity-timeouts)
- [Set a Start-To-Close Timeout using the Python SDK](/develop/python/activities/timeouts)
- [Set a Start-To-Close Timeout using the Ruby SDK](/develop/ruby/activities/timeouts)
- [Set a Start-To-Close Timeout using the Rust SDK](/develop/rust/activities/timeouts)
- [Set a Start-To-Close Timeout using the TypeScript SDK](/develop/typescript/activities/timeouts)

**The Start-To-Close Timeout has no default value of its own.**
If you set only the [Schedule-To-Close Timeout](#schedule-to-close-timeout), the Temporal Service uses that value as the Start-To-Close Timeout.
If you set neither timeout, the Temporal Service rejects the Activity.
For a Workflow Activity, the Workflow Task fails with a [Bad Schedule Activity Attributes](/references/workflow-task-errors#bad-schedule-activity-attributes) error.

We recommend always setting this timeout; however, make sure that Start-To-Close Timeout is always set to be longer than the maximum possible time for the Activity Execution to complete.
For long running Activity Executions, we recommend also using [Activity Heartbeats](#activity-heartbeat) and [Heartbeat Timeouts](#heartbeat-timeout).

> **💡 Tip:**
>
> We strongly recommend setting a Start-To-Close Timeout.
>
> The Temporal Server doesn't detect failures when a Worker loses communication with the Server or crashes.
> Therefore, the Temporal Server relies on the Start-To-Close Timeout to force Activity retries.
>

The main use case for the Start-To-Close timeout is to detect when a Worker crashes after it has started executing an Activity Task.

![Start-To-Close Timeout period](https://docs.temporal.io/diagrams/start-to-close-timeout.svg)

A [Retry Policy](/encyclopedia/retry-policies) attached to an Activity Execution retries an Activity Task Execution.
Thus, the Start-To-Close Timeout is applied to each Activity Task Execution within an Activity Execution.

If the first Activity Task Execution returns an error the first time, then the full Activity Execution might look like this:

![Start-To-Close Timeout period with retries](https://docs.temporal.io/diagrams/start-to-close-timeout-with-retry.svg)

If this timeout is reached, the following actions occur:

- For a Workflow Activity, an [ActivityTaskTimedOut](/references/events#activitytasktimedout) Event is written to the
  Workflow Execution's mutable state. A [Standalone Activity](/standalone-activity) has no Workflow Execution, so this
  isn't recorded the same way — see [Activity Execution](/activity-execution#standalone-activity-executions) for what
  Standalone Activities provide instead.
- If a Retry Policy dictates a retry, the Temporal Service schedules another Activity Task.
  - For a Workflow Activity, the attempt count increments by 1 in the Workflow Execution's mutable state.
  - The Start-To-Close Timeout timer is reset.

## Schedule-To-Close Timeout 

**What is a Schedule-To-Close Timeout in Temporal?**

A Schedule-To-Close Timeout is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution), from when the first [Activity Task](/tasks#activity-task) is scheduled to when the last Activity Task, in the chain of Activity Tasks that make up the Activity Execution, reaches a Closed status.

**Related:**

- [Set a Schedule-To-Close Timeout using the .NET SDK](/develop/dotnet/activities/timeouts)
- [Set a Schedule-To-Close Timeout using the Go SDK](/develop/go/activities/timeouts)
- [Set a Schedule-To-Close Timeout using the Java SDK](/develop/java/activities/timeouts)
- [Set a Schedule-To-Close Timeout using the PHP SDK](/develop/php/activities/timeouts#activity-timeouts)
- [Set a Schedule-To-Close Timeout using the Python SDK](/develop/python/activities/timeouts)
- [Set a Schedule-To-Close Timeout using the Ruby SDK](/develop/ruby/activities/timeouts)
- [Set a Schedule-To-Close Timeout using the Rust SDK](/develop/rust/activities/timeouts)
- [Set a Schedule-To-Close Timeout using the TypeScript SDK](/develop/typescript/activities/timeouts)

![Schedule-To-Close Timeout period](https://docs.temporal.io/diagrams/schedule-to-close-timeout.svg)

Example Schedule-To-Close Timeout period for an Activity Execution that has a chain Activity Task Executions:

![Schedule-To-Close Timeout period with a retry](https://docs.temporal.io/diagrams/schedule-to-close-timeout-with-retry.svg)

**The default Schedule-To-Close Timeout is ∞ (infinity).**

An Activity Execution must have either this timeout (Schedule-To-Close) or [Start-To-Close](#start-to-close-timeout) set.
This timeout can be used to control the overall duration of an Activity Execution in the face of failures (repeated Activity Task Executions), without altering the Maximum Attempts field of the Retry Policy.

> **💡 Tip:**
>
> We strongly recommend setting a Start-To-Close Timeout.
>
> The Temporal Server doesn't detect failures when a Worker loses communication with the Server or crashes.
> Therefore, the Temporal Server relies on the Start-To-Close Timeout to force Activity retries.
>

## Activity Heartbeat 

**What is an Activity Heartbeat in Temporal?**

An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Service.
Each ping informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed.

**Related:**

- [Heartbeat an Activity using the .NET SDK](/develop/dotnet/activities/timeouts#activity-heartbeats)
- [Heartbeat an Activity using the Go SDK](/develop/go/activities/timeouts#activity-heartbeats)
- [Heartbeat an Activity using the Java SDK](/develop/java/activities/timeouts#activity-heartbeats)
- [Heartbeat an Activity using the PHP SDK](/develop/php/activities/timeouts#activity-heartbeats)
- [Heartbeat an Activity using the Python SDK](/develop/python/activities/timeouts#activity-heartbeats)
- [Heartbeat an Activity using the Ruby SDK](/develop/ruby/activities/timeouts#activity-heartbeats)
- [Heartbeat an Activity using the Rust SDK](/develop/rust/activities/timeouts#activity-heartbeats)
- [Heartbeat an Activity using the TypeScript SDK](/develop/typescript/activities/timeouts#activity-heartbeat-timeout)

Activity Heartbeats work in conjunction with a [Heartbeat Timeout](#heartbeat-timeout).

Activity Heartbeats are implemented within the Activity Definition.
Custom progress information can be included in the Heartbeat which can then be used by the Activity Execution should a retry occur.

An Activity Heartbeat can be recorded as often as needed (for example, once a minute or every loop iteration).
It is often a good practice to Heartbeat on anything but the shortest Activity Execution.
Temporal SDKs control the rate at which Heartbeats are sent to the Temporal Service.

Heartbeating is not required from [Local Activities](/local-activity), and does nothing.

For _long-running_ Activities, we recommend using a relatively short Heartbeat Timeout and a frequent Heartbeat.
That way if a Worker fails it can be handled in a timely manner.

A Heartbeat can include an application layer payload that can be used to _save_ Activity Execution progress.
If an [Activity Task Execution](/tasks#activity-task-execution) times out due to a missed Heartbeat, the next Activity Task can access and continue with that payload.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation.
Heartbeat throttling may lead to Cancellation getting delivered later than expected.

### Throttling

Heartbeats may not always be sent to the Temporal Service—they may be throttled by the Worker.
The throttle interval is the smaller of the following:

- If `heartbeatTimeout` is provided, `heartbeatTimeout * 0.8`; otherwise, `defaultHeartbeatThrottleInterval`
- `maxHeartbeatThrottleInterval`

`defaultHeartbeatThrottleInterval` is 30 seconds by default, and `maxHeartbeatThrottleInterval` is 60 seconds by default.
Each can be set in Worker options.

Throttling is implemented as follows:

- After sending a Heartbeat, the Worker sets a timer for the throttle interval.
- The Worker stops sending Heartbeats, but continues receiving Heartbeats from the Activity and remembers the most recent one.
- When the timer fires, the Worker:
  - Sends the most recent Heartbeat.
  - Sets the timer again.

Throttling allows the Worker to reduce network traffic and load on the Temporal Service by suppressing Heartbeats that aren’t necessary to prevent a Heartbeat Timeout.
Throttling does not apply to the final Heartbeat message in the case of Activity Failure.
If an Activity fails just after recording progress information in a Heartbeat message, that progress information will be available during the next retry attempt, provided that the Worker itself did not crash before delivering it to the Temporal Service.

### Which Activities should Heartbeat?

Heartbeating is best thought about not in terms of time, but in terms of "How do you know you are making progress?"
For short-term operations, progress updates are not a requirement.
However, checking the progress and status of Activity Executions that run over long periods is almost always useful.

Consider the following when setting Activity Heartbeats:

- Your underlying task must be able to report definite progress.
  Note that the caller cannot read this progress information while the Activity is still executing (or it would have to store it in Event History).
  You can report progress to external sources if you need it exposed to the user.

- Your Activity Execution is long-running, and you need to verify whether the Worker that is processing your Activity is still alive and has not run out of memory or silently crashed.

For example, the following scenarios are suitable for Heartbeating:

- Reading a large file from Amazon S3.
- Running a ML training job on some local GPUs.

And the following scenarios are not suitable for Heartbeating:

- Making a quick API call.
- Reading a small file from disk.

### Heartbeat Timeout 

**What is a Heartbeat Timeout in Temporal?**

A Heartbeat Timeout is the maximum time between [Activity Heartbeats](#activity-heartbeat).

**Related:**

- [Set a Heartbeat Timeout using the .NET SDK](/develop/dotnet/activities/timeouts#heartbeat-timeout)
- [Set a Heartbeat Timeout using the Go SDK](/develop/go/activities/timeouts#activity-heartbeats)
- [Set a Heartbeat Timeout using the Java SDK](/develop/java/activities/timeouts#heartbeat-timeout)
- [Set a Heartbeat Timeout using the PHP SDK](/develop/php/activities/timeouts#heartbeat-timeout)
- [Set a Heartbeat Timeout using the Python SDK](/develop/python/activities/timeouts#heartbeat-timeout)
- [Set a Heartbeat Timeout using the Ruby SDK](/develop/ruby/activities/timeouts#heartbeat-timeout)
- [Set a Heartbeat Timeout using the Rust SDK](/develop/rust/activities/timeouts#heartbeat-timeout)
- [Set a Heartbeat Timeout using the TypeScript SDK](/develop/typescript/activities/timeouts#activity-heartbeat-timeout)

![Heartbeat Timeout periods](https://docs.temporal.io/diagrams/heartbeat-timeout.svg)

If this timeout is reached, the Activity Task fails and a retry occurs if a [Retry Policy](/encyclopedia/retry-policies) dictates it.

#### What a Heartbeat Timeout of `0s` means 

`0s` means the Heartbeat Timeout is disabled. No Heartbeat timer is started, so the Activity never fails for missing a Heartbeat. It does not mean an unbounded timeout, and it does not mean the Activity times out immediately.

`0s` is also the normalized value the Temporal Service stores when you leave the Heartbeat Timeout unset. An unset Heartbeat Timeout and an explicit `0s` are therefore indistinguishable once the Activity is scheduled: both are reported as `0s` and both behave as disabled.

The Heartbeat Timeout is also capped at the [Start-To-Close Timeout](#start-to-close-timeout). Setting a Heartbeat Timeout longer than the Start-To-Close Timeout stores the Start-To-Close Timeout instead.

---

# Detecting Workflow failures

Source: https://docs.temporal.io/encyclopedia/detecting-workflow-failures

> Learn about Workflow Execution Timeout, Workflow Run Timeout, and Workflow Task Timeout in Temporal. Maximize Workflow efficiency and manage durations effectively.

Each Workflow Timeout controls the maximum duration of a different aspect of a Workflow Execution. Workflow Timeouts are set when starting the Workflow Execution.

Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient.
Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes.
If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer.

- [Workflow Execution Timeout](#workflow-execution-timeout)
- [Workflow Run Timeout](#workflow-run-timeout)
- [Workflow Task Timeout](#workflow-task-timeout)

## Workflow Execution Timeout 

**What is a Workflow Execution Timeout in Temporal?**

A Workflow Execution Timeout is the maximum time that a Workflow Execution can be executing (have an Open status) including retries and any usage of Continue As New.

**Related:**

- [Set a Workflow Execution Timeout using the Go SDK](/develop/go/workflows/timeouts)
- [Set a Workflow Execution Timeout using the Java SDK](/develop/java/workflows/timeouts)
- [Set a Workflow Execution Timeout using the PHP SDK](/develop/php/workflows/timeouts#workflow-timeouts)
- [Set a Workflow Execution Timeout using the Python SDK](/develop/python/workflows/timeouts)
- [Set a Workflow Execution Timeout using the TypeScript SDK](/develop/typescript/workflows/timeouts)
- [Set a Workflow Execution Timeout using the .NET SDK](/develop/dotnet/workflows/timeouts)
- [Set a Workflow Execution Timeout using the Ruby SDK](/develop/ruby/workflows/timeouts)
- [Set a Workflow Execution Timeout using the Rust SDK](/develop/rust/workflows/timeouts)

![Workflow Execution Timeout period](https://docs.temporal.io/diagrams/workflow-execution-timeout.svg)

**The default value is ∞ (infinite).**
If this timeout is reached, the Workflow Execution changes to a Timed Out status.
This timeout is different from the [Workflow Run Timeout](#workflow-run-timeout).
This timeout is most commonly used for stopping the execution of a [Temporal Cron Job](/cron-job) after a certain amount of time has passed.

## Workflow Run Timeout 

**What is a Workflow Run Timeout in Temporal?**

A Workflow Run is the instance of a specific Workflow Execution.

Due to the potential for Workflow Retries or Continue-as-New, a Workflow Execution may have multiple Workflow runs. For example, if a Workflow that specifies a Retry Policy initially fails and then succeeds during the next retry attempt, there is a single Workflow Execution that spans two Workflow Runs. Both runs will share the same Workflow ID but have a unique Run ID to distinguish them.

A Workflow Run Timeout restricts the maximum duration of a single Workflow Run. If the Workflow Run Timeout is reached, the Workflow Execution will be Timed Out. Because this Timeout only applies to an individual Workflow Run, this does not include retries or Continue-As-New.

**Related:**

- [Set a Workflow Timeout using the .NET SDK](/develop/dotnet/workflows/timeouts)
- [Set a Workflow Run Timeout using the Go SDK](/develop/go/workflows/timeouts)
- [Set a Workflow Run Timeout using the Java SDK](/develop/java/workflows/timeouts)
- [Set a Workflow Run Timeout using the PHP SDK](/develop/php/workflows/timeouts#workflow-timeouts)
- [Set a Workflow Run Timeout using the Python SDK](/develop/python/workflows/timeouts)
- [Set a Workflow Timeout using the Ruby SDK](/develop/ruby/workflows/timeouts)
- [Set a Workflow Timeout using the Rust SDK](/develop/rust/workflows/timeouts)
- [Set a Workflow Run Timeout using the TypeScript SDK](/develop/typescript/workflows/timeouts)

![Workflow Run Timeout period](https://docs.temporal.io/diagrams/workflow-run-timeout.svg)

**The default is set to the same value as the [Workflow Execution Timeout](#workflow-execution-timeout).**
This timeout is most commonly used to limit the execution time of a single [Temporal Cron Job Execution](/cron-job).

If the Workflow Run Timeout is reached, the Workflow Execution will be Timed Out.

## Workflow Task Timeout 

**What is a Workflow Task Timeout in Temporal?**

A Workflow Task Timeout is the maximum amount of time allowed for a [Worker](/workers#worker) to execute a [Workflow Task](/tasks#workflow-task) after the Worker has pulled that Workflow Task from the [Task Queue](/task-queue).

This Timeout is primarily available to recognize whether a Worker has gone down so that the Workflow Execution can be recovered on a different Worker.

**Related:**

- [Set a Workflow Task Timeout using the .NET SDK](/develop/dotnet/workflows/timeouts)
- [Set a Workflow Task Timeout using the Go SDK](/develop/go/workflows/timeouts)
- [Set a Workflow Task Timeout using the Java SDK](/develop/java/workflows/timeouts)
- [Set a Workflow Task Timeout using the PHP SDK](/develop/php/workflows/timeouts#workflow-timeouts)
- [Set a Workflow Task Timeout using the Python SDK](/develop/python/workflows/timeouts)
- [Set a Workflow Task Timeout using the Ruby SDK](/develop/ruby/workflows/timeouts)
- [Set a Workflow Task Timeout using the Rust SDK](/develop/rust/workflows/timeouts)
- [Set a Workflow Task Timeout using the TypeScript SDK](/develop/typescript/workflows/timeouts)

![Workflow Task Timeout period](https://docs.temporal.io/diagrams/workflow-task-timeout.svg)

**The default value is 10 seconds.**
This timeout is primarily available to recognize whether a Worker has gone down so that the Workflow Execution can be recovered on a different Worker.
The main reason for increasing the default value is to accommodate a Workflow Execution that has an extensive Workflow Execution History, requiring more than 10 seconds for the Worker to load.
It's worth mentioning that although you can extend the timeout up to the maximum value of 120 seconds, it's not recommended to move beyond the default value.

## Detecting Workflow Task Failures

Use the `TemporalReportedProblems` Search Attribute to detect Workflows with failed Workflow Tasks. 
A failed Workflow Task does not cause the Workflow to fail. Some Tasks within a Workflow may be intended to fail.
For example, a Workflow Task may check a remote data source for new messages. If there aren't any, the Task will fail as intended.
If your Task has code to handle the failure, the Workflow will proceed.
However, if your Workflow has a Task that fails and the failure is not handled, the Workflow will continue to run, but will not complete.
Detecting Workflows in this state is a common troubleshooting issue.

To identify Workflows with Task failures, you can use the Temporal Web UI. See [Task Failures View](/web-ui/#task-failures-view) for more details.

You can also detect Workflows with Task failures by searching for the `TemporalReportedProblems` search attribute with your observability tools.

> **⚠️ Warning:**
> Activating Workflow Task Failure
> To enable the Task Failures View for a Namespace, you need to update the Dynamic Config for that Namespace. See [Activating Task Failures View](/web-ui/#activate-task-failures-view).

---

# Event History

Source: https://docs.temporal.io/encyclopedia/event-history

> Discover how Temporal uses the Event History to recreate a Workflow's state in the case of failure, such as a Worker crash, and how it uses replay to restore the Workflow's state to the point of failure.

With Temporal, your Workflows recover from crashes automatically. This is made possible by the [Event History](/workflow-execution/event), a complete and durable log of everything that has happened in the lifecycle of a Workflow Execution, as well as the ability of the Temporal Service to durably persist the Events during Replay.

Temporal uses the Event History to record every step taken along the way. Each time your Workflow Definition makes an API call to execute an Activity or start a Timer for instance, it doesn’t perform the action directly. Instead, it sends a Command to the Temporal Service.

A Command is a requested action issued by a Worker to the Temporal Service after a Workflow Task Execution completes. The Temporal Service will act on these Commands such as scheduling an Activity or scheduling a timer. These Commands are then mapped to Events which are persisted in case of failure. For example, if the Worker crashes, the Worker uses the Event History to replay the code and recreate the state of the Workflow Execution to what it was immediately before the crash. It then resumes progress from the point of failure as if the failure never occurred.

For a deep dive on how the Event History works, refer to the walkthroughs in the dropdown.

- [Go](/encyclopedia/event-history/event-history-go)
- [Java](/encyclopedia/event-history/event-history-java)
- [Python](/encyclopedia/event-history/event-history-python)
- [TypeScript](/encyclopedia/event-history/event-history-typescript)
- [.NET](/encyclopedia/event-history/event-history-dotnet)

---

# Event History walkthrough with the .NET SDK

Source: https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet

In order to understand how Workflow Replay works, this page will go through the following walkthroughs:

1. [How Workflow code maps to Commands](#How-Workflow-Code-Maps-To-Commands)
2. [How Workflow Commands map to Events](#How-Workflow-Commands-Map-To-Events)
3. [How History Replay provides Durable Execution](#How-History-Replay-Provides-Durable-Execution)
4. [Example of a non-deterministic Workflow Definition](#Example-of-Non-Deterministic-Workflow)

## How Workflow code maps to Commands 

This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the
Temporal Service know what to do.

Step through the Workflow Definition below. Each step highlights the statements the Worker is running and shows the
Commands it has issued so far.

```csharp
[Workflow]
public class PizzaWorkflow
{
    [WorkflowRun]
    public async Task<OrderConfirmation> RunAsync(PizzaOrder order)
    {
        // Activity Options omitted for brevity
        var totalPrice = order.Items.Sum(pizza => pizza.Price);

        var distance = await Workflow.ExecuteActivityAsync(
            (Activities act) => act.GetDistanceAsync(order.Address),
            options);

        if (order.IsDelivery && distance.Kilometers > 25)
        {
            throw new ApplicationFailureException("Customer too far away for delivery");
        }

        await Workflow.DelayAsync(TimeSpan.FromMinutes(30));

        var bill = new Bill(
            CustomerId: order.Customer.CustomerId,
            OrderNumber: order.OrderNumber,
            Description: "Pizza",
            Amount: totalPrice);

        var confirmation = await Workflow.ExecuteActivityAsync(
            (Activities act) => act.SendBillAsync(bill),
            options);

        return confirmation;
    }
}
```
#### Step 1: A basic Workflow Definition

This Workflow Definition takes a pizza order and does the work listed here. Step through it to see which
statements the Worker handles on its own and which ones send a Command to the Temporal Service.

- Calculates the total price of the pizzas
- Determines the distance to the customer
- Fails if the customer is too far away for delivery
- Sleeps for 30 minutes
- Populates a record with billing information
- Sends a bill to the customer

#### Step 2: Internal steps

*Internal step*

These steps are internal to the Workflow. The Worker runs them in your process and never contacts the Temporal
Service.

#### Step 3: Steps that reach the Temporal Service

*Sends a Command*

These steps do involve the Temporal Service. Requesting an Activity Execution generates a Command to schedule
the Activity Task, and returning from the Workflow tells the Temporal Service that the Workflow Execution is
complete.

#### Step 4: Total the price of the order

*Internal step*

The walkthrough starts here. Calculating the total price of the pizzas is an internal step, so it doesn't
require any interaction with the Temporal Service.

#### Step 5: Request the GetDistance Activity

*Sends a Command*

The Worker reaches a statement that requires the Temporal Service: a request to execute an Activity. It issues
a ScheduleActivityTask Command with the details the Temporal Service needs, such as the Task Queue name, the
Activity Type, and the input values. An Activity can take hours or days to complete, and the Worker holds no
resources while it waits.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", GetDistance, { Line1: "123 Oak St.", Line2: "", ... })

#### Step 6: Evaluate the distance

*Internal step*

The Worker evaluates the distance returned by the Activity. If the customer lived too far away, the Workflow
would throw an exception, which sends a Command asking the Temporal Service to fail the Workflow Execution.
This order is going to a nearby customer, so execution continues.

#### Step 7: Start a Timer

*Sends a Command*

The call to delay execution is another statement that involves the Temporal Service. The Worker issues a
StartTimer Command that includes the duration, and this Workflow Execution pauses for 30 minutes until the
Timer fires.

**Commands issued:** `StartTimer` (30 minutes)

#### Step 8: Populate the bill

*Internal step*

The Timer fires and execution resumes. These lines create and populate the record that holds the input for the
next Activity. The record relates to an Activity, but building it doesn't involve the Temporal Service.

#### Step 9: Request the SendBill Activity

*Sends a Command*

The next statement requests execution of an Activity, so the Worker issues another ScheduleActivityTask
Command to the Temporal Service.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", SendBill, { Amount: 2750, Description: "Pizzas", ... })

#### Step 10: Return from the Workflow

*Sends a Command*

Returning from the Workflow method also results in a Command. The Worker issues CompleteWorkflowExecution to
the Temporal Service, which includes the value returned from the method.

**Commands issued:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

Four statements in this Workflow Definition produce a Command:

| Statement                                         | Command                     |
| ------------------------------------------------- | --------------------------- |
| `Workflow.ExecuteActivityAsync(GetDistanceAsync)` | `ScheduleActivityTask`      |
| `Workflow.DelayAsync(TimeSpan.FromMinutes(30))`   | `StartTimer`                |
| `Workflow.ExecuteActivityAsync(SendBillAsync)`    | `ScheduleActivityTask`      |
| `return confirmation;`                            | `CompleteWorkflowExecution` |

Everything else is an internal step. Totaling the order price, evaluating the distance, and populating the bill record
all run in the Worker without contacting the Temporal Service.

## How Workflow Commands map to Events 

The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The
Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution
of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so
they're durable, and will even survive a crash of the Temporal Service itself.

These Events are what are used to recreate a Workflow Execution's state in the case of failure.

Step through the same Workflow Definition to see each Command the Worker issues and the Events the Temporal Service
records in response.

```csharp
[Workflow]
public class PizzaWorkflow
{
    [WorkflowRun]
    public async Task<OrderConfirmation> RunAsync(PizzaOrder order)
    {
        // Activity Options omitted for brevity
        var totalPrice = order.Items.Sum(pizza => pizza.Price);

        var distance = await Workflow.ExecuteActivityAsync(
            (Activities act) => act.GetDistanceAsync(order.Address),
            options);

        if (order.IsDelivery && distance.Kilometers > 25)
        {
            throw new ApplicationFailureException("Customer too far away for delivery");
        }

        await Workflow.DelayAsync(TimeSpan.FromMinutes(30));

        var bill = new Bill(
            CustomerId: order.Customer.CustomerId,
            OrderNumber: order.OrderNumber,
            Description: "Pizza",
            Amount: totalPrice);

        var confirmation = await Workflow.ExecuteActivityAsync(
            (Activities act) => act.SendBillAsync(bill),
            options);

        return confirmation;
    }
}
```
#### Step 1: Commands and the Events they produce

This walkthrough keeps a running list of the Commands the Worker issues and the Events the Temporal Service records in response. Blue Events are the direct result of a Command. Pink Events are an indirect result.

#### Step 2: The GetDistance Activity is scheduled

*Sends a Command*

The call to the Activity is the first line of code in the Workflow that causes a Command to be issued. In response, the Temporal Service creates an Activity Task, adds it to the Task Queue, and appends the ActivityTaskScheduled Event to the Event History.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Events:** `ActivityTaskScheduled` (GetDistance)

#### Step 3: A Worker polls for the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to an available Worker. It responds to the poll request with the Task, and the Worker begins executing the code needed to complete it. Nothing is recorded in the Event History yet.

#### Step 4: The Activity Task closes

*Temporal Service*

The ActivityTaskStarted Event is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event. It is an indirect result of the Command. The Start-to-Close Timeout sets how long the Activity has to complete.

**Events:** `ActivityTaskStarted`

#### Step 5: The Activity reports its result

*Temporal Service*

When the Activity Definition returns a result, the Worker sends a message to the Temporal Service to say the Task is complete. This is a notification, not a Command, because it does not ask the Temporal Service to do anything that lets the Workflow Execution progress. The Temporal Service records ActivityTaskCompleted.

**Events:** `ActivityTaskCompleted` (distance = 15)

#### Step 6: The Worker starts a Timer

*Sends a Command*

The next statement that results in a Command is the call to start a Timer. The Temporal Service starts a 30-minute Timer and records a TimerStarted Event, a direct result of the StartTimer Command.

**Commands:** `StartTimer` (30 minutes)

**Events:** `TimerStarted` (30 minutes)

#### Step 7: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires on the Temporal Service, which records a TimerFired Event. The Workflow Execution continues with the next statement, but populating the bill is an internal step that does not interact with the Temporal Service.

**Events:** `TimerFired`

#### Step 8: The SendBill Activity is scheduled

*Sends a Command*

The Worker reaches the call to the SendBill Activity and issues another ScheduleActivityTask Command. The Temporal Service adds an Activity Task to the Task Queue and records an ActivityTaskScheduled Event.

**Commands:** `ScheduleActivityTask` (SendBill)

**Events:** `ActivityTaskScheduled` (SendBill)

#### Step 9: A Worker dequeues the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to the Worker. The Worker removes the Task from the Task Queue and begins working on it, and the Temporal Service records an ActivityTaskStarted Event to signify that the Task was dequeued.

**Events:** `ActivityTaskStarted`

#### Step 10: The bill is sent

*Temporal Service*

When the Activity returns, the Task is complete and the Worker notifies the Temporal Service, which records the ActivityTaskCompleted Event. Execution continues until the Workflow completes. The next walkthrough covers the full Event History, including the Workflow Task Events.

**Events:** `ActivityTaskCompleted`

Blue Events are the direct result of a Command. Pink Events are an indirect result, such as the Events the Temporal
Service records when a Worker starts or finishes a Task:

| Command                | Direct Event            | Indirect Events                                |
| ---------------------- | ----------------------- | ---------------------------------------------- |
| `ScheduleActivityTask` | `ActivityTaskScheduled` | `ActivityTaskStarted`, `ActivityTaskCompleted` |
| `StartTimer`           | `TimerStarted`          | `TimerFired`                                   |

## How History Replay provides Durable Execution 

Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look
at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of
a failure.

This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and
Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of
the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed.

```csharp
[Workflow]
public class PizzaWorkflow
{
    [WorkflowRun]
    public async Task<OrderConfirmation> RunAsync(PizzaOrder order)
    {
        // Activity Options omitted for brevity
        var totalPrice = order.Items.Sum(pizza => pizza.Price);

        var distance = await Workflow.ExecuteActivityAsync(
            (Activities act) => act.GetDistanceAsync(order.Address),
            options);

        if (order.IsDelivery && distance.Kilometers > 25)
        {
            throw new ApplicationFailureException("Customer too far away for delivery");
        }

        await Workflow.DelayAsync(TimeSpan.FromMinutes(30));

        var bill = new Bill(
            CustomerId: order.Customer.CustomerId,
            OrderNumber: order.OrderNumber,
            Description: "Pizza",
            Amount: totalPrice);

        var confirmation = await Workflow.ExecuteActivityAsync(
            (Activities act) => act.SendBillAsync(bill),
            options);

        return confirmation;
    }
}
```
### Original execution

#### Step 1: A Client requests the Workflow Execution

*Temporal Service*

The walkthrough begins with a request to execute this Workflow Definition with input data about the customer and the pizzas ordered. The Temporal Service records WorkflowExecutionStarted, always the first Event of a Workflow Execution, and that Event holds the input data.

**Event History:** `WorkflowExecutionStarted` (customer, pizzas ordered)

#### Step 2: A Workflow Task is queued

*Temporal Service*

The Temporal Service adds a Workflow Task to the Task Queue and records a WorkflowTaskScheduled Event.

**Event History:** `WorkflowTaskScheduled`

#### Step 3: A Worker accepts the Workflow Task

*Temporal Service*

The Temporal Service dispatches the Task to a Worker that is polling the Task Queue. The Worker accepts it, and the Temporal Service records a WorkflowTaskStarted Event.

**Event History:** `WorkflowTaskStarted`

#### Step 4: The Worker runs the Workflow code

*Internal step*

The Worker invokes the Workflow code and runs it one statement at a time. The first few lines are internal steps that do not interact with the Temporal Service.

#### Step 5: The Worker completes the Workflow Task

*Sends a Command*

The Worker encounters a request to execute an Activity, so it completes the current Workflow Task. It makes a single gRPC call, RespondWorkflowTaskCompleted, that signals completion of the Workflow Task and carries any Commands with it, so WorkflowTaskCompleted and the Command that follows are technically one call.

**Event History:** `WorkflowTaskCompleted`

#### Step 6: The GetDistance Activity Task is scheduled

*Sends a Command*

In response to the ScheduleActivityTask Command, the Temporal Service queues an Activity Task and records an ActivityTaskScheduled Event, a direct result of the Command.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Event History:** `ActivityTaskScheduled` (GetDistance)

#### Step 7: A Worker starts the Activity Task

*Temporal Service*

The Temporal Service dispatches the Activity Task to an available Worker, which starts running the code in the GetDistance Activity. ActivityTaskStarted is an indirect result of the Command, and it is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event.

**Event History:** `ActivityTaskStarted`

#### Step 8: The Activity returns a distance of 15

*Temporal Service*

When the Activity returns, the Worker notifies the Temporal Service that the Activity Execution is complete. The Temporal Service records an ActivityTaskCompleted Event, which contains the result of the Activity.

**Event History:** `ActivityTaskCompleted` (distance = 15)

#### Step 9: The result goes back to the Workflow

*Temporal Service*

To deliver the result to the Workflow, the Temporal Service creates another Workflow Task that includes it. A Worker dequeues that Task, resumes execution, and evaluates the distance. This order is going to a nearby customer, so execution continues.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 10: The Worker starts a Timer

*Sends a Command*

The Worker reaches the request to start a Timer, so it completes the current Workflow Task and sends the StartTimer Command with it. The Temporal Service starts the Timer and records TimerStarted. The Workflow does not progress until the Timer fires.

**Commands:** `StartTimer` (30 minutes)

**Event History:**
- `WorkflowTaskCompleted`
- `TimerStarted` (30 minutes)

#### Step 11: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires and the Temporal Service records TimerFired. It then queues a new Workflow Task to deliver that Event to the Workflow and drive progress forward.

**Event History:**
- `TimerFired`
- `WorkflowTaskScheduled`

### Worker crash

#### Step 12: The Worker crashes

*Worker crash*

The Worker polls for the Task, dequeues it, and continues executing the Workflow code — and then crashes right here. How does Temporal recover the state of this Workflow Execution? First, how does the Temporal Service know the Worker crashed?

**Event History:** `WorkflowTaskStarted`

#### Step 13: The Workflow Task times out

*Worker crash*

Once a Worker accepts a Task, it is expected to complete it within a predefined duration. That Workflow Task Timeout, 10 seconds by default, is how a crashed Worker is recognized. The Worker failed to complete the Task in time, so the Temporal Service records WorkflowTaskTimedOut.

**Event History:** `WorkflowTaskTimedOut`

#### Step 14: A new Workflow Task is scheduled

*Temporal Service*

The Temporal Service schedules a new Workflow Task. The Worker that polls for it might be another Worker in the fleet or a new Worker process created by restarting the one that crashed. Everything above the line is the Event History as it stood when the Worker crashed.

**Event History:**
- `WorkflowTaskScheduled`
- *— Event History at the time of the crash —*

### History Replay

#### Step 15: The Worker requests the Event History

*Replay*

Either way, the Worker needs the current Event History for this Workflow Execution, so it requests it from the Temporal Service. The Temporal Service provides the History.

#### Step 16: Replay re-executes the code

*Replay*

The Worker begins re-executing the code with the same input, which was stored in the WorkflowExecutionStarted Event. Because the Workflow Definition is deterministic, the state of every variable encountered so far is identical to what it was before the crash. For example, totalPrice holds the same value.

#### Step 17: The Activity result comes from the History

*Replay*

When Replay reaches the call to schedule GetDistance, it creates a ScheduleActivityTask Command but does not issue it. Instead, the Worker inspects the Event History and finds ActivityTaskScheduled for this Activity Type, ActivityTaskStarted showing a Worker dequeued the Task, and ActivityTaskCompleted with a result of 15. The Activity already ran, so the Command is not issued.

**Commands:** ✓ `ScheduleActivityTask` (GetDistance) — created during Replay, not issued

#### Step 18: Replay assigns the stored result

*Replay*

The Worker assigns the value stored in the ActivityTaskCompleted Event, 15, to the distance variable. The Activity is not re-executed, so there is no way for it to behave differently during Replay than it did during the original execution. The conditional evaluates to false, just as it did before.

#### Step 19: The Timer is already done

*Replay*

The Worker reaches the request to start a Timer and creates a StartTimer Command, which again it does not issue. The Event History contains TimerStarted and TimerFired, so the Worker knows the Timer already started and fired during the previous execution.

**Commands:** ✓ `StartTimer` (30 minutes) — created during Replay, not issued

#### Step 20: State is restored to the point of the crash

*Replay*

The Worker has reached the point where the crash occurred, and replaying the code has completely restored the state of the Workflow Execution. Every variable, including totalPrice, holds the value it held before the crash.

### Execution resumes

#### Step 21: Execution moves past the crash

*Sends a Command*

The Worker reaches a statement beyond where the crash occurred, which is evident because the Event History contains no Events for the SendBill Activity. Execution continues as if the crash never happened: the Worker completes the current Workflow Task and includes the Command with it.

**Event History:** `WorkflowTaskCompleted`

#### Step 22: The SendBill Activity runs

*Sends a Command*

The Worker issues the Command, the Temporal Service queues the Activity Task, and a Worker dequeues and runs it. When the Activity returns, the Worker notifies the Temporal Service, which records ActivityTaskCompleted with the result from SendBill.

**Commands:** `ScheduleActivityTask` (SendBill)

**Event History:**
- `ActivityTaskScheduled` (SendBill)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 23: One more Workflow Task delivers the result

*Temporal Service*

The Temporal Service has not received a Command saying the Workflow Execution completed or failed, so it schedules another Workflow Task to continue progress. A Worker accepts it and resumes the Workflow code.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 24: The Workflow Execution completes

*Sends a Command*

The Workflow returns, so the Worker completes the current Workflow Task and issues a CompleteWorkflowExecution Command that contains the result. The Temporal Service records WorkflowExecutionCompleted as the final Event. The result is identical to an execution that never crashed.

**Commands:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

**Event History:**
- `WorkflowTaskCompleted`
- `WorkflowExecutionCompleted`

The walkthrough covers four phases:

1. **Original execution**: the Client starts the Workflow Execution, and the Commands the Worker issues become Events in
   the Event History.
2. **Worker crash**: the Worker dies partway through a Workflow Task. When the Workflow Task Timeout elapses, 10 seconds
   by default, the Temporal Service records `WorkflowTaskTimedOut` and schedules a new Workflow Task.
3. **History Replay**: a Worker requests the Event History and re-executes the Workflow code with the original input,
   which the `WorkflowExecutionStarted` Event stores. Commands the Worker creates during Replay are matched against the
   Event History instead of being issued, so Activities don't run again. The Worker uses the results stored in the
   `ActivityTaskCompleted` Events.
4. **Execution resumes**: past the point of the crash, the Event History holds no matching Events, so the Worker issues
   Commands for real again until the Workflow Execution completes. The result is identical to an execution that never
   crashed.

## Example of a non-deterministic Workflow Definition 

Now that Replay has been covered, this section will explain why a Workflow Definition needs to be
[deterministic](/workflow-definition#deterministic-constraints) in order for Replay to work.

A Workflow Definition is deterministic if every execution of it produces the same Commands in the same
sequence given the same input.

As mentioned in the [`How History Replay Provides Durable Execution`](#How-History-Replay-Provides-Durable-Execution)
walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs
the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event
History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History
and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue.

To better understand why a Workflow Definition needs to be deterministic, it's helpful to look at one that violates
it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random
number generator.

```csharp
[Workflow]
public class GenerateDailyReport
{
    private static readonly Random random = new Random();

    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        // Activity Options and logger declaration omitted for brevity
        var salesData = await Workflow.ExecuteActivityAsync(
            (Activities act) => act.ImportSalesDataAsync(),
            options);

        if (random.Next(100) >= 50)
        {
            await Workflow.DelayAsync(TimeSpan.FromHours(4));
        }

        Logger.LogInformation("Preparing to run daily report");

        return await Workflow.ExecuteActivityAsync(
            (Activities act) => act.RunDailyReportAsync(),
            options);
    }
}
```
### First execution

#### Step 1: The ImportSalesData Activity runs

*Sends a Command*

As this Workflow executes step by step, the first line that results in a Command is the call to the ImportSalesData Activity. The Activity Execution succeeds, so the Temporal Service logs three Events to the Event History.

**Commands created:** `ScheduleActivityTask` (ImportSalesData)

**Relevant History Events:**
- `ActivityTaskScheduled` (ImportSalesData)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 2: A random number decides the next branch

*Internal step*

The Worker reaches a conditional statement that evaluates a randomly generated number. The random number generator returns 84 during this execution, so the expression evaluates to true and execution continues with the next line.

#### Step 3: The Workflow starts a 4-hour Timer

*Sends a Command*

The next line requests a Timer, so the Worker issues a StartTimer Command. The Temporal Service starts the Timer and records TimerStarted, then records TimerFired when the Timer fires.

**Commands created:** `StartTimer` (4 hours)

**Relevant History Events:**
- `TimerStarted` (4 hours)
- `TimerFired`

#### Step 4: The Worker crashes

*Worker crash*

The Worker crashes once it reaches the next line, so another Worker takes over. That Worker uses Replay to restore the current state before continuing execution of the lines that follow.

### History Replay

#### Step 5: Replay establishes the expected Commands

*Replay*

The Worker requests the Event History and determines the sequence of Commands needed to restore the current state. Based on the History, it expects to encounter ScheduleActivityTask (ImportSalesData) and then StartTimer (4 hours).

#### Step 6: The first Command matches

*Replay*

As the Worker executes the code during Replay, it reaches the first call to execute an Activity and creates a ScheduleActivityTask Command. It is the right type of Command and it occurs at the right position in the expected sequence, so Replay proceeds.

**Commands created:** ✓ `ScheduleActivityTask` (ImportSalesData) — created during Replay

#### Step 7: The random number returns something different

*Replay*

The Worker reaches the conditional statement again. This time the random number generator returns 14, so the expression evaluates to false and execution skips the call that starts the Timer.

#### Step 8: The next Command does not match

*Worker crash*

The Worker reaches the request to execute the RunDailyReport Activity and creates another ScheduleActivityTask Command. That is not the Command it expected at this position in the sequence, so the Worker cannot restore the previous state.

**Commands created:** ✗ `ScheduleActivityTask` (RunDailyReport) — created during Replay (expected: StartTimer (4 hours))

#### Step 9: Replay fails with a non-determinism error

*Worker crash*

The Workflow produced a different sequence of Commands during Replay than the Event History recorded before the crash, so the Workflow Execution cannot be replayed. The random number generator is the source of the non-determinism.

During the first execution, the random number is 84, so the Workflow starts a Timer and the Event History records
`TimerStarted` and `TimerFired`. During Replay, the random number is 14, so the Workflow skips the Timer and produces a
`ScheduleActivityTask` Command where the Event History expects `StartTimer`. That mismatch is what makes Replay fail.

Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is
considered a [Workflow Task Failure](/references/failures#workflow-task-failures) which is considered a transient
failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the
Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a
strategy called versioning to address this non-determinism error. See [versioning](/develop/dotnet/workflows/versioning)
to learn more.

For more information on how Temporal handles Durable Execution or to see these walkthroughs in video format with more
explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and
[Versioning Workflows](https://learn.temporal.io/courses/versioning/).

## Temporal Applications support non-deterministic operations

We want to emphasize that although your Workflow Definition needs to be deterministic, your application itself does not!

Remember that pretty much anything that interacts with the external world is inherently non-deterministic:

- Calling LLM APIs
- Querying databases
- Reading or writing files
- Making HTTP requests to external services

**Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow Definition must be
deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed
above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities
which have built-in support for retries.

---

# Event History walkthrough with the Go SDK

Source: https://docs.temporal.io/encyclopedia/event-history/event-history-go

In order to understand how Workflow Replay works, this page will go through the following walkthroughs:

1. [How Workflow code maps to Commands](#How-Workflow-Code-Maps-To-Commands)
2. [How Workflow Commands map to Events](#How-Workflow-Commands-Map-To-Events)
3. [How History Replay provides Durable Execution](#How-History-Replay-Provides-Durable-Execution)
4. [Example of a non-deterministic Workflow Definition](#Example-of-Non-Deterministic-Workflow)

## How Workflow code maps to Commands 

This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the
Temporal Service know what to do.

Step through the Workflow Definition below. Each step highlights the statements the Worker is running and shows the
Commands it has issued so far.

```go
func PizzaWorkflow(ctx workflow.Context, order Order) (OrderConfirmation, error) {

    options := workflow.ActivityOptions{
        StartToCloseTimeout: 5 * time.Second,
    }
    ctx = workflow.WithActivityOptions(ctx, options)

    // Total the cost of the order
    var totalPrice int
    for _, pizza := range order.Items {
        totalPrice += pizza.Price
    }

    var distance Distance
    err := workflow.ExecuteActivity(ctx, GetDistance, order.Address).Get(ctx, &distance)
    if err != nil {
        return OrderConfirmation{}, err
    }

    if order.IsDelivery && distance.Kilometers > 25 {
        return OrderConfirmation{}, errors.New("customer too far away for delivery")
    }

    // Wait 30 minutes before billing the customer
    workflow.Sleep(ctx, 30 * time.Minute)

    bill := Bill{
        CustomerId:  order.Customer.CustomerId,
        Amount:      totalPrice,
        Description: order.OrderNumber,
    }

    var confirmation OrderConfirmation
    err = workflow.ExecuteActivity(ctx, SendBill, bill).Get(ctx, &confirmation)
    if err != nil {
        return OrderConfirmation{}, err
    }

    return confirmation, nil
}
```
#### Step 1: A basic Workflow Definition

This Workflow Definition takes a pizza order and does the work listed here. Step through it to see which
statements the Worker handles on its own and which ones send a Command to the Temporal Service.

- Defines a Start-to-Close Timeout
- Calculates the total price of the pizzas
- Determines the distance to the customer
- Fails if the customer is too far away for delivery
- Sleeps for 30 minutes
- Populates a struct with billing information
- Sends a bill to the customer

#### Step 2: Internal steps

*Internal step*

These steps are internal to the Workflow. The Worker runs them in your process and never contacts the Temporal
Service.

#### Step 3: Steps that reach the Temporal Service

*Sends a Command*

These steps do involve the Temporal Service. Requesting an Activity Execution generates a Command to schedule
the Activity Task, and returning from the Workflow tells the Temporal Service that the Workflow Execution is
complete.

#### Step 4: Total the price of the order

*Internal step*

The walkthrough starts here. Setting the Start-to-Close Timeout and adding up the price of each pizza are
internal steps, so they don't require any interaction with the Temporal Service.

#### Step 5: Request the GetDistance Activity

*Sends a Command*

The Worker reaches a statement that requires the Temporal Service: a request to execute an Activity. It issues
a ScheduleActivityTask Command with the details the Temporal Service needs, such as the Task Queue name, the
Activity Type, and the input values. An Activity can take hours or days to complete, and the Worker holds no
resources while it waits.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", GetDistance, { Line1: "123 Oak St.", Line2: "", ... })

#### Step 6: Evaluate the distance

*Internal step*

The Worker evaluates the distance returned by the Activity. If the customer lived too far away, the Workflow
would return an error, which sends a Command asking the Temporal Service to fail the Workflow Execution. This
order is going to a nearby customer, so execution continues.

#### Step 7: Start a Timer

*Sends a Command*

The call to sleep is another statement that involves the Temporal Service. The Worker issues a StartTimer
Command that includes the duration, and this Workflow Execution pauses for 30 minutes until the Timer fires.

**Commands issued:** `StartTimer` (30 minutes)

#### Step 8: Populate the bill

*Internal step*

The Timer fires and execution resumes. These lines create and populate the struct that holds the input for the
next Activity. The struct relates to an Activity, but building it doesn't involve the Temporal Service.

#### Step 9: Request the SendBill Activity

*Sends a Command*

The next statement requests execution of an Activity, so the Worker issues another ScheduleActivityTask
Command to the Temporal Service.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", SendBill, { Amount: 2750, Description: "Pizzas", ... })

#### Step 10: Return from the Workflow

*Sends a Command*

Returning from the Workflow function also results in a Command. The Worker issues CompleteWorkflowExecution to
the Temporal Service, which includes the value returned from the function.

**Commands issued:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

Four statements in this Workflow Definition produce a Command:

| Statement                                                   | Command                     |
| ----------------------------------------------------------- | --------------------------- |
| `workflow.ExecuteActivity(ctx, GetDistance, order.Address)` | `ScheduleActivityTask`      |
| `workflow.Sleep(ctx, 30 * time.Minute)`                     | `StartTimer`                |
| `workflow.ExecuteActivity(ctx, SendBill, bill)`             | `ScheduleActivityTask`      |
| `return confirmation, nil`                                  | `CompleteWorkflowExecution` |

Everything else is an internal step. Setting the Start-to-Close Timeout, totaling the order price, evaluating the
distance, and populating the bill struct all run in the Worker without contacting the Temporal Service.

## How Workflow Commands map to Events 

The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The
Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution
of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so
they're durable, and will even survive a crash of the Temporal Service itself.

These Events are what are used to recreate a Workflow Execution's state in the case of failure.

Step through the same Workflow Definition to see each Command the Worker issues and the Events the Temporal Service
records in response.

```go
func PizzaWorkflow(ctx workflow.Context, order Order) (OrderConfirmation, error) {

    options := workflow.ActivityOptions{
        StartToCloseTimeout: 5 * time.Second,
    }
    ctx = workflow.WithActivityOptions(ctx, options)

    // Total the cost of the order
    var totalPrice int
    for _, pizza := range order.Items {
        totalPrice += pizza.Price
    }

    var distance Distance
    err := workflow.ExecuteActivity(ctx, GetDistance, order.Address).Get(ctx, &distance)
    if err != nil {
        return OrderConfirmation{}, err
    }

    if order.IsDelivery && distance.Kilometers > 25 {
        return OrderConfirmation{}, errors.New("customer too far away for delivery")
    }

    // Wait 30 minutes before billing the customer
    workflow.Sleep(ctx, 30 * time.Minute)

    bill := Bill{
        CustomerId:  order.Customer.CustomerId,
        Amount:      totalPrice,
        Description: order.OrderNumber,
    }

    var confirmation OrderConfirmation
    err = workflow.ExecuteActivity(ctx, SendBill, bill).Get(ctx, &confirmation)
    if err != nil {
        return OrderConfirmation{}, err
    }

    return confirmation, nil
}
```
#### Step 1: Commands and the Events they produce

This walkthrough keeps a running list of the Commands the Worker issues and the Events the Temporal Service records in response. Blue Events are the direct result of a Command. Pink Events are an indirect result.

#### Step 2: The GetDistance Activity is scheduled

*Sends a Command*

The call to the Activity is the first line of code in the Workflow that causes a Command to be issued. In response, the Temporal Service creates an Activity Task, adds it to the Task Queue, and appends the ActivityTaskScheduled Event to the Event History.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Events:** `ActivityTaskScheduled` (GetDistance)

#### Step 3: A Worker polls for the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to an available Worker. It responds to the poll request with the Task, and the Worker begins executing the code needed to complete it. Nothing is recorded in the Event History yet.

#### Step 4: The Activity Task closes

*Temporal Service*

The ActivityTaskStarted Event is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event. It is an indirect result of the Command. The Start-to-Close Timeout sets how long the Activity has to complete.

**Events:** `ActivityTaskStarted`

#### Step 5: The Activity reports its result

*Temporal Service*

When the Activity Definition returns a result, the Worker sends a message to the Temporal Service to say the Task is complete. This is a notification, not a Command, because it does not ask the Temporal Service to do anything that lets the Workflow Execution progress. The Temporal Service records ActivityTaskCompleted.

**Events:** `ActivityTaskCompleted` (distance = 15)

#### Step 6: The Worker starts a Timer

*Sends a Command*

The next statement that results in a Command is the call to start a Timer. The Temporal Service starts a 30-minute Timer and records a TimerStarted Event, a direct result of the StartTimer Command.

**Commands:** `StartTimer` (30 minutes)

**Events:** `TimerStarted` (30 minutes)

#### Step 7: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires on the Temporal Service, which records a TimerFired Event. The Workflow Execution continues with the next statement, but populating the bill is an internal step that does not interact with the Temporal Service.

**Events:** `TimerFired`

#### Step 8: The SendBill Activity is scheduled

*Sends a Command*

The Worker reaches the call to the SendBill Activity and issues another ScheduleActivityTask Command. The Temporal Service adds an Activity Task to the Task Queue and records an ActivityTaskScheduled Event.

**Commands:** `ScheduleActivityTask` (SendBill)

**Events:** `ActivityTaskScheduled` (SendBill)

#### Step 9: A Worker dequeues the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to the Worker. The Worker removes the Task from the Task Queue and begins working on it, and the Temporal Service records an ActivityTaskStarted Event to signify that the Task was dequeued.

**Events:** `ActivityTaskStarted`

#### Step 10: The bill is sent

*Temporal Service*

When the Activity returns, the Task is complete and the Worker notifies the Temporal Service, which records the ActivityTaskCompleted Event. Execution continues until the Workflow completes. The next walkthrough covers the full Event History, including the Workflow Task Events.

**Events:** `ActivityTaskCompleted`

Blue Events are the direct result of a Command. Pink Events are an indirect result, such as the Events the Temporal
Service records when a Worker starts or finishes a Task:

| Command                | Direct Event            | Indirect Events                                |
| ---------------------- | ----------------------- | ---------------------------------------------- |
| `ScheduleActivityTask` | `ActivityTaskScheduled` | `ActivityTaskStarted`, `ActivityTaskCompleted` |
| `StartTimer`           | `TimerStarted`          | `TimerFired`                                   |

## How History Replay provides Durable Execution 

Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look
at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of
a failure.

This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and
Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of
the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed.

```go
func PizzaWorkflow(ctx workflow.Context, order Order) (OrderConfirmation, error) {

    options := workflow.ActivityOptions{
        StartToCloseTimeout: 5 * time.Second,
    }
    ctx = workflow.WithActivityOptions(ctx, options)

    // Total the cost of the order
    var totalPrice int
    for _, pizza := range order.Items {
        totalPrice += pizza.Price
    }

    var distance Distance
    err := workflow.ExecuteActivity(ctx, GetDistance, order.Address).Get(ctx, &distance)
    if err != nil {
        return OrderConfirmation{}, err
    }

    if order.IsDelivery && distance.Kilometers > 25 {
        return OrderConfirmation{}, errors.New("customer too far away for delivery")
    }

    // Wait 30 minutes before billing the customer
    workflow.Sleep(ctx, 30 * time.Minute)

    bill := Bill{
        CustomerId:  order.Customer.CustomerId,
        Amount:      totalPrice,
        Description: order.OrderNumber,
    }

    var confirmation OrderConfirmation
    err = workflow.ExecuteActivity(ctx, SendBill, bill).Get(ctx, &confirmation)
    if err != nil {
        return OrderConfirmation{}, err
    }

    return confirmation, nil
}
```
### Original execution

#### Step 1: A Client requests the Workflow Execution

*Temporal Service*

The walkthrough begins with a request to execute this Workflow Definition with input data about the customer and the pizzas ordered. The Temporal Service records WorkflowExecutionStarted, always the first Event of a Workflow Execution, and that Event holds the input data.

**Event History:** `WorkflowExecutionStarted` (customer, pizzas ordered)

#### Step 2: A Workflow Task is queued

*Temporal Service*

The Temporal Service adds a Workflow Task to the Task Queue and records a WorkflowTaskScheduled Event.

**Event History:** `WorkflowTaskScheduled`

#### Step 3: A Worker accepts the Workflow Task

*Temporal Service*

The Temporal Service dispatches the Task to a Worker that is polling the Task Queue. The Worker accepts it, and the Temporal Service records a WorkflowTaskStarted Event.

**Event History:** `WorkflowTaskStarted`

#### Step 4: The Worker runs the Workflow code

*Internal step*

The Worker invokes the Workflow code and runs it one statement at a time. The first few lines are internal steps that do not interact with the Temporal Service.

#### Step 5: The Worker completes the Workflow Task

*Sends a Command*

The Worker encounters a request to execute an Activity, so it completes the current Workflow Task. It makes a single gRPC call, RespondWorkflowTaskCompleted, that signals completion of the Workflow Task and carries any Commands with it, so WorkflowTaskCompleted and the Command that follows are technically one call.

**Event History:** `WorkflowTaskCompleted`

#### Step 6: The GetDistance Activity Task is scheduled

*Sends a Command*

In response to the ScheduleActivityTask Command, the Temporal Service queues an Activity Task and records an ActivityTaskScheduled Event, a direct result of the Command.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Event History:** `ActivityTaskScheduled` (GetDistance)

#### Step 7: A Worker starts the Activity Task

*Temporal Service*

The Temporal Service dispatches the Activity Task to an available Worker, which starts running the code in the GetDistance Activity. ActivityTaskStarted is an indirect result of the Command, and it is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event.

**Event History:** `ActivityTaskStarted`

#### Step 8: The Activity returns a distance of 15

*Temporal Service*

When the Activity returns, the Worker notifies the Temporal Service that the Activity Execution is complete. The Temporal Service records an ActivityTaskCompleted Event, which contains the result of the Activity.

**Event History:** `ActivityTaskCompleted` (distance = 15)

#### Step 9: The result goes back to the Workflow

*Temporal Service*

To deliver the result to the Workflow, the Temporal Service creates another Workflow Task that includes it. A Worker dequeues that Task, resumes execution, and evaluates the distance. This order is going to a nearby customer, so execution continues.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 10: The Worker starts a Timer

*Sends a Command*

The Worker reaches the request to start a Timer, so it completes the current Workflow Task and sends the StartTimer Command with it. The Temporal Service starts the Timer and records TimerStarted. The Workflow does not progress until the Timer fires.

**Commands:** `StartTimer` (30 minutes)

**Event History:**
- `WorkflowTaskCompleted`
- `TimerStarted` (30 minutes)

#### Step 11: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires and the Temporal Service records TimerFired. It then queues a new Workflow Task to deliver that Event to the Workflow and drive progress forward.

**Event History:**
- `TimerFired`
- `WorkflowTaskScheduled`

### Worker crash

#### Step 12: The Worker crashes

*Worker crash*

The Worker polls for the Task, dequeues it, and continues executing the Workflow code — and then crashes right here. How does Temporal recover the state of this Workflow Execution? First, how does the Temporal Service know the Worker crashed?

**Event History:** `WorkflowTaskStarted`

#### Step 13: The Workflow Task times out

*Worker crash*

Once a Worker accepts a Task, it is expected to complete it within a predefined duration. That Workflow Task Timeout, 10 seconds by default, is how a crashed Worker is recognized. The Worker failed to complete the Task in time, so the Temporal Service records WorkflowTaskTimedOut.

**Event History:** `WorkflowTaskTimedOut`

#### Step 14: A new Workflow Task is scheduled

*Temporal Service*

The Temporal Service schedules a new Workflow Task. The Worker that polls for it might be another Worker in the fleet or a new Worker process created by restarting the one that crashed. Everything above the line is the Event History as it stood when the Worker crashed.

**Event History:**
- `WorkflowTaskScheduled`
- *— Event History at the time of the crash —*

### History Replay

#### Step 15: The Worker requests the Event History

*Replay*

Either way, the Worker needs the current Event History for this Workflow Execution, so it requests it from the Temporal Service. The Temporal Service provides the History.

#### Step 16: Replay re-executes the code

*Replay*

The Worker begins re-executing the code with the same input, which was stored in the WorkflowExecutionStarted Event. Because the Workflow Definition is deterministic, the state of every variable encountered so far is identical to what it was before the crash. For example, totalPrice holds the same value.

#### Step 17: The Activity result comes from the History

*Replay*

When Replay reaches the call to schedule GetDistance, it creates a ScheduleActivityTask Command but does not issue it. Instead, the Worker inspects the Event History and finds ActivityTaskScheduled for this Activity Type, ActivityTaskStarted showing a Worker dequeued the Task, and ActivityTaskCompleted with a result of 15. The Activity already ran, so the Command is not issued.

**Commands:** ✓ `ScheduleActivityTask` (GetDistance) — created during Replay, not issued

#### Step 18: Replay assigns the stored result

*Replay*

The Worker assigns the value stored in the ActivityTaskCompleted Event, 15, to the distance variable. The Activity is not re-executed, so there is no way for it to behave differently during Replay than it did during the original execution. The conditional evaluates to false, just as it did before.

#### Step 19: The Timer is already done

*Replay*

The Worker reaches the request to start a Timer and creates a StartTimer Command, which again it does not issue. The Event History contains TimerStarted and TimerFired, so the Worker knows the Timer already started and fired during the previous execution.

**Commands:** ✓ `StartTimer` (30 minutes) — created during Replay, not issued

#### Step 20: State is restored to the point of the crash

*Replay*

The Worker has reached the point where the crash occurred, and replaying the code has completely restored the state of the Workflow Execution. Every variable, including totalPrice, holds the value it held before the crash.

### Execution resumes

#### Step 21: Execution moves past the crash

*Sends a Command*

The Worker reaches a statement beyond where the crash occurred, which is evident because the Event History contains no Events for the SendBill Activity. Execution continues as if the crash never happened: the Worker completes the current Workflow Task and includes the Command with it.

**Event History:** `WorkflowTaskCompleted`

#### Step 22: The SendBill Activity runs

*Sends a Command*

The Worker issues the Command, the Temporal Service queues the Activity Task, and a Worker dequeues and runs it. When the Activity returns, the Worker notifies the Temporal Service, which records ActivityTaskCompleted with the result from SendBill.

**Commands:** `ScheduleActivityTask` (SendBill)

**Event History:**
- `ActivityTaskScheduled` (SendBill)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 23: One more Workflow Task delivers the result

*Temporal Service*

The Temporal Service has not received a Command saying the Workflow Execution completed or failed, so it schedules another Workflow Task to continue progress. A Worker accepts it and resumes the Workflow code.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 24: The Workflow Execution completes

*Sends a Command*

The Workflow returns, so the Worker completes the current Workflow Task and issues a CompleteWorkflowExecution Command that contains the result. The Temporal Service records WorkflowExecutionCompleted as the final Event. The result is identical to an execution that never crashed.

**Commands:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

**Event History:**
- `WorkflowTaskCompleted`
- `WorkflowExecutionCompleted`

The walkthrough covers four phases:

1. **Original execution**: the Client starts the Workflow Execution, and the Commands the Worker issues become Events in
   the Event History.
2. **Worker crash**: the Worker dies partway through a Workflow Task. When the Workflow Task Timeout elapses, 10 seconds
   by default, the Temporal Service records `WorkflowTaskTimedOut` and schedules a new Workflow Task.
3. **History Replay**: a Worker requests the Event History and re-executes the Workflow code with the original input,
   which the `WorkflowExecutionStarted` Event stores. Commands the Worker creates during Replay are matched against the
   Event History instead of being issued, so Activities don't run again. The Worker uses the results stored in the
   `ActivityTaskCompleted` Events.
4. **Execution resumes**: past the point of the crash, the Event History holds no matching Events, so the Worker issues
   Commands for real again until the Workflow Execution completes. The result is identical to an execution that never
   crashed.

## Example of a non-deterministic Workflow Definition 

Now that Replay has been covered, this section will explain why a Workflow Definition needs to be
[deterministic](/workflow-definition#deterministic-constraints) in order for Replay to work.

A Workflow Definition is deterministic if every execution of it produces the same Commands in the same
sequence given the same input.

As mentioned in the [`How History Replay Provides Durable Execution`](#How-History-Replay-Provides-Durable-Execution)
walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs
the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event
History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History
and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue.

To better understand why a Workflow Definition needs to be deterministic, it's helpful to look at one that violates
it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random
number generator.

```go
func GenerateDailyReport(ctx workflow.Context) error {

    options := workflow.ActivityOptions{
        StartToCloseTimeout: 45 * time.Minute,
    }
    ctx = workflow.WithActivityOptions(ctx, options)

    // this Activity is always executed
    err := workflow.ExecuteActivity(ctx, ImportSalesData).Get(ctx, nil)
    if err != nil {
        return err
    }

    if rand.Intn(100) >= 50 {
        workflow.Sleep(ctx, 4 * time.Hour)
    }

    workflow.GetLogger(ctx).Info("Preparing to run daily report")
    err = workflow.ExecuteActivity(ctx, RunDailyReport).Get(ctx, nil)
    if err != nil {
        return err
    }

    return nil
}
```
### First execution

#### Step 1: The ImportSalesData Activity runs

*Sends a Command*

As this Workflow executes step by step, the first line that results in a Command is the call to the ImportSalesData Activity. The Activity Execution succeeds, so the Temporal Service logs three Events to the Event History.

**Commands created:** `ScheduleActivityTask` (ImportSalesData)

**Relevant History Events:**
- `ActivityTaskScheduled` (ImportSalesData)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 2: A random number decides the next branch

*Internal step*

The Worker reaches a conditional statement that evaluates a randomly generated number. The random number generator returns 84 during this execution, so the expression evaluates to true and execution continues with the next line.

#### Step 3: The Workflow starts a 4-hour Timer

*Sends a Command*

The next line requests a Timer, so the Worker issues a StartTimer Command. The Temporal Service starts the Timer and records TimerStarted, then records TimerFired when the Timer fires.

**Commands created:** `StartTimer` (4 hours)

**Relevant History Events:**
- `TimerStarted` (4 hours)
- `TimerFired`

#### Step 4: The Worker crashes

*Worker crash*

The Worker crashes once it reaches the next line, so another Worker takes over. That Worker uses Replay to restore the current state before continuing execution of the lines that follow.

### History Replay

#### Step 5: Replay establishes the expected Commands

*Replay*

The Worker requests the Event History and determines the sequence of Commands needed to restore the current state. Based on the History, it expects to encounter ScheduleActivityTask (ImportSalesData) and then StartTimer (4 hours).

#### Step 6: The first Command matches

*Replay*

As the Worker executes the code during Replay, it reaches the first call to execute an Activity and creates a ScheduleActivityTask Command. It is the right type of Command and it occurs at the right position in the expected sequence, so Replay proceeds.

**Commands created:** ✓ `ScheduleActivityTask` (ImportSalesData) — created during Replay

#### Step 7: The random number returns something different

*Replay*

The Worker reaches the conditional statement again. This time the random number generator returns 14, so the expression evaluates to false and execution skips the call that starts the Timer.

#### Step 8: The next Command does not match

*Worker crash*

The Worker reaches the request to execute the RunDailyReport Activity and creates another ScheduleActivityTask Command. That is not the Command it expected at this position in the sequence, so the Worker cannot restore the previous state.

**Commands created:** ✗ `ScheduleActivityTask` (RunDailyReport) — created during Replay (expected: StartTimer (4 hours))

#### Step 9: Replay fails with a non-determinism error

*Worker crash*

The Workflow produced a different sequence of Commands during Replay than the Event History recorded before the crash, so the Workflow Execution cannot be replayed. The random number generator is the source of the non-determinism.

During the first execution, the random number is 84, so the Workflow starts a Timer and the Event History records
`TimerStarted` and `TimerFired`. During Replay, the random number is 14, so the Workflow skips the Timer and produces a
`ScheduleActivityTask` Command where the Event History expects `StartTimer`. That mismatch is what makes Replay fail.

Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is
considered a [Workflow Task Failure](/references/failures#workflow-task-failures) which is considered a transient
failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the
Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a
strategy called versioning to address this non-determinism error. See [versioning](/develop/go/workflows/versioning) to
learn more.

For more information on how Temporal handles Durable Execution or to see these walkthroughs in video format with more
explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and
[Versioning Workflows](https://learn.temporal.io/courses/versioning/).

## Temporal Applications support non-deterministic operations

We want to emphasize that although your Workflow Definition needs to be deterministic, your application itself does not!

Remember that pretty much anything that interacts with the external world is inherently non-deterministic:

- Calling LLM APIs
- Querying databases
- Reading or writing files
- Making HTTP requests to external services

**Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow Definition must be
deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed
above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities
which have built-in support for retries.

---

# Event History walkthrough with the Java SDK

Source: https://docs.temporal.io/encyclopedia/event-history/event-history-java

In order to understand how Workflow Replay works, this page will go through the following walkthroughs:

1. [How Workflow code maps to Commands](#How-Workflow-Code-Maps-To-Commands)
2. [How Workflow Commands map to Events](#How-Workflow-Commands-Map-To-Events)
3. [How History Replay provides Durable Execution](#How-History-Replay-Provides-Durable-Execution)
4. [Example of a non-deterministic Workflow Definition](#Example-of-Non-Deterministic-Workflow)

## How Workflow code maps to Commands 

This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the
Temporal Service know what to do.

Step through the Workflow Definition below. Each step highlights the statements the Worker is running and shows the
Commands it has issued so far.

```java
public class PizzaWorkflowImpl implements PizzaWorkflow {

    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(5))
        .build();

    private final PizzaActivities activities =
        Workflow.newActivityStub(PizzaActivities.class, options);

    @Override
    public String pizzaWorkflow(Order order) {

        // Iterate over the items and calculate the cost of the order
        int totalPrice = 0;
        for (Pizza pizza : order.getItems()) {
            totalPrice += pizza.getPrice();
        }

        int distance = activities.getDistance(order.getAddress());

        if (order.isDelivery() && distance > 25) {
            String message = "Customer lives outside the service area";
            throw ApplicationFailure.newFailure(message,
                OutOfServiceAreaException.class.getName());
        }

        // Wait for 30 minutes before billing the customer
        Workflow.sleep(Duration.ofMinutes(30));

        Bill bill = new Bill();
        bill.setCustomerId(order.getCustomer().getCustomerId());
        bill.setAmount(totalPrice);
        bill.setDescription(order.getOrderNumber());

        String confirmation = activities.sendBill(bill);

        return confirmation;
    }
}
```
#### Step 1: A basic Workflow Definition

This Workflow Definition takes a pizza order and does the work listed here. Step through it to see which
statements the Worker handles on its own and which ones send a Command to the Temporal Service.

- Defines a Start-to-Close Timeout
- Calculates the total price of the pizzas
- Determines the distance to the customer
- Fails if the customer is too far away for delivery
- Sleeps for 30 minutes
- Populates a class with billing information
- Sends a bill to the customer

#### Step 2: Internal steps

*Internal step*

These steps are internal to the Workflow. The Worker runs them in your process and never contacts the Temporal
Service.

#### Step 3: Steps that reach the Temporal Service

*Sends a Command*

These steps do involve the Temporal Service. Requesting an Activity Execution generates a Command to schedule
the Activity Task, and returning from the Workflow tells the Temporal Service that the Workflow Execution is
complete.

#### Step 4: Total the price of the order

*Internal step*

The walkthrough starts here. Setting the Start-to-Close Timeout and adding up the price of each pizza are
internal steps, so they don't require any interaction with the Temporal Service.

#### Step 5: Request the GetDistance Activity

*Sends a Command*

The Worker reaches a statement that requires the Temporal Service: a request to execute an Activity. It issues
a ScheduleActivityTask Command with the details the Temporal Service needs, such as the Task Queue name, the
Activity Type, and the input values. An Activity can take hours or days to complete, and the Worker holds no
resources while it waits.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", GetDistance, { Line1: "123 Oak St.", Line2: "", ... })

#### Step 6: Evaluate the distance

*Internal step*

The Worker evaluates the distance returned by the Activity. If the customer lived too far away, the Workflow
would throw an exception, which sends a Command asking the Temporal Service to fail the Workflow Execution.
This order is going to a nearby customer, so execution continues.

#### Step 7: Start a Timer

*Sends a Command*

The call to sleep is another statement that involves the Temporal Service. The Worker issues a StartTimer
Command that includes the duration, and this Workflow Execution pauses for 30 minutes until the Timer fires.

**Commands issued:** `StartTimer` (30 minutes)

#### Step 8: Populate the bill

*Internal step*

The Timer fires and execution resumes. These lines create and populate the object that holds the input for the
next Activity. The object relates to an Activity, but building it doesn't involve the Temporal Service.

#### Step 9: Request the SendBill Activity

*Sends a Command*

The next statement requests execution of an Activity, so the Worker issues another ScheduleActivityTask
Command to the Temporal Service.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", SendBill, { Amount: 2750, Description: "Pizzas", ... })

#### Step 10: Return from the Workflow

*Sends a Command*

Returning from the Workflow method also results in a Command. The Worker issues CompleteWorkflowExecution to
the Temporal Service, which includes the value returned from the method.

**Commands issued:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

Four statements in this Workflow Definition produce a Command:

| Statement                                    | Command                     |
| -------------------------------------------- | --------------------------- |
| `activities.getDistance(order.getAddress())` | `ScheduleActivityTask`      |
| `Workflow.sleep(Duration.ofMinutes(30))`     | `StartTimer`                |
| `activities.sendBill(bill)`                  | `ScheduleActivityTask`      |
| `return confirmation;`                       | `CompleteWorkflowExecution` |

Everything else is an internal step. Setting the Start-to-Close Timeout, totaling the order price, evaluating the
distance, and populating the bill object all run in the Worker without contacting the Temporal Service.

## How Workflow Commands map to Events 

The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The
Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution
of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so
they're durable, and will even survive a crash of the Temporal Service itself.

These Events are what are used to recreate a Workflow Execution's state in the case of failure.

Step through the same Workflow Definition to see each Command the Worker issues and the Events the Temporal Service
records in response.

```java
public class PizzaWorkflowImpl implements PizzaWorkflow {

    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(5))
        .build();

    private final PizzaActivities activities =
        Workflow.newActivityStub(PizzaActivities.class, options);

    @Override
    public String pizzaWorkflow(Order order) {

        // Iterate over the items and calculate the cost of the order
        int totalPrice = 0;
        for (Pizza pizza : order.getItems()) {
            totalPrice += pizza.getPrice();
        }

        int distance = activities.getDistance(order.getAddress());

        if (order.isDelivery() && distance > 25) {
            String message = "Customer lives outside the service area";
            throw ApplicationFailure.newFailure(message,
                OutOfServiceAreaException.class.getName());
        }

        // Wait for 30 minutes before billing the customer
        Workflow.sleep(Duration.ofMinutes(30));

        Bill bill = new Bill();
        bill.setCustomerId(order.getCustomer().getCustomerId());
        bill.setAmount(totalPrice);
        bill.setDescription(order.getOrderNumber());

        String confirmation = activities.sendBill(bill);

        return confirmation;
    }
}
```
#### Step 1: Commands and the Events they produce

This walkthrough keeps a running list of the Commands the Worker issues and the Events the Temporal Service records in response. Blue Events are the direct result of a Command. Pink Events are an indirect result.

#### Step 2: The GetDistance Activity is scheduled

*Sends a Command*

The call to the Activity is the first line of code in the Workflow that causes a Command to be issued. In response, the Temporal Service creates an Activity Task, adds it to the Task Queue, and appends the ActivityTaskScheduled Event to the Event History.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Events:** `ActivityTaskScheduled` (GetDistance)

#### Step 3: A Worker polls for the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to an available Worker. It responds to the poll request with the Task, and the Worker begins executing the code needed to complete it. Nothing is recorded in the Event History yet.

#### Step 4: The Activity Task closes

*Temporal Service*

The ActivityTaskStarted Event is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event. It is an indirect result of the Command. The Start-to-Close Timeout sets how long the Activity has to complete.

**Events:** `ActivityTaskStarted`

#### Step 5: The Activity reports its result

*Temporal Service*

When the Activity Definition returns a result, the Worker sends a message to the Temporal Service to say the Task is complete. This is a notification, not a Command, because it does not ask the Temporal Service to do anything that lets the Workflow Execution progress. The Temporal Service records ActivityTaskCompleted.

**Events:** `ActivityTaskCompleted` (distance = 15)

#### Step 6: The Worker starts a Timer

*Sends a Command*

The next statement that results in a Command is the call to start a Timer. The Temporal Service starts a 30-minute Timer and records a TimerStarted Event, a direct result of the StartTimer Command.

**Commands:** `StartTimer` (30 minutes)

**Events:** `TimerStarted` (30 minutes)

#### Step 7: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires on the Temporal Service, which records a TimerFired Event. The Workflow Execution continues with the next statement, but populating the bill is an internal step that does not interact with the Temporal Service.

**Events:** `TimerFired`

#### Step 8: The SendBill Activity is scheduled

*Sends a Command*

The Worker reaches the call to the SendBill Activity and issues another ScheduleActivityTask Command. The Temporal Service adds an Activity Task to the Task Queue and records an ActivityTaskScheduled Event.

**Commands:** `ScheduleActivityTask` (SendBill)

**Events:** `ActivityTaskScheduled` (SendBill)

#### Step 9: A Worker dequeues the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to the Worker. The Worker removes the Task from the Task Queue and begins working on it, and the Temporal Service records an ActivityTaskStarted Event to signify that the Task was dequeued.

**Events:** `ActivityTaskStarted`

#### Step 10: The bill is sent

*Temporal Service*

When the Activity returns, the Task is complete and the Worker notifies the Temporal Service, which records the ActivityTaskCompleted Event. Execution continues until the Workflow completes. The next walkthrough covers the full Event History, including the Workflow Task Events.

**Events:** `ActivityTaskCompleted`

Blue Events are the direct result of a Command. Pink Events are an indirect result, such as the Events the Temporal
Service records when a Worker starts or finishes a Task:

| Command                | Direct Event            | Indirect Events                                |
| ---------------------- | ----------------------- | ---------------------------------------------- |
| `ScheduleActivityTask` | `ActivityTaskScheduled` | `ActivityTaskStarted`, `ActivityTaskCompleted` |
| `StartTimer`           | `TimerStarted`          | `TimerFired`                                   |

## How History Replay provides Durable Execution 

Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look
at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of
a failure.

This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and
Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of
the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed.

```java
public class PizzaWorkflowImpl implements PizzaWorkflow {

    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(5))
        .build();

    private final PizzaActivities activities =
        Workflow.newActivityStub(PizzaActivities.class, options);

    @Override
    public String pizzaWorkflow(Order order) {

        // Iterate over the items and calculate the cost of the order
        int totalPrice = 0;
        for (Pizza pizza : order.getItems()) {
            totalPrice += pizza.getPrice();
        }

        int distance = activities.getDistance(order.getAddress());

        if (order.isDelivery() && distance > 25) {
            String message = "Customer lives outside the service area";
            throw ApplicationFailure.newFailure(message,
                OutOfServiceAreaException.class.getName());
        }

        // Wait for 30 minutes before billing the customer
        Workflow.sleep(Duration.ofMinutes(30));

        Bill bill = new Bill();
        bill.setCustomerId(order.getCustomer().getCustomerId());
        bill.setAmount(totalPrice);
        bill.setDescription(order.getOrderNumber());

        String confirmation = activities.sendBill(bill);

        return confirmation;
    }
}
```
### Original execution

#### Step 1: A Client requests the Workflow Execution

*Temporal Service*

The walkthrough begins with a request to execute this Workflow Definition with input data about the customer and the pizzas ordered. The Temporal Service records WorkflowExecutionStarted, always the first Event of a Workflow Execution, and that Event holds the input data.

**Event History:** `WorkflowExecutionStarted` (customer, pizzas ordered)

#### Step 2: A Workflow Task is queued

*Temporal Service*

The Temporal Service adds a Workflow Task to the Task Queue and records a WorkflowTaskScheduled Event.

**Event History:** `WorkflowTaskScheduled`

#### Step 3: A Worker accepts the Workflow Task

*Temporal Service*

The Temporal Service dispatches the Task to a Worker that is polling the Task Queue. The Worker accepts it, and the Temporal Service records a WorkflowTaskStarted Event.

**Event History:** `WorkflowTaskStarted`

#### Step 4: The Worker runs the Workflow code

*Internal step*

The Worker invokes the Workflow code and runs it one statement at a time. The first few lines are internal steps that do not interact with the Temporal Service.

#### Step 5: The Worker completes the Workflow Task

*Sends a Command*

The Worker encounters a request to execute an Activity, so it completes the current Workflow Task. It makes a single gRPC call, RespondWorkflowTaskCompleted, that signals completion of the Workflow Task and carries any Commands with it, so WorkflowTaskCompleted and the Command that follows are technically one call.

**Event History:** `WorkflowTaskCompleted`

#### Step 6: The GetDistance Activity Task is scheduled

*Sends a Command*

In response to the ScheduleActivityTask Command, the Temporal Service queues an Activity Task and records an ActivityTaskScheduled Event, a direct result of the Command.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Event History:** `ActivityTaskScheduled` (GetDistance)

#### Step 7: A Worker starts the Activity Task

*Temporal Service*

The Temporal Service dispatches the Activity Task to an available Worker, which starts running the code in the GetDistance Activity. ActivityTaskStarted is an indirect result of the Command, and it is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event.

**Event History:** `ActivityTaskStarted`

#### Step 8: The Activity returns a distance of 15

*Temporal Service*

When the Activity returns, the Worker notifies the Temporal Service that the Activity Execution is complete. The Temporal Service records an ActivityTaskCompleted Event, which contains the result of the Activity.

**Event History:** `ActivityTaskCompleted` (distance = 15)

#### Step 9: The result goes back to the Workflow

*Temporal Service*

To deliver the result to the Workflow, the Temporal Service creates another Workflow Task that includes it. A Worker dequeues that Task, resumes execution, and evaluates the distance. This order is going to a nearby customer, so execution continues.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 10: The Worker starts a Timer

*Sends a Command*

The Worker reaches the request to start a Timer, so it completes the current Workflow Task and sends the StartTimer Command with it. The Temporal Service starts the Timer and records TimerStarted. The Workflow does not progress until the Timer fires.

**Commands:** `StartTimer` (30 minutes)

**Event History:**
- `WorkflowTaskCompleted`
- `TimerStarted` (30 minutes)

#### Step 11: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires and the Temporal Service records TimerFired. It then queues a new Workflow Task to deliver that Event to the Workflow and drive progress forward.

**Event History:**
- `TimerFired`
- `WorkflowTaskScheduled`

### Worker crash

#### Step 12: The Worker crashes

*Worker crash*

The Worker polls for the Task, dequeues it, and continues executing the Workflow code — and then crashes right here. How does Temporal recover the state of this Workflow Execution? First, how does the Temporal Service know the Worker crashed?

**Event History:** `WorkflowTaskStarted`

#### Step 13: The Workflow Task times out

*Worker crash*

Once a Worker accepts a Task, it is expected to complete it within a predefined duration. That Workflow Task Timeout, 10 seconds by default, is how a crashed Worker is recognized. The Worker failed to complete the Task in time, so the Temporal Service records WorkflowTaskTimedOut.

**Event History:** `WorkflowTaskTimedOut`

#### Step 14: A new Workflow Task is scheduled

*Temporal Service*

The Temporal Service schedules a new Workflow Task. The Worker that polls for it might be another Worker in the fleet or a new Worker process created by restarting the one that crashed. Everything above the line is the Event History as it stood when the Worker crashed.

**Event History:**
- `WorkflowTaskScheduled`
- *— Event History at the time of the crash —*

### History Replay

#### Step 15: The Worker requests the Event History

*Replay*

Either way, the Worker needs the current Event History for this Workflow Execution, so it requests it from the Temporal Service. The Temporal Service provides the History.

#### Step 16: Replay re-executes the code

*Replay*

The Worker begins re-executing the code with the same input, which was stored in the WorkflowExecutionStarted Event. Because the Workflow Definition is deterministic, the state of every variable encountered so far is identical to what it was before the crash. For example, totalPrice holds the same value.

#### Step 17: The Activity result comes from the History

*Replay*

When Replay reaches the call to schedule GetDistance, it creates a ScheduleActivityTask Command but does not issue it. Instead, the Worker inspects the Event History and finds ActivityTaskScheduled for this Activity Type, ActivityTaskStarted showing a Worker dequeued the Task, and ActivityTaskCompleted with a result of 15. The Activity already ran, so the Command is not issued.

**Commands:** ✓ `ScheduleActivityTask` (GetDistance) — created during Replay, not issued

#### Step 18: Replay assigns the stored result

*Replay*

The Worker assigns the value stored in the ActivityTaskCompleted Event, 15, to the distance variable. The Activity is not re-executed, so there is no way for it to behave differently during Replay than it did during the original execution. The conditional evaluates to false, just as it did before.

#### Step 19: The Timer is already done

*Replay*

The Worker reaches the request to start a Timer and creates a StartTimer Command, which again it does not issue. The Event History contains TimerStarted and TimerFired, so the Worker knows the Timer already started and fired during the previous execution.

**Commands:** ✓ `StartTimer` (30 minutes) — created during Replay, not issued

#### Step 20: State is restored to the point of the crash

*Replay*

The Worker has reached the point where the crash occurred, and replaying the code has completely restored the state of the Workflow Execution. Every variable, including totalPrice, holds the value it held before the crash.

### Execution resumes

#### Step 21: Execution moves past the crash

*Sends a Command*

The Worker reaches a statement beyond where the crash occurred, which is evident because the Event History contains no Events for the SendBill Activity. Execution continues as if the crash never happened: the Worker completes the current Workflow Task and includes the Command with it.

**Event History:** `WorkflowTaskCompleted`

#### Step 22: The SendBill Activity runs

*Sends a Command*

The Worker issues the Command, the Temporal Service queues the Activity Task, and a Worker dequeues and runs it. When the Activity returns, the Worker notifies the Temporal Service, which records ActivityTaskCompleted with the result from SendBill.

**Commands:** `ScheduleActivityTask` (SendBill)

**Event History:**
- `ActivityTaskScheduled` (SendBill)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 23: One more Workflow Task delivers the result

*Temporal Service*

The Temporal Service has not received a Command saying the Workflow Execution completed or failed, so it schedules another Workflow Task to continue progress. A Worker accepts it and resumes the Workflow code.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 24: The Workflow Execution completes

*Sends a Command*

The Workflow returns, so the Worker completes the current Workflow Task and issues a CompleteWorkflowExecution Command that contains the result. The Temporal Service records WorkflowExecutionCompleted as the final Event. The result is identical to an execution that never crashed.

**Commands:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

**Event History:**
- `WorkflowTaskCompleted`
- `WorkflowExecutionCompleted`

The walkthrough covers four phases:

1. **Original execution**: the Client starts the Workflow Execution, and the Commands the Worker issues become Events in
   the Event History.
2. **Worker crash**: the Worker dies partway through a Workflow Task. When the Workflow Task Timeout elapses, 10 seconds
   by default, the Temporal Service records `WorkflowTaskTimedOut` and schedules a new Workflow Task.
3. **History Replay**: a Worker requests the Event History and re-executes the Workflow code with the original input,
   which the `WorkflowExecutionStarted` Event stores. Commands the Worker creates during Replay are matched against the
   Event History instead of being issued, so Activities don't run again. The Worker uses the results stored in the
   `ActivityTaskCompleted` Events.
4. **Execution resumes**: past the point of the crash, the Event History holds no matching Events, so the Worker issues
   Commands for real again until the Workflow Execution completes. The result is identical to an execution that never
   crashed.

## Example of a non-deterministic Workflow Definition 

Now that Replay has been covered, this section will explain why a Workflow Definition needs to be
[deterministic](/workflow-definition#deterministic-constraints) in order for Replay to work.

A Workflow Definition is deterministic if every execution of it produces the same Commands in the same
sequence given the same input.

As mentioned in the [`How History Replay Provides Durable Execution`](#How-History-Replay-Provides-Durable-Execution)
walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs
the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event
History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History
and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue.

To better understand why a Workflow Definition needs to be deterministic, it's helpful to look at one that violates
it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random
number generator.

```java
public class GenerateDailyReportImpl implements GenerateDailyReport {

    ActivityOptions options = ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofMinutes(45))
        .build();

    private final ReportActivities activities =
        Workflow.newActivityStub(ReportActivities.class, options);

    @Override
    public void generateDailyReport() {

        String salesData = activities.importSalesData();

        Random random = new Random();
        if (random.nextInt(101) >= 50) {
            // Sleep for 4 hours
            Workflow.sleep(Duration.ofHours(4));
        }

        Workflow.getLogger(this.getClass()).info("Preparing to run daily report");
        DailyReport report = activities.runDailyReport(salesData);
    }
}
```
### First execution

#### Step 1: The importSalesData Activity runs

*Sends a Command*

As this Workflow executes step by step, the first line that results in a Command is the call to the importSalesData Activity. The Activity Execution succeeds, so the Temporal Service logs three Events to the Event History.

**Commands created:** `ScheduleActivityTask` (importSalesData)

**Relevant History Events:**
- `ActivityTaskScheduled` (importSalesData)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 2: A random number decides the next branch

*Internal step*

The Worker reaches a conditional statement that evaluates a randomly generated number. The random number generator returns 84 during this execution, so the expression evaluates to true and execution continues with the next line.

#### Step 3: The Workflow starts a 4-hour Timer

*Sends a Command*

The next line requests a Timer, so the Worker issues a StartTimer Command. The Temporal Service starts the Timer and records TimerStarted, then records TimerFired when the Timer fires.

**Commands created:** `StartTimer` (4 hours)

**Relevant History Events:**
- `TimerStarted` (4 hours)
- `TimerFired`

#### Step 4: The Worker crashes

*Worker crash*

The Worker crashes once it reaches the next line, so another Worker takes over. That Worker uses Replay to restore the current state before continuing execution of the lines that follow.

### History Replay

#### Step 5: Replay establishes the expected Commands

*Replay*

The Worker requests the Event History and determines the sequence of Commands needed to restore the current state. Based on the History, it expects to encounter ScheduleActivityTask (importSalesData) and then StartTimer (4 hours).

#### Step 6: The first Command matches

*Replay*

As the Worker executes the code during Replay, it reaches the first call to execute an Activity and creates a ScheduleActivityTask Command. It is the right type of Command and it occurs at the right position in the expected sequence, so Replay proceeds.

**Commands created:** ✓ `ScheduleActivityTask` (importSalesData) — created during Replay

#### Step 7: The random number returns something different

*Replay*

The Worker reaches the conditional statement again. This time the random number generator returns 14, so the expression evaluates to false and execution skips the call that starts the Timer.

#### Step 8: The next Command does not match

*Worker crash*

The Worker reaches the request to execute the runDailyReport Activity and creates another ScheduleActivityTask Command. That is not the Command it expected at this position in the sequence, so the Worker cannot restore the previous state.

**Commands created:** ✗ `ScheduleActivityTask` (runDailyReport) — created during Replay (expected: StartTimer (4 hours))

#### Step 9: Replay fails with a non-determinism error

*Worker crash*

The Workflow produced a different sequence of Commands during Replay than the Event History recorded before the crash, so the Workflow Execution cannot be replayed. The random number generator is the source of the non-determinism.

During the first execution, the random number is 84, so the Workflow starts a Timer and the Event History records
`TimerStarted` and `TimerFired`. During Replay, the random number is 14, so the Workflow skips the Timer and produces a
`ScheduleActivityTask` Command where the Event History expects `StartTimer`. That mismatch is what makes Replay fail.

Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is
considered a [Workflow Task Failure](/references/failures#workflow-task-failures) which is considered a transient
failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the
Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a
strategy called versioning to address this non-determinism error. See [versioning](/develop/java/workflows/versioning)
to learn more.

For more information on how Temporal handles Durable Execution or to see these walkthroughs in video format with more
explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and
[Versioning Workflows](https://learn.temporal.io/courses/versioning/).

## Temporal Applications support non-deterministic operations

We want to emphasize that although your Workflow Definition needs to be deterministic, your application itself does not!

Remember that pretty much anything that interacts with the external world is inherently non-deterministic:

- Calling LLM APIs
- Querying databases
- Reading or writing files
- Making HTTP requests to external services

**Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow Definition must be
deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed
above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities
which have built-in support for retries.

---

# Event History walkthrough with the Python SDK

Source: https://docs.temporal.io/encyclopedia/event-history/event-history-python

In order to understand how Workflow Replay works, this page will go through the following walkthroughs:

1. [How Workflow code maps to Commands](#How-Workflow-Code-Maps-To-Commands)
2. [How Workflow Commands map to Events](#How-Workflow-Commands-Map-To-Events)
3. [How History Replay provides Durable Execution](#How-History-Replay-Provides-Durable-Execution)
4. [Example of a non-deterministic Workflow Definition](#Example-of-Non-Deterministic-Workflow)

## How Workflow code maps to Commands 

This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the
Temporal Service know what to do.

Step through the Workflow Definition below. Each step highlights the statements the Worker is running and shows the
Commands it has issued so far.

```python
@workflow.defn
class PizzaOrderWorkflow:
    @workflow.run
    async def order_pizza(self, order: PizzaOrder) -> OrderConfirmation:
        total_price = sum(pizza.price for pizza in order.items)

        distance = await workflow.execute_activity_method(
            PizzaOrderActivities.get_distance,
            order.address,
            start_to_close_timeout=timedelta(seconds=5),
        )

        if order.is_delivery and distance.kilometers > 25:
            error_message = "customer lives outside the service area"
            raise ApplicationError(error_message)

        # Wait 30 minutes before billing the customer
        await asyncio.sleep(timedelta(minutes=30).total_seconds())

        bill = Bill(
            customer_id=order.customer.customer_id,
            order_number=order.order_number,
            description="Pizza order",
            amount=total_price,
        )

        confirmation = await workflow.execute_activity_method(
            PizzaOrderActivities.send_bill,
            bill,
            start_to_close_timeout=timedelta(seconds=5),
        )

        return confirmation
```
#### Step 1: A basic Workflow Definition

This Workflow Definition takes a pizza order and does the work listed here. Step through it to see which
statements the Worker handles on its own and which ones send a Command to the Temporal Service.

- Calculates the total price of the pizzas
- Determines the distance to the customer
- Fails if the customer is too far away for delivery
- Sleeps for 30 minutes
- Populates a data structure with billing information
- Sends a bill to the customer

#### Step 2: Internal steps

*Internal step*

These steps are internal to the Workflow. The Worker runs them in your process and never contacts the Temporal
Service.

#### Step 3: Steps that reach the Temporal Service

*Sends a Command*

These steps do involve the Temporal Service. Requesting an Activity Execution generates a Command to schedule
the Activity Task, and returning from the Workflow tells the Temporal Service that the Workflow Execution is
complete.

#### Step 4: Total the price of the order

*Internal step*

The walkthrough starts here. Calculating the total price of the pizzas is an internal step, so it doesn't
require any interaction with the Temporal Service.

#### Step 5: Request the GetDistance Activity

*Sends a Command*

The Worker reaches a statement that requires the Temporal Service: a request to execute an Activity. It issues
a ScheduleActivityTask Command with the details the Temporal Service needs, such as the Task Queue name, the
Activity Type, and the input values. An Activity can take hours or days to complete, and the Worker holds no
resources while it waits.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", GetDistance, { Line1: "123 Oak St.", Line2: "", ... })

#### Step 6: Evaluate the distance

*Internal step*

The Worker evaluates the distance returned by the Activity. If the customer lived too far away, the Workflow
would raise an exception, which sends a Command asking the Temporal Service to fail the Workflow Execution.
This order is going to a nearby customer, so execution continues.

#### Step 7: Start a Timer

*Sends a Command*

The call to sleep is another statement that involves the Temporal Service. The Worker issues a StartTimer
Command that includes the duration, and this Workflow Execution pauses for 30 minutes until the Timer fires.

**Commands issued:** `StartTimer` (30 minutes)

#### Step 8: Populate the bill

*Internal step*

The Timer fires and execution resumes. These lines create and populate the data structure that holds the input
for the next Activity. The structure relates to an Activity, but building it doesn't involve the Temporal
Service.

#### Step 9: Request the SendBill Activity

*Sends a Command*

The next statement requests execution of an Activity, so the Worker issues another ScheduleActivityTask
Command to the Temporal Service.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", SendBill, { Amount: 2750, Description: "Pizzas", ... })

#### Step 10: Return from the Workflow

*Sends a Command*

Returning from the Workflow method also results in a Command. The Worker issues CompleteWorkflowExecution to
the Temporal Service, which includes the value returned from the method.

**Commands issued:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

Four statements in this Workflow Definition produce a Command:

| Statement                                             | Command                     |
| ----------------------------------------------------- | --------------------------- |
| `workflow.execute_activity_method(get_distance, ...)` | `ScheduleActivityTask`      |
| `asyncio.sleep(timedelta(minutes=30)...)`             | `StartTimer`                |
| `workflow.execute_activity_method(send_bill, ...)`    | `ScheduleActivityTask`      |
| `return confirmation`                                 | `CompleteWorkflowExecution` |

Everything else is an internal step. Totaling the order price, evaluating the distance, and populating the bill all run
in the Worker without contacting the Temporal Service.

## How Workflow Commands map to Events 

The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The
Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution
of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so
they're durable, and will even survive a crash of the Temporal Service itself.

These Events are what are used to recreate a Workflow Execution's state in the case of failure.

Step through the same Workflow Definition to see each Command the Worker issues and the Events the Temporal Service
records in response.

```python
@workflow.defn
class PizzaOrderWorkflow:
    @workflow.run
    async def order_pizza(self, order: PizzaOrder) -> OrderConfirmation:
        total_price = sum(pizza.price for pizza in order.items)

        distance = await workflow.execute_activity_method(
            PizzaOrderActivities.get_distance,
            order.address,
            start_to_close_timeout=timedelta(seconds=5),
        )

        if order.is_delivery and distance.kilometers > 25:
            error_message = "customer lives outside the service area"
            raise ApplicationError(error_message)

        # Wait 30 minutes before billing the customer
        await asyncio.sleep(timedelta(minutes=30).total_seconds())

        bill = Bill(
            customer_id=order.customer.customer_id,
            order_number=order.order_number,
            description="Pizza order",
            amount=total_price,
        )

        confirmation = await workflow.execute_activity_method(
            PizzaOrderActivities.send_bill,
            bill,
            start_to_close_timeout=timedelta(seconds=5),
        )

        return confirmation
```
#### Step 1: Commands and the Events they produce

This walkthrough keeps a running list of the Commands the Worker issues and the Events the Temporal Service records in response. Blue Events are the direct result of a Command. Pink Events are an indirect result.

#### Step 2: The GetDistance Activity is scheduled

*Sends a Command*

The call to the Activity is the first line of code in the Workflow that causes a Command to be issued. In response, the Temporal Service creates an Activity Task, adds it to the Task Queue, and appends the ActivityTaskScheduled Event to the Event History.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Events:** `ActivityTaskScheduled` (GetDistance)

#### Step 3: A Worker polls for the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to an available Worker. It responds to the poll request with the Task, and the Worker begins executing the code needed to complete it. Nothing is recorded in the Event History yet.

#### Step 4: The Activity Task closes

*Temporal Service*

The ActivityTaskStarted Event is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event. It is an indirect result of the Command. The Start-to-Close Timeout sets how long the Activity has to complete.

**Events:** `ActivityTaskStarted`

#### Step 5: The Activity reports its result

*Temporal Service*

When the Activity Definition returns a result, the Worker sends a message to the Temporal Service to say the Task is complete. This is a notification, not a Command, because it does not ask the Temporal Service to do anything that lets the Workflow Execution progress. The Temporal Service records ActivityTaskCompleted.

**Events:** `ActivityTaskCompleted` (distance = 15)

#### Step 6: The Worker starts a Timer

*Sends a Command*

The next statement that results in a Command is the call to start a Timer. The Temporal Service starts a 30-minute Timer and records a TimerStarted Event, a direct result of the StartTimer Command.

**Commands:** `StartTimer` (30 minutes)

**Events:** `TimerStarted` (30 minutes)

#### Step 7: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires on the Temporal Service, which records a TimerFired Event. The Workflow Execution continues with the next statement, but populating the bill is an internal step that does not interact with the Temporal Service.

**Events:** `TimerFired`

#### Step 8: The SendBill Activity is scheduled

*Sends a Command*

The Worker reaches the call to the SendBill Activity and issues another ScheduleActivityTask Command. The Temporal Service adds an Activity Task to the Task Queue and records an ActivityTaskScheduled Event.

**Commands:** `ScheduleActivityTask` (SendBill)

**Events:** `ActivityTaskScheduled` (SendBill)

#### Step 9: A Worker dequeues the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to the Worker. The Worker removes the Task from the Task Queue and begins working on it, and the Temporal Service records an ActivityTaskStarted Event to signify that the Task was dequeued.

**Events:** `ActivityTaskStarted`

#### Step 10: The bill is sent

*Temporal Service*

When the Activity returns, the Task is complete and the Worker notifies the Temporal Service, which records the ActivityTaskCompleted Event. Execution continues until the Workflow completes. The next walkthrough covers the full Event History, including the Workflow Task Events.

**Events:** `ActivityTaskCompleted`

Blue Events are the direct result of a Command. Pink Events are an indirect result, such as the Events the Temporal
Service records when a Worker starts or finishes a Task:

| Command                | Direct Event            | Indirect Events                                |
| ---------------------- | ----------------------- | ---------------------------------------------- |
| `ScheduleActivityTask` | `ActivityTaskScheduled` | `ActivityTaskStarted`, `ActivityTaskCompleted` |
| `StartTimer`           | `TimerStarted`          | `TimerFired`                                   |

## How History Replay provides Durable Execution 

Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look
at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of
a failure.

This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and
Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of
the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed.

```python
@workflow.defn
class PizzaOrderWorkflow:
    @workflow.run
    async def order_pizza(self, order: PizzaOrder) -> OrderConfirmation:
        total_price = sum(pizza.price for pizza in order.items)

        distance = await workflow.execute_activity_method(
            PizzaOrderActivities.get_distance,
            order.address,
            start_to_close_timeout=timedelta(seconds=5),
        )

        if order.is_delivery and distance.kilometers > 25:
            error_message = "customer lives outside the service area"
            raise ApplicationError(error_message)

        # Wait 30 minutes before billing the customer
        await asyncio.sleep(timedelta(minutes=30).total_seconds())

        bill = Bill(
            customer_id=order.customer.customer_id,
            order_number=order.order_number,
            description="Pizza order",
            amount=total_price,
        )

        confirmation = await workflow.execute_activity_method(
            PizzaOrderActivities.send_bill,
            bill,
            start_to_close_timeout=timedelta(seconds=5),
        )

        return confirmation
```
### Original execution

#### Step 1: A Client requests the Workflow Execution

*Temporal Service*

The walkthrough begins with a request to execute this Workflow Definition with input data about the customer and the pizzas ordered. The Temporal Service records WorkflowExecutionStarted, always the first Event of a Workflow Execution, and that Event holds the input data.

**Event History:** `WorkflowExecutionStarted` (customer, pizzas ordered)

#### Step 2: A Workflow Task is queued

*Temporal Service*

The Temporal Service adds a Workflow Task to the Task Queue and records a WorkflowTaskScheduled Event.

**Event History:** `WorkflowTaskScheduled`

#### Step 3: A Worker accepts the Workflow Task

*Temporal Service*

The Temporal Service dispatches the Task to a Worker that is polling the Task Queue. The Worker accepts it, and the Temporal Service records a WorkflowTaskStarted Event.

**Event History:** `WorkflowTaskStarted`

#### Step 4: The Worker runs the Workflow code

*Internal step*

The Worker invokes the Workflow code and runs it one statement at a time. The first few lines are internal steps that do not interact with the Temporal Service.

#### Step 5: The Worker completes the Workflow Task

*Sends a Command*

The Worker encounters a request to execute an Activity, so it completes the current Workflow Task. It makes a single gRPC call, RespondWorkflowTaskCompleted, that signals completion of the Workflow Task and carries any Commands with it, so WorkflowTaskCompleted and the Command that follows are technically one call.

**Event History:** `WorkflowTaskCompleted`

#### Step 6: The GetDistance Activity Task is scheduled

*Sends a Command*

In response to the ScheduleActivityTask Command, the Temporal Service queues an Activity Task and records an ActivityTaskScheduled Event, a direct result of the Command.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Event History:** `ActivityTaskScheduled` (GetDistance)

#### Step 7: A Worker starts the Activity Task

*Temporal Service*

The Temporal Service dispatches the Activity Task to an available Worker, which starts running the code in the GetDistance Activity. ActivityTaskStarted is an indirect result of the Command, and it is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event.

**Event History:** `ActivityTaskStarted`

#### Step 8: The Activity returns a distance of 15

*Temporal Service*

When the Activity returns, the Worker notifies the Temporal Service that the Activity Execution is complete. The Temporal Service records an ActivityTaskCompleted Event, which contains the result of the Activity.

**Event History:** `ActivityTaskCompleted` (distance = 15)

#### Step 9: The result goes back to the Workflow

*Temporal Service*

To deliver the result to the Workflow, the Temporal Service creates another Workflow Task that includes it. A Worker dequeues that Task, resumes execution, and evaluates the distance. This order is going to a nearby customer, so execution continues.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 10: The Worker starts a Timer

*Sends a Command*

The Worker reaches the request to start a Timer, so it completes the current Workflow Task and sends the StartTimer Command with it. The Temporal Service starts the Timer and records TimerStarted. The Workflow does not progress until the Timer fires.

**Commands:** `StartTimer` (30 minutes)

**Event History:**
- `WorkflowTaskCompleted`
- `TimerStarted` (30 minutes)

#### Step 11: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires and the Temporal Service records TimerFired. It then queues a new Workflow Task to deliver that Event to the Workflow and drive progress forward.

**Event History:**
- `TimerFired`
- `WorkflowTaskScheduled`

### Worker crash

#### Step 12: The Worker crashes

*Worker crash*

The Worker polls for the Task, dequeues it, and continues executing the Workflow code — and then crashes right here. How does Temporal recover the state of this Workflow Execution? First, how does the Temporal Service know the Worker crashed?

**Event History:** `WorkflowTaskStarted`

#### Step 13: The Workflow Task times out

*Worker crash*

Once a Worker accepts a Task, it is expected to complete it within a predefined duration. That Workflow Task Timeout, 10 seconds by default, is how a crashed Worker is recognized. The Worker failed to complete the Task in time, so the Temporal Service records WorkflowTaskTimedOut.

**Event History:** `WorkflowTaskTimedOut`

#### Step 14: A new Workflow Task is scheduled

*Temporal Service*

The Temporal Service schedules a new Workflow Task. The Worker that polls for it might be another Worker in the fleet or a new Worker process created by restarting the one that crashed. Everything above the line is the Event History as it stood when the Worker crashed.

**Event History:**
- `WorkflowTaskScheduled`
- *— Event History at the time of the crash —*

### History Replay

#### Step 15: The Worker requests the Event History

*Replay*

Either way, the Worker needs the current Event History for this Workflow Execution, so it requests it from the Temporal Service. The Temporal Service provides the History.

#### Step 16: Replay re-executes the code

*Replay*

The Worker begins re-executing the code with the same input, which was stored in the WorkflowExecutionStarted Event. Because the Workflow Definition is deterministic, the state of every variable encountered so far is identical to what it was before the crash. For example, totalPrice holds the same value.

#### Step 17: The Activity result comes from the History

*Replay*

When Replay reaches the call to schedule GetDistance, it creates a ScheduleActivityTask Command but does not issue it. Instead, the Worker inspects the Event History and finds ActivityTaskScheduled for this Activity Type, ActivityTaskStarted showing a Worker dequeued the Task, and ActivityTaskCompleted with a result of 15. The Activity already ran, so the Command is not issued.

**Commands:** ✓ `ScheduleActivityTask` (GetDistance) — created during Replay, not issued

#### Step 18: Replay assigns the stored result

*Replay*

The Worker assigns the value stored in the ActivityTaskCompleted Event, 15, to the distance variable. The Activity is not re-executed, so there is no way for it to behave differently during Replay than it did during the original execution. The conditional evaluates to false, just as it did before.

#### Step 19: The Timer is already done

*Replay*

The Worker reaches the request to start a Timer and creates a StartTimer Command, which again it does not issue. The Event History contains TimerStarted and TimerFired, so the Worker knows the Timer already started and fired during the previous execution.

**Commands:** ✓ `StartTimer` (30 minutes) — created during Replay, not issued

#### Step 20: State is restored to the point of the crash

*Replay*

The Worker has reached the point where the crash occurred, and replaying the code has completely restored the state of the Workflow Execution. Every variable, including totalPrice, holds the value it held before the crash.

### Execution resumes

#### Step 21: Execution moves past the crash

*Sends a Command*

The Worker reaches a statement beyond where the crash occurred, which is evident because the Event History contains no Events for the SendBill Activity. Execution continues as if the crash never happened: the Worker completes the current Workflow Task and includes the Command with it.

**Event History:** `WorkflowTaskCompleted`

#### Step 22: The SendBill Activity runs

*Sends a Command*

The Worker issues the Command, the Temporal Service queues the Activity Task, and a Worker dequeues and runs it. When the Activity returns, the Worker notifies the Temporal Service, which records ActivityTaskCompleted with the result from SendBill.

**Commands:** `ScheduleActivityTask` (SendBill)

**Event History:**
- `ActivityTaskScheduled` (SendBill)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 23: One more Workflow Task delivers the result

*Temporal Service*

The Temporal Service has not received a Command saying the Workflow Execution completed or failed, so it schedules another Workflow Task to continue progress. A Worker accepts it and resumes the Workflow code.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 24: The Workflow Execution completes

*Sends a Command*

The Workflow returns, so the Worker completes the current Workflow Task and issues a CompleteWorkflowExecution Command that contains the result. The Temporal Service records WorkflowExecutionCompleted as the final Event. The result is identical to an execution that never crashed.

**Commands:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

**Event History:**
- `WorkflowTaskCompleted`
- `WorkflowExecutionCompleted`

The walkthrough covers four phases:

1. **Original execution**: the Client starts the Workflow Execution, and the Commands the Worker issues become Events in
   the Event History.
2. **Worker crash**: the Worker dies partway through a Workflow Task. When the Workflow Task Timeout elapses, 10 seconds
   by default, the Temporal Service records `WorkflowTaskTimedOut` and schedules a new Workflow Task.
3. **History Replay**: a Worker requests the Event History and re-executes the Workflow code with the original input,
   which the `WorkflowExecutionStarted` Event stores. Commands the Worker creates during Replay are matched against the
   Event History instead of being issued, so Activities don't run again. The Worker uses the results stored in the
   `ActivityTaskCompleted` Events.
4. **Execution resumes**: past the point of the crash, the Event History holds no matching Events, so the Worker issues
   Commands for real again until the Workflow Execution completes. The result is identical to an execution that never
   crashed.

## Example of a non-deterministic Workflow Definition 

Now that Replay has been covered, this section will explain why a Workflow Definition needs to be
[deterministic](/workflow-definition#deterministic-constraints) in order for Replay to work.

A Workflow Definition is deterministic if every execution of it produces the same Commands in the same
sequence given the same input.

As mentioned in the [`How History Replay Provides Durable Execution`](#How-History-Replay-Provides-Durable-Execution)
walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs
the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event
History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History
and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue.

To better understand why a Workflow Definition needs to be deterministic, it's helpful to look at one that violates
it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random
number generator.

```python
@workflow.defn
class GenerateDailyReport:
    @workflow.run
    async def run(self) -> str:
        sales_data = await workflow.execute_activity_method(
            ReportActivities.import_sales_data,
            start_to_close_timeout=timedelta(minutes=45),
        )

        if non_deterministic_number_generator(100) >= 50:
            # sleep for 4 hours
            await asyncio.sleep(timedelta(hours=4).total_seconds())

        workflow.logger.info("Preparing to run daily report")

        return await workflow.execute_activity_method(
            ReportActivities.run_daily_report,
            sales_data,
            start_to_close_timeout=timedelta(minutes=45),
        )
```
### First execution

#### Step 1: The import_sales_data Activity runs

*Sends a Command*

As this Workflow executes step by step, the first line that results in a Command is the call to the import_sales_data Activity. The Activity Execution succeeds, so the Temporal Service logs three Events to the Event History.

**Commands created:** `ScheduleActivityTask` (import_sales_data)

**Relevant History Events:**
- `ActivityTaskScheduled` (import_sales_data)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 2: A random number decides the next branch

*Internal step*

The Worker reaches a conditional statement that evaluates a randomly generated number. The random number generator returns 84 during this execution, so the expression evaluates to true and execution continues with the next line.

#### Step 3: The Workflow starts a 4-hour Timer

*Sends a Command*

The next line requests a Timer, so the Worker issues a StartTimer Command. The Temporal Service starts the Timer and records TimerStarted, then records TimerFired when the Timer fires.

**Commands created:** `StartTimer` (4 hours)

**Relevant History Events:**
- `TimerStarted` (4 hours)
- `TimerFired`

#### Step 4: The Worker crashes

*Worker crash*

The Worker crashes once it reaches the next line, so another Worker takes over. That Worker uses Replay to restore the current state before continuing execution of the lines that follow.

### History Replay

#### Step 5: Replay establishes the expected Commands

*Replay*

The Worker requests the Event History and determines the sequence of Commands needed to restore the current state. Based on the History, it expects to encounter ScheduleActivityTask (import_sales_data) and then StartTimer (4 hours).

#### Step 6: The first Command matches

*Replay*

As the Worker executes the code during Replay, it reaches the first call to execute an Activity and creates a ScheduleActivityTask Command. It is the right type of Command and it occurs at the right position in the expected sequence, so Replay proceeds.

**Commands created:** ✓ `ScheduleActivityTask` (import_sales_data) — created during Replay

#### Step 7: The random number returns something different

*Replay*

The Worker reaches the conditional statement again. This time the random number generator returns 14, so the expression evaluates to false and execution skips the call that starts the Timer.

#### Step 8: The next Command does not match

*Worker crash*

The Worker reaches the request to execute the run_daily_report Activity and creates another ScheduleActivityTask Command. That is not the Command it expected at this position in the sequence, so the Worker cannot restore the previous state.

**Commands created:** ✗ `ScheduleActivityTask` (run_daily_report) — created during Replay (expected: StartTimer (4 hours))

#### Step 9: Replay fails with a non-determinism error

*Worker crash*

The Workflow produced a different sequence of Commands during Replay than the Event History recorded before the crash, so the Workflow Execution cannot be replayed. The random number generator is the source of the non-determinism.

During the first execution, the random number is 84, so the Workflow starts a Timer and the Event History records
`TimerStarted` and `TimerFired`. During Replay, the random number is 14, so the Workflow skips the Timer and produces a
`ScheduleActivityTask` Command where the Event History expects `StartTimer`. That mismatch is what makes Replay fail.

Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is
considered a [Workflow Task Failure](/references/failures#workflow-task-failures) which is considered a transient
failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the
Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a
strategy called versioning to address this non-determinism error. See [versioning](/develop/python/workflows/versioning)
to learn more.

For more information on how Temporal handles Durable Execution or to see these walkthroughs in video format with more
explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and
[Versioning Workflows](https://learn.temporal.io/courses/versioning/).

## Temporal Applications support non-deterministic operations

We want to emphasize that although your Workflow Definition needs to be deterministic, your application itself does not!

Remember that pretty much anything that interacts with the external world is inherently non-deterministic:

- Calling LLM APIs
- Querying databases
- Reading or writing files
- Making HTTP requests to external services

**Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow Definition must be
deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed
above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities
which have built-in support for retries.

---

# Event History walkthrough with the TypeScript SDK

Source: https://docs.temporal.io/encyclopedia/event-history/event-history-typescript

In order to understand how Workflow Replay works, this page will go through the following walkthroughs:

1. [How Workflow code maps to Commands](#How-Workflow-Code-Maps-To-Commands)
2. [How Workflow Commands map to Events](#How-Workflow-Commands-Map-To-Events)
3. [How History Replay provides Durable Execution](#How-History-Replay-Provides-Durable-Execution)
4. [Example of a non-deterministic Workflow Definition](#Example-of-Non-Deterministic-Workflow)

## How Workflow code maps to Commands 

This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the
Temporal Service know what to do.

Step through the Workflow Definition below. Each step highlights the statements the Worker is running and shows the
Commands it has issued so far.

```typescript
const { sendBill, getDistance } = proxyActivities<typeof activities>({
  startToCloseTimeout: '5 seconds',
});

export async function pizzaWorkflow(order: Order): Promise<string> {
  let distance: Distance | undefined = undefined;
  let totalPrice = 0;

  // compute distance
  distance = await getDistance(order.address);

  if (distance.kilometers > 25) {
    throw new ApplicationFailure('Customer too far away for delivery');
  }

  // Iterate over the items and calculate the cost of the order
  for (const pizza of order.items) {
    totalPrice += pizza.price;
  }

  // Wait 30 minutes before billing the customer
  await sleep('30 minutes');

  const bill = {
    customerID: order.customer.customerID,
    orderNumber: order.orderNumber,
    amount: totalPrice,
    description: 'Pizza',
  };

  const confirmation = await sendBill(bill);

  return confirmation;
}
```
#### Step 1: A basic Workflow Definition

This Workflow Definition takes a pizza order and does the work listed here. Step through it to see which
statements the Worker handles on its own and which ones send a Command to the Temporal Service.

- Defines a Start-to-Close Timeout
- Determines the distance to the customer
- Fails if the customer is too far away for delivery
- Calculates the total price of the pizzas
- Sleeps for 30 minutes
- Populates an object with billing information
- Sends a bill to the customer

#### Step 2: Internal steps

*Internal step*

These steps are internal to the Workflow. The Worker runs them in your process and never contacts the Temporal
Service.

#### Step 3: Steps that reach the Temporal Service

*Sends a Command*

These steps do involve the Temporal Service. Requesting an Activity Execution generates a Command to schedule
the Activity Task, and returning from the Workflow tells the Temporal Service that the Workflow Execution is
complete.

#### Step 4: Total the price of the order

*Internal step*

The walkthrough starts here. Setting the Start-to-Close Timeout and declaring the variables are internal steps,
so they don't require any interaction with the Temporal Service.

#### Step 5: Request the GetDistance Activity

*Sends a Command*

The Worker reaches a statement that requires the Temporal Service: a request to execute an Activity. It issues
a ScheduleActivityTask Command with the details the Temporal Service needs, such as the Task Queue name, the
Activity Type, and the input values. An Activity can take hours or days to complete, and the Worker holds no
resources while it waits.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", GetDistance, { Line1: "123 Oak St.", Line2: "", ... })

#### Step 6: Evaluate the distance

*Internal step*

The Worker evaluates the distance returned by the Activity. If the customer lived too far away, the Workflow
would throw an exception, which sends a Command asking the Temporal Service to fail the Workflow Execution.
This order is going to a nearby customer, so execution continues.

#### Step 7: Total the price of the order

*Internal step*

Adding up the price of each pizza is another internal step, so it does not involve the Temporal Service
either.

#### Step 8: Start a Timer

*Sends a Command*

The call to sleep is another statement that involves the Temporal Service. The Worker issues a StartTimer
Command that includes the duration, and this Workflow Execution pauses for 30 minutes until the Timer fires.

**Commands issued:** `StartTimer` (30 minutes)

#### Step 9: Populate the bill

*Internal step*

The Timer fires and execution resumes. These lines create and populate the object that holds the input for the
next Activity. The object relates to an Activity, but building it doesn't involve the Temporal Service.

#### Step 10: Request the SendBill Activity

*Sends a Command*

The next statement requests execution of an Activity, so the Worker issues another ScheduleActivityTask
Command to the Temporal Service.

**Commands issued:** `ScheduleActivityTask` ("pizza-tasks", SendBill, { Amount: 2750, Description: "Pizzas", ... })

#### Step 11: Return from the Workflow

*Sends a Command*

Returning from the Workflow function also results in a Command. The Worker issues CompleteWorkflowExecution to
the Temporal Service, which includes the value returned from the function.

**Commands issued:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

Four statements in this Workflow Definition produce a Command:

| Statement                          | Command                     |
| ---------------------------------- | --------------------------- |
| `await getDistance(order.address)` | `ScheduleActivityTask`      |
| `await sleep('30 minutes')`        | `StartTimer`                |
| `await sendBill(bill)`             | `ScheduleActivityTask`      |
| `return confirmation`              | `CompleteWorkflowExecution` |

Everything else is an internal step. Setting the Start-to-Close Timeout, declaring the variables, evaluating the
distance, totaling the order price, and populating the bill object all run in the Worker without contacting the Temporal
Service.

## How Workflow Commands map to Events 

The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The
Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution
of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so
they're durable, and will even survive a crash of the Temporal Service itself.

These Events are what are used to recreate a Workflow Execution's state in the case of failure.

Step through the same Workflow Definition to see each Command the Worker issues and the Events the Temporal Service
records in response.

```typescript
const { sendBill, getDistance } = proxyActivities<typeof activities>({
  startToCloseTimeout: '5 seconds',
});

export async function pizzaWorkflow(order: Order): Promise<string> {
  let distance: Distance | undefined = undefined;
  let totalPrice = 0;

  // compute distance
  distance = await getDistance(order.address);

  if (distance.kilometers > 25) {
    throw new ApplicationFailure('Customer too far away for delivery');
  }

  // Iterate over the items and calculate the cost of the order
  for (const pizza of order.items) {
    totalPrice += pizza.price;
  }

  // Wait 30 minutes before billing the customer
  await sleep('30 minutes');

  const bill = {
    customerID: order.customer.customerID,
    orderNumber: order.orderNumber,
    amount: totalPrice,
    description: 'Pizza',
  };

  const confirmation = await sendBill(bill);

  return confirmation;
}
```
#### Step 1: Commands and the Events they produce

This walkthrough keeps a running list of the Commands the Worker issues and the Events the Temporal Service records in response. Blue Events are the direct result of a Command. Pink Events are an indirect result.

#### Step 2: The GetDistance Activity is scheduled

*Sends a Command*

The call to the Activity is the first line of code in the Workflow that causes a Command to be issued. In response, the Temporal Service creates an Activity Task, adds it to the Task Queue, and appends the ActivityTaskScheduled Event to the Event History.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Events:** `ActivityTaskScheduled` (GetDistance)

#### Step 3: A Worker polls for the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to an available Worker. It responds to the poll request with the Task, and the Worker begins executing the code needed to complete it. Nothing is recorded in the Event History yet.

#### Step 4: The Activity Task closes

*Temporal Service*

The ActivityTaskStarted Event is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event. It is an indirect result of the Command. The Start-to-Close Timeout sets how long the Activity has to complete.

**Events:** `ActivityTaskStarted`

#### Step 5: The Activity reports its result

*Temporal Service*

When the Activity Definition returns a result, the Worker sends a message to the Temporal Service to say the Task is complete. This is a notification, not a Command, because it does not ask the Temporal Service to do anything that lets the Workflow Execution progress. The Temporal Service records ActivityTaskCompleted.

**Events:** `ActivityTaskCompleted` (distance = 15)

#### Step 6: The Worker starts a Timer

*Sends a Command*

The next statement that results in a Command is the call to start a Timer. The Temporal Service starts a 30-minute Timer and records a TimerStarted Event, a direct result of the StartTimer Command.

**Commands:** `StartTimer` (30 minutes)

**Events:** `TimerStarted` (30 minutes)

#### Step 7: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires on the Temporal Service, which records a TimerFired Event. The Workflow Execution continues with the next statement, but populating the bill is an internal step that does not interact with the Temporal Service.

**Events:** `TimerFired`

#### Step 8: The SendBill Activity is scheduled

*Sends a Command*

The Worker reaches the call to the SendBill Activity and issues another ScheduleActivityTask Command. The Temporal Service adds an Activity Task to the Task Queue and records an ActivityTaskScheduled Event.

**Commands:** `ScheduleActivityTask` (SendBill)

**Events:** `ActivityTaskScheduled` (SendBill)

#### Step 9: A Worker dequeues the Activity Task

*Temporal Service*

The Temporal Service dispatches this Activity Task to the Worker. The Worker removes the Task from the Task Queue and begins working on it, and the Temporal Service records an ActivityTaskStarted Event to signify that the Task was dequeued.

**Events:** `ActivityTaskStarted`

#### Step 10: The bill is sent

*Temporal Service*

When the Activity returns, the Task is complete and the Worker notifies the Temporal Service, which records the ActivityTaskCompleted Event. Execution continues until the Workflow completes. The next walkthrough covers the full Event History, including the Workflow Task Events.

**Events:** `ActivityTaskCompleted`

Blue Events are the direct result of a Command. Pink Events are an indirect result, such as the Events the Temporal
Service records when a Worker starts or finishes a Task:

| Command                | Direct Event            | Indirect Events                                |
| ---------------------- | ----------------------- | ---------------------------------------------- |
| `ScheduleActivityTask` | `ActivityTaskScheduled` | `ActivityTaskStarted`, `ActivityTaskCompleted` |
| `StartTimer`           | `TimerStarted`          | `TimerFired`                                   |

## How History Replay provides Durable Execution 

Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look
at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of
a failure.

This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and
Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of
the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed.

```typescript
const { sendBill, getDistance } = proxyActivities<typeof activities>({
  startToCloseTimeout: '5 seconds',
});

export async function pizzaWorkflow(order: Order): Promise<string> {
  let distance: Distance | undefined = undefined;
  let totalPrice = 0;

  // compute distance
  distance = await getDistance(order.address);

  if (distance.kilometers > 25) {
    throw new ApplicationFailure('Customer too far away for delivery');
  }

  // Iterate over the items and calculate the cost of the order
  for (const pizza of order.items) {
    totalPrice += pizza.price;
  }

  // Wait 30 minutes before billing the customer
  await sleep('30 minutes');

  const bill = {
    customerID: order.customer.customerID,
    orderNumber: order.orderNumber,
    amount: totalPrice,
    description: 'Pizza',
  };

  const confirmation = await sendBill(bill);

  return confirmation;
}
```
### Original execution

#### Step 1: A Client requests the Workflow Execution

*Temporal Service*

The walkthrough begins with a request to execute this Workflow Definition with input data about the customer and the pizzas ordered. The Temporal Service records WorkflowExecutionStarted, always the first Event of a Workflow Execution, and that Event holds the input data.

**Event History:** `WorkflowExecutionStarted` (customer, pizzas ordered)

#### Step 2: A Workflow Task is queued

*Temporal Service*

The Temporal Service adds a Workflow Task to the Task Queue and records a WorkflowTaskScheduled Event.

**Event History:** `WorkflowTaskScheduled`

#### Step 3: A Worker accepts the Workflow Task

*Temporal Service*

The Temporal Service dispatches the Task to a Worker that is polling the Task Queue. The Worker accepts it, and the Temporal Service records a WorkflowTaskStarted Event.

**Event History:** `WorkflowTaskStarted`

#### Step 4: The Worker runs the Workflow code

*Internal step*

The Worker invokes the Workflow code and runs it one statement at a time. The first few lines are internal steps that do not interact with the Temporal Service.

#### Step 5: The Worker completes the Workflow Task

*Sends a Command*

The Worker encounters a request to execute an Activity, so it completes the current Workflow Task. It makes a single gRPC call, RespondWorkflowTaskCompleted, that signals completion of the Workflow Task and carries any Commands with it, so WorkflowTaskCompleted and the Command that follows are technically one call.

**Event History:** `WorkflowTaskCompleted`

#### Step 6: The GetDistance Activity Task is scheduled

*Sends a Command*

In response to the ScheduleActivityTask Command, the Temporal Service queues an Activity Task and records an ActivityTaskScheduled Event, a direct result of the Command.

**Commands:** `ScheduleActivityTask` (GetDistance)

**Event History:** `ActivityTaskScheduled` (GetDistance)

#### Step 7: A Worker starts the Activity Task

*Temporal Service*

The Temporal Service dispatches the Activity Task to an available Worker, which starts running the code in the GetDistance Activity. ActivityTaskStarted is an indirect result of the Command, and it is not written to the Event History until the Task closes, because the number of retry attempts is an attribute of that Event.

**Event History:** `ActivityTaskStarted`

#### Step 8: The Activity returns a distance of 15

*Temporal Service*

When the Activity returns, the Worker notifies the Temporal Service that the Activity Execution is complete. The Temporal Service records an ActivityTaskCompleted Event, which contains the result of the Activity.

**Event History:** `ActivityTaskCompleted` (distance = 15)

#### Step 9: The result goes back to the Workflow

*Temporal Service*

To deliver the result to the Workflow, the Temporal Service creates another Workflow Task that includes it. A Worker dequeues that Task, resumes execution, and evaluates the distance. This order is going to a nearby customer, so execution continues.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 10: The Worker starts a Timer

*Sends a Command*

The Worker reaches the request to start a Timer, so it completes the current Workflow Task and sends the StartTimer Command with it. The Temporal Service starts the Timer and records TimerStarted. The Workflow does not progress until the Timer fires.

**Commands:** `StartTimer` (30 minutes)

**Event History:**
- `WorkflowTaskCompleted`
- `TimerStarted` (30 minutes)

#### Step 11: The Timer fires

*Temporal Service*

After 30 minutes elapse, the Timer fires and the Temporal Service records TimerFired. It then queues a new Workflow Task to deliver that Event to the Workflow and drive progress forward.

**Event History:**
- `TimerFired`
- `WorkflowTaskScheduled`

### Worker crash

#### Step 12: The Worker crashes

*Worker crash*

The Worker polls for the Task, dequeues it, and continues executing the Workflow code — and then crashes right here. How does Temporal recover the state of this Workflow Execution? First, how does the Temporal Service know the Worker crashed?

**Event History:** `WorkflowTaskStarted`

#### Step 13: The Workflow Task times out

*Worker crash*

Once a Worker accepts a Task, it is expected to complete it within a predefined duration. That Workflow Task Timeout, 10 seconds by default, is how a crashed Worker is recognized. The Worker failed to complete the Task in time, so the Temporal Service records WorkflowTaskTimedOut.

**Event History:** `WorkflowTaskTimedOut`

#### Step 14: A new Workflow Task is scheduled

*Temporal Service*

The Temporal Service schedules a new Workflow Task. The Worker that polls for it might be another Worker in the fleet or a new Worker process created by restarting the one that crashed. Everything above the line is the Event History as it stood when the Worker crashed.

**Event History:**
- `WorkflowTaskScheduled`
- *— Event History at the time of the crash —*

### History Replay

#### Step 15: The Worker requests the Event History

*Replay*

Either way, the Worker needs the current Event History for this Workflow Execution, so it requests it from the Temporal Service. The Temporal Service provides the History.

#### Step 16: Replay re-executes the code

*Replay*

The Worker begins re-executing the code with the same input, which was stored in the WorkflowExecutionStarted Event. Because the Workflow Definition is deterministic, the state of every variable encountered so far is identical to what it was before the crash. For example, totalPrice holds the same value.

#### Step 17: The Activity result comes from the History

*Replay*

When Replay reaches the call to schedule GetDistance, it creates a ScheduleActivityTask Command but does not issue it. Instead, the Worker inspects the Event History and finds ActivityTaskScheduled for this Activity Type, ActivityTaskStarted showing a Worker dequeued the Task, and ActivityTaskCompleted with a result of 15. The Activity already ran, so the Command is not issued.

**Commands:** ✓ `ScheduleActivityTask` (GetDistance) — created during Replay, not issued

#### Step 18: Replay assigns the stored result

*Replay*

The Worker assigns the value stored in the ActivityTaskCompleted Event, 15, to the distance variable. The Activity is not re-executed, so there is no way for it to behave differently during Replay than it did during the original execution. The conditional evaluates to false, just as it did before.

#### Step 19: The Timer is already done

*Replay*

The Worker reaches the request to start a Timer and creates a StartTimer Command, which again it does not issue. The Event History contains TimerStarted and TimerFired, so the Worker knows the Timer already started and fired during the previous execution.

**Commands:** ✓ `StartTimer` (30 minutes) — created during Replay, not issued

#### Step 20: State is restored to the point of the crash

*Replay*

The Worker has reached the point where the crash occurred, and replaying the code has completely restored the state of the Workflow Execution. Every variable, including totalPrice, holds the value it held before the crash.

### Execution resumes

#### Step 21: Execution moves past the crash

*Sends a Command*

The Worker reaches a statement beyond where the crash occurred, which is evident because the Event History contains no Events for the SendBill Activity. Execution continues as if the crash never happened: the Worker completes the current Workflow Task and includes the Command with it.

**Event History:** `WorkflowTaskCompleted`

#### Step 22: The SendBill Activity runs

*Sends a Command*

The Worker issues the Command, the Temporal Service queues the Activity Task, and a Worker dequeues and runs it. When the Activity returns, the Worker notifies the Temporal Service, which records ActivityTaskCompleted with the result from SendBill.

**Commands:** `ScheduleActivityTask` (SendBill)

**Event History:**
- `ActivityTaskScheduled` (SendBill)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 23: One more Workflow Task delivers the result

*Temporal Service*

The Temporal Service has not received a Command saying the Workflow Execution completed or failed, so it schedules another Workflow Task to continue progress. A Worker accepts it and resumes the Workflow code.

**Event History:**
- `WorkflowTaskScheduled`
- `WorkflowTaskStarted`

#### Step 24: The Workflow Execution completes

*Sends a Command*

The Workflow returns, so the Worker completes the current Workflow Task and issues a CompleteWorkflowExecution Command that contains the result. The Temporal Service records WorkflowExecutionCompleted as the final Event. The result is identical to an execution that never crashed.

**Commands:** `CompleteWorkflowExecution` ({ ConfirmationNumber: "TPD-26074139" })

**Event History:**
- `WorkflowTaskCompleted`
- `WorkflowExecutionCompleted`

The walkthrough covers four phases:

1. **Original execution**: the Client starts the Workflow Execution, and the Commands the Worker issues become Events in
   the Event History.
2. **Worker crash**: the Worker dies partway through a Workflow Task. When the Workflow Task Timeout elapses, 10 seconds
   by default, the Temporal Service records `WorkflowTaskTimedOut` and schedules a new Workflow Task.
3. **History Replay**: a Worker requests the Event History and re-executes the Workflow code with the original input,
   which the `WorkflowExecutionStarted` Event stores. Commands the Worker creates during Replay are matched against the
   Event History instead of being issued, so Activities don't run again. The Worker uses the results stored in the
   `ActivityTaskCompleted` Events.
4. **Execution resumes**: past the point of the crash, the Event History holds no matching Events, so the Worker issues
   Commands for real again until the Workflow Execution completes. The result is identical to an execution that never
   crashed.

## Example of a non-deterministic Workflow Definition 

Now that Replay has been covered, this section will explain why a Workflow Definition needs to be
[deterministic](/workflow-definition#deterministic-constraints) in order for Replay to work.

A Workflow Definition is deterministic if every execution of it produces the same Commands in the same
sequence given the same input.

As mentioned in the [`How History Replay Provides Durable Execution`](#How-History-Replay-Provides-Durable-Execution)
walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs
the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event
History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History
and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue.

To better understand why a Workflow Definition needs to be deterministic, it's helpful to look at one that violates
it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random
number generator.

```typescript
const { importSalesData, runDailyReport } = proxyActivities<typeof activities>({
  startToCloseTimeout: '45 minutes',
});

export async function generateDailyReport(): Promise<void> {
  await importSalesData();

  if (getRandomNumber(1, 100) >= 50) {
    await sleep('4 hours');
  }

  log.info('Preparing to run daily report', {});

  await runDailyReport();
}
```
### First execution

#### Step 1: The importSalesData Activity runs

*Sends a Command*

As this Workflow executes step by step, the first line that results in a Command is the call to the importSalesData Activity. The Activity Execution succeeds, so the Temporal Service logs three Events to the Event History.

**Commands created:** `ScheduleActivityTask` (importSalesData)

**Relevant History Events:**
- `ActivityTaskScheduled` (importSalesData)
- `ActivityTaskStarted`
- `ActivityTaskCompleted`

#### Step 2: A random number decides the next branch

*Internal step*

The Worker reaches a conditional statement that evaluates a randomly generated number. The random number generator returns 84 during this execution, so the expression evaluates to true and execution continues with the next line.

#### Step 3: The Workflow starts a 4-hour Timer

*Sends a Command*

The next line requests a Timer, so the Worker issues a StartTimer Command. The Temporal Service starts the Timer and records TimerStarted, then records TimerFired when the Timer fires.

**Commands created:** `StartTimer` (4 hours)

**Relevant History Events:**
- `TimerStarted` (4 hours)
- `TimerFired`

#### Step 4: The Worker crashes

*Worker crash*

The Worker crashes once it reaches the next line, so another Worker takes over. That Worker uses Replay to restore the current state before continuing execution of the lines that follow.

### History Replay

#### Step 5: Replay establishes the expected Commands

*Replay*

The Worker requests the Event History and determines the sequence of Commands needed to restore the current state. Based on the History, it expects to encounter ScheduleActivityTask (importSalesData) and then StartTimer (4 hours).

#### Step 6: The first Command matches

*Replay*

As the Worker executes the code during Replay, it reaches the first call to execute an Activity and creates a ScheduleActivityTask Command. It is the right type of Command and it occurs at the right position in the expected sequence, so Replay proceeds.

**Commands created:** ✓ `ScheduleActivityTask` (importSalesData) — created during Replay

#### Step 7: The random number returns something different

*Replay*

The Worker reaches the conditional statement again. This time the random number generator returns 14, so the expression evaluates to false and execution skips the call that starts the Timer.

#### Step 8: The next Command does not match

*Worker crash*

The Worker reaches the request to execute the runDailyReport Activity and creates another ScheduleActivityTask Command. That is not the Command it expected at this position in the sequence, so the Worker cannot restore the previous state.

**Commands created:** ✗ `ScheduleActivityTask` (runDailyReport) — created during Replay (expected: StartTimer (4 hours))

#### Step 9: Replay fails with a non-determinism error

*Worker crash*

The Workflow produced a different sequence of Commands during Replay than the Event History recorded before the crash, so the Workflow Execution cannot be replayed. The random number generator is the source of the non-determinism.

During the first execution, the random number is 84, so the Workflow starts a Timer and the Event History records
`TimerStarted` and `TimerFired`. During Replay, the random number is 14, so the Workflow skips the Timer and produces a
`ScheduleActivityTask` Command where the Event History expects `StartTimer`. That mismatch is what makes Replay fail.

Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is
considered a [Workflow Task Failure](/references/failures#workflow-task-failures) which is considered a transient
failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the
Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a
strategy called versioning to address this non-determinism error. See
[versioning](/develop/typescript/workflows/versioning) to learn more.

For more information on how Temporal handles Durable Execution or to see these walkthroughs in video format with more
explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and
[Versioning Workflows](https://learn.temporal.io/courses/versioning/).

## Temporal Applications support non-deterministic operations

We want to emphasize that although your Workflow Definition needs to be deterministic, your application itself does not!

Remember that pretty much anything that interacts with the external world is inherently non-deterministic:

- Calling LLM APIs
- Querying databases
- Reading or writing files
- Making HTTP requests to external services

**Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow Definition must be
deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed
above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities
which have built-in support for retries.

---

# Extensibility

Source: https://docs.temporal.io/encyclopedia/extensibility

> Temporal offers mechanisms to augment the functionality of Workflows and Activities, including data conversion, context propagation, interceptors, and plugins.

Temporal offers many mechanisms to augment the functionality of Workflows and Activities.
These allow you to customize how data is serialized, propagate metadata across execution boundaries, and inject cross-cutting behavior like tracing and logging.

- [Data Conversion](/dataconversion) - Customize how arguments and return values are serialized, compressed, or encrypted
- [Context Propagation](/encyclopedia/context-propagation) - Pass custom metadata (tracing IDs, tenant IDs, auth tokens) across Workflow, Activity, and Child Workflow boundaries
- [Interceptors](/encyclopedia/interceptors) - Add cross-cutting behavior (observability, authorization, header manipulation) before and after SDK operations
- [Plugins](/encyclopedia/plugins) - Bundle interceptors, context propagators, data converters, and built-in definitions into reusable packages

---

# Failures and error handling

Source: https://docs.temporal.io/encyclopedia/failures-and-error-handling

> Learn how Temporal represents and handles failures, from platform-level retries to application-level error handling strategies.

Temporal provides Durable Execution, which guarantees that your application runs to completion despite infrastructure issues, network problems, and Worker crashes.
However, not all failures are handled automatically by the platform.
Understanding the difference between platform-level and application-level failures is key to building reliable Temporal applications.

This section covers:

- **[Application failures](/encyclopedia/application-failures)**: What application failures are, how Temporal represents them, and how they propagate through your code.
- **[Detecting Activity failures](/encyclopedia/detecting-activity-failures)**: How to use timeouts and Heartbeats to detect when Activities fail or become unresponsive.
- **[Detecting Workflow failures](/encyclopedia/detecting-workflow-failures)**: How Workflow-level timeouts work and when they apply.
- **[Retry Policies](/encyclopedia/retry-policies)**: How to configure automatic retry behavior for Activities and Workflows.

For prescriptive guidance on building an error handling strategy, see [Error handling](/best-practices/error-handling).

---

# Interceptors

Source: https://docs.temporal.io/encyclopedia/interceptors

> Add cross-cutting behavior like observability, authorization, and header manipulation before and after SDK operations.

Interceptors let you add cross-cutting behavior before and after SDK operations such as starting a Workflow, executing an Activity, or handling a Signal. They work like middleware: each interceptor wraps the next, forming a chain that executes around the underlying operation.

Common use cases:

- Observability (logging, metrics, tracing)
- Authorization and authentication checks
- Header manipulation (propagating metadata)
- Input/output validation

## Implementing Interceptors

Here are SDK-specific guides:

- [Go](/develop/go/workers/interceptors)
- [Java](/develop/java/workers/interceptors)
- [Python](/develop/python/workers/interceptors)
- [TypeScript](/develop/typescript/workers/interceptors)
- [.NET](/develop/dotnet/workers/interceptors)

---

# Plugins

Source: https://docs.temporal.io/encyclopedia/plugins

> Bundle interceptors, context propagators, data converters, and built-in definitions into reusable packages.

A Plugin bundles multiple extensibility primitives - interceptors, context propagators, data converters, and built-in Workflow/Activity/Nexus definitions - into a single reusable package. Plugins let platform teams and library authors ship ready-made functionality that application developers can adopt with a single registration call.

Common use cases:

- AI Agent SDKs (for example, OpenAI Agents, Pydantic AI, LangGraph)
- Observability packages (tracing, logging, metrics)
- Encryption or compliance middleware
- Shared infrastructure integrations (messaging, payments, LLM calls)

## Implementing Plugins

See the [Plugins guide](/develop/plugins-guide) for how to build and use plugins across all supported SDKs.

---

# What is a Temporal Retry Policy?

Source: https://docs.temporal.io/encyclopedia/retry-policies

A Retry Policy is a collection of settings that tells Temporal how and when to try again after something fails in a Workflow Execution or Activity Task Execution.

## Overview

Temporal's default behavior is to automatically retry an Activity that fails, so transient or intermittent failures require no action on your part. This behavior is defined by the Retry Policy.

A Retry Policy is declarative. You do not need to implement your own logic for handling the retries; you only need to specify the desired behavior and Temporal will provide it.

In contrast to Activities, a Workflow Execution itself is not associated with a Retry Policy by default. This may seem counterintuitive, but Workflows and Activities perform different roles. Activities are intended for operations that may fail, so having a default Retry Policy increases the likelihood that they will ultimately complete successfully, even if the initial attempt failed.

On the other hand, a Workflow Definition must be deterministic to support replay, and failure-prone or non-deterministic operations (API calls, LLM invocations, etc.) should be placed in Activities, which have built-in retry support. While it is possible to assign a Retry Policy to a Workflow Execution,
this is not the default and it is uncommon to do so.

Retry Policies do not apply to Workflow Task Executions, which retry until the Workflow Execution Timeout (which is
unlimited by default) with an exponential backoff and a max interval of 10 minutes. A Retry Policy instructs the
Temporal Service how to retry a failure of either a [Workflow Execution](/workflow-execution) or an
[Activity Task Execution](/tasks#activity-task-execution).

Try out the [Activity retry simulator](/demos/activity-retry-simulator) to visualize how a Retry Policy works.

---

**Related:**

- [Set a custom Retry Policy for an Activity in Go](/develop/go/activities/timeouts#activity-retries)
- [Set a custom Retry Policy for an Activity in Java](/develop/java/activities/timeouts#activity-retries)
- [Set a custom Retry Policy for an Activity in PHP](/develop/php/activities/timeouts#activity-retries)
- [Set a custom Retry Policy for an Activity in Python](/develop/python/activities/timeouts#activity-retries)
- [Set a custom Retry Policy for an Activity in Ruby](/develop/ruby/activities/timeouts#activity-retries)
- [Set a custom Retry Policy for an Activity in Rust](/develop/rust/activities/timeouts#activity-retries)
- [Set a custom Retry Policy for an Activity in TypeScript](/develop/typescript/activities/timeouts#activity-retries)

---

**Related:**

- [Set a Retry Policy for a Workflow in Go](/develop/go/workflows/timeouts#workflow-retries)
- [Set a Retry Policy for a Workflow in Java](/develop/java/workflows/timeouts#workflow-retries)
- [Set a Retry Policy for a Workflow in PHP](/develop/php/workflows/timeouts#workflow-retries)
- [Set a Retry Policy for a Workflow in Python](/develop/python/workflows/timeouts#workflow-retries)
- [Set a Retry Policy for a Workflow in Ruby](/develop/ruby/workflows/timeouts#workflow-retries)
- [Set a Retry Policy for a Workflow in Rust](/develop/rust/workflows/timeouts#workflow-retries)
- [Set a Retry Policy for a Workflow in TypeScript](/develop/typescript/workflows/timeouts#workflow-retries)

## Default behavior

Activities in Temporal are associated with a Retry Policy by default, while Workflows are not. The Temporal SDK provides
a Retry Policy instance with default behavior. While this object is not specific to either a Workflow or Activity,
you'll use different methods to apply it to the execution of each.

This section details the default retry behavior for both Activities and Workflows to provide context for any further
customization.

### Activity Execution

Temporal's default behavior is to automatically retry an Activity, with a short delay between each attempt that
increases exponentially, until it either succeeds or is canceled. When a subsequent attempt succeeds, execution
continues normally: a Workflow that called the Activity resumes as if the failure never occurred, and for a
[Standalone Activity](/standalone-activity), the Activity Execution completes and returns its result to the Client.

When an Activity Task Execution is retried, the Temporal Service places a new [Activity Task](/tasks#activity-task) into
its respective [Activity Task Queue](/task-queue), which results in a new Activity Task Execution.

The default Retry Policy uses exponential backoff with a 2.0 backoff coefficient, starting with a 1-second initial
interval and capping at a maximum interval of 100 seconds. By default, the maximum attempt of retries are set to zero
which is evaluated as unlimited and non-retryable errors default to none. For detailed information about all Retry
Policy attributes and their default values, see the [Properties](#properties) section.

These defaults are the same whether the Activity is a Workflow Activity or a [Standalone Activity](/standalone-activity)
— a Retry Policy is a property of the Activity Task Execution itself, not of what started it.

### Workflow Execution

Unlike Activities, Workflow Executions do not retry by default. When a Workflow Execution is spawned, it is not
associated with a default Retry Policy and thus does not retry by default.

Temporal provides guidance around idempotence of Activity code with the expectation that Activities will need to
re-execute upon failure; this is not typically true of Workflows. In most use cases, a Workflow failure would indicate
an issue with the design or deployment of your application; for example, a permanent failure that may require different
input data.

Retrying an entire Workflow Execution is not recommended due to the deterministic nature of Workflow replay. Since Workflows replay the same sequence of events to reach the same state, retrying the whole Workflow would repeat the same logic without resolving the underlying issue that caused the failure. This repetition doesn't address problems related to external dependencies or unchanged conditions and can lead to unnecessary resource consumption and higher costs.

Instead, retry failed Activities within the Workflow, which is Temporal's default behavior. This approach targets specific points of failure, allowing the Workflow to progress without redundant operations, thereby saving on resources and ensuring a more focused and effective error recovery process. If you need to retry parts of your Workflow Definition, we recommend you implement this in your Workflow code.

## Custom Retry Policy

To use a custom Retry Policy, provide it as an options parameter when starting a Workflow Execution or Activity
Execution. Only certain scenarios merit starting a Workflow Execution with a custom Retry Policy, such as the following:

- A [Temporal Cron Job](/cron-job) or some other stateless, always-running Workflow Execution that can benefit from
  retries.
- A file-processing or media-encoding Workflow Execution that downloads files to a host.

## Properties

### Default values for Retry Policy

```
Initial Interval     = 1 second
Backoff Coefficient  = 2.0
Maximum Interval     = 100 × Initial Interval
Maximum Attempts     = ∞
Non-Retryable Errors = []
```

### Initial Interval

- **Description:** Amount of time that must elapse before the first retry occurs.
  - **The default value is 1 second.**
- **Use case:** This is used as the base interval time for the [Backoff Coefficient](#backoff-coefficient) to multiply
  against.

### Backoff Coefficient

- **Description:** The value dictates how much the _retry interval_ increases.
  - **The default value is 2.0.**
  - A backoff coefficient of 1.0 means that the retry interval always equals the [Initial Interval](#initial-interval).
- **Use case:** Use this attribute to increase the interval between retries. By having a backoff coefficient greater
  than 1.0, the first few retries happen relatively quickly to overcome intermittent failures, but subsequent retries
  happen farther and farther apart to account for longer outages. Use the [Maximum Interval](#maximum-interval)
  attribute to prevent the coefficient from increasing the retry interval too much.

### Maximum Interval

- **Description:** Specifies the maximum interval between retries.
  - **The default value is 100 times the [Initial Interval](#initial-interval).**
- **Use case:** This attribute is useful for [Backoff Coefficients](#backoff-coefficient) that are greater than 1.0
  because it prevents the retry interval from growing infinitely.

### Maximum Attempts

- **Description:** Specifies the maximum number of execution attempts that can be made in the presence of failures.
  - **The default is unlimited.**
  - If this limit is exceeded, the execution fails without retrying again. When this happens an error is returned.
  - Setting the value to 0 also means unlimited.
  - Setting the value to 1 means a single execution attempt and no retries.
  - Setting the value to a negative integer results in an error when the execution is invoked.
- **Use case:** Use this attribute to ensure that retries do not continue indefinitely. In most cases, we recommend
  using the Workflow Execution Timeout for [Workflows](/workflows) or the Schedule-To-Close Timeout for Activities to
  limit the total duration of retries, rather than using this attribute.

### Non-Retryable Errors 

Non-Retryable Errors specify errors that shouldn't be retried. By default, none are specified. Errors are matched
against the `type` field of the [Application Failure](/references/failures#application-failure). If one of those errors
occurs, a retry does not occur. If you know of errors that should not trigger a retry, you can specify that and if they
occur, the execution is not retried.

#### Non-Retryable Errors for Activities

When writing software applications, you will encounter three types of failures: transient, intermittent, and permanent.
While transient and intermittent failures may resolve themselves upon retrying without further intervention, permanent
failures will not. Permanent failures, by definition, require you to make some change to your logic or your input.
Therefore, it is better to surface them than to retry them.

Non-Retryable Errors are errors that will not be retried, regardless of a Retry Policy.

**Ruby**

To raise a non-retryable error, specify the `non_retryable` flag when raising an `ApplicationError`:

```ruby
raise Temporalio::Error::ApplicationError.new(
  "Invalid credit card number: #{credit_card_number}",
  type: 'InvalidChargeAmount',
  non_retryable: true
)
```

This will designate the `ApplicationError` as non-retryable.

**Python**

To raise a non-retryable error, specify the `non_retryable` flag when raising an `ApplicationError`:

```python
raise ApplicationError(
    f"Invalid credit card number: {credit_card_number}",
    type="InvalidChargeAmount",
    non_retryable=True,
)
```

This will designate the `ApplicationError` as non-retryable.

**TypeScript**

To throw a non-retryable error, add `nonRetryable: true` to `ApplicationFailure.create({})`:

```typescript
throw ApplicationFailure.create({
  message: `Invalid charge amount: ${chargeAmount} (must be above zero)`,
  details: [chargeAmount],
  nonRetryable: true,
});
```

This will designate the Error as non-retryable.

**Java**

To throw a non-retryable error, use the `newNonRetryableFailure` method:

```java
throw ApplicationFailure.newNonRetryableFailure(
    "Invalid credit card number: " + creditCardNumber,
    InvalidChargeAmountException.class.getName()
);
```

This will designate the `ApplicationFailure` as non-retryable.

**Go**

To return a non-retryable error, replace your call to `NewApplicationError()` with `NewNonRetryableApplicationError()`:

```go
temporal.NewNonRetryableApplicationError("Credit Card Charge Error", "CreditCardError", nil, nil)
```

This will designate the Error as non-retryable.

**.NET**

To throw a non-retryable error, specify the `nonRetryable` flag when throwing an `ApplicationFailureException`:

```csharp
var attempt = ActivityExecutionContext.Current.Info.Attempt;

throw new ApplicationFailureException(
    $"Something bad happened on attempt {attempt}",
    errorType: "my_failure_type",
    nonRetryable: true
);
```

This will designate the `ApplicationFailureException` as non-retryable.

Use non-retryable errors in your code sparingly.

**Ruby**

If you do not specify the failure as non-retryable within the definition, you can always mark that error type as
non-retryable in your Activity's Retry Policy, but an `ApplicationError` with the `non_retryable` keyword argument set
to `true` will always be non-retryable.

**Python**

If you do not specify the failure as non-retryable within the definition, you can always mark that error type as
non-retryable in your Activity's Retry Policy, but an `ApplicationError` with the `non_retryable` keyword argument set
to `True` will always be non-retryable.

**TypeScript**

If you do not specify the failure as non-retryable within the definition, you can always mark that error type as
non-retryable in your Activity's Retry Policy, but an error with `nonRetryable: true` set will always be non-retryable.

**Java**

If you throw a regular `newFailure()`, you can always mark that error _type_ as non-retryable in your Activity's Retry
Policy, but a `newNonRetryableFailure()` will always be non-retryable.

**Go**

If you return a regular `NewApplicationError()`, you can always mark that error _type_ as non-retryable in your
Activity's Retry Policy, but a `NewNonRetryableApplicationError()` will always be non-retryable.

**.NET**

If you do not specify the failure as non-retryable within the definition, you can always mark that error type as
non-retryable in your Activity's Retry Policy, but an `ApplicationFailureException` with the `nonRetryable` parameter
set to `true` will always be non-retryable.

For example, checking for bad input data is a reasonable time to use a non-retryable error. If the Activity cannot
proceed with the input it has, that error should be surfaced immediately so that the input can be corrected on the next
attempt.

If responsibility for your application is distributed across multiple maintainers, or if you are developing a library to
integrate into somebody else's application, you can think of the decision to hardcode non-retryable errors as following
a "caller versus implementer" dichotomy. Anyone who is calling your Activity would be able to make decisions about their
Retry Policy, but only the implementer can decide whether an error should never be retryable out of the box.

## Retry interval

The wait time before a retry is the _retry interval_. A retry interval is the smaller of two values:

- The [Initial Interval](#initial-interval) multiplied by the [Backoff Coefficient](#backoff-coefficient) raised to the
  power of the number of retries.
- The [Maximum Interval](#maximum-interval).

![Diagram that shows the retry interval and its formula](https://docs.temporal.io/img/info/retry-interval-diagram.png)

### Per-error next Retry delay

Sometimes, your Activity or Workflow raises a special exception that needs a different retry interval from the Retry
Policy. To accomplish this, you may throw an [Application Failure](/references/failures#application-failure) with the
next Retry delay field set. This value will replace and override whatever the retry interval would be on the Retry
Policy. Note that your retries will still cap out under the Retry Policy's Maximum Attempts, as well as overall
timeouts. For an Activity, its Schedule-to-Close Timeout applies. For a Workflow, the Execution Timeout applies.

**Related:**

- [Customize retry delays per error in the Java SDK.](/develop/java/activities/timeouts#activity-next-retry-delay)
- [Customize retry delays per error in the Rust SDK](/develop/rust/activities/timeouts#next-retry-delay)
- [Customize retry delays per error in the TypeScript SDK](/develop/typescript/activities/timeouts#activity-next-retry-delay)

## Event History

There are some subtle nuances to how Events are recorded to an Event History when a Retry Policy comes into play.

- For an Activity Execution, the [ActivityTaskStarted](/references/events#activitytaskstarted) Event will not show up in
  the Workflow Execution Event History until the Activity Execution has completed or failed (having exhausted all
  retries). This is to avoid filling the Event History with noise. Use the Describe API to get a pending Activity
  Execution's attempt count.

- When a Workflow fails and has a Retry Policy, the failed run ends with `WorkflowExecutionFailed`, with
  `retryState=IN_PROGRESS` and `newExecutionRunId` set, and the Temporal Service starts a new Workflow Execution. The
  new Workflow Execution is created immediately, but the first Workflow Task won’t be scheduled until the backoff
  duration is exhausted. That duration is recorded as the `first_workflow_task_backoff` field on the new run’s
  `WorkflowExecutionStartedEventAttributes`.

---

# Temporal Client

Source: https://docs.temporal.io/encyclopedia/temporal-client

> A Temporal Client acts as the bridge for communication between your applications and the Temporal Service, enabling you to start Workflow Executions, send Signals and Queries, and retrieve results.

A Temporal Client allows you to communicate with the [Temporal Service](/temporal-service).

The most common operations that a Temporal Client allows you to perform are the following:
- Start a Workflow Execution
- Get the result of Workflow Execution
- List Workflow Executions
- Query a Workflow Execution
- Signal a Workflow Execution
- Start and manage [Standalone Activities](/standalone-activity) directly, without involving a Workflow
- Start a [Standalone Nexus Operation](/standalone-nexus-operation) directly, without using a caller Workflow

A Standalone Activity is a top-level [Activity Execution](/activity-execution) started directly by a Client, without using a Workflow.
A Standalone Nexus Operation is a top-level [Nexus Operation Execution](/nexus/operations) started directly by a Client, without using a caller Workflow.

## SDK guides 

- [Temporal Client - Go](/develop/go/client/temporal-client)
- [Temporal Client - Java](/develop/java/client/temporal-client)
- [Temporal Client - PHP](/develop/php/client/temporal-client)
- [Temporal Client - Python](/develop/python/client/temporal-client)
- [Temporal Client - Ruby](/develop/ruby/client/temporal-client)
- [Temporal Client - Rust](/develop/rust/client/temporal-client)
- [Temporal Client - TypeScript](/develop/typescript/client/temporal-client)
- [Temporal Client - .NET](/develop/dotnet/client/temporal-client)

---

# Worker versioning (legacy)

Source: https://docs.temporal.io/encyclopedia/worker-versioning-legacy

> Remember how to use the now-deprecated pre-release version of Worker Versioning

> **💡 Tip:**
> Support, stability, and dependency info
>
> - This document refers to the 2023 draft of Worker Versioning, which was deprecated
> - It was not made available in Temporal Cloud
> - The 2024 draft was available in Cloud on an opt-in basis, and is documented in this [Pre-release README.md](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md).
>
> For newer revisions of this feature set, please see [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) instead.
>

Worker Versioning simplifies the process of deploying changes to [Workflow Definitions](/workflow-definition).
It does this by letting you define sets of versions that are compatible with each other, and then assigning a Build ID to the code that defines a Worker.
The Temporal Server uses the Build ID to determine which versions of a Workflow Definition a Worker can process.

We recommend that you read about Workflow Definitions before proceeding, because Workflow Versioning is largely concerned with helping to manage nondeterministic changes to those definitions.

Worker Versioning helps manage nondeterministic changes by providing a convenient way to ensure that [Workers](/workers) with different Workflow and Activity Definitions operating on the same Task Queue don't attempt to process [Workflow Tasks](/tasks#workflow-task) and [Activity Tasks](/tasks#activity-task-execution) that they can't successfully process, according to sets of versions associated with that Task Queue that you've defined.

Accomplish this goal by assigning a Build ID (a free-form string) to the code that defines a Worker, and specifying which Build IDs are compatible with each other by updating the version sets associated with the Task Queue, stored by the Temporal Server.

## When and why you should use Worker Versioning

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

The main reason to use this feature is to deploy incompatible changes to short-lived [Workflows](/workflows).
On Task Queues using this feature, the Workflow starter doesn't have to know about the introduction of new versions.

The new code in the newly deployed Workers executes new [Workflow Executions](/workflow-execution), while only Workers with an appropriate version process old Workflow Executions.

### Decommission old Workers

You can decommission old Workers after you archive all open Workflows using their version.
If you have no need to query closed Workflows, you can decommission them when no open Workflows remain at that version.

For example, if you have a Workflow that completes within a day, a good strategy is to assign a new Build ID to every new Worker build and add it as the new overall default in the version sets.

Because your Workflow completes in a day, you know that you won't need to keep older Workers running for more than a day after you deploy the new version (assuming availability).
You can apply this technique to longer-lived Workflows too; however, you might need to run multiple Worker versions simultaneously while open Workflows complete.

Version sets have a maximum size limit, which defaults to 100 Build IDs across all sets.
Operations to add new Build IDs to the sets will fail if they exceed this limit.
There is also a limit on the number of Version Sets, which defaults to 10.
A version can only be garbage collected after a Workflow Execution is deleted.

### Deploy code changes to Workers

The feature also lets you implement compatible changes to or prevent a buggy code path from executing on currently open Workflows.
You can achieve this by adding a new version to an existing set and defining it as _compatible_ with an existing version, which shouldn't execute any future Workflow Tasks.
Because the new version processes existing [Event Histories](/workflow-execution/event#event-history), it must adhere to the usual [deterministic constraints](/workflow-definition#deterministic-constraints), and you might need to use one of the [versioning APIs](/workflow-definition#workflow-versioning).

Moreover, this feature lets you make incompatible changes to Activity Definitions in conjunction with incompatible changes to Workflow Definitions that use those Activities.
This functionality works because any Activity that a Workflow schedules on the same Task Queue gets dispatched by default only to Workers compatible with the Workflow that scheduled it.
If you want to change an Activity Definition's type signature while creating a new incompatible Build ID for a Worker, you can do so without worrying about the Activity failing to execute on some other Worker with an incompatible definition.
The same principle applies to Child Workflows.
For both Activities and Child Workflows, you can override the default behavior and run the Activity or Child Workflow on latest default version.

> **💡 Tip:**
>
> Public-facing Workflows on a versioned Task Queue shouldn't change their signatures because doing so contradicts the purpose of Workflow-launching Clients remaining unaware of changes in the Workflow Definition.
> If you need to change a Workflow's signature, use a different Workflow Type or a completely new Task Queue.
>

> **📝 Note:**
>
> If you schedule an Activity or a Child Workflow on _a different_ Task Queue from the one the Workflow runs on, the system doesn't assign a specific version.
> This means if the target queue is versioned, they run on the latest default, and if it's unversioned, they operate as they would have without this feature.
>

**Continue-As-New and Worker Versioning**

By default, a versioned Task Queue's Continue-as-New function starts the continued Workflow on the same compatible set as the original Workflow.

If you continue-as-new onto a different Task Queue, the system doesn't assign any particular version.
You also have the option to specify that the continued Workflow should start using the Task Queue's latest default version.

## How to use Worker Versioning

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. See [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

To use Worker Versioning, follow these steps:

1. Define Worker build-identifier version sets for the Task Queue.
   You can use either the Temporal CLI or your choice of SDK.
2. Enable the feature on your Worker by specifying a Build ID.

### Defining the version sets

Whether you use [Temporal CLI](/cli/) or an SDK, updating the version sets feels the same.
You specify the Task Queue that you're targeting, the Build ID that you're adding (or promoting), whether it becomes the new default version, and any existing versions it should be considered compatible with.

The rest of this section uses updates to one Task Queue's version sets as examples.

By default, both Task Queues and Workers are in an unversioned state.
[Unversioned Worker](#unversioned-workers) can poll unversioned Task Queues and receive tasks.
To use this feature, both the Task Queue and the Worker must be associated with Build IDs.

If you run a Worker using versioning against a Task Queue that has not been set up to use versioning (or is missing that Worker's Build ID), it won't get any tasks.
Likewise, an unversioned Worker polling a Task Queue with versioning won't work either.

> **📝 Note:**
> Versions don't need to follow semver or any other semantic versioning scheme!
>
> The versions in the following examples look like semver versions for clarity, but they don't need to be.
> Versions can be any arbitrary string.
>

First, add a version `1.0` to the Task Queue as the new default.
Your version sets now look like this:

| set 1 (default) |
| --------------- |
| 1.0 (default)   |

All new Workflows started on the Task Queue have their first tasks assigned to version `1.0`.
Workers with their Build ID set to `1.0` receive these Tasks.

If Workflows that don't have an assigned version are still running on the Task Queue, Workers without a version take those tasks.
So ensure that such Workers are still operational if any Workflows were open when you added the first version.
If you deployed any Workers with a _different_ version, those Workers receive no Tasks.

Now, imagine you need to change the Workflow for some reason.

Add `2.0` to the sets as the new default:

| set 1         | set 2 (default) |
| ------------- | --------------- |
| 1.0 (default) | 2.0 (default)   |

All new Workflows started on the Task Queue have their first tasks assigned to version `2.0`.
Existing `1.0` Workflows keep generating tasks targeting `1.0`.
Each deployment of Workers receives their respective Tasks.
This same concept carries forward for each new incompatible version.

Maybe you have a bug in `2.0`, and you want to make sure all open `2.0` Workflows switch to some new code as fast as possible.
So, you add `2.1` to the sets, marking it as compatible with `2.0`.
Now your sets look like this:

| set 1         | set 2 (default) |
| ------------- | --------------- |
| 1.0 (default) | 2.0             |
|               | 2.1 (default)   |

All new Workflow Tasks that are generated for Workflows whose last Workflow Task completion was on version `2.0` are now assigned to version `2.1`.
Because you specified that `2.1` is compatible with `2.0`, Temporal Server assumes that Workers with this version can process the existing Event Histories successfully.

Continue with your normal development cycle, adding a `3.0` version.
Nothing new here:

| set 1         | set 2         | set 3 (default) |
| ------------- | ------------- | --------------- |
| 1.0 (default) | 2.0           | 3.0 (default)   |
|               | 2.1 (default) |                 |

Now imagine that version `3.0` doesn't have an explicit bug, but something about the business logic
is less than ideal.
You are okay with existing `3.0` Workflows running to completion, but you want new Workflows to use the old `2.x` branch.
This operation is supported by performing an update targeting `2.1` (or `2.0`) and setting its set as the current default, which results in these sets:

| set 1         | set 3         | set 2 (default) |
| ------------- | ------------- | --------------- |
| 1.0 (default) | 3.0 (default) | 2.0             |
|               |               | 2.1 (default)   |

Now new Workflows start on `2.1`.

### Permitted and forbidden operations on version sets

A request to change the sets can do one of the following:

- Add a version to the sets as the new default version in a new overall-default compatible set.
- Add a version to an existing set that's compatible with an existing version.
  - Optionally making it the default for that set.
  - Optionally making that set the overall-default set.
- Promote a version within an existing set to become the default for that set.
- Promote a set to become the overall-default set.

You can't explicitly delete versions. This helps you avoid the situation in which Workflows accidentally become stuck with no means of making progress because the version they're associated with no longer exists.

However, sometimes you might want to do this intentionally.
If you _want_ to make sure that all Workflows currently being processed by, say, `2.0` stop (even if you don't yet have a new version ready), you can add a new version `2.1` to the sets marked as compatible with `2.0`.
New tasks will target `2.1`, but because you haven't deployed any `2.1` Workers, they won't make any progress.

### Set constraints

The sets have a maximum size limit, which defaults to 100 build IDs across all sets.
This limit is configurable on Temporal Server via the `limit.versionBuildIdLimitPerQueue` dynamic config property.
Operations to add new Build IDs to the sets fail if the limit would be exceeded.

There is also a limit on the number of sets, which defaults to 10.
This limit is configurable via the `limit.versionCompatibleSetLimitPerQueue` dynamic config property.

In practice, these limits should rarely be a concern because a version is no longer needed after no open Workflows are using that version, and a background process will delete IDs and sets that are no longer needed.

There is also a limit on the size of each Build ID or version string, which defaults to 255 characters.
This limit is configurable on the server via the `limit.workerBuildIdSize` dynamic config property.

## Build ID reachability

> **⚠️ Caution:**
>
> This section is for a deprecated Worker Versioning API. See [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
>

Eventually, you'll want to know whether you can retire the old Worker versions.
Temporal provides functionality to help you determine whether a version is still in use by open or closed Workflows.
You can use the Temporal CLI to do this with the following command:

```command
temporal task-queue get-build-id-reachability
```

The command determines, for each Task Queue, whether the Build ID in question is unreachable, only reachable by closed Workflows, or reachable by open and new Workflows.
For example, this "2.0" Build ID is shown here by the Temporal CLI to be reachable by both new Workflows and some existing Workflows:

```command
temporal task-queue get-build-id-reachability --build-id "2.0"
```

```output
BuildId                         TaskQueue                                   Reachability
    2.0  build-id-versioning-dc0068f6-0426-428f-b0b2-703a7e409a97  [NewWorkflows
                                                                   ExistingWorkflows]
```

For more information, see the [CLI documentation](/cli/) or help output.

You can also use this API `GetWorkerTaskReachability` directly from within language SDKs.

## Unversioned Workers

Unversioned Workers refer to Workers that have not opted into the Worker Versioning feature in their configuration.
They receive tasks only from Task Queues that do not have any version sets defined on them, or that have open Workflows that began executing before versions were added to the queue.

To migrate from an unversioned Task Queue, add a new default Build ID to the Task Queue.
From there, deploy Workers with the same Build ID.
Unversioned Workers will continue processing open Workflows, while Workers with the new Build ID will process new Workflow Executions.

---

# Worker Shutdown Behavior

Source: https://docs.temporal.io/encyclopedia/workers/worker-shutdown

> Learn how a Temporal Worker shuts down, and how graceful shutdown affects in-flight Workflow Tasks and Activities.

When a Worker shuts down, it stops polling for new tasks and begins the shutdown sequence.
In the case of in-flight Workflow Tasks, shutdown may cause them to fail if they aren’t completed in time, after exhausting Retry Policy attempts.

There are two types of shutdown behavior that can occur, depending on whether an idea of “graceful shutdown” is configured.

## Graceful shutdown

Graceful shutdown configures how much time a Worker has to complete its current task before shutting down.
An Activity is able to determine that the Worker it’s running on is being shut down, through the Activity context.

> Core SDKs - `graceful_shutdown_period`

> Go - `WorkerStopTimeout`

> Java - `shutdown()` followed by `awaitTermination(timeout, unit)`

In the Java SDK, Activities can't Heartbeat during graceful shutdown by default. `heartbeat()` throws `ActivityWorkerShutdownException` instead.
See [Heartbeat during shutdown](/develop/java/workers/run-worker-process#heartbeat-during-shutdown) to change this.

### Workflow tasks

Any in-flight Workflow Tasks are (attempted to be) completed.
The only reason they may not immediately, is if Workflow code is (incorrectly) blocking, or because of Local Activities (see below).

### Activities

Activities are allowed to complete during the graceful shutdown period.

### Local Activities

Because Local Activities run within a Workflow Task, current and future Local Activities within the same Workflow Task will be allowed to run and complete, assuming there is no additional command to yield to.

If the Local Activity is unable to complete by the graceful shutdown period, the Local Activity attempt is sent a cancel signal.
In this case, no new Local Activities will be retried or started, and the Worker is shut down.
The Worker still waits for the current Workflow Task to complete, meaning you can eventually hit your Workflow Task or execution timeout, unless another Worker is spun up.

## Non-graceful period shutdown

This behavior is for either no graceful period being specified, or if the shutdown has taken longer than the configured graceful period.
In all cases, the Activity context is canceled and the Worker will finish shutdown when the current Workflow Task completes (with either success or failure).

> **📝 Note:**
> Go and Core SDKs behave differently when we pass task timeout and the Activity or Local Activity is still running:
>
> **Go** - The shutdown completes, but the Activity will continue to run and use a slot.
>
> **Core** - The Worker shutdown will not complete while the Activity completes.

### Local Activities

The Local Activity is sent a cancel signal, then the Workflow Task heartbeats stop, and no new Local Activities will be retried or started.
The Worker still waits for the current Workflow Task to complete, meaning you can eventually hit your Workflow Task or execution timeout, unless another Worker is spun up.

## General developer guidance

- Ensure Activities and Local Activities **honor context cancellation** or other shutdown signals.
- Expect that **long or hung Local Activities may block shutdown** unless you fail early.
  It is recommended that Local Activities should already generally be used for short Activities.

---

# Temporal Workflow message passing - Signals, Queries, & Updates

Source: https://docs.temporal.io/encyclopedia/workflow-message-passing

> Signals, Queries, and Updates facilitate interactions with Workflow Executions.

Workflows can be thought of as stateful web services that can receive messages. The Workflow can have message handlers, akin to endpoints, that react to incoming messages together with the Workflow's current state.
Temporal supports three types of messages: Signals, Queries, and Updates:

- Queries are read requests. They can read the current state of the Workflow but cannot block in doing so.
- Signals are asynchronous write requests. They cause changes in the running Workflow, but you cannot await any response
  or error.
- Updates are synchronous, tracked write requests. The sender of the Update can wait for a response on completion or an
  error on failure.

## How to choose between Signals, Updates, and Queries as a Workflow author? 

This section will help you write Workflows that receive messages.

### For write requests

Unlike Signals, Updates must be synchronous and must wait for the Worker running the Workflow to acknowledge the
request.

The following table compares when to use **Signals** versus **Updates**.

| **Requirement type**           | **Use Signals when...**                                                                     | **Use Updates when...**                                                                                                                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Asynchronous communication** | Clients want to quickly move on after sending an asynchronous message.                      | Clients want to track the completion of the message.                                                                                                                                       |
| **Result handling**            | Clients are okay with “fire and forget” — no result or exception needed.                    | Clients need a result or exception without performing a query.                                                                                                                             |
| **Worker availability**        | Clients don't depend on the Worker being available.                                         | You want to validate the Update before accepting it into the Workflow and its history.                                                                                                     |
| **Concurrency and throughput** | You don’t want to limit the number of messages processed concurrently by a single Workflow. | You don’t need more concurrent Updates per Workflow than the allowed limits for [Cloud](/evaluate/cloud/limits#per-workflow-execution-update-limits) or [Self-Hosted](/self-hosted-guide/defaults). |
| **Latency sensitivity**        | Since clients don’t expect a result, latency is often not relevant when using Signals.      | Clients want a low-latency end-to-end operation and are willing to wait for completion or validation.                                                                                      |

### For read requests

You normally want to do a Query, because:

- Queries are efficient–they never add entries to the [Workflow Event History](/workflow-execution/event#event-history),
  whereas an Update would (if accepted).
- Queries can operate on completed Workflows.

However, because Queries cannot block, sometimes Updates are best. When your goal is to do a read once the Workflow
achieves a certain desired state, you have two options:

- You could poll periodically with Queries until the Workflow is ready.
- You could write your read operation as an Update, which will give you better efficiency and latency, though it will
  write an entry to the [Workflow Event History](/workflow-execution/event#event-history).

### For read/write requests

Use an Update for synchronous read/write requests. If your request must be asynchronous, consider sending a Signal
followed by polling with a Query.

---

# Cancellation and Termination

Source: https://docs.temporal.io/encyclopedia/workflow/cancellation-and-termination

Cancellation and Termination both stop a [Workflow Execution](/workflow-execution) before it finishes on its own. They
differ in whether your Workflow code gets a say.

Cancellation is cooperative. The Temporal Service delivers the request to the Workflow, and the Workflow decides what to
do with it, which means it can release resources, compensate for completed steps, and choose its own closing status.
Termination is forceful. The Workflow Execution is closed immediately and Workflow code never sees the request, so no
cleanup runs.

Prefer Cancellation. Reach for Termination when a Workflow can't process a Cancellation request, such as when it's
blocked on a non-deterministic error or has no Workers to run its Tasks.

## Operations summary

| Operation                       | What it does                                                        | CLI                                                                     |
| ------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| [Cancellation](#cancellation)   | Asks the Workflow to stop, letting it run cleanup first.            | [`temporal workflow cancel`](/cli/command-reference/workflow#cancel)     |
| [Termination](#termination)     | Closes the Workflow Execution immediately, with no cleanup.         | [`temporal workflow terminate`](/cli/command-reference/workflow#terminate) |

Both operations are also available through the Web UI, the SDK Clients, and the gRPC API. Both accept a
[List Filter](/list-filter) instead of a Workflow Id to act on matching Workflow Executions in bulk.

## Cancellation

### What happens when you cancel a Workflow

- A [WorkflowExecutionCancelRequested](/references/events#workflowexecutioncancelrequested) Event is recorded, carrying
  the requester's identity and the reason.
- The Temporal Service schedules a new [Workflow Task](/tasks#workflow-task), so the Workflow can react to the request.
- The SDK surfaces the request to Workflow code as a Cancelled Failure. See
  [How Temporal represents failures](/encyclopedia/application-failures#failure-representation).
- The Workflow runs whatever cleanup its implementation supports.
- The closing status depends on what the Workflow does next.

Because Cancellation is delivered as a failure, unhandled Cancellation closes the Workflow Execution as **Canceled**
rather than Failed, recording a [WorkflowExecutionCanceled](/references/events#workflowexecutioncanceled) Event. A
Workflow that handles the request and returns normally closes as **Completed**. Both are valid outcomes; which one you
want is a business decision, not a technical one.

Cleanup that calls Activities has to run outside the canceled scope, because a canceled scope can't schedule new work.
Each SDK provides a mechanism for this, such as `workflow.NewDisconnectedContext` in Go. See the SDK guides below.

### Cancellation and Activities

Activity Cancellations are delivered to the Activity when it [Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat).
This has two consequences worth planning around:

- An Activity that doesn't Heartbeat can't receive a Cancellation. It runs to completion, fails, or times out.
- Heartbeat throttling can delay delivery, so Cancellation may reach the Activity later than the request was made.

If you need a long-running Activity to stop when its Workflow is canceled, it has to Heartbeat.

### Cancellation across Workflow boundaries

A Workflow can request Cancellation of another Workflow Execution. This records a
[RequestCancelExternalWorkflowExecutionInitiated](/references/events#requestcancelexternalworkflowexecutioninitiated)
Event in the requesting Workflow, followed by
[ExternalWorkflowExecutionCancelRequested](/references/events#externalworkflowexecutioncancelrequested) once the Service
delivers it. When the target is a [Child Workflow](/child-workflows), the parent records
[ChildWorkflowExecutionCanceled](/references/events#childworkflowexecutioncanceled).

## Termination

### What happens when you terminate a Workflow

- A [WorkflowExecutionTerminated](/references/events#workflowexecutionterminated) Event is recorded with the reason and
  the requester's identity. This is the closing Event in the Workflow Execution's Event History.
- Workflow code can't see or respond to the termination, so no cleanup runs and no compensation happens.
- Pending Activities are not given a chance to stop cleanly.

The reason is optional and defaults to the current user's name. Provide one — it's the only record of why the Workflow
Execution was stopped.

Anything a terminated Workflow would have cleaned up is left as it was: reserved inventory stays reserved, held locks
stay held, partial writes stay written. If that matters for your use case, cancel instead, and terminate only if the
Cancellation doesn't take effect.

## Which one should you use?

|                              | Cancellation                                    | Termination                        |
| ---------------------------- | ----------------------------------------------- | ---------------------------------- |
| **Workflow code notified?**  | Yes, as a Cancelled Failure                     | No                                 |
| **Cleanup and compensation** | Runs, if the Workflow implements it             | Doesn't run                        |
| **Closing status**           | Canceled, or Completed if the Workflow handles it and returns normally | Terminated    |
| **Closing Event**            | WorkflowExecutionCanceled                       | WorkflowExecutionTerminated        |
| **Takes effect**             | On the next Workflow Task                       | Immediately                        |
| **Works on a stuck Workflow** | No, the Workflow has to process the request     | Yes                                |
| **Failure type**             | Cancelled Failure                               | Terminated Failure                 |

## Child Workflows

A [Parent Close Policy](/parent-close-policy) decides what happens to a Child Workflow Execution when its parent reaches
a Closed status. Two of the three values are these same operations:

- **Terminate** (the default): the child is forcefully Terminated.
- **Request Cancel**: a Cancellation request is sent to the child.
- **Abandon**: the child isn't affected.

Each child sets its own policy, so a parent can terminate some children, cancel others, and leave the rest running.

## Related controls

These operations also stop a Workflow Execution from progressing, but they aren't Cancellation or Termination:

- **[Workflow Pause](/encyclopedia/workflow/workflow-pause)** holds a Workflow Execution in place without closing it or
  losing state. A cancel request sent to a Paused Workflow is recorded and takes effect after the Workflow is Unpaused. A terminate request takes effect immediately.
- **[Reset](/cli/command-reference/workflow#reset)** terminates the current Run and starts a new Run from a point you
  choose in the Event History. Use it when a Workflow is blocked by a non-deterministic error you've since fixed.
- **Timeouts** close a Workflow Execution as Timed Out rather than Canceled or Terminated. See
  [Detecting Workflow failures](/encyclopedia/detecting-workflow-failures).

## Handle Cancellation in your SDK

Cancellation is the half of this that needs Workflow code. Termination needs none.

- [.NET SDK](/develop/dotnet/workflows/cancellation)
- [Go SDK](/develop/go/workflows/cancellation)
- [Java SDK](/develop/java/workflows/cancellation)
- [PHP SDK](/develop/php/workflows/cancellation)
- [Python SDK](/develop/python/workflows/cancellation)
- [Ruby SDK](/develop/ruby/workflows/cancellation)
- [Rust SDK](/develop/rust/workflows/cancellation)
- [TypeScript SDK](/develop/typescript/workflows/cancellation)

---

# Workflow Pause

Source: https://docs.temporal.io/encyclopedia/workflow/workflow-pause

> Pause and Unpause a Workflow Execution to stop new progress without terminating the Workflow or losing state.

This page discusses the following:

- [Pause](#pause)
- [Unpause](#unpause)
- [Observability](#observability)
- [What Workflow Pause is not](#what-workflow-pause-is-not)
- [Important considerations](#important-considerations)
- [Limitations](#limitations)

Workflow Pause is an operational control that stops a specific [Workflow Execution](/workflow-execution) from making new
progress until it is Unpaused.

Use Workflow Pause when you need to hold a Workflow in place without terminating the Workflow Execution or losing Workflow state. For example, during an incident, investigation, or dependency outage.

You can Pause and Unpause Workflow Executions through the [CLI](/cli/command-reference/workflow), the UI, or directly
through the gRPC API.

> **Pre-release**
> In Temporal
> Cloud, Pre-release access is invite-only; contact your Temporal account team or
> [open a support ticket](/evaluate/cloud/support#support-ticket). For self-hosted Temporal, Workflow Pause requires Temporal
> Server v1.30.0+ with `frontend.WorkflowPauseEnabled` enabled. The Temporal CLI requires v1.6.0+. Self-hosted UI support
> requires v2.47.2+.

## Operations summary

| Operation         | What it does                                         | CLI                                                                  |
| ----------------- | ---------------------------------------------------- | -------------------------------------------------------------------- |
| [Pause](#pause)   | Stops a Workflow Execution from making new progress. | [`temporal workflow pause`](/cli/command-reference/workflow#pause)   |
| [Unpause](#unpause) | Resumes a Paused Workflow Execution.                 | [`temporal workflow unpause`](/cli/command-reference/workflow#unpause) |

## Pause

Pause stops a Workflow Execution from making new progress until it is Unpaused.

### When to pause

- A downstream dependency is unhealthy, and you want to stop the Workflow from continuing until the dependency recovers.
- You need time to inspect or fix an issue before the Workflow schedules more work.
- You're rolling out a Worker change and want to hold specific Workflow Executions until the deploy is complete.
- You want to prevent a Workflow from continuing without terminating it or losing its current state.

### What happens when you Pause a Workflow

- No new Workflow Tasks are dispatched. Workflow code doesn't make progress while the Workflow is Paused.
- No new Activity Tasks are dispatched. Activity retries and newly scheduled Activity Tasks don't start while the
  Workflow is Paused.
- In-flight Activity attempts aren't interrupted. Activity attempts that are already running can complete, fail,
  time out, and Heartbeat normally.
- Activity completion, failure, and timeout events can be recorded. Workflow code doesn't process those events until
  the Workflow is Unpaused.
- Signals are accepted and recorded. Signal handlers run after the Workflow is Unpaused.
- Timers keep advancing. Timers that fire while Paused are processed by Workflow code after the Workflow is
  Unpaused.
- Updates and Queries are rejected.
- Cancel requests are recorded. Cancellation takes effect after the Workflow is Unpaused.
- Terminate requests still terminate the Workflow immediately.
- Pause is recorded in Event History. The Event includes the identity, reason, and request ID.

### CLI usage

```bash
temporal workflow pause \
  --workflow-id YourWorkflowId \
  --reason "Pausing while downstream service recovers"
```

See the [CLI reference for `temporal workflow pause`](/cli/command-reference/workflow#pause) for all options.

## Unpause

Unpause resumes a Paused Workflow Execution.

### When to unpause

- The dependency or service that caused the Pause has recovered.
- Investigation is complete and the Workflow can continue.
- A deploy or configuration change is complete.
- You're ready for queued Signals, recorded cancellation, or blocked work to proceed.

### What happens when you Unpause a Workflow

- Workflow Tasks and Activity Tasks can be dispatched again.
- Signals received and Timers that fired while Paused are processed by the Workflow.
- Pending Activity retries can proceed unless the Activity itself is Paused.
- The Workflow continues from its existing state.
- Unpause is recorded in Event History. The Event includes the identity, reason, and request ID.

### CLI usage

```bash
temporal workflow unpause \
  --workflow-id YourWorkflowId \
  --reason "Downstream service recovered"
```

See the [CLI reference for `temporal workflow unpause`](/cli/command-reference/workflow#unpause) for all options.

## Observability

Paused Workflow Executions have `ExecutionStatus="Paused"`.

You can find Paused Workflow Executions with a [List Filter](/list-filter):

```sql
ExecutionStatus = "Paused"
```

You can also inspect a Workflow Execution with:

```bash
temporal workflow describe \
  --workflow-id YourWorkflowId
```

Pause and Unpause are recorded in [Event History](/workflow-execution/event#event-history) with the identity, reason, and
request ID.

## What Workflow Pause is not

- Workflow Pause isn't Activity Pause. To interrupt a Heartbeating Activity attempt, use
  [Activity Pause](/activity-operations/pause).
- Workflow Pause isn't bulk Workflow Pause. It applies to one Workflow Execution at a time.
- Workflow Pause doesn't pause [Schedules](/schedule), Task Queues, or Namespaces.
- Workflow Pause isn't an auto-pause policy. Operators must Pause and Unpause manually through an operational interface.

## Important considerations

### Scope

- Workflow Pause applies to a single Workflow Execution.
- It doesn't pause [Child Workflows](/child-workflows), Activities, Schedules, Task Queues, or Namespaces.

### Timeouts and in-flight work

- Workflow Pause doesn't stop time from passing. Workflow Execution timeouts, Workflow Run timeouts, Activity timeouts,
  and Timer deadlines continue to advance.
- Activity attempts that are already running aren't interrupted. They can complete, fail, Heartbeat, or time out while
  the Workflow is Paused.

### Messages and lifecycle operations

- Signals are accepted while Paused and processed after the Workflow is Unpaused.
- Updates and Queries are rejected while Paused.
- Cancel requests are recorded while Paused, but cancellation takes effect after the Workflow is Unpaused.
- [Terminate](/encyclopedia/workflow/cancellation-and-termination#termination) requests still terminate the Workflow
  immediately.
- Resetting a Paused Workflow terminates the current Run and starts a new Run from the selected reset point.

### Schedules, cron, and operation timing

- A Paused Run is still considered active for [Schedule](/schedule) overlap policy decisions.
- For [Cron jobs](/cron-job), Pause affects the current Run. Missed cron intervals are not backfilled.
- Pause and Unpause aren't a transactional boundary. Work can make progress while the Workflow is Unpaused.

### Billing

In Temporal Cloud, pausing a Workflow Execution stops progress but doesn't end the Workflow Execution.
Its Event History continues to count toward [Active Storage](/cloud/pricing#storage) until the Workflow Execution
completes, fails, times out, is canceled, or is terminated.

## Limitations

- Workflow Pause must be enabled before it can be used.
- Workflow Pause applies to a single Workflow Execution. Bulk Workflow Pause is not supported.
- Auto-pause policies are not supported.
- Workflow Pause is an operational control. It isn't intended to be called from Workflow code.
- There is no indicator that all in-flight Activity attempts have completed after a Workflow is Paused.

## Interaction with Activity operations

Workflow Pause and [Activity Operations](/activity-operations) are separate controls.

Workflow Pause stops progress for a Workflow Execution. Activity Operations act on a specific Activity Execution.

If a Workflow is Paused, Activity retries in that Workflow are blocked. If an individual Activity is also Paused, both
the Workflow and the Activity must be Unpaused before that Activity can proceed. If a Workflow is Unpaused, all pending
Activities will have available retries dispatched, if the Activity is Paused it has to be Unpaused before any new
attempts are dispatched.

Workflow Pause doesn't interrupt Activity attempts that are already running. To interrupt a Heartbeating Activity
attempt, use [Activity Pause](/activity-operations/pause).

---

# Evaluate Temporal

Source: https://docs.temporal.io/evaluate

> Temporal enhances distributed application development with clear code structure, fault-tolerance, and execution guarantees, trusted by thousands for mission-critical workloads.

Temporal gives developers a clear way to structure distributed application code and see the state of a running application.
Applications get fault tolerance and execution guarantees.
Companies of all sizes run Temporal for both mission-critical and standard workloads.

- [Why Temporal](/evaluate/why-temporal): Durable Execution, simplified code structure, and state visibility, and what each one changes about building reliable applications.
- [Understanding Temporal](/evaluate/understanding-temporal): How the Temporal Service, Workers, Workflows, and Activities fit together to run your application code.
- [Features](/evaluate/features): Development and production capabilities available through a Temporal SDK, from core primitives to observability and data encryption.
- [Temporal Cloud](/evaluate/cloud): A fully managed durable execution platform on AWS and GCP, with consumption-based pricing and migration from self-hosted deployments.
- [Product release stages](/evaluate/product-release-stages): Pre-release, Public Preview, and General Availability compared by feature access, API stability, support, pricing, and regional coverage.
- [Security](/security): Security practices for Temporal Cloud and for self-hosted deployments, plus company compliance certifications.
- [Use cases](/evaluate/use-cases-design-patterns): How organizations run Temporal in production, and the architectural design patterns behind those applications.

---

# Overview

Source: https://docs.temporal.io/evaluate/cloud

> Managed Durable Execution on AWS and GCP, with cell-based isolation, three-zone replication, consumption-based pricing, and migration from self-hosted.

Temporal Cloud is a managed Durable Execution platform.
Temporal operates the Temporal Service for you: persistence, replication, version upgrades, and availability.

Your Workers stay in your environment.
They run your Workflow and Activity code and handle your data, while Temporal Cloud stores Workflow state and schedules the work.

## How Temporal Cloud works

Temporal Cloud runs the Temporal Service for your applications:

1. **Your environment**: You run Workers that execute your Workflow and Activity code. Deploy them on Kubernetes, virtual machines, serverless platforms, or on-premises hardware.
2. **Temporal Cloud**: Stores Workflow state and Event History, queues Tasks, and schedules work. Data is encrypted in transit and at rest.
3. **Temporal SDKs**: Your applications use an SDK to talk to Temporal Cloud over TLS-encrypted gRPC connections.

Because the two halves scale separately, you size the compute for your Workers and Temporal sizes the orchestration layer.

## Architecture

### Cell-based infrastructure

Temporal Cloud uses a cell-based architecture to achieve isolation and scalability.
Each cell is a self-contained deployment unit with its own:

- Dedicated cloud account and VPC
- Kubernetes cluster running Temporal services
- Primary database with synchronous replication across three availability zones
- Elasticsearch for Workflow visibility and search
- Load balancers and ingress management
- Observability and certificate infrastructure

Cells are failure domains.
When infrastructure inside a cell degrades, only the Namespaces in that cell are affected, and cells scale independently of each other.

### Data plane and control plane

**Data plane**: Where your Workflows run. Each cell processes Workflow operations, persists state, and manages Task Queues.

**Control plane**: Handles provisioning, configuration, and lifecycle operations. When you create a Namespace, the control plane:

1. Selects a cell in the region you chose
2. Provisions database resources and roles
3. Generates and deploys mTLS certificates
4. Configures ingress routes and validates connectivity

The control plane runs those steps on Temporal itself.

### Multi-cloud availability

Temporal Cloud runs on both AWS and GCP:

- <strong><CloudRegionCount provider="aws" /> AWS regions</strong> spanning North America, Europe, Asia Pacific, and South America
- <strong><CloudRegionCount provider="gcp" /> GCP regions</strong> in North America, Europe, and Asia Pacific

You can create Namespaces in any supported AWS region. New Namespaces in some GCP regions are temporarily restricted; see [GCP capacity restriction](/evaluate/cloud/regions#gcp-capacity-restriction).
For disaster recovery, replicate across regions within a cloud provider or across the two providers.

See [Service regions](/evaluate/cloud/regions) for the complete list.

## Built-in reliability

Every Temporal Cloud Namespace includes baseline high availability:

- **Three-zone replication**: Workflow state replicates synchronously across three availability zones before a write is acknowledged
- **Automatic failover**: If one zone becomes unavailable, operations continue on the remaining zones
- **Contractual SLA**: Uptime commitment for standard Namespaces, defined in the [Temporal Cloud SLA](https://temporal.io/sla)

### High Availability features

For workloads that need stronger guarantees, Temporal Cloud offers three replication options:

| Deployment | Description | Use case |
|------------|-------------|----------|
| **Same-region** | Replicate across isolated cells within one region | Single-region applications needing cell-level isolation |
| **Multi-region** | Replicate across regions within one cloud provider | Geographic redundancy and compliance requirements |
| **Multi-cloud** | Replicate across cloud providers (AWS ↔️ GCP) | Resilience to a provider-wide outage |

High Availability Namespaces include:

- **Higher contractual SLA**: Uptime commitment defined in the [Temporal Cloud SLA](https://temporal.io/sla)
- **Sub-1-minute RPO**: Recovery Point Objective for data loss
- **20-minute RTO**: Recovery Time Objective for failover completion
- **Automatic or manual failover**: Choose the failover strategy per Namespace

See [High Availability](/cloud/high-availability) for configuration details.

## Security model

Temporal Cloud secures your applications at several layers.

### Your code stays with you

Temporal Cloud does not execute your application code.
Workers run in your environment and connect to Temporal Cloud over encrypted channels, so access to your compute resources and secrets stays with you.

### Client-side encryption

The [Data Converter](/dataconversion) encrypts Payloads before they leave your Workers, so Temporal Cloud stores ciphertext and the data stays encrypted even if the service is compromised.
Deploy a [Codec Server](/production-deployment/data-encryption) to decrypt data in the Web UI without sharing keys with Temporal.

### Network isolation

- **mTLS authentication**: Per-Namespace certificate-based authentication for gRPC endpoints
- **API key authentication**: Alternative to certificates for simpler key management
- **Private connectivity**: AWS PrivateLink and GCP Private Service Connect for traffic that never traverses the public internet

### Compliance

Temporal Technologies maintains SOC 2 Type 2 certification and complies with GDPR and HIPAA regulations.
Audit logs capture supported operations in the Temporal Cloud control plane and export to your security
monitoring systems.
See [Audit Logging](/cloud/audit-logs) for the supported operations and coverage.

See [Security model](/evaluate/cloud/security) for the full model.

## Consumption-based pricing

Temporal Cloud bills for what you use.

### Actions

Actions are the primary billing unit.
Actions are billable operations like starting Workflows, sending Signals, recording Heartbeats, and completing Activities. Pricing starts at $50 per million Actions. Volume and commitment discounts are available on the Business, Enterprise, and Mission Critical plans.

### Storage

- **Active Storage**: Event History for running Workflows
- **Retained Storage**: Event History for closed Workflows, for a Retention Period you set per Namespace, up to 90 days

### Plans

Four tiers—Developer, Business, Enterprise, and Mission Critical—with increasing support levels. The Developer plan starts at $0/month. The Business plan and above add monthly Actions and Storage allocations and commitment discounts.
Self-serve signup and plan management available at [cloud.temporal.io](https://cloud.temporal.io).

See [Pricing](/cloud/pricing) for rates and examples.

## Move workloads between deployments

Temporal Cloud runs the same Temporal Server as the open-source distribution, so applications built for a self-hosted Temporal Service run on Temporal Cloud without code changes.
Point your connection configuration at your Cloud Namespace.

[Automated migration](/cloud/migrate/automated) uses Workflow replication to move running Workflows from self-hosted to Cloud, or between Cloud regions, with no Workflow restarts, data loss, or downtime.
[Manual migration](/cloud/migrate/manual) updates Clients and Workers to use new Namespace endpoints and lets existing Workflows finish where they are.

The same tooling works in every direction: self-hosted to Cloud, Cloud to self-hosted, and between Cloud regions and providers.

## Run Cloud operations yourself

Standard operations need no support ticket:

- **Web UI**: Create Namespaces, manage users, and configure settings at [cloud.temporal.io](https://cloud.temporal.io)
- **[Temporal CLI Cloud extension](/cli/cloud)**: Automate operations from the Temporal CLI
- **CLI (`tcld`)**: Automate operations from the command line
- **Terraform provider**: Manage Namespaces, users, and configuration as code
- **Cloud Ops API**: Programmatic access for custom tooling and automation

## Get started

1. [Sign up for Temporal Cloud](https://temporal.io/get-cloud)
2. [Create your first Namespace](/cloud/namespaces)
3. [Connect your Workers](/cloud/get-started#set-up-your-clients-and-workers)
4. [Run your first Workflow](/cloud/get-started#run-your-first-workflow)

To move existing self-hosted workloads, see [Migration](/cloud/migrate).

## Evaluate Temporal Cloud

- [Pricing](/cloud/pricing): Plans, Action and Storage rates, commitment pricing, and worked examples for estimating a monthly bill.
- [Actions](/evaluate/cloud/actions): Which operations count as billable Actions, by category, with the metric and Event History type for each.
- [Limits](/evaluate/cloud/limits): Rate, resource, and configuration limits by scope, which ones you can raise, and what happens when you exceed them.
- [Uptime, latency, and SLOs](/cloud/slo): Published objectives for uptime, latency, and Visibility availability, and measured latency for common operations.
- [SLA](https://temporal.io/sla): The contractual uptime commitment: covered APIs, how availability is calculated, exclusions, and service credits.
- [Security model](/evaluate/cloud/security): Namespace isolation, authentication, encryption in transit and at rest, access control, and compliance certifications.
- [Service regions](/evaluate/cloud/regions): The AWS and GCP regions where you can create a Temporal Cloud Namespace.
- [Support](/evaluate/cloud/support): Response times, channels, and technical guidance included with each plan, plus how to open a support ticket.

---

# Temporal Cloud Actions

Source: https://docs.temporal.io/evaluate/cloud/actions

Actions are the primary unit of consumption-based pricing for Temporal Cloud. They track billable operations within the
Temporal Cloud Service, such as starting Workflows, recording a Heartbeat, or sending Signals.

Most Actions fall into one of these categories:

- [Workflow](#workflow)
- [Activity](#activity)
- [Timer](#timer)
- [Signal](#signal)
- [Query](#query)
- [Update](#update)
- [Schedule](#schedule)
- [Nexus](#nexus)

Some additional Temporal Cloud features are billed as Actions:

- [Export](#export)
- [Fairness](#fairness)
- [Capacity](#capacity)

Actions that occur during [Workflow Replay](/workflow-execution#replay) do not count towards billed Actions. Replay
happens only on the Worker side to reconstruct Workflow state from Event History and does not generate new server-side
operations.

Billable Actions are visible in the [OpenMetrics endpoint](/cloud/metrics/openmetrics),
[Event History](/cloud/actions-usage#actions-in-workflows) and [Usage](https://cloud.temporal.io/usage) Dashboards,
and the [Billing API](/cloud/billing-api). Use Action types and categories to estimate usage, find the Action types
driving it, track spikes, and troubleshoot errors. For example, you can see whether a new Workflow Type on a Namespace
is driving a usage increase, or find the aggregate usage of Activity Heartbeats on a Namespace.

Which Action Types you see depends on the endpoint. Action Categories and Action Types are not available in the Billing
API.

Usage and billing data carry the most complete Actions data, while Event History and metrics support detailed
_estimates_. [History Event Types](/workflow-execution/event) are listed for transparency, but they do not always map
1:1 to Action Types. The following sections list each category, its Action types, and where each one is available.

## Compare Action metrics with metering 

Action metrics are estimates and can differ by up to 5% from the absolute Action counts recorded by Temporal Cloud
metering. Use the [Usage dashboard](/cloud/actions-usage#usage) or [Billing API](/cloud/billing-api) for metered usage when
you need exact totals. Some billable Action types are not included in the
[Billable Action Count metric](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count), so
its total might also differ from the total across all Action types.

## Workflow 

- **Workflow started**
  - Occurs via client start, [Continue-As-New](/workflow-execution/continue-as-new), [Child Workflow](/child-workflows)
    start.
  - If a Workflow start fails, an Action is not recorded.
  - De-duplicated Workflow starts that share a Workflow ID do _not_ count as an Action.
- **Workflow reset**
  - Occurs when a [Workflow](/workflows) is reset.
  - Actions that occur before a [Reset](/workflow-execution/event#reset) are counted on the original workflow. These
    events are not counted on the newly created workflow.
- **Search Attribute upsert requested**
  - Occurs for each invocation of `UpsertSearchAttributes` command.
  - Multiple Search Attributes updated in a single `UpsertSearchAttributes` command count as one Action.
  - Search Attributes specified during Workflow start are _excluded_ from Action counts.
  - The `TemporalChangeVersion` Search Attribute, used for Workflow versioning, is also exempt from Action counting.
- **Workflow Execution Options updated.**
  - Occurs for every [Workflow-Execution-Options-Updated](/references/events#workflowexecutionoptionsupdated) event.
  - This includes attaching a Workflow completion callback or modifying a Workflow versioning override.
- **Start Child Workflow Execution initiated.**
  - Occurs when the parent Workflow durably records the intent to start a Child Workflow.
  - Results in two Actions, one for durably storing the intent to start a Child Workflow and one for the attempt to
    start it.
- **Workflow started via Multi-Operation.**
  - Occurs when a Workflow is started as part of an ExecuteMultiOperation request — a single atomic RPC that bundles a
    Workflow start with one or more Workflow Updates. The start counts as one Action.
  - Each bundled Update that is accepted or rejected counts as an additional Action.
  - If the Workflow was already running and the start is de-duplicated, no Action is charged for the start (though an
    `update_workflow_options_via_start` Action may still apply).

| Usage Name                          | Metric Name                         | History Event Type                                                                                                                                                                                          |
| ----------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Start Workflow                      | `start_workflow`                    | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED`                                                                                                                                                                     |
| Start Child Workflow                | `start_child_workflow`              | `EVENT_TYPE_START_CHILD_WORKFLOW_EXECUTION_INITIATED`                                                                                                                                                       |
| Continue As New Workflow            | `continue_as_new_workflow`          | `EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW` Note: The UI shows a billable action only for the `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` event of the newly started Workflow, not for this history event. |
| Start Workflow (Multi-Operation)    | `start_workflow_multi_operation`    | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED`                                                                                                                                                                     |
| Update Workflow Options             | `update_workflow_options`           | `EVENT_TYPE_WORKFLOW_EXECUTION_OPTIONS_UPDATED`                                                                                                                                                             |
| Update Workflow Options (Via Start) | `update_workflow_options_via_start` | `EVENT_TYPE_WORKFLOW_EXECUTION_OPTIONS_UPDATED`                                                                                                                                                             |
| Upsert Workflow Search Attributes   | `upsert_workflow_search_attributes` | `EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES`                                                                                                                                                              |
| Reset Workflow                      | `reset_workflow`                    | `EVENT_TYPE_WORKFLOW_TASK_FAILED` Note: When `cause == WORKFLOW_TASK_FAILED_CAUSE_RESET_WORKFLOW`                                                                                                           |
| Record Workflow Heartbeat           | `record_workflow_heartbeat`         | N/A                                                                                                                                                                                                         |

## Activity 

- **Activity started or retried**. Occurs each time a [**Workflow Activity**](/workflow-activity) or [**Standalone Activity**](/standalone-activity) is started or retried.
  - De-duplicated Standalone Activity starts that return an already-running Activity (sharing an Activity ID) do _not_
    count as an Action, unless the start request performs an Activity options update, like attaching a callback to the running Activity — for example,
    within a Nexus handler using the `USE_EXISTING` conflict policy.
- **Standalone Activity pause, reset, and update options**. Each count as one Action. Unpause does _not_ count as an Action.
  See [Activity Operations](/activity-operations).
- **Local Activity started**. All [Local Activities](/local-activity) associated with one Workflow Task count as a
  single Action. Temporal Cloud counts all [RecordMarkers](/references/commands#recordmarker) from each Workflow Task as
  one action, and not _N_ actions. Note:
  - Each Workflow Task Heartbeat counts as an additional Action.
  - Local Activities retried following a Workflow Task Heartbeat count as one Action (capped at 100 Actions).
- **Activity Heartbeat recorded**. A Heartbeat call from Activity code counts as an Action only if it reaches the
  [Temporal Server](/temporal-service/temporal-server). Temporal SDKs throttle
  [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat). The default throttle is 80% of
  the [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout). Heartbeats don't apply to Local
  Activities.

| Usage Name                                 | Metric Name                                  | History Event Type                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Schedule Activity                          | `schedule_activity`                          | `EVENT_TYPE_ACTIVITY_TASK_SCHEDULED`                                                                                                                                                                                                                                                                                                                                                                                                          |
| Retry Activity                             | `retry_activity`                             | `EVENT_TYPE_ACTIVITY_TASK_STARTED` Note: Intermediate retry attempts do not produce separate history events. Total attempt count is recorded in the `ActivityTaskStartedEventAttributes.attempt` field on this event. The user should subtract 1, because the first attempt is not a retry. If the Workflow is terminated while Activity retry is in progress, this `STARTED` event is not written, so this data may not always be available. |
| Retry Local Activity                       | `retry_activity_local`                       | `EVENT_TYPE_WORKFLOW_TASK_COMPLETED ` Note: the retry count is recorded in `WorkflowTaskCompletedEventAttributes.metering_metadata.nonfirst_local_activity_execution_attempts` field on this event.                                                                                                                                                                                                                                           |
| Record Activity Heartbeat                  | `record_activity_heartbeat`                  | N/A                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Record Activity Heartbeat by ID            | `record_activity_heartbeat_by_id`            | N/A                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Record Activity Markers                    | `record_activity_markers`                    | `EVENT_TYPE_MARKER_RECORDED`                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Start Standalone Activity                  | `start_standalone_activity`                  | N/A                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Retry Standalone Activity                  | `retry_standalone_activity`                  | N/A                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Record Standalone Activity Heartbeat       | `record_standalone_activity_heartbeat`       | N/A                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Record Standalone Activity Heartbeat By ID | `record_standalone_activity_heartbeat_by_id` | N/A                                                                                                                                                                                                                                                                                                                                                                                                                                           |

## Timer 

- **Timer started**. Includes implicit Timers that are started by a Temporal SDK when timeouts are set, such as
  `AwaitWithTimeout` in Go or `condition` in TypeScript.

| Usage Name  | Metric Name   | History Event Type         |
| ----------- | ------------- | -------------------------- |
| Start Timer | `start_timer` | `EVENT_TYPE_TIMER_STARTED` |

## Signal 

- **Signal sent**. An Action occurs for every [Signal](/sending-messages#sending-signals),
  whether sent from a Client or from a Workflow. Also, one total action occurs for any
  [Signal-With-Start](/sending-messages#signal-with-start), regardless of whether the Workflow
  starts.

| Usage Name                 | Metric Name                  | History Event Type                                                                                                                                                                                                                                                           |
| -------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Signal Workflow            | `signal_workflow`            | `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED`                                                                                                                                                                                                                                     |
| Signal External Workflow   | `signal_external_workflow`   | `EVENT_TYPE_SIGNAL_EXTERNAL_WORKFLOW_EXECUTION_INITIATED` (signaler event history) `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED` (signaled Event history) Note: The UI shows a billable action only in the signaled Event history, not in the signaler Event history. |
| Signal Workflow With Start | `signal_workflow_with_start` | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` followed by `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED` if the workflow didn’t exist, or only `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED` if the Workflow existed.                                                                         |

## Query 

- **Query received by Worker**. An Action occurs for every
  [Query](/sending-messages#sending-queries), including viewing the call stack in the Temporal
  Cloud UI, which results in a Query behind the scenes. `__temporal_workflow_metadata` is a built-in query used to
  retrieve workflow metadata at runtime and is excluded.

| Usage Name     | Metric Name      | History Event Type |
| -------------- | ---------------- | ------------------ |
| Query Workflow | `query_workflow` | N/A                |

## Update 

- **Update received by Worker**.
  - Occurs for every successful [Update](/sending-messages#sending-updates) and every
    [rejected](/handling-messages#update-validators) Update.
  - This includes [Update-With-Start](/sending-messages#update-with-start), and is in addition
    to the start Action in the case when the Workflow starts as well.
  - De-duplicated Updates that share an Update ID do _not_ count as an Action.

| Name                   | Action ID                | History Event Type                                                                            |
| ---------------------- | ------------------------ | --------------------------------------------------------------------------------------------- |
| Accept Workflow Update | `accept_workflow_update` | `EVENT_TYPE_WORKFLOW_EXECUTION_UPDATE_ACCEPTED`                                               |
| Reject Workflow Update | `reject_workflow_update` | `EVENT_TYPE_WORKFLOW_EXECUTION_UPDATE_REJECTED` Note: this event is never written to history. |

## Schedule 

A [Schedule](/schedule) starts a Workflow at a particular time. Each execution of a Schedule accrues three Actions:

- **Schedule Start**. This accounts for two actions.
- **Workflow started**. This is a single action to start the target Workflow. It includes initial Search Attributes as
  part of the start request.

| Usage Name        | Metric Name         | History Event Type                                                                                                                                                               |
| ----------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Create Schedule   | `create_schedule`   | N/A                                                                                                                                                                              |
| Update Schedule   | `update_schedule`   | N/A                                                                                                                                                                              |
| Patch Schedule    | `patch_schedule`    | N/A                                                                                                                                                                              |
| Delete Schedule   | `delete_schedule`   | N/A                                                                                                                                                                              |
| Schedule Workflow | `schedule_workflow` | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` Note: `event.WorkflowExecutionStartedEventAttributes.indexedFields.TemporalScheduledById` tells if a Workflow was started by a Schedule. |

## Temporal Nexus 

- For [Nexus Operation scheduled](/references/events#nexusoperationscheduled), the caller Workflow starting a Nexus
  Operation results in one Action on the caller Namespace.
- For [Nexus Operation canceled](/references/events#nexusoperationcanceled), the caller Workflow canceling a Nexus
  Operation results in one Action on the caller Namespace.
- The underlying Temporal primitives (such as Workflows, Activities, and Signals) created by a Nexus Operation handler
  (directly or indirectly) result in the normal Actions for those primitives billed to the handler’s Namespace. This
  includes retries for underlying Temporal primitives like Activities but _not_ for handling the Nexus Operation itself
  or a retry of the Nexus Operation itself.

| Usage Name                           | Metric Name                            | History Event Type                            |
| ------------------------------------ | -------------------------------------- | --------------------------------------------- |
| Schedule Nexus Operation             | `schedule_nexus_operation`             | `EVENT_TYPE_NEXUS_OPERATION_SCHEDULED`        |
| Request Nexus Operation Cancellation | `request_nexus_operation_cancellation` | `EVENT_TYPE_NEXUS_OPERATION_CANCEL_REQUESTED` |

## Export 

[Workflow History Export](/cloud/export) enables you to export closed Workflow Histories to a cloud storage sink of your
choice.

- **Workflow exported**. Each Workflow exported accrues a single action.
- Excluded from APS calculations.

| Usage Name              | Metric Name | History Event Type |
| ----------------------- | ----------- | ------------------ |
| Export Workflow History | N/A         | N/A                |

## Fairness 

For each hour a Namespace has the [Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) feature enabled,
an additional `0.1` Action is charged per Action in the Namespace.

- Excluded from APS calculations.

| Usage Name      | Metric Name | History Event Type |
| --------------- | ----------- | ------------------ |
| Enable Fairness | N/A         | N/A                |

## Capacity 

- For Namespace Capacity Temporal Resource Units (TRUs), Actions are generated up to the included hourly allocation for
  TRUs in any hour where TRUs are set and actual usage falls beneath the included hourly Action allocation.
- Excluded from APS calculations.

| Usage Name                  | Metric Name | History Event Type |
| --------------------------- | ----------- | ------------------ |
| Enable Provisioned Capacity | N/A         | N/A                |

Track Actions usage at the granularity you need. See [Billing and Usage](/cloud/billing-and-usage) for the available
views.

---

# System limits

Source: https://docs.temporal.io/evaluate/cloud/limits

> Rate, resource, and configuration limits for Temporal Cloud by scope, which ones you can raise, and what happens when you exceed them.

Temporal Cloud enforces three kinds of limits to keep the service reliable: rate limits (how often something can happen in a unit of time), resource limits (how many of a resource can exist at once), and configuration limits (the minimum or maximum value for a setting).

Every limit applies at one scope:

- The Temporal Cloud [Account level](#account-level)
- The [Project level](#project-level)
- The [Namespace level](#namespace-level)
- The [Nexus Endpoint level](#nexus-endpoint-level)
- The [programming model](#programming-model-level) itself

## Account level

The following limits apply at the Temporal Cloud Account level (per account).

### Users

- Scope: Account
- Default limit: 300 Users
- How to increase: [Contact support](/evaluate/cloud/support#support-ticket) 

### Projects

- Scope: Account
- Default limit: 5 Projects
- How to increase: [Contact support](/evaluate/cloud/support#support-ticket)

### Namespaces

- Scope: Account
- Default limit: 10 Namespaces
- How to increase:
  - Automatically increased as you start creating Namespaces
  - [Contact support](/evaluate/cloud/support#support-ticket) for large-scale needs

### Connectivity Rules 

Without [Projects](/cloud/projects), the Account allows 50 private Connectivity Rules and 1 public Connectivity Rule, for a total of 51.

- Scope: Account
- Default limit: 50 private Connectivity Rules and 1 public Connectivity Rule (51 total)
- How to increase: [Contact support](/evaluate/cloud/support#support-ticket)

With [Projects](/cloud/projects) enabled, each Project has its own Connectivity Rule limits. Connectivity Rules across all Projects still count toward this Account total of 51. See [Project Connectivity Rules](#project-connectivity-rules).

## Project level 

> **Pre-release**

The following limits apply at the Project level for any new Project created when [Projects](/cloud/projects) are enabled. The Default Project uses the Account-level resource limits rather than these per-Project defaults for Namespaces, Service Accounts, and Nexus Endpoints. Connectivity Rule limits are listed below.

### Namespaces and Project-scoped Service accounts

- Scope: Project
- Default: 10
- How to increase: [Contact support](/evaluate/cloud/support#support-ticket)

### Nexus Endpoints

- Scope: Project
- Default: 5
- How to increase: [Contact support](/evaluate/cloud/support#support-ticket)

### Connectivity rules 

These limits apply when [Projects](/cloud/projects) are enabled.

- Each Project allows 1 public Connectivity Rule.
- Each Project allows up to 5 private Connectivity Rules.
- The total number of Connectivity Rules, public and private, across all Projects is subject to the Account limit of 51.
- How to increase: [Contact support](/evaluate/cloud/support#support-ticket)

## Namespace level

The following limits apply at the Namespace level.

### Actions per second

- Scope: Namespace
- Default limit: 500 actions per second (APS). With On-Demand Capacity, the default limit is the floor your APS limit automatically scales above based on the last 7 days of usage; your limit never falls below the default limit. With Provisioned Capacity, your limit is set by the Temporal Resource Units you provision.
- See your current limit: View it in the Temporal Cloud UI under the Namespace overview, retrieve it with the [Temporal CLI](/cli/command-reference/cloud/namespace#capacity) (`temporal cloud namespace capacity get`), or track the `temporal_cloud_v1_action_limit` metric. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits).
- How to increase: See [Capacity Modes](/cloud/capacity-modes) or [contact support](/evaluate/cloud/support#support-ticket).
- What happens when you exceed the limit: See [Throttling behavior](#throttling-behavior) below.

See the [Actions page](/evaluate/cloud/actions) for the list of actions. 

### Requests per second

- Scope: Namespace
- Limit: Your requests per second (RPS) limit is dynamic and depends on your [capacity mode](/cloud/capacity-modes). With On-Demand Capacity it automatically increases (and decreases) based on the last 7 days of RPS usage and never falls below the On-Demand floor; with Provisioned Capacity it is set by the Temporal Resource Units you provision.
- See your current limit: Track the `temporal_cloud_v1_service_request_limit` metric. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits).
- How to increase: See [Capacity Modes](/cloud/capacity-modes) or [contact support](/evaluate/cloud/support#support-ticket).

See the [glossary](/glossary#requests-per-second-rps) for more about RPS.

### Operations per second

- Scope: Namespace
- Limit: Your operations per second (OPS) limit is dynamic and depends on your [capacity mode](/cloud/capacity-modes). With On-Demand Capacity it automatically increases (and decreases) based on the last 7 days of OPS usage and never falls below the On-Demand floor; with Provisioned Capacity it is set by the Temporal Resource Units you provision.
- See your current limit: Track the `temporal_cloud_v1_operations_limit` metric. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits).
- How to increase: See [Capacity Modes](/cloud/capacity-modes) or [contact support](/evaluate/cloud/support#support-ticket).

See the [operations list](/references/operation-list) for the list of operations.

### Throttling behavior

When you exceed your APS, RPS, or OPS limits, Temporal Cloud throttles requests:

1. **Priority-based throttling**: Low-priority operations are throttled first. Higher-priority operations such as `StartWorkflowExecution`, `SignalWorkflowExecution`, and `UpdateWorkflowExecution` continue when possible. Temporal Cloud uses [throttling priorities similar to the open source server](https://github.com/temporalio/temporal/blob/main/service/frontend/configs/quotas.go#L66).
2. **Throttling latency**: Rate limiting is not instantaneous, so usage can briefly exceed your limit before throttling takes effect.
3. **ResourceExhausted errors**: A throttled request returns a `ResourceExhausted` gRPC error, which SDK clients retry under the default gRPC retry policy.
4. **Potential failure**: If throttling outlasts the SDK's retry limit, client calls fail, and work _can_ be lost if you don't handle those failures.

To handle throttling:

- Log every failed `StartWorkflowExecution`, `SignalWorkflowExecution`, and `UpdateWorkflowExecution` call on the Client side, including the Payload, so you can retry or backfill it later.
- Alert on [Cloud metrics](/cloud/metrics/openmetrics/metrics-reference#limit-metrics) when throttling occurs and when you approach a limit.
- Switch to [Provisioned Capacity](/cloud/capacity-modes#provisioned-capacity) if you have predictable spikes or need guaranteed throughput.

### Schedules rate limit

- Scope: Namespace
- Default limit: 10 schedule requests per second (RPS)
- How to increase: [Contact support](/evaluate/cloud/support#support-ticket)

To avoid throttling, don't schedule all your Workflow Executions to start at the same time (such as daily, weekly, or monthly).
Every Temporal SDK supports jittering, which adds small random delays to Schedule specifications, helping to reduce load at any specific moment.
Set the `jitter` value to the largest delay you will permit before your Workflow Execution must begin.
This approach uniformly distributes the scheduled Workflow Execution launches through that period and reduces your Schedule Workflow Execution RPS load.

### Per-primitive Id reuse rate limits 

- Scope: Namespace, primitive type, and Id
- Primitives covered: Workflows, Standalone Activities, Schedules, and Nexus Operations
- Default limit: 1 new Execution per second with a burst allowance
- Not configurable

When executing [Workflows](/workflows), [Standalone Activities](/standalone-activity), and [Nexus Operations](/standalone-nexus-operation), and when creating [Schedules](/schedule), the system enforces a per-primitive limit on successive Executions with the same Id.

Every operation that creates an Execution counts toward the limit:

- `StartWorkflowExecution`
- `SignalWithStartWorkflowExecution`
- `ExecuteMultiOperation` (Update)
- `ResetWorkflowExecution`
- [Continue-As-New](/workflow-execution/continue-as-new), whether requested by the Workflow or triggered by a [Retry Policy](/encyclopedia/retry-policies) or [Cron Job](/cron-job)
- `CreateSchedule`
- `StartActivityExecution`
- `StartNexusOperationExecution`

This limit applies only to requests that will create new Executions, and so this limit applies after the deduplication check.
A duplicate start that returns `WorkflowExecutionAlreadyStarted` has no effect on the limit.

Exceeding this limit returns a `ResourceExhausted` error.

### Visibility API rate limit

- Scope: Namespace
- Default limit: 30 Visibility API calls per second
- Not configurable

The Visibility API rate limit applies to every read API that lists, searches, or counts across executions. These are:

- **Workflow search and count:** `ListWorkflowExecutions`, `ListOpenWorkflowExecutions`, `ListClosedWorkflowExecutions`, `ScanWorkflowExecutions`, `CountWorkflowExecutions`
- **Schedules:** `ListSchedules`, `CountSchedules`
- **Batch operations:** `ListBatchOperations`
- **Task Queue and worker reachability:** `GetWorkerTaskReachability`, and `DescribeTaskQueue` (subject to this limit since Server v1.24)
- **Worker deployments:** `ListDeployments`, `GetDeploymentReachability`, `ListWorkerDeployments`

Any API that returns a set of executions or a count falls under this limit. Single-entity lookups by ID, such as `DescribeWorkflowExecution` and `DescribeSchedule`, do not. Those read the primary store and count against the general namespace rate limit.

### Nexus rate limit 

Nexus requests (such as starting a Nexus Operation or sending a Nexus completion callback) are counted as part of the overall Namespace RPS limit.
If too many Nexus requests are sent at once, they may be throttled, along with other requests to the Namespace.
Throttling limits the rate at which Nexus requests are processed, ensuring the RPS limit isn't exceeded.

You can request this limit be manually raised by [opening a support ticket](/evaluate/cloud/support#support-ticket).

> **📝 Note:**
>
> For the target Namespace of a Nexus Endpoint, even though there are no Action results for handling a Nexus Operation itself, the Nexus requests on a target Namespace do count towards the overall RPS limit for the Namespace as a whole.
>

### Certificates

Temporal Cloud limits each Namespace to a total of 32 KB or 16 certificates, whichever is reached first.

### Concurrent Task pollers

Temporal Cloud limits each Namespace to 20,000 Activity pollers and 20,000 Workflow Task pollers concurrently. 

Each SDK offers a way to configure Workers for per-Worker maximum Activity and Workflow Task pollers.
Those values do not affect the global Namespace limit.

### Default Retention Period

The [Retention Period](/temporal-service/temporal-server#retention-period) is set per Namespace.

Temporal Cloud sets the default Retention Period to 30 days.
This is configurable in the Temporal Web UI.

[Navigate to your list of Namespaces](https://cloud.temporal.io/namespaces), choose the Namespace you want to update, and select edit:

![Choose your Namespace and select Edit](https://docs.temporal.io/img/cloud/cloud-guide/edit-namespace-option.png)

![Update the Retention Period](https://docs.temporal.io/img/cloud/cloud-guide/edit-retention-period.png)

You can set the Retention Period between 1 and 90 days.

### Batch jobs

A Namespace can have just one [Batch job](/cli/command-reference/batch) running at a time.

Each batch job operates on a maximum of 50 Workflow Executions per second.

### Number of Custom Search Attributes

There is a limit to the number of custom Search Attributes per attribute type per Namespace:

| Search Attribute type | Limit |
| --------------------- | ----- |
| Bool                  | 20    |
| Datetime              | 20    |
| Double                | 20    |
| Int                   | 20    |
| Keyword               | 40    |
| KeywordList           | 5     |
| Text                  | 5     |

### Custom Search Attribute names

When creating custom Search Attributes in Temporal Cloud, the attribute names must adhere to the following constraints:

- Maximum characters: 64
- Allowed characters: `[a-zA-Z0-9.,:-_\/@ ]`.

For more information on custom Search Attributes see [Custom Search Attributes limits](/search-attribute#custom-search-attribute).

### Custom Roles limits 

Each account can create up to 25 [Custom Roles](/cloud/manage-access/custom-roles).

The UI can display up to 100 Custom Roles.

A single principal (such as a user) can be assigned up to 10 Custom Roles.

A Custom Role can contain up 20 permissions. Each permission is a pair of
- A list of resource IDs of a single type (for example, specific Namespace IDs)
- A list of actions that apply to those resources (for example, cloud.namespace.get, cloud.namespace.update)
For example, 
- 1 permission: Namespaces [ns-prod, ns-staging] with actions [cloud.namespace.get, cloud.namespace.update] 
- 2 permissions: Namespace [ns-prod] with action [cloud.namespace.get]; Namespace [ns-staging] with action [cloud.namespace.update] defined separately.

There is no limit on the number of resource IDs or actions within a single permission pair. Applying a permission to all resources of a type counts as one permission regardless of how many resources exist in your account.

## Nexus Endpoint level

### Nexus Endpoints limits

By default, each account is provisioned with 100 Nexus Endpoints.
You can request further increases beyond the initial 100 Endpoint limit by [opening a support ticket](/evaluate/cloud/support#support-ticket).

### Nexus caller Namespace limits 

By default, a single Nexus Endpoint can have a maximum of 1,000 caller Namespaces in its [Access Policy](/nexus/security#runtime-access-controls), the allowlist of Namespaces permitted to use the Endpoint.
You can request further increases beyond the initial 1,000 caller Namespace limit by [opening a support ticket](/evaluate/cloud/support#support-ticket).

## Programming model level

The following limits apply at the programming model level.
See also: [Self-hosted Temporal Service defaults](/self-hosted-guide/defaults).

### Identifier length limit

Identifiers, such as Workflow Id, Workflow Type, and Task Queue names, are limited to a maximum length of 1,000 bytes.
Note that Unicode characters may use multiple bytes.

### Per message gRPC limit

Each gRPC message received has a limit of 4 MB.
This limit applies to all gRPC endpoints across the Temporal Platform.

### Event History transaction size limit

An Event History transaction encompasses a set of operations such as initiating a new Workflow, scheduling an Activity, processing a Signal, or starting a Child Workflow.
These operations create Events that are then logged in the Event History.
The transaction size limit restricts the total size of Events that can be accommodated within a single transaction.

The size limit for any given [Event History](/workflow-execution/event#event-history) transaction is 4 MB.
This limit is non-configurable for Temporal Cloud.

### Transaction Payload size limit

Blob size limit for Payloads, including Workflow context and each Workflow and Activity argument and return value:
  - The max payload for a single request is 2 MB.
  - The max size limit for any given [Event History](/workflow-execution/event#event-history) transaction is 4 MB.

This limit is non-configurable for Temporal Cloud.

The [BlobSizeLimitError guide](/troubleshooting/blob-size-limit-error) provides solutions for handling large payloads.

> **📝 Note:**
>
> The 2 MB limit is the maximum size for a single request, not a sustained-throughput target.
> Sustained, high-volume use of large payloads can degrade Namespace performance, and Temporal reserves the right to rate limit such traffic to protect the service.
> For workloads that regularly produce large payloads, offload them to [External Storage](/external-storage) and keep inline payloads small.
>

### Per Workflow Execution concurrency limits

If a Workflow Execution has 2,000 incomplete Activities, Signals, Child Workflows, or external Workflow Cancellation requests, additional [Commands](/workflow-execution#command) of that type will fail to be applied to that Workflow Execution:

- `ScheduleActivityTask`
- `SignalExternalWorkflowExecution`
- `StartChildWorkflowExecution`
- `RequestCancelExternalWorkflowExecution`

For optimal performance, limit concurrent operations to 500 or fewer.
This reduces Workflow's Event History size and decreases the loading time in the Web UI.

### Per Workflow Execution Signal limit

A single Workflow Execution may receive up to 10,000 Signals.
After that limit is reached, no more Signals will be processed for that Workflow Execution.

### Per Workflow Execution Update limits

A single Workflow Execution can have a maximum of 10 in-flight Updates and 2000 total Updates in History.

### Workflow Execution Event History limits

As a precautionary measure, a Workflow Execution's Event History is limited to 51,200 Events or 50 MB.
It warns you after 10,240 Events or 10 MB.
This limit applies to all Temporal Workflow Executions, whether on Temporal Cloud or other deployments.

This limit is non-configurable for Temporal Cloud.

Read more about [Temporal Workflow Execution limits](/workflow-execution/limits) on the [Temporal Workflow](/workflows) documentation page.

### Per Workflow Callback limits 

A single Workflow Execution can have a maximum of 2000 total Callbacks.

These limits may be exceeded when [multiple Nexus callers attach to the same handler Workflow](/nexus/operations#attaching-multiple-nexus-callers).

See the Nexus Encyclopedia entry for [additional details](/workflow-execution/limits#workflow-execution-callback-limits).

### Per Workflow Nexus Operation limits 

A single Workflow Execution can have a maximum of 30 in-flight Nexus Operations.

See the Nexus Encyclopedia entry for [additional details](/workflow-execution/limits#workflow-execution-nexus-operation-limits).

### Nexus Operation request timeout 

Less than 10 seconds is the maximum duration for a Nexus handler to process a single Nexus start or cancel request.

The timeout is measured from the calling History Service and the request must go through matching, so the available time for a handler to respond is often much less than 10 seconds.
Handlers should observe the context deadline and ensure they don't exceed it.
This includes fully processing a synchronous Nexus operation and starting an asynchronous Nexus operation, for example one that starts a Workflow.

If a Nexus handler doesn’t process a start or cancel request within 10 seconds, it will receive a context deadline exceeded error, and the caller will retry, with an exponential backoff, for the ScheduleToClose duration for the overall Nexus Operation.
This has a default and maximum as defined below in [Nexus Operation duration limits](/evaluate/cloud/limits#nexus-operation-duration-limits).

### Nexus Operation duration limits 

Each Nexus Operation has a maximum ScheduleToClose duration of 60 days.
This is most applicable to asynchronous Nexus Operations completed with an asynchronous callback using a separate Nexus request from the handler back to the caller Namespace.

For enhanced security, you may sign completion callbacks with a single-use token in the future, and the 60 day maximum allows you to rotate the asymmetric encryption keys used for completion callback request signing.

While the caller of a Nexus Operation can configure the ScheduleToClose duration to be shorter than 60 days, the maximum duration can not extend beyond 60 days and capped by the server to 60 days.

### Timer duration limit

Timers have a maximum duration of 100 years in Temporal Cloud.

## Worker Versioning level

### Max Worker deployments limits 

The maximum number of Worker deployments that the server allows to be registered in a single Namespace. Defaults to 100.

### Max versions in deployment limits 

The maximum number of versions that the server allows to be registered in a single Worker deployments at a given time. Note that unused versions will be deleted by the system automatically when this limit is reached. Defaults to 100.

### Max Task Queues in Deployment Version limits 

The maximum number of Task Queues that the server allows to be registered in a single Worker Deployment Version. Defaults to 100.

> **⚠️ Caution:**
>
> **Do not enable Worker Versioning for Task Queues that are generated per Worker** (for example, a Task Queue that has a UUID in its name) — this pattern will quickly exhaust the limit.
> A per-worker Task Queue's lifetime is tied to the Worker itself. When the Worker is updated to a new version, it gets a new Task Queue and the old one is gone, so versioning is not relevant.
>

---

# Service regions

Source: https://docs.temporal.io/evaluate/cloud/regions

> Supported AWS and GCP regions for Temporal Cloud Namespaces, any current restrictions on creating them, and how region choice affects latency to your Workers.

You can reach Temporal Cloud from anywhere with Internet connectivity, wherever your Namespaces are located, and your applications can run in the cloud environment or data center of your choice.
Create Namespaces in a region close to where you host your Workers to reduce latency.

This page lists supported Temporal Cloud regions and any restrictions on creating Namespaces.

> **💡 Tip:**
> Service availability
>
> Check the status of supported regions at [status.temporal.io](https://status.temporal.io).
> Subscribe there to get an email whenever Temporal creates, updates, or resolves an incident.
>

## GCP capacity restriction 

> **⚠️ Caution:**
>
> New Temporal Cloud Namespace provisioning is temporarily restricted in Iowa (`us-central1`), Northern Virginia (`us-east4`), and Frankfurt (`europe-west3`) due to limited GCP compute capacity.
> Provisioning is available to all accounts in Mumbai (`asia-south1`), Jakarta (`asia-southeast2`), and Oregon (`us-west1`).
>

Existing GCP Namespaces continue to operate as normal.
For Namespaces using [Provisioned Capacity](/cloud/capacity-modes#provisioned-capacity), additional Temporal Resource Units (TRUs) are limited and may be unavailable, depending on the region.

If your new Namespace must be in a restricted GCP region, [create a support ticket](/evaluate/cloud/support#support-ticket) to request an exception for your account.
Include the required GCP regions, expected Action volume, and business requirements for your workload.

Multi-Cloud Replication to Mumbai (`asia-south1`) and Oregon (`us-west1`) is available for the [supported region pairs](#gcp-service-regions) without an account exception.
Jakarta (`asia-southeast2`) does not support High Availability features.
To add a replica in a restricted GCP region using Multi-Region Replication or Multi-Cloud Replication, [create a support ticket](/evaluate/cloud/support#support-ticket), even if your account already has an approved exception.
In restricted regions, an approved exception enables self-service creation of single-region GCP Namespaces only; adding a GCP replica still requires assistance from Temporal Dev Success.
You can still add an AWS replica to an existing GCP Namespace without an exception, where that replication option is [supported](#gcp-service-regions).
See [Enable and manage High Availability](/cloud/high-availability/enable) for instructions.

You can also deploy new workloads in an [AWS region](#aws-service-regions).

## AWS service regions

Temporal Cloud operates in the following Amazon Web Services (AWS) regions:

### Asia Pacific - Tokyo (`ap-northeast-1`)

- **Cloud API Code**: `aws-ap-northeast-1`
- **Regional Endpoint**: `ap-northeast-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Seoul (`ap-northeast-2`)

- **Cloud API Code**: `aws-ap-northeast-2`
- **Regional Endpoint**: `ap-northeast-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Mumbai (`ap-south-1`)

- **Cloud API Code**: `aws-ap-south-1`
- **Regional Endpoint**: `ap-south-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Hyderabad (`ap-south-2`)

- **Cloud API Code**: `aws-ap-south-2`
- **Regional Endpoint**: `ap-south-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Singapore (`ap-southeast-1`)

- **Cloud API Code**: `aws-ap-southeast-1`
- **Regional Endpoint**: `ap-southeast-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Asia Pacific - Sydney (`ap-southeast-2`)

- **Cloud API Code**: `aws-ap-southeast-2`
- **Regional Endpoint**: `ap-southeast-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08`
- **Same Region Replication**: Available
- **Multi-Region Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
- **Multi-Cloud Replication**:
  - `gcp-asia-south1`

### Europe - Frankfurt (`eu-central-1`)

- **Cloud API Code**: `aws-eu-central-1`
- **Regional Endpoint**: `eu-central-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-eu-west-1`
  - `aws-eu-west-2`
- **Multi-Cloud Replication**:
  - `gcp-europe-west3`

### Europe - Ireland (`eu-west-1`)

- **Cloud API Code**: `aws-eu-west-1`
- **Regional Endpoint**: `eu-west-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-eu-central-1`
  - `aws-eu-west-2`
- **Multi-Cloud Replication**:
  - `gcp-europe-west3`

### Europe - London (`eu-west-2`)

- **Cloud API Code**: `aws-eu-west-2`
- **Regional Endpoint**: `eu-west-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-eu-central-1`
  - `aws-eu-west-1`
- **Multi-Cloud Replication**:
  - `gcp-europe-west3`

### North America - Central Canada (`ca-central-1`)

- **Cloud API Code**: `aws-ca-central-1`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d`
- **Regional Endpoint**: `ca-central-1.aws.api.temporal.io:7233`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### North America - Northern Virginia (`us-east-1`)

- **Cloud API Code**: `aws-us-east-1`
- **Regional Endpoint**: `us-east-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f`
- **Same Region Replication**:  Available
- **Multi-Region Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-2`
  - `aws-us-west-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### North America - Ohio (`us-east-2`)

- **Cloud API Code**: `aws-us-east-2`
- **Regional Endpoint**: `us-east-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-west-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### North America - Oregon (`us-west-2`)

- **Cloud API Code**: `aws-us-west-2`
- **Regional Endpoint**: `us-west-2.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94`
- **Same Region Replication**:  Available
- **Multi-Region Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
- **Multi-Cloud Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
  - `gcp-us-east4`

### South America - São Paulo (`sa-east-1`)

- **Cloud API Code**: `aws-sa-east-1`
- **Regional Endpoint**: `sa-east-1.aws.api.temporal.io:7233`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - None

## GCP service regions

Temporal Cloud operates in the following Google Cloud (GCP) regions:

> **⚠️ Caution:**
> GCP capacity restriction
>
> Provisioning is available in Mumbai (`asia-south1`), Jakarta (`asia-southeast2`), and Oregon (`us-west1`).
> New Namespaces in Iowa (`us-central1`), Northern Virginia (`us-east4`), and Frankfurt (`europe-west3`) require an approved account exception.
> Adding a replica in a restricted region also requires assistance from Temporal Dev Success, even with an approved exception.
> See [GCP capacity restriction](#gcp-capacity-restriction).
>

### North America - Iowa (`us-central1`)

- **Cloud API Code**: `gcp-us-central1`
- **Regional Endpoint**: `us-central1.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `gcp-us-west1`
  - `gcp-us-east4`
- **Multi-Cloud Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`

### North America - Oregon (`us-west1`)

- **Cloud API Code**: `gcp-us-west1`
- **Regional Endpoint**: `us-west1.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `gcp-us-central1`
  - `gcp-us-east4`
- **Multi-Cloud Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`

### North America - Northern Virginia (`us-east4`)

- **Cloud API Code**: `gcp-us-east4`
- **Regional Endpoint**: `us-east4.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - `gcp-us-central1`
  - `gcp-us-west1`
- **Multi-Cloud Replication**:
  - `aws-ca-central-1`
  - `aws-us-east-1`
  - `aws-us-east-2`
  - `aws-us-west-2`

### Europe - Frankfurt (`europe-west3`)

- **Cloud API Code**: `gcp-europe-west3`
- **Regional Endpoint**: `europe-west3.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - `aws-eu-central-1`
  - `aws-eu-west-1`
  - `aws-eu-west-2`

### Asia Pacific - Mumbai (`asia-south1`)

- **Cloud API Code**: `gcp-asia-south1`
- **Regional Endpoint**: `asia-south1.gcp.api.temporal.io:7233`
- **Private Service Connect Service Attachment URI**: `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - `aws-ap-northeast-1`
  - `aws-ap-northeast-2`
  - `aws-ap-south-1`
  - `aws-ap-south-2`
  - `aws-ap-southeast-1`
  - `aws-ap-southeast-2`

### Asia Pacific - Jakarta (`asia-southeast2`)

- **Cloud API Code**: `gcp-asia-southeast2`
- **Regional Endpoint**: `gcp-asia-southeast2.region.tmprl.cloud`
- **Private Service Connect Service Attachment URI**: `projects/prod-bsbyrfwqqq885qkcr3s43y524/regions/asia-southeast2/serviceAttachments/pl-c3ayi`
- **Same Region Replication**:  Not Available
- **Multi-Region Replication**:
  - None
- **Multi-Cloud Replication**:
  - None

---

# Security model

Source: https://docs.temporal.io/evaluate/cloud/security

> Namespace isolation, mTLS and API key authentication, client-side Payload encryption, private connectivity, access control, and SOC 2 Type 2 compliance.

The security model of [Temporal Cloud](/cloud) covers three things: your applications, your data, and the Temporal Cloud platform itself.

> **ℹ️ Info:**
> General platform security
>
> For the security features of Temporal that aren't specific to Cloud, see [Platform security](/security).
>

## Applications and data 

### Code execution boundaries

Temporal Cloud runs the Temporal Server as a managed service.
It does not run or manage your applications or [Workers](/workers#worker).
Applications written with a [Temporal SDK](/encyclopedia/architecture/temporal-sdks) run in your computing environment, whether that's containers (Docker, Kubernetes) or virtual machines in any hosting environment, so you control how they're secured.

### Data Converter: client-side encryption

The optional [Data Converter](/dataconversion) encrypts data before it's sent to Temporal Cloud and decrypts it on the way back.

Data conversion runs on your Workers and [Clients](/encyclopedia/temporal-client), so Temporal Cloud cannot see or decrypt your data.
Data stored in Temporal Cloud stays encrypted even if the service itself is compromised.

Deploy a [Codec Server](/production-deployment/data-encryption) to decrypt data in the [Temporal Web UI](/web-ui) without sharing encryption keys with Temporal.

Alternatively, [Temporal Proxy](/production-deployment/temporal-proxy) encrypts payloads in your environment before
forwarding them to Temporal Cloud, without changing Worker or Client code. The proxy has access to plaintext payloads
and your KMS; Temporal Cloud receives encrypted payloads. See the proxy's
[encryption guide](/production-deployment/temporal-proxy/encrypt-payloads) for configuration and coverage limits.

## The platform 

### Namespace isolation

The base unit of isolation in Temporal is a [Namespace](/namespaces).
A Temporal Cloud account can hold many Namespaces, and each one is isolated from the rest.

#### Authentication

Each Namespace is secured with the authentication method you choose:

- **mTLS certificates**: Namespace-specific X.509 certificates for mutual TLS authentication
- **API keys**: Namespace-scoped keys for bearer-token authentication

See [API Keys](/cloud/api-keys) and [mTLS Certificates](/cloud/certificates) to configure authentication for a Namespace.

#### Rate limiting

Each Namespace has its own rate limits, which keep one Namespace from becoming a noisy neighbor:

- **Actions Per Second (APS)** limits the rate of [Actions](/best-practices/managing-aps-limits) performed in your Workflows
- **Operations Per Second (OPS)** limits the rate of all [operations](/references/operation-list) that put load on the Temporal Server

A traffic spike in one Namespace cannot affect the performance or reliability of another, whether the two belong to the same Temporal Cloud account or to different accounts.

See [Rate limiting](/evaluate/cloud/limits) for more information about Temporal Cloud limits, and [Monitoring trends against limits](/cloud/service-health#rps-aps-rate-limits) for monitoring best practices.

#### Inter-Namespace communication

Namespaces are isolated by default.
Workflows in one Namespace reach Workflows in another only through [Temporal Nexus](/nexus), which routes cross-Namespace calls through Nexus Endpoints you control.

See [Nexus Security](/nexus/security) for how Nexus secures that communication.

#### Logical segregation

Temporal Cloud is a multi-tenant service.
Namespaces in the same environment are logically segregated.
Namespaces do not share data processing or data storage across regional boundaries.

### Private connectivity

Connect to Temporal Cloud from a secured network without traversing the public internet.

See [Connectivity](/cloud/connectivity) for how to use AWS PrivateLink and GCP Private Service Connect with Temporal Cloud.

### Temporal Nexus

Like Namespaces, the Nexus Registry is account-scoped and global within a Temporal Cloud account.
Nexus Endpoint names remain unique Account-wide, but each Endpoint is Project-scoped for management.
An Account Developer (or higher), or a user with Project Contribute (or higher) on the Endpoint's Project, can manage (create, update, or delete) an Endpoint only when they also have Namespace Admin on its target Namespace.
Changing an Endpoint's target Namespace requires Namespace Admin on both the current target Namespace and the new target Namespace.
All users with a Read-only role (or higher) in an account, can view and browse the full list of Endpoints.

Runtime access from a Workflow in a caller Namespace to a Nexus Endpoint is controlled by an allowlist policy (of caller Namespaces) for each Endpoint in the Nexus API registry.
Workers authenticate with Temporal Cloud as they do today with mTLS certificates or API keys as allowed by the Namespace configuration.
Nexus requests are sent from the caller’s Namespace to the handler’s Namespace over a secure multi-region mTLS Envoy mesh.

For payload encryption, the DataConverter works the same for a Nexus Operation as it does for other payloads sent between a Worker and Temporal Cloud.

See [Nexus Security](/nexus/security) for more information.

### Encryption

> **💡 Tip:**
> TLS vs mTLS
>
> **TLS** (Transport Layer Security) encrypts data in transit. **mTLS** (mutual TLS) is an authentication method where both client and server present certificates to verify identity. All Temporal Cloud connections use TLS encryption. When you choose "mTLS authentication," you're choosing how to prove your identity, not whether your connection is encrypted.
>

**In transit**: All connections to Temporal Cloud use TLS 1.2 or higher (TLS 1.3 preferred), regardless of your authentication method ([API keys](/cloud/api-keys) or [mTLS certificates](/cloud/certificates)).

**At rest**: Data is stored in two locations: an Elasticsearch instance (used when filtering Workflows in SDK clients, the [CLI](/cloud/tcld), or the Web UI) and the core Temporal Cloud persistence layer.
Both are encrypted at rest with AES-256-GCM.

### Identity

Authentication to Temporal Cloud gRPC endpoints supports two methods:

- **[API keys](/cloud/api-keys)**: Identity-based authentication using bearer tokens. Recommended for most cases.
- **[mTLS certificates](/cloud/certificates)**: Mutual TLS authentication using client certificates issued by your CA.

Both connections are encrypted, so choose based on your organization's security requirements and key management preferences.

For user authentication to the Temporal Cloud UI, see [SAML authentication](/cloud/manage-access/saml).

### Access

Authorization is managed at the account and Namespace level, and users and systems are assigned one or more preconfigured roles.
Users hold [account-level Roles](/cloud/manage-access/roles-and-permissions#account-level-roles) of administrator, developer, or read-only.
Systems and application processes hold their own distinct roles.

### Monitoring

Alongside system monitoring for operations and availability, Temporal collects audit logs from the AWS environment and from every call to the gRPC API, which the SDKs, CLI, and Web UI use.
Those audit logs can be exported to your own security monitoring system.

### Testing

Temporal contracts with a third party for an annual full-scope pentest, excluding social engineering.
Targeted third-party and internal testing happens as needed, such as when a significant feature is released.

### Internal Temporal access

Access to production systems is restricted to the team that maintains the production infrastructure.
All access is logged, shared accounts are not allowed, and every production system requires SSO with MFA.

Access to Temporal's cloud environments is granted for a limited period, up to a maximum of 8 hours.
For the reasoning behind that, see [Rolling out access hours at Temporal](https://temporal.io/blog/rolling-out-access-hours-at-temporal).

Temporal engineering systems are secured by GitHub credentials, which require both membership in the Temporal GitHub organization and MFA.
Access grants are reviewed quarterly.

### Compliance

Temporal Technologies is SOC 2 Type 2 certified and compliant with GDPR and HIPAA regulations.
Request compliance audits through the [Contact](https://pages.temporal.io/contact-us) page.

---

# Services, support, and training

Source: https://docs.temporal.io/evaluate/cloud/support

> Response times, channels, and technical guidance included with each Temporal Cloud plan, plus how to open a ticket in the Temporal Support Portal.

Every Temporal Cloud plan includes support, and the plan you choose sets the response times, the channels you can reach Temporal through, and the depth of technical guidance you receive.
Coverage ranges from break/fix support and general guidance to code reviews, pre-production optimization, and operational readiness.

> **📝 Note:**
>
> This page applies to Temporal Cloud customers only.
>

## Services offered by Temporal Cloud 

|                             | Developer                                                                       | Business                                                                                                                                                            | Enterprise                                                                                      | Mission Critical                                                                                                                  |
| --------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Support Staff               | Trained staff providing break-fix support and general guidance.                 | Trained staff providing break-fix support and general guidance.                                                                                                     | Developer experts who provide advanced support                                                  | Developer experts who provide advanced support                                                                                    |
| Technical Guidance          | Core platform config, platform access, documented features, and basic inquiries | Advanced technical support, Workflow troubleshooting, SDK implementations, and Worker configuration, Quarterly code review or design implementation best practices. | Business+ expert-led code reviews and design implementation best practices, available as needed | Enterprise+ expert guidance on Workflow latency monitoring and optimization; performance recommendations based on real time tests |
| Billing & Cost Optimization | Generic Billing Questions                                                       | Generic Billing Questions                                                                                                                                           | Quarterly review of spend                                                                       | Quarterly review of spend, proactive cost optimization                                                                            |

## Temporal Cloud support guarantees 

Temporal Cloud customers get break/fix support with an agreed set of SLAs for prioritized issues, entered, tracked, and closed through a ticketing system.
Depending on your plan, you can also reach support through a dedicated Slack channel, the community forums, and the knowledge base.

Availability and response times by plan:

|                                                  | Developer                                                                                                      | Business                                                                                                           | Enterprise                                                                                                   | Mission Critical                                                                                             |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| **Availability**<br />(Based on<br />Time-zones) | **P0–3**: 9–5 Mon–Fri                                                                                          | **P0–3**: 9–5 Mon–Fri                                                                                              | **P0**: 24×7, <br />(On Page Service)<br /> **P1–3**: 9–5 Mon–Fri                                            | **P0**: 24×7 (On Page Service)<br /> **P1**: 9-5, 7 days/week<br /> **P2–3**: Mon–Fri                        |
| **Response Time**                                | **P0**: 1 business day<br /> **P1**: 1 business day<br /> **P2**: 1 business day<br /> **P3**: 2 business days | **P0**: 2 business hours<br /> **P1**: 2 business hours<br /> **P2**: 1 business day<br /> **P3**: 2 business days | **P0**: 30 minutes<br /> **P1**: 1 business hour<br /> **P2**: 4 business hours<br /> **P3**: 1 business day | **P0**: 15 minutes<br /> **P1**: 1 business hour<br /> **P2**: 4 business hours<br /> **P3**: 1 business day |
| **Dedicated Platform Architect**                                          | -                                                                                                              | -                                                                                                                  | Add-on                                                                                                       | Included (1 Unit)                                                                                        |
| **Channels**                                     | Community<br/>Temporal Support Portal                                                                          | Community<br />Temporal Support Portal                                                                             | Community<br />Temporal Support Portal<br />Private Slack                                                    | Community<br />Temporal Support Portal<br />Private Slack                                                    |

> **ℹ️ Info:**
> Business Hours Timezones
>
> Business Hours will be specified in your contract, including one of three locations: US Pacific time, European Central time, Australia Eastern time
>

**Priority definitions**

- **P0 - Critical** (Production impacted)
  - The Temporal Cloud service is unavailable or degraded with a significant impact.
- **P1 - High** (Production issue)
  - An issue related to production workloads running on the Temporal Cloud service, or a significant project is blocked.
- **P2 - Normal** (General issues)
  - General Temporal Cloud service or other issues where there is no production impact, or a workaround exists to mitigate the impact.
- **P3 - Low** (General guidance)
  - Questions or an issue with the Temporal Cloud service that is not impacting system availability or functionality.

> **📝 Note:**
> On Page Service
>
> P0: 24×7 (On Page Service) is offered for Enterprise and Mission Critical accounts.
>

For the cost of each support level, see the [pricing page](/cloud/pricing).

## Temporal dedicated platform architect 

<a id="dedicated-support-engineer"></a>

Mission Critical customers, and Enterprise customers who opt in, get a Dedicated Platform Architect:

- A senior developer expert joins your Temporal account team, and because they already know your implementation, issues resolve faster.
  - Advisory on best practices, and development pairing on your code.
  - Regular reviews with recommendations to improve performance and efficiency.
  - Input on implementation design and operations.
- Priority access to that engineer for up to 20 hours per month, for one business unit or major group within a single region.

Engineers are assigned in your local time zone.
Buy additional service units to cover more groups or regions at &#36;6,000/Mo/Unit.
One unit of Mission Critical Support includes:

- Up to 20 hours per month
- One major group or business unit
- Limited to one region
- Quarterly onsite visits

## Ticketing

Temporal Cloud customers file tickets in the Temporal Support Portal.
For questions outside a ticket, use the [community Slack](https://temporalio.slack.com) and the [community Discourse forum](https://community.temporal.io/).

> **ℹ️ Info:**
>
> The Temporal Support Portal is for Cloud customers only, and all Cloud customers pay for support as part of their plan.
> Other Temporal users have full community access except for the `#support-cloud` channel.
>

### Access Temporal support

1. Go to [support.temporal.io](https://support.temporal.io/).
2. If prompted, log in to Temporal Cloud with the method you normally use, such as Google, Microsoft, or email and password.
3. View open and closed tickets for your Temporal account, or submit a new one.

To request assistance from Temporal Support, see [Create a ticket](#support-ticket).

### Create a ticket 

> **ℹ️ Info:**
>
> This procedure applies only to Temporal Cloud customers whose contracts include paid support.
> Without paid support, post your request in the [Temporal Community Forum](https://community.temporal.io) or the `#support-cloud` channel of the [Temporal workspace](https://t.mp/slack) in Slack.
>

To create a ticket in the Temporal Support Portal:

1. Go to [support.temporal.io](https://support.temporal.io/).
2. If prompted, log in to Temporal Cloud using the same method you normally use (for example, Google, Microsoft, email-password, or other methods).
3. Click the **Create Ticket** button in the top right corner.
4. On the **Submit a ticket** page, enter the details of your request into the form. **Name**, **Subject**, and **Description** are required.
5. At the bottom of the form, choose **Submit**.

## Developer resources 

Hands-on material for getting started and for going deeper:

- [Get started with Temporal](https://learn.temporal.io/getting_started): Set up a development environment, run an existing Temporal application, then build one from scratch with an SDK.
- [Courses](https://learn.temporal.io/courses): Free, self-paced courses that apply Temporal concepts in code.
- [Tutorials](https://learn.temporal.io/tutorials): Build real-world applications step by step.
- [Example applications](https://learn.temporal.io/examples): Complete applications that show how Temporal concepts fit together at scale.

---

# Temporal development and production features

Source: https://docs.temporal.io/evaluate/features

> Leverage a Temporal SDK to develop robust applications featuring Workflows, Activities, Workers, testing suites, Temporal Clients, Scheduled Workflows, and advanced observability tools.

Through a Temporal SDK, Temporal provides a wide range of features that enable developers to build applications that serve a wide range of use cases.

- [Core primitives](/evaluate/features/core-primitives): Develop and run your application with Workflows, Activities, and Workers.
- [Timeouts and retries](/evaluate/features/timeouts-and-retries): Detect failures with timeouts and configure automatic retries to mitigate them.
- [Child Workflows](/evaluate/features/child-workflows): Break up business processes by data streams, team ownership, or other organization factors.
- [Temporal Nexus](/evaluate/features/nexus): Connect Temporal Applications across (and within) isolated Namespaces for improved modularity, security, debugging, and fault isolation. Nexus supports cross-team, cross-domain, and multi-region use cases.
- [Workflow message passing](/evaluate/features/workflow-message-passing): Build responsive applications that react to events at runtime and enable data retrieval from ongoing Workflows.
- [Cancellation and termination](/evaluate/features/cancellation-and-termination): Cancel or terminate a business process (Workflow) that is already in progress and compensate for any steps already taken.
- [Observability](/evaluate/features/observability): List business processes, view their state, and set up dashboards with metrics.
- [Data encryption](/evaluate/features/data-encryption): Transform data and protect the privacy of the users of your application.
- [Scheduled Workflows](/evaluate/features/schedules): Start a business process at a specific time or on a given time interval.
- [Job queues](/evaluate/features/job-queue): Run any Activity as a durable background job, without the overhead of a Workflow.

Temporal also provides these features:

- **Runtime safeguards**: Prevent avoidable errors and issues from executing during runtime.
- **Versioning**: Support multiple versions of your business logic for long-running business processes.

For detailed information on Temporal feature release stages and criteria, see this [Product Release Stages Guide](/evaluate/product-release-stages).

---

# Cancellation and termination

Source: https://docs.temporal.io/evaluate/features/cancellation-and-termination

A business process that's already running sometimes has to stop: an order is retracted, a customer closes an account, a
deploy goes wrong. Temporal gives you two ways to stop a
[Workflow Execution](/workflow-execution), and the difference between them is whether your code gets to clean up after
itself.

**Cancellation** asks the Workflow to stop. The Workflow receives the request, so it can release resources, undo steps it
has already taken, and notify other systems before closing. The steps it takes are your code, so you decide what
unwinding a half-finished process means.

**Termination** closes the Workflow Execution immediately. Workflow code never sees the request and no cleanup runs. It
works on a Workflow that can't respond to anything, which is what makes it the fallback when Cancellation won't land.

## What this means for your design

- **You write the cleanup.** Temporal delivers the Cancellation request and
  schedules a Workflow Task so your code can act on it. You can specify what to undo, and in what order. If your
  process reserves inventory, charges cards, or provisions infrastructure, plan the unwind path alongside the forward
  one. See the [Saga Pattern](/design-patterns/saga-pattern) for the usual structure.
- **Long-running Activities need Heartbeats to be cancellable.** Cancellation reaches an Activity when it Heartbeats, so
  an Activity that never Heartbeats runs to completion regardless. This shapes how you write anything long-running.
- **You always have a way out.** A Workflow blocked by a bug, a bad deploy, or a missing Worker can still be terminated,
  so a stuck process is never permanently stuck.
- **The record survives the stop.** Both operations are recorded in
  [Event History](/workflow-execution/event#event-history) with who requested it and why, which matters for audit and
  incident review.

Cancellation and Termination close a Workflow for good. To hold one in place and resume it later, see
[Workflow Pause](/encyclopedia/workflow/workflow-pause).

## Resources

Read the Temporal Encyclopedia for conceptual depth:

- [Cancellation and Termination](/encyclopedia/workflow/cancellation-and-termination)
- [Parent Close Policy](/parent-close-policy)

Or jump straight to the SDK feature guide for implementation details:

- [.NET SDK](/develop/dotnet/workflows/cancellation): Handle Cancellation and Termination in C# and .NET.
- [Go SDK](/develop/go/workflows/cancellation): Handle Cancellation and Termination in Go.
- [Java SDK](/develop/java/workflows/cancellation): Handle Cancellation and Termination in Java and other JVM languages.
- [PHP SDK](/develop/php/workflows/cancellation): Handle Cancellation and Termination in PHP.
- [Python SDK](/develop/python/workflows/cancellation): Handle Cancellation and Termination in Python.
- [Ruby SDK](/develop/ruby/workflows/cancellation): Handle Cancellation and Termination in Ruby.
- [Rust SDK](/develop/rust/workflows/cancellation): Handle Cancellation and Termination in Rust.
- [TypeScript SDK](/develop/typescript/workflows/cancellation): Handle Cancellation and Termination in TypeScript and JavaScript.

---

# Child Workflows

Source: https://docs.temporal.io/evaluate/features/child-workflows

> Child Workflows give part of a business process its own Event History, Workers, and lifecycle, with the limits and trade-offs versus one Workflow.

A **[Child Workflow](/child-workflows)** is a [Workflow Execution](/workflow-execution) started by another Workflow.
The child gets its own [Event History](/workflow-execution/event#event-history), can run on a separate set of [Workers](/workers#worker), and has its own lifecycle.
That lets a Parent Workflow hand off part of a business process instead of running every step itself.

## When to use a Child Workflow

- **[Create a separate service](/child-workflows#create-a-separate-service)**: because a child can be processed by a completely separate set of Workers than its parent, it can act as its own service. Parent and child share no local state and communicate only through asynchronous [Signals](/sending-messages#sending-signals). To compose across Namespace boundaries, or between teams that own separate Temporal Applications, use [Temporal Nexus](/evaluate/features/nexus) instead.
- **[Partition a large workload](/child-workflows#partition-problems-into-smaller-chunks)**: each child has its own Event History, so splitting work across children raises the ceiling on how many steps one business process can take.
- **[Represent a single resource](/child-workflows#represent-a-single-resource)**: map one child to one resource and use its [Workflow Id](/workflow-execution/workflowid-runid) to guarantee uniqueness, which serializes every operation on that resource.
- **[Run periodic logic](/child-workflows#periodic-logic-execution)**: a child can call [Continue-As-New](/workflow-execution/continue-as-new) as many times as it needs and then complete. From the parent's point of view it was a single invocation, so the periodic work never fills the parent's Event History.

## When not to use a Child Workflow

- **For code organization alone.** Use the object-oriented structure and other organizing techniques your language already offers.
- **When the workload has a bounded size.** Start from a single Workflow Definition when you can put a ceiling on the number of Activity Executions and Signals. One Workflow is simpler to reason about than several communicating asynchronously.
- **When Activities would do.** Child Workflow Executions record more Events overall than Activities do, and each Event in an Event History costs compute. Start with a single Workflow that calls Activities, and add Child Workflows once you have a clear reason.

### Child Workflow or Activity?

Both are started from a Workflow, which makes the choice easy to get wrong.
A Child Workflow has access to all Workflow APIs but is subject to the same [deterministic constraints](/workflow-definition#deterministic-constraints) as any other Workflow.
An Activity has the inverse trade-off: no access to Workflow APIs, but no determinism constraints either.
See [Child Workflow versus an Activity](/child-workflows#child-workflow-versus-an-activity) for the full comparison.

## Limits to plan for

These numbers shape how you partition a business process:

- A Workflow Execution's Event History is capped at 51,200 Events or 50 MB, with a warning at 10,240 Events or 10 MB.
- A Workflow Execution can have at most 2,000 incomplete Child Workflows at a time, by default.
- A Parent Workflow's Event History records Events for each child's status, so a single parent should not spawn more than about 1,000 Child Workflow Executions.
- Those figures compound: one parent running 1,000 children that each run 1,000 Activity Executions reaches 1,000,000 Activity Executions, far past what one Workflow Execution can hold.

See [Workflow Execution limits](/workflow-execution/limits) for the full set.

## Parent Close Policy

Every Child Workflow Execution carries a [Parent Close Policy](/parent-close-policy) that decides what happens to the child when its parent reaches a Closed status:

- **Terminate** (the default): the child is forcefully Terminated.
- **Request Cancel**: a Cancellation request is sent to the child.
- **Abandon**: the child keeps running.

Each child can set its own policy, so a parent can terminate some children and leave others running.

## Resources

Read the Temporal Encyclopedia for conceptual depth:

- [Child Workflows](/child-workflows)
- [Parent Close Policy](/parent-close-policy)

Or jump straight to the SDK feature guide for implementation details:

- [.NET SDK](/develop/dotnet/workflows/child-workflows): Start and manage Child Workflows in C# and .NET.
- [Go SDK](/develop/go/workflows/child-workflows): Start and manage Child Workflows in Go.
- [Java SDK](/develop/java/workflows/child-workflows): Start and manage Child Workflows in Java and other JVM languages.
- [PHP SDK](/develop/php/workflows/child-workflows): Start and manage Child Workflows in PHP.
- [Python SDK](/develop/python/workflows/child-workflows): Start and manage Child Workflows in Python.
- [Ruby SDK](/develop/ruby/workflows/child-workflows): Start and manage Child Workflows in Ruby.
- [Rust SDK](/develop/rust/workflows/child-workflows): Start and manage Child Workflows in Rust.
- [TypeScript SDK](/develop/typescript/workflows/child-workflows): Start and manage Child Workflows in TypeScript and JavaScript.

---

# Temporal's production deployment features

Source: https://docs.temporal.io/evaluate/features/cloud-vs-self-hosted-features

> Transform your Temporal apps into production-ready systems by deploying Workflows, Activities, and Workers with either our managed Temporal Cloud or self-hosted service solutions.

Transform your Temporal applications into production-ready systems by deploying your application code, Workflows, Activities, and Workers for operational use.
When your application is ready to start serving production traffic, we offer two Temporal Service options:

- **[Choose Temporal Cloud for your Temporal Service](/cloud)**
  Let us handle the Temporal Service operations so you can focus on your applications.
- **[Self-host a Temporal Service](/self-hosted-guide)**
  Deploy your own production level Temporal Service to orchestrate your durable applications.

| Feature                            | Temporal Cloud                                                            | Self-hosted                                      |
| ---------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------ |
| **Multi-tenant**                   | ✅ Up to 100 Namespaces                                                   | ✅ Unlimited Namespaces                          |
| **High availability and failover** | ✅ [Namespaces with High Availability features](/cloud/high-availability) | ✅ Global Namespaces & Multi-Cluster Replication |
| **Application state persistence**  | ✅ 30-90 day Retention                                                    | ✅ Unlimited                                     |
| **Long term state retention**      | ✅ Workflow History Export                                                | ✅ Archival                                      |
| **Community support**              | ✅ Slack, Forum                                                           | ✅ Slack, Forum                                  |
| **Paid support**                   | ✅ Prioritized responses                                                  | ✖️                                                |

---

# Core primitives

Source: https://docs.temporal.io/evaluate/features/core-primitives

> Discover Temporal's Workflow, Activity, and Worker framework; orchestrate steps, encapsulate business logic, and execute code efficiently using the Temporal SDK in your favorite language.

[Workflows](/workflows), [Activities](/activities), [Standalone Activities](/standalone-activity), and [Workers](/workers) form the core parts of a Temporal Application.

**[Workflows](/workflows)**: A Workflow defines the overall flow of the application.
You write it in your programming language of choice using the Temporal SDK.
Conceptually, a Workflow specifies a sequence of steps and orchestrates the execution of Activities.

**[Activities](/activities)**: An Activity is a method or function that encapsulates business logic prone to failure (for example, calling a service that may go down).
The system can automatically retry these Activities upon some failures.
Activities perform a single, well-defined action, such as calling another service, transcoding a media file, or sending an email message.

**[Standalone Activities](/standalone-activity)**: A Standalone Activity is a top-level [Activity Execution](/activity-execution) started directly by a [Client](/encyclopedia/temporal-client), without using a Workflow. Use a [Workflow](/workflows) to orchestrate multiple Activities, but if you just need to execute a single Activity, use a [Standalone Activity](/standalone-activity). Standalone Activities are Temporal's job queue.

**[Workers](/workers)**: A Worker executes your Workflow and Activity code.

## Resources

Follow one of our [tutorials](https://learn.temporal.io/getting_started/) to develop Workflows and Activities and run them in Worker Processes, or enroll in one of [our courses](https://learn.temporal.io/courses/).

Read the Temporal Encyclopedia pages for each core primitive:

- [Temporal Workflows](/workflows)
- [Temporal Activities](/activities)
- [Temporal Standalone Activities](/standalone-activity)
- [Temporal Workers](/workers)

Or jump straight to a Temporal SDK develop docs:

- [.NET SDK](/develop/dotnet): Write Workflows, Activities, and Workers in C# and .NET.
- [Go SDK](/develop/go): Write Workflows, Activities, and Workers in Go.
- [Java SDK](/develop/java): Write Workflows, Activities, and Workers in Java and other JVM languages.
- [PHP SDK](/develop/php): Write Workflows, Activities, and Workers in PHP.
- [Python SDK](/develop/python): Write Workflows, Activities, and Workers in Python.
- [Ruby SDK](/develop/ruby): Write Workflows, Activities, and Workers in Ruby.
- [Rust SDK](/develop/rust): Write Workflows, Activities, and Workers in Rust.
- [TypeScript SDK](/develop/typescript): Write Workflows, Activities, and Workers in TypeScript and JavaScript.

---

# Data encryption

Source: https://docs.temporal.io/evaluate/features/data-encryption

> Payload Codecs encrypt Workflow inputs, outputs, and Signals before they reach the Temporal Service, so plaintext stays on hosts you control.

The Temporal Service persists what your Workflows send it.
Workflow inputs and results, Activity results, Signal and Query payloads, and memos are all recorded in the [Event History](/workflow-execution/event#event-history) and stay there for the length of the Namespace [Retention Period](/temporal-service/temporal-server#retention-period).

A [Payload Codec](/payload-codec) encrypts those Payloads before they leave your process.
The Service stores ciphertext, and the data exists unencrypted only on the Clients and Workers you run, on hosts you control.
Temporal ships no default codec: the default [Data Converter](/dataconversion) serializes to JSON without encrypting, so encryption is something you add.

Encrypted Payloads are unreadable in the Web UI and CLI, which is what a [Codec Server](/codec-server) is for.
It's an HTTP server you run that holds the same decode logic as your codec.
The Web UI and CLI send Payloads to it and display what it returns, so decrypted values reach the browser and the terminal only. The copy on the Temporal Service stays encrypted.

You can also use [Temporal Proxy](/production-deployment/temporal-proxy) to encrypt payloads without changing Worker or
Client code. You run the proxy in your environment, where it has access to plaintext payloads and your KMS. See the
proxy's [encryption guide](/production-deployment/temporal-proxy/encrypt-payloads) for its encryption design and coverage
limits.

## What you operate

When you implement encryption with a Payload Codec, each piece of it is yours to run:

- **The codec.** You choose the cipher and write the `encode` and `decode` logic. Temporal doesn't hold or generate your keys.
- **Key distribution.** Your Workers and your Codec Server need the same keys, which usually means [key management](/key-management) infrastructure you already have or have to stand up.
- **The Codec Server.** You host it, secure it, and control access to it. Because it decodes sensitive data through a single API call, restrict it by putting it behind a VPN or adding authentication if it's reachable from the internet. It also needs [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) enabled to answer the Web UI.

Where the endpoint gets configured depends on where the Temporal Service runs:

- **Temporal Cloud.** A [Namespace Admin](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) sets the Codec Server endpoint on the Namespace, which enables it for everyone on that Namespace. Cloud can pass a JWT access token with each request so your server can decide, per user, whether to return plaintext.
- **Self-hosted.** The endpoint is set for the whole Temporal Service, though one Codec Server can route per Namespace. You configure Web UI authorization yourself.

Expect the Web UI to send several requests per Workflow Execution, and expect the round trip to add latency when someone views an Event History.

## What a codec doesn't cover

- **Search Attributes** are persisted unencoded so the Visibility store can index them. Anything you need to search on is readable by the Service, so keep sensitive values out of [Search Attributes](/search-attribute) and in encrypted Payloads.
- **Failure messages and stack traces** aren't encoded as codec-capable Payloads by default. Encoding them is an explicit opt-in through the [Failure Converter](/failure-converter).
- **Payloads above the Service size limit** still need [External Storage](/external-storage), whether or not they're encrypted.
- **Nexus Operations** pass through the Data Converter like any other Payload, so caller and handler Workers need compatible converters. See [Nexus security](/nexus/security#payload-encryption-data-converter).

For how converters and codecs work, see the Encyclopedia pages on [data conversion](/dataconversion).
For setting up and securing a Codec Server, see [Codecs and encryption](/production-deployment/data-encryption).

## Resources

- [Data conversion](/dataconversion): How Payloads are serialized, what the default Data Converter does, and where you can substitute your own.
- [Payload Codec](/payload-codec): The encode and decode step that transforms Payloads for encryption or compression.
- [Codec Server](/codec-server): What a Codec Server is, how the Web UI and CLI use it, and how it works with External Storage.
- [Codecs and encryption](/production-deployment/data-encryption): API contract, CORS, authorization, and endpoint setup for running a Codec Server in production.

Or jump straight to the SDK feature guide for implementation details:

- [.NET SDK](/develop/dotnet/best-practices/data-handling/data-encryption): Implement a Payload Codec and attach it to the Client's Data Converter in C# and .NET.
- [Go SDK](/develop/go/data-handling/data-encryption): Implement a Payload Codec and attach it to the Client's Data Converter in Go.
- [Java SDK](/develop/java/best-practices/data-handling/data-encryption): Implement a Payload Codec and attach it to the Client's Data Converter in Java and other JVM languages.
- [Python SDK](/develop/python/data-handling/data-encryption): Implement a Payload Codec and attach it to the Client's Data Converter in Python.
- [Ruby SDK](/develop/ruby/best-practices/data-handling/data-encryption): Implement a Payload Codec and attach it to the Client's Data Converter in Ruby.
- [TypeScript SDK](/develop/typescript/best-practices/data-handling/data-encryption): Implement a Payload Codec and attach it to the Client's Data Converter in TypeScript and JavaScript.

---

# Job queues

Source: https://docs.temporal.io/evaluate/features/job-queue

> Run background jobs as Standalone Activities with durable retries, timeouts, priority and fairness dispatch, and per-job visibility without a Workflow.

A job is a single, discrete unit of work that runs asynchronously in the background such as sending an email, processing a webhook, syncing data, or executing a single function reliably.

A job queue is the system that manages these jobs: accepting work, dispatching it to workers, retrying on failure, and providing visibility into what's running and what failed.

**[Standalone Activities](/standalone-activity) are Temporal's job queue.**

They let you use Temporal Activities as background jobs, in addition to using the same Activities as steps inside a Workflow. You write an Activity once and can run it either as a background job or as part of a multi-step Workflow.

Under the hood, Standalone Activities use Temporal's [Task Queues](/task-queue) for dispatching work to Workers.

## What you get that a queue library doesn't give you

Each job is a durably persisted [Activity Execution](/activity-execution), not a message in a broker you also have to run.

That changes four things a job queue usually leaves to you:

- **Delivery semantics you choose.** At-least-once by default, with a [Retry Policy](/encyclopedia/retry-policies), timeouts, and exponential backoff enforced by the Temporal Service. Set retry maximum attempts to 1 for at-most-once.
- **Addressable jobs.** Every job has an Activity Id and Run Id. Use them to fetch the result, cancel, or terminate a specific job, rather than searching a log for it.
- **Deduplication at submit time.** A conflict policy and a reuse policy decide what happens when a job with the same Id is already running or has already run, so a retried submit doesn't create a second job.
- **Dispatch under load.** Jobs carry priority and fairness weights, and a slow job doesn't block dispatch of the ones behind it.

For queries, `temporal activity list` and [List Filters](/list-filter) search Executions by type, status, and Task Queue, and standard [Activity metrics](/cloud/metrics/openmetrics/metrics-reference#activity-metrics) cover scheduled, started, completed, failed, timed out, and canceled counts.

## When to use a Standalone Activity instead of a Workflow

Use a Standalone Activity to run **one** Activity. Use a [Workflow](/workflows) to orchestrate **several**, or to hold state between steps.

Running a single Activity as a Standalone Activity rather than wrapping it in a Workflow costs fewer [Billable Actions](/cloud/actions-usage#actions-in-workflows) on Temporal Cloud, and for short jobs it's lower latency, because there are fewer Worker round-trips.

You don't have to decide up front. The same Activity Function runs both ways with no code changes, so a job that later becomes one step of a longer process moves into a Workflow without a rewrite.

## What you still operate

Temporal doesn't replace your compute. You run and scale the Workers that poll [Task Queues](/task-queue) and execute the Activities; the Temporal Service handles persistence, dispatch, retries, and timeouts.

Long jobs also stay your responsibility to instrument: heartbeats are optional, and without them a job that dies mid-run isn't detected until its Start-To-Close Timeout expires.

## Release stages and limitations

Standalone Activities are Generally Available, including Start Delay, Request Cancel, Terminate, and Delete. These capabilities are in [Public Preview](/evaluate/product-release-stages#public-preview):

- [Operator commands](/activity-operations): Pause, Unpause, Reset, and Update Options.
- Batch operations by [List Filter](/list-filter): Request Cancel, Terminate, and Delete.

The `TerminateExisting` conflict policy is not supported yet. Use `Fail` or `UseExisting` instead. See [Limitations](/standalone-activity#limitations) for the full list.

## Resources

Read about [Standalone Activity concepts, features, and limitations](/standalone-activity), or jump to a language-specific quickstart:

- [.NET SDK](/develop/dotnet/activities/standalone-activities-quickstart): Start and manage a Standalone Activity Execution in C# and .NET, with a runnable code sample.
- [Go SDK](/develop/go/activities/standalone-activities-quickstart): Start and manage a Standalone Activity Execution in Go, with a runnable code sample.
- [Java SDK](/develop/java/activities/standalone-activities-quickstart): Start and manage a Standalone Activity Execution in Java and other JVM languages, with a runnable code sample.
- [Python SDK](/develop/python/activities/standalone-activities-quickstart): Start and manage a Standalone Activity Execution in Python, with a runnable code sample.
- [Ruby SDK](/develop/ruby/activities/standalone-activities-quickstart): Start and manage a Standalone Activity Execution in Ruby, with a runnable code sample.
- [Rust SDK](/develop/rust/activities/standalone-activities-quickstart): Start and manage a Standalone Activity Execution in Rust, with a runnable code sample.
- [TypeScript SDK](/develop/typescript/activities/standalone-activities-quickstart): Start and manage a Standalone Activity Execution in TypeScript, with a runnable code sample.

---

# Temporal Nexus

Source: https://docs.temporal.io/evaluate/features/nexus

> Nexus connects Temporal Applications across Namespace, team, and region boundaries.

**[Temporal Nexus](/nexus)** connects Temporal Applications across isolated Namespaces, regions, and clouds.
Each team keeps its own Namespace, Workers, and deployment lifecycle, and exposes a service contract that other teams call with the same durability, observability, and access control as the rest of Temporal.
Callers get a durable, atomic handoff instead of a network call they have to make reliable themselves.
Watch the [Nexus overview](https://www.youtube.com/watch?v=tJ1OwSFokOg&t=117s) for a walkthrough.

## When to use Nexus 

Use Nexus when a business process crosses a boundary you put in place on purpose:

- **Separate teams or domains** - Each team owns its Namespace, Workers, and release cadence, and shares capability through a service contract instead of shared internals.
- **Security and fault isolation** - Grant access to a Nexus Endpoint rather than to a whole Namespace. A misbehaving Worker affects its own Namespace, not its callers.
- **Multiple regions or clouds** - Temporal Cloud routes Nexus requests across Namespaces in AWS and GCP over a global mTLS-secured mesh.
- **Compliance and data residency** - Isolated Namespaces keep sensitive data (PCI, PII) under their own encryption, access controls, and audit trail.
- **Shared internal services** - A registered Endpoint is discoverable, so teams can adopt an existing service instead of rebuilding it.

Use the following decision tree to help determine if Nexus is right for your use case:

## When not to use Nexus

- **One team in one Namespace** - [Child Workflows](/evaluate/features/child-workflows) and [Activities](/activities) already compose work inside a Namespace, and an Endpoint is one more thing to register and operate. If you expect to split into separate Namespaces later, Nexus is still worth adopting now. It separates the contract from the implementation, so the migration is mostly configuration.
- **Synchronous calls that can't finish fast** - A [Synchronous Operation](/nexus/operations#operation-lifecycle) has to complete within the 10-second handler deadline. Anything slower or less predictable belongs in an Asynchronous Operation.

### Nexus, Child Workflows, or Activities?

Connecting Namespaces without Nexus is possible, but each option costs you something:

- **Child Workflows** are limited to the same Namespace. Using them across Namespaces leaks implementation details because the caller has to manage the target Namespace, Task Queue, and Workflow options.
- **Activity wrappers** need a per-target mTLS client, so you take on configuration and certificate management. They tend to be over-permissioned, have no built-in cross-Namespace observability, and need error-prone boilerplate for asynchronous results.
- **A gateway service** is not durable, is hard to debug across services, and is one more service to run and patch.

Nexus replaces all three with a contract between caller and handler, less code, and first-class observability.

## Limits to plan for

- Nexus is available in the Go, Java, Python, TypeScript, .NET, and Rust SDKs. PHP and Ruby don't support it.
- A Synchronous Operation must complete within a [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout). An Asynchronous Operation has a [maximum ScheduleToClose duration of 60 days](/evaluate/cloud/limits#nexus-operation-duration-limits).
- A Workflow Execution can have 30 in-flight Nexus Operations at a time, and a handler Workflow accepts 2,000 callbacks, which sets how many callers can attach to it.
- Temporal Cloud defaults to 100 Nexus Endpoints per Account and 1,000 caller Namespaces per Endpoint. Support can raise both.
- The global Nexus Registry, runtime access controls, audit logging, and cross-region routing are Temporal Cloud features. On a self-hosted Temporal Service, you enable the HTTP API and Nexus dynamic configuration yourself, and external Nexus calls are still experimental.

See [Nexus limits](/cloud/nexus/limits) for the full set.

## How teams use Nexus

- **Duolingo** uses Nexus for self-service infrastructure ([case study](https://temporal.io/resources/case-studies/duolingo-temporal-nexus) | [webinar](https://www.youtube.com/watch?v=tJ1OwSFokOg&t=524s)).
- **Netflix** uses it to orchestrate infrastructure across teams ([Replay talk](https://www.youtube.com/watch?v=izR9dQ_eIe4&t=470s) | [webinar](https://www.youtube.com/watch?v=At1FfqGQiu0&t=1295s)).
- **Miro** uses it for cross-region data migration ([Replay talk](https://youtu.be/YLmFR-IAC3M?feature=shared&t=1488)).

For more architectural patterns built on Temporal, see [use cases and design patterns](/evaluate/use-cases-design-patterns).

## Resources 

Join the [#nexus](https://temporalio.slack.com/archives/C07LQN0JK9B) channel in [Temporal Slack](https://t.mp/slack) to connect with the Nexus community.

Read the Temporal Encyclopedia for conceptual depth:

- [How Nexus works](/nexus)
- [Nexus Security](/nexus/security)
- [Nexus Patterns](/nexus/patterns)

Or jump straight to an SDK quick start:

- [.NET SDK](/develop/dotnet/nexus): Define, register, and call Nexus Services in C# and .NET.
- [Go SDK](/develop/go/nexus): Define, register, and call Nexus Services in Go.
- [Java SDK](/develop/java/nexus): Define, register, and call Nexus Services in Java and other JVM languages.
- [Python SDK](/develop/python/nexus): Define, register, and call Nexus Services in Python.
- [Rust SDK](/develop/rust/nexus): Define, register, and call Nexus Services in Rust.
- [TypeScript SDK](/develop/typescript/nexus): Define, register, and call Nexus Services in TypeScript and JavaScript.

Then set up Nexus on the deployment you run:

- [Temporal Cloud](/cloud/nexus): Global Nexus Registry, runtime access controls, audit logging, and multi-region connectivity.
- [Self-hosted deployment](/production-deployment/self-hosted-guide/nexus): Enable the HTTP API and Nexus dynamic configuration on a self-hosted Temporal Service.

---

# Observability

Source: https://docs.temporal.io/evaluate/features/observability

> Query live and closed Workflow Executions by your own business identifiers, export Prometheus-compatible metrics, and trace Executions across Worker processes.

Temporal records every Workflow Execution as a durable [Event History](/workflow-execution/event#event-history): the inputs, every Activity result, every timer, and every message, in order.
Logs and metrics tell you what a process reported while it was running.
The Event History lets you replay what a Workflow actually did, including one that finished last week.

Around that history, Temporal gives you three things you would otherwise assemble yourself.

## Query Executions by your own identifiers

[Visibility](/visibility) indexes running and closed Workflow Executions so you can list, filter, and count them with a SQL-like [List Filter](/list-filter).
[Custom Search Attributes](/search-attribute#custom-search-attribute) put your own identifiers into that index like, an order Id, customer, region, or tier.
An operator can then answer "which orders are stuck in payment retry" from the [Web UI](/web-ui) or the [CLI](/cli), without querying your application database.

Visibility is a search index that Temporal updates asynchronously, so a List or Count query can return results that are a few seconds stale.
It's built for finding Executions across many, not for reading the authoritative state of one.

## Metrics from both sides of the connection

Temporal emits two independent, Prometheus-compatible metric sets, and you need both:

- **[SDK metrics](/references/sdk-metrics)** come from your Workers and Clients: task slot availability, Schedule-To-Start latency, Activity failures. They tell you whether your Workers are keeping up with the work.
- **Temporal Service metrics** come from the Service: request rates, latencies, and errors. On Temporal Cloud these stream from the [OpenMetrics endpoint](/cloud/metrics/openmetrics) into Datadog, Grafana Cloud, New Relic, or your own Prometheus. Self-hosted deployments scrape [Temporal Service metrics](/references/service-metrics) directly.

Watching one side alone hides the most common failure mode: the Service is accepting and serving work normally while Tasks pile up in a Task Queue because the Workers are under-provisioned.

## Tracing and logs that survive process boundaries

A Workflow Execution can span days, several Activities, and multiple Worker restarts, which is where hand-rolled tracing usually loses the thread.
Temporal SDKs ship OpenTelemetry interceptors that propagate trace context from the Client into the Workflow and out to each Activity, across Worker processes, so one trace covers the whole Execution.

SDKs also expose a logger for Workflow and Activity code that suppresses duplicate output during replay, so a retried or resumed Workflow doesn't repeat every log line it already emitted.

## What you operate

What you run yourself depends on where the Temporal Service runs:

- **Temporal Cloud.** Visibility is enabled by default, and metrics come from the OpenMetrics endpoint. There's no Visibility store to size or upgrade. Both the [number of custom Search Attributes](/evaluate/cloud/limits#number-of-custom-search-attributes) and the [Visibility API rate limit](/evaluate/cloud/limits#visibility-api-rate-limit) are scoped per Namespace.
- **Self-hosted.** You choose and operate the [Visibility store](/self-hosted-guide/visibility): MySQL v8.0.17+ or PostgreSQL v12+ on Temporal Server v1.20 and later, or Elasticsearch and OpenSearch. [Dual Visibility](/dual-visibility) covers migrating between them without downtime.

On both, the Namespace [Retention Period](/temporal-service/temporal-server#retention-period) bounds how long closed Workflow Executions stay queryable. To keep Event Histories past that window, configure [Archival](/temporal-service/archival).

## Resources

- [Visibility](/visibility): How Temporal indexes Workflow Executions, what the index can answer, and when to use something else.
- [Search Attributes](/search-attribute): Default and custom Search Attributes, supported types, and the limits on how many you can define.
- [List Filter](/list-filter): The SQL-like query syntax for filtering Workflow Executions, and the operators it supports.
- [Monitor Temporal Cloud](/cloud/monitor): Set up Cloud and SDK metrics to monitor service and Worker health.
- [Worker alerting](/best-practices/worker-alerting): A recommended alert set for Temporal Workers, with tag filters, thresholds, and triage guidance.
- [Performance bottlenecks](/troubleshooting/performance-bottlenecks): Which metric to read when Workflows are slow, and what each symptom points to.

Or jump straight to the SDK feature guide for implementation details:

- [.NET SDK](/develop/dotnet/platform/observability): Emit metrics, set up tracing, configure logging, and use Search Attributes in C# and .NET.
- [Go SDK](/develop/go/platform/observability): Emit metrics, set up tracing, configure logging, and use Search Attributes in Go.
- [Java SDK](/develop/java/platform/observability): Emit metrics, set up tracing, configure logging, and use Search Attributes in Java and other JVM languages.
- [PHP SDK](/develop/php/platform/observability): Emit metrics, set up tracing, configure logging, and use Search Attributes in PHP.
- [Python SDK](/develop/python/platform/observability): Emit metrics, set up tracing, configure logging, and use Search Attributes in Python.
- [Ruby SDK](/develop/ruby/platform/observability): Emit metrics, set up tracing, configure logging, and use Search Attributes in Ruby.
- [TypeScript SDK](/develop/typescript/platform/observability): Emit metrics, set up tracing, configure logging, and use Search Attributes in TypeScript and JavaScript.

---

# Schedules

Source: https://docs.temporal.io/evaluate/features/schedules

> A Schedule starts Workflows on an interval or calendar expression, with Overlap Policy, Catchup Window, backfill, and pause controls the Temporal Service runs for you.

A [Schedule](/schedule) starts a [Workflow Execution](/workflow-execution) at times you define: on an interval, on a calendar expression, or on a combination of both.

A Schedule is its own object in the Temporal Service with its own Id, separate from the Workflow Executions it starts.
You can update, pause, backfill, or trigger it without changing or redeploying the Workflow code it runs.

That separation is what you get instead of running a scheduler alongside your application.
There's no second system to deploy, monitor, and reconcile with your Workers, and every start the Schedule makes is an ordinary Workflow Execution with the same durability, retries, and [Event History](/workflow-execution/event#event-history) as one you start by hand.

## What a Schedule controls

- **Spec.** When Actions happen: an interval (`45m`, or `6h/5h` for every six hours offset into the fifth hour) or a calendar expression, given as a cron string (`0 8 * * 1-5`, weekdays at 8:00 UTC) or as JSON with named fields (`{"dayOfMonth": "1,15", "hour": "11-14"}`). One Spec can combine several of each and add start and end times, exclusions, jitter, and a time zone. Exclusions and embedded time zone data are available through the SDKs and API, but not the CLI or Web UI.
- **[Overlap Policy](/schedule#overlap-policy).** What happens when it's time to start and the previous Execution is still running: `Skip` (the default), `BufferOne`, `BufferAll`, `CancelOther`, `TerminateOther`, or `AllowAll`.
- **[Catchup Window](/schedule#catchup-window).** Which missed Actions to take when the Temporal Service was unavailable at the scheduled time. The default is one year; the minimum is ten seconds.
- **[Pause-on-failure](/schedule#pause-on-failure).** Pause the Schedule automatically when a scheduled Execution ends in failure or timeout.
- **[Backfill](/schedule#backfill).** Run every Action for a past time range now, including a range from before the Schedule existed.
- **[Action limit](/schedule#limit-number-of-actions).** Stop after a set number of scheduled Actions, after which the Schedule behaves as paused.

Every Execution a Schedule starts carries the `TemporalScheduledStartTime` and `TemporalScheduledById` [Search Attributes](/search-attribute), so you can query scheduled runs with a [List Filter](/list-filter) the same way you query anything else.

## What a Schedule doesn't do

- **Pausing a Schedule doesn't pause what's already running.** It stops future Actions. Executions the Schedule already started keep going.
- **A Paused Workflow Execution still counts as running.** When the Schedule evaluates its Overlap Policy, a Paused Execution is an open Execution, so `Skip` skips and `BufferOne` buffers behind it. See [Interaction with Workflow Pause](/schedule#workflow-pause).
- **Listing Schedules is eventually consistent.** `ListSchedules` and `CountSchedules` are served by [Visibility](/visibility#operations-that-use-visibility) and share its rate limit, so a Schedule you just created or deleted may not appear right away.

## How to choose between a Schedule, a Cron Job, and Start Delay

- **Use a Schedule when the same Workflow has to run more than once.** It's the right choice as soon as you need to pause the series during an incident, change the timing without a deploy, or run the Actions an outage skipped. Use it for new applications even when the timing is a plain cron expression, because you get those controls whether or not you need them yet.
- **Use [Start Delay](/workflow-execution/timers-delays#delay-workflow-execution) when there's exactly one run, at a time you know when you start it.** A trial expiry, a cancellation deadline, a reminder. It isn't recurring, and it's incompatible with both Schedules and Cron Jobs, so it's not a way to hold off the first Action of a Schedule. Set a start time on the Schedule Spec for that.
- **Keep a [Temporal Cron Job](/cron-job) if you have one running, but don't write new ones.** A cron string is a property of the Workflow Execution rather than a separate object, so the next Run starts only after the current one closes, and changing or stopping the series means terminating the Workflow. A Schedule covers the same cases and lets you update it in place.

If the waiting happens inside a Workflow that's already running, none of these apply. Use a [Timer](/workflow-execution/timers-delays#timer).

## Resources

- [Schedule](/schedule): Full reference for Spec, Policies, Backfill, last completion result, and the limitations of Schedules.
- [Temporal Cron Job](/cron-job): How Cron Schedules behave, the supported cron syntax, time zone caveats, and how to stop a Cron Job.
- [Timers and Start Delay](/workflow-execution/timers-delays): Durable Timers inside a Workflow, and Start Delay for a one-time start at a future point.
- [temporal schedule](/cli/command-reference/schedule): CLI commands to create, backfill, delete, describe, list, pause, trigger, and update a Schedule.
- [Missed Schedule Actions](/troubleshooting/schedule-missed-actions): Alert on the missed catchup window metric, then narrow down which Schedule skipped an Action and why.

Or jump straight to the SDK feature guide for implementation details:

- [.NET SDK](/develop/dotnet/workflows/schedules): Create, backfill, delete, describe, list, pause, trigger, and update Schedules in C# and .NET.
- [Go SDK](/develop/go/workflows/schedules): Create, backfill, delete, describe, list, pause, trigger, and update Schedules in Go.
- [Java SDK](/develop/java/workflows/schedules): Create, backfill, delete, describe, list, pause, trigger, and update Schedules in Java and other JVM languages.
- [PHP SDK](/develop/php/workflows/schedules): Create, backfill, delete, describe, list, pause, trigger, and update Schedules in PHP.
- [Python SDK](/develop/python/workflows/schedules): Create, backfill, delete, describe, list, pause, trigger, and update Schedules in Python.
- [Ruby SDK](/develop/ruby/workflows/schedules): Create, backfill, delete, describe, list, pause, trigger, and update Schedules in Ruby.
- [TypeScript SDK](/develop/typescript/workflows/schedules): Create, backfill, delete, describe, list, pause, trigger, and update Schedules in TypeScript.

---

# Serverless Workers

Source: https://docs.temporal.io/evaluate/features/serverless-workers

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview. Amazon Bedrock AgentCore Runtime support is in
> Pre-release. To request AgentCore Runtime access on Temporal Cloud, create a
> [support ticket](/evaluate/cloud/support#support-ticket) or contact your account team. You can also
> [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or try them through self-hosting.

Serverless Workers let you run Temporal Workers on AWS Lambda, Amazon Bedrock AgentCore Runtime, and GCP Cloud Run.
There are no servers to provision, no Worker fleet to scale, and no idle Worker fleet to maintain. Temporal starts
Worker capacity when Tasks arrive and scales it down when the work drains.

Serverless Workers use the same Temporal SDKs as traditional long-lived Workers. You register Workflows and Activities
the same way. The difference is in the lifecycle: Temporal manages it instead of you running a Worker process. How that
lifecycle works depends on the provider. On AWS Lambda, Temporal invokes a function per unit of work, and the Worker
exits when the invocation ends. On AgentCore Runtime, Temporal invokes Runtime sessions that host standard long-running
Workers. On GCP Cloud Run, Temporal resizes a pool of long-lived instances, scaling it to zero when there is no work.

For a deeper look at how Serverless invocation works under the hood, see [Serverless Workers](/serverless-workers) in
the encyclopedia.

> **💡 Tip:**
> Use the Serverless Workers Skill
>
> The [Temporal Serverless Workers Skill](https://github.com/temporalio/skill-temporal-serverless) gives your AI coding
> agent the knowledge to write, package, deploy, and troubleshoot Serverless Workers. See
> [Develop with AI](/with-ai#skills) for installation instructions.
>

## Why use Serverless Workers?

Serverless Workers are a good fit for many workloads. They offer several advantages compared to long-lived Workers on
dedicated compute.

### Reduce operational overhead

Long-lived Workers require you to provision infrastructure, configure scaling policies, manage deployments, and monitor
host-level health. Serverless Workers reduce this burden by offloading invocation and scaling to Temporal and the
compute provider. You still deploy the Worker package and configure the compute provider, but there is no always-on
infrastructure to manage and no autoscaling policies to tune.

Worker management is one of the most common sources of support questions for Temporal users. Serverless Workers offer a
prescriptive deployment path that reduces the operational surface area and lets you focus on writing Workflows instead
of managing infrastructure.

### Get started faster

Running a long-lived Worker requires choosing a hosting strategy, configuring compute resources, and setting up
deployment pipelines before you can execute your first Workflow in production.

With Serverless Workers, you package your Worker code, deploy it to your serverless provider, and configure the
connection to Temporal. There is no need to set up Kubernetes, manage container orchestration, or design a scaling
strategy.

### Scale automatically

Temporal scales the compute for you. When Task volume increases, it asks the provider for more capacity. When traffic
drops, it scales back down. When there is no work, there is no compute running.

This automatic scaling is especially useful for bursty, event-driven workloads where traffic patterns are unpredictable
or highly variable.

### Pay only for what you use

Long-lived Workers run continuously, whether or not there is work to process. Serverless Workers run only when Tasks are
available, so you pay for compute while work is in flight rather than around the clock. For workloads with low or
intermittent volume, that can cut compute costs.

## When to use Serverless Workers

Serverless Workers are a good fit when:

- **Workloads are bursty or event-driven.** Order processing, notifications, webhook handlers, and similar workloads
  that experience spiky traffic benefit from automatic scaling without over-provisioning.
- **Traffic is low or intermittent.** If Workers spend most of their time idle, Serverless Workers eliminate the cost of
  always-on compute.
- **You want a simpler getting-started path.** Deploying a function is simpler than setting up a container orchestration
  platform. Serverless Workers reduce the steps between writing Worker code and running your first Workflow.
- **Your organization has standardized on serverless.** Teams that already run services on Lambda, AgentCore Runtime,
  Cloud Run, or similar platforms can run Temporal Workers using the same deployment patterns and tooling.
- **You serve multiple tenants with infrequent workloads.** Platforms that run Workflows on behalf of many users or
  customers can avoid running dedicated Workers per tenant.

Serverless Workers may not be ideal when:

- **Activities are long-running and cannot be interrupted.** Some serverless platforms enforce execution time limits.
  For example, AWS Lambda has a 15-minute execution limit. Activities that run longer than the provider's timeout and
  cannot be broken into smaller steps need a different hosting strategy or a provider with a longer compute lifetime.
  AgentCore Runtime sessions can provide up to 8 hours of uninterrupted compute. Long-running Workflows are not
  affected because Temporal maintains their state and execution progress across Worker processes.
- **Workloads require sustained high throughput.** For consistently high-volume Task Queues, long-lived Workers on
  dedicated compute may be more cost-effective and performant.
- **You need a persistent connection to Temporal.** Some features require the Worker to hold a connection open. On AWS
  Lambda each invocation connects fresh, so those features do not apply. AgentCore Runtime Workers and Cloud Run
  instances hold a connection for the lifetime of their compute.

> **⚠️ Caution:**
> Multi-cloud Replication is not supported
>
> Serverless Workers do not support [Multi-cloud Replication](/cloud/high-availability#multi-cloud-replication).
> They must be invoked from the same cloud provider where the Namespace is currently active. After a failover to a
> different cloud provider, Serverless Workers can no longer be invoked. This is a temporary restriction during Public Preview.
>

## How Serverless Workers compare to long-lived Workers

|                | Long-lived Worker                                          | Serverless Worker                                                                                  |
| -------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| **Lifecycle**  | Long-lived process that runs continuously.                 | Temporal starts and stops it according to the compute provider's lifecycle.                         |
| **Scaling**    | You manage scaling (Kubernetes HPA, instance count, etc.). | Temporal adds capacity as needed, within the compute provider's limits.                            |
| **Connection** | Persistent connection to Temporal.                         | Fresh per Lambda invocation. Held for the compute lifetime on AgentCore Runtime and Cloud Run.     |

## Supported providers

All three providers can serve long-running Workflows because Workflow lifetime is independent of Worker lifetime. The
provider's compute limit applies to an individual Worker invocation or compute instance, not to the total Workflow
duration.

| Provider | Worker lifecycle | Consider this provider when |
| --- | --- | --- |
| [AWS Lambda](/serverless-workers/aws-lambda) | Temporal invokes a function per unit of work. Each invocation can run for up to 15 minutes. | You want general-purpose, event-driven AWS compute, and individual Activity attempts fit within the Lambda invocation window. |
| [Amazon Bedrock AgentCore Runtime](/serverless-workers/agentcore) | Temporal invokes Runtime sessions that host standard long-running Workers. Each session can provide up to 8 hours of uninterrupted compute. | You are building agents on AWS, want to use AgentCore services, need longer Activity attempts, or benefit from process-local reuse while the Runtime compute remains available. |
| [GCP Cloud Run](/serverless-workers/cloud-run) | Temporal adjusts the size of a Worker Pool of long-lived instances. | You want a managed pool of container-based Workers on Google Cloud. |

## Next steps

- [Interactive demo](/demos/serverless-workers) to explore the configuration and invocation flow.
- [How Serverless Workers work](/serverless-workers) for a deeper look at the invocation lifecycle, compute providers,
  and architecture.
- [Test a Serverless Worker locally with LocalStack](/production-deployment/worker-deployments/serverless-workers/aws-lambda/local-development)
  to run the full AWS Lambda invocation loop on your machine with no AWS account.
- [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers) for the end-to-end
  deployment guide.

For the Worker code itself, pick your SDK and provider:

| SDK | AWS Lambda | Amazon Bedrock AgentCore Runtime | GCP Cloud Run |
| --- | --- | --- | --- |
| Go | [Lambda Workers in Go](/develop/go/workers/serverless-workers/aws-lambda) |  | [Cloud Run Workers in Go](/develop/go/workers/serverless-workers/cloud-run) |
| Python | [Lambda Workers in Python](/develop/python/workers/serverless-workers/aws-lambda) | [AgentCore Runtime Workers in Python](/develop/python/workers/serverless-workers/agentcore) | [Cloud Run Workers in Python](/develop/python/workers/serverless-workers/cloud-run) |
| TypeScript | [Lambda Workers in TypeScript](/develop/typescript/workers/serverless-workers/aws-lambda) |  | [Cloud Run Workers in TypeScript](/develop/typescript/workers/serverless-workers/cloud-run) |
| Java | [Lambda Workers in Java](/develop/java/workers/serverless-workers/aws-lambda) |  | [Cloud Run Workers in Java](/develop/java/workers/serverless-workers/cloud-run) |
| .NET | [Lambda Workers in .NET](/develop/dotnet/workers/serverless-workers/aws-lambda) |  | [Cloud Run Workers in .NET](/develop/dotnet/workers/serverless-workers/cloud-run) |
| Ruby |  |  | [Cloud Run Workers in Ruby](/develop/ruby/workers/serverless-workers/cloud-run) |
| Rust |  |  | [Cloud Run Workers in Rust](/develop/rust/workers/serverless-workers/cloud-run) |

---

# Timeouts and Retry Policies

Source: https://docs.temporal.io/evaluate/features/timeouts-and-retries

> Detect Workflow and Activity failures with timeouts, then mitigate them with automatic retries. Configuration guides for every Temporal SDK.

In Temporal, timeouts detect application failures.
The system can then automatically mitigate these failures through retries.
Both major application function primitives, **[Workflows](/workflows)** and **[Activities](/activities)**, have dedicated **timeout configurations** and can be configured with a **[Retry Policy](/encyclopedia/retry-policies)**.

Follow one of our tutorials to [Get started](https://learn.temporal.io/getting_started/) exploring timeouts and Retry Policies.

## Workflow timeouts

Each Workflow timeout limits a different aspect of a [Workflow Execution](/workflow-execution), and you set them when you start the Workflow Execution.

- **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)**: the maximum time a Workflow Execution can stay Open, including retries and any use of [Continue-As-New](/workflow-execution/continue-as-new). The default is infinite.
- **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout)**: the maximum duration of a single [Workflow Run](/workflow-execution/workflowid-runid#run-id), which excludes retries and Continue-As-New. The default matches the Workflow Execution Timeout.
- **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout)**: the maximum time a [Worker](/workers#worker) has to execute a [Workflow Task](/tasks#workflow-task) after picking it up from the [Task Queue](/task-queue). The default is 10 seconds, with a maximum of 120 seconds.

Workflows are built to run for long periods and to survive failures, so most applications don't need a Workflow Execution Timeout or Workflow Run Timeout.
A timeout limits how long a Workflow can absorb delays. To act after a set period inside a Workflow, use a [Timer](/workflow-execution/timers-delays) instead.
Workflow Execution and Workflow Run Timeouts are most often used to bound a [Temporal Cron Job](/cron-job).

Set Workflow timeouts and Retry Policies in your language:

- [.NET SDK](/develop/dotnet/workflows/timeouts): Set Workflow timeouts and Retry Policies in C# and .NET.
- [Go SDK](/develop/go/workflows/timeouts): Set Workflow timeouts and Retry Policies in Go.
- [Java SDK](/develop/java/workflows/timeouts): Set Workflow timeouts and Retry Policies in Java and other JVM languages.
- [PHP SDK](/develop/php/workflows/timeouts): Set Workflow timeouts and Retry Policies in PHP.
- [Python SDK](/develop/python/workflows/timeouts): Set Workflow timeouts and Retry Policies in Python.
- [Ruby SDK](/develop/ruby/workflows/timeouts): Set Workflow timeouts and Retry Policies in Ruby.
- [Rust SDK](/develop/rust/workflows/timeouts): Set Workflow timeouts and Retry Policies in Rust.
- [TypeScript SDK](/develop/typescript/workflows/timeouts): Set Workflow timeouts and Retry Policies in TypeScript and JavaScript.

## Activity timeouts

Activity timeouts tell the caller — a Workflow, or the Client for a [Standalone Activity](/standalone-activity) — which kind of Activity failure occurred, and the Activity's [Retry Policy](/encyclopedia/retry-policies) decides what happens next.

- **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout)**: the maximum time an [Activity Task](/tasks#activity-task) can sit in the Task Queue before a Worker picks it up. The default is infinite, and this timeout never triggers a retry, because a retry would return the Activity Task to the same Task Queue.
- **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout)**: the maximum time for a single [Activity Task Execution](/tasks#activity-task-execution). It has no default of its own. If you set only a Schedule-To-Close Timeout, Start-To-Close uses that value.
- **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout)**: the maximum time for the whole [Activity Execution](/activity-execution), from the first Activity Task being scheduled to the last one closing. The default is infinite.
- **[Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout)**: the maximum time between [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat), the pings a Worker sends to report that an Activity Execution is still making progress.

Every Activity Execution needs either a Start-To-Close or a Schedule-To-Close Timeout.
Set Start-To-Close: the [Temporal Service](/temporal-service) can't tell when a Worker crashes or loses contact, so it relies on this timeout to retry the Activity.
Long-running Activities should also Heartbeat and set a Heartbeat Timeout, which detects a lost Worker sooner than a long Start-To-Close Timeout can.

Set Activity timeouts and Retry Policies in your language:

- [.NET SDK](/develop/dotnet/activities/timeouts): Set Activity timeouts and Retry Policies in C# and .NET.
- [Go SDK](/develop/go/activities/timeouts): Set Activity timeouts and Retry Policies in Go.
- [Java SDK](/develop/java/activities/timeouts): Set Activity timeouts and Retry Policies in Java and other JVM languages.
- [PHP SDK](/develop/php/activities/timeouts): Set Activity timeouts and Retry Policies in PHP.
- [Python SDK](/develop/python/activities/timeouts): Set Activity timeouts and Retry Policies in Python.
- [Ruby SDK](/develop/ruby/activities/timeouts): Set Activity timeouts and Retry Policies in Ruby.
- [Rust SDK](/develop/rust/activities/timeouts): Set Activity timeouts and Retry Policies in Rust.
- [TypeScript SDK](/develop/typescript/activities/timeouts): Set Activity timeouts and Retry Policies in TypeScript and JavaScript.

## Resources

For a deep dive into timeouts and Retry Policies visit the following Temporal Encyclopedia pages or enroll in one of [our courses](https://learn.temporal.io/courses/).

- [Detecting Workflow failures](/encyclopedia/detecting-workflow-failures): Workflow Execution, Workflow Run, and Workflow Task Timeouts in detail, plus how to find Workflows with failed Workflow Tasks.
- [Detecting Activity failures](/encyclopedia/detecting-activity-failures): Schedule-To-Start, Start-To-Close, Schedule-To-Close, and Heartbeat Timeouts in detail, plus which Activities should Heartbeat.
- [Retry Policies](/encyclopedia/retry-policies): How attempts, intervals, backoff coefficients, and non-retryable errors shape a retry.

---

# Workflow message passing

Source: https://docs.temporal.io/evaluate/features/workflow-message-passing

> Signals write to a running Workflow without waiting, Queries read its current state, and Updates do both and return a result to the caller.

A running [Workflow Execution](/workflow-execution) can receive messages from outside itself.
Its message handlers act on the Workflow's current state, which makes a Workflow behave like a stateful service with its own endpoints.
Temporal has three message types: **Signals** are asynchronous write requests, **Queries** are read requests, and **Updates** are synchronous write requests that return a result.

Each one fits a different job:

- Your shipment-tracking Workflow needs to know when an item leaves the warehouse. **Signal** the Workflow when the driver scans the barcode.
- Your team wants to track the progress of a data migration. **Query** the running batch Workflow for the numbers behind a progress bar.
- Your shopping cart Workflow needs to add an item and render the cart. **Update** it to add the item and get the current contents back in the same call.

## Choose a message type

|                                                        | Signal            | Query | Update              |
| ------------------------------------------------------ | ----------------- | ----- | ------------------- |
| Reads Workflow state                                   | No                | Yes   | Yes                 |
| Changes Workflow state                                 | Yes               | No    | Yes                 |
| Caller waits for the handler                           | No                | Yes   | Yes                 |
| Returns a value to the caller                          | No                | Yes   | Yes                 |
| Recorded in the [Event History](/workflow-execution/event#event-history) | Yes | No    | Yes, once accepted  |
| Handler can block                                      | Yes               | No    | Yes                 |

For **write requests**, the choice is whether the caller needs to know the outcome.
Send a Signal when the caller can move on without a result and shouldn't depend on a Worker being available to accept the request.
Send an Update when the caller needs a result or an error, wants low end-to-end latency, or should be rejected up front by a validator before the request enters the Workflow's Event History.

For **read requests**, start with a Query.
Queries never add Events to the Event History, and they work against completed Workflow Executions.
Because a Query handler cannot block, reading a value that only exists once the Workflow reaches a certain state means either polling with Queries or writing the read as an Update, which is more efficient but does write to the Event History.

For **combined read and write requests**, use an Update.
If the request has to be asynchronous, send a Signal and poll with a Query.

To stream a sequence of events out of a Workflow rather than answer a single request, see [Workflow Streams](/workflow-streams).

## When not to use a message

- **To change state from a Query.** A Query handler can inspect Workflow state but must not mutate it. Use a Signal or an Update.
- **To carry data that's known at start time.** Pass it as a Workflow argument instead of sending a message after the Workflow starts.
- **To reach a Child Workflow from inside a Workflow.** Send Updates from an Activity using a Temporal Client, not from Workflow code.
- **For high fan-in with Updates.** A Workflow Execution can have only a small number of Updates in flight at once, while Signals have no equivalent per-message concurrency cap. See [Limits to plan for](#limits-to-plan-for).
- **To stop a Workflow.** Cancellation and Termination are their own operations. See [Cancellation and Termination](/evaluate/features/cancellation-and-termination).

## Send a message and start a Workflow in one call

Both write types have a variant that starts a Workflow Execution if one isn't already running under the given [Workflow Id](/workflow-execution/workflowid-runid):

- **[Signal-With-Start](/sending-messages#signal-with-start)** Signals a running Workflow Execution, or starts one and immediately Signals it. The operation is atomic, which makes it a way to lazily initialize a Workflow.
- **[Update-With-Start](/sending-messages#update-with-start)** sends an Update and starts the Workflow if needed, in a single round trip, and requires a [Workflow Id conflict policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy). Unlike Signal-With-Start it is not atomic: the SDK retries the request, but the Update is not guaranteed to succeed. Self-hosted deployments should run [Temporal Server v1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) or later.

## Limits to plan for

These figures are for Temporal Cloud. Self-hosted deployments configure their own, starting from the [self-hosted defaults](/self-hosted-guide/defaults).

- A Workflow Execution can receive up to 10,000 Signals. It stops processing Signals after that.
- A Workflow Execution can have at most 10 in-flight Updates, and 2,000 Updates total in its Event History.
- A Workflow Execution can have at most 2,000 incomplete `SignalExternalWorkflowExecution` Commands at a time, which bounds how many Signals it can send to other Workflows at once.
- A single request payload is capped at 2 MB, and any one Event History transaction at 4 MB. Offload larger payloads to [External Storage](/external-storage).

See [System limits](/evaluate/cloud/limits) for the full set.

## Resources

For conceptual depth on the three message types, read the Temporal Encyclopedia or enroll in one of [our courses](https://learn.temporal.io/courses/interacting_with_workflows).

- [Workflow message passing](/encyclopedia/workflow-message-passing): How Signals, Queries, and Updates are delivered and handled, and the guarantees each one carries.
- [Sending messages](/sending-messages): Sending Signals, Queries, and Updates from a Client, the Temporal CLI, or another Workflow.
- [Handling messages](/handling-messages): Writing handlers, validating Updates, and processing a message exactly once.
- [Workflow messaging patterns](/design-patterns/workflow-messaging-patterns): Which pattern to reach for when a caller sends data into a running Workflow and waits on a response.

Or jump straight to the SDK feature guide for implementation details:

- [.NET SDK](/develop/dotnet/workflows/message-passing): Send and handle Signals, Queries, and Updates in C# and .NET.
- [Go SDK](/develop/go/workflows/message-passing): Send and handle Signals, Queries, and Updates in Go.
- [Java SDK](/develop/java/workflows/message-passing): Send and handle Signals, Queries, and Updates in Java and other JVM languages.
- [PHP SDK](/develop/php/workflows/message-passing): Send and handle Signals, Queries, and Updates in PHP.
- [Python SDK](/develop/python/workflows/message-passing): Send and handle Signals, Queries, and Updates in Python.
- [Ruby SDK](/develop/ruby/workflows/message-passing): Send and handle Signals, Queries, and Updates in Ruby.
- [Rust SDK](/develop/rust/workflows/message-passing): Send and handle Signals, Queries, and Updates in Rust.
- [TypeScript SDK](/develop/typescript/workflows/message-passing): Send and handle Signals, Queries, and Updates in TypeScript and JavaScript.

---

# Temporal product release stages

Source: https://docs.temporal.io/evaluate/product-release-stages

> Pre-release, Public Preview, and General Availability compared by feature access, API stability, support, pricing, and regional coverage.

> **💡 Tip:**
> CHANGELOG
> To stay up-to-date with the latest feature changes, visit the [changelog](https://temporal.io/change-log).

This Product Release Stages Guide provides an understanding of how Temporal features are released. It describes and lists the criteria for each release stage, so that you can make informed decisions about the adoption of each new feature.

Product Release Guide Expectations:

|                                 | Pre-release                                                        | Public Preview                                                                              | General Availability                             |
| ------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| **Features access**             | Self-hosted Temporal users: Everyone; Temporal Cloud: Invite only. | Everyone. Temporal Cloud may limit the number of users being onboarded to ensure stability. | Everyone.                                        |
| **Feature completeness**        | Limited functionality.                                             | Core functionality is complete.                                                             | Mature and feature complete.                     |
| **API stability**               | Experimental; API is subject to change.                            | API breaking changes are kept to a minimum.                                                 | API is stable.                                   |
| **Feature region Availability** | Limited regions.                                                   | Most regions.                                                                               | All [regions](/evaluate/cloud/regions).                   |
| **Feature support**             | Community and engineering team.                                    | [Formal support](/evaluate/cloud/support#support-ticket).                                            | [Formal support](/evaluate/cloud/support#support-ticket). |
| **Feature recommended usage**   | Experimental.                                                      | Production use cases.                                                                       | Production usage.                                |
| **Feature Cloud pricing**       | No additional cost.                                                | Pricing changes are kept to a minimum.                                                      | Pricing is stable.                               |
| **Feature Interoperability**    | Limited.                                                           | Features are compatible with each other, unless otherwise stated.                           | Features are compatible with each other.         |

## Pre-release 

**Access:** Most Pre-release features are released in the open source Temporal software and are publicly available.
However, some features which are explicit to hosting Temporal Services, such as [API Keys](/cloud/api-keys), may be specific to Temporal Cloud.

In Temporal Cloud, Pre-release features are invite-only: Temporal will work directly with a group of existing Temporal Cloud customers to be part of testing of each Pre-release feature.
These customers are invited to provide feedback to the Temporal team.

**Classification:** New features in Pre-release may not be fully mature and may have bugs.
Users acknowledge and agree that Pre-release features are provided on an “as-is” basis, and that they are provided without any indemnification, support, warranties, or representation of any kind.

**Feedback:** Feedback is highly encouraged and important for guiding Temporal feature development.
We encourage you to share your experience so that you can influence the future direction of Temporal.

**Availability:** Temporal may modify features before they become Generally Available, or may even decide to remove them.
This means there is no guarantee that a new feature will become Generally Available.
A Pre-release feature can be deprecated at any time.

Pre-release features may be disabled by default, and can be enabled via configuration.
Temporal Cloud customers can contact the Temporal account team or [Temporal Support Team](/evaluate/cloud/support#support-ticket) to gain Pre-release access.

## Public Preview 

**Access:** New features in Public preview are available to everyone.

**Classification:** Features in public preview may undergo further development and testing before they are made Generally Available.
These features are being refined and are recommended for production usage.

**Feedback:** Temporal users are invited to share feedback via the [Community Slack](http://t.mp/slack), by reaching out directly to the Temporal team at product@temporal.io, by creating issues in the relevant [GitHub repository](https://github.com/temporalio), or by posting a feature request on the [Temporal Feature Request Board](https://github.com/temporalio/temporal/discussions).
Temporal also encourages Temporal Cloud users to submit feedback via [support ticket](/evaluate/cloud/support#support-ticket).
This feedback will assist in guiding the improvements for General Availability.

**Availability:** New Features in Public Preview may evolve.
The APIs may undergo changes; however, Temporal's goal is to maintain backward compatibility.

## General Availability 

**Access:** Features in General Availability are available to everyone.

**Classification:** The feature is now fully developed, tested, and available for use without further anticipated changes.

**Feedback:** Temporal users are invited to share feedback via the [Community Slack](http://t.mp/slack), by reaching out directly to the Temporal team at product@temporal.io, by creating issues in the relevant [GitHub repository](https://github.com/temporalio), or by posting a feature request on the [Temporal Feature Request Board](https://github.com/temporalio/temporal/discussions).

**Availability:** Features in General Availability are released with stable APIs and recommended for production use.

Features in General Availability may be covered by the [Temporal Cloud Service Level Agreement (SLA)](https://temporal.io/sla) or have a public [service level objective (SLO)](/cloud/slo), where relevant.

> **ℹ️ Info:**
> Exceptions
>
> There may be exceptions for different features, but this is the typical expectation.
> Any variation will be documented.
>

---

# Understanding Temporal

Source: https://docs.temporal.io/evaluate/understanding-temporal

> Workflows, Activities, Workers, Task Queues, and Event History. The parts of a Temporal application and how they fit together to run code to completion.

A Temporal application has two halves: the code you write, and the Temporal Service that runs it.
This page introduces the pieces of each and how they fit together.

[Why Temporal](/evaluate/why-temporal) covers what the platform replaces and why you would adopt it.
Read this page first for the vocabulary (Workflow, Activity, Worker, Task Queue, Event History),
then go deeper with [Temporal architecture](/encyclopedia/architecture/temporal-architecture) and
[How Temporal works](/encyclopedia/architecture/how-temporal-works).

## Durable Execution

Temporal runs your application code as a Durable Execution. Once a [Workflow](/workflows) starts, it
runs to completion, whether that takes a second or a year. If the process running it crashes, the
Temporal Service hands the work to another process, which rebuilds the state of the execution and
resumes at the point where it stopped, with local variables and progress intact.

Without that guarantee, a crash loses the state of an in-flight process. Recovering it takes a state
table, checkpoint columns, a reconciliation job, and retry logic at every call site — code that has
nothing to do with the problem you set out to solve, and that you have to continuously maintain. Temporal moves
that layer into the platform, so what you write is the business logic.

## Workflows

A Workflow is your business process defined in code, with each step written as ordinary control
flow: loops, conditionals, and function calls. You have probably used one today without calling it
that:

- Transferring money in a banking app
- Booking a vacation
- Filing an expense report
- Onboarding a new employee
- Deploying cloud infrastructure
- Training a model

Temporal is not a no-code Workflow engine. You write Workflows in your own language, editor, and
version control, and you keep the control flow of a normal program.

A Workflow Definition must be deterministic because Temporal reruns it to rebuild state after a failure.
Anything that talks to the outside world belongs in an Activity. See
[deterministic constraints](/workflow-definition#deterministic-constraints) for the rules.

A running Workflow can also:

- Wait on a durable [Timer](/workflow-execution/timers-delays#timer) for minutes or months.
- Receive [Signals and Updates, and answer Queries](/evaluate/features/workflow-message-passing),
  while it runs.
- Start [Child Workflows](/evaluate/features/child-workflows) to break a large process into parts.
- Run on a [Schedule](/schedule) instead of being started by a Client.
- Be held in place with [Workflow Pause](/encyclopedia/workflow/workflow-pause) during an incident,
  without terminating it or losing state.

## Activities

Activities are the units of work that touch the outside world: calling an API, writing to a
database, sending an email, charging a card. They are plain functions or methods, depending on the
language, and you call them from Workflow code.

Because those calls fail, Temporal supervises them. Each Activity gets a
[Retry Policy](/encyclopedia/retry-policies),
[timeouts](/evaluate/features/timeouts-and-retries), and optional heartbeats from configuration
rather than from code you wrote. A failed Activity retries with backoff, and an Activity whose
Worker dies is rescheduled.

You can also run an Activity on its own. A [Standalone Activity](/standalone-activity) is a
top-level [Activity Execution](/activity-execution) started directly by a
[Client](/encyclopedia/temporal-client), with no Workflow around it. It gives you durable retries, a
Run ID you can cancel or query, and deduplication at submit time, which makes it Temporal's
[job queue](/evaluate/features/job-queue) for teams moving off Celery, Sidekiq, or a homegrown
broker. Deploy the Activity to a Worker once, then invoke it standalone or from inside a Workflow.

Use a Workflow when you need to orchestrate several Activities, keep state between steps, or
compensate for a partial failure. Use a Standalone Activity when you need to run one unit of work
reliably.

## Temporal SDKs

A Temporal SDK is the open-source library you add to your application. It provides the APIs to
define Workflows and Activities, to configure and run Workers, and to create a
[Temporal Client](/encyclopedia/temporal-client) — the object your application uses to start
Workflows, send Signals, run Queries, and fetch results.

Temporal has eight SDKs: [.NET](/develop/dotnet), [Go](/develop/go), [Java](/develop/java),
[PHP](/develop/php), [Python](/develop/python), [Ruby](/develop/ruby), [Rust](/develop/rust), and
[TypeScript](/develop/typescript). Workflows written in different languages can call each other, so
a polyglot organization does not have to standardize on one language first. Adding an SDK to an
existing project does not change how you build or deploy it. For what each SDK provides, see
[About Temporal SDKs](/encyclopedia/architecture/temporal-sdks).

## Workers

A Worker is the part of your application that runs Workflow and Activity code. You build it with an
SDK, and you deploy it on your own infrastructure.

A common misconception is that the Temporal Service runs your code. It does not. Workers run your
code and handle your data, inside your own network, with your own encryption libraries and keys.
The Temporal Service schedules the work and records what happened.

Workers scale horizontally: one while you develop, and dozens, hundreds, or thousands in production.
They have historically been long-running processes on bare metal, virtual machines, or in
containers. [Serverless Workers](/serverless-workers) also run them on AWS Lambda and GCP Cloud Run,
with no process to keep alive between invocations.

Long-running Workflows outlive the code that started them, which is the hardest part of operating
one. [Worker Versioning](/worker-versioning) pins each Workflow Execution to the Worker Deployment
Version it started on, so a running execution finishes on the code it began with while new
executions go to the new version.

### Task Queues

Workers do not receive work; they ask for it. Each Worker polls one or more named
[Task Queues](/task-queue), and the Temporal Service matches pending [Tasks](/tasks) to the Workers
polling that queue. Nothing is pushed to a Worker, so a Worker that is down or saturated stops
polling and its Tasks wait until a Worker is available.

Task Queues are also how you route and isolate work: a queue per service, per Worker fleet, or per
hardware type. [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness) dispatches
urgent Tasks ahead of bulk work and keeps one tenant or customer from monopolizing a queue.

## The Temporal Service

The Temporal Service is the backend that orchestrates executions and stores their state. It is
several services behind a single gRPC endpoint:

- The **Frontend Service** is the API that Clients, Workers, the CLI, and the Web UI all connect to.
  It handles authorization, rate limiting, and routing.
- The **History Service** owns each Workflow Execution: its state and its Event History, persisted
  to a database.
- The **Matching Service** manages the Task Queues and hands Tasks to polling Workers.
- The **Worker Service** runs Temporal's own internal Workflows, such as archival and Visibility
  indexing. It is separate from the Workers that run your code.

Persistence is Cassandra, MySQL, PostgreSQL, or SQLite, with Elasticsearch as an option for advanced
Visibility.

Run the Temporal Service yourself, or use [Temporal Cloud](/evaluate/cloud) and skip operating the
database, the shards, and multi-region failover. For a walkthrough of the services and what each one
does on a request, see [Temporal architecture](/encyclopedia/architecture/temporal-architecture).

## Event History

The [Event History](/encyclopedia/event-history) is the durable log behind Durable Execution: a
complete, ordered record of everything that happened in a Workflow Execution.

Workflow code does not perform actions directly. When your Workflow calls an SDK API to execute an
Activity or start a Timer, the SDK records a [Command](/references/commands), and the Worker sends
it to the Temporal Service when the Workflow Task completes. The Temporal Service acts on the
Command, scheduling the Activity or setting the Timer, and appends the corresponding Events to the
Event History.

That is why a crash is recoverable. A new Worker fetches the Event History,
[replays](/workflow-execution#replay) the Workflow code against it to rebuild local state, and
continues from the point of failure as if the failure never happened.

For the step-by-step exchange between Client, Temporal Service, and Worker, including an interactive
demo, see [How Temporal works](/encyclopedia/architecture/how-temporal-works).

## Tools to inspect and steer executions

Every Workflow Execution carries a durable record of what happened, so debugging starts from
evidence rather than reconstruction.

### Temporal UI

The Temporal UI, also called the Web UI, is a browser interface for the state of your application:
which Workflow Executions are running, their inputs and outputs, and the full Event History of any
execution, including which Activity failed and with what error.

![Recent Workflows page](https://docs.temporal.io/img/webui/workflow-details-page-hiw.avif)

[List Filters](/list-filter) and Search Attributes query executions by your own business data, so
"every order stuck awaiting fraud review" is one query.

### Temporal CLI

The [Temporal CLI](/cli) does the same work from a terminal or a script. Through it you can:

- Start a Workflow and pass it input
- Trace the progress of a Workflow Execution
- Inspect or export an Event History
- Cancel or terminate an execution
- Manage Namespaces, Schedules, and Search Attributes

`temporal server start-dev` also runs a local development Temporal Service and Web UI on your
machine, which is how most people write their first Workflow.

For the aggregate view, SDK and Temporal Service
[metrics](/evaluate/features/observability) feed Prometheus, Datadog, Elastic, and other
OpenMetrics consumers.

## Next steps

- [How Temporal works](/encyclopedia/architecture/how-temporal-works): The end-to-end lifecycle of a Workflow and Activity, step by step, with an interactive demo of each exchange.
- [Temporal architecture](/encyclopedia/architecture/temporal-architecture): The Frontend, History, Matching, and Worker services, the Persistence Layer, and how a Client reaches them.
- [Features](/evaluate/features): The development and production capabilities available through a Temporal SDK, from core primitives to observability.
- [Design patterns](/design-patterns): Reusable Workflow and Activity techniques with runnable code: entity Workflows, fan-out, polling, sagas, and retry shapes.

> **💡 Tip:**
>
> To write your first Workflow, follow a [getting started tutorial](https://learn.temporal.io/getting_started/)
> or take the [Introduction to Temporal 101](https://learn.temporal.io/courses/temporal_101/) course.
>

---

# Temporal use cases and design patterns

Source: https://docs.temporal.io/evaluate/use-cases-design-patterns

> Temporal in production at Stripe, Coinbase, Netflix, and others, with the design patterns and Guides that implement each application shape.

Across industries, Temporal applications fall into a handful of shapes: a transaction that must not
half-complete, a process that waits days for a person, an entity that accumulates state for years, a
pipeline that fans out across GPUs. This page groups those shapes, names companies running each one
in production, and points to the pattern or Guide that implements it.

[Why Temporal](/evaluate/why-temporal) covers what the platform replaces, and
[Understanding Temporal](/evaluate/understanding-temporal) covers the vocabulary.
[Design Patterns](/design-patterns) is the code-level catalog, and [Guides](/guides) are full
implementation walkthroughs built on top of those patterns.

## Temporal in production

These are the use cases where Temporal runs at scale today. For more companies and a deeper look at
each, see [Temporal in use](https://temporal.io/in-use).

### Transactions

Operations across two or more parties where a partial completion leaves money or data in the wrong
state.

- [Payment processing at Stripe](https://temporal.io/resources/on-demand/stripe)
- [Money movement at Coinbase](https://temporal.io/in-use/coinbase)
- [Content management at Box](https://temporal.io/resources/case-studies/box)

Start with the [Saga pattern](/design-patterns/saga-pattern) to compensate for steps that already
succeeded, and [Non-Retryable Errors](/design-patterns/non-retryable-errors) to fail fast on a
rejection that no retry will fix. See
[distributed transaction patterns](/design-patterns/distributed-transaction-patterns) to choose
between them.

### Business processes

A sequence of steps that ends in a service or product delivered to a customer, often with a person
in the middle.

- [Bookings at Turo](https://temporal.io/replay/videos/temporal-adoption-and-integration-at-turo)
- [Orders and logistics at Maersk](https://temporal.io/replay/videos/building-a-time-machine-for-the-logistics-industry)
- [Marketing campaigns at Airbnb](https://medium.com/airbnb-engineering/journey-platform-a-low-code-tool-for-creating-interactive-user-workflows-9954f51fa3f8)
- [Human-in-the-loop at Checkr](https://temporal.io/in-use/checkr)

Use the [Approval](/design-patterns/approval) pattern for the steps that block on a decision, and
[Updatable Timer](/design-patterns/updatable-timer) for deadlines that move. The
[reliable document approvals](/guides/reliable-document-approvals) Guide walks through a full
implementation.

### Entity lifecycle

A long-lived business entity — an account, a device, a loan application — that accumulates state
over months or years.

- [Mortgage underwriting at ANZ](https://temporal.io/in-use/anz-story)
- [Menu versioning at Yum! Brands](https://temporal.io/replay-2023/videos/synchronizing-concurrent-workflows)

Give each instance one [Entity Workflow](/design-patterns/entity-workflow) and drive every state
transition with a Signal or an Update, then use
[Continue-As-New](/design-patterns/continue-as-new) to keep Event History bounded. The
[loyalty points](/guides/entity-pattern-loyalty-points) and
[gaming sessions](/guides/durable-gaming-sessions) Guides show the pattern end to end.

### Operations and infrastructure

Repeatable internal automation: provisioning, deployments, and scheduled maintenance.

- [Infrastructure services at Datadog](https://www.youtube.com/watch?v=Hz7ZZzafBoE)
- [Custom CI/CD at Netflix](https://temporal.io/replay-2023/videos/actor-workflows-reliably-orchestrating-thousands-of-flink-clusters-at)

Run recurring work on a [Schedule](/schedule), wait on external state with the
[Polling](/design-patterns/polling) pattern, and route work to the right hardware with
[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue). The
[route specialized workloads](/guides/route-specialized-workloads) and
[temporary rate limit increases](/guides/temporary-rate-limit-increases) Guides cover the routing
and capacity cases.

### Data and AI pipelines

Multi-step extraction, embedding, transcoding, or batch inference work that fans out and must resume
without reprocessing what already finished.

- [Orchestrating video processing at Descript](https://temporal.io/blog/ai-ml-and-data-engineering-workflows-with-temporal#descript)
- [Automating data pipelines at Neosync](https://temporal.io/blog/ai-ml-and-data-engineering-workflows-with-temporal#neosync)

Fan out with [Child Workflows](/design-patterns/fanout-child-workflows) or a
[MapReduce Tree](/design-patterns/mapreduce-tree), page through unbounded input with the
[Batch Iterator](/design-patterns/batch-iterator), and cap concurrency with a
[Sliding Window](/design-patterns/sliding-window). When pipelines share Workers across models or
tenants, [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness) dispatches urgent
work ahead of bulk work and keeps one tenant from starving the others.

### AI agents

Agent loops are long-running and stateful, and the calls inside them fail: model requests time out,
tools rate-limit, approvals take a day. The loop is a Workflow, each model call and tool call is an
Activity, and a crash resumes the conversation instead of restarting it.

- [Reliable, observable agents at Lindy](https://temporal.io/resources/case-studies/lindy-reliability-observability-ai-agents-temporal-cloud)
- [Long-running, durable agents at Dust](https://temporal.io/blog/how-dust-builds-agentic-ai-temporal)
- [Account summaries with agents at ZoomInfo](https://temporal.io/resources/on-demand/account-summaries-gen-ai)

[Durable AI](/ai) is the entry point for this work: runnable [AI Cookbook](/ai/cookbook) recipes for
tool calling, MCP, structured output, and human-in-the-loop, plus the agent framework integrations
for each SDK.

## Common application patterns

Several shapes recur regardless of industry. Each one below has a reusable implementation in the
[Design Patterns](/design-patterns) catalog.

### Human in the loop

Some steps can't proceed until a person acts: an onboarding form, an invoice approval, a manual
fraud review. The wait is the hard part, because it can last days and the connection between the
person and the rest of the system is unreliable.

A Workflow handles this without a callback table or a polling job. It blocks on a
[Signal or Update](/evaluate/features/workflow-message-passing), sets a durable
[Timer](/workflow-execution/timers-delays#timer) for the reminder or the deadline, and keeps its
place across Worker restarts in between. See the [Approval](/design-patterns/approval) pattern for
the reusable version and the [background checks example](https://learn.temporal.io/examples/go/background-checks/)
for a complete application in Go.

### Background jobs and long-running work

To run a single unit of work durably, use a [Standalone Activity](/standalone-activity): a top-level
Activity Execution started directly by a Client, with no Workflow around it. It gives you retries
with backoff, an Activity ID and Run ID you can cancel or query, and deduplication at submit time,
without running a broker. This is Temporal's [job queue](/evaluate/features/job-queue), and
Standalone Activities are Generally Available. Teams moving off Celery, Sidekiq, or a homegrown
queue can start with the [Celery migration Guide](/guides/celery-to-standalone-activity).

Jobs that take longer than a broker's poll timeout are where this matters most. A message queue
with a five-minute visibility timeout hands the same message to a second consumer while the first
is still working; Temporal tracks the Activity Execution itself, so a long job doesn't get
duplicated. Report progress with an
[Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) so a job that
dies mid-run is detected before its Start-To-Close Timeout expires, and resumes from its last
reported checkpoint. See the [Long-Running Activity](/design-patterns/long-running-activity)
pattern.

### Polyglot systems

Temporal has eight SDKs — [.NET](/develop/dotnet), [Go](/develop/go), [Java](/develop/java),
[PHP](/develop/php), [Python](/develop/python), [Ruby](/develop/ruby), [Rust](/develop/rust), and
[TypeScript](/develop/typescript) — and Workflows written in different languages can call each
other, so teams keep the language they already use.

The [polyglot sample](https://github.com/temporalio/temporal-polyglot) shows Workflow Executions in
Go, Java, PHP, and TypeScript sending messages to each other, including how errors propagate across
the language boundary. For calls between teams that don't share a Namespace, deploy schedule, or
database, [Temporal Nexus](/evaluate/features/nexus) puts a service contract between them.

### State machines

A state machine changes a system's behavior in response to changes in its state. Applying one to a
real business process usually means a transition table, a persistence layer for the current state,
and a job to detect rows that got stuck — none of which is the process you set out to model.

In Temporal, the current state is the position of a running Workflow. Express transitions as
ordinary control flow and hold state in local variables, and Temporal persists it for you. The
[Entity Workflow](/design-patterns/entity-workflow) pattern is the general form, and
[entity lifecycle patterns](/design-patterns/entity-lifecycle-patterns) covers the variants. For the
longer argument, see the
[State Machines Simplified whitepaper](https://pages.temporal.io/download-state-machines-simplified.html).

## Find an implementation

- [Design patterns](/design-patterns): Reusable Workflow and Activity techniques with runnable code across SDKs, grouped by problem domain.
- [Guides](/guides): Step-by-step walkthroughs for specific use cases, written and verified by Temporal staff.
- [Durable AI](/ai): Agents, processing pipelines, and model training on Temporal, with cookbook recipes and framework integrations.
- [Code Exchange](https://temporal.io/code-exchange): Community and Temporal-built sample applications you can run and adapt.

> **💡 Tip:**
>
> To write your first Workflow, follow a [getting started tutorial](https://learn.temporal.io/getting_started/),
> then work through the [project-based tutorials](https://learn.temporal.io/tutorials/) and
> [example applications](https://learn.temporal.io/examples/).
>

---

# Why Temporal?

Source: https://docs.temporal.io/evaluate/why-temporal

> Durable Execution runs your code to completion across crashes and outages, keeping retries, state recovery, and queue plumbing out of your business logic.

A process crashes halfway through a payment. An API times out on the third of five calls. A step
waits three days for a person to approve it. The usual answer is retry logic, a state table, a
message queue, a scheduler, and a reconciliation job — code that has nothing to do with the problem
you set out to solve, and that you have to test and operate forever.

Temporal runs your application code as a _Durable Execution_. Once started, a
[Workflow](/workflows) runs to completion, whether that takes seconds or months. When a
[Worker](/workers) crashes, the Temporal Service hands the work to another Worker, which
replays the [Event History](/encyclopedia/event-history) and resumes at the line where execution
stopped, with local variables and progress intact.

> **💡 Tip:**
> See Temporal in action
> Watch the following video to see how Temporal ensures an order-fulfillment system can recover from various failures, from process crashes to unreachable APIs.
>
> [Watch: Recovering an order-fulfillment system from failures with Temporal](https://www.youtube.com/watch?v=dNVmRfWsNkM)

## What Temporal replaces

Most of what Temporal gives you is code you no longer have to maintain.

| Problem                        | Without Temporal                                                    | With Temporal                                                                                            |
| ------------------------------ | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| A step fails                   | Retry loops and backoff in every call site                          | A [Retry Policy](/encyclopedia/retry-policies) enforced by the Temporal Service                          |
| A process crashes mid-sequence | State table, checkpoint columns, recovery job                       | Event History and [Replay](/workflow-execution#replay)                                                   |
| A step takes days              | Scheduler, polling job, timeout sweeper                             | A [Timer](/workflow-execution/timers-delays#timer) in Workflow code, durable across restarts             |
| A step needs a human           | Callback endpoint plus a table row to correlate it                  | [Signals, Updates, and Queries](/evaluate/features/workflow-message-passing) on a running Workflow       |
| A transaction half-completes   | Hand-written compensation and cleanup                               | The [Saga pattern](/design-patterns/saga-pattern) in ordinary control flow                               |
| Background jobs                | A broker, workers, a dead-letter queue, a dashboard                 | [Standalone Activities](/standalone-activity) as a [job queue](/evaluate/features/job-queue)             |
| "What is this thing doing?"    | Log aggregation and inference                                       | The [Web UI](/web-ui), the [CLI](/cli), and a durable record of every step                               |

## Failure handling you don't write

Temporal moves failure handling from your application into the platform. An
[Activity](/activities) that calls an external service gets timeouts, retries with exponential
backoff, and heartbeats from configuration rather than from code you wrote. A Workflow that
orchestrates several Activities keeps its place across Worker restarts, deploys, and
[Temporal Service](/temporal-service) upgrades.

What you write is the business logic: call this service, wait for that approval, then charge the
card. What you skip is the layer underneath it.

## One programming model, from one job to a whole system

The same Activity function runs at three scales, so an early decision doesn't lock you in:

- **One job.** A [Standalone Activity](/standalone-activity) is a single Activity Execution started
  directly by a Client, with no Workflow around it. It's Temporal's
  [job queue](/evaluate/features/job-queue): durable retries, a Run ID you can cancel or query,
  and deduplication at submit time, without running a broker. Teams moving off Celery, Sidekiq, or
  a homegrown queue can start here — see the
  [Celery migration guide](/guides/celery-to-standalone-activity).
- **One process.** Wrap several Activities in a Workflow when you need ordering, state between
  steps, or compensation. The Activity code doesn't change.
- **Many teams.** [Temporal Nexus](/evaluate/features/nexus) connects Workflows across isolated
  Namespaces through a service contract, so one team can call another team's process without
  sharing a Namespace, a database, or a deploy schedule.

The [Design Patterns](/design-patterns) library covers some common cases like, entity Workflows,
fan-out, sliding windows, polling, early return, distributed transactions.

## Deploy changes to code that's already running

Long-running Workflows outlive the code that started them, which is the hardest part of operating
one. [Worker Versioning](/worker-versioning) pins each Workflow Execution to the Worker Deployment
Version it started on, so a running execution finishes on the code it began with while new
executions go to the new version. You get gradual traffic ramps, verification before production
traffic, and instant rollback. For most teams deploying Workflow code changes, this is the default
approach.

## Keep work moving under load

A shared Task Queue eventually has a noisy neighbor: one tenant's bulk import starving everyone
else, or a batch job crowding out interactive requests.
[Task Queue Priority and Fairness](/develop/task-queue-priority-fairness) handles both in the
platform. Priority dispatches urgent Tasks ahead of bulk work, and Fairness keeps one key, like a
tenant, a customer, a model, from monopolizing a priority level. Both work on Temporal Cloud and
self-hosted deployments, and Priority is on by default.

## See and steer what's running

Every Workflow Execution carries a complete, durable record of what happened, so debugging starts
from evidence instead of reconstruction.

- The [Web UI](/web-ui) shows the input, output, and Event History of any execution, including
  which Activity failed and with what error.
- The [Temporal CLI](/cli) starts, inspects, cancels, and terminates executions from a terminal or
  a script.
- [List Filters](/list-filter) and Search Attributes query executions by business data, so "every
  order stuck awaiting fraud review" is one query.
- [Workflow Pause](/encyclopedia/workflow/workflow-pause) holds a specific execution in place during
  an incident or a dependency outage, without terminating it or losing state.
- SDK and Temporal Service [metrics](/evaluate/features/observability) feed Prometheus, Datadog,
  Elastic, and other OpenMetrics consumers for the aggregate view.

## Durable AI agents

Agent loops are long-running, stateful, and full of calls that fail: LLM requests that time out,
tools that rate-limit, approvals that take a day. These are the failure modes Temporal already
handles, which is why agents run on the same primitives as everything else. The loop is a
Workflow, each model call and tool call is an Activity, and a crash resumes the conversation
instead of restarting it.

[Durable AI](/ai) covers the cookbook recipes, the design patterns for agent workloads, and
integrations with the SDKs and frameworks teams build agents on. If instead you want a coding
assistant to help you write Temporal code, see [Develop with AI](/with-ai).

## Where your code runs

Temporal has eight SDKs — [.NET](/develop/dotnet), [Go](/develop/go), [Java](/develop/java),
[PHP](/develop/php), [Python](/develop/python), [Ruby](/develop/ruby), [Rust](/develop/rust), and
[TypeScript](/develop/typescript) — and Workflows in different languages can call each other, so a
polyglot organization doesn't have to standardize first. Your code runs in your own infrastructure: Workers execute Workflows and Activities and see
your data, while the Temporal Service orchestrates and records.

Run the Temporal Service yourself, or use [Temporal Cloud](/evaluate/cloud) and skip operating the
database, the shards, and the multi-region failover. For Workers,
[Serverless Workers](/evaluate/features/serverless-workers) run on AWS Lambda or GCP Cloud Run with
no Worker process to keep alive and no idle compute to pay for.

## Next steps

- [Understanding Temporal](/evaluate/understanding-temporal): How the Temporal Service, Workers, Workflows, and Activities fit together to run your application code.
- [Features](/evaluate/features): The development and production capabilities available through a Temporal SDK, from core primitives to observability.
- [Use cases](/evaluate/use-cases-design-patterns): How organizations run Temporal in production, and the architectural design patterns behind those applications.

---

# External Storage

Source: https://docs.temporal.io/external-storage

> **Public Preview**
> APIs and configuration may change before General Availability. Join the
> [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for
> help.

External Storage offloads payloads to an external store (such as Amazon S3) and passes a small reference token through
the Event History instead. This is called the
[claim check pattern](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern).

For SDK-specific usage guides, see:

- [Go SDK: Large payload storage](/develop/go/data-handling/external-storage)
- [Java SDK: Large payload storage](/develop/java/data-handling/external-storage)
- [Python SDK: Large payload storage](/develop/python/data-handling/external-storage)
- [TypeScript SDK: Large payload storage](/develop/typescript/best-practices/data-handling/external-storage)

## Why use External Storage

The Temporal Service enforces a maximum per-payload size. The default and recommended limit is 2 MB. Self-hosted users
can [configure this limit](/self-hosted-guide/defaults), but it is fixed at 2 MB on Temporal Cloud. Payloads that exceed
this limit fail the operation. Without External Storage, you must restructure your code to work around the limit, for
example by splitting data across multiple Workflows.

Even when individual payloads stay under the hard limit, payload data accumulates in Event History. Every Activity input
and output is persisted, so Workflows that pass data through many Activities can see history size grow quickly. Large
histories degrade Workflow Task latency. You may use [Continue-as-New](/workflow-execution/continue-as-new) to work
around this problem, but that comes with other tradeoffs.

External Storage addresses several common scenarios:

- **Data processing pipelines.** Workflows that process documents, images, or other large blobs can exceed the
  per-payload limit.
- **AI agent conversations.** Long conversation histories grow with each turn, and the cumulative size can degrade
  Workflow performance.
- **Spiky data sizes.** Some Workflows handle data that is usually small but occasionally large. The Claim check pattern
  handles these spikes transparently, offloading only the payloads that reach the size threshold.
- **Migration to Temporal Cloud.** Self-hosted deployments may have higher configured payload limits. External Storage
  lets you migrate to Cloud without restructuring Workflows that exceed the 2 MB limit.
- **Data governance.** While Temporal supports end-to-end client-side encryption, some organizations prefer to store
  payload data in infrastructure they control. Lower the offload size threshold to externalize all payloads regardless
  of size. The setting varies by SDK; see the SDK-specific usage guides below.

For SDK-specific usage guides, see:

- [Go SDK: Large payload storage](/develop/go/data-handling/external-storage)
- [Java SDK: Large payload storage](/develop/java/data-handling/external-storage)
- [Python SDK: Large payload storage](/develop/python/data-handling/external-storage)
- [TypeScript SDK: Large payload storage](/develop/typescript/best-practices/data-handling/external-storage)

## How External Storage fits in the data conversion pipeline 

During [Data Conversion](/dataconversion), External Storage sits at the end of the pipeline, after both the
[Payload Converter](/payload-converter) and the [Payload Codec](/payload-codec):

![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)

When a Temporal Client sends a payload that reaches or exceeds the configured size threshold, the storage driver uploads
the payload to your external store and replaces it with a lightweight reference. Payloads below the threshold stay
inline in the Event History.

When the Temporal Service dispatches Tasks to the Worker, the process reverses. The Worker downloads the referenced
payloads from external storage in parallel, then passes them back through the Payload Codec and Payload Converter to
reconstruct the original data.

The SDK parallelizes uploads and downloads to minimize latency. When a single Workflow Task involves multiple payloads
that reach the threshold, the SDK uploads or downloads all of them concurrently rather than one at a time. This allows
external storage operations to scale well even when a Task carries many large payloads.

When a payload is offloaded to external storage, the Temporal UI displays a reference token instead of the actual data.
This is expected. Your application code receives the fully decoded result because the SDK transparently retrieves the
payload from external storage before returning it to your Workflow or Client.

Because External Storage runs after the Payload Codec, if you use an encryption codec, payloads are already encrypted
before upload to your store.

## Choose a storage system 

A production storage system should meet the following requirements:

- Store payload data durably and retain it for the full Workflow lifetime plus the Namespace retention period. See
  [Lifecycle management](#lifecycle) for details.
- Be reachable from every Client, Worker, and Codec Server that encodes or decodes payloads.
- Support your expected payload sizes.
- Return consistent data immediately after a write completes.
- Meet your latency and throughput requirements under realistic load.
- Provide appropriate controls for authentication, encryption, monitoring, and backup.

Start with an object store such as Amazon S3, Google Cloud Storage, or Azure Blob Storage unless you have a specific
reason to use a different system, such as lower latency or existing infrastructure constraints.

- **Amazon S3, Google Cloud Storage, Azure Blob Storage:** Default choice for durable payload storage. We provide a
  first-party S3 storage driver for the [Go](/develop/go/data-handling/external-storage#store-and-retrieve-large-payloads), [Java](/develop/java/data-handling/external-storage#store-and-retrieve-large-payloads-with-amazon-s3), [Python](/develop/python/data-handling/external-storage#store-and-retrieve-large-payloads-with-amazon-s3), and [TypeScript](/develop/typescript/best-practices/data-handling/external-storage#store-and-retrieve-large-payloads) SDKs,
  and a first-party Google Cloud Storage driver for the [Go](/develop/go/data-handling/external-storage#store-and-retrieve-large-payloads) and [TypeScript](/develop/typescript/best-practices/data-handling/external-storage#store-and-retrieve-large-payloads) SDKs.
- **Google Cloud Bigtable:** Low-latency reads on Google Cloud, but payloads must fit within Bigtable's cell and row size limits.
- **Redis:** Suitable when configured for durability (such as with AOF persistence), not as an evicting cache. Refer to the [Python Redis storage driver sample](https://github.com/temporalio/samples-python/tree/main/external_storage_redis) for an example implementation. 

## Storage drivers

A storage driver connects External Storage to a backing store. Each driver provides two operations:

- **Store**. Upload payloads and return a claim, which is a set of key-value pairs the driver uses to locate the payload
  later.
- **Retrieve**. Download payloads using the claims that `store` produced.

The S3 driver also includes diagnostic metadata in error messages, such as the AWS region, to help with troubleshooting
storage failures.

Temporal SDKs include built-in drivers for common storage systems like Amazon S3. You can configure multiple storage
drivers and use a selector function to route payloads to different drivers based on size, type, or other criteria such
as hot and cold storage tiers.

### Custom storage drivers

If the built-in drivers don't support your storage backend, you can implement a custom driver. For SDK-specific
examples, see:

- [Go SDK: Implement a custom storage driver](/develop/go/data-handling/external-storage#implement-a-custom-storage-driver)
- [Java SDK: Implement a custom storage driver](/develop/java/data-handling/external-storage#implement-a-custom-storage-driver)
- [Python SDK: Implement a custom storage driver](/develop/python/data-handling/external-storage#implement-a-custom-storage-driver)
- [TypeScript SDK: Implement a custom storage driver](/develop/typescript/best-practices/data-handling/external-storage#implement-a-custom-storage-driver)

For example, see the
[Redis storage driver sample](https://github.com/temporalio/samples-python/tree/main/external_storage_redis).

## Key configuration settings

Configure External Storage on the Data Converter. The key settings are:

- **Size threshold**. The driver offloads payloads whose serialized size reaches or exceeds this value. Defaults to
  256 KiB.
- **Drivers**. One or more storage driver implementations.
- **Driver selector**. When using multiple drivers, you must provide a function that chooses which driver handles each
  payload.

## Lifecycle management for external storage 

Temporal does not automatically delete payloads from your external store. Payloads can also be orphaned if a request
fails after the upload completes. We recommend you configure a lifecycle policy that both ensures these payloads are
eventually cleaned up and provides a grace period for debugging and recovery.

Your TTL must be long enough that payloads remain available for the entire lifetime of the Workflow plus its retention
window:

```
TTL > Maximum Workflow Run Timeout + Namespace Retention Period
```

For example, if your longest-running Workflow has a Run Timeout of 14 days and your Namespace retention period is 30
days, configure your lifecycle rule to expire objects after at least 44 days.

If your Workflows run indefinitely (no Run Timeout), there is no finite TTL that guarantees safety. Set a generous TTL
based on your operational needs. Use [Continue-as-New](/workflow-execution/continue-as-new) for Workflows that need to
run longer. The new run uploads fresh payloads, and the old run's payloads only need to survive through its retention
period.

## Durable External Storage 

External Storage stores payloads in a single storage backend. If that backend becomes unavailable, Workers can't
retrieve payloads. To protect against regional or provider failures, implement your drivers in a way that takes
advantage of your storage provider's redundancy features, such as cross-region replication and multi-region routing.

> **ℹ️ Info:**
> Storage failures are not retried within a Task attempt
>
> The SDK does not retry a failed `Store` or `Retrieve` call within the same Task attempt. If your storage driver
> returns an error, that failure fails the current Workflow Task or Activity Task attempt. Temporal retries the Task as
> a whole, and the new attempt retries the storage operation along with the rest of the Task. For Activities, the
> [Retry Policy](/encyclopedia/retry-policies) controls the timing of that retry.
>

### S3 Multi-Region Access Points

The built-in S3 storage driver supports
[S3 Multi-Region Access Points (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/) as the bucket
endpoint. Combined with
[Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html), this gives you
automatic failover across regions: if a bucket or region becomes unavailable, requests route to the closest healthy
bucket.

For setup instructions, see
[Configuring S3 Multi-Region Access Points with replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointBucketReplication.html)
in the AWS documentation. After creating the MRAP, configure the External Storage S3 driver to use the MRAP ARN in place
of the bucket name.

### Replication tradeoffs

Cross-region replication introduces eventual consistency. After a write, a read in another region may temporarily miss
the object. To mitigate:

- Ensure Activities that read from External Storage have appropriate retry policies, so they recover from transient
  unavailability caused by replication lag.
- If an Activity needs to read a payload immediately after it is written, prefer scheduling it on the same Worker or in
  the same region to avoid the lag window.

By default, S3 CRR has no SLA on replication time. If you need stronger guarantees, enable
[Replication Time Control](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-time-control.html).

Replication and versioning can also significantly increase storage costs. Check your provider's pricing before enabling.

---

# Failure Converter

Source: https://docs.temporal.io/failure-converter

> A Failure Converter transforms error messages and call stacks into encoded formats to enhance security and observability.

This page discusses [Failure Converter](#failure-converter).

## What is a Failure Converter? 

As with input and output, Temporal also uses its own default converter logic for errors that are generated by Workflows.
The default Failure Converter copies error messages and call stacks as plain text, and this text output is then directly accessible in the `Message` field of these Workflow Executions.

This may be undesirable for your application. In some cases, errors could contain privileged or sensitive information that you would need to prevent from leaking or being available via a side channel.
Failure messages and call stacks are not encoded as codec-capable Payloads by default; you must explicitly enable encoding these common attributes on failures.

If your errors might contain sensitive information, you can encrypt the message and call stack by configuring the default Failure Converter to use your encoding.
This moves your `message` and `stack_trace` fields to a Payload that's run through your codec.

For example, with the Temporal Go SDK, you can do this by adding a `FailureConverter` parameter to your `client.Options{}` array when calling `client.Dial()`.
The `FailureConverter` should override the `DefaultFailureConverterOptions{}` by setting `EncodeCommonAttributes: true` like so:

```go
c, err := client.Dial(client.Options{
	// Set DataConverter here to ensure that workflow inputs and results are
	// encoded as required.
	DataConverter: mycustom.DataConverter,
	FailureConverter: temporal.NewDefaultFailureConverter(temporal.DefaultFailureConverterOptions{
		EncodeCommonAttributes: true,
	}),
})
```

If for some reason you need to specify a different set of Converter logic for your Failures, you can replace the `NewDefaultFailureConverter` with a custom method.
For example, if you are both working with highly sensitive data and using a sophisticated logging/observability implementation, you may need to implement different encryption methods for each of them.

---

# Global Namespace

Source: https://docs.temporal.io/global-namespace

> This guide covers everything about Global Namespaces within the Temporal Platform.

This page provides an overview of Global Namespace.

## What is a Global Namespace? 

A Global Namespace is a [Namespace](/namespaces) that exists across Clusters when [Multi-Cluster Replication](/temporal-service/multi-cluster-replication) is set up.

- [How to register a Global Namespace](/cli/command-reference/operator#create)
- [How to fail over a Global Namespace](/self-hosted-guide/multi-cluster-replication#how-to-fail-over-a-global-namespace)

The Global Namespace feature enables Workflow Executions to progress through another Cluster in the event of a failover.

A Global Namespace may be replicated to any number of Clusters, but only one Cluster is active at a time during normal operation.
During a failover, the source and destination Clusters can briefly disagree about which one is active until the failover replicates; see [How to fail over a Global Namespace](/self-hosted-guide/multi-cluster-replication#how-to-fail-over-a-global-namespace) for planned and forced failover procedures.

For a failover to be successful, Worker Processes must be polling for Tasks for the Global Namespace on all Clusters.

A Global Namespace has a failover version.
Because a failover can be triggered from any Cluster, the failover version prevents certain conflicts from occurring if a failover is mistakenly triggered simultaneously on two Clusters.

Only the active Cluster dispatches [Tasks](/tasks#task); however, certain conflicts are possible.
Unlike regular Namespaces, which provide at-most-once semantics for an Activity Execution, Global Namespaces can support only at-least-once semantics (see [Conflict resolution](/temporal-service/multi-cluster-replication#conflict-resolution)).
Worker Processes on the standby Clusters are idle until a failover occurs and their Cluster becomes active.

Temporal Application API calls made to a non-active Cluster are rejected with a **NamespaceNotActiveError** which contains the name of the current active Cluster.
It is the responsibility of the Temporal Application to call the Cluster that is currently active.

---

# Glossary

Source: https://docs.temporal.io/glossary

> The following terms have specific definitions within the context of the Temporal Platform.

The following terms are used in [Temporal Platform](/temporal) documentation.

#### [Action](/cloud/pricing#action)

An Action is the fundamental pricing unit in Temporal Cloud. Temporal Actions are the building blocks for Workflow
Executions. When you execute a Temporal Workflow, its Actions create the ongoing state and progress of your Temporal
Application.

<!-- _Tags: [term](/tags/term), [pricing](/tags/pricing), [temporal-cloud](/tags/temporal-cloud), [explanation](/tags/explanation)_ -->

#### [Actions per second (APS)](/evaluate/cloud/limits#actions-per-second)

APS, or Actions per second, is specific to Temporal Cloud. Each Temporal Cloud Namespace enforces a rate limit, which is
measured in Actions per second (APS). This is the number of Actions, such as starting or signaling a Workflow, that can
be performed per second within a specific Namespace.

<!-- _Tags: [term](/tags/term), [pricing](/tags/pricing), [temporal-cloud](/tags/temporal-cloud), [explanation](/tags/explanation)_ -->

#### [Activity](/activities)

In day-to-day conversation, the term "Activity" denotes an Activity Type, Activity Definition, or Activity Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Activity Definition](/activity-definition)

An Activity Definition is the code that defines the constraints of an Activity Task Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Activity Execution](/activity-execution)

An Activity Execution is the full chain of Activity Task Executions.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat)

An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Service.

Each ping informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Activity ID](/activity-execution#activity-id)

A unique identifier for an Activity Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Activity Task](/tasks#activity-task)

An Activity Task contains the context needed to make an Activity Task Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Activity Task Execution](/tasks#activity-task-execution)

An Activity Task Execution occurs when a Worker uses the context provided from the Activity Task and executes the
Activity Definition.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Activity Type](/activity-definition#activity-type)

An Activity Type is the mapping of a name to an Activity Definition.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Archival](/temporal-service/archival)

Archival is a feature specific to a Self-hosted Temporal Service that automatically backs up Event Histories from
Temporal Service persistence to a custom blob store after the Closed Workflow Execution retention period is reached.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion)

Asynchronous Activity Completion occurs when an external system provides the final result of a computation, started by
an Activity, to the Temporal System.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Audit Logging](/cloud/audit-logs)

Audit Logging is a feature that provides forensic access information for accounts, users, and Namespaces.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [temporal-cloud](/tags/temporal-cloud), [operations](/tags/operations)_ -->

#### [Authorizer Plugin](/self-hosted-guide/security#authorizer-plugin)

The `Authorizer` plugin contains a single `Authorize` method, which is invoked for each incoming API call. `Authorize`
receives information about the API call, along with the role and permission claims of the caller.

<!-- _Tags: [term](/tags/term)_ -->

#### [Availability Zone](/cloud/high-availability)

An availability zone is a part of the Temporal system where tasks or operations are handled and executed. This design
helps manage workloads and ensure tasks are completed. Temporal Cloud Namespaces are automatically distributed across
three availability zones, which is the basis of the uptime commitment in the Temporal Cloud [SLA](https://temporal.io/sla).

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Child Workflow](/child-workflows)

A Child Workflow Execution is a Workflow Execution that is spawned from within another Workflow.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [child-workflow](/tags/child-workflow)_ -->

#### [Claim Mapper](/self-hosted-guide/security#claim-mapper)

The Claim Mapper component is a pluggable component that extracts Claims from JSON Web Tokens (JWTs).

<!-- _Tags: [term](/tags/term)_ -->

#### [Codec Server](/codec-server)

A Codec Server is an HTTP server that uses your custom Payload Codec to encode and decode your data remotely through
endpoints.

<!-- _Tags: [term](/tags/term)_ -->

#### [Command](/workflow-execution#command)

A Command is a requested action issued by a Worker to the Temporal Service after a Workflow Task Execution completes.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Continue-As-New](/workflow-execution/continue-as-new)

Continue-As-New is the mechanism by which all relevant state is passed to a new Workflow Execution with a fresh Event
History.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [continue-as-new](/tags/continue-as-new)_ -->

#### [Core SDK](https://temporal.io/blog/why-rust-powers-core-sdk)

The Core SDK is a shared common core library used by several Temporal SDKs. Written in Rust, the Core SDK provides
complex concurrency management and state machine logic among its standout features. Centralizing development enables the
Core SDK to support quick and reliable deployment of new features to existing SDKs, and to more easily add new SDK
languages to the Temporal ecosystem.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [continue-as-new](/tags/continue-as-new)_ -->

#### [Custom Data Converter](/default-custom-data-converters#custom-data-converter)

A custom Data Converter extends the default Data Converter with custom logic for Payload conversion or Payload
encryption.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Data Converter](/dataconversion)

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Service.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Default Data Converter](/default-custom-data-converters#default-data-converter)

The default Data Converter is used by the Temporal SDK to convert objects into bytes using a series of Payload
Converters.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Delay Workflow Execution](/workflow-execution/timers-delays)

Start Delay determines the amount of time to wait before initiating a Workflow Execution. If the Workflow receives a
Signal-With-Start or Update-With-Start during the delay, it dispatches a Workflow Task and the remaining delay is
bypassed.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [delay-workflow](/tags/delay-workflow)_ -->

#### [Dual Visibility](/dual-visibility)

Dual Visibility is a feature, specific to a Self-hosted Temporal Service, that lets you set a secondary Visibility store
in your Temporal Service to facilitate migrating your Visibility data from one database to another.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [filtered-lists](/tags/filtered-lists), [visibility](/tags/visibility)_ -->

#### [Durable Execution](/temporal#durable-execution)

Durable Execution in the context of Temporal refers to the ability of a Workflow Execution to maintain its state and
progress even in the face of failures, crashes, or server outages.

<!-- _Tags: [temporal](/tags/temporal), [durable-execution](/tags/durable-execution), [term](/tags/term)_ -->

#### [Dynamic Handler](/dynamic-handler)

Dynamic Handlers are Workflows, Activities, Signals, or Queries that are unnamed and invoked when no other named handler
matches the call from the Server at runtime.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Event](/workflow-execution/event#event)

Events are created by a Temporal Service in response to external occurrences and Commands generated by a Workflow
Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Event History](/workflow-execution/event#event-history)

An append-only log of Events that represents the full state a Workflow Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Failback](/cloud/high-availability)

After Temporal Cloud has resolved an outage or incident involving a failover, a failback process shifts Workflow
Execution processing back to the original region that was active before the incident.

#### [Failover](/cloud/high-availability)

A failover shifts Workflow Execution processing from an active Temporal Namespace region to a standby Temporal Namespace
region during outages or other incidents. Standby Namespace regions use replication to duplicate data and prevent data
loss during failover.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Failure](/references/failures)

Temporal Failures are representations of various types of errors that occur in the system.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Failure Converter](/failure-converter)

A Failure Converter converts error objects to proto Failures and back.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Failures](/references/failures)

A Failure is Temporal's representation of various types of errors that occur in the system.

<!-- _Tags: [failure](/tags/failure), [explanation](/tags/explanation), [term](/tags/term)_ -->

#### [Frontend Service](/temporal-service/temporal-server#frontend-service)

The Frontend Service is a stateless gateway service that exposes a strongly typed Proto API. The Frontend Service is
responsible for rate limiting, authorizing, validating, and routing all inbound calls.

<!-- _Tags: [term](/tags/term)_ -->

#### [General Availability](/evaluate/product-release-stages#general-availability)

Learn more about the General Availability release stage

<!-- _Tags: [product-release-stages](/tags/product-release-stages), [term](/tags/term)_ -->

#### [Global Namespace](/global-namespace)

A Global Namespace is a Namespace that duplicates data from an active [Temporal Service](#temporal-cluster) to a standby
Service using the replication to keep both Namespaces in sync. Global Namespaces are designed to respond to service
issues like network congestion. When service to the primary Cluster is compromised, a [failover](#failover) transfers
control from the active to the standby cluster.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout)

A Heartbeat Timeout is the maximum time between Activity Heartbeats.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [High Availability](/cloud/high-availability/)

High availability ensures that a system remains operational with minimal downtime. It achieves this with redundancy and
failover mechanisms that handle failures, so end-users remain unaware of incidents. Temporal Cloud guarantees this high
availability with its Service Level Agreements ([SLA](https://temporal.io/sla))

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [High Availability features](/cloud/high-availability#high-availability-features)

High Availability features automatically synchronize your data between a primary Namespace and its replica, keeping them
in sync. In case of an incident or an outage, Temporal will automatically failover your Namespace from the primary to
the replica. This supports high levels of business continuity, allowing Workflow Executions to continue with minimal
interruptions or data loss.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [History Service](/temporal-service/temporal-server#history-service)

The History Service is responsible for persisting Workflow Execution state and determining what to do next to progress
the Workflow Execution through History Shards.

<!-- _Tags: [term](/tags/term)_ -->

#### [History Shard](/temporal-service/temporal-server#history-shard)

A History Shard is an important unit within a Temporal Service by which the scale of concurrent Workflow Execution
throughput can be measured.

<!-- _Tags: [term](/tags/term)_ -->

#### [Idempotency](/activity-definition#idempotency)

An "idempotent" approach avoids process duplication that could withdraw money twice or ship extra orders by mistake.
Idempotency keeps operations from producing additional effects, protecting your processes from accidental or repeated
actions, for more reliable execution. Design your activities to succeed once and only once. Run-once actions maintain
data integrity and prevent costly errors.

<!-- _Tags: [term](/tags/term)_ -->

#### [List Filter](/list-filter)

A List Filter is the SQL-like string that is provided as the parameter to an advanced Visibility List API.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [filtered-lists](/tags/filtered-lists), [visibility](/tags/visibility)_ -->

#### [Local Activity](/local-activity)

A Local Activity is an Activity Execution that executes in the same process as the Workflow Execution that spawns it.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Matching Service](/temporal-service/temporal-server#matching-service)

The Matching Service is responsible for hosting external Task Queues for Task dispatching.

<!-- _Tags: [term](/tags/term)_ -->

#### [Memo](/workflow-execution#memo)

A Memo is a non-indexed user-supplied set of Workflow Execution metadata that is returned when you describe or list
Workflow Executions.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication)

Multi-Cluster Replication is a feature which asynchronously replicates Workflow Executions from active Clusters to other
passive Clusters, for backup and state reconstruction.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Multi-cloud Replication](/cloud/high-availability/enable)

Multi-cloud Replication replicates Workflows and metadata to a different cloud provider (AWS or GCP). This is
particularly beneficial for organizations required to be highly available across regions for compliance purposes.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Multi-region Replication](/cloud/high-availability/enable)

Multi-region Replication replicates Workflows and metadata to a different region that is not co-located with the primary
Namespace. This is particularly beneficial for organizations with multi-regional architectures or those required to be
highly available across regions for compliance purposes.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Namespace](/namespaces)

A Namespace is a unit of isolation within the Temporal Platform.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [NexGen](/nexus/nexgen)

NexGen generates client code for Go, Java, Python, and TypeScript from a schema file that defines the contract. Both
sides can then use code generated from the same file, which gives data validation and type safety across the languages
and helps prevent drift.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus async completion callback

A Nexus async completion callback is the completion callback for an asynchronous Nexus Operation.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Endpoint

A Nexus Endpoint is a reverse proxy that can serve one or more Nexus Services. It routes Nexus requests to a target
Namespace and Task Queue, that a Nexus Worker is polling. This allows service providers to present a clean service
contract and hide the underlying implementation, which may consist of many internal Workflows. Multiple Nexus Endpoints
can target the same Namespace.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Machinery

Temporal has built-in Nexus Machinery to guarantee at-least-once execution of Nexus Operations with state-machine-based
invocation and completion callbacks. The Nexus Machinery uses [Nexus RPC](/glossary#nexus-rpc), a protocol designed with
Durable Execution in mind, to communicate across Namespace boundaries. Caller Workflows and Nexus handlers don't have to
use Nexus RPC directly, since the Temporal SDK provides a streamlined developer experience to build, run, and use Nexus
Services.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Operation

An arbitrary-duration operation that may be synchronous or asynchronous, short-lived, or long-lived, and used to connect
durable executions within and across Namespaces, clusters, regions, and clouds. Unlike a traditional RPC, an
asynchronous Nexus Operation has an operation token that can be used to re-attach to a long-lived Nexus Operation, for
example, one backed by a Temporal Workflow. Nexus Operations support a uniform interface to get the status of an
operation or its result, receive a completion callback, or cancel the operation – all of which are fully integrated into
the Temporal Platform.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Operation Events

Nexus Operations Events are history events that surface in the Caller Workflow to indicate the state of an Operation
including `Nexus Operation Scheduled`, `Nexus Operation Started`, `Nexus Operation Completed`.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Operation handler

The Nexus handler code in a Temporal Worker typically created using Temporal SDK builder functions that make it easy to
abstract Temporal primitives and expose a clean service contract for others to use.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Registry

The Nexus Registry manages Nexus Endpoints and provides lookup services for resolving Nexus requests at runtime. In the
open source version of Temporal, the Registry is scoped to a Cluster, while in Temporal Cloud, it is scoped to an
Account. Endpoint names must be unique within the Registry. When the Temporal Service dispatches a Nexus request, it
resolves the request's Endpoint to a Namespace and Task Queue through the Registry.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Nexus RPC](https://github.com/nexus-rpc/api/blob/main/SPEC.md)

Nexus RPC is a protocol designed with durable execution in mind. It supports arbitrary-duration Operations that extend
beyond a traditional RPC — a key underpinning to connect durable executions within and across Namespaces, clusters,
regions, and cloud boundaries.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Service

A Nexus Service is a named collection of arbitrary-duration Nexus Operations that provide a microservice contract
suitable for sharing across team and application boundaries. Nexus Services are registered with a Temporal Worker that
is polling a Nexus Endpoint's target Namespace and Task Queue.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### Nexus Service contract

A common code package, schema, or documentation that a Caller can use to obtain Service and Operation names as
associated input/output types a Service will accept for a given Operation.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Nexus Standalone Activity](/nexus/standalone-activity)

An Activity-backed Nexus Operation runs a Standalone Activity and completes when that Activity returns. Use it when the
work behind an Operation is one durable step rather than a process: calling an external API, running a computation,
writing to another system.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Parent Close Policy](/parent-close-policy)

If a Workflow Execution is a Child Workflow Execution, a Parent Close Policy determines what happens to the Workflow
Execution if its Parent Workflow Execution changes to a Closed status (Completed, Failed, Timed out).

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [child-workflow-executions](/tags/child-workflow-executions)_ -->

#### [Payload](/dataconversion#payload)

A Payload represents binary data such as input and output from Activities and Workflows.

<!-- _Tags: [term](/tags/term), [payloads](/tags/payloads), [explanation](/tags/explanation)_ -->

#### [Payload Codec](/payload-codec)

A Payload Codec transforms an array of Payloads into another array of Payloads.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Payload Converter](/payload-converter)

A Payload Converter serializes data, converting objects or values to bytes and back.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Pre-release](/evaluate/product-release-stages#pre-release)

Learn more about the Pre-release stage

<!-- _Tags: [product-release-stages](/tags/product-release-stages), [term](/tags/term)_ -->

#### [Public Preview](/evaluate/product-release-stages#public-preview)

Learn more about the Public Preview release stage

<!-- _Tags: [product-release-stages](/tags/product-release-stages), [term](/tags/term)_ -->

#### [Query](/sending-messages#sending-queries)

A Query is a synchronous operation that is used to report the state of a Workflow Execution.

<!-- _Tags: [term](/tags/term), [queries](/tags/queries), [explanation](/tags/explanation)_ -->

#### [Remote data encoding](/remote-data-encoding)

Remote data encoding is using your custom Data Converter to decode (and encode) your Payloads remotely through
endpoints.

<!-- _Tags: [term](/tags/term), [queries](/tags/queries), [explanation](/tags/explanation)_ -->

#### [Replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99)

The transmission delay of Workflow updates and history events from the active region to the standby region.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Requests per second (RPS)](/references/dynamic-configuration#service-level-rps-limits)

RPS, or Requests per second, is used in the Temporal Service (both in self-hosted Temporal and Temporal Cloud). This is
a measure that controls the rate of requests at the service level, such as the Frontend, History, or Matching Service.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [temporal](/tags/temporal)_ -->

#### [Reset](/workflow-execution/event#reset)

A Reset terminates a Workflow Execution and creates a new Workflow Execution with the same Workflow Type and Workflow
ID. The new Workflow Execution starts from a copy of the Event History up to the reset point, so progress made after the
reset point is discarded.

<!-- _Tags: [term](/tags/term), [resets](/tags/resets), [explanation](/tags/explanation)_ -->

#### [Retention Period](/temporal-service/temporal-server#retention-period)

A Retention Period is the amount of time a Workflow Execution Event History remains in the Temporal Service's
persistence store.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Retry Policy](/encyclopedia/retry-policies)

A Retry Policy is a collection of attributes that instructs the Temporal Server how to retry a failure of a Workflow
Execution or an Activity Task Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Run ID](/workflow-execution/workflowid-runid#run-id)

A Run ID is a globally unique, platform-level identifier for a Workflow Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Same-region Replication](/cloud/high-availability/enable)

Same-region Replication replicates Workflows and metadata to a separate cell within the same region as the primary
Namespace. It provides a reliable failover mechanism while maintaining deployment simplicity.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Schedule](/schedule)

A Schedule enables the scheduling of Workflow Executions.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout)

A Schedule-To-Close Timeout is the maximum amount of time allowed for the overall Activity Execution, from when the
first Activity Task is scheduled to when the last Activity Task, in the chain of Activity Tasks that make up the
Activity Execution, reaches a Closed status.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout)

A Schedule-To-Start Timeout is the maximum amount of time that is allowed from when an Activity Task is placed in a Task
Queue to when a Worker picks it up from the Task Queue. This timeout is non-retryable by design.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [Search Attribute](/search-attribute)

A Search Attribute is an indexed name used in List Filters to filter a list of Workflow Executions that have the Search
Attribute in their metadata.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [filtered-lists](/tags/filtered-lists), [visibility](/tags/visibility)_ -->

#### [Side Effect](/workflow-execution/event#side-effect)

A Side Effect is a way to execute a short, non-deterministic code snippet, such as generating a UUID, that executes the
provided function once and records its result into the Workflow Execution Event History.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Signal](/sending-messages#sending-signals)

A Signal is an asynchronous request to a Workflow Execution.

<!-- _Tags: [term](/tags/term), [signals](/tags/signals), [explanation](/tags/explanation)_ -->

#### [Signal-With-Start](/sending-messages#signal-with-start)

Signal-With-Start starts and Signals a Workflow Execution, or just Signals it if it already exists.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Standalone Activity](/standalone-activity)

A Standalone Activity is an Activity Execution invoked outside of a Workflow, directly from a Temporal Client. It has a
separate ID space from Workflows and other Temporal primitives. Existing Activity Functions can be invoked as
Standalone with no code changes.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout)

A Start-To-Close Timeout is the maximum time allowed for a single Activity Task Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [State Transition](/workflow-execution#state-transition)

A State Transition is a unit of progress by a Workflow Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Sticky Execution](/sticky-execution)

A Sticky Execution is a when a Worker Entity caches the Workflow Execution Event History and creates a dedicated Task
Queue to listen on.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Tag](/references/sdk-metrics)

A Tag is a key-value pair that a Temporal SDK or the Temporal Service attaches to an emitted metric, such as `namespace` or `task_queue`.
Once a metric is scraped by Prometheus or another OpenMetrics-compatible system, the same key-value pair is called a label.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Task](/tasks#task)

A Task is the context needed to make progress with a specific Workflow Execution or Activity Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Task Queue](/task-queue)

A Task Queue is a first-in, first-out queue that a Worker Process polls for Tasks.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Task Routing](/task-routing)

Task Routing is when a Task Queue is paired with one or more Worker Processes, primarily for Activity Task Executions.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Task Token](/activity-execution#task-token)

A Task Token is a unique identifier for an Activity Task Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal](/temporal)

Temporal is a scalable and reliable runtime for Reentrant Processes called Temporal Workflow Executions.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Application](/temporal#temporal-application)

A Temporal Application is a set of Workflow Executions.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal CLI](/cli) 

The Temporal CLI is the most recent version of Temporal's command-line tool.

<!-- _Tags: [term](/tags/term), [cli](/tags/cli)_ -->

#### [Temporal Client](/encyclopedia/temporal-client)

A Temporal Client, provided by a Temporal SDK, provides a set of APIs to communicate with a Temporal Service.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Cloud](/evaluate/cloud)

Temporal Cloud is a managed, hosted Temporal environment that provides a platform for Temporal Applications.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Cloud Account ID](/cloud/namespaces#temporal-cloud-account-id)

A Temporal Cloud Account ID is a unique identifier for a customer.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Cloud Namespace ID](/cloud/namespaces#temporal-cloud-namespace-id)

A Cloud Namespace ID is a globally unique identifier for a Namespace in Temporal Cloud.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Cloud Namespace name](/cloud/namespaces#temporal-cloud-namespace-name)

A Cloud Namespace name is a customer-supplied name for a Namespace in Temporal Cloud.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Cloud gRPC Endpoint](/cloud/namespaces#temporal-cloud-grpc-endpoint)

A Cloud gRPC Endpoint is a Namespace-specific address used to access Temporal Cloud from your code.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Cluster](/temporal-service)

The term "Temporal Cluster" is being phased out. Instead the term [Temporal Service](#temporal-service) is now being
used.

#### [Temporal Service](/temporal-service)

A Temporal Service is a Temporal Server paired with Persistence and Visibility stores.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Service configuration](/temporal-service/configuration)

Temporal Service configuration is the setup and configuration details of your Temporal Service, defined using YAML.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Cron Job](/cron-job)

A Temporal Cron Job is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to
spawn a Workflow Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Operation Handler](/nexus/temporal-operation-handler)

The Temporal Operation Handler is a single handler type that backs a Nexus Operation with a Workflow, an Update, or an
Activity, and links every Execution back to the caller. What runs behind an Operation remains private to the handler,
so you can change it later without touching the contract or any caller.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Platform](/temporal#temporal-platform)

The Temporal Platform consists of a Temporal Service and Worker Processes.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal SDK](/encyclopedia/architecture/temporal-sdks)

A Temporal SDK is a language-specific library that offers APIs to construct and use a Temporal Client to communicate
with a Temporal Service, develop Workflow Definitions, and develop Worker Programs.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Server](/temporal-service/temporal-server)

The Temporal Server is a grouping of four horizontally scalable services.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Temporal Web UI](/web-ui)

The Temporal Web UI provides users with Workflow Execution state and metadata for debugging purposes.

<!-- _Tags: [term](/tags/term), [web-ui](/tags/web-ui)_ -->

#### [Timer](/workflow-execution/timers-delays)

Temporal SDKs offer Timer APIs so that the Workflow Definition is deterministic in its handling of time values.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Update](/sending-messages#sending-updates)

An Update is a request to and a response from Workflow Execution.

<!-- _Tags: [term](/tags/term), [updates](/tags/updates), [explanation](/tags/explanation)_ -->

#### [Visibility](/temporal-service/visibility)

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view
Workflow Executions that currently exist within a Temporal Service.

<!-- _Tags: [term](/tags/term)_ -->

#### [Worker](/workers#worker)

In day-to-day conversations, the term Worker is used to denote both a Worker Program and a Worker Process. Temporal
documentation aims to be explicit and differentiate between them.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Worker Entity](/workers#worker-entity)

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Worker Process](/workers#worker-process)

A Worker Process is responsible for polling a Task Queue, dequeueing a Task, executing your code in response to a Task,
and responding to the Temporal Server with the results.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Worker Program](/workers#worker-program)

A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a
Temporal SDK.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Worker Service](/temporal-service/temporal-server#worker-service)

The Worker Service runs background processing for the replication queue, system Workflows, and (in versions older than
1.5.0) the Kafka visibility processor.

<!-- _Tags: [term](/tags/term)_ -->

#### [Worker Session](/task-routing#worker-session)

A Worker Session is a feature provided by some SDKs that provides a straightforward way to ensure that Activity Tasks
are executed with the same Worker without requiring you to manually specify Task Queue names.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow](/workflows)

In day-to-day conversations, the term "Workflow" frequently denotes either a Workflow Type, a Workflow Definition, or a
Workflow Execution.

#### [Workflow cache](/workflow-execution#workflow-cache)

An in-memory cache on a Worker that holds the state of Workflow Executions it has processed so later Workflow Tasks can avoid a full Event History replay. Used with [Sticky Execution](/sticky-execution). See also [Workflow Cache Tuning](/develop/worker-performance/workflow-cache).

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow Activity](/workflow-activity)

A Workflow Activity is an Activity Execution orchestrated by a Workflow. Its results are persisted to the Workflow and
Events are added to the Workflow Execution's Event History. Contrast with
[Standalone Activity](/standalone-activity) and [Local Activity](/local-activity).

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow Definition](/workflow-definition)

A Workflow Definition is the code that defines the constraints of a Workflow Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow Execution](/workflow-execution)

A Temporal Workflow Execution is a durable, scalable, reliable, and reactive function execution. It is the main unit of
execution of a Temporal Application.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)

A Workflow Execution Timeout is the maximum time that a Workflow Execution can be executing (have an Open status)
including retries and any usage of Continue As New.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [Workflow History Export](/cloud/export)

Workflow History export allows users to export Closed Workflow Histories to a user's Cloud Storage Sink.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [temporal-cloud](/tags/temporal-cloud), [operations](/tags/operations)_ -->

#### [Workflow ID](/workflow-execution/workflowid-runid#workflow-id)

A Workflow ID is a customizable, application-level identifier for a Workflow Execution that is unique to an Open
Workflow Execution within a Namespace.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow ID Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy)

A Workflow ID Conflict Policy determines how to resolve the conflict when spawning a new Workflow Execution with a
particular Workflow ID that is used by an Open Workflow Execution already.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow ID Reuse Policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy)

A Workflow ID Reuse Policy determines whether a Workflow Execution is allowed to spawn with a particular Workflow ID, if
that Workflow ID has been used with a previous, and now Closed, Workflow Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout)

This is the maximum amount of time that a single Workflow Run is restricted to.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [Workflow Task](/tasks#workflow-task)

A Workflow Task is a Task that contains the context needed to make progress with a Workflow Execution.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow Task Execution](/tasks#workflow-task-execution)

A Workflow Task Execution occurs when a Worker picks up a Workflow Task and uses it to make progress on the execution of
a Workflow Definition.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

#### [Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout)

A Workflow Task Timeout is the maximum amount of time that the Temporal Server will wait for a Worker to start
processing a Workflow Task after the Task has been pulled from the Task Queue.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation), [timeouts](/tags/timeouts)_ -->

#### [Workflow Type](/workflow-definition#workflow-type)

A Workflow Type is a name that maps to a Workflow Definition.

<!-- _Tags: [term](/tags/term), [explanation](/tags/explanation)_ -->

## Deprecated terms

#### `tctl` (_deprecated_)

tctl is a command-line tool that you can use to interact with a Temporal Service. It is superseded by the
[Temporal CLI utility](#cli).

---

# Guides

Source: https://docs.temporal.io/guides

> End-to-end Temporal implementations of distributed locks, API rate limits, approvals, loyalty programs, and Celery migration in Python, TypeScript, and Go.

Temporal Guides offer step-by-step solutions to specific implementation use cases. These Guides are written by Temporal staff and verified for accuracy. If you don't see your specific use case here, see the [Design Patterns](/design-patterns) for other solutions.

Browse Guides for Temporal SDKs and Temporal Cloud.
Filter by SDK, tag, or search to find what you need.

- [Durable agent on AgentCore](/guides/durable-agent-on-agentcore): Run a long-lived Strands agent with Temporal and serverless Worker compute on Amazon Bedrock AgentCore. (Python, AI agents)
- [Customer loyalty program](/guides/entity-pattern-loyalty-points): How to run a customer loyalty program with the entity pattern and durable workflows. (Python, Entity Pattern)
- [Recover without restart](/guides/recover-without-restart): Build business processes that pause on errors and recover without restarting. (TypeScript, Saga Pattern)
- [Route specialized workloads](/guides/route-specialized-workloads): Direct resource-intensive workloads to appropriate Task Queues to optimize resources. (Python, Task Queues)
- [Worker execution affinity](/guides/worker-execution-affinity): Direct multiple Activities to execute on the same Worker to maintain data locality. (Python, Worker management)
- [Temporary rate limit increases](/guides/temporary-rate-limit-increases): Handle temporary spikes in usage by dynamically provisioning extra capacity. (Go, Namespace management)
- [Reliable document approvals](/guides/reliable-document-approvals): Build durable human-in-the-loop Workflows. (Python, Human-in-the-loop)
- [Rate-limit downstream APIs](/guides/rate-limit-downstream-apis): Protect limited resources and avoid Workflow failures with separate Task Queues. (Python, Workflow Management)
- [Durable gaming sessions](/guides/durable-gaming-sessions): Protect player sessions from backend failures by using the Actor pattern. (Python, Actor Pattern)
- [Distributed locking](/guides/lock-shared-resources): Coordinate access to shared resources with a distributed lock. (Python, Workflow Management)
- [Migrate from Celery](/guides/celery-to-standalone-activity): Convert a Celery task queue to Temporal Standalone Activities, with retries and monitoring. (Python, Migration)

---

# Migrate a Celery task queue to a Temporal Standalone Activity

Source: https://docs.temporal.io/guides/celery-to-standalone-activity

> Map Celery tasks, .delay() and .get() calls, retries, and Flower monitoring to Standalone Activities started from a Temporal Client, with Python code.

[Celery](https://docs.celeryq.dev/en/stable/) is a distributed task queue that runs background jobs by pushing messages through a broker (such as Redis or RabbitMQ) to a pool of worker processes. Most Celery tasks are self-contained: send one email, resize one image, call one API. For that kind of single-step job, you want Durable Execution and automatic retries without having to stand up an orchestration layer around each task.

[**Standalone Activities**](/standalone-activity) are Temporal's [job queue](/evaluate/features/job-queue). A Standalone Activity is an Activity you start directly from a Temporal Client, without wrapping it in a Workflow. You get Temporal's durability, retries, timeouts, and visibility for an individual unit of work, which maps almost one-to-one onto a Celery task. Because there is no Workflow to run a single Activity, Standalone Activities also use fewer resources than orchestrating one Activity through a Workflow.

In this guide, you will migrate a Celery task to a Temporal Standalone Activity. You will convert the task into an Activity, run a Worker to process it, execute it both synchronously and fire-and-forget in place of your `.get()` and `.delay()` calls, migrate its retries to a Retry Policy, and inspect running Activities in place of Flower. By the end, you will have a working Temporal Application that reproduces the behavior of your Celery app with no Workflow code.

## How Celery concepts map to Standalone Activities

Before you start, it helps to know which Temporal building block replaces each Celery concept. You will implement each row of this table in the steps that follow.

| Celery | Temporal Standalone Activity | Purpose |
|---|---|---|
| Task (`@app.task`) | Activity (`@activity.defn`) | A single unit of work (I/O, API calls) |
| Worker (`celery worker`) | Worker (activities only, no Workflows) | Process that executes your code |
| Broker + result backend | Temporal Service | Durably stores queue state and results |
| `task.delay(...)` | `client.start_activity(...)` | Kick off work without waiting |
| `AsyncResult.get()` | `client.execute_activity(...)` or `handle.result()` | Retrieve the return value |
| `max_retries` / `self.retry()` | `RetryPolicy` | Automatic retries |
| Flower / `celery inspect` | `client.list_activities()` / `client.count_activities()` | Monitor jobs |
| `countdown` / `eta` | Start Delay | Run a job later instead of immediately |
| `task_routes` priority queues | [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness) | Control dispatch order under contention |
| `celery control` (`revoke`, rate limits) | [Activity Operations](/activity-operations) | Intervene in running jobs |

## Prerequisites

Before you begin, you will need the following:

- Python 3.10 or higher installed on your machine.
- Temporal Server v1.32.0 or higher (for Standalone Activities)
- The Temporal Python SDK, version 1.33.0 or higher (installed in Step 2).
- The Temporal CLI, version 1.9.1 or higher (installed in Step 2).
- An existing Celery task you want to migrate, or the sample task shown in Step 4 if you are following along from scratch.

## Step 1: Set up your project directory

In this step, you will create a small project layout. Because Standalone Activities need no Workflow code, the structure is flatter than a Workflow-based Temporal project: one module for the Activity, one Worker, and one script for each way of invoking the Activity.

Create a new project directory and move into it:

```bash
mkdir temporal-standalone && cd temporal-standalone
```

Your project will grow into the following files as you work through the tutorial:

```
temporal-standalone/
├── my_activity.py        # The Activity (your former Celery task)
├── worker.py             # Runs the Worker
├── execute_activity.py   # Runs the Activity and waits for the result
├── start_activity.py     # Starts the Activity without waiting
└── inspect_activities.py # Lists and counts Activities
```

With the directory in place, you can install the tools you need.

## Step 2: Install the Temporal SDK and CLI

In this step, you will install the Python SDK your code depends on and the Temporal CLI you will use to run a local server. Standalone Activities require specific minimum versions, so this step matters more than a normal install.

Install the Temporal Python SDK (version 1.33.0 or higher) with `pip`:

```bash
pip install "temporalio>=1.33.0"
```

Next, install the Temporal CLI (version 1.9.1 or higher). On macOS or Linux with [Homebrew](https://brew.sh/), run:

```bash
brew install temporal
```

If you aren't using Homebrew, download the binary for your platform from the [Temporal CLI install guide](/cli/setup-cli) and add it to your `PATH`.

Verify the CLI version, since Standalone Activities require 1.9.1 or higher:

```bash
temporal --version
```

Confirm the printed version is at least 1.9.1. With the tools installed, you can start a local Temporal Service.

## Step 3: Start the Temporal Development Server

In Celery, work flows through a broker such as Redis. In Temporal, work flows through the Temporal Service, which also stores each Activity's durable state. In this step, you will start a local development server that stands in for that service.

Start the development server:

```bash
temporal server start-dev
```

You will see output confirming the server is running, including two addresses:

```bash
Server:  localhost:7233
UI:      http://localhost:8233
```

Your application code will connect to `localhost:7233`. The Web UI at `http://localhost:8233` lets you inspect Activities and their results; Standalone Activities appear under their own item in the UI's navigation. Leave this process running and open a new terminal for the remaining steps.

## Step 4: Convert a Celery task into an Activity

In this step, you will take a Celery task and rewrite it as a Temporal Activity. The work inside (network calls, database writes, file I/O) stays the same. You write a Standalone Activity exactly the way you would write any Temporal Activity; nothing about the function marks it as "standalone." What makes it standalone is how you invoke it, which you will do in Step 6.

Consider a typical Celery task that sends a welcome email. Its `tasks.py` might look like this:

```python
# Celery version — tasks.py
from celery import Celery

app = Celery("myapp", broker="redis://localhost:6379/0")

@app.task(bind=True, max_retries=5, default_retry_delay=10)
def send_welcome_email(self, user_id):
    try:
        user = get_user(user_id)
        deliver_email(user.email, "Welcome!")
        return f"sent to {user.email}"
    except TransientError as exc:
        raise self.retry(exc=exc)
```

Create `my_activity.py` and add the Temporal equivalent:

```python
# my_activity.py
from dataclasses import dataclass
from temporalio import activity

@dataclass
class User:
    user_id: int
    email: str

@dataclass
class WelcomeEmailInput:
    user_id: int

# --- Mock helpers ---------------------------------------------------------
# Stand-ins for your real user lookup and email delivery. Replace these with
# your database query and email provider when you adapt the tutorial.

def get_user(user_id: int) -> User:
    return User(user_id=user_id, email=f"user{user_id}@example.com")

def deliver_email(address: str, subject: str) -> None:
    print(f"Delivering '{subject}' to {address}")
# --------------------------------------------------------------------------

@activity.defn
def send_welcome_email(input: WelcomeEmailInput) -> str:
    user = get_user(input.user_id)
    deliver_email(user.email, "Welcome!")
    return f"sent to {user.email}"
```

Two changes are worth noting. First, the retry boilerplate is gone: you no longer catch `TransientError` or call `self.retry()`, because Temporal retries a failed Activity automatically. You will configure how it retries in Step 8. Second, the Activity takes a single `dataclass` argument instead of positional parameters. Passing one structured argument is the recommended Temporal pattern, because it lets you add fields later without breaking callers.

The `User` dataclass and the two mock helpers let this file run end to end without a database or email provider. When you migrate your own task, swap `get_user` and `deliver_email` for your real implementations; the Activity itself does not change. Now you need a Worker to run it.

## Step 5: Run a Worker to process the Activity

Just as `celery worker` pulls jobs from a broker, a Temporal Worker polls a Task Queue for work. A Worker for Standalone Activities is an ordinary Temporal Worker with your Activities registered and no Workflows. In this step, you will create and start that Worker.

Create `worker.py`:

```python
# worker.py
import asyncio
from concurrent.futures import ThreadPoolExecutor

from temporalio.client import Client
from temporalio.worker import Worker

from my_activity import send_welcome_email

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue="email-tasks",
        activities=[send_welcome_email],
        activity_executor=ThreadPoolExecutor(max_workers=5),
    )
    print("Worker running...")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

The `task_queue` name is the routing key that ties your Worker and your invocation scripts together, similar to a Celery queue name. Because your Activity is a synchronous function, you pass a `ThreadPoolExecutor` as the `activity_executor`; `max_workers` controls how many Activities run concurrently, much like Celery's `--concurrency` flag.

Start the Worker:

```bash
python worker.py
```

The Worker begins polling the `email-tasks` Task Queue and waits for work. Leave it running and open another terminal to invoke it.

## Step 6: Execute an Activity in place of a blocking `.get()`

In Celery, running a task and waiting for its result looks like `send_welcome_email.delay(42).get()`. The Temporal equivalent is a single client call, `execute_activity`, which durably enqueues the Activity, waits for a Worker to run it, and returns the result. In this step, you will run your Activity and print its result.

Create `execute_activity.py`:

```python
# execute_activity.py
import asyncio
from datetime import timedelta

from temporalio.client import Client

from my_activity import WelcomeEmailInput, send_welcome_email

async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_activity(
        send_welcome_email,
        args=[WelcomeEmailInput(42)],
        id="welcome-email-42",
        task_queue="email-tasks",
        start_to_close_timeout=timedelta(seconds=30),
    )
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```

Run it:

```bash
python execute_activity.py
```

You will see the Activity's return value:

```
[secondary_label Output]
Result: sent to user42@example.com
```

A few details map directly from Celery. You pass the Activity's arguments through `args=[...]`, so the single `WelcomeEmailInput` goes in a list. The `id` you provide is a business identifier you choose (an order number, a user ID). Temporal uses it, along with the conflict policy and reuse policy, to guarantee the same Activity is not started twice, which is a built-in form of deduplication. Every Activity requires a timeout — `start_to_close_timeout` caps how long one attempt may run, replacing Celery's `task_time_limit`.

## Step 7: Start an Activity in place of `.delay()`

Celery's `.delay()` is fire-and-forget: it enqueues the task and returns immediately. The Temporal equivalent is `start_activity`, which durably enqueues the Activity and hands back a handle you can use later to fetch the result. In this step, you will start an Activity without blocking on it. (Note that the activity ID is changed to avoid deduplication effects.)

Create `start_activity.py`:

```python
# start_activity.py
import asyncio
from datetime import timedelta

from temporalio.client import Client

from my_activity import WelcomeEmailInput, send_welcome_email

async def main():
    client = await Client.connect("localhost:7233")

    handle = await client.start_activity(
        send_welcome_email,
        args=[WelcomeEmailInput(42)],
        id="welcome-email-delay",
        task_queue="email-tasks",
        start_to_close_timeout=timedelta(seconds=30),
    )
    print("Activity started")

    # Later, when you actually need the value, await the handle:
    result = await handle.result()
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```

Run it:

```bash
python start_activity.py
```

The pattern mirrors Celery precisely. `start_activity` corresponds to `.delay()` and returns a handle immediately, the way Celery returns an `AsyncResult`. Calling `handle.result()` corresponds to `AsyncResult.get()`. If you need to reconnect to an Activity from a different process — for example, a web request started it and a later request checks on it — recreate the handle from the Activity's ID and run ID (the run ID is available on the handle returned by `start_activity`):

```python
handle = client.get_activity_handle(
    activity_id="welcome-email-42",
    run_id="the-run-id",
)
```

## Step 8: Migrate task retries to a Retry Policy

In Celery, retries are your responsibility: you set `max_retries` and call `self.retry()` inside the task. With Temporal, retries are automatic and declarative. The default [Retry Policy](/develop/python/activities/timeouts#activity-retries) for Activities retries indefinitely with exponential backoff. This is sufficient in most cases, and typically doesn't need adjusting. 

However, if you want to replicate the Celery workflow exactly, migrating means adding a limit back in to match Celery's `max_retries`. In this step, you will attach a retry policy to the invocation. Note that Celery counts *retries* after the first attempt, whereas Temporal counts total attempts. Therefore, you'll set `maximum_attempts` to 6, which mimics Celery's `max_retries` of 5.

Update `execute_activity.py` to pass a `retry_policy`:

```python
# execute_activity.py (updated)
import asyncio
from datetime import timedelta

from temporalio.client import Client
from temporalio.common import RetryPolicy

from my_activity import WelcomeEmailInput, send_welcome_email

async def main():
    client = await Client.connect("localhost:7233")

    result = await client.execute_activity(
        send_welcome_email,
        args=[WelcomeEmailInput(42)],
        id="welcome-email-42",
        task_queue="email-tasks",
        start_to_close_timeout=timedelta(seconds=30),
        retry_policy=RetryPolicy(
            maximum_attempts=6,
            maximum_interval=timedelta(minutes=1),
            non_retryable_error_types=["InvalidUserError"],
        ),
    )
    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```

Here, `maximum_attempts=6` is similar to Celery's `max_retries`, and `maximum_interval` caps the backoff between attempts. The `non_retryable_error_types` list names errors that should fail immediately without retrying — the equivalent of *not* calling `self.retry()` for a permanent failure. To raise such an error from the Activity, use `ApplicationError` with `non_retryable=True` in `my_activity.py`:

```python
# my_activity.py (excerpt)
from temporalio.exceptions import ApplicationError

@activity.defn
def send_welcome_email(input: WelcomeEmailInput) -> str:
    user = get_user(input.user_id)
    if user is None:
        raise ApplicationError("No such user", type="InvalidUserError", non_retryable=True)
    deliver_email(user.email, "Welcome!")
    return f"sent to {user.email}"
```

Because the mock `deliver_email` in this tutorial never fails, the happy path completes on the first attempt. To watch a retry happen, make `deliver_email` raise an exception on its first call or two; Temporal will re-run the Activity automatically according to the policy above. The same `retry_policy` argument works on `start_activity` as well.

## Step 9: Inspect Activities in place of Flower

Celery users reach for Flower or `celery inspect` to see what is running. Temporal provides equivalent visibility directly through the client: you can list and count Standalone Activities that match a filter, the same way you would query Workflow Executions. In this step, you will write a small script to inspect your Activities.

Create `inspect_activities.py`:

```python
# inspect_activities.py
import asyncio

from temporalio.client import Client

async def main():
    client = await Client.connect("localhost:7233")

    query = "TaskQueue = 'email-tasks'"

    # List: like `celery inspect active`, but durable and queryable.
    async for info in client.list_activities(query=query):
        print(f"{info.activity_id} | {info.activity_type} | {info.status.name}")

    # Count: total executions (running, completed, failed), not queued tasks.
    resp = await client.count_activities(query=query)
    print(f"Total activities: {resp.count}")

if __name__ == "__main__":
    asyncio.run(main())
```

Run it:

```bash
python inspect_activities.py
```

You will see one line per Activity execution, followed by a total count:

```bash
welcome-email-42 | send_welcome_email | COMPLETED
Total activities: 1
```

The `query` uses the same [List Filter](/list-filter) syntax as Workflow visibility, so you can filter by attributes such as `ActivityType` and `ExecutionStatus` — for example, `"ActivityType = 'send_welcome_email' AND ExecutionStatus = 'Running'"`. These calls return only Standalone Activities; Activities running inside Workflows are excluded. The Temporal CLI offers the same views with `temporal activity list` and `temporal activity count`.

## Step 10: When to use a Workflow

Standalone Activities replace the common case: a Celery task that does one independent thing. They deliberately have no orchestration, so there is one situation they do not cover — multi-step pipelines. 

If your Celery app uses [Canvas](https://docs.celeryq.dev/en/stable/userguide/canvas.html) primitives — chaining tasks so one result feeds the next (`chain`), fanning work out in parallel (`group`), or running a callback after a group finishes (`chord`) — that coordination logic needs somewhere to live durably. A Standalone Activity cannot call another Activity or guarantee progress across several steps. For those pipelines, wrap your Activities in a Temporal **Workflow**, where sequencing is ordinary `await` statements and parallelism is `asyncio.gather`. See the [Temporal Python documentation](/develop/python) for building Workflows.

A rule of thumb: migrate a task to a Standalone Activity when it stands on its own, and to a Workflow when it coordinates other tasks. Most Celery tasks are the former.

## Conclusion

In this tutorial, you migrated a Celery task to a Temporal Standalone Activity. You converted the task into an Activity, ran a Worker to execute it, invoked it both synchronously and fire-and-forget in place of your `.get()` and `.delay()` calls, replaced hand-written retries with a retry policy, and inspected your Activities in place of Flower — all without writing a single Workflow. Your jobs now survive Worker crashes, retry on well-defined policies, and remain queryable through the client and Web UI.

Useful next topics:
- [Python SDK quickstart - Standalone Activities](/develop/python/activities/standalone-activities-quickstart) to get running on your machine.
- [Python SDK feature guide - Standalone Activities](/develop/python/activities/standalone-activities) for the full API surface.
- [Python SDK tutorial - Standalone Activities](https://learn.temporal.io/tutorials/python/standalone-activities/) to build a job queue with Standalone Activities.

## Before you benchmark

If you plan to compare throughput against your Celery deployment, match the Worker poller configuration first. Celery
prefetches several tasks per worker process by default (`worker_prefetch_multiplier`), while a Temporal Worker fetches
per poll. We recommend [poller autoscaling](/develop/worker-performance/configuration#configuring-poller-options) for 
more efficient poller usage, better throughput, and schedule-to-start latency improvements.

---

# Build a durable agent on Amazon Bedrock AgentCore

Source: https://docs.temporal.io/guides/durable-agent-on-agentcore

> Run a Strands agent as a Temporal Workflow while AgentCore Runtime supplies serverless Worker compute and a code execution tool.

> **Pre-release**
> Amazon Bedrock AgentCore Runtime support is in Pre-release, and its APIs may change in backwards-incompatible ways.
> To request Temporal Cloud access, create a [support ticket](/evaluate/cloud/support#support-ticket) or contact your
> account team. You can also [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or
> [try it through self-hosting](/production-deployment/worker-deployments/serverless-workers/agentcore/self-hosted-setup).

This guide deploys a Strands agent as a Temporal Serverless Worker on Amazon Bedrock AgentCore Runtime. The agent uses
Amazon Bedrock for model inference and AgentCore Code Interpreter to run Python.

If you already have an AgentCore application and are only interested in deploying a Worker Runtime, see
[Deploy a Serverless Worker on Amazon Bedrock AgentCore
Runtime](/production-deployment/worker-deployments/serverless-workers/agentcore). This guide uses a complete agent
sample to explain why the Workflow, Activities, Runtime, and Worker Deployment are structured this way.

## What you will build

You will build an agent that can respond to prompts by using Amazon Bedrock for model inference and AgentCore Code
Interpreter to run Python when needed. The sample accepts one prompt and runs one Workflow Execution. The Workflow asks
the model to answer the prompt. The model can call Code Interpreter through a Temporal Activity before returning its
answer.

Your Temporal Client starts the Workflow. When the Task Queue needs a Worker, Temporal starts an
AgentCore Runtime session. The Worker processes the Workflow and Activity Tasks, then drains after 60 seconds without
an Activity starting or finishing.

The sample handles one turn so the tutorial can focus on deployment. Both the model request and Code Interpreter call
run as Temporal Activities. If the Worker stops before an Activity completes, Temporal can retry the Activity on
another Worker according to its Retry Policy.

## Architecture

Temporal Cloud owns the agent's execution state and capacity control. The application starts or signals the Workflow.
The Workflow records agent decisions and schedules model and tool work as Temporal Tasks. When the Task Queue needs
capacity, the [Worker Controller Instance (WCI)](/serverless-workers#worker-controller-instance) starts AgentCore
Runtime sessions.

Each Runtime session hosts a Temporal Worker that polls the versioned Task Queue. Workers can call AgentCore services,
but the sessions and their process-local state remain replaceable. The sample in this guide uses AgentCore Code
Interpreter. It does not use every AgentCore service shown in the reference architecture.

![An application starts or signals a durable Strands agent Workflow in Temporal Cloud. The Workflow sends Tasks to a versioned Task Queue. A Worker Controller Instance observes demand and starts AgentCore Runtime sessions. Temporal Workers in those sessions poll and complete Tasks and can use AgentCore services.](https://docs.temporal.io/diagrams/temporal-agentcore-reference-architecture.png)

Place state according to how long it must remain available:

| State | Location | Reason |
|---|---|---|
| Agent progress and bounded working context | Temporal Workflow | Temporal reconstructs Workflow state from Event History when another Worker continues the execution. |
| Model calls and tool operations | Temporal Activities | Each operation gets its own timeout, Retry Policy, and recorded result. |
| Large conversations, uploads, and generated artifacts | External durable storage, with references in the Workflow | Large or unbounded data should not cause Event History to grow without limit. |
| Process-local caches and temporary files | AgentCore Runtime session | Runtime sessions are replaceable, so the agent must tolerate losing this state. |

This division is what lets the Workflow outlive any one Runtime session. A Worker can stop after the current work is
complete, and a later Worker can reconstruct the Workflow before continuing it.

To extend the sample to multiple turns, use one Workflow ID per conversation and send later prompts to that Workflow
through Updates. Any compatible Runtime session can process the next turn.

## Prerequisites

- A Temporal Cloud account with an AWS-hosted Namespace and access to the AgentCore Serverless Workers Pre-release.
- A Temporal Cloud API key that can connect to the Namespace.
- [Temporal CLI v1.8.3](https://github.com/temporalio/cli/releases/tag/v1.8.3) or later.
- Python 3.10 or later and [`uv`](https://docs.astral.sh/uv/).
- Node.js 20 or later and the
  [AgentCore CLI](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-get-started-cli.html).
- The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured for your AWS
  account.
- The [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting-started.html) installed and bootstrapped in an
  [AgentCore-supported Region](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-regions.html).
- AWS permissions to deploy AgentCore resources, CloudFormation stacks, and IAM roles. See
  [IAM permissions for AgentCore Runtime](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-permissions.html).
- Access to the Amazon Bedrock model that Strands selects in the target Region.

## 1. Get the sample

Clone the [Python samples repository](https://github.com/temporalio/samples-python/tree/main/bedrock_agentcore/strands_agent),
then install the sample's Python dependencies:

```bash
git clone --depth 1 https://github.com/temporalio/samples-python.git
cd samples-python/bedrock_agentcore/strands_agent
uv sync
```

The cloned directory contains the Workflow, Activity, Runtime handler, AgentCore configuration, deployment scripts,
and IAM template used throughout this guide.

The sample uses AgentCore's CodeZip build instead of a container image. AgentCore packages the Python project and runs
it on its managed Python runtime, so this path does not require a Dockerfile. The checked-in files define the
application and Runtime settings. The deployment script generates the AgentCore CDK project when you first run it.

## 2. Examine the agent Workflow

The sample creates a
[`TemporalAgent`](/develop/python/integrations/strands-agents) with a
system prompt and the `execute_code` tool:

<!--SNIPSTART python-agentcore-strands-workflow-->
[bedrock_agentcore/strands_agent/workflows.py](https://github.com/temporalio/samples-python/blob/main/bedrock_agentcore/strands_agent/workflows.py)
```py
@workflow.defn
class StrandsAgentWorkflow:
    def __init__(self) -> None:
        # Configure with the plugin's default BedrockModel(), custom system
        # prompt and code interpreter tool.
        self.agent = TemporalAgent(
            start_to_close_timeout=timedelta(seconds=60),
            system_prompt=SYSTEM_PROMPT,
            tools=[
                activity_as_tool(
                    execute_code,
                    start_to_close_timeout=timedelta(minutes=2),
                )
            ],
        )

    @workflow.run
    async def run(self, prompt: str) -> str:
        # invoke_async, not agent(prompt) -- the sync form spawns a worker thread the
        # Workflow sandbox blocks.
        result = await self.agent.invoke_async(prompt)
        return str(result)

```
<!--SNIPEND-->

`TemporalAgent` adapts the Strands agent loop to run in Workflow code. The
[Temporal Strands plugin](/develop/python/integrations/strands-agents) schedules each model call as an Activity.
`activity_as_tool` makes `execute_code` another Activity when the model selects that tool.

The Workflow owns the sequence of model and tool decisions because that sequence must resume correctly after a
failure. The model calls themselves do not run as ordinary Workflow code. They run as Activities because they perform
network I/O, can fail independently, and are not deterministic.

The `execute_code` Activity creates a Code Interpreter session using the Workflow ID as its session name:

<!--SNIPSTART python-agentcore-code-interpreter-activity-->
[bedrock_agentcore/strands_agent/activities.py](https://github.com/temporalio/samples-python/blob/main/bedrock_agentcore/strands_agent/activities.py)
```py
# Use AgentCore Code Interpreter to provide a code sandbox and execute LLM generated solution
@activity.defn
def execute_code(
    code: str, language: LanguageType = LanguageType.PYTHON
) -> dict[str, Any]:
    """Run code in this Sessions's sandbox (workflow ID) and return the Code Interpreter result."""
    interpreter = AgentCoreCodeInterpreter(
        region=os.environ.get("AWS_REGION", "us-west-2"),
        session_name=activity.info().workflow_id,
    )
    return interpreter.execute_code(
        ExecuteCodeAction(type="executeCode", code=code, language=language)
    )

```
<!--SNIPEND-->

Using the Workflow ID gives each Workflow Execution its own Code Interpreter sandbox.

## 3. Configure and deploy the Runtime

Open `agentcore/aws-targets.json`. Replace the account number and Region with the AWS account and Region where you
will deploy the Runtime:

```json
[
  {
    "name": "default",
    "description": "AWS account and Region for the Runtime",
    "account": "<AWS_ACCOUNT_ID>",
    "region": "<AWS_REGION>"
  }
]
```

Open `agentcore/agentcore.json` and replace the placeholder values for `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, and
`TEMPORAL_API_KEY`. Set `AWS_REGION` to the same Region used in `aws-targets.json`. Keep these sample values unchanged:

| Setting | Value |
|---|---|
| `TEMPORAL_TASK_QUEUE` | `agentcore-strands-task-queue` |
| `TEMPORAL_DEPLOYMENT_NAME` | `agentcore-strands-agent-python` |
| `TEMPORAL_BUILD_ID` | `1.0.0` |
| Runtime endpoint name | `temporal` |

These values connect two separately configured systems. The Runtime uses the Task Queue, deployment name, and Build ID
when its Worker registers with Temporal. The Worker Deployment Version created in Step 5 uses the same deployment name
and Build ID and points Temporal back to this Runtime endpoint. If the values differ, Temporal can start compute that
does not register as the version waiting for work.

Putting the API key in `agentcore.json` keeps the tutorial short. Do not commit the populated file. For a production
deployment, store the key in AWS Secrets Manager and load it when the Runtime starts.

Set the Region for the AWS CLI commands in this guide:

```bash
export AWS_REGION="<AWS_REGION>"
```

Deploy the Runtime and its named endpoint:

```bash
./bin/create-runtime.sh
```

The script creates the AgentCore CDK project on its first run, validates the configuration, packages the sample, and
deploys it. The sample uses public network mode so the Worker can make an outbound connection to Temporal Cloud. The
named endpoint is for capacity requests from Temporal, not prompts from the application.

Retrieve the Runtime and endpoint ARNs:

```bash
export AGENT_RUNTIME_ARN="$(
  aws bedrock-agentcore-control list-agent-runtimes \
    --region "$AWS_REGION" \
    --query "agentRuntimes[?agentRuntimeName=='TemporalStrandsAgent_temporal_strands_worker'].agentRuntimeArn | [0]" \
    --output text
)"
export AGENT_RUNTIME_ID="${AGENT_RUNTIME_ARN##*/}"
export RUNTIME_ENDPOINT_ARN="$(
  aws bedrock-agentcore-control list-agent-runtime-endpoints \
    --agent-runtime-id "$AGENT_RUNTIME_ID" \
    --region "$AWS_REGION" \
    --query "runtimeEndpoints[?name=='temporal'].agentRuntimeEndpointArn | [0]" \
    --output text
)"
echo "$AGENT_RUNTIME_ARN"
echo "$RUNTIME_ENDPOINT_ARN"
```

Both commands must print an ARN before you continue.

> **❗ Important:**
> Runtime ARN and endpoint ARN are different
>
> `AGENT_RUNTIME_ARN` ends with `/runtime/<runtime-id>`. It identifies the Runtime resource. The IAM role in Step 4 uses
> this ARN with a trailing wildcard to allow access to the Runtime and its endpoints.
>
> `RUNTIME_ENDPOINT_ARN` adds `/runtime-endpoint/temporal` to the Runtime ARN. It identifies the named endpoint that
> routes invocations to its configured Runtime version. The Worker Deployment Version in Step 5 uses this ARN to tell
> Temporal what to invoke.
>
> Do not use `AGENT_RUNTIME_ARN` as the `--aws-agentcore-endpoint-arn` value.
>

## 4. Grant Temporal access to the Runtime

Set the External ID, then use the sample's CloudFormation script to create the IAM role that Temporal Cloud assumes:

```bash
export EXTERNAL_ID="temporal-agentcore-tutorial"
export INVOCATION_STACK="ac-strands-invoke"

./bin/mk-invoke-role.sh \
  "$INVOCATION_STACK" \
  "$EXTERNAL_ID" \
  "${AGENT_RUNTIME_ARN}*"
```

The fixed External ID makes the tutorial commands repeatable. Use a unique External ID for a production deployment.

> **⚠️ Caution:**
>
> The sample names the IAM role `Temporal-Cloud-Serverless-Worker-<stack-name>`. An [IAM role name can contain at most 64
> characters](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-iam-role.html). Keep
> `INVOCATION_STACK` to 31 characters or fewer. The `ac-strands-invoke` value above is within the limit.
>

Wait for the stack and retrieve the role ARN:

```bash
aws cloudformation wait stack-create-complete \
  --stack-name "$INVOCATION_STACK" \
  --region "$AWS_REGION"

export INVOCATION_ROLE_ARN="$(
  aws cloudformation describe-stacks \
    --stack-name "$INVOCATION_STACK" \
    --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' \
    --output text \
    --region "$AWS_REGION"
)"
echo "$INVOCATION_ROLE_ARN"
```

This invocation role lets Temporal get the named endpoint and invoke the Runtime. It is separate from the Runtime
execution role that AgentCore created to run the Worker and access Code Interpreter.

Keeping the roles separate gives each side only the permissions it needs. Temporal assumes the invocation role to
start capacity. AgentCore assumes the execution role inside that capacity when the Worker calls Bedrock and Code
Interpreter. The trailing wildcard on the Runtime ARN allows the invocation role to cover the named endpoint as well
as the Runtime.

## 5. Create the Serverless Worker deployment

The values in `agentcore.json` configure the Worker inside AgentCore Runtime. They do not configure the Temporal CLI or
the sample client process. Before continuing, export `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and
`AWS_REGION` for those processes. From the `samples-python/bedrock_agentcore/strands_agent` directory, run this command
to read and export the values from `agentcore/agentcore.json`:

```bash
eval "$(python3 -c 'import json, shlex; env = {item["name"]: item["value"] for item in json.load(open("agentcore/agentcore.json"))["runtimes"][0]["envVars"]}; print("\n".join(f"export {name}={shlex.quote(env[name])}" for name in ("TEMPORAL_ADDRESS", "TEMPORAL_NAMESPACE", "TEMPORAL_API_KEY", "AWS_REGION")))')"
```

Create a Worker Deployment and a version that points to the AgentCore endpoint:

```bash
temporal worker deployment create \
  --name agentcore-strands-agent-python

temporal worker deployment create-version \
  --deployment-name agentcore-strands-agent-python \
  --build-id 1.0.0 \
  --aws-agentcore-endpoint-arn "$RUNTIME_ENDPOINT_ARN" \
  --aws-agentcore-assume-role-arn "$INVOCATION_ROLE_ARN" \
  --aws-agentcore-assume-role-external-id "$EXTERNAL_ID"

temporal worker deployment set-current-version \
  --deployment-name agentcore-strands-agent-python \
  --build-id 1.0.0 \
  --yes
```

Creating the version causes Temporal to invoke the Runtime and wait for the Worker to register. The deployment name
and Build ID match the values in `agentcore.json`. Setting the version as current lets it receive new Tasks on the
`agentcore-strands-task-queue` Task Queue.

The Worker Deployment Version binds one version of the Worker code to one compute configuration. The sample registers
Workflows with `PINNED` behavior, so a Workflow continues on its assigned version instead of moving to a newer version
while it is running. Marking `1.0.0` as current sends new Workflow Executions to that version.

## 6. Run the agent

Run the sample client with its default prompt:

```bash
uv run python starter.py
```

Or provide a prompt:

```bash
uv run python starter.py \
  "Calculate the first 10 Fibonacci numbers and verify the result with Python."
```

`starter.py` starts `StrandsAgentWorkflow` and waits for its result. Temporal starts AgentCore Worker capacity, the
Workflow calls the model and Code Interpreter Activities, and the client prints the answer. The Workflow then
completes. After 60 seconds without an Activity starting or finishing, the Worker drains.

Inspect the completed Workflow Execution:

```bash
temporal workflow show \
  --workflow-id agentcore-strands-workflow-id-1
```

The Event History contains the model and `execute_code` Activities. Search the previous hour of AgentCore Runtime logs:

```bash
agentcore logs --runtime temporal_strands_worker --since 1h
```

The Workflow history and AgentCore logs show the two sides of the integration. Event History records what the agent
did. The AgentCore logs show which replaceable Worker process performed the work and when that Worker drained.

---

# Build durable multiplayer game sessions with the actor model

Source: https://docs.temporal.io/guides/durable-gaming-sessions

> Model each player as an Actor Workflow that holds session state, runs game actions, and survives server crashes, with a complete Python implementation.

**by Cecil Phillip**

Multiplayer game backends run into the same wall at scale: player state lives in memory, pinned to a server, and when that server goes down the session goes with it. Reconnection logic is painful, distributed state is expensive, and cross-player coordination is worse. This guide shows how to model each player as their own durable, independently-running process — one that survives crashes, executes game actions, and stays addressable for as long as the session lasts.

## Problem statement

When a game server goes down, every player on it loses their session. Inventory, position, active buffs, current room — gone. Reconnection has to reconstruct state from whatever made it to the database before the crash, and something always gets missed. Scaling makes this worse: keeping per-player state consistent across a cluster requires distributed caches, locking, and sync logic that grows in complexity with every new game feature. Multi-step operations between players — trading items, triggering combat, changing rooms — need coordination logic that's hard to get right and even harder to debug when it goes wrong.

## Solution

Each player gets their own Workflow — a durable, uniquely-addressable process identified by their player ID. It holds inventory, health, position, and room state in memory. It doesn't just store state: it acts. Joining a room, executing a combat move, updating the leaderboard — each of these happens through an Activity the Workflow drives directly. When the server goes down, the Workflow resumes on another Worker from exactly the point it left off, with no state lost and no reconnection logic required. Sessions that run for weeks don't accumulate unbounded history — Continue-As-New handles that automatically.

## Outcomes

After working through this guide, you'll have a session system where:

- **Server crashes don't end player sessions**: state is durable and resumes on any available Worker without custom recovery logic
- **Game actions are reliable**: room joins, combat moves, and leaderboard updates execute through Activities with automatic retries — partial failures don't corrupt session state
- **Sessions run as long as needed**: Continue-As-New manages history growth so a session that lasts weeks costs no more to run than one that lasts minutes
- **Every player is independently addressable**: any service can send a command, Query state, or trigger an Update using the player's Workflow Id
- **Active sessions are queryable by operators**: Search Attributes let you find who's in which room, how long a session has been running, or which players are idle

## Background and best practices

### What is an Actor Workflow?

The Actor Workflow pattern extends the [Entity Workflow pattern](/design-patterns/entity-workflow). An Entity Workflow is a long-running Workflow that represents a *thing*: it holds state, responds to messages, and exposes that state through Queries. An Actor Workflow is an Entity Workflow that *does things*. The distinction is behavioral:

- **An entity is a thing. An actor is a thing that does things.**
- An Entity Workflow is analogous to a distributed data cache: it stores and retrieves state.
- An Actor Workflow is analogous to a distributed object with operations: it stores state *and* executes side effects on behalf of the entity it represents.

An Entity Workflow that takes action is an Actor Workflow. Concrete examples of actors include a Workflow that scales up a database when load increases, starts a car engine when a driver authenticates, submits an order when a customer confirms checkout, or manages a player session in a multiplayer game by joining rooms and executing combat moves.

AI agents are often modeled as Actor Workflows. They receive instructions (Temporal Signals or Updates), maintain conversational state, and take action by calling external services (Activities) such as language model APIs, tool integrations, and data retrieval systems.

### The actor model and Temporal

The actor model, originally formalized by Carl Hewitt in 1973, defines computation in terms of *actors*: autonomous units that communicate exclusively through asynchronous messages. Each actor:

1. Has a private, encapsulated state that other actors can not directly access.
2. Processes messages one at a time from an inbox (mailbox).
3. Can send messages to other actors.
4. Can create new actors.
5. Can change its own internal state in response to a message.

Temporal Workflows map naturally to this model:

| Actor model concept | Temporal equivalent |
|---|---|
| Actor identity | Workflow Id |
| Actor mailbox | Workflow Event History (Signals or Updates) |
| Processing a message | Signal handler, Update handler |
| Sending a message to another actor | Signaling an external Workflow |
| Creating a new actor | Starting a new Workflow Execution |
| Actor supervision | Retry Policies, parent-child relationships |
| Actor state persistence | Durable Execution (automatic via Event History) |

Traditional actor frameworks such as Akka, and Orleans require you to configure persistence, message delivery guarantees, and supervision hierarchies. Temporal provides these capabilities as platform primitives. Every Workflow Execution is durably persisted. Every Signal is reliably delivered. Every Activity execution is automatically retried according to its Retry Policy. You do not need to implement a persistence plugin, configure a journal, or write custom supervision strategies.

### Why actors matter for player sessions

A player session maps almost perfectly onto the actor model. Each player has private state that nothing else should write to directly — inventory, health, position, active buffs. They receive messages from clients, servers, and other players. They take real-world actions in response. And their lifecycle is unpredictable: a session might last twenty minutes or three months, with idle stretches in between.

Modeling each player as an Actor Workflow gives you a single authoritative representation of that session. There is no cache to invalidate, no database row to lock, no sticky session to maintain. The Workflow *is* the session, and it's always reachable by player ID.

### Event History and Continue-As-New

Every Workflow Execution in Temporal produces an append-only Event History. This history has a limit of 51,200 Events or 50 MB. For actor Workflows that run indefinitely, Event History growth must be managed proactively.

Continue-As-New atomically completes the current Workflow Execution and starts a new one with the same Workflow Id, carrying forward any state you provide as arguments. From the perspective of external callers, the Workflow Id remains the same. Signals sent during the transition are not lost; the Temporal Service buffers them for the new execution.

Before executing Continue-As-New, you must ensure that all in-progress Signal and Update handlers have finished processing. The `workflow.all_handlers_finished` predicate provides this guarantee.

### Signal volume limits

Two hard limits govern Signal volume per Workflow Execution:

- **10,000 total Signals** per Execution. Continue-As-New resets this counter, so an entity that transitions regularly is not constrained by it in practice.
- **2,000 pending Signals** (unprocessed Signals buffered by the server) at any one time. If this limit is reached, new Signals are rejected.

Beyond these limits, the practical throughput ceiling is Worker-side: each Signal triggers a Workflow Task, and a Workflow processes one Workflow Task at a time. For typical short tasks, this yields a few Workflow Tasks per second per Workflow Execution.

If your use case requires higher Signal ingestion rates — for example, streaming real-time game telemetry — consider one of these approaches:

- **Batch events into a single Signal payload.** Instead of one Signal per game event, batch several events into a list and send them as a single Signal. The handler appends the entire batch to the queue in one shot.
- **Use an aggregation layer.** Route high-frequency event streams through a service (such as a message broker or aggregator) that batches events before forwarding them as Signals. This decouples the producer's throughput from the Workflow's processing rate.

### Activity Heartbeating

Long-running Activities, such as maintaining a WebSocket connection to a game room or processing a large batch of leaderboard updates, should emit Heartbeats. A Heartbeat serves two purposes:

1. It tells the Temporal Service that the Activity is still making progress. If heartbeats stop arriving within the configured Heartbeat Timeout, the Temporal Service considers the Activity failed and schedules a retry.
2. It carries a custom payload that captures the Activity's progress. When the Activity is retried after a failure, the new attempt can read the last heartbeat payload and resume from where it left off.

Configure a short Heartbeat Timeout (for example, 30 seconds) and emit heartbeats frequently (for example, every 5 to 10 seconds). The SDK throttles Heartbeat calls to avoid overwhelming the Temporal Service, so you can call `activity.heartbeat()` as often as needed without performance concern.

### Workflow Definition determinism

In Temporal, your Activities can include non-deterministic code, but the Workflow Definition must remain deterministic. This is because Temporal restores Workflow state through *replay*. When a Worker restarts, it re-executes the Workflow code from the beginning, matching Commands against Events stored in the history. If the code produces different Commands than what the history contains, the Worker raises a `NondeterminismError`.

In practice, this means:

- Use `workflow.now()` instead of `datetime.now()`.
- Use `workflow.uuid4()` instead of `uuid.uuid4()`.
- Use `workflow.random()` instead of the `random` module.
- Do not perform I/O, network calls, or file system access inside Workflow code. Delegate all side effects to Activities.
- Use `workflow.logger` instead of `print()` for replay-safe logging.

The Python SDK's Workflow sandbox provides automatic protection against many of these violations, but understanding the underlying mechanism helps you write correct code.

### Activity idempotency

Activities may be retried by the Temporal Service due to timeouts, Worker crashes, or transient failures. Every Activity that interacts with an external system must be designed so that executing it twice produces the same result as executing it once. Common strategies include:

- Passing a unique identifier (the Workflow Id, an Activity-specific identifier, or a business identifier) as an idempotency key to external APIs.
- Checking the current state of the external system before making changes.
- Using conditional writes or upserts instead of blind inserts.

## Target audience

This guide references the following roles:

- **Game backend developers** who design and implement player session management, matchmaking, and real-time game logic. They will write the Workflow, Activity, and data model code.
- **Platform engineers** who deploy and operate Temporal Workers, and manage Temporal infrastructure.
- **Technical architects** who evaluate distributed system patterns for multiplayer game backends. They will use this guide to understand how the actor model maps to Temporal and when to apply it.

## Prerequisites

### Required software and infrastructure

- **Python 3.11 or later**
- **Temporal Python SDK (`temporalio`)** version 1.9.0 or later
- **Pydantic** version 2.0 or later for data validation
- **Temporal CLI** for running a local development server (`temporal server start-dev`)

### Required concepts

- Familiarity with Python `async`/`await` and the `asyncio` event loop
- Familiarity with Temporal Workflows, Activities, and Workers
- Familiarity with Temporal Signals, Queries, and Updates
- Basic understanding of the actor model (message-passing concurrency)
- Familiarity with `dataclasses` or Pydantic models for structured data

---

## Architecture diagram

The following sequence diagram illustrates the lifecycle of a player Actor Workflow, including interactions with game systems and Events.

### Narrative

1. The game client starts a player Actor Workflow using the player's unique identifier as the Workflow Id.
2. The Workflow initializes player state and enters a main event loop, waiting for messages.
3. When the client sends a `join_room` Signal, the Workflow executes an Activity that communicates with the game room service. This is the key distinction from an Entity Workflow: the actor *does something* by executing an Activity with a real-world side effect.
4. Combat moves trigger Activities that process game logic and update leaderboards.
5. Queries provide read-only access to the player's current state for dashboards and client polling.
6. When the Event History approaches the size limit, the Workflow waits for all handlers to finish and then executes Continue-As-New, carrying forward the player's state to a fresh execution.

```mermaid
sequenceDiagram
    participant Client as Game Client
    participant TS as Temporal Service
    participant PW as Player Actor Workflow<br/>(player-session-alice)
    participant A as Activities

    Client->>TS: Start Workflow (player-session-alice)
    TS->>PW: Execute @workflow.run

    Note over PW: Actor initializes state,<br/>enters main event loop

    Client->>TS: Signal: join_room("dungeon-7")
    TS->>PW: Deliver Signal
    PW->>A: execute_activity(join_game_room)
    A-->>PW: Room joined, room state returned
    Note over PW: Update internal state<br/>with room information

    Client->>TS: Signal: execute_move(attack, target)
    TS->>PW: Deliver Signal
    PW->>A: execute_activity(process_combat_move)
    A-->>PW: Move result (damage dealt, XP earned)
    PW->>A: execute_activity(update_leaderboard)
    A-->>PW: Leaderboard updated

    Client->>TS: Query: get_player_state
    TS->>PW: Deliver Query
    PW-->>TS: Return current state
    TS-->>Client: Player state

    Note over PW: Event History approaching limit
    PW->>PW: await all_handlers_finished
    PW->>TS: Continue-As-New(carry forward state)
    TS->>PW: New execution with same Workflow Id
```

## Implementation plan

This section walks you through building a complete player session Actor Workflow system. The implementation is organized into the following phases:

1. Define the data models
2. Define the Activities
3. Define the Player Actor Workflow
4. Configure and start the Worker
5. Start and interact with player sessions from a client

### Phase 1: Define the data models

Create a file named `models.py` to hold all data structures used by the Workflows and Activities. Using `dataclasses` keeps payloads lightweight and avoids additional dependencies, though Pydantic models work equally well if you need validation.

```python
# models.py
from __future__ import annotations

import enum
from dataclasses import dataclass, field

class PlayerStatus(str, enum.Enum):
    """Represents the current lifecycle state of a player session."""

    ONLINE = "online"
    IN_ROOM = "in_room"
    IN_COMBAT = "in_combat"
    IDLE = "idle"
    OFFLINE = "offline"

class MoveType(str, enum.Enum):
    """Types of combat moves a player can execute."""

    ATTACK = "attack"
    DEFEND = "defend"
    HEAL = "heal"
    SPECIAL = "special"

@dataclass
class InventoryItem:
    """A single item in a player's inventory."""

    item_id: str
    name: str
    quantity: int = 1

@dataclass
class CombatMoveRequest:
    """A request from a player to execute a combat move."""

    move_type: MoveType
    target_player_id: str | None = None
    event_id: str = ""  # Idempotency key; callers should always provide one

@dataclass
class CombatMoveResult:
    """The outcome of a combat move."""

    damage_dealt: int = 0
    damage_received: int = 0
    xp_earned: int = 0
    move_description: str = ""

@dataclass
class LeaderboardEntry:
    """A player's leaderboard record."""

    player_id: str
    score: int
    rank: int = 0

@dataclass
class RoomInfo:
    """Information about a game room the player has joined."""

    room_id: str
    room_name: str
    player_count: int = 0
    max_players: int = 20

@dataclass
class PlayerState:
    """The complete state of a player session.

    This dataclass is the state that the Actor Workflow carries through
    Continue-As-New cycles. It contains everything needed to reconstruct
    the player's session context.
    """

    player_id: str
    display_name: str
    status: PlayerStatus = PlayerStatus.ONLINE
    current_room: RoomInfo | None = None
    inventory: list[InventoryItem] = field(default_factory=list)
    health: int = 100
    max_health: int = 100
    xp: int = 0
    level: int = 1
    score: int = 0
    session_actions_count: int = 0
    total_actions_count: int = 0
    pending_notifications: list[str] = field(default_factory=list)
```

The `PlayerState` dataclass is the single source of truth for a player's session. It is passed as the argument to the Workflow's `@workflow.run` method and carried forward through each Continue-As-New cycle. The `session_actions_count` field tracks the number of actions in the *current* execution (reset on Continue-As-New), while `total_actions_count` tracks the lifetime total.

The separation of `session_actions_count` from `total_actions_count` gives you two useful signals: the session count helps you decide when to trigger Continue-As-New (since each action generates Events in the history), and the total count provides a lifetime metric for the player.

### Phase 2: Define the Activities

Create a file named `activities.py`. Activities contain all non-deterministic code: network calls, database writes, game server interactions, and notifications. Each Activity is designed to be idempotent so that retries do not produce duplicate effects.

```python
# activities.py
from __future__ import annotations

import uuid
from dataclasses import replace

from temporalio import activity

from models import (
    CombatMoveRequest,
    CombatMoveResult,
    InventoryItem,
    LeaderboardEntry,
    MoveType,
    RoomInfo,
)

@activity.defn
def join_game_room(player_id: str, room_id: str) -> RoomInfo:
    """Join a game room and return room information.

    In a production system, this Activity would call the game room
    microservice API to register the player in the room. The player_id
    serves as an idempotency key: joining the same room twice is a
    no-op that returns the current room state.
    """
    activity.logger.info(
        f"Player {player_id} joining room {room_id}"
    )
    # Production: call game room service API
    # room_service.join(player_id=player_id, room_id=room_id)
    return RoomInfo(
        room_id=room_id,
        room_name=f"Room {room_id}",
        player_count=5,
        max_players=20,
    )

@activity.defn
def leave_game_room(player_id: str, room_id: str) -> None:
    """Remove a player from a game room.

    Idempotent: leaving a room you are not in is a no-op.
    """
    activity.logger.info(
        f"Player {player_id} leaving room {room_id}"
    )
    # Production: call game room service API
    # room_service.leave(player_id=player_id, room_id=room_id)

@activity.defn
def process_combat_move(
    player_id: str, move: CombatMoveRequest
) -> CombatMoveResult:
    """Process a combat move and return the result.

    This Activity calls the game logic service to resolve the combat
    action. The combination of player_id and a server-assigned move
    identifier ensures idempotency.
    """
    activity.logger.info(
        f"Player {player_id} executing {move.move_type.value}"
        f" targeting {move.target_player_id}"
    )
    # Production: call game logic service
    # result = combat_service.resolve_move(player_id, move)
    damage = 25 if move.move_type == MoveType.ATTACK else 0
    healing = 15 if move.move_type == MoveType.HEAL else 0
    xp = 10

    return CombatMoveResult(
        damage_dealt=damage,
        damage_received=0,
        xp_earned=xp,
        move_description=(
            f"{move.move_type.value} executed"
            f"{' against ' + move.target_player_id if move.target_player_id else ''}"
        ),
    )

@activity.defn
def update_leaderboard(entry: LeaderboardEntry) -> LeaderboardEntry:
    """Update the leaderboard with the player's current score.

    Uses the player_id as an upsert key so that repeated calls with the
    same score do not create duplicate entries.
    """
    activity.logger.info(
        f"Updating leaderboard for {entry.player_id} "
        f"with score {entry.score}"
    )
    # Production: upsert into leaderboard database
    # db.leaderboard.upsert(player_id=entry.player_id, score=entry.score)
    return replace(entry, rank=42)

@activity.defn
def send_player_notification(
    target_player_id: str, message: str
) -> None:
    """Send a push notification to a player.

    Notifications are delivered through an external push service.
    The message includes a client-generated identifier for deduplication.
    """
    activity.logger.info(
        f"Sending notification to {target_player_id}: {message}"
    )
    # Production: call push notification service
    # push_service.send(player_id=target_player_id, message=message)

@activity.defn
def record_session_metrics(
    player_id: str,
    total_actions: int,
    session_duration_minutes: int,
) -> None:
    """Record session metrics to the analytics system.

    Called during Continue-As-New to capture session telemetry.
    Uses an append-only metrics store, so duplicate writes are harmless.
    """
    activity.logger.info(
        f"Recording session metrics for {player_id}: "
        f"{total_actions} actions, {session_duration_minutes} min"
    )
    # Production: write to metrics/analytics pipeline
    # analytics.record_session(player_id, total_actions, duration)
```

Each Activity follows a consistent pattern:

1. **Logging with `activity.logger`** provides automatic correlation with the Workflow Id and Activity attempt number.
2. **Idempotency annotations** in the docstrings explain how the Activity handles retries. In production code, each Activity passes an idempotency key to the external service it calls.
3. **Stub implementations** return realistic data structures. In a production system, each stub would be replaced by a call to the corresponding game backend service.

Activities are defined as synchronous functions. The Python SDK runs synchronous Activities in a `ThreadPoolExecutor`, which is safer and easier to debug than async Activities. Async Activities are only necessary when the Activity must use `async`-native libraries throughout its implementation.

### Phase 3: Define the player Actor Workflow

Create a file named `player_workflow.py`. This is the core of the system: a long-running Actor Workflow that represents a single player's game session.

```python
# player_workflow.py
from __future__ import annotations

import asyncio
from datetime import timedelta

from temporalio import workflow
from temporalio.common import RetryPolicy, SearchAttributeKey

with workflow.unsafe.imports_passed_through():
    from activities import (
        join_game_room,
        leave_game_room,
        process_combat_move,
        record_session_metrics,
        send_player_notification,
        update_leaderboard,
    )
    from models import (
        CombatMoveRequest,
        CombatMoveResult,
        InventoryItem,
        LeaderboardEntry,
        PlayerState,
        PlayerStatus,
    )

# Search Attribute keys for operational visibility
PLAYER_STATUS_KEY = SearchAttributeKey.for_keyword("PlayerStatus")
CURRENT_ROOM_KEY = SearchAttributeKey.for_keyword("CurrentRoom")
PLAYER_LEVEL_KEY = SearchAttributeKey.for_int("PlayerLevel")

# Threshold for triggering Continue-As-New.
# Each Signal/Update handler execution and each Activity execution
# generates Events in the history. A conservative threshold of
# 10,000 events provides ample headroom below the 51,200 limit.
CONTINUE_AS_NEW_THRESHOLD = 10_000

# Default Retry Policy for Activities in this Workflow.
DEFAULT_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=1),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(seconds=30),
    maximum_attempts=5,
)

@workflow.defn
class PlayerSessionWorkflow:
    """Actor Workflow representing a single player's game session.

    This Workflow is the player's durable, autonomous agent in the game
    backend. It maintains the player's state and actively performs
    operations by executing Activities. External systems interact with
    it through Signals (fire-and-forget commands), Updates (synchronous
    request-response), and Queries (read-only state inspection).

    Unlike an Entity Workflow that passively holds state, this Actor
    Workflow takes action: it joins game rooms, executes combat moves,
    updates leaderboards, and sends notifications.
    These side effects are all executed through Activities.

    Lifecycle:
    1. The Workflow starts when a player logs in.
    2. It enters a main loop that waits for Signals and Updates.
    3. It carries forward state through Continue-As-New cycles.
    4. It completes when the player explicitly logs out.
    """

    def __init__(self) -> None:
        # Pending action queues. Signal handlers append to these queues,
        # and the main loop processes them. This serializes action
        # processing and avoids concurrent Activity executions that
        # could conflict with each other.
        self._pending_room_joins: list[str] = []
        self._pending_moves: list[CombatMoveRequest] = []
        self._pending_notifications: list[str] = []
        self._shutdown_requested: bool = False
        self._state: PlayerState | None = None
        # In-memory set for Signal deduplication within this execution.
        # Temporal delivers Signals at least once, so the same Signal may
        # arrive more than once. This set catches duplicates that arrive
        # within the same Execution. It is not persisted across
        # Continue-As-New transitions, which is acceptable: the window
        # for a delayed duplicate to span a CAN boundary is very small.
        self._seen_move_ids: set[str] = set()

    # ──────────────────────────────────────────────
    # Signals: fire-and-forget commands from clients
    # ──────────────────────────────────────────────

    @workflow.signal
    async def join_room(self, room_id: str) -> None:
        """Request the player to join a game room.

        The Signal handler appends the room identifier to a queue. The
        main loop picks it up and executes the join_game_room Activity.
        This pattern ensures that room joins are processed serially,
        preventing race conditions where a player could be in two rooms.

        If the room is already queued or is the player's
        current room, the Signal is silently dropped. If the caller
        needs confirmation that the join succeeded, use an Update instead.
        """
        if room_id in self._pending_room_joins:
            return
        if (
            self._state is not None
            and self._state.current_room is not None
            and self._state.current_room.room_id == room_id
        ):
            return
        self._pending_room_joins.append(room_id)

    @workflow.signal
    async def execute_move(self, move: CombatMoveRequest) -> None:
        """Request the player to execute a combat move.

        Moves are queued and processed in order by the main loop.
        Callers must supply a unique event_id for deduplication. Without
        one, duplicate Signals cannot be detected and may result in
        double-processing. If at-most-once execution or a result is
        required, use an Update instead of a Signal.
        """
        if move.event_id:
            if move.event_id in self._seen_move_ids:
                return
            self._seen_move_ids.add(move.event_id)
        self._pending_moves.append(move)

    @workflow.signal
    async def receive_notification(self, message: str) -> None:
        """Receive a notification from another player or the game system.

        Notifications are stored in the player's state and can be
        retrieved through the get_player_state Query.
        """
        if self._state is not None:
            self._state.pending_notifications.append(message)

    @workflow.signal
    async def logout(self) -> None:
        """Request the player to log out and end the session."""
        self._shutdown_requested = True

    # ──────────────────────────────────────────────
    # Queries: read-only state inspection
    # ──────────────────────────────────────────────

    @workflow.query
    def get_player_state(self) -> PlayerState | None:
        """Return the complete player state.

        Queries are read-only and must not modify Workflow state.
        They execute synchronously on the Worker and return immediately.
        Game clients can poll this Query to display inventory, health,
        score, and pending notifications.
        """
        return self._state

    @workflow.query
    def get_status(self) -> str:
        """Return the player's current status as a string."""
        if self._state is None:
            return PlayerStatus.OFFLINE.value
        return self._state.status.value

    @workflow.query
    def get_inventory(self) -> list[InventoryItem]:
        """Return the player's current inventory."""
        if self._state is None:
            return []
        return self._state.inventory

    # ──────────────────────────────────────────────
    # Main Workflow logic
    # ──────────────────────────────────────────────

    @workflow.run
    async def run(self, state: PlayerState) -> str:
        """Main entry point for the Player Actor Workflow.

        This method initializes state and enters a loop that processes
        queued actions and checks for Continue-As-New conditions. The
        loop runs indefinitely until the player logs out or the
        session is terminated.

        Args:
            state: The player's state. On the first execution, this is
                the initial state. On Continue-As-New, this is the state
                carried forward from the previous execution.
        """
        self._state = state
        self._state.status = PlayerStatus.ONLINE

        # Update Search Attributes so operators can find this player
        self._upsert_search_attributes()

        workflow.logger.info(
            f"Player session started for {state.player_id} "
            f"(total actions: {state.total_actions_count})"
        )

        # Main event loop: process queued actions until shutdown
        while not self._shutdown_requested:
            # Wait for any pending action or a shutdown request.
            # The timeout ensures we periodically check the Event
            # History length even during idle periods.
            try:
                await workflow.wait_condition(
                    lambda: (
                        bool(self._pending_room_joins)
                        or bool(self._pending_moves)
                        or self._shutdown_requested
                    ),
                    timeout=timedelta(minutes=5),
                )
            except asyncio.TimeoutError:
                # No actions for 5 minutes. Check Continue-As-New
                # threshold and loop again.
                pass

            # Process all pending room joins
            await self._process_room_joins()

            # Process all pending combat moves
            await self._process_combat_moves()

            # Check whether Event History is approaching the limit
            if self._should_continue_as_new():
                await self._perform_continue_as_new()
                # continue_as_new raises an exception that exits the method
                return ""  # unreachable, but satisfies the type checker

        # Shutdown path: player logged out
        await self._handle_logout()

        return f"Player {state.player_id} session ended"

    # ──────────────────────────────────────────────
    # Private helper methods
    # ──────────────────────────────────────────────

    async def _process_room_joins(self) -> None:
        """Process all pending room join requests."""
        while self._pending_room_joins:
            room_id = self._pending_room_joins.pop(0)

            # Leave the current room first if the player is already in one
            if self._state.current_room is not None:
                await workflow.execute_activity(
                    leave_game_room,
                    args=[
                        self._state.player_id,
                        self._state.current_room.room_id,
                    ],
                    start_to_close_timeout=timedelta(seconds=30),
                    retry_policy=DEFAULT_RETRY_POLICY,
                )

            # Join the new room. This Activity calls the game room
            # service, which is a real-world side effect. This is
            # what makes this an Actor Workflow, not an Entity Workflow.
            room_info = await workflow.execute_activity(
                join_game_room,
                args=[self._state.player_id, room_id],
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=DEFAULT_RETRY_POLICY,
            )

            self._state.current_room = room_info
            self._state.status = PlayerStatus.IN_ROOM
            self._increment_action_count()
            self._upsert_search_attributes()

            workflow.logger.info(
                f"Player {self._state.player_id} joined room "
                f"{room_info.room_name}"
            )

    async def _process_combat_moves(self) -> None:
        """Process all pending combat moves in order."""
        while self._pending_moves:
            move = self._pending_moves.pop(0)
            self._state.status = PlayerStatus.IN_COMBAT

            # Execute the combat move. This Activity calls the game
            # logic service to resolve damage, XP, and other effects.
            result: CombatMoveResult = await workflow.execute_activity(
                process_combat_move,
                args=[self._state.player_id, move],
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=DEFAULT_RETRY_POLICY,
            )

            # Apply the results to the player's state
            self._state.xp += result.xp_earned
            self._state.health = max(
                0,
                min(
                    self._state.max_health,
                    self._state.health - result.damage_received,
                ),
            )
            self._state.score += result.damage_dealt + result.xp_earned

            # Check for level up (every 100 XP)
            new_level = (self._state.xp // 100) + 1
            if new_level > self._state.level:
                self._state.level = new_level
                workflow.logger.info(
                    f"Player {self._state.player_id} reached "
                    f"level {new_level}"
                )

            # Update the leaderboard. This is another real-world side
            # effect: the leaderboard is an external system that other
            # players and the game client query.
            await workflow.execute_activity(
                update_leaderboard,
                LeaderboardEntry(
                    player_id=self._state.player_id,
                    score=self._state.score,
                ),
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=DEFAULT_RETRY_POLICY,
            )

            # Notify the target player of the incoming attack.
            # This demonstrates cross-actor communication: one Actor
            # Workflow triggers an Activity that sends a message to
            # another player's session via an external push service.
            if move.target_player_id:
                await workflow.execute_activity(
                    send_player_notification,
                    args=[
                        move.target_player_id,
                        f"{self._state.display_name} attacked you "
                        f"for {result.damage_dealt} damage "
                        f"({result.move_description})",
                    ],
                    start_to_close_timeout=timedelta(seconds=10),
                    retry_policy=DEFAULT_RETRY_POLICY,
                )

            self._increment_action_count()
            self._state.status = (
                PlayerStatus.IN_ROOM
                if self._state.current_room
                else PlayerStatus.ONLINE
            )

    def _should_continue_as_new(self) -> bool:
        """Check whether the Event History is approaching the limit."""
        current_length = workflow.info().get_current_history_length()
        return current_length > CONTINUE_AS_NEW_THRESHOLD

    async def _perform_continue_as_new(self) -> None:
        """Execute Continue-As-New with the current player state.

        Before continuing, this method:
        1. Waits for all in-progress Signal and Update handlers to finish.
        2. Records session metrics to the analytics system.
        3. Resets the session-level action counter.
        """
        workflow.logger.info(
            f"Continue-As-New for player {self._state.player_id} "
            f"at {workflow.info().get_current_history_length()} events"
        )

        # Wait for all handlers to complete before continuing.
        # This prevents data loss from in-flight Signal or Update handlers.
        await workflow.wait_condition(workflow.all_handlers_finished)

        # Record session metrics before resetting the counter
        await workflow.execute_activity(
            record_session_metrics,
            args=[
                self._state.player_id,
                self._state.total_actions_count,
                0,  # duration computed from workflow start time in production
            ],
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=DEFAULT_RETRY_POLICY,
        )

        # Reset session-level counter; keep total lifetime counter
        self._state.session_actions_count = 0

        # Continue-As-New carries the full player state to the new execution
        workflow.continue_as_new(args=[self._state])

    async def _handle_logout(self) -> None:
        """Clean up resources when the player logs out."""
        assert self._state is not None

        # Wait for all handlers to finish processing
        await workflow.wait_condition(workflow.all_handlers_finished)

        # Leave the current room
        if self._state.current_room is not None:
            await workflow.execute_activity(
                leave_game_room,
                args=[
                    self._state.player_id,
                    self._state.current_room.room_id,
                ],
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=DEFAULT_RETRY_POLICY,
            )
            self._state.current_room = None

        self._state.status = PlayerStatus.OFFLINE
        self._upsert_search_attributes()

        workflow.logger.info(
            f"Player {self._state.player_id} logged out "
            f"after {self._state.total_actions_count} total actions"
        )

    def _increment_action_count(self) -> None:
        """Increment both session and total action counters."""
        self._state.session_actions_count += 1
        self._state.total_actions_count += 1

    def _upsert_search_attributes(self) -> None:
        """Update Search Attributes for operational visibility.

        Search Attributes allow operators to query running Workflows.
        For example: find all players in a specific room, find all
        players above a certain level, or find all players with a
        particular status.
        """
        updates = [
            PLAYER_STATUS_KEY.value_set(self._state.status.value),
            PLAYER_LEVEL_KEY.value_set(self._state.level),
        ]
        if self._state.current_room:
            updates.append(
                CURRENT_ROOM_KEY.value_set(self._state.current_room.room_id)
            )
        workflow.upsert_search_attributes(updates)
```

The `PlayerSessionWorkflow` is the central piece of this system. Several design decisions are worth examining in detail:

**Message queue pattern for Signals.** Signal handlers do not execute Activities directly. Instead, they append to internal queues (`_pending_room_joins`, `_pending_moves`), and the main loop drains these queues. This serializes side effects and prevents concurrent Activity executions that could conflict with each other. For example, if two `join_room` Signals arrive simultaneously, processing them through a queue guarantees that the player leaves the first room before joining the second.

**Signal deduplication.** Temporal delivers Signals at least once: the same Signal may arrive more than once under certain failure conditions. Without deduplication, a duplicate `execute_move` Signal could apply a combat move twice, corrupting XP and health state. The `join_room` handler drops a Signal if the target room is already queued or is the current room. The `execute_move` handler uses the `event_id` field on `CombatMoveRequest` as an idempotency key, tracking seen identifiers in `_seen_move_ids`. Callers must generate and provide a unique `event_id` for every move Signal. If a Signal has no `event_id`, duplicates cannot be detected and the move is appended unconditionally — this is the fallback behavior and should be avoided in production. For operations where the caller needs confirmation or strict at-most-once semantics, use a Workflow Update instead of a Signal: Updates support validators that run before the operation is accepted, and the caller blocks until the handler returns a result.

**Continue-As-New with handler draining.** Before executing `workflow.continue_as_new()`, the Workflow calls `await workflow.wait_condition(workflow.all_handlers_finished)`. This ensures that any Signal or Update handler that is currently executing finishes before the current execution ends. Without this step, in-flight handlers could be interrupted, causing data loss.

**Search Attribute updates.** The Workflow upserts Search Attributes whenever the player's status, room, or level changes. This enables operational Queries such as:

```
PlayerStatus = "in_room" AND CurrentRoom = "dungeon-7"
```

```
PlayerLevel >= 10 AND PlayerStatus = "online"
```

These Queries are invaluable for live operations: finding all players in a room that needs to be shut down for maintenance, identifying high-level players for a special event, or monitoring the distribution of player statuses across the system.

### Phase 4: Configure and start the Worker

Create a file named `worker.py`. The Worker registers the `PlayerSessionWorkflow` and all Activities on a single Task Queue.

```python
# worker.py
from __future__ import annotations

import asyncio
import concurrent.futures
import logging

from temporalio.client import Client
from temporalio.worker import Worker

from activities import (
    join_game_room,
    leave_game_room,
    process_combat_move,
    record_session_metrics,
    send_player_notification,
    update_leaderboard,
)
from player_workflow import PlayerSessionWorkflow

# Configure logging for the Worker process
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)

async def main() -> None:
    """Start a Worker that handles player session Workflows."""
    client = await Client.connect("localhost:7233", namespace="default")

    # Use a ThreadPoolExecutor for synchronous Activities.
    # Size the pool based on expected concurrent Activity executions.
    # Each concurrent Activity occupies one thread.
    with concurrent.futures.ThreadPoolExecutor(
        max_workers=100
    ) as activity_executor:
        worker = Worker(
            client,
            task_queue="player-session-queue",
            workflows=[PlayerSessionWorkflow],
            activities=[
                join_game_room,
                leave_game_room,
                process_combat_move,
                update_leaderboard,
                send_player_notification,
                record_session_metrics,
            ],
            activity_executor=activity_executor,
            # Limit concurrent Workflow Tasks. Each Workflow Task is
            # lightweight (in-memory replay), but limiting concurrency
            # prevents memory pressure under high load.
            max_concurrent_workflow_tasks=200,
            # Limit concurrent Activity executions. Activities perform
            # I/O and consume threads from the executor pool.
            max_concurrent_activities=100,
        )

        logging.info(
            "Worker started on task queue 'player-session-queue'"
        )
        await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

The Worker configuration makes several deliberate choices:

- **ThreadPoolExecutor sizing:** The `max_workers=100` setting supports up to 100 concurrent Activity executions. Size this based on the expected ratio of concurrent players to Activity execution time. If Activities are fast (under 100 milliseconds), fewer threads are needed. If Activities involve slower external service calls, increase the pool size.
- **Concurrency limits:** `max_concurrent_workflow_tasks=200` and `max_concurrent_activities=100` prevent the Worker from being overwhelmed. These values should be tuned based on the Worker's available memory and CPU.

### Phase 5: Start and interact with player sessions from a client

Create a file named `starter.py`. This module demonstrates how a game backend API would start player sessions and send commands to them.

```python
# starter.py
from __future__ import annotations

import asyncio

from temporalio.client import Client
from temporalio.common import (
    SearchAttributeKey,
    SearchAttributePair,
    TypedSearchAttributes,
)

from models import (
    CombatMoveRequest,
    InventoryItem,
    MoveType,
    PlayerState,
)
from player_workflow import PlayerSessionWorkflow

# Search Attribute keys (must match the Workflow definitions)
PLAYER_STATUS_KEY = SearchAttributeKey.for_keyword("PlayerStatus")
PLAYER_LEVEL_KEY = SearchAttributeKey.for_int("PlayerLevel")

async def start_player_session(
    client: Client, player_id: str, display_name: str
) -> None:
    """Start a new player session Actor Workflow.

    The Workflow Id is derived from the player_id, ensuring that each
    player has exactly one active session. If a session with this
    Workflow Id already exists, this call will raise an error. Use
    the Workflow Id Reuse Policy to control this behavior.
    """
    initial_state = PlayerState(
        player_id=player_id,
        display_name=display_name,
        inventory=[
            InventoryItem(
                item_id="sword-001",
                name="Iron Sword",
                quantity=1,
            ),
            InventoryItem(
                item_id="potion-001",
                name="Health Potion",
                quantity=5,
            ),
        ],
    )

    handle = await client.start_workflow(
        PlayerSessionWorkflow.run,
        initial_state,
        id=f"player-session-{player_id}",
        task_queue="player-session-queue",
        search_attributes=TypedSearchAttributes([
            SearchAttributePair(PLAYER_STATUS_KEY, "online"),
            SearchAttributePair(PLAYER_LEVEL_KEY, 1),
        ]),
    )
    print(f"Started session for {display_name} ({player_id})")
    print(f"  Workflow Id: {handle.id}")

async def join_room(
    client: Client, player_id: str, room_id: str
) -> None:
    """Send a join_room Signal to a player's Actor Workflow."""
    handle = client.get_workflow_handle(
        f"player-session-{player_id}"
    )
    await handle.signal(PlayerSessionWorkflow.join_room, room_id)
    print(f"Player {player_id} sent join_room Signal for {room_id}")

async def attack(
    client: Client,
    player_id: str,
    target_player_id: str,
) -> None:
    """Send a combat move Signal to a player's Actor Workflow."""
    handle = client.get_workflow_handle(
        f"player-session-{player_id}"
    )
    move = CombatMoveRequest(
        move_type=MoveType.ATTACK,
        target_player_id=target_player_id,
    )
    await handle.signal(PlayerSessionWorkflow.execute_move, move)
    print(
        f"Player {player_id} sent attack Signal "
        f"targeting {target_player_id}"
    )

async def check_player_state(
    client: Client, player_id: str
) -> None:
    """Query a player's current state."""
    handle = client.get_workflow_handle(
        f"player-session-{player_id}"
    )
    state = await handle.query(
        PlayerSessionWorkflow.get_player_state
    )
    if state is not None:
        print(f"Player: {state.display_name}")
        print(f"  Status: {state.status.value}")
        print(f"  Health: {state.health}/{state.max_health}")
        print(f"  Level: {state.level}")
        print(f"  Score: {state.score}")
        print(f"  XP: {state.xp}")
        print(
            f"  Room: "
            f"{state.current_room.room_name if state.current_room else 'None'}"
        )
        print(f"  Inventory: {len(state.inventory)} item(s)")
        print(f"  Total actions: {state.total_actions_count}")

async def logout_player(
    client: Client, player_id: str
) -> None:
    """Send a logout Signal to gracefully end a player session."""
    handle = client.get_workflow_handle(
        f"player-session-{player_id}"
    )
    await handle.signal(PlayerSessionWorkflow.logout)
    print(f"Player {player_id} sent logout Signal")

async def main() -> None:
    """Demonstrate a complete player session lifecycle."""
    client = await Client.connect("localhost:7233", namespace="default")

    # Start two player sessions
    await start_player_session(client, "alice", "Alice the Brave")
    await start_player_session(client, "bob", "Bob the Bold")

    # Give the Workers a moment to pick up the Workflow Tasks
    await asyncio.sleep(2)

    # Alice joins a game room
    await join_room(client, "alice", "dungeon-7")
    await asyncio.sleep(1)

    # Bob joins the same room
    await join_room(client, "bob", "dungeon-7")
    await asyncio.sleep(1)

    # Alice attacks Bob
    await attack(client, "alice", "bob")
    await asyncio.sleep(1)

    # Check Alice's state
    await check_player_state(client, "alice")

    # Logout both players
    await logout_player(client, "alice")
    await logout_player(client, "bob")

if __name__ == "__main__":
    asyncio.run(main())
```

The client code demonstrates the two interaction patterns available to external callers:

- **Signals** for fire-and-forget commands: `join_room`, `execute_move`, `logout`. The caller does not wait for the Workflow to process the Signal.
- **Queries** for read-only state inspection: `get_player_state`. The caller reads the Workflow's current state without affecting its execution.

### (Optional) Phase 6: Configure Search Attributes

Before starting the Worker, register the custom Search Attributes with the Temporal Service. When using the Temporal CLI development server, run the following commands:

```bash
temporal operator search-attribute create --name PlayerStatus --type Keyword
temporal operator search-attribute create --name CurrentRoom --type Keyword
temporal operator search-attribute create --name PlayerLevel --type Int
```

Once registered, you can query running player sessions. For example, to find all players currently in room `dungeon-7`:

```bash
temporal workflow list --query 'PlayerStatus = "in_room" AND CurrentRoom = "dungeon-7"'
```

To find all players at level 10 or higher:

```bash
temporal workflow list --query 'PlayerLevel >= 10'
```

### (Optional) Phase 7: Add Heartbeating to long-running Activities

If your game includes Activities that run for extended periods, such as maintaining a persistent connection to a game room server or processing a large batch of analytics events, add Heartbeating to track progress and enable resumption after failures.

```python
# activities.py (additional long-running Activity example)

@activity.defn
def process_batch_leaderboard_update(
    entries: list[LeaderboardEntry],
) -> int:
    """Process a large batch of leaderboard updates with heartbeating.

    This Activity processes entries one at a time and heartbeats after
    each entry with the index of the last processed entry. If the Worker
    fails mid-batch, the next attempt reads the heartbeat payload and
    resumes from where processing stopped.

    Configure this Activity with a heartbeat_timeout of 30 seconds.
    The SDK throttles heartbeat calls, so calling activity.heartbeat()
    on every iteration is safe and does not overload the Temporal Service.
    """
    heartbeat_details = activity.info().heartbeat_details
    start_index = heartbeat_details[0] if heartbeat_details else 0

    processed_count = start_index
    for i in range(start_index, len(entries)):
        entry = entries[i]
        # Production: upsert into leaderboard database
        activity.logger.info(
            f"Processing leaderboard entry {i + 1}/{len(entries)} "
            f"for player {entry.player_id}"
        )
        processed_count += 1

        # Heartbeat with progress. The heartbeat payload (the current
        # index) is available to the next attempt if this attempt fails.
        activity.heartbeat(processed_count)

    return processed_count
```

When calling this Activity from a Workflow, configure the `heartbeat_timeout` parameter:

```python
# In the Workflow
processed = await workflow.execute_activity(
    process_batch_leaderboard_update,
    entries,
    start_to_close_timeout=timedelta(minutes=30),
    heartbeat_timeout=timedelta(seconds=30),
    retry_policy=DEFAULT_RETRY_POLICY,
)
```

The `heartbeat_timeout` of 30 seconds means that if the Activity does not emit a Heartbeat for 30 seconds, the Temporal Service considers it stalled and schedules a retry. The new attempt reads the last heartbeat payload (the index of the last processed entry) and resumes from that point.

---

## Outcomes

You've built a session system where player state is durable, game actions are reliable, and server failures are non-events. The player Workflow doesn't just hold state — it drives action. That distinction matters: when a player joins a room or executes a combat move, the Workflow owns the operation end-to-end, retrying if needed and preserving consistency if something fails partway through.

The approach isn't limited to games. Any long-lived entity that has to take action on its own behalf — an IoT device scaling infrastructure in response to sensor data, an e-commerce order submitting payment and triggering fulfillment, an AI agent calling tools and external APIs — fits the same model. The player session implementation here gives you the complete pattern.

---

## Related resources

- [Entity Workflow Pattern](/design-patterns/entity-workflow) — pattern catalog reference for the underlying Entity Workflow pattern this Actor Workflow extends
- [Continue-As-New Pattern](/design-patterns/continue-as-new) — pattern catalog reference covering common pitfalls and history-size triggers
- [Temporal Python SDK documentation](/develop/python)
- [Message Passing — Signals, Queries, Updates](/develop/python/workflows/message-passing)
- [Continue-As-New](/develop/python/workflows/continue-as-new)
- [Child Workflows](/develop/python/workflows/child-workflows)
- [Failure Detection — Timeouts, Activity Heartbeating, and Retry Policies](/develop/python/best-practices/error-handling)
- [Search Attributes](/search-attribute)
- [Temporal Python SDK API Reference](https://python.temporal.io)
- [Temporal Python SDK samples](https://github.com/temporalio/samples-python)
- Companion article: [Entity Workflow Pattern](/guides/entity-pattern-loyalty-points)

---

# Build a customer loyalty program with Entity Workflows

Source: https://docs.temporal.io/guides/entity-pattern-loyalty-points

> Give each customer a long-running Workflow that accrues points, recalculates tiers, and handles redemptions, using Continue-As-New and Worker Versioning.

**by Cecil Phillip**

Loyalty programs look simple until you're running them at scale: points have to accrue correctly across every purchase, tiers have to recalculate at the right moment, redemptions have to be consistent, and all of it has to hold up across years of customer interactions. This guide shows how to give each customer their own dedicated, durable account — one that holds state, enforces business rules, responds to purchases and redemptions in real time, and never needs a cron job to stay current.

## Problem statement

Every customer in a loyalty program is their own entity. Their points accumulate across purchases, their tier changes when they hit thresholds, and their redemptions need to be validated and applied without double-spending. Doing this with a shared database means locking, retry logic across services, and cron jobs that periodically recalculate who qualifies for what. When a points accrual succeeds in one service but the tier upgrade notification fails in another, you have an inconsistency that's difficult to detect and expensive to fix. The longer the customer relationship, the more chances there are for these gaps to accumulate.

## Solution

Each customer gets their own Workflow — a persistent, running process identified by their customer ID. It holds the account's points balance, tier status, and activity history in memory. When a purchase happens, a Signal adds the points. When a customer redeems at checkout, an Update validates and applies the redemption synchronously. When a mobile app needs to display the current balance, a Query reads the state without touching the database. Tier upgrades happen as part of normal business logic, not from a nightly job. The Workflow runs for as long as the customer relationship does — years if needed — and resets its internal history automatically to stay within platform limits.

## Outcomes

After working through this guide, you'll have a loyalty system where:

- **Each customer account is independent and durable**: state is held per-customer, survives infrastructure failures, and picks up exactly where it left off
- **Tier upgrades happen in real time**: business rules run in the Workflow itself, triggered by point accruals — no scheduler, no batch job
- **Redemptions are consistent**: synchronous Updates validate and apply changes atomically, with the response going back to the caller
- **In-memory Balance reads**: Queries read in-memory state without generating Events or hitting a database
- **The Workflow runs for the life of the customer**: Continue-As-New keeps history size in check so nothing needs to be migrated or restarted
- **Failures are isolated**: a notification failing doesn't roll back a point accrual; each concern fails independently

## Background and best practices

### What is an Entity Workflow?

Although "Workflow" is a Temporal SDK primitive, when we talk about an "entity workflow," we're describing an architectural pattern. An entity workflow represents something that persists over time — a customer, a device, an order, a bank account. Contrast this with a "process workflow," which has a definite end. The key distinction is that an entity is a thing, whereas a process workflow does a thing. An entity workflow has an indefinite lifetime and reacts to events as they arrive. In the loyalty context, the customer account is the entity: it exists as long as the customer relationship does, and it responds to purchases, redemptions, and status queries throughout that lifetime.

Entity Workflows share three defining characteristics:

- They run for an **indefinite duration**. You do not know at start time when or if they will complete.
- They **react to external messages** at any point in their lifecycle via Signals, Updates, and Queries.
- They **maintain mutable state** that evolves over time in response to those messages.

In the loyalty points use case, each customer has their own Entity Workflow. The Workflow holds the points balance and tier status. External services send Signals to add points after a purchase, send Updates to redeem points at checkout, and send Queries to display the current balance in a mobile app. The Workflow runs for as long as the customer account exists.

### How Entity Workflows differ from actors

The [Entity Workflow pattern](/design-patterns/entity-workflow) shares characteristics with the Actor Model: both receive messages, maintain state, and can create new instances. However, there is an important distinction. An entity **represents state** and responds to interactions. It is closer to a data cache than a process orchestrator. It holds and serves state, evaluates business rules against that state, and delegates side effects to Activities. An actor, by contrast, **takes actions**. It may write to databases, call external APIs, or send emails as part of its core behavior.

In this pattern, the loyalty Entity Workflow does not directly write to a database or send an email. When it needs a side effect, such as sending a tier upgrade notification, it delegates that work to an Activity. The Workflow itself remains a pure state container with business rules.

### Why Temporal is well suited for this pattern

Temporal provides several capabilities that make Entity Workflows practical:

- **Durable Timers**: A Workflow can sleep for hours, days, or months without consuming compute resources. The Timer is persisted by the Temporal Server; the Worker is freed to handle other tasks.
- **Message passing**: Signals, Queries, and Updates provide three distinct interaction patterns — fire-and-forget, read-only, and request-response — covering the full range of entity interactions.
- **Event History**: Every state transition is recorded in an append-only log, giving a complete audit trail for compliance and debugging.
- **Continue-As-New**: When the Event History approaches size limits, the Workflow resets itself with a fresh history while preserving current state, allowing Entity Workflows to run for years.
- **Deterministic replay**: If a Worker crashes, the Workflow resumes from its last known state by replaying the Event History — no state is lost and no external coordination is needed.

### Operational considerations

Before implementing this pattern, keep the following constraints in mind:

**Event History limits.** Temporal enforces a hard limit of 51,200 Events and 50 MB per Workflow Execution, with warnings starting around 10,000 Events. Each Signal generates approximately 4 Events; each Activity execution approximately 11. A loyalty entity processing 100 point accruals per day (each triggering one Activity) generates roughly 1,500 Events per day — plan your Continue-As-New threshold accordingly.

**Payload size limits.** Individual payloads for Workflow and Activity inputs and outputs are limited to 2 MB. If the serialized entity state approaches this limit, store large data externally and reference it by identifier.

**Workflow Definition determinism.** Activities can include non-deterministic code, but the Workflow Definition must remain deterministic. Use `workflow.now()` instead of `datetime.now()`, `workflow.random()` instead of `random`, and `workflow.uuid4()` instead of `uuid.uuid4()`. All I/O must be performed in Activities.

**Signal delivery guarantees.** Temporal delivers Signals at least once. Include a deduplication key in Signal payloads and make your Signal processing logic idempotent.

## Target audience

The following roles are involved in this pattern:

- **Temporal Workflow and Activity developers**: Responsible for implementing the loyalty Entity Workflow, defining Activities, and writing client code to interact with the entity
- **Platform operators**: Responsible for deploying and managing Temporal Workers, configuring Task Queues, and monitoring Event History growth
- **Domain architects**: Responsible for deciding which loyalty business rules belong in the Workflow versus in Activities, and defining the message contracts for Signals, Updates, and Queries

This guide requires Python, a Temporal Worker, and either a local development server or Temporal Cloud.

## Prerequisites

### Required software and infrastructure

- Python 3.9 or later
- Temporal Python SDK (`temporalio`) version 1.7.0 or later (required for `@workflow.init`, `workflow.all_handlers_finished`, and Workflow Updates)
- A running Temporal Server, either via the [Temporal CLI](/cli) for local development or [Temporal Cloud](https://temporal.io/cloud) for production
- `pip` or `uv` for Python dependency management

### Resources and access privileges

- Temporal Cloud account with Namespace Admin role, or local development server access via `temporal server start-dev`
- Write access to your Worker deployment environment

### Required concepts

You should be familiar with the following before proceeding:

- [Temporal Workflows](/workflows): The durable, stateful functions that Temporal orchestrates
- [Temporal Activities](/activities): The functions that perform side effects such as API calls and database writes
- [Temporal Workers](/workers): The processes that host and execute Workflow and Activity code
- [Temporal Task Queues](/task-queue): The named queues that route work from the Temporal Server to Workers

## Architecture diagrams

The following diagrams illustrate how external services interact with a loyalty Entity Workflow through Signals, Updates, and Queries.

### Signal and Continue-As-New flow

This diagram shows how a client adds points to a customer's loyalty account via a Signal, and how the Workflow performs Continue-As-New when the Event History grows large.

**Narrative:**

1. An external service starts a loyalty Entity Workflow with a stable Workflow Id tied to the customer identifier, such as `loyalty-cust-123`.
2. The Temporal Server persists the start request and schedules a Workflow Task on the `loyalty-points-queue` Task Queue.
3. A Python Worker picks up the Task, initializes the loyalty state, and enters a wait loop.
4. Throughout the customer's lifetime, external services send Signals to add points, Updates to redeem points, and Queries to read the current balance and tier.
5. When the Event History approaches the suggested threshold, the Workflow serializes its current state and calls `continue_as_new`, starting a fresh Execution with the same Workflow Id.
6. When the customer account is closed, a shutdown Signal causes the Workflow to drain its queue, await all handlers, and complete.

```mermaid
sequenceDiagram
    participant Client as External Service
    participant TS as Temporal Server
    participant W as Worker (Python)

    Client->>TS: start_workflow(LoyaltyEntity,<br/>id="loyalty-cust-123")
    TS->>W: Schedule Workflow Task
    W->>W: Initialize loyalty state (0 points, Bronze tier)
    W-->>TS: Blocked on wait_condition (waiting for messages)

    Client->>TS: signal("loyalty-cust-123",<br/>add_points, {points: 500})
    TS->>W: Deliver Signal
    W->>W: Append to internal queue
    W->>W: Process: update points balance
    W->>TS: Schedule Activity (record_points_transaction)
    TS->>W: Execute Activity
    W-->>TS: Activity complete, resume Workflow

    Client->>TS: query("loyalty-cust-123", get_loyalty_summary)
    TS->>W: Deliver Query
    W-->>Client: Return {points: 500, tier: "Bronze"}<br/>(no Event recorded)

    Note over W: Event History approaching threshold
    W->>W: Drain pending signals
    W->>W: await all_handlers_finished()
    W->>TS: continue_as_new(current_state)
    Note over TS: Same Workflow Id,<br/>new Run Id, fresh Event History
```

### Update flow for point redemption

This diagram shows a synchronous Update interaction where a client redeems points and receives confirmation of the new balance.

```mermaid
sequenceDiagram
    participant Client as External Service
    participant TS as Temporal Server
    participant W as Worker (Python)

    Client->>TS: execute_update("loyalty-cust-123",<br/>redeem_points, {points: 200})
    TS->>W: Deliver Update
    W->>W: Run validator (check points > 0)
    W->>W: Check sufficient balance
    W->>W: Deduct points from state
    W->>TS: Schedule Activity (record_redemption)
    TS->>W: Execute Activity
    W-->>Client: Return {redeemed: 200, remaining: 300}
```

## Implementation plan

This section walks you through building a complete customer loyalty Entity Workflow in Python. You will define the data model, implement the Workflow with Signal, Query, and Update handlers, write Activities for side effects, configure Continue-As-New, set up the Worker, and write client code to interact with the entity.

### Phase 1: Define the loyalty state and data models

Entity Workflows carry their current state as their primary input, so Continue-As-New can resume with that state intact. Use Python `dataclass` objects — they serialize cleanly with Temporal's default JSON converter and allow new fields with defaults without breaking existing Executions.

Create a file named `models.py` with the following content:

```python
# models.py
from dataclasses import dataclass, field
from enum import Enum

class LoyaltyTier(str, Enum):
    """Loyalty tiers based on lifetime points earned."""
    BRONZE = "Bronze"
    SILVER = "Silver"
    GOLD = "Gold"
    PLATINUM = "Platinum"

@dataclass
class PointsEvent:
    """A single points transaction to be processed by the entity."""
    event_id: str          # Unique identifier for idempotent processing
    points: int            # Positive for accrual, negative for redemption
    reason: str            # Human-readable description (e.g., "purchase", "signup_bonus")
    source: str = ""       # Originating service or channel

@dataclass
class LoyaltyState:
    """The complete state of a customer's loyalty account.

    This is the data that survives Continue-As-New transitions.
    Keep it small and serializable — well under the 2 MB payload limit.
    """
    customer_id: str
    points_balance: int = 0
    lifetime_points: int = 0
    tier: str = LoyaltyTier.BRONZE.value
    is_active: bool = True
    processed_event_ids: list[str] = field(default_factory=list)

    # Unprocessed events carried across Continue-As-New transitions
    pending_events: list[dict] = field(default_factory=list)

@dataclass
class LoyaltySummary:
    """Read-only view of the loyalty account returned by Queries."""
    customer_id: str
    points_balance: int
    lifetime_points: int
    tier: str
    is_active: bool

@dataclass
class RedemptionRequest:
    """Input for the redeem_points Update."""
    event_id: str       # Idempotency key for the redemption
    points: int         # Number of points to redeem (must be positive)
    reward: str         # What the points are being redeemed for

@dataclass
class RedemptionResult:
    """Output from the redeem_points Update."""
    redeemed: int
    remaining_balance: int
    reward: str
```

`LoyaltyState` holds only what's needed to reconstruct the entity after a Continue-As-New transition. `processed_event_ids` enables idempotent Signal processing — in production, limit this to a recent window (for example, last 1,000 IDs) and rely on an external store for long-term deduplication. `pending_events` carries any Signals that arrived but weren't processed before the transition.

### Phase 2: Implement the Activities

Activities hold all non-deterministic code: database writes, API calls, and notifications. The Workflow stays a pure state container.

Create a file named `activities.py`:

```python
# activities.py
import logging

from temporalio import activity

logger = logging.getLogger(__name__)

@activity.defn
async def record_points_transaction(transaction: dict) -> bool:
    """Record a points transaction in the loyalty database.

    Must be idempotent: use event_id as the key to prevent duplicate records
    on retry. Example: INSERT ... ON CONFLICT (event_id) DO NOTHING.
    """
    activity.logger.info(
        "Recording points transaction for customer %s: %s points (%s)",
        transaction["customer_id"],
        transaction["points"],
        transaction["reason"],
    )
    # await db.execute("INSERT INTO points_transactions ... ON CONFLICT (event_id) DO NOTHING", ...)
    return True

@activity.defn
async def send_tier_change_notification(notification: dict) -> bool:
    """Send a notification when a customer's loyalty tier changes.

    Non-critical: if this Activity fails after all retries, the Workflow
    logs the failure and continues rather than terminating the account.
    """
    activity.logger.info(
        "Sending tier change notification to customer %s: %s -> %s",
        notification["customer_id"],
        notification["old_tier"],
        notification["new_tier"],
    )
    # await email_service.send(template="tier_change", to=..., context={...})
    return True

@activity.defn
async def record_redemption(redemption: dict) -> bool:
    """Record a points redemption in the loyalty database.

    Must be idempotent: use event_id as the key to prevent duplicate records
    on retry. Example: INSERT ... ON CONFLICT (event_id) DO NOTHING.
    """
    activity.logger.info(
        "Recording redemption for customer %s: %s points for %s",
        redemption["customer_id"],
        redemption["points"],
        redemption["reward"],
    )
    # await db.execute("INSERT INTO redemptions ... ON CONFLICT (event_id) DO NOTHING", ...)
    return True
```

Each Activity uses `@activity.defn` and `activity.logger` for context-aware logging. All three are idempotent by design: calling them twice with the same `event_id` produces the same result, which is essential because Temporal may retry Activities on failure.

### Phase 3: Implement the Entity Workflow

The Entity Workflow is the core of the pattern: it initializes state, waits for messages, processes them sequentially, and performs Continue-As-New when history grows large.

Create a file named `workflows.py`:

```python
# workflows.py
import asyncio
from collections import deque
from dataclasses import asdict
from datetime import timedelta

from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ApplicationError

with workflow.unsafe.imports_passed_through():
    from activities import (
        record_points_transaction,
        record_redemption,
        send_tier_change_notification,
    )
    from models import (
        LoyaltyState,
        LoyaltySummary,
        LoyaltyTier,
        PointsEvent,
        RedemptionRequest,
        RedemptionResult,
    )

# Retry policy for database writes: retry transient failures with backoff
DB_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=1),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(seconds=30),
    maximum_attempts=10,
)

# Retry policy for notifications: more aggressive retry since these are non-critical
NOTIFICATION_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=2),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=1),
    maximum_attempts=5,
)

@workflow.defn
class LoyaltyEntityWorkflow:
    """Manages a single customer's loyalty account as a long-lived Entity Workflow.

    Each customer has their own instance identified by a stable Workflow Id
    (e.g. "loyalty-cust-123"). All side effects are delegated to Activities.
    """

    @workflow.init
    def __init__(self, state: LoyaltyState | None = None) -> None:
        """Initialize Workflow state.

        The @workflow.init decorator ensures this runs before any Signal
        or Update handler, preventing race conditions with Signal-with-Start.

        Args:
            state: Existing state from Continue-As-New, or None for a new account.
        """
        if state is not None:
            self._state = state
        else:
            # First start only (not Continue-As-New); customer_id set in run().
            self._state = LoyaltyState(customer_id="")

        # Signal queue: handlers append, main loop pops one at a time.
        self._pending_signals: deque[dict] = deque(self._state.pending_events)
        self._state.pending_events = []  # Clear after restoring

        self._shutdown_requested: bool = False

    @workflow.run
    async def run(self, state: LoyaltyState | None = None) -> LoyaltyState:
        """Main entity loop: wait for messages, process them, and Continue-As-New when needed."""
        if state is None:
            # First start: derive customer_id from the Workflow Id convention.
            self._state.customer_id = workflow.info().workflow_id.replace("loyalty-", "", 1)

        workflow.logger.info(
            "Loyalty entity started for customer %s (tier: %s, balance: %d)",
            self._state.customer_id,
            self._state.tier,
            self._state.points_balance,
        )

        while not self._shutdown_requested:
            # Block until a Signal arrives, shutdown is requested,
            # CAN is suggested, or 24 hours pass.
            await workflow.wait_condition(
                lambda: (
                    bool(self._pending_signals)
                    or self._shutdown_requested
                    or workflow.info().is_continue_as_new_suggested()
                ),
                timeout=timedelta(hours=24),
            )

            # Process all queued Signals sequentially
            while self._pending_signals:
                event_data = self._pending_signals.popleft()
                await self._process_points_event(event_data)

                # Check if we should Continue-As-New after each event
                if workflow.info().is_continue_as_new_suggested():
                    break

            # Exit if shutdown was requested
            if self._shutdown_requested:
                break

            # Trigger Continue-As-New when Temporal suggests it
            if workflow.info().is_continue_as_new_suggested():
                await self._do_continue_as_new()

        # Wait for all in-flight handlers before completing.
        await workflow.wait_condition(workflow.all_handlers_finished)

        workflow.logger.info(
            "Loyalty entity shutting down for customer %s (final balance: %d)",
            self._state.customer_id,
            self._state.points_balance,
        )
        return self._state

    @workflow.signal
    def add_points(self, event: dict) -> None:
        """Enqueue a points accrual event for processing in the main loop.

        Signal handlers must be synchronous — they only append to the queue.
        All processing happens in the main loop to ensure sequential execution.

        Args:
            event: dict with keys event_id, points, reason, and optionally source.
        """
        self._pending_signals.append(event)

    @workflow.signal
    def shutdown(self) -> None:
        """Signal the entity to drain pending events and terminate gracefully."""
        self._shutdown_requested = True

    # ------------------------------------------------------------------
    # Query handlers
    # ------------------------------------------------------------------

    @workflow.query
    def get_loyalty_summary(self) -> LoyaltySummary:
        """Return a read-only summary (no Events generated, safe at high frequency)."""
        return LoyaltySummary(
            customer_id=self._state.customer_id,
            points_balance=self._state.points_balance,
            lifetime_points=self._state.lifetime_points,
            tier=self._state.tier,
            is_active=self._state.is_active,
        )

    @workflow.query
    def get_points_balance(self) -> int:
        """Return the current points balance."""
        return self._state.points_balance

    @workflow.query
    def get_tier(self) -> str:
        """Return the current loyalty tier."""
        return self._state.tier

    @workflow.update
    async def redeem_points(self, request: RedemptionRequest) -> RedemptionResult:
        """Redeem loyalty points for a reward.

        Blocks until the Update completes and the caller receives the result,
        making it ideal for checkout flows that need confirmed point deduction.

        Raises:
            ApplicationError: If the account is inactive or has insufficient points.
        """
        if not self._state.is_active:
            raise ApplicationError(
                "Cannot redeem points: account is inactive",
                type="InactiveAccount",
                non_retryable=True,
            )

        if request.points > self._state.points_balance:
            raise ApplicationError(
                f"Insufficient points: requested {request.points}, "
                f"available {self._state.points_balance}",
                type="InsufficientPoints",
                non_retryable=True,
            )

        # Idempotency: return without re-processing if already applied.
        if request.event_id in self._state.processed_event_ids:
            return RedemptionResult(
                redeemed=request.points,
                remaining_balance=self._state.points_balance,
                reward=request.reward,
            )

        self._state.points_balance -= request.points

        # Record the redemption via an Activity (idempotent on event_id).
        await workflow.execute_activity(
            record_redemption,
            {
                "customer_id": self._state.customer_id,
                "event_id": request.event_id,
                "points": request.points,
                "reward": request.reward,
            },
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=DB_RETRY_POLICY,
        )

        # Track this event as processed.
        self._state.processed_event_ids.append(request.event_id)
        self._trim_processed_ids()

        workflow.logger.info(
            "Customer %s redeemed %d points for %s (remaining: %d)",
            self._state.customer_id,
            request.points,
            request.reward,
            self._state.points_balance,
        )

        return RedemptionResult(
            redeemed=request.points,
            remaining_balance=self._state.points_balance,
            reward=request.reward,
        )

    @redeem_points.validator
    def validate_redemption(self, request: RedemptionRequest) -> None:
        """Run before the Update is accepted into Event History. Rejection here writes no Event."""
        if request.points <= 0:
            raise ApplicationError(
                "Redemption amount must be a positive number",
                type="ValidationError",
            )

    async def _process_points_event(self, event_data: dict) -> None:
        """Process a single points event: check for duplicates, record the
        transaction via an Activity, update in-memory state, and evaluate tier.
        """
        event_id = event_data.get("event_id", "")
        points = event_data.get("points", 0)
        reason = event_data.get("reason", "unknown")

        if event_id and event_id in self._state.processed_event_ids:
            workflow.logger.info(
                "Skipping duplicate event %s for customer %s",
                event_id,
                self._state.customer_id,
            )
            return

        # Record the transaction BEFORE updating in-memory state to avoid
        # double-counting if the Activity retries.
        await workflow.execute_activity(
            record_points_transaction,
            {
                "customer_id": self._state.customer_id,
                "event_id": event_id,
                "points": points,
                "reason": reason,
            },
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=DB_RETRY_POLICY,
        )

        # Update state after Activity confirms success.
        self._state.points_balance += points
        if points > 0:
            self._state.lifetime_points += points

        if event_id:
            self._state.processed_event_ids.append(event_id)
            self._trim_processed_ids()

        workflow.logger.info(
            "Processed event %s for customer %s: %+d points (balance: %d)",
            event_id,
            self._state.customer_id,
            points,
            self._state.points_balance,
        )

        await self._evaluate_tier()

    async def _evaluate_tier(self) -> None:
        """Recalculate tier; send notification if it changed (non-critical)."""
        new_tier = self._calculate_tier(self._state.lifetime_points)

        if new_tier != self._state.tier:
            old_tier = self._state.tier
            self._state.tier = new_tier

            workflow.logger.info(
                "Customer %s tier changed: %s -> %s",
                self._state.customer_id,
                old_tier,
                new_tier,
            )

            # Notification is non-critical — log and continue on failure.
            try:
                await workflow.execute_activity(
                    send_tier_change_notification,
                    {
                        "customer_id": self._state.customer_id,
                        "old_tier": old_tier,
                        "new_tier": new_tier,
                    },
                    start_to_close_timeout=timedelta(seconds=30),
                    retry_policy=NOTIFICATION_RETRY_POLICY,
                )
            except Exception:
                workflow.logger.warning(
                    "Failed to send tier change notification for customer %s; "
                    "continuing entity lifecycle",
                    self._state.customer_id,
                )

    def _calculate_tier(self, lifetime_points: int) -> str:
        """Determine tier from lifetime points — pure logic, no side effects."""
        if lifetime_points >= 10_000:
            return LoyaltyTier.PLATINUM.value
        if lifetime_points >= 5_000:
            return LoyaltyTier.GOLD.value
        if lifetime_points >= 1_000:
            return LoyaltyTier.SILVER.value
        return LoyaltyTier.BRONZE.value

    def _trim_processed_ids(self) -> None:
        """Keep only the most recent 1,000 processed event IDs to prevent unbounded growth."""
        max_ids = 1_000
        if len(self._state.processed_event_ids) > max_ids:
            self._state.processed_event_ids = self._state.processed_event_ids[
                -max_ids:
            ]

    async def _do_continue_as_new(self) -> None:
        """Perform Continue-As-New: drain excess pending signals, serialize
        remaining unprocessed signals into state, wait for all handlers to
        finish, then call continue_as_new with the current state.
        """
        # Drain excess pending signals before serializing into the CAN input.
        # Keeps the LoyaltyState payload well under the 2 MB limit.
        MAX_PENDING_CARRY = 500
        while len(self._pending_signals) > MAX_PENDING_CARRY:
            event_data = self._pending_signals.popleft()
            await self._process_points_event(event_data)

        # Carry remaining unprocessed Signals to the next Execution.
        self._state.pending_events = list(self._pending_signals)

        # Wait for async handlers to finish before transitioning.
        await workflow.wait_condition(workflow.all_handlers_finished)

        workflow.logger.info(
            "Continuing as new for customer %s (balance: %d, pending: %d)",
            self._state.customer_id,
            self._state.points_balance,
            len(self._state.pending_events),
        )

        workflow.continue_as_new(self._state)
```

Two design decisions in this Workflow deserve extra attention:

**Activity call inside an Update handler.** The `redeem_points` Update calls `record_redemption` directly via `await` rather than enqueuing it for the main loop. The Python and TypeScript SDKs support `async` Update handlers, making this safe. The trade-off keeps redemption logic self-contained and returns the confirmed result to the caller in a single round trip. It is safe here because `record_redemption` is idempotent — a replay detects the duplicate `event_id` and returns without creating a second record. If you are using the Java SDK, Update handlers cannot be `async`; enqueue the work for the main loop instead.

**Continue-As-New payload size.** Unprocessed Signals are serialized into the CAN input, which must stay under the 2 MB payload limit. The `MAX_PENDING_CARRY` guard drains the queue to a safe size before serializing. Lower the limit if your Signal payloads are large. If the guard fires frequently, Signals are arriving faster than Workers can process them — scale your Worker pool.

### Phase 4: Configure and run the Worker

The Worker hosts your Workflow and Activity code, polls a Task Queue, and executes work.

Create a file named `worker.py`:

```python
# worker.py
import asyncio
import concurrent.futures
import logging

from temporalio.client import Client
from temporalio.worker import Worker

from activities import (
    record_points_transaction,
    record_redemption,
    send_tier_change_notification,
)
from workflows import LoyaltyEntityWorkflow

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

TASK_QUEUE = "loyalty-points-queue"
TEMPORAL_ADDRESS = "localhost:7233"
TEMPORAL_NAMESPACE = "default"

async def main() -> None:
    """Start a Worker that hosts the LoyaltyEntityWorkflow and its Activities."""
    client = await Client.connect(
        TEMPORAL_ADDRESS,
        namespace=TEMPORAL_NAMESPACE,
    )

    logger.info("Starting loyalty Worker on task queue: %s", TASK_QUEUE)

    with concurrent.futures.ThreadPoolExecutor(max_workers=50) as activity_executor:
        worker = Worker(
            client,
            task_queue=TASK_QUEUE,
            workflows=[LoyaltyEntityWorkflow],
            activities=[
                record_points_transaction,
                record_redemption,
                send_tier_change_notification,
            ],
            activity_executor=activity_executor,
            max_concurrent_workflow_tasks=200,
            max_concurrent_activities=100,
        )
        await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

The Worker registers the `LoyaltyEntityWorkflow` class and all three Activity functions on the `loyalty-points-queue` Task Queue. Setting `max_concurrent_workflow_tasks` higher than the default makes sense here because Entity Workflows spend most of their time blocked on `wait_condition`, so one Worker can host many concurrent instances.

To start the Worker, run:

```
python worker.py
```

The Worker connects to the Temporal Server and begins polling. It will continue running until you stop the process.

### Phase 5: Write client code to interact with the entity

The client code starts Entity Workflows, sends Signals, executes Updates, and runs Queries. In production this typically lives in an API server or background service.

Create a file named `starter.py`:

```python
# starter.py
import asyncio

from temporalio.client import Client, WorkflowUpdateFailedError

from models import RedemptionRequest
from workflows import LoyaltyEntityWorkflow

TASK_QUEUE = "loyalty-points-queue"
TEMPORAL_ADDRESS = "localhost:7233"
TEMPORAL_NAMESPACE = "default"

async def main() -> None:
    client = await Client.connect(
        TEMPORAL_ADDRESS,
        namespace=TEMPORAL_NAMESPACE,
    )

    customer_id = "cust-42"
    workflow_id = f"loyalty-{customer_id}"

    # Start the entity — Workflow Id is stable per customer.
    # If it's already running, start_workflow raises WorkflowAlreadyStartedError.
    handle = await client.start_workflow(
        LoyaltyEntityWorkflow.run,
        id=workflow_id,
        task_queue=TASK_QUEUE,
        # No initial state — the Workflow derives customer_id from the Workflow Id.
    )
    print(f"Started loyalty entity: {workflow_id}")

    # Signals are fire-and-forget: the call returns once the Server accepts them.
    await handle.signal(
        LoyaltyEntityWorkflow.add_points,
        {
            "event_id": "evt-001",
            "points": 500,
            "reason": "initial_purchase",
            "source": "ecommerce",
        },
    )
    print("Sent Signal: +500 points (initial_purchase)")

    await handle.signal(
        LoyaltyEntityWorkflow.add_points,
        {
            "event_id": "evt-002",
            "points": 750,
            "reason": "referral_bonus",
            "source": "referral_service",
        },
    )
    print("Sent Signal: +750 points (referral_bonus)")

    # Allow time for the Worker to process the Signals
    await asyncio.sleep(2)

    # Queries are synchronous and read-only — no Events generated.
    summary = await handle.query(LoyaltyEntityWorkflow.get_loyalty_summary)
    print(
        f"Query result: {summary.points_balance} points, "
        f"tier={summary.tier}, lifetime={summary.lifetime_points}"
    )

    # Updates block until the Workflow processes them and returns a result —
    # ideal for checkout where the caller needs confirmation.
    try:
        result = await handle.execute_update(
            LoyaltyEntityWorkflow.redeem_points,
            RedemptionRequest(
                event_id="redeem-001",
                points=200,
                reward="free_shipping",
            ),
        )
        print(
            f"Redemption successful: {result.redeemed} points for "
            f"{result.reward}, remaining: {result.remaining_balance}"
        )
    except WorkflowUpdateFailedError as e:
        print(f"Redemption failed: {e}")

    # Query again to confirm the updated balance.
    balance = await handle.query(LoyaltyEntityWorkflow.get_points_balance)
    tier = await handle.query(LoyaltyEntityWorkflow.get_tier)
    print(f"After redemption: balance={balance}, tier={tier}")

    print("\nSending batch of points events...")
    for i in range(10):
        await handle.signal(
            LoyaltyEntityWorkflow.add_points,
            {
                "event_id": f"batch-{i:04d}",
                "points": 100,
                "reason": f"batch_purchase_{i}",
            },
        )
    print("Sent 10 batch events (+1000 points total)")

    await asyncio.sleep(3)

    summary = await handle.query(LoyaltyEntityWorkflow.get_loyalty_summary)
    print(
        f"Final state: {summary.points_balance} points, "
        f"tier={summary.tier}, lifetime={summary.lifetime_points}"
    )

    # Uncomment to shut down the entity gracefully.
    # await handle.signal(LoyaltyEntityWorkflow.shutdown)
    # print("Sent shutdown Signal")

if __name__ == "__main__":
    asyncio.run(main())
```

This client code demonstrates the three interaction patterns:

- **Signals** (`add_points`): Fire-and-forget; returns immediately after the Server accepts the Signal.
- **Queries** (`get_loyalty_summary`, `get_points_balance`, `get_tier`): Read-only, no Events generated — safe at high frequency.
- **Updates** (`redeem_points`): Synchronous request-response; the caller blocks until the Workflow returns a confirmed result.

### Phase 6: Handle Continue-As-New for indefinite execution

Continue-As-New allows an Entity Workflow to run for years without unbounded history. When `workflow.info().is_continue_as_new_suggested()` returns `True`, the history is approaching the suggested threshold (~4,096 Events).

The `LoyaltyEntityWorkflow` checks this flag in two places:

1. **In the main wait condition.** The Workflow wakes up even if no Signals are pending, handling the case where history has grown during idle periods.
2. **After processing each event.** Avoids processing an entire batch before detecting an oversized history.

The `_do_continue_as_new` method handles the transition in three steps:

1. **Unprocessed Signals are preserved.** Any Signals still in the queue are serialized into `pending_events` on the state. The next Execution restores them in its constructor.
2. **All handlers complete.** `workflow.all_handlers_finished` ensures any in-flight Update handlers finish before the transition — cancelling them mid-execution would lose data.
3. **The transition occurs.** `workflow.continue_as_new(self._state)` ends the current Execution and starts a new one with the same Workflow Id, a new Run Id, and a fresh Event History.

After the transition, clients continue interacting with the same Workflow Id. Temporal automatically routes Signals, Queries, and Updates to the latest Run. Do not specify a Run Id when obtaining a Workflow handle, or your interactions will target a stale Execution.

### Phase 7: Ensure Workflow Definition determinism

Temporal replays Event History to resume a Workflow after a crash. The code must produce the same sequence of Commands on every replay.

The Python SDK sandbox blocks most non-deterministic operations, but you must still follow these rules:

| Do not use | Use instead | Reason |
|---|---|---|
| `datetime.now()` | `workflow.now()` | Returns the same value during replay |
| `random.random()` | `workflow.random().random()` | Seeded deterministically by the SDK |
| `uuid.uuid4()` | `workflow.uuid4()` | Produces the same value during replay |
| `time.sleep()` | `asyncio.sleep()` or `workflow.sleep()` | Converts to a Durable Timer |
| `print()` | `workflow.logger.info()` | Replay-safe; only logs on initial execution |
| Direct I/O (HTTP, file reads) | Activities | All I/O must happen in Activities |

**Testing for determinism.** Use the `Replayer` to verify code changes against a saved Event History:

```python
# test_determinism.py
import asyncio

from temporalio.client import WorkflowHistory
from temporalio.worker import Replayer

from workflows import LoyaltyEntityWorkflow

async def test_replay_compatibility() -> None:
    """Replay a saved Event History against the current code.

    Export history first:
        temporal workflow show --workflow-id loyalty-cust-42 --output json > history.json
    """
    replayer = Replayer(workflows=[LoyaltyEntityWorkflow])

    with open("history.json") as f:
        history = WorkflowHistory.from_json("loyalty-cust-42", f.read())

    # This raises NondeterminismError if the code is incompatible
    await replayer.replay_workflow(history)
    print("Replay succeeded: code is compatible with saved history")

if __name__ == "__main__":
    asyncio.run(test_replay_compatibility())
```

Run this test as part of your continuous integration pipeline before deploying Workflow code changes. If the replay fails with a `NondeterminismError`, the code change is not backward-compatible. Use Worker Versioning (see Phase 8) to deploy the change safely.

### (Optional) Phase 8: Manage Workflow evolution with Worker Versioning

Entity Workflows are the hardest case for code evolution: a loyalty account can span years and dozens of deployments. Any change that alters the sequence of Commands the Workflow generates causes a `NondeterminismError` during replay for in-progress Executions.

[Worker Versioning](/production-deployment/worker-deployments/worker-versioning) is the recommended solution: run multiple builds simultaneously, route each Execution to the build it started on, and drain old builds without any code-level branching.

> Worker Versioning requires Python SDK v1.11.0 or later, Temporal Server v1.29.1 or later, and Temporal CLI v1.4.1 or later.

#### Why PINNED is the right choice for Entity Workflows

`PINNED` is the canonical versioning behavior for Entity Workflows: each execution runs entirely on the Worker Deployment Version where it started. At a Continue-As-New boundary the Workflow can optionally upgrade to the latest build.

| Behavior | Guarantee | When to use |
|----------|-----------|-------------|
| `PINNED` | Each execution completes on the version where it started | Long-lived entities; no patching needed within a run |
| `AUTO_UPGRADE` | Moves to the latest version automatically | Short Workflows that complete before the next deploy |

#### Step 1: Annotate the Workflow

Add `versioning_behavior=VersioningBehavior.PINNED` to `@workflow.defn` in `workflows.py`:

```python
from temporalio.common import VersioningBehavior

@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class LoyaltyEntityWorkflow:
    ...
```

#### Step 2: Configure the Worker

Update `worker.py` to declare the deployment name and build ID:

```python
import os
from temporalio.common import WorkerDeploymentVersion
from temporalio.worker import Worker, WorkerDeploymentConfig

# BUILD_ID should be injected by CI/CD (e.g. a Git SHA). All Workers from the same build must match.
BUILD_ID = os.environ["BUILD_ID"]
DEPLOYMENT_NAME = os.getenv("TEMPORAL_DEPLOYMENT", "loyalty-points")

async def main() -> None:
    client = await Client.connect(TEMPORAL_ADDRESS, namespace=TEMPORAL_NAMESPACE)

    with concurrent.futures.ThreadPoolExecutor(max_workers=50) as activity_executor:
        worker = Worker(
            client,
            task_queue=TASK_QUEUE,
            workflows=[LoyaltyEntityWorkflow],
            activities=[
                record_points_transaction,
                record_redemption,
                send_tier_change_notification,
            ],
            activity_executor=activity_executor,
            deployment_config=WorkerDeploymentConfig(
                version=WorkerDeploymentVersion(
                    deployment_name=DEPLOYMENT_NAME,
                    build_id=BUILD_ID,
                ),
                use_worker_versioning=True,
            ),
            max_concurrent_workflow_tasks=200,
            max_concurrent_activities=100,
        )
        await worker.run()
```

#### Step 3: Roll out changes with the CLI

**Initial deployment** — start the first versioned build and make it Current:

```bash
BUILD_ID=1.0.0 python worker.py &

temporal worker deployment set-current-version \
  --deployment-name "loyalty-points" \
  --build-id "1.0.0"
```

**Deploying a change** — run the new build alongside the old one, ramp gradually, then promote:

```bash
# Start Workers on the new build
BUILD_ID=1.1.0 python worker.py &

# Canary: send 10% of new Workflows to the new build
temporal worker deployment set-ramping-version \
  --deployment-name "loyalty-points" \
  --build-id "1.1.0" \
  --percentage 10

# Promote to Current once satisfied
temporal worker deployment set-current-version \
  --deployment-name "loyalty-points" \
  --build-id "1.1.0"
```

Old `1.0.0` Workers keep running, draining their pinned Executions automatically.

Check drain status before stopping old Workers:

```bash
temporal worker deployment describe-version \
  --deployment-name "loyalty-points" \
  --build-id "1.0.0"
# DrainageStatus: drained  ← safe to shut down
```

#### Upgrading at the Continue-As-New boundary

Because `LoyaltyEntityWorkflow` already calls `workflow.continue_as_new()` to manage Event History, you can upgrade long-running Executions to a new build at that boundary — without patching.

In `_do_continue_as_new`, check `is_target_worker_deployment_version_changed()` before transitioning:

```python
from temporalio.workflow import ContinueAsNewVersioningBehavior

async def _do_continue_as_new(self) -> None:
    MAX_PENDING_CARRY = 500
    while len(self._pending_signals) > MAX_PENDING_CARRY:
        event_data = self._pending_signals.popleft()
        await self._process_points_event(event_data)

    self._state.pending_events = list(self._pending_signals)
    await workflow.wait_condition(workflow.all_handlers_finished)

    initial_versioning_behavior = None
    if workflow.info().is_target_worker_deployment_version_changed():
        # Newer build available — upgrade on this CaN boundary.
        initial_versioning_behavior = ContinueAsNewVersioningBehavior.AUTO_UPGRADE

    workflow.continue_as_new(
        self._state, initial_versioning_behavior=initial_versioning_behavior
    )
```

When `is_target_worker_deployment_version_changed()` is `False`, the CaN run stays on the same pinned build. This flag is refreshed after each Workflow Task, so upgrades are picked up on the next natural CaN boundary.

## Outcomes

You've built a loyalty account system where each customer is an independent, self-managing process. Points accrue consistently, tier upgrades happen in real time, redemptions can't be double-spent, and the account keeps running through deployments, restarts, and schema changes. The approach isn't specific to loyalty programs — any domain object with a long lifecycle and ongoing interactions can be modeled the same way: user profiles, IoT devices, subscription accounts, open orders. The loyalty implementation here gives you the full blueprint.

## Related resources

- [Entity Workflow Pattern](/design-patterns/entity-workflow) — pattern catalog reference for this Workflow with Go, Java, Python, and TypeScript implementations
- [Continue-As-New Pattern](/design-patterns/continue-as-new) — pattern catalog reference covering common pitfalls and history-size triggers
- [Temporal Python SDK documentation](/develop/python)
- [Temporal Python SDK API Reference](https://python.temporal.io)
- [Message Passing — Signals, Queries, Updates](/develop/python/workflows/message-passing)
- [Continue-As-New](/develop/python/workflows/continue-as-new)
- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning)
- [Failure Detection — Timeouts, Activity Heartbeating, and Retry Policies](/develop/python/best-practices/error-handling)
- [Temporal Python SDK samples](https://github.com/temporalio/samples-python)
- Companion article: [Player Sessions That Survive Anything](/guides/durable-gaming-sessions) — covers the related Actor Workflow pattern for entities that take action

---

# Build a distributed lock and semaphore with Workflow IDs

Source: https://docs.temporal.io/guides/lock-shared-resources

> Acquire permits by starting Child Workflows whose IDs name resource slots, with lease Timers for crash recovery and no external database or lock service.

**by Keith Tenzer**

This guide details a reusable, durable distributed lock for Temporal Workflows. The design doesn't rely on an external database, a central limiter, or any shared state outside Temporal itself.

This pattern is best for small, fixed pools of scarce resources where each permit is held for minutes, not milliseconds: GPU devices, lab hardware, tenant migrations, or similar operational resources. The latency maybe too high to qualify as a general-purpose high-throughput mutex or rate limiter.
Common approaches such as a database, cache-layer, or a central rate-limiter services introduce extra infrastructure and new failure modes. Holders of locks die without releasing leak orphan locks. Lock state drifts out of sync with the Workflows that hold them. Capacity changes require restarting limiter services.

This pattern represents each held permit or lock as its own short-lived child Workflow.
The child Workflow's ID encapsulates the resource and slot, for example `permit:gpu-pool:gpu-2`.
Temporal does not allow two running Workflows with the same ID, so acquiring a permit is an atomic operation using Temporal's own state.
The Workflow releases on a Signal pinned to the run.  If no release Signal is received, the permit Workflow's lease timeout handles orphan recovery.

By following this implementation plan, you will gain:

- **Atomic Acquire**: Take a permit by starting a child Workflow whose ID is the lock. The operation succeeds or fails atomically against Temporal state.
- **Lease-Based Recovery**: A per-permit lease Timer makes the slot reusable if the holder crashes before releasing. For external resources that can be corrupted by overlap, pass a fencing token to reject stale holders.
- **Horizontal Scale**: Permit traffic spreads across the slots in a scarce resource pool, which is useful for long-held resource coordination.

## Background and best practices

This section explains why Temporal Workflow IDs make a good distributed lock primitive and how the design recovers from the failure modes that hurt typical lock services.

**Distributed locking is a classical problem**
Solutions such as a database, cache-layer, and central rate-limiting services share a structural weakness: the lock state lives outside the runtime that holds it. The locking system and the work it gates can fall out of sync. A holder crash leaves an external lock entry in place until a separate cleanup job runs. A limiter outage blocks every gated Workflow, or lets every request through unchecked. Scaling the limiter is a separate operational concern from scaling the workload it gates.

Inside Temporal, by default, every running Workflow execution already has a globally unique, atomically enforced identifier. You cannot start two running Workflows with the same Workflow ID at the same time. That guarantee is the primitive a distributed mutex requires (guarantee only exists if not changing WorkflowId Conflict Policy from its default of FAIL). Treating each permit as its own child Workflow places the lock and the work it gates on the same platform: Temporal's history, durability, and Worker fleet.

**The Workflow ID is the lock**
Encapsulating the resource and slot into the Workflow ID (`permit:{resource}:{slot}`) makes the act of starting the child Workflow the acquire operation. The start either succeeds and the caller holds the permit, or it fails with `WorkflowAlreadyStartedError` and the caller tries the next slot.

If the parent Workflow closes for any reason without sending the release Signal (operator termination, execution timeout, unhandled exception), the default `ParentClosePolicy.TERMINATE` cancels the permit child immediately and the slot frees right away. The permit Workflow also runs a `wait_condition` with a lease timeout for cases where the holder is alive but went silent, and `start_child_workflow` sets `execution_timeout=lease` as a server-side backstop for permits that no Worker ever picks up. Together these mechanisms handle orphan recovery internally, eliminating the need for a separate cleanup process.

The `Semaphore` pins the release Signal to the specific run started at acquire time by holding the `ChildWorkflowHandle` from `start_child_workflow` and signalling on that handle. Without this pin, a late release Signal from a holder whose lease already expired could revoke a subsequent acquirer's permit. The `Semaphore` shuffles its slot list using `workflow.random()` so Workflows always probe in a random order. The shuffle produces the same order on every replay, as the Workflow Definition requires, and is uniform across callers. This spreads contention across the pool rather than concentrating on slot 0, for example.

Three operational characteristics matter before adopting the pattern:

- Acquire is not first-in-first-out (FIFO). When a slot frees, whomever claims it first, wins.
- Capacity changes apply only to acquires after the change. Already-held permits run to release or lease expiry on their original capacity.
- For production, host `PermitSlotWorkflow` on a dedicated Task Queue so application Worker backpressure does not slow permit acquire and release.
- Lease expiry is recovery, not proof that the old holder stopped using the external resource. For resources that can be corrupted by overlap, pass the permit run ID as a fencing token to the Activity or downstream system and reject stale holders there. 

## Target audience

This guide references the following roles:

- **Application developers**: Build the Workflows that acquire permits. Import the `Semaphore` helper and wrap critical sections in `async with sem.acquire(): ...`.
- **Platform operators**: Run the Worker that hosts `PermitSlotWorkflow`, tune lease and backoff defaults, and decide the Task Queue topology.

The process outlined in this document requires Python code changes and Worker configuration.
This pattern requires no additional infrastructure.

## Prerequisites

To complete the implementation plan, you will need:

- **Required software and tools**:
    - Python 3.12 or later ([download](https://www.python.org/downloads/))
    - Temporal Python SDK 1.7 or later ([github](https://github.com/temporalio/sdk-python))
    - `uv` 0.9.18 for Python dependency management ([install](https://docs.astral.sh/uv/))
    - Temporal CLI 1.6.2 for the local development server ([install](/cli))
    - Access to a Temporal Cluster 1.3.2 (local development server or Temporal Cloud).
- **Resources and access privileges**:
    - A Temporal Namespace where you can register Workers and start Workflows.
    - For Temporal Cloud: an API key with the Namespace Admin role for the target Namespace.
- **Required concepts**:
    - Familiarity with Temporal Workflows and Activities ([docs](/workflows)).
    - Child Workflows and parent close policies ([docs](/child-workflows)).
    - Signals and Queries ([docs](/encyclopedia/workflow-message-passing)).
    - Workflow Definition determinism ([docs](/workflow-definition#deterministic-constraints)).

## Architecture diagram

The following diagram shows the acquire and release flow when two Workflows compete for slots in a 2-slot pool. The first attempt to get a slot collides with an already held slot; the second succeeds.

```mermaid
sequenceDiagram
    autonumber
    actor App as Application Workflow
    participant Sem as Semaphore (helper)
    participant Permit as PermitSlotWorkflow (child)
    participant Temporal as Temporal Service

    App->>Sem: acquire(lease=10m, backoff=5s)
    Sem->>Temporal: start_child_workflow(id="permit:gpu-pool:2")
    Temporal-->>Sem: WorkflowAlreadyStartedError (slot held)
    Sem->>Temporal: start_child_workflow(id="permit:gpu-pool:3")
    Temporal-->>Sem: ChildWorkflowHandle (pinned to this run)
    Temporal->>Permit: dispatch run(PermitSlotInput)
    Sem-->>App: yield "3"
    Note over App: execute gated work
    App->>Sem: release on context exit
    Sem->>Temporal: signal "release" on the child handle
    Temporal->>Permit: deliver release Signal
    Permit-->>Temporal: complete ("released")
    Note over Permit,Temporal: slot ID becomes reusable
```

1. The application Workflow calls `Semaphore.acquire`. The `Semaphore` shuffles slot names using `workflow.random()` and attempts to acquire a slot.
2. The first attempt to acquire a slot starts a child Workflow with ID `permit:gpu-pool:2`. Temporal already has a running Workflow at that ID and rejects the start with `WorkflowAlreadyStartedError`.
3. The `Semaphore` falls through to the next slot and starts `permit:gpu-pool:3`. Temporal accepts the start and returns a `ChildWorkflowHandle` to the `Semaphore`.
4. Temporal dispatches the new execution to a Worker, and `PermitSlotWorkflow` enters its `wait_condition` with the lease timeout running.
5. The `Semaphore` yields the slot name (`"3"`) to the caller.
6. The application Workflow runs the gated work using the yielded slot identity.
7. On context-manager exit, the `Semaphore` sends a `release` Signal on the recorded `ChildWorkflowHandle`, which targets the specific run it started.
8. The permit Workflow's `wait_condition` returns, the Workflow completes with `released`, and the Workflow ID is reusable.

## Implementation plan

This section outlines the sequential phases to build the distributed lock library and integrate it into a Worker. 

### Define shared configuration

The library needs a small config module with lease and backoff defaults, a function that encodes the resource and slot into a Workflow ID, and a function that picks auth mode from environment variables.

Create a file named `distributed_lock/config.py`:

```python
# config.py

from __future__ import annotations

import os
from datetime import timedelta

from temporalio.client import Client
from temporalio.envconfig import ClientConfig

TEMPORAL_TASK_QUEUE: str = os.environ.get("TEMPORAL_TASK_QUEUE", "distributed-lock-tq")

DEFAULT_LEASE: timedelta = timedelta(minutes=10)
DEFAULT_BACKOFF: timedelta = timedelta(seconds=5)
DEFAULT_MAX_BACKOFF: timedelta = timedelta(minutes=1)
DEFAULT_BACKOFF: timedelta = timedelta(seconds=5)

PERMIT_WORKFLOW_ID_PREFIX: str = "permit"

def permit_workflow_id(resource: str, slot: str) -> str:
    return f"{PERMIT_WORKFLOW_ID_PREFIX}:{resource}:{slot}"

async def connect_temporal_client() -> Client:
    return await Client.connect(**ClientConfig.load_client_connect_config())
```

The `permit_workflow_id` function produces IDs like `permit:gpu-pool:2`. That string is the lock: Temporal will reject any second attempt to start a Workflow with the same ID while the first is running.
The default lease and backoff values are conservative starting points. Adjust these values based on the resource you're gating.
The `connect_temporal_client` function delegates to `temporalio.envconfig`, which reads connection settings from a TOML profile at `~/.config/temporalio/temporal.toml` (or the path in `TEMPORAL_CONFIG_FILE`) and applies environment variable overrides such as `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and the `TEMPORAL_TLS_*` family. This is the same configuration source the Temporal CLI uses, so a developer's existing CLI profile drives both the CLI and this Worker. API-key authentication enables Transport Layer Security (TLS) automatically; for self-hosted TLS without an API key, set the relevant `TEMPORAL_TLS_*` variables; for `temporal server start-dev`, leave the variables unset and the connection falls back to plain Transmission Control Protocol (TCP).

### Define the permit primitive

The permit primitive is a child Workflow whose ID is the lock. It accepts a resource name, slot name, and lease duration, then blocks on a `release` Signal with a timeout equal to the lease. On Signal it completes; on timeout it auto-completes for orphan recovery.

Create a file named `distributed_lock/workflow.py`:

```python
# workflow.py

from __future__ import annotations

from dataclasses import dataclass
from datetime import timedelta

from temporalio import workflow

@dataclass
class PermitSlotInput:
    resource: str
    slot: str
    lease_seconds: float

@workflow.defn(name="PermitSlotWorkflow")
class PermitSlotWorkflow:
    def __init__(self) -> None:
        self._released: bool = False

    @workflow.signal(name="release")
    def release(self) -> None:
        self._released = True

    @workflow.run
    async def run(self, input: PermitSlotInput) -> str:
        workflow.logger.info(
            "permit acquired",
            extra={"resource": input.resource, "slot": input.slot},
        )
        try:
            await workflow.wait_condition(
                lambda: self._released,
                timeout=timedelta(seconds=input.lease_seconds),
            )
            return "released"
        except TimeoutError:
            workflow.logger.warning(
                "permit lease expired - auto-releasing slot",
                extra={"resource": input.resource, "slot": input.slot},
            )
            return "lease_expired"
```

The `@workflow.signal(name="release")` decorator exposes the Signal the caller sends on context exit. The `workflow.wait_condition(predicate, timeout=lease)` is the durable wait: Temporal persists the wait, so a Worker restart does not lose the held permit. The Workflow does not set its own ID. The caller sets it via `start_child_workflow(id=...)`. The return value (`"released"` or `"lease_expired"`) lets operators distinguish clean releases from orphan recoveries directly in the Temporal UI.

### Build the caller-side Semaphore helper

The `Semaphore` is a thin async context manager around `workflow.start_child_workflow`. It is not a Workflow itself. Construct it with a fixed `capacity`; the `Semaphore` auto-generates slot names `"0"` through `"N-1"` and yields the name of the slot it acquired. Activities that need to map the slot to an external resource (CUDA device, database shard, etc.) translate the string with `int(slot)`.

Create a file named `distributed_lock/semaphore.py`:

```python
# semaphore.py

from __future__ import annotations

from contextlib import asynccontextmanager
from datetime import timedelta
from typing import AsyncIterator

from temporalio import workflow
from temporalio.exceptions import FailureError, WorkflowAlreadyStartedError
from temporalio.workflow import ChildWorkflowHandle

from distributed_lock.config import DEFAULT_BACKOFF, DEFAULT_LEASE, permit_workflow_id
from distributed_lock.workflow import PermitSlotInput, PermitSlotWorkflow

class Semaphore:
    def __init__(
        self,
        resource: str,
        *,
        capacity: int,
        task_queue: str | None = None,
    ) -> None:
        if capacity < 1:
            raise ValueError("capacity must be >= 1")
        self._resource = resource
        self._slots: list[str] = [str(i) for i in range(capacity)]
        self._task_queue = task_queue

    @asynccontextmanager
    async def acquire(
        self,
        *,
        lease: timedelta = DEFAULT_LEASE,
        backoff: timedelta = DEFAULT_BACKOFF,
    ) -> AsyncIterator[str]:
        slot, handle = await self._acquire_one(lease=lease, backoff=backoff)
        try:
            yield slot
        finally:
            await self._release(slot, handle)

    async def _acquire_one(
        self, *, lease: timedelta, backoff: timedelta
    ) -> tuple[str, ChildWorkflowHandle]:
        rng = workflow.random()
        lease_seconds = lease.total_seconds()
        while True:
            order = list(self._slots)
            rng.shuffle(order)
            for slot in order:
                wf_id = permit_workflow_id(self._resource, slot)
                try:
                    handle = await workflow.start_child_workflow(
                        PermitSlotWorkflow.run,
                        PermitSlotInput(
                            resource=self._resource,
                            slot=slot,
                            lease_seconds=lease_seconds,
                        ),
                        id=wf_id,
                        task_queue=self._task_queue,
                        execution_timeout=lease,
                    )
                    return slot, handle
                except WorkflowAlreadyStartedError:
                    continue
            await workflow.sleep(backoff)

    async def _release(self, slot: str, handle: ChildWorkflowHandle) -> None:
        try:
            await handle.signal("release")
        except FailureError as e:
            workflow.logger.info(
                "release signal target already finished (lease likely expired)",
                extra={
                    "resource": self._resource,
                    "slot": slot,
                    "error": str(e),
                },
            )
```

The constructor takes a fixed `capacity` and rejects values below 1. The `Semaphore` auto-generates slot names `"0"` through `"N-1"` and reshuffles them on every attempt using `workflow.random()` so long-running contention spreads evenly across slots. The `Semaphore` keeps the `ChildWorkflowHandle` returned by `start_child_workflow` and signals release on that handle, so the Signal cannot land on a subsequent acquirer's execution.
The `FailureError` caught in `_release` covers the lease-already-expired case: the slot is already free and the release is a no-op, logged at info level for observability.
Three orphan-recovery mechanisms work together to bound slot occupancy. The default `ParentClosePolicy.TERMINATE` is the fastest, when the parent Workflow closes for any reason without releasing, Temporal terminates the permit child immediately and the slot frees right away. The in-workflow timer `wait_condition(timeout=lease)` covers the case where the parent is still alive but the holder is silent, the permit Workflow exits cleanly as `"lease_expired"`, distinguishable in the Temporal UI. That Timer only starts ticking once a Worker runs the permit's first Task, however, so the workflow timeout `execution_timeout=lease` is a server-enforced backstop that ticks from Workflow creation regardless of whether a Worker ever picks it up. Together these guarantee the slot is never leaked, even if the permit Workflow never runs.

### Register the permit Workflow on a Worker

The Worker that runs application Workflows must also register `PermitSlotWorkflow`.
A single-queue topology like the one below is fine for development. For production deployments, use the dedicated Task Queue described in the Background and People and process sections.

Create a file named `worker/main.py`:

```python
# main.py

from __future__ import annotations

import asyncio

from temporalio.worker import Worker

from distributed_lock.config import TEMPORAL_TASK_QUEUE, connect_temporal_client
from distributed_lock.workflow import PermitSlotWorkflow

async def _run_worker() -> None:
    client = await connect_temporal_client()
    worker = Worker(
        client,
        task_queue=TEMPORAL_TASK_QUEUE,
        workflows=[PermitSlotWorkflow],
        activities=[],
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(_run_worker())
```

`PermitSlotWorkflow` is in the `workflows=[...]` list and the `activities=[]` list is empty.
The permit lifecycle is pure Workflow code, so this library needs no Activities. `worker.run()` blocks until the awaited Task is cancelled, which is sufficient for `temporal server start-dev` and CI: Ctrl-C raises `KeyboardInterrupt` and the Worker drains in-flight work as the cancellation propagates. For containerized production deployments, add a `SIGTERM` handler that cancels the run Task so pod restarts and rolling deploys also drain gracefully. The `connect_temporal_client` function picks up environment variables to choose between Temporal Cloud API key auth, self-hosted TLS, and plain TCP for `temporal server start-dev`, so the same Worker source runs against any environment.

### Example application Workflows

The two examples below show how application Workflows apply the pattern in different use cases. The API is the same in both: construct a `Semaphore` with a `capacity`, wrap the gated section in `async with sem.acquire(...)`, and use the yielded slot string however the workload requires. Source files live at `examples/gate_workflow.py` and `examples/gpu_workflow.py` in the reference implementation.

#### Generic Workflow throttling

Cap the number of concurrent holders without caring about slot identity. The yielded slot is a string `"0"` through `"N-1"`; the Activity passes it through as a label.

```python
# gate_workflow.py

from __future__ import annotations

from datetime import timedelta

from temporalio import workflow

from distributed_lock.semaphore import Semaphore

@workflow.defn(name="ThrottledGateWorkflow")
class ThrottledGateWorkflow:
    @workflow.run
    async def run(self, job_id: str) -> str:
        sem = Semaphore("app-gate", capacity=4)
        async with sem.acquire(lease=timedelta(minutes=10)) as slot:
            return await workflow.execute_activity(
                "do_gated_work",
                args=[job_id, slot],
                start_to_close_timeout=timedelta(minutes=5),
                heartbeat_timeout=timedelta(seconds=30),
            )
```

#### Resource locks (GPU, database shard, tenant)

Hand the yielded slot to an Activity that maps it to a specific resource. The slot is a string `"0"` through `"N-1"`; convert it with `int(slot)` if the resource is indexed numerically (CUDA device, shard ID).

```python
# gpu_workflow.py

from __future__ import annotations

from datetime import timedelta

from temporalio import workflow

from distributed_lock.semaphore import Semaphore

@workflow.defn(name="GpuTrainingWorkflow")
class GpuTrainingWorkflow:
    @workflow.run
    async def run(self, model_id: str) -> str:
        sem = Semaphore("gpu-pool", capacity=4)
        async with sem.acquire(lease=timedelta(minutes=45)) as slot:
            return await workflow.execute_activity(
                "run_training",
                args=[model_id, slot],
                start_to_close_timeout=timedelta(minutes=30),
                heartbeat_timeout=timedelta(minutes=3),
            )
```

#### Heartbeat the long-running Activity

Both Activities run inside the gated critical section, so a Worker crash mid-Activity must be detected promptly to free the slot. Set `heartbeat_timeout` on the Activity invocation and call `activity.heartbeat(...)` periodically inside the Activity body. Temporal fails the Activity attempt as soon as a Heartbeat is missed for longer than the timeout, the holder Workflow then exits its `async with` block, the `Semaphore` sends `release` to the slot Workflow, and the slot returns to the pool. The lease Timer remains as a coarse last-resort recovery for cases where Heartbeats cannot run.

```python
# activities.py

from __future__ import annotations

import asyncio

from temporalio import activity

@activity.defn(name="do_gated_work")
async def do_gated_work(job_id: str, slot: str) -> str:
    total_seconds = 60
    interval = 5
    elapsed = 0
    while elapsed < total_seconds:
        # `asyncio.sleep` simulates the real gated work
        # (API call, batch processing, etc.). Replace with the actual workload.
        await asyncio.sleep(interval)
        elapsed += interval
        activity.heartbeat(
            {"job_id": job_id, "slot": slot, "elapsed_seconds": elapsed}
        )
    return f"{job_id} done on slot {slot}"

@activity.defn(name="run_training")
async def run_training(model_id: str, slot: str) -> str:
    gpu_index = int(slot)  # map "0".."3" to a CUDA device index
    epochs = 10
    for epoch in range(1, epochs + 1):
        # `asyncio.sleep` simulates one training epoch on cuda:{gpu_index}.
        # Replace with the actual training step.
        await asyncio.sleep(60)
        activity.heartbeat(
            {"model_id": model_id, "gpu_index": gpu_index, "epoch": epoch}
        )
    return f"trained {model_id} on cuda:{gpu_index}"
```

Set the Heartbeat interval shorter than `heartbeat_timeout` (a 2x to 3x ratio is a common starting point) so a single delayed Heartbeat does not fail a healthy Activity. Pass progress data into `activity.heartbeat(...)` so retries can resume from the last reported point and operators can observe progress in the Temporal UI.

## Outcomes

By following this guide, you have built a durable distributed lock for Temporal Workflows backed by Temporal's own Workflow ID uniqueness guarantee.

You now have the capability to:

- Cap concurrent access to any shared resource or critical section without a database, cache-layer, or a central limiter service.
- Recover orphan permits automatically when holders crash, using per-permit lease Timers.
- Scale lock acquisition horizontally. Permit traffic distributes across short-lived child Workflows rather than concentrating on one limiter.

## Related resources

- [Temporal Python SDK Documentation](/develop/python). SDK reference for Workflows, Signals, child Workflows, and Worker setup.
- [Child Workflow Executions](/child-workflows). How parent and child Workflows interact, including parent close policies.
- [Workflow Message Passing](/encyclopedia/workflow-message-passing). Reference for Signals and Queries.
- [Reference implementation source repository](https://github.com/temporal-sa/temporal-workflow-throttler). The upstream repository bundles `PermitSlotWorkflow`, the `Semaphore` helper, demo Workers, a FastAPI service, and a React UI. The package directory is named `throttler/` in the source.

---

# Rate-limit downstream APIs with separate Task Queues

Source: https://docs.temporal.io/guides/rate-limit-downstream-apis

> Avoid HTTP 429 errors from APIs such as Stripe, OpenAI, and Twilio by giving each API its own Task Queue with a queue-wide Activity rate limit, in Python.

**by Cecil Phillip**

### Problem statement

Modern applications integrate with numerous external APIs (SendGrid, Stripe, OpenAI, Twilio) that enforce rate limits to protect their infrastructure. These limits vary by provider service and plan, for example:

- **SendGrid:** 100 emails/minute on free tier, 1000/minute on paid
- **Stripe:** 100 requests/second globally
- **OpenAI:** 60 requests/minute, 90,000 tokens/minute
- **Twilio:** 1 request/second per phone number

Without proactive rate limiting when calling downstream services, users may experience the following issues:
- **HTTP 429 errors:** Activities overwhelm APIs causing "Too Many Requests" errors
- **Account suspension:** Repeated violations can lead to temporary or permanent bans from the downstream service
- **Failed Workflows:** Without proper retry handling, Workflows may not be able to continue to make progress
- **Wasted execution:** Activities that will fail due to rate limits consume Worker resources
- **Cascading failures:** One Workflow's excessive API calls could affect others

### Solution

Use separate Task Queues with rate limiting configuration to protect downstream APIs. Create one Task Queue per rate-limited API and configure:

- `max_task_queue_activities_per_second`: Global rate limit across all Workers on the queue
- `max_activities_per_second`: Per-Worker rate limit (optional, for additional control)
- `max_concurrent_activities`: Limit concurrent executions when API has concurrency limits

This ensures Activities calling external APIs never exceed their rate limits, preventing 429 errors and account issues.

### Outcomes

- **Rate limit compliance:** Activities respect API rate limits, eliminating 429 errors and preventing account suspension
- **Improved reliability:** Workflows complete successfully without failures caused by rate limits
- **Resource efficiency:** Workers don't waste resources on doomed-to-fail Activities
- **Better API relationships:** Consistent rate limit compliance maintains good standing with API providers
- **Independent scaling:** Add more Workers without exceeding API rate limits (global queue limit applies)

## Background and best practices

### Task Queue fundamentals

Task Queues in Temporal are dynamically created when first referenced. Rate limiting is configured at the Worker level and enforced by the Temporal Server.

**Recommended practice:** Create one Task Queue per rate-limited API to isolate rate limits.

### Rate limiting configuration options

The Python SDK provides three rate limiting controls:

1. **max_task_queue_activities_per_second** (global): Limits Activity dispatch across ALL Workers on this queue. Enforced by Temporal Server. Best for API rate limits. Last value wins.

2. **max_activities_per_second** (per-worker): Limits Activities per Worker. Can be combined with global limit for finer control.

3. **max_concurrent_activities** (concurrency): Limits concurrent executions. Use when API has concurrent connection limits (for example, database pool size).

**Important:** Eager Activity execution can bypass a Task Queue's rate limit. It applies only when an Activity is scheduled on the same Task Queue as its Workflow and the Worker running the Workflow has the Activity registered. The Temporal Service then returns the Activity Task directly to that Worker instead of dispatching it through the Task Queue. A Worker that sets `max_task_queue_activities_per_second` never requests eager execution, so set it on every Worker that polls a rate-limited queue. This guide schedules each API's Activities on a dedicated Task Queue, separate from the Workflow Task Queue, so its Activities never run eagerly.

### Handling HTTP 429 responses

Even with rate limiting, occasional 429 errors may occur due to:
- Other systems using the same API key
- API provider reducing limits temporarily
- Burst traffic patterns

**Recommended practice:** Raise an exception with a specific next retry delay from the API's `Retry-After` header, and fallback to exponential backoff when the header is not available:

1. When catching 429 errors, check for the `Retry-After` response header
2. If present, raise an `ApplicationError` with `next_retry_delay` set to the header value
3. If not present, raise the exception normally to use the Activity's Retry Policy

```python
from temporalio.common import RetryPolicy

retry_policy=RetryPolicy(
    initial_interval=timedelta(seconds=1),
    maximum_interval=timedelta(minutes=10),
    backoff_coefficient=2.0,
    maximum_attempts=5,
)
```

**Understanding the retry cadence:**

With this configuration, the retry intervals follow this pattern:
- Attempt 1: 1 second
- Attempt 2: 2 seconds (1s × 2.0)
- Attempt 3: 4 seconds (2s × 2.0)
- Attempt 4: 8 seconds (4s × 2.0)
- Attempt 5: 16 seconds (8s × 2.0)

Total time before failure: ~31 seconds across 5 attempts.

**Key differences from default Retry Policy:**
- **maximum_attempts**: Set to `5` instead of the default `unlimited`. This prevents infinite retries for persistent rate limiting issues and ensures timely failure detection.
- **maximum_interval**: Set to `10 minutes` instead of the default `100x initial_interval`. This caps retry delays at a reasonable duration.

For rate-limited APIs, capping `maximum_attempts` ensures that Activities don't retry indefinitely if an API is experiencing extended downtime or if your rate limits have been permanently reduced.

This approach respects the API's rate limit guidance while providing a sensible fallback strategy.

### Operational considerations

- **Monitor API usage:** Track actual API calls vs rate limits to adjust configured rate limits in your Temporal Task Queues
- **API limit changes:** API providers may change rate limits; monitor and update Worker configuration
- **Burst allowance:** Some APIs allow short bursts above stated limits; test to determine safe Temporal limits
- **Multiple environments:** Use separate API keys and Task Queues for dev/staging/prod

#### Task Queue backlog and draining strategies

During throttling events, Task Queues can grow significantly. Operators need to switch to **draining mitigation mode** when backlogs occur.

**Critical considerations:**
- **Determine acceptable SLA/SLO for draining:** How long is acceptable for the queue to drain? Hours? Days?
- **Cascading failures:** Simply increasing downstream API or Temporal Cloud rate limits may move the bottleneck elsewhere, potentially overwhelming the next component in line

**Mitigation strategies (in order of consideration):**

1. **Request downstream API rate limit increases** - Work with API providers to increase your rate limits
2. **Request Temporal Cloud rate limit increases** - If using Temporal Cloud, request higher limits
3. **Scale Worker pools** - Add more Workers or adjust Worker configuration
4. **Increase internal resources** - Scale your infrastructure
5. **Identify the next bottleneck** - Determine what will become throttled next to prevent cascading failures

#### CLI commands for operators

**Adjust Task Queue rate limits dynamically:**

```bash
# Set rate limit for a specific task queue
temporal task-queue config set --queue-rps-limit 99 --task-queue sendgrid-api
```

This allows operators to adjust rate limits without redeploying workers.

**Reset stuck activities:**

When activities are stuck at their maximum retry intervals:

```bash
# Reset activities to retry immediately
temporal activity reset --workflow-id <workflow-id> --activity-id <activity-id>
```

This is useful when downstream APIs recover from outages and you want to immediately retry activities that are waiting at long backoff intervals (for example, 10 minutes).

## Target audience

- **Temporal Workflow & Activity developers:** Implementing API integrations with rate limiting
- **Platform operators:** Configuring and monitoring rate-limited Workers
- **API integration engineers:** Ensuring compliance with third-party API limits
- **SRE teams:** Preventing API-related outages

This implementation requires Worker configuration, Activity error handling, and monitoring of API usage against limits.

## Prerequisites

### Required software, infrastructure, and tools

- Temporal Server v1.17+ (for `max_task_queue_activities_per_second` support)
- Python 3.8 or later
- Temporal Python SDK v1.0.0 or later (`pip install temporalio`)
- API keys and documentation for external services

### Resources and access privileges

- Temporal Namespace with permissions to start Workflows and register Workers
- API keys with known rate limits for external services
- Access to API provider dashboards to monitor usage

### Required concepts

- Temporal Workflows, Activities, and Task Queues
- HTTP client libraries (httpx, requests)
- API authentication (API keys, OAuth)
- Exponential backoff and retry strategies

## Architecture diagram(s)

### Rate-limited Task Queues architecture

```mermaid
flowchart TB
    subgraph Workflow["Workflow Execution"]
        WF[Notification<br/>Workflow]
    end

    subgraph Routing["Activity Routing"]
        Decision{Which<br/>API?}
    end

    subgraph Queues["Rate-Limited Task Queues"]
        SGTQ[sendgrid-api<br/>100/min = 1.67/sec]
        TWTQ[twilio-api<br/>60/min = 1/sec]
        AITQ[openai-api<br/>60/min = 1/sec]
    end

    subgraph Workers["Worker Pools"]
        subgraph SGWP["SendGrid Workers"]
            SGW1[Worker 1]
            SGW2[Worker 2]
            SGW3[Global Limit:<br/>1.5/sec]
        end
        subgraph TWWP["Twilio Workers"]
            TWW1[Worker 1]
            TWW2[Worker 2]
            TWW3[Global Limit:<br/>0.9/sec]
        end
        subgraph AIWP["OpenAI Workers"]
            AIW1[Worker 1]
            AIW2[Worker 2]
            AIW3[Global Limit:<br/>0.9/sec]
        end
    end

    subgraph APIs["External APIs"]
        SGAPI[SendGrid API<br/>Rate Limit: 100/min]
        TWAPI[Twilio API<br/>Rate Limit: 60/min]
        AIAPI[OpenAI API<br/>Rate Limit: 60/min]
    end

    WF --> Decision

    Decision -->|Email| SGTQ
    Decision -->|SMS| TWTQ
    Decision -->|AI| AITQ

    SGTQ --> SGWP
    TWTQ --> TWWP
    AITQ --> AIWP

    SGWP -.->|Respects<br/>100/min limit| SGAPI
    TWWP -.->|Respects<br/>60/min limit| TWAPI
    AIWP -.->|Respects<br/>60/min limit| AIAPI

    classDef blue stroke-width:1px
    classDef amber stroke-width:1px
    classDef purple stroke-width:1px
    class SGTQ blue
    class TWTQ amber
    class AITQ purple
```

## Implementation plan

### Step 1: Identify API rate limits

Document rate limits for all external APIs your application uses.

**Example rate limit inventory:**

| API | Rate Limit | Concurrency Limit | Plan/Tier | Notes |
|-----|-----------|-------------------|-----------|-------|
| SendGrid | 100 req/min | None | Free | Burst: 300/min for 1 min |
| Stripe | 100 req/sec | None | Standard | Per API key globally |
| OpenAI | 60 req/min | None | Tier 1 | Also: 90K tokens/min |
| Twilio | 60 req/min | None | Free | Per phone number |
| Database | None | 100 connections | N/A | Connection pool limit |

**Actions:**
1. Review API documentation for each integrated service
2. Check your current plan/tier limits
3. Determine if limits are per API key, per account, or per resource
4. Set Temporal rate limits to 90% of API limits (leave safety buffer)

### Step 2: Define Task Queue constants

**File: `task_queues.py`**

```python
"""Task Queue constants for rate-limited APIs."""

# Rate-limited API task queues
SENDGRID_API_QUEUE = "sendgrid-api"
STRIPE_API_QUEUE = "stripe-api"
OPENAI_API_QUEUE = "openai-api"
TWILIO_API_QUEUE = "twilio-api"
```

### Step 3: Configure Workers with rate limiting

**File: `worker_sendgrid.py`**

```python
"""Worker for rate-limited SendGrid email activities."""
import asyncio
import logging
from temporalio.client import Client
from temporalio.worker import Worker

from task_queues import SENDGRID_API_QUEUE
from activities import send_email, send_batch_email

logging.basicConfig(level=logging.INFO)

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue=SENDGRID_API_QUEUE,
        activities=[send_email, send_batch_email],

        # SendGrid free tier: 100 emails/minute = 1.67/sec
        # Set to 1.5/sec for safety buffer
        # This limit applies GLOBALLY across all workers on this queue
        max_task_queue_activities_per_second=1.5,

        # Optional: Also limit per-worker to prevent single worker bursts
        max_activities_per_second=0.5,

        # Limit concurrent connections (SendGrid allows many concurrent)
        max_concurrent_activities=20,
    )

    logging.info(
        f"Starting SendGrid worker on {SENDGRID_API_QUEUE} "
        f"(max 1.5/sec globally)"
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**File: `worker_openai.py`**

```python
"""Worker for rate-limited OpenAI API activities."""
import asyncio
import logging
from temporalio.client import Client
from temporalio.worker import Worker

from task_queues import OPENAI_API_QUEUE
from activities import generate_text, generate_embeddings

logging.basicConfig(level=logging.INFO)

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue=OPENAI_API_QUEUE,
        activities=[generate_text, generate_embeddings],

        # OpenAI Tier 1: 60 requests/minute = 1/sec
        # Set to 0.9/sec for safety
        max_task_queue_activities_per_second=0.9,

        # Limit concurrent requests
        max_concurrent_activities=5,
    )

    logging.info(
        f"Starting OpenAI worker on {OPENAI_API_QUEUE} "
        f"(max 0.9/sec globally)"
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**File: `worker_stripe.py`**

```python
"""Worker for rate-limited Stripe API activities."""
import asyncio
import logging
from temporalio.client import Client
from temporalio.worker import Worker

from task_queues import STRIPE_API_QUEUE
from activities import create_payment, refund_payment

logging.basicConfig(level=logging.INFO)

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue=STRIPE_API_QUEUE,
        activities=[create_payment, refund_payment],

        # Stripe: 100 requests/second globally
        # Set to 90/sec for safety
        max_task_queue_activities_per_second=90,

        # High concurrency allowed
        max_concurrent_activities=50,
    )

    logging.info(
        f"Starting Stripe worker on {STRIPE_API_QUEUE} "
        f"(max 90/sec globally)"
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**Deployment guidance:**
- Deploy 2-5 Workers per API-specific Task Queue
- Rate limit is enforced globally by Temporal Service across all Workers
- More Workers = better fault tolerance, but rate limit still applies
- Monitor Worker CPU/memory; rate-limited Workers typically have low utilization

### Step 4: Implement Activities with proper error handling

**File: `activities.py`**

```python
"""Activities that call rate-limited external APIs."""
import httpx
from temporalio import activity
from temporalio.exceptions import ApplicationError
from datetime import timedelta

# API keys (use environment variables in production)
SENDGRID_API_KEY = "your-sendgrid-key"
STRIPE_API_KEY = "your-stripe-key"
OPENAI_API_KEY = "your-openai-key"

@activity.defn
async def send_email(email_data: dict) -> dict:
    """
    Send email via SendGrid.

    Rate limited by worker configuration to 1.5/sec globally.
    """
    activity.logger.info(f"Sending email to {email_data['to']}")

    async with httpx.AsyncClient() as client:
        try:
            response = await client.post(
                "https://api.sendgrid.com/v3/mail/send",
                headers={
                    "Authorization": f"Bearer {SENDGRID_API_KEY}",
                    "Content-Type": "application/json",
                },
                json={
                    "personalizations": [{"to": [{"email": email_data["to"]}]}],
                    "from": {"email": "noreply@example.com"},
                    "subject": email_data["subject"],
                    "content": [{"type": "text/html", "value": email_data["body"]}],
                },
                timeout=30.0,
            )

            response.raise_for_status()
            activity.logger.info(f"Email sent to {email_data['to']}")

            return {"status": "sent", "email": email_data["to"]}

        except httpx.HTTPStatusError as e:
            if e.response.status_code == 429:
                # Check for Retry-After header (in seconds)
                retry_after = e.response.headers.get("Retry-After")

                if retry_after:
                    # Use specific retry delay from API
                    retry_delay = timedelta(seconds=int(retry_after))
                    activity.logger.warning(
                        f"Rate limit hit for {email_data['to']}, "
                        f"retry after {retry_after}s"
                    )
                    raise ApplicationError(
                        "Rate limit exceeded",
                        non_retryable=False,
                        next_retry_delay=retry_delay,
                    )
                else:
                    # Fallback to exponential backoff via retry policy
                    activity.logger.warning(
                        f"Rate limit hit for {email_data['to']}, "
                        f"using exponential backoff"
                    )
                    raise

            # Other HTTP errors
            activity.logger.error(f"HTTP error {e.response.status_code}: {e}")
            raise

        except Exception as e:
            activity.logger.error(f"Failed to send email: {e}")
            raise

@activity.defn
async def generate_text(prompt: str) -> dict:
    """
    Generate text using OpenAI API.

    Rate limited to 0.9/sec globally.
    """
    activity.logger.info(f"Generating text for prompt: {prompt[:50]}...")

    async with httpx.AsyncClient() as client:
        try:
            response = await client.post(
                "https://api.openai.com/v1/chat/completions",
                headers={
                    "Authorization": f"Bearer {OPENAI_API_KEY}",
                    "Content-Type": "application/json",
                },
                json={
                    "model": "gpt-3.5-turbo",
                    "messages": [{"role": "user", "content": prompt}],
                },
                timeout=60.0,
            )

            response.raise_for_status()
            result = response.json()

            activity.logger.info("Text generated successfully")
            return {
                "text": result["choices"][0]["message"]["content"],
                "usage": result["usage"],
            }

        except httpx.HTTPStatusError as e:
            if e.response.status_code == 429:
                # Check for Retry-After header
                retry_after = e.response.headers.get("Retry-After")
                error_data = e.response.json()

                if retry_after:
                    # Use specific retry delay from API
                    retry_delay = timedelta(seconds=int(retry_after))
                    activity.logger.warning(
                        f"OpenAI rate limit, retry after {retry_after}s: "
                        f"{error_data.get('error', {}).get('message')}"
                    )
                    raise ApplicationError(
                        "OpenAI rate limit exceeded",
                        non_retryable=False,
                        next_retry_delay=retry_delay,
                    )
                else:
                    # Fallback to exponential backoff
                    activity.logger.warning(
                        f"OpenAI rate limit, using exponential backoff: "
                        f"{error_data.get('error', {}).get('message')}"
                    )
                    raise

            activity.logger.error(f"OpenAI error {e.response.status_code}: {e}")
            raise

@activity.defn
async def create_payment(payment_data: dict) -> dict:
    """
    Create payment using Stripe API.

    Rate limited to 90/sec globally.
    """
    activity.logger.info(
        f"Creating payment for ${payment_data['amount']/100:.2f}"
    )

    async with httpx.AsyncClient() as client:
        try:
            response = await client.post(
                "https://api.stripe.com/v1/payment_intents",
                headers={
                    "Authorization": f"Bearer {STRIPE_API_KEY}",
                },
                data={
                    "amount": payment_data["amount"],
                    "currency": payment_data["currency"],
                    "payment_method": payment_data["payment_method"],
                    "confirm": "true",
                },
                timeout=30.0,
            )

            response.raise_for_status()
            result = response.json()

            activity.logger.info(f"Payment created: {result['id']}")
            return {
                "payment_id": result["id"],
                "status": result["status"],
            }

        except httpx.HTTPStatusError as e:
            if e.response.status_code == 429:
                # Check for Retry-After header
                retry_after = e.response.headers.get("Retry-After")

                if retry_after:
                    # Use specific retry delay from API
                    retry_delay = timedelta(seconds=int(retry_after))
                    activity.logger.warning(
                        f"Stripe rate limit hit, retry after {retry_after}s"
                    )
                    raise ApplicationError(
                        "Stripe rate limit exceeded",
                        non_retryable=False,
                        next_retry_delay=retry_delay,
                    )
                else:
                    # Fallback to exponential backoff
                    activity.logger.warning(
                        "Stripe rate limit hit, using exponential backoff"
                    )
                    raise

            activity.logger.error(f"Stripe error {e.response.status_code}")
            raise
```

### Step 5: Route Activities to rate-limited queues in Workflows

**File: `notification_workflow.py`**

```python
"""Workflow that routes to rate-limited API queues."""
from datetime import timedelta
from dataclasses import dataclass
from temporalio import workflow
from temporalio.common import RetryPolicy

with workflow.unsafe.imports_passed_through():
    from task_queues import SENDGRID_API_QUEUE, OPENAI_API_QUEUE, STRIPE_API_QUEUE

@dataclass
class NotificationRequest:
    user_email: str
    notification_type: str
    data: dict

@workflow.defn
class NotificationWorkflow:
    """
    Send notifications via multiple rate-limited APIs.

    Routes to appropriate queues based on notification type.
    """

    @workflow.run
    async def run(self, request: NotificationRequest) -> dict:
        workflow.logger.info(
            f"Processing {request.notification_type} notification "
            f"for {request.user_email}"
        )

        results = {}

        # Send email (rate-limited to 1.5/sec)
        if "email" in request.notification_type:
            email_result = await workflow.execute_activity(
                "send_email",
                {
                    "to": request.user_email,
                    "subject": f"{request.notification_type} Notification",
                    "body": str(request.data),
                },
                task_queue=SENDGRID_API_QUEUE,
                start_to_close_timeout=timedelta(minutes=2),
                retry_policy=RetryPolicy(
                    initial_interval=timedelta(seconds=1),
                    maximum_interval=timedelta(minutes=10),
                    backoff_coefficient=2.0,
                    maximum_attempts=5,
                ),
            )
            results["email"] = email_result

        # Generate AI content (rate-limited to 0.9/sec)
        if "ai" in request.notification_type:
            ai_result = await workflow.execute_activity(
                "generate_text",
                f"Generate notification: {request.data}",
                task_queue=OPENAI_API_QUEUE,
                start_to_close_timeout=timedelta(minutes=5),
                retry_policy=RetryPolicy(
                    initial_interval=timedelta(seconds=1),
                    maximum_interval=timedelta(minutes=10),
                    backoff_coefficient=2.0,
                    maximum_attempts=5,
                ),
            )
            results["ai"] = ai_result

        workflow.logger.info(f"Notification sent to {request.user_email}")
        return results
```

**Starter example:**

```python
# start_workflow.py
"""Start notification workflow."""
import asyncio
from temporalio.client import Client
from notification_workflow import NotificationWorkflow, NotificationRequest

async def main():
    client = await Client.connect("localhost:7233")

    # Send 100 notifications - rate limiting prevents API overload
    for i in range(100):
        request = NotificationRequest(
            user_email=f"user{i}@example.com",
            notification_type="email",
            data={"message": f"Notification {i}"},
        )

        handle = await client.start_workflow(
            NotificationWorkflow.run,
            request,
            id=f"notification-{i}",
            task_queue="workflows",
        )

        print(f"Started workflow {i}: {handle.id}")

    print("All workflows started. Rate limiting ensures API compliance.")

if __name__ == "__main__":
    asyncio.run(main())
```

## Conclusion

By implementing rate-limited separate Task Queues for external APIs, you have achieved:

1. **Rate limit compliance:** Activities respect API rate limits (for example, SendGrid 1.5/sec, OpenAI 0.9/sec), eliminating 429 errors and preventing account suspension

2. **Improved reliability:** Workflows complete successfully without rate limit failures. Global rate limiting ensures that adding more Workers doesn't violate API limits.

3. **Better API relationships:** Consistent rate limit compliance maintains good standing with API providers, avoiding temporary bans or throttling

4. **Independent scaling:** Add more Workers to increase throughput for other operations without exceeding API rate limits (global queue limit still applies)

Your application now safely integrates with rate-limited external APIs, ensuring compliance and reliability.

## Related resources

### Official documentation
- [Temporal Documentation - Task Routing](/task-routing)
- [Temporal Best Practices - Separate Task Queues](/best-practices/worker#separate-task-queues-logically)
- [Python SDK Worker Configuration](https://python.temporal.io/temporalio.worker.Worker.html)

### Related patterns
- [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting) - Pattern catalog reference for this technique, covering `MaxTaskQueueActivitiesPerSecond` versus per-worker and concurrency-slot alternatives
- [Separate Task Queues - Worker Affinity](worker-execution-affinity) - For activities on same Worker

### Community resources
- [Forum: When to Use Multiple Task Queues](https://community.temporal.io/t/in-what-situation-should-we-use-multiple-separated-task-queues/1254)
- [Slack: Rate Limiting Activities](https://temporal.io/slack) - Search for "rate limiting" discussions

### API documentation
- [SendGrid Rate Limits](https://docs.sendgrid.com/api-reference/rate-limits)
- [Stripe Rate Limits](https://docs.stripe.com/rate-limits)
- [OpenAI Rate Limits](https://platform.openai.com/docs/guides/rate-limits)

---

# Pause a Workflow on failure and resume it after a fix

Source: https://docs.temporal.io/guides/recover-without-restart

> A TypeScript loan pipeline that pauses on permanent failures, takes a corrective Signal, resumes in place, and unwinds side effects with saga compensation.

**by Tao Guo**

This pattern shows how to build a multi-step business process that pauses when it hits a permanent failure — a malformed identifier, an exceeded policy limit, a failed compliance check — allows a human or automated agent to correct the underlying issue, and resumes from where it stopped without restarting from the beginning.
Queryable metadata routes each blocked process to the right resolution resource, and operators get full visibility into the state of every individual execution and across the entire pipeline.

![A blocked Workflow showing the failure details and a form to send a corrective Signal](https://docs.temporal.io/guides/assets/recover-without-restart/keep-business-moving-list-view.png)

## Problem statement

Business processes such as loan origination, insurance claims, and onboarding involve sequential steps where certain failures — invalid input, regulatory violations, business policy breaches — cannot be resolved by automated retries.
An invalid Social Security number will never pass a credit check regardless of how often you retry it.
A title search against a missing property identifier will return the same error no matter how many times you repeat it.
These failures require someone to inspect the error, correct the data, and instruct the process to continue.

Without a durable orchestration layer, teams typically implement this by writing failed records to a dead-letter queue, building a separate reconciliation service, and stitching together polling loops to resume processing.
This approach is fragile: the process state lives across multiple systems, corrections are ad-hoc, and operators have no single view of which applications need attention and why.
When something goes wrong, the common workaround is to fix the data and restart the entire process from the beginning, losing all progress from steps that already completed.

## Solution

You will use a Temporal Workflow to orchestrate a six-step home loan processing pipeline.
When a step encounters a problem that retries alone cannot solve — a missing document, a policy limit breach, or a regulatory hold — it throws a non-retryable `ApplicationFailure` and the Workflow pauses, preserving all progress.
Custom Search Attributes advertise the blocked state so that operators or AI agents can find and prioritize the right cases.
An operator sends a Signal containing corrected data, which patches the application in-place and wakes the Workflow to retry the failed step.
The process picks up where it left off: completed steps do not re-execute, and the full correction history is preserved.

## Architectural benefit

While you can use traditional architectures to implement a similar solution, they often involve significantly more complexity and demand higher implementation and operational costs. A Temporal-based solution is typically much leaner and more robust.

![Architectural comparison between a traditional implementation and the Temporal-based solution](https://docs.temporal.io/guides/assets/recover-without-restart/keep-business-moving-architectural-comparison.png)

## What you will achieve

By completing this pattern, you will:

- Pause a business process on a non-retryable failure and resume after correction without restarting from the beginning.
- Unwind side-effecting steps through a LIFO **[saga](/design-patterns/saga-pattern)** compensation stack when forward progress is not possible — for example, a compliance block or an explicit cancellation from the applicant.
- Recover from compensation failures using the same pause-and-fix loop so a stuck rollback does not leave the process in a partially unwound state.
- Route blocked processes to the right human or automated agent using queryable Search Attributes.
- Inspect the current state of any individual process or filter across all processes through a single API.
- Maintain a complete audit trail of every data correction and every compensation applied during the lifecycle of each process.

## Background and best practices

Temporal Workflows persist their execution state, including local variables, call stacks, and pending Timers, to the Temporal Cluster.
When a Workflow suspends on `condition()`, the Worker completes the Workflow Task and the execution can be evicted from the Worker's in-memory cache to make room for other Workflows, freeing the associated resources.
The Workflow resumes where it left off when the condition becomes true, regardless of how much wall-clock time passes.
This means thousands of loan applications can sit in a `PENDING_FIX` state simultaneously without consuming Worker capacity.

The primary architectural challenge is distinguishing between transient failures that automated retries can resolve and permanent failures that require human intervention.
Temporal addresses this with [`ApplicationFailure.nonRetryable()`](/references/failures#application-failure), which instructs the SDK to skip the Retry Policy and propagate the error to the Workflow code without delay.
The Workflow then has full control over how to handle the failure: it can log the error, update Search Attributes for visibility, and suspend until a [Signal](/develop/typescript/workflows/message-passing#signals) arrives.

Custom [Search Attributes](/search-attribute) provide the mechanism for routing blocked Workflows to the correct resolution resource.
By upserting attributes like `LoanStatus` and `FailedActivity` at each State Transition, you create a queryable index across all active Workflows.
An operations dashboard can filter for `LoanStatus = 'PENDING_FIX' AND FailedActivity = 'runCreditCheck'` to show which applications need a credit-related data correction.
This same query interface supports automated agents that poll for specific failure categories and apply corrections programmatically.

Temporal [Queries](/develop/typescript/workflows/message-passing#queries) provide synchronous read access to the current state of any running Workflow.
Unlike Search Attributes, which expose denormalized metadata for cross-Workflow queries, Queries return the full internal state of a single execution.
Together, Search Attributes and Queries give you both the aggregate view across your entire pipeline and the detailed view into any individual Workflow.

## Target audience

Engineers familiar with Temporal foundations who need to handle long-running, complex business processes.

## Prerequisites

To execute the steps in this pattern, you must have:

- **Required software, infrastructure, and tools:** Temporal TypeScript SDK v1.13.0 or later, Node.js v18 or later, Temporal CLI v1.6.1 or later.
- **Resources and access privileges:** A running Temporal Cluster (local dev server or Temporal Cloud) with permissions to create custom Search Attributes and start Workflows.
- **Required concepts:** Familiarity with Temporal Workflows, Activities, Signals, Queries, Search Attributes, and Workers.

Install the Temporal CLI and start a local dev server:

```bash
# macOS
brew install temporal

# Or download directly from https://docs.temporal.io/cli#install

# Start the local development server
temporal server start-dev
```

This starts a Temporal Server on `localhost:7233` with the Web UI at `http://localhost:8233`.

**Note:** This guide uses the TypeScript SDK but the pattern applies to any Temporal SDK.

## Architecture diagram

The following diagram illustrates the flow of a loan application through the six-step pipeline, including the pause-and-resume cycle when an Activity encounters a permanent failure.

```mermaid
sequenceDiagram
    participant Client
    participant WF as Workflow
    participant Act as Activities
    participant Op as Operator / Agent

    Client->>WF: Start homeLoanWorkflow
    WF->>+Act: Execute Activity
    Act-->>-WF: ApplicationFailure.nonRetryable()
    WF->>WF: Set PENDING_FIX, await condition()

    Note over Op, Act: Correction cycle
    Op->>WF: Query blocked Workflows
    WF-->>Op: PENDING_FIX list
    Op->>WF: Signal: retry({key, value})
    WF->>WF: Patch data, wake condition
    WF->>+Act: Retry Activity
    Act-->>-WF: Success

    WF->>WF: Advance to next Activity
```

1. A client starts `homeLoanWorkflow` with the loan application data.
2. The Workflow executes Activities sequentially. On a non-retryable failure, the Activity throws `ApplicationFailure.nonRetryable()`.
3. The Workflow catches the error, sets Search Attributes to `PENDING_FIX`, and suspends via `await condition(() => retryRequested)`.
4. An operator or agent queries for blocked Workflows, sends a corrective Signal with the field name and new value.
5. The Signal handler patches the data, records the fix, and wakes the Workflow.
6. The Workflow retries the failed Activity. On success it advances; on failure the cycle repeats from step 3.

## Implementation plan

Prior to executing this plan, ensure you have your Temporal Cluster running and the custom Search Attributes created.

### Create custom Search Attributes

Before writing any code, register the custom Search Attributes that your Workflow will use to advertise its state.
These attributes enable the visibility queries that route blocked Workflows to the appropriate resolution resource.

```bash
temporal operator search-attribute create --name LoanStatus --type Keyword
temporal operator search-attribute create --name FailedActivity --type Keyword
```

`LoanStatus` tracks the current pipeline stage: `STARTED`, `INCOME_VERIFIED`, `CREDIT_CHECKED`, `APPRAISAL_ORDERED`, `TITLE_SEARCHED`, `UNDERWRITTEN`, `CLOSED`, or `PENDING_FIX`.
`FailedActivity` records the name of the Activity that caused the failure.
Together, these two attributes allow you to write queries like `LoanStatus = 'PENDING_FIX' AND FailedActivity = 'runCreditCheck'` to find every application blocked on a credit check.

### Define data models

Create the interfaces that define the data flowing through Workflows and Activities.
It is a best practice to use a single serializable input to Workflows and Activities.

```typescript
// src/models.ts

// Input data for the loan processing pipeline
export interface LoanApplication {
  applicationId: string;
  applicantName: string;
  ssn: string;
  employerName: string;
  annualIncome: number;
  propertyAddress: string;
  propertyId: string;
  loanAmount: number;
  downPayment: number;
}

// Tracks the current pipeline stage; PENDING_FIX means the Workflow is waiting for a correction
export type LoanStatus =
  | 'STARTED'
  | 'INCOME_VERIFIED'
  | 'CREDIT_CHECKED'
  | 'APPRAISAL_ORDERED'
  | 'TITLE_SEARCHED'
  | 'UNDERWRITTEN'
  | 'CLOSED'
  | 'PENDING_FIX'
  | 'FAILED';

// Records a single data correction applied during execution
export interface FixEntry {
  activity: string;   // which Activity was blocked
  field: string;      // which field was corrected
  oldValue: string;
  newValue: string;
  error: string;      // the failure message that triggered the fix
  id?: string;        // client-supplied idempotency key for duplicate Signal detection
}

// Complete queryable state of a pipeline execution
export interface LoanState {
  status: LoanStatus;
  failedActivity: string;
  failureMessage: string;
  completedActivities: string[];  // Activities that have already succeeded
  fixHistory: FixEntry[];         // audit trail of all corrections
  application: LoanApplication;   // current data including patches
}

// Signal payload: the field to correct and its new value
export interface RetryUpdate {
  key?: keyof LoanApplication | '';
  value?: string;
  id?: string;  // optional idempotency key — Signals are at-least-once
}
```

### Define the Activities

Create functions that validate loan application data at each pipeline stage.
Temporal Activities automatically recover from transient failures — network timeouts, temporary service unavailability, rate limits — through their built-in [Retry Policy](/encyclopedia/retry-policies) and [timeout management](/develop/typescript/activities/timeouts#activity-timeouts).
When an Activity encounters a permanent failure that retries cannot fix — invalid input data, a policy violation, or a missing record — it throws `ApplicationFailure.nonRetryable()` to bypass the Retry Policy and propagate the error directly to the Workflow.

```typescript
// src/activities.ts

import { ApplicationFailure } from '@temporalio/activity';

// Step 1: Validate the applicant's employment and income
export async function verifyIncome(
  applicantName: string,
  employerName: string,
  annualIncome: number
): Promise<string> {
  // Unrecognized employer — cannot verify
  if (employerName === 'UNKNOWN_EMPLOYER') {
    throw ApplicationFailure.nonRetryable(
      `Employer "${employerName}" not found in verification database for ${applicantName}`
    );
  }
  if (annualIncome <= 0) {
    throw ApplicationFailure.nonRetryable(
      `Invalid annual income: $${annualIncome} for ${applicantName}`
    );
  }
  return `Income verified: ${applicantName} earns $${annualIncome}/yr at ${employerName}`;
}

// Step 2: Pull credit report using the applicant's SSN
export async function runCreditCheck(
  applicantName: string,
  ssn: string
): Promise<string> {
  // Malformed SSN — no retry will fix this
  if (ssn === '000-00-0000' || ssn.length < 11) {
    throw ApplicationFailure.nonRetryable(
      `Invalid SSN "${ssn}" for ${applicantName} — cannot pull credit report`
    );
  }
  return `Credit check passed for ${applicantName}: score 750`;
}

// Step 3: Order a property appraisal
export async function orderAppraisal(
  propertyAddress: string,
  loanAmount: number
): Promise<string> {
  // Invalid property address
  if (propertyAddress === '' || propertyAddress === 'INVALID_ADDRESS') {
    throw ApplicationFailure.nonRetryable(
      `Cannot order appraisal — invalid property address: "${propertyAddress}"`
    );
  }
  return `Appraisal completed for ${propertyAddress}: valued at $${loanAmount * 1.1}`;
}

// Step 4: Verify the property title is clear
export async function performTitleSearch(
  propertyId: string,
  propertyAddress: string
): Promise<string> {
  // Missing or invalid property ID
  if (propertyId === '' || propertyId === 'MISSING') {
    throw ApplicationFailure.nonRetryable(
      `Title search failed — missing or invalid property ID: "${propertyId}" for ${propertyAddress}`
    );
  }
  return `Title is clear for property ${propertyId} at ${propertyAddress}`;
}

// Step 5: Check debt-to-income ratio against lending policy
export async function underwrite(
  applicantName: string,
  annualIncome: number,
  loanAmount: number,
  downPayment: number
): Promise<string> {
  const dti = ((loanAmount - downPayment) / annualIncome) * 100;
  // DTI above 400% exceeds policy limit — needs a larger down payment or lower loan amount
  if (dti > 400) {
    throw ApplicationFailure.nonRetryable(
      `Underwriting denied for ${applicantName} — debt-to-income ratio ${dti.toFixed(0)}% exceeds 400% limit`
    );
  }
  return `Underwriting approved for ${applicantName}: DTI ${dti.toFixed(0)}%`;
}

// Step 6: Finalize and fund the loan
export async function closeLoan(
  applicationId: string,
  applicantName: string,
  loanAmount: number
): Promise<string> {
  return `Loan ${applicationId} closed for ${applicantName}: $${loanAmount} funded`;
}
```

Each Activity validates its inputs against business rules and throws `ApplicationFailure.nonRetryable()` when the data is fundamentally invalid.
The `nonRetryable` designation is critical: it instructs the Temporal SDK to skip the Retry Policy entirely and propagate the error directly to the Workflow.
This distinguishes failures that require human judgment — corrupt input, policy breaches, compliance blocks — from transient infrastructure failures that retries can resolve.

### Implement the Workflow with the recoverableStep pattern

You will now create the Workflow that orchestrates the six-step pipeline.
The central mechanism is the `recoverableStep` helper function that wraps each Activity call in a pause-and-resume loop.

```typescript
// src/workflows.ts

import {
  proxyActivities,
  defineSignal,
  defineQuery,
  setHandler,
  condition,
  upsertSearchAttributes,
  log,
  ActivityFailure,
  isCancellation,
} from '@temporalio/workflow';
import { defineSearchAttributeKey } from '@temporalio/common';
import type * as activities from './activities';
import type { FixEntry, LoanApplication, LoanState, LoanStatus, RetryUpdate } from './models';

// Typed Search Attribute keys — used in both upsertSearchAttributes and client start options
const LoanStatusKey = defineSearchAttributeKey('LoanStatus', 'KEYWORD');
const FailedActivityKey = defineSearchAttributeKey('FailedActivity', 'KEYWORD');

const {
  verifyIncome,
  runCreditCheck,
  orderAppraisal,
  performTitleSearch,
  underwrite,
  closeLoan,
} = proxyActivities<typeof activities>({
  // In production, split into multiple proxies and tune per Activity; long-running steps
  // should also call heartbeat() with a heartbeatTimeout.
  startToCloseTimeout: '10 seconds',
  // Use the default retry policy for transient failures.
  // ApplicationFailure.nonRetryable() bypasses retries for permanent failures.
});

// Signal to deliver corrected data; Query to read current pipeline state
export const retrySignal = defineSignal<[RetryUpdate]>('retry');
export const getStateQuery = defineQuery<LoanState>('getState');

export async function homeLoanWorkflow(application: LoanApplication): Promise<LoanState> {
  // Mutable copy — patched in-place by the Signal handler when corrections arrive
  const app = { ...application };
  let status: LoanStatus = 'STARTED';
  let failedActivity = '';
  let failureMessage = '';
  let retryRequested = false;
  const completedActivities: string[] = [];
  const fixHistory: FixEntry[] = [];

  // Publish pipeline state as Search Attributes so operators can query across all Workflows
  const updateStatus = (newStatus: LoanStatus, activity = '', message = '') => {
    status = newStatus;
    failedActivity = activity;
    failureMessage = message;
    upsertSearchAttributes([
      { key: LoanStatusKey, value: newStatus },
      { key: FailedActivityKey, value: activity },
    ]);
  };

  // Query handler — returns a snapshot of the full pipeline state without side effects
  setHandler(getStateQuery, () => ({
    status,
    failedActivity,
    failureMessage,
    completedActivities: [...completedActivities],
    fixHistory: [...fixHistory],
    application: { ...app },
  }));

  // Signal handler — patches the application data and wakes the suspended Workflow.
  // Signals are at-least-once: clients can retry on transport hiccups, so dedupe
  // via the caller-supplied id stored on FixEntry.
  setHandler(retrySignal, (update: RetryUpdate) => {
    if (update.id && fixHistory.some((f) => f.id === update.id)) {
      log.warn(`Duplicate retry signal ignored: ${update.id}`);
      return;
    }
    if (update.key) {
      const key = update.key as keyof LoanApplication;
      const oldValue = String((app as any)[key]);
      if (key === 'annualIncome' || key === 'loanAmount' || key === 'downPayment') {
        (app as any)[key] = parseFloat(update.value ?? '0');
      } else {
        (app as any)[key] = update.value ?? '';
      }
      fixHistory.push({
        activity: failedActivity,
        field: key,
        oldValue,
        newValue: update.value ?? '',
        error: failureMessage,
        id: update.id,
      });
      log.info(`Fix received ${key}: ${oldValue} -> ${update.value}`);
    } else {
      log.info('Retry requested without patch');
    }
    retryRequested = true; // unblocks the condition() below
  });

  // Core pattern: wrap each Activity in a pause-and-resume loop.
  // On failure, advertise PENDING_FIX via Search Attributes and suspend
  // until a Signal delivers corrected data, then retry the same Activity.
  const recoverableStep = async <T>(
    activityName: string,
    fn: () => Promise<T>
  ): Promise<T> => {
    while (true) {
      try {
        const result = await fn();
        return result;
      } catch (e) {
        // Anything that isn't an ActivityFailure (workflow-side bug, non-determinism)
        // is treated as a workflow task failure by Temporal and retried — let it surface.
        if (!(e instanceof ActivityFailure)) throw e;
        // Cancellation arrives wrapped as ActivityFailure; propagate so the workflow unwinds.
        if (isCancellation(e)) throw e;
        // Activity errors are wrapped in ActivityFailure; the original message is in .cause
        const message = e.cause?.message || e.message || String(e);
        log.warn(`Activity ${activityName} failed: ${message}`);
        updateStatus('PENDING_FIX', activityName, message);
        retryRequested = false;
        // Suspend the Workflow — no resources consumed while waiting
        await condition(() => retryRequested);
        updateStatus('STARTED', '', '');
        log.info(`Retrying activity ${activityName} after fix`);
      }
    }
  };

  await recoverableStep('verifyIncome', () =>
    verifyIncome(app.applicantName, app.employerName, app.annualIncome)
  );
  completedActivities.push('verifyIncome');
  updateStatus('INCOME_VERIFIED');

  await recoverableStep('runCreditCheck', () =>
    runCreditCheck(app.applicantName, app.ssn)
  );
  completedActivities.push('runCreditCheck');
  updateStatus('CREDIT_CHECKED');

  await recoverableStep('orderAppraisal', () =>
    orderAppraisal(app.propertyAddress, app.loanAmount)
  );
  completedActivities.push('orderAppraisal');
  updateStatus('APPRAISAL_ORDERED');

  await recoverableStep('performTitleSearch', () =>
    performTitleSearch(app.propertyId, app.propertyAddress)
  );
  completedActivities.push('performTitleSearch');
  updateStatus('TITLE_SEARCHED');

  await recoverableStep('underwrite', () =>
    underwrite(app.applicantName, app.annualIncome, app.loanAmount, app.downPayment)
  );
  completedActivities.push('underwrite');
  updateStatus('UNDERWRITTEN');

  await recoverableStep('closeLoan', () =>
    closeLoan(app.applicationId, app.applicantName, app.loanAmount)
  );
  completedActivities.push('closeLoan');
  updateStatus('CLOSED');

  return {
    status,
    failedActivity,
    failureMessage,
    completedActivities: [...completedActivities],
    fixHistory: [...fixHistory],
    application: { ...app },
  };
}
```

The `homeLoanWorkflow` orchestrates the entire loan processing pipeline.
The `recoverableStep` helper function wraps each Activity call in a `while (true)` loop.
When an Activity throws, the helper updates the `LoanStatus` Search Attribute to `PENDING_FIX` and the `FailedActivity` Search Attribute to the name of the failed Activity.
It then calls `await condition(() => retryRequested)`, which suspends the Workflow until a Signal delivers corrected data.

The catch block narrows the error before suspending. It first re-throws anything that is not an `ActivityFailure` — a Workflow-side bug or a non-determinism error should fail the Workflow Task and let Temporal retry it, not be silently parked in `PENDING_FIX`. It then re-throws on `isCancellation(e)`, since cancellation arrives wrapped as an `ActivityFailure` and must propagate so the framework can unwind the Workflow cleanly.

The Signal handler receives a `RetryUpdate` containing the field name and corrected value.
It patches the `app` object in-place, records the correction in `fixHistory`, and sets `retryRequested = true` to wake the suspended `condition()`.
The loop then retries the Activity with the patched data.
If the Activity succeeds, the loop exits and the pipeline advances.
If it fails again, the cycle repeats.

Temporal delivers Signals with at-least-once semantics: a client retry, a flaky network, or an over-eager UI can submit the same correction twice and the Workflow will see both copies.
Without protection, the second delivery would overwrite the field again — usually a no-op, but harmful when the operator has since sent a different value, and always noisy in `fixHistory`.
The handler defends against this by checking the caller-supplied `id` against the audit trail and dropping anything it has already applied.
The client (the dashboard, CLI script, or upstream agent) is responsible for generating a stable `id` per logical correction — typically a UUID minted when the operator clicks **Patch and Retry**, reused across any retries of that same submission.
[Workflow Updates](/develop/typescript/workflows/message-passing#updates) would give you this deduplication for free via the SDK's built-in `updateId` handling, along with a synchronous result to the caller; this pattern sticks with Signals because the dashboard is fire-and-forget and the audit trail already provides the dedup index.

The Query handler returns the complete `LoanState` at any point during execution.
This includes the current status, the list of completed Activities, the full fix history, and the current application data.
External systems can poll this Query to display real-time pipeline progress.

By calling `upsertSearchAttributes` at every State Transition, the Workflow maintains a denormalized index that the Temporal Visibility API can query across all active Workflows.
This enables an operations dashboard to display aggregate statistics and filter by any combination of status and failed Activity.

### Add saga compensation for side-effecting steps

Not every failure can be resolved by correcting input data.
An OFAC match, a withdrawn offer, or a regulatory hold demands that the pipeline roll back rather than pause for a fix.
This is where the [saga pattern](/design-patterns/saga-pattern) comes in: each forward Activity that produces an external side effect — a credit bureau inquiry, an appraiser booking, a title company fee, a reserved lending slot, a funded loan — registers a compensating Activity **before** it executes.
If the forward pipeline aborts, the Workflow unwinds the registered compensations in reverse order.

The pattern has three key discipline points that the implementation must honor:

1. **Register before execution.** Compensations are registered before the forward call to handle partial failures. Consider `orderAppraisal`: the Worker POSTs a booking to the appraisal vendor, the vendor records the booking and reserves the fee, then the response is lost to a network blip on the way back. The Activity throws because it never received a response — yet the booking exists on the vendor side. If the Workflow only registered the compensation after a successful forward call, that booking would be orphaned by a saga rollback. Pre-registering guarantees `cancelAppraisal` runs during rollback either way; idempotency makes it a safe no-op when the side effect never actually landed.
2. **LIFO unwinding.** The last step to touch external state is the first to undo. In TypeScript this is expressed by pushing onto the front of the array with `unshift()` and iterating forward, or by iterating a `push()`-built array in reverse.
3. **Recoverable compensation.** Compensations can fail — vendor APIs go down, external systems reject requests. Running each compensation through the same recoverable wrapper as forward steps means a stuck rollback pauses with `ROLLBACK_PENDING_FIX` and can be patched or retried by an operator.

Extend the Workflow to register compensations alongside forward Activities:

```typescript
interface Compensation {
  forwardActivity: string;
  compensationActivity: string;
  run: () => Promise<string>;
}
const compensations: Compensation[] = [];

const runForward = async <T>(
  activityName: string,
  forward: () => Promise<T>,
  compensation?: { name: string; fn: () => Promise<string> }
): Promise<T> => {
  if (cancelRequested) {
    throw new Error(`Cancelled before ${activityName}: ${cancelReason}`);
  }
  // Register BEFORE execution — handles partial side effects if the Activity aborts mid-flight
  if (compensation) {
    compensations.unshift({
      forwardActivity: activityName,
      compensationActivity: compensation.name,
      run: compensation.fn,
    });
  }
  return recoverableStep(activityName, forward, 'forward');
};
```

Two paths now trigger a rollback:

**Explicit cancellation.** A `cancelApplication` Signal sets `cancelRequested = true` and wakes any paused `condition()`. The `recoverableStep` loop checks this flag on wake-up and throws out of the pause to the outer `try/catch`.

**RollbackRequired failure type.** An Activity can throw `ApplicationFailure.nonRetryable(message, 'RollbackRequired')` to signal that no data correction will help — the application must be withdrawn. The Workflow catches this specific failure type and routes it to the compensation phase instead of the pause loop:

```typescript
export async function underwrite(
  applicantName: string, ssn: string, ...
): Promise<string> {
  if (ssn.startsWith('999')) {
    throw ApplicationFailure.nonRetryable(
      `Compliance block: OFAC/sanctions match — application must be withdrawn`,
      'RollbackRequired'
    );
  }
  // ... normal DTI check
}
```

The compensation loop runs in LIFO order through the same recoverable wrapper:

```typescript
try {
  // ... forward pipeline: verifyIncome, runCreditCheck, orderAppraisal, ...
} catch (err: any) {
  const trigger = cancelReason || err.message || String(err);
  updateStatus('COMPENSATING', '', trigger);

  for (const comp of compensations) {
    // Safe even without this check thanks to idempotency, but skipping keeps the audit clean
    if (!completedActivities.includes(comp.forwardActivity)) continue;

    const result = await recoverableStep(comp.forwardActivity, comp.run, 'compensation');
    compensationHistory.push({
      forwardActivity: comp.forwardActivity,
      compensationActivity: comp.compensationActivity,
      result,
    });
    compensatedActivities.push(comp.forwardActivity);
    updateStatus('COMPENSATING');
  }

  updateStatus('ROLLED_BACK', '', trigger);
}
```

Not every step needs a compensation.
`verifyIncome` is a read-only lookup against the employer verification database — there is no external state to undo.
`runCreditCheck` records a hard inquiry that lowers the applicant's score, so its compensation submits a withdrawal request.
`orderAppraisal` books an appraiser and charges a fee — the compensation cancels the booking and issues a partial refund.
`performTitleSearch` pays the title company and places a placeholder hold — the compensation releases the hold.
`underwrite` reserves lending capacity against portfolio limits — the compensation returns the capacity to the pool.
`closeLoan` is the most consequential: funds are disbursed and a lien is recorded at the county — the compensation initiates a clawback and files the lien release.

Each compensation Activity is intentionally written to be idempotent.
`withdrawCreditInquiry` simply resubmits the withdrawal request; the bureau accepts duplicates.
`cancelAppraisal` checks for an existing booking before cancelling.
`releaseTitleHold` is safe to call multiple times on an already-released hold.
This means the "register before execution" discipline cannot corrupt state even if the forward Activity never produced the side effect.

After the compensations finish, a `notifyApplicantCancelled` Activity runs to inform the applicant that the application was withdrawn and to surface the trigger reason.
This step runs through the same recoverable wrapper as compensations: if the email provider is down, the Workflow pauses with `ROLLBACK_PENDING_FIX` so an operator can retry once the provider recovers rather than leaving the applicant uninformed.

### Implement the web service

Connect the Workflow to your application's API layer.
You will write the code that receives incoming requests, starts Workflows, queries state, and sends corrective Signals.

```typescript
// src/web-service.ts

import express from 'express';
import { Connection, Client } from '@temporalio/client';
import { defineSearchAttributeKey } from '@temporalio/common';
import { homeLoanWorkflow, retrySignal, getStateQuery } from './workflows';
import type { LoanApplication, RetryUpdate, LoanState } from './models';

const LoanStatusKey = defineSearchAttributeKey('LoanStatus', 'KEYWORD');
const FailedActivityKey = defineSearchAttributeKey('FailedActivity', 'KEYWORD');

async function run() {
  const connection = await Connection.connect({ address: 'localhost:7233' });
  const client = new Client({ connection });

  const app = express();
  app.use(express.json());

  app.get('/api/workflows', async (_req, res) => {
    try {
      const workflows: any[] = [];
      const iterator = client.workflow.list({
        query: `TaskQueue = 'recoverable-activity' AND ExecutionStatus != 'Terminated'`,
      });
      for await (const wf of iterator) {
        const entry: any = {
          workflowId: wf.workflowId,
          wfStatus: wf.status.name,
          // Read Search Attributes from the Visibility store (eventually consistent)
          loanStatus: wf.searchAttributes?.LoanStatus?.[0] ?? '',
          failedActivity: wf.searchAttributes?.FailedActivity?.[0] ?? '',
        };
        if (wf.status.name === 'RUNNING') {
          const handle = client.workflow.getHandle(wf.workflowId);
          // Query returns the strongly consistent internal state from the Workflow
          entry.state = await handle.query(getStateQuery);
        }
        workflows.push(entry);
      }
      res.json({ workflows });
    } catch (error) {
      res.status(500).json({ error: (error as Error).message });
    }
  });

  app.get('/api/workflows/search', async (req, res) => {
    try {
      const { failedActivity, status } = req.query;
      const clauses = [`TaskQueue = 'recoverable-activity'`];
      if (failedActivity) {
        clauses.push(`FailedActivity = '${failedActivity}'`);
      }
      if (status) {
        clauses.push(`LoanStatus = '${status}'`);
      }
      const workflows: any[] = [];
      const iterator = client.workflow.list({ query: clauses.join(' AND ') });
      for await (const wf of iterator) {
        workflows.push({
          workflowId: wf.workflowId,
          loanStatus: wf.searchAttributes?.LoanStatus?.[0] ?? '',
          failedActivity: wf.searchAttributes?.FailedActivity?.[0] ?? '',
        });
      }
      res.json({ workflows });
    } catch (error) {
      res.status(500).json({ error: (error as Error).message });
    }
  });

  app.post('/api/workflows', async (req, res) => {
    try {
      const application = req.body as LoanApplication;
      const handle = await client.workflow.start(homeLoanWorkflow, {
        taskQueue: 'recoverable-activity',
        workflowId: application.applicationId,
        args: [application],
        // Set initial Search Attributes so the Workflow is queryable from the start
        typedSearchAttributes: [
          { key: LoanStatusKey, value: 'STARTED' },
          { key: FailedActivityKey, value: '' },
        ],
      });
      res.json({ success: true, workflowId: handle.workflowId });
    } catch (error) {
      res.status(500).json({ error: (error as Error).message });
    }
  });

  app.post('/api/workflows/:workflowId/fix', async (req, res) => {
    try {
      const { key, value } = req.body as RetryUpdate;
      // getHandle creates a reference to a running Workflow — no server call yet
      const handle = client.workflow.getHandle(req.params.workflowId);
      // signal() delivers the correction asynchronously; the Workflow wakes and retries
      await handle.signal(retrySignal, { key, value });
      res.json({ success: true, message: `Fix sent: ${key} = ${value}` });
    } catch (error) {
      res.status(500).json({ error: (error as Error).message });
    }
  });

  app.listen(3000, () => {
    console.log('Recoverable Activity UI running on http://localhost:3000');
  });
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

The search endpoint demonstrates how custom Search Attributes enable routing to human operators or automated agents for specific failure categories.

### Create the Temporal Worker

Now that you have built the Activities and Workflow, here is the complete Worker setup required to register and run the system.

```typescript
// src/worker.ts

import { NativeConnection, Worker } from '@temporalio/worker';
import * as activities from './activities';

async function run() {
  const connection = await NativeConnection.connect({ address: 'localhost:7233' });

  const worker = await Worker.create({
    connection,
    namespace: 'default',
    taskQueue: 'recoverable-activity',
    workflowsPath: require.resolve('./workflows'),
    activities,
  });

  console.log('Worker started, ctrl+c to exit');
  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

The `taskQueue` value must match the Task Queue used when starting Workflows from the client.

Run the following commands sequentially, each in their own terminal:

```bash
temporal server start-dev
```

```bash
npm start
```

### Launch the demo dashboard

The project includes a single-page operations dashboard at [`public/index.html`](https://github.com/temporal-sa/validated-pattern-keep-business-moving/blob/main/public/index.html) that exercises the web service endpoints you built above. Start it alongside the Worker:

```bash
npm run web
```

Open `http://localhost:3000` in your browser. The dashboard provides:

- **Stats bar** — live counts of Total, Pending Fix, Running, and Completed Workflows.
- **Filters** — narrow by failed Activity or loan status, powered by the Search Attribute Query endpoint.
- **Pipeline visualization** — per-Workflow step indicators showing which Activities have completed, which failed, and which are pending.
- **Detail panel** — click any Workflow to inspect its full state (via the Query API), view the error message, and send a corrective Signal through a form that pre-suggests which field to fix.
- **Fix history** — an audit trail of every correction applied during the lifecycle of the Workflow.

The dashboard polls the web service every 3 seconds and refreshes the detail panel in real time after you send a fix, so you can watch the pipeline advance step by step.

### Test the Workflow execution

To verify the pipeline and the pause-and-resume cycle, you must submit loan applications with known data errors and business rule violations.
You will write a client script that starts Workflows covering single-issue and multi-issue failure scenarios.

```typescript
// src/client.ts

import { Connection, Client } from '@temporalio/client';
import { defineSearchAttributeKey } from '@temporalio/common';
import { homeLoanWorkflow } from './workflows';
import type { LoanApplication } from './models';

const LoanStatusKey = defineSearchAttributeKey('LoanStatus', 'KEYWORD');
const FailedActivityKey = defineSearchAttributeKey('FailedActivity', 'KEYWORD');

const scenarios: { name: string; application: LoanApplication }[] = [
  {
    name: 'Clean — all activities pass',
    application: {
      applicationId: 'LOAN-001',
      applicantName: 'Alice Johnson',
      ssn: '123-45-6789',
      employerName: 'Acme Corp',
      annualIncome: 120000,
      propertyAddress: '123 Oak St, Springfield',
      propertyId: 'PROP-001',
      loanAmount: 350000,
      downPayment: 70000,
    },
  },
  {
    name: 'Bad SSN — credit check fails',
    application: {
      applicationId: 'LOAN-002',
      applicantName: 'Bob Smith',
      ssn: '000-00-0000',
      employerName: 'TechCo',
      annualIncome: 95000,
      propertyAddress: '456 Elm Ave, Shelbyville',
      propertyId: 'PROP-002',
      loanAmount: 280000,
      downPayment: 56000,
    },
  },
  {
    name: 'Invalid address — appraisal fails',
    application: {
      applicationId: 'LOAN-003',
      applicantName: 'Carol Davis',
      ssn: '987-65-4321',
      employerName: 'HealthPlus',
      annualIncome: 105000,
      propertyAddress: 'INVALID_ADDRESS',
      propertyId: 'PROP-003',
      loanAmount: 320000,
      downPayment: 64000,
    },
  },
  {
    name: 'Multi-issue — bad employer + bad SSN + invalid address + high DTI',
    application: {
      applicationId: 'LOAN-009',
      applicantName: 'Irene Tanaka',
      ssn: '000-00-0000',
      employerName: 'UNKNOWN_EMPLOYER',
      annualIncome: 40000,
      propertyAddress: 'INVALID_ADDRESS',
      propertyId: 'PROP-009',
      loanAmount: 600000,
      downPayment: 5000,
    },
  },
];

async function run() {
  const connection = await Connection.connect({ address: 'localhost:7233' });
  const client = new Client({ connection });

  for (const scenario of scenarios) {
    const handle = await client.workflow.start(homeLoanWorkflow, {
      taskQueue: 'recoverable-activity',
      workflowId: scenario.application.applicationId,
      args: [scenario.application],
      typedSearchAttributes: [
        { key: LoanStatusKey, value: 'STARTED' },
        { key: FailedActivityKey, value: '' },
      ],
    });
    console.log(`Started: ${handle.workflowId} — ${scenario.name}`);
  }
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Execute the client from your command line:

```bash
npm run workflow
```

The script will complete and print the confirmation for each Workflow:

```
Started: LOAN-001 — Clean — all activities pass
Started: LOAN-002 — Bad SSN — credit check fails
Started: LOAN-003 — Invalid address — appraisal fails
Started: LOAN-009 — Multi-issue — bad employer + bad SSN + invalid address + high DTI
```

After this output appears, `LOAN-001` completes without pausing because all data is valid.
`LOAN-002` pauses at `runCreditCheck` because of the invalid Social Security number (SSN).
`LOAN-003` pauses at `orderAppraisal` because of the invalid address.
`LOAN-009` pauses at `verifyIncome` because of the unknown employer, and will require four separate corrections across four Activities to complete.

### Send a corrective Signal

To resume a paused Workflow, send a Signal containing the field name and corrected value.
You can do this through the demo UI, the web service API, or the Temporal CLI.

#### Using the demo UI

The dashboard shows all Workflows and their pipeline status. Click a blocked Workflow to inspect the failure details and send a correction.

![Dashboard showing 3 blocked Workflows and 1 completed](https://docs.temporal.io/guides/assets/recover-without-restart/keep-business-moving-list-view.png)

Click a `PENDING_FIX` Workflow to open its detail panel. The UI shows the failure message and suggests a fix. Select the field to patch, enter the corrected value, and click **Patch and Retry** to send the Signal.

![Sending a corrective Signal to fix an unknown employer](https://docs.temporal.io/guides/assets/recover-without-restart/keep-business-moving-fix-employer.png)

After the Signal is delivered, the Workflow wakes up, retries the failed Activity with the corrected data, and advances through the remaining pipeline steps. The detail panel shows the updated state including the fix history.

![Workflow completed after fixing the employer name](https://docs.temporal.io/guides/assets/recover-without-restart/keep-business-moving-success.png)

You can also verify the full Event History — including the Signal and subsequent Activity completions — in the Temporal Web UI.

![Event History in the Temporal Web UI showing the Signal and Activity completions](https://docs.temporal.io/guides/assets/recover-without-restart/keep-business-moving-event-history.png)

#### Using the Temporal CLI

```bash
temporal workflow signal \
  --workflow-id LOAN-002 \
  --name retry \
  --input '{"key":"ssn","value":"222-33-4444"}'
```

#### Using the web service API

```bash
curl -X POST http://localhost:3000/api/workflows/LOAN-002/fix \
  -H 'Content-Type: application/json' \
  -d '{"key":"ssn","value":"222-33-4444"}'
```

### Query pipeline status using Search Attributes

Use the Temporal CLI or the web service to query blocked Workflows across the entire pipeline.

Find all Workflows waiting for a fix:

```bash
temporal workflow list --query "LoanStatus = 'PENDING_FIX'"
```

Find all Workflows blocked on credit checks:

```bash
temporal workflow list \
  --query "LoanStatus = 'PENDING_FIX' AND FailedActivity = 'runCreditCheck'"
```

Using the web service search endpoint:

```bash
curl "http://localhost:3000/api/workflows/search?failedActivity=runCreditCheck&status=PENDING_FIX"
```

These Queries return results in real time as Workflows update their Search Attributes.
An operations dashboard can poll these endpoints to display aggregate statistics: the number of applications awaiting correction, which Activities cause the most failures, and how long applications sit in `PENDING_FIX` status.

This query capability supports automated resolution agents.
An AI agent can periodically query for Workflows blocked on a specific Activity, fetch the error message and application data through the Query API, apply a correction, and send the Signal.
The same Search Attribute interface that routes work to human operators also routes work to automated agents.

### Query individual Workflow state

Use the Query API to inspect the full internal state of any running Workflow without affecting its execution.

```bash
temporal workflow query \
  --workflow-id LOAN-009 \
  --name getState
```

The Query returns the complete `LoanState` object:

```json
{
  "status": "PENDING_FIX",
  "failedActivity": "verifyIncome",
  "failureMessage": "Employer \"UNKNOWN_EMPLOYER\" not found in verification database for Irene Tanaka",
  "completedActivities": [],
  "fixHistory": [],
  "application": {
    "applicationId": "LOAN-009",
    "applicantName": "Irene Tanaka",
    "ssn": "000-00-0000",
    "employerName": "UNKNOWN_EMPLOYER",
    "annualIncome": 40000,
    "propertyAddress": "INVALID_ADDRESS",
    "propertyId": "PROP-009",
    "loanAmount": 600000,
    "downPayment": 5000
  }
}
```

## Outcomes

By following this guide, you have implemented a recoverable and compensatable pipeline including:

- **Durable pause on permanent failure.** Activities throw `ApplicationFailure.nonRetryable()` to signal non-retryable failures — whether invalid data, a breached lending limit, or a blocked compliance check. The Workflow catches these errors and suspends using `await condition()`, consuming no Worker resources while waiting for correction. The Workflow resumes where it left off regardless of how much time passes.
- **Signal-driven recovery.** External operators or automated agents send corrective Signals containing the field name and new value. The Signal handler patches the application data in-place, records the correction in an auditable fix history, and wakes the Workflow to retry the failed Activity.
- **Saga compensation on abort.** When a `RollbackRequired` failure or a `cancelApplication` Signal aborts the forward pipeline, the Workflow unwinds registered compensations in LIFO order. Compensations are registered before execution and written to be idempotent so partial side effects are always cleaned up. Each compensation runs through the same recoverable wrapper, so a stuck rollback pauses with `ROLLBACK_PENDING_FIX` for operator intervention rather than leaving the process half-unwound.
- **Search Attribute routing.** Custom Search Attributes `LoanStatus` and `FailedActivity` are updated at every State Transition, creating a real-time queryable index across all active Workflows. Operations teams use visibility Queries to filter blocked Workflows by failure type and route them to the appropriate resolution resource, be it human or agents.
- **Full pipeline visibility.** Queries return the complete internal state of any running Workflow, including completed Activities, compensated Activities, fix history, compensation history, and the triggering cancellation reason. Search Attributes provide the denormalized aggregate view across all Workflows. Together, these mechanisms give you both the individual and cross-pipeline views needed to operate the system at scale.

## Related resources
- [Source code](https://github.com/temporal-sa/validated-pattern-keep-business-moving)
- [Saga Pattern](/design-patterns/saga-pattern) — pattern catalog reference for the LIFO compensation technique used to roll back the loan pipeline
- [Resumable Activity](/design-patterns/resumable-activity) — pattern catalog reference for the pause-on-failure, resume-via-Signal technique behind `recoverableStep`
- [Non-Retryable Errors](/design-patterns/non-retryable-errors) — pattern catalog reference for marking permanent failures so Temporal fails fast instead of retrying
- [Temporal TypeScript SDK Documentation](/develop/typescript) — Complete reference for building Workflows, Activities, and Workers with the TypeScript SDK.
- [Temporal Signals](/develop/typescript/workflows/message-passing#signals) — Guide to defining and sending Signals to running Workflow Executions.
- [Temporal Queries](/develop/typescript/workflows/message-passing#queries) — Guide to defining and handling synchronous read-only Queries on Workflows.
- [Temporal Search Attributes](/search-attribute) — Reference for creating and using custom Search Attributes for Workflow visibility.
- [Temporal Visibility](/visibility) — Overview of the Visibility subsystem and List Filter Query syntax.
- [ApplicationFailure Reference](/references/failures#application-failure) — API reference for `ApplicationFailure` including retryable and non-retryable error handling.

---

# Build a human-in-the-loop document approval Workflow

Source: https://docs.temporal.io/guides/reliable-document-approvals

> Wait durably for reviewer decisions, enforce SLAs with Timers, escalate to a backup approver, handle resubmissions, and keep an audit trail, in Python.

**by Cecil Phillip**

Document approval processes fail in predictable ways: requests go unanswered, deadlines pass silently, context gets lost across restarts, and audit trails end up incomplete. The usual fixes — polling loops, cron jobs, reminder emails kicked off by a scheduler — create coordination debt that compounds over time. This guide shows how to build an approval system where none of that is possible: decisions are waited on durably, SLAs are enforced automatically, escalations happen without a scheduler, and every action is recorded whether or not your infrastructure stays up.

### Problem statement

An approval request may not get a response until the following week — or ever. Between submission and decision, a lot can go wrong: the request gets buried in an inbox, the original approver is out, the system restarts and loses state, or nobody notices the deadline passed. Patching this with a database, a cron job, a notification service, and custom reconciliation logic means maintaining four systems to do one thing.

### Solution

The approval process runs as a single Workflow that holds its own state and waits — without consuming compute — for as long as it takes. Deadlines are tracked by durable timers that fire whether or not Workers restart. Reminders go out on schedule. If no one responds in time, escalation happens automatically. Resubmissions loop back cleanly. A complete audit record is written at every step. The infrastructure can fail and recover; the approval process continues from exactly where it left off.

### Outcomes

After working through this guide, you'll have a running approval system where:

- **Requests don't disappear when infrastructure does**: Temporal's Event History replays the Workflow to its exact pre-failure state, so no approval decisions or audit entries are ever lost.
- **Deadlines enforce themselves**: Durable timers track every SLA and fire precisely when a deadline expires — no external scheduler, no polling loop, no cron job.
- **Escalations happen automatically when reviewers go quiet**: If an approver doesn't respond within the SLA window, the Workflow escalates to a backup approver without any external trigger.
- **Resubmissions work without restarting the process from scratch**: Change requests loop back cleanly, preserving full context and the existing audit trail.
- **Every action is in the audit record**: Decisions, reminders, escalations, and status changes are all recorded — whether from a Signal, an Activity, or a timer firing.

## Background and best practices

### Why human-in-the-loop matters

Some decisions can't be automated — a contract requiring legal sign-off, a patient record needing clinical approval, an AI-generated document awaiting human review. A system that drops a request, misses a deadline, or produces an incomplete audit trail creates real business and legal exposure.

### How Temporal enables the "wait" state

When a Workflow calls `workflow.wait_condition()`, **the Worker returns the current task to the Temporal Server and becomes idle** — consuming no compute. The Workflow's state is persisted and suspended on the Server. When a Signal arrives or a timeout fires, the Server schedules a new Workflow Task, the Worker replays the Event History to reconstruct the Workflow's state, and execution resumes from the `wait_condition` call. This mechanism works identically whether the wait is five seconds or five months — durable timers are persisted to the Server's database and survive Worker restarts, deployments, and infrastructure migrations.

### Event History management

Temporal's Event History has a hard limit of 51,200 events or 50 MB ([Workflow Execution limits](/workflow-execution/limits)). The Server emits warnings at ~10,240 events or 10 MB; hitting either hard limit terminates the Workflow. For this [approval pattern](/design-patterns/approval):

| Workflow operation | Approximate events generated |
|--------------------|------------------------------|
| Workflow start (baseline) | 3 events |
| Single Activity execution | ~8 events |
| Timer (`asyncio.sleep` / `workflow.sleep`) | ~4 events |
| Signal received | ~4 events |
| Query received | 0 events (not recorded) |
| Continue-As-New | 1 event on the old run; fresh history on the new run |

A single approval with reminders and escalation generates roughly 40–80 events — well within limits. If your process supports resubmission loops or runs for months, use `workflow.info().is_continue_as_new_suggested()` to detect when the Server recommends a Continue-As-New, and act on it at a safe boundary.

### Workflow Definition determinism

The Workflow Definition must be deterministic because Temporal uses *history replay* to reconstruct state after Worker restarts. If the code generates different commands on replay, the SDK raises a `NondeterminismError`.

Key rules:

- Use `workflow.now()` instead of `datetime.now()` for timestamps
- Use `workflow.uuid4()` instead of `uuid.uuid4()` for generating unique identifiers
- Use `workflow.random()` instead of Python's `random` module
- Never perform I/O (network calls, file reads, database queries) directly in Workflow code — delegate to Activities
- Use `workflow.logger` instead of `print()` for replay-safe logging

### Activity idempotency

Activities may be retried if a Worker crashes mid-execution. Each Activity must be designed so that executing it twice with the same inputs produces the same outcome:

- **Idempotency keys**: Include the Workflow Id and a step identifier in every external API call.
- **Check-before-act**: Query the external system's state before performing a mutation.
- **Upsert semantics**: Use database upsert operations for audit log entries keyed on Workflow Id and approval stage.

### Long running Activities and Heartbeating

For Activities that may take minutes (PDF report generation, rate-limited API calls):

- Set `start_to_close_timeout` to the maximum time for a single attempt.
- Set `heartbeat_timeout` to the maximum acceptable interval between progress reports. If the Worker crashes, the Server retries on another Worker within this window.
- Call `activity.heartbeat()` periodically with a progress payload. On retry, read `activity.info().heartbeat_details` to resume from where it left off.

## Prerequisites

### Required software and infrastructure

- Python **3.11** or later
- Temporal Python SDK (`temporalio`) **1.7.0** or later
- A running Temporal Server — either the [Temporal CLI](/cli) development server (`temporal server start-dev`) or [Temporal Cloud](https://temporal.io/cloud)
- `uv` or `pip` for Python dependency management

### Resources and access privileges

- (Optional) SMTP server or third-party notification API credentials for the notification Activity

### Required concepts

Readers should be familiar with the following concepts:

- [Signals](/develop/python/workflows/message-passing#signals) — the mechanism for sending data to a running Workflow
- [Queries](/develop/python/workflows/message-passing#queries) — the mechanism for reading Workflow state without affecting execution
- [Workers](/workers) — the processes that execute Workflow and Activity code
- [Task Queues](/task-queue) — the named queues that route work to Workers
- Python `asyncio` — coroutines, `await`, and the event loop
- Python dataclasses or [Pydantic](https://docs.pydantic.dev/) for structured data

## Architecture diagram

The following sequence diagram illustrates the complete lifecycle of a document approval Workflow, from submission through approval to final disposition.

```mermaid
sequenceDiagram
    participant Submitter as Document Submitter
    participant API as Web API / Client
    participant TS as Temporal Server
    participant W as Worker (Python)
    participant NS as Notification Service
    participant AuditDB as Audit Log Store

    Submitter->>API: Submit document for approval
    API->>TS: start_workflow(DocumentApprovalWorkflow, document)
    TS->>W: Schedule Workflow Task
    W->>TS: Schedule Activity: store_document
    TS->>W: Execute store_document Activity
    W-->>TS: Document stored
    W->>TS: Schedule Activity: send_notification (to approver)
    TS->>W: Execute send_notification Activity
    W->>NS: Send approval request email
    NS-->>W: Notification sent
    W-->>TS: Workflow waiting for Signal (approval decision)

    Note over TS,W: Workflow is suspended. Zero compute consumed.

    alt Approver responds within SLA
        Submitter->>API: Approve document
        API->>TS: signal(workflow_id, "submit_decision", APPROVED)
        TS->>W: Deliver Signal + Schedule Workflow Task
        W->>TS: Schedule Activity: record_audit_entry
        TS->>W: Execute record_audit_entry Activity
        W->>AuditDB: Write audit entry
        W->>TS: Schedule Activity: generate_approval_report
        Note over W: Generate final report and complete...
    else SLA deadline expires
        TS->>W: Timer fires, Workflow Task scheduled
        W->>W: Handle TimeoutError — escalate
        W->>TS: Schedule Activity: send_notification (to escalation contact)
    else Approver rejects
        Submitter->>API: Reject document
        API->>TS: signal(workflow_id, "submit_decision", REJECTED)
        TS->>W: Deliver Signal
        W->>TS: Schedule Activity: record_audit_entry
        W->>TS: Schedule Activity: send_notification (rejection notice)
        W-->>TS: Workflow completes with REJECTED status
    end
```

## Implementation plan

The following phases walk through the complete implementation. Each phase builds on the previous one.

### Phase 1: Define the data models

Start by defining the data structures that represent documents, approval decisions, and the state that flows through the Workflow. Using Python dataclasses keeps the models serializable by Temporal's default JSON data converter without additional configuration.

Create a file named `models.py`:

```python
"""Data models for the document approval system."""

from __future__ import annotations

import enum
from dataclasses import dataclass, field
from datetime import datetime

class ApprovalStatus(str, enum.Enum):
    """Possible statuses for an approval decision."""

    PENDING = "PENDING"
    APPROVED = "APPROVED"
    REJECTED = "REJECTED"
    ESCALATED = "ESCALATED"
    TIMED_OUT = "TIMED_OUT"
    CHANGES_REQUESTED = "CHANGES_REQUESTED"

class DocumentStatus(str, enum.Enum):
    """Overall status of the document approval process."""

    SUBMITTED = "SUBMITTED"
    IN_REVIEW = "IN_REVIEW"
    APPROVED = "APPROVED"
    REJECTED = "REJECTED"
    WITHDRAWN = "WITHDRAWN"

@dataclass
class ApproverConfig:
    """Configuration for the approver."""

    approver_email: str
    approver_name: str
    sla_seconds: int  # Maximum time to wait for the approver's decision
    escalation_email: str | None = None  # Backup approver if SLA expires
    reminder_interval_seconds: int = 86400  # Send reminders every 24 hours
    max_reminders: int = 3  # Maximum number of reminders to send
    resubmission_timeout_seconds: int = 604800  # Time to allow resubmission after changes requested (default 7 days)

@dataclass
class ApprovalDecision:
    """A decision submitted by an approver."""

    approver_email: str
    status: ApprovalStatus
    comment: str = ""
    decided_at: str = ""  # ISO 8601 timestamp, set by the Workflow

@dataclass
class AuditEntry:
    """A single entry in the approval audit trail."""

    timestamp: str  # ISO 8601
    workflow_id: str
    action: str
    actor: str
    details: str = ""
    approval_level: int = 0

@dataclass
class DocumentSubmission:
    """Input to the document approval Workflow."""

    document_id: str
    title: str
    submitter_email: str
    submitter_name: str
    content_url: str  # Reference to the stored document (Claim Check pattern)
    approver: ApproverConfig = field(default_factory=lambda: ApproverConfig(
        approver_email="",
        approver_name="",
        sla_seconds=86400,
    ))
    metadata: dict[str, str] = field(default_factory=dict)

@dataclass
class ApprovalState:
    """Mutable state carried through the approval process and passed to Continue-As-New."""

    document: DocumentSubmission
    status: DocumentStatus = DocumentStatus.SUBMITTED
    decision: ApprovalDecision | None = None
    audit_trail: list[AuditEntry] = field(default_factory=list)
    resubmission_count: int = 0
    max_resubmissions: int = 3
    document_stored: bool = False
```

The `ApprovalState` dataclass carries everything the Workflow needs to resume after a Continue-As-New.

### Phase 2: Define the Activities

Create a file named `activities.py`:

```python
"""Activities for the document approval system.

Each Activity performs a side-effecting operation and is designed to be idempotent.
"""

from __future__ import annotations

from dataclasses import dataclass
from datetime import timedelta

from temporalio import activity
from temporalio.exceptions import ApplicationError

from models import AuditEntry, DocumentSubmission

@dataclass
class NotificationRequest:
    """Input for the send_notification Activity."""

    recipient_email: str
    recipient_name: str
    subject: str
    body: str
    idempotency_key: str  # Prevents duplicate sends on retry

@dataclass
class StoreDocumentRequest:
    """Input for the store_document Activity."""

    document_id: str
    title: str
    content_url: str
    submitter_email: str

@activity.defn
def send_notification(request: NotificationRequest) -> bool:
    """Send a notification; raises ApplicationError for invalid input (non-retryable)."""
    # Validate input — non-retryable business logic error
    if not request.recipient_email or "@" not in request.recipient_email:
        raise ApplicationError(
            f"Invalid recipient email: {request.recipient_email}"
        )

    if not request.subject or not request.subject.strip():
        raise ApplicationError("Notification subject cannot be empty")

    activity.logger.info(
        "Sending notification",
        extra={
            "recipient": request.recipient_email,
            "subject": request.subject,
            "idempotency_key": request.idempotency_key,
        },
    )

    # --- Add your notification provider ----

    activity.logger.info(
        "Notification sent successfully",
        extra={"recipient": request.recipient_email},
    )
    return True

@activity.defn
def record_audit_entry(entry: AuditEntry) -> bool:
    """Write an audit entry; uses upsert semantics to prevent duplicates on retry."""
    activity.logger.info(
        "Recording audit entry",
        extra={
            "workflow_id": entry.workflow_id,
            "action": entry.action,
            "actor": entry.actor,
            "level": entry.approval_level,
        },
    )

    # --- Add your audit log store ---

    return True

@activity.defn
def store_document(request: StoreDocumentRequest) -> str:
    """Persist document metadata; idempotent — storing the same document_id twice overwrites with identical data."""
    activity.logger.info(
        "Storing document",
        extra={
            "document_id": request.document_id,
            "title": request.title,
        },
    )

    # --- Add your document store ---

    return f"doc-store://{request.document_id}"

@activity.defn
def generate_approval_report(
    document_id: str,
    decisions: list[dict],
) -> str:
    """Generate an approval summary report; uses heartbeating to support long runs."""
    activity.logger.info(
        "Generating approval report",
        extra={"document_id": document_id, "decision_count": len(decisions)},
    )

    total_steps = len(decisions)
    for i, decision in enumerate(decisions):
        # Simulate report generation work for each decision
        activity.logger.info(
            f"Processing decision {i + 1}/{total_steps}",
            extra={"approver": decision.get("approver_email", "unknown")},
        )

        # Heartbeat with progress so the server can detect Worker crashes.
        activity.heartbeat(i + 1)

    report_url = f"reports://{document_id}/approval-summary"
    activity.logger.info(
        "Approval report generated",
        extra={"document_id": document_id, "report_url": report_url},
    )
    return report_url
```

### Phase 3: Define the approval Workflow

Create a file named `workflows.py`:

```python
"""Document approval Workflow with SLA enforcement, escalation, resubmission handling, and audit logging."""

from __future__ import annotations

import asyncio
from dataclasses import asdict
from datetime import timedelta

from temporalio import workflow
from temporalio.common import RetryPolicy, SearchAttributeKey

with workflow.unsafe.imports_passed_through():
    from activities import (
        NotificationRequest,
        StoreDocumentRequest,
        generate_approval_report,
        record_audit_entry,
        send_notification,
        store_document,
    )
    from models import (
        ApprovalDecision,
        ApprovalState,
        ApprovalStatus,
        ApproverConfig,
        AuditEntry,
        DocumentStatus,
        DocumentSubmission,
    )

# Custom Search Attribute that tracks where the document is in the approval process.
APPROVAL_STATUS_KEY = SearchAttributeKey.for_keyword("ApprovalStatus")

# Retry policy for notification Activities.
NOTIFICATION_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=5),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=2),
    maximum_attempts=5,
)

# Retry policy for audit logging.
AUDIT_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=2),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=5),
    maximum_attempts=10,
)

# Retry policy for document storage.
STORAGE_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=1),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=1),
    maximum_attempts=5,
)

# Retry policy for long-running report generation.
REPORT_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=10),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=5),
    maximum_attempts=3,
)

@workflow.defn
class DocumentApprovalWorkflow:

    @workflow.init
    def __init__(
        self, input_data: DocumentSubmission | ApprovalState
    ) -> None:
        if isinstance(input_data, ApprovalState):
            self._state = input_data
        else:
            self._state = ApprovalState(document=input_data)

        self._pending_decision: ApprovalDecision | None = None
        self._processed_update_ids: set[str] = set()

    @workflow.signal
    async def submit_decision(self, decision: ApprovalDecision) -> None:
        workflow.logger.info(
            "Decision received via Signal",
            extra={
                "approver": decision.approver_email,
                "status": decision.status,
            },
        )
        self._pending_decision = decision

    @workflow.signal
    async def withdraw(self) -> None:
        workflow.logger.info("Document withdrawal requested")
        self._state.status = DocumentStatus.WITHDRAWN

    @workflow.query
    def get_status(self) -> dict:
        return {
            "document_id": self._state.document.document_id,
            "title": self._state.document.title,
            "status": self._state.status.value,
            "decision": asdict(self._state.decision) if self._state.decision else None,
            "resubmission_count": self._state.resubmission_count,
        }

    @workflow.query
    def get_audit_trail(self) -> list[dict]:
        """Return the in-memory audit trail for this Workflow."""
        return [asdict(e) for e in self._state.audit_trail]

    @workflow.update
    async def resubmit_document(self, update_id: str, new_content_url: str) -> dict:
        # Prevent duplicate processing of the same update
        if update_id in self._processed_update_ids:
            return {
                "accepted": True,
                "reason": "Resubmission already processed",
                "duplicate": True,
            }

        if self._state.status != DocumentStatus.REJECTED:
            return {
                "accepted": False,
                "reason": (
                    f"Document is in {self._state.status.value} state. "
                    "Resubmission is only allowed after rejection."
                ),
            }

        if self._state.resubmission_count >= self._state.max_resubmissions:
            return {
                "accepted": False,
                "reason": (
                    f"Maximum resubmissions ({self._state.max_resubmissions}) "
                    "reached."
                ),
            }

        self._state.document.content_url = new_content_url
        self._state.resubmission_count += 1
        self._state.status = DocumentStatus.SUBMITTED
        self._pending_decision = None
        self._processed_update_ids.add(update_id)

        workflow.logger.info(
            "Document resubmitted",
            extra={
                "resubmission_count": self._state.resubmission_count,
                "new_content_url": new_content_url,
            },
        )

        return {
            "accepted": True,
            "resubmission_count": self._state.resubmission_count,
        }

    @resubmit_document.validator
    def validate_resubmit(self, update_id: str, new_content_url: str) -> None:
        """Validate the resubmission URL before accepting the Update."""
        if not update_id or not update_id.strip():
            raise ValueError("update_id must not be empty")
        if not new_content_url or not new_content_url.strip():
            raise ValueError("new_content_url must not be empty")

    @workflow.run
    async def run(
        self, input_data: DocumentSubmission | ApprovalState
    ) -> dict:
        workflow_id = workflow.info().workflow_id

        await self._record_audit(
            workflow_id=workflow_id,
            action="WORKFLOW_STARTED",
            actor=self._state.document.submitter_email,
            details=(
                f"Document '{self._state.document.title}' submitted "
                f"for approval (resubmission #{self._state.resubmission_count})"
            ),
        )

        if not self._state.document_stored:
            await self._store_document(workflow_id)
            self._state.document_stored = True

        if workflow.info().is_continue_as_new_suggested():
            workflow.logger.info("Continue-As-New suggested, resetting Event History")
            await workflow.wait_condition(workflow.all_handlers_finished)
            workflow.continue_as_new(args=[self._state])

        self._state.status = DocumentStatus.IN_REVIEW
        approver_config = self._state.document.approver

        workflow.logger.info(
            "Approval process started",
            extra={
                "approver": approver_config.approver_email,
                "sla_seconds": approver_config.sla_seconds,
            },
        )

        await self._send_approval_request(workflow_id, approver_config)

        result: dict = {}
        while True:
            decision = await self._wait_for_decision(workflow_id, approver_config)

            if self._state.status == DocumentStatus.WITHDRAWN:
                await self._record_audit(
                    workflow_id=workflow_id,
                    action="DOCUMENT_WITHDRAWN",
                    actor=self._state.document.submitter_email,
                    details="Document withdrawn by submitter",
                )
                result = self._build_result("Document withdrawn by submitter")
                break

            result = await self._handle_decision(workflow_id, approver_config, decision)

            if self._state.status != DocumentStatus.SUBMITTED:
                break

            if workflow.info().is_continue_as_new_suggested():
                workflow.logger.info(
                    "Continue-As-New suggested before resubmission cycle"
                )
                await workflow.wait_condition(workflow.all_handlers_finished)
                workflow.continue_as_new(args=[self._state])

            self._state.status = DocumentStatus.IN_REVIEW
            await self._send_approval_request(workflow_id, approver_config)

        # Wait for all in-flight Signal and Update handlers to complete
        # before returning
        await workflow.wait_condition(workflow.all_handlers_finished)

        # Record the final status so you can filter closed Workflows by it
        workflow.upsert_search_attributes(
            [APPROVAL_STATUS_KEY.value_set(self._state.status.value)]
        )

        return result

    async def _wait_for_decision(
        self, workflow_id: str, approver_config: ApproverConfig
    ) -> ApprovalDecision:
        sla_timeout = timedelta(seconds=approver_config.sla_seconds)
        reminder_interval = timedelta(
            seconds=approver_config.reminder_interval_seconds
        )

        self._pending_decision = None

        reminder_task = asyncio.create_task(
            self._send_reminders(workflow_id, approver_config, reminder_interval)
        )

        try:
            await workflow.wait_condition(
                lambda: (
                    self._pending_decision is not None
                    or self._state.status == DocumentStatus.WITHDRAWN
                ),
                timeout=sla_timeout,
            )

            reminder_task.cancel()
            try:
                await reminder_task
            except asyncio.CancelledError:
                pass

            if self._state.status == DocumentStatus.WITHDRAWN:
                return ApprovalDecision(
                    approver_email=self._state.document.submitter_email,
                    status=ApprovalStatus.REJECTED,
                    comment="Document withdrawn",
                    decided_at=workflow.now().isoformat(),
                )

            decision = self._pending_decision
            assert decision is not None
            self._pending_decision = None
            return decision

        except asyncio.TimeoutError:
            reminder_task.cancel()
            try:
                await reminder_task
            except asyncio.CancelledError:
                pass

            return await self._handle_escalation(workflow_id, approver_config)

    async def _send_reminders(
        self, workflow_id: str, approver_config: ApproverConfig, interval: timedelta
    ) -> None:
        reminder_count = 0
        while True:
            await asyncio.sleep(interval.total_seconds())
            reminder_count += 1

            if reminder_count > approver_config.max_reminders:
                workflow.logger.info(
                    "Max reminders reached, stopping reminder loop"
                )
                break

            workflow.logger.info(
                "Sending reminder",
                extra={
                    "reminder_count": reminder_count,
                    "approver": approver_config.approver_email,
                },
            )

            await workflow.execute_activity(
                send_notification,
                NotificationRequest(
                    recipient_email=approver_config.approver_email,
                    recipient_name=approver_config.approver_name,
                    subject=(
                        f"Reminder: Approval needed for "
                        f"'{self._state.document.title}'"
                    ),
                    body=f"Reminder #{reminder_count}: please review '{self._state.document.title}'.",
                    idempotency_key=(
                        f"{workflow_id}-reminder-R{reminder_count}"
                    ),
                ),
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=NOTIFICATION_RETRY_POLICY,
            )

            await self._record_audit(
                workflow_id=workflow_id,
                action="REMINDER_SENT",
                actor="system",
                details=(
                    f"Reminder #{reminder_count} sent to "
                    f"{approver_config.approver_email}"
                ),
            )

    async def _handle_escalation(
        self, workflow_id: str, approver_config: ApproverConfig
    ) -> ApprovalDecision:
        await self._record_audit(
            workflow_id=workflow_id,
            action="SLA_EXPIRED",
            actor="system",
            details=(
                f"SLA expired for {approver_config.approver_email} "
                f"after {approver_config.sla_seconds} seconds"
            ),
        )

        if approver_config.escalation_email:
            workflow.logger.info(
                "Escalating to backup approver",
                extra={
                    "original_approver": approver_config.approver_email,
                    "escalation_contact": approver_config.escalation_email,
                },
            )

            await workflow.execute_activity(
                send_notification,
                NotificationRequest(
                    recipient_email=approver_config.escalation_email,
                    recipient_name="Escalation Contact",
                    subject=(
                        f"ESCALATION: Approval needed for "
                        f"'{self._state.document.title}'"
                    ),
                    body=f"Original approver ({approver_config.approver_email}) did not respond within the SLA. Please review and approve or reject.",
                    idempotency_key=f"{workflow_id}-escalation",
                ),
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=NOTIFICATION_RETRY_POLICY,
            )

            await self._record_audit(
                workflow_id=workflow_id,
                action="ESCALATION_SENT",
                actor="system",
                details=f"Escalated to {approver_config.escalation_email}",
            )

            self._pending_decision = None
            try:
                await workflow.wait_condition(
                    lambda: (
                        self._pending_decision is not None
                        or self._state.status == DocumentStatus.WITHDRAWN
                    ),
                    timeout=timedelta(seconds=approver_config.sla_seconds),
                )

                if self._state.status == DocumentStatus.WITHDRAWN:
                    return ApprovalDecision(
                        approver_email=self._state.document.submitter_email,
                        status=ApprovalStatus.REJECTED,
                        comment="Document withdrawn during escalation",
                        decided_at=workflow.now().isoformat(),
                    )

                decision = self._pending_decision
                assert decision is not None
                self._pending_decision = None
                return decision

            except asyncio.TimeoutError:
                await self._record_audit(
                    workflow_id=workflow_id,
                    action="ESCALATION_TIMEOUT",
                    actor="system",
                    details=(
                        f"Escalation contact {approver_config.escalation_email} "
                        f"also timed out"
                    ),
                )

                return ApprovalDecision(
                    approver_email="system",
                    status=ApprovalStatus.TIMED_OUT,
                    comment=(
                        "Auto-rejected: both the original approver and "
                        "escalation contact failed to respond within the SLA"
                    ),
                    decided_at=workflow.now().isoformat(),
                )
        else:
            workflow.logger.info("No escalation contact configured, auto-rejecting")

            await self._record_audit(
                workflow_id=workflow_id,
                action="AUTO_REJECTED",
                actor="system",
                details=(
                    f"No escalation contact configured for {approver_config.approver_email}"
                ),
            )

            return ApprovalDecision(
                approver_email="system",
                status=ApprovalStatus.TIMED_OUT,
                comment=(
                    f"Auto-rejected: approver {approver_config.approver_email} "
                    f"did not respond within {approver_config.sla_seconds} seconds"
                ),
                decided_at=workflow.now().isoformat(),
            )

    async def _handle_decision(
        self,
        workflow_id: str,
        approver_config: ApproverConfig,
        decision: ApprovalDecision,
    ) -> dict:
        # Stamp the decision with processing time using the deterministic Workflow clock
        decision.decided_at = workflow.now().isoformat()
        self._state.decision = decision

        await self._record_audit(
            workflow_id=workflow_id,
            action=f"DECISION_{decision.status.value}",
            actor=decision.approver_email,
            details=decision.comment,
        )

        if decision.status == ApprovalStatus.APPROVED:
            workflow.logger.info(
                "Document approved",
                extra={"approver": decision.approver_email},
            )

            report_url = await self._finalize_approval(workflow_id)

            await workflow.execute_activity(
                send_notification,
                NotificationRequest(
                    recipient_email=self._state.document.submitter_email,
                    recipient_name=self._state.document.submitter_name,
                    subject=(
                        f"Document approved: '{self._state.document.title}'"
                    ),
                    body=f"Your document has been approved by {decision.approver_email}. Approval report: {report_url}",
                    idempotency_key=f"{workflow_id}-final-approval",
                ),
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=NOTIFICATION_RETRY_POLICY,
            )

            return self._build_result(
                message="Document approved",
                report_url=report_url,
            )

        elif decision.status == ApprovalStatus.CHANGES_REQUESTED:
            workflow.logger.info(
                "Changes requested",
                extra={"approver": decision.approver_email},
            )

            self._state.status = DocumentStatus.REJECTED

            await workflow.execute_activity(
                send_notification,
                NotificationRequest(
                    recipient_email=self._state.document.submitter_email,
                    recipient_name=self._state.document.submitter_name,
                    subject=(
                        f"Changes requested: "
                        f"'{self._state.document.title}'"
                    ),
                    body=f"Approver {decision.approver_email} has requested changes: {decision.comment}. You may resubmit.",
                    idempotency_key=(
                        f"{workflow_id}-changes-requested-"
                        f"S{self._state.resubmission_count}"
                    ),
                ),
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=NOTIFICATION_RETRY_POLICY,
            )

            resubmission_timeout = timedelta(
                seconds=approver_config.resubmission_timeout_seconds
            )
            try:
                await workflow.wait_condition(
                    lambda: (
                        self._state.status == DocumentStatus.SUBMITTED
                        or self._state.status == DocumentStatus.WITHDRAWN
                    ),
                    timeout=resubmission_timeout,
                )

                if self._state.status == DocumentStatus.WITHDRAWN:
                    await self._record_audit(
                        workflow_id=workflow_id,
                        action="DOCUMENT_WITHDRAWN",
                        actor=self._state.document.submitter_email,
                        details="Document withdrawn after changes requested",
                    )
                    return self._build_result(
                        message="Document withdrawn by submitter"
                    )

                workflow.logger.info(
                    "Resubmission accepted",
                    extra={"resubmission_count": self._state.resubmission_count},
                )

                if self._state.resubmission_count >= self._state.max_resubmissions:
                    self._state.status = DocumentStatus.REJECTED
                    await self._record_audit(
                        workflow_id=workflow_id,
                        action="MAX_RESUBMISSIONS_EXCEEDED",
                        actor="system",
                        details=f"Maximum resubmissions ({self._state.max_resubmissions}) exceeded",
                    )
                    return self._build_result(
                        message="Maximum resubmission attempts exceeded"
                    )

                return self._build_result(message="Resubmission accepted")

            except asyncio.TimeoutError:
                workflow.logger.info(
                    "Resubmission window expired"
                )
                self._state.status = DocumentStatus.REJECTED
                await self._record_audit(
                    workflow_id=workflow_id,
                    action="RESUBMISSION_TIMEOUT",
                    actor="system",
                    details=(
                        "Resubmission window expired after 7 days"
                    ),
                )
                return self._build_result(
                    message="Resubmission deadline expired, document rejected"
                )

        else:
            workflow.logger.info(
                "Document rejected",
                extra={"status": decision.status.value},
            )

            self._state.status = DocumentStatus.REJECTED

            await workflow.execute_activity(
                send_notification,
                NotificationRequest(
                    recipient_email=self._state.document.submitter_email,
                    recipient_name=self._state.document.submitter_name,
                    subject=(
                        f"Document rejected: "
                        f"'{self._state.document.title}'"
                    ),
                    body=f"Your document was {decision.status.value.lower()}. Reason: {decision.comment}",
                    idempotency_key=(
                        f"{workflow_id}-rejected-"
                        f"S{self._state.resubmission_count}"
                    ),
                ),
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=NOTIFICATION_RETRY_POLICY,
            )
            return self._build_result(
                message="Document rejected by approver"
            )

    async def _finalize_approval(self, workflow_id: str) -> str:
        """Generate the approval report and return the report URL."""
        self._state.status = DocumentStatus.APPROVED

        await self._record_audit(
            workflow_id=workflow_id,
            action="DOCUMENT_APPROVED",
            actor="system",
            details="Document approved by approver",
        )

        decision_data = asdict(self._state.decision)
        report_url = await workflow.execute_activity(
            generate_approval_report,
            args=[self._state.document.document_id, [decision_data]],
            start_to_close_timeout=timedelta(minutes=5),
            heartbeat_timeout=timedelta(minutes=2),
            retry_policy=REPORT_RETRY_POLICY,
        )

        return report_url

    async def _store_document(self, workflow_id: str) -> None:
        """Store the document and record the audit entry."""
        storage_ref = await workflow.execute_activity(
            store_document,
            StoreDocumentRequest(
                document_id=self._state.document.document_id,
                title=self._state.document.title,
                content_url=self._state.document.content_url,
                submitter_email=self._state.document.submitter_email,
            ),
            start_to_close_timeout=timedelta(seconds=60),
            retry_policy=STORAGE_RETRY_POLICY,
        )

        workflow.logger.info(
            "Document stored",
            extra={"storage_ref": storage_ref},
        )

        await self._record_audit(
            workflow_id=workflow_id,
            action="DOCUMENT_STORED",
            actor=self._state.document.submitter_email,
            details=f"Document stored at {storage_ref}",
        )

    async def _send_approval_request(
        self, workflow_id: str, approver_config: ApproverConfig
    ) -> None:
        """Send the initial approval request to the approver."""
        await workflow.execute_activity(
            send_notification,
            NotificationRequest(
                recipient_email=approver_config.approver_email,
                recipient_name=approver_config.approver_name,
                subject=(
                    f"Approval requested: "
                    f"'{self._state.document.title}'"
                ),
                body=f"'{self._state.document.title}' by {self._state.document.submitter_name} requires your approval (SLA: {approver_config.sla_seconds}s).",
                idempotency_key=(
                    f"{workflow_id}-approval-request-"
                    f"S{self._state.resubmission_count}"
                ),
            ),
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=NOTIFICATION_RETRY_POLICY,
        )

        await self._record_audit(
            workflow_id=workflow_id,
            action="APPROVAL_REQUESTED",
            actor="system",
            details=(
                f"Approval request sent to "
                f"{approver_config.approver_email}"
            ),
        )

    async def _record_audit(
        self,
        workflow_id: str,
        action: str,
        actor: str,
        details: str = "",
        approval_level: int = 0,
    ) -> None:
        """Record an audit entry both in-memory and via an Activity."""
        entry = AuditEntry(
            timestamp=workflow.now().isoformat(),
            workflow_id=workflow_id,
            action=action,
            actor=actor,
            details=details,
            approval_level=approval_level,
        )

        self._state.audit_trail.append(entry)

        await workflow.execute_activity(
            record_audit_entry,
            entry,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=AUDIT_RETRY_POLICY,
        )

    def _build_result(
        self, message: str, report_url: str = ""
    ) -> dict:
        """Build the final result returned by the Workflow."""
        return {
            "document_id": self._state.document.document_id,
            "status": self._state.status.value,
            "message": message,
            "decision": (
                asdict(self._state.decision)
                if self._state.decision
                else None
            ),
            "resubmission_count": self._state.resubmission_count,
            "report_url": report_url,
        }
```

### Phase 4: Configure and run the Worker

The Worker is the process that executes Workflow and Activity code. Create a file named `worker.py`:

```python
"""Worker process for the document approval system."""

from __future__ import annotations

import asyncio
import concurrent.futures
import logging
import os

from temporalio.client import Client
from temporalio.worker import Worker

from activities import (
    generate_approval_report,
    record_audit_entry,
    send_notification,
    store_document,
)
from workflows import DocumentApprovalWorkflow

TASK_QUEUE = os.getenv("TEMPORAL_TASK_QUEUE", "document-approval")
NAMESPACE = os.getenv("TEMPORAL_NAMESPACE", "document-approval")

async def main() -> None:
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    )

    logging.info(f"Connecting to Temporal Server with namespace: {NAMESPACE}")
    client = await Client.connect("localhost:7233", namespace=NAMESPACE)

    with concurrent.futures.ThreadPoolExecutor(
        max_workers=50
    ) as activity_executor:
        worker = Worker(
            client,
            task_queue=TASK_QUEUE,
            workflows=[DocumentApprovalWorkflow],
            activities=[
                send_notification,
                record_audit_entry,
                store_document,
                generate_approval_report,
            ],
            activity_executor=activity_executor,
            max_concurrent_workflow_tasks=100,
            max_concurrent_activities=50,
        )

        logging.info("Starting Worker on Task Queue: %s", TASK_QUEUE)
        await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

To run the Worker, start the Temporal development server in one terminal and the Worker in another:

```
temporal server start-dev
```

```
python worker.py
```

### Phase 5: Start a Workflow and send Signals

Create a file named `starter.py` that demonstrates how to start the approval Workflow and interact with it:

```python
"""Client code to start a document approval Workflow and interact with it."""

from __future__ import annotations

import asyncio
import os

from temporalio.client import Client
from temporalio.common import (
    SearchAttributeKey,
    SearchAttributePair,
    TypedSearchAttributes,
)

from models import (
    ApprovalDecision,
    ApprovalStatus,
    ApproverConfig,
    DocumentSubmission,
)
from workflows import APPROVAL_STATUS_KEY, DocumentApprovalWorkflow

TASK_QUEUE = os.getenv("TEMPORAL_TASK_QUEUE", "document-approval")
NAMESPACE = os.getenv("TEMPORAL_NAMESPACE", "document-approval")

DOCUMENT_ID_KEY = SearchAttributeKey.for_keyword("DocumentId")
SUBMITTER_EMAIL_KEY = SearchAttributeKey.for_keyword("SubmitterEmail")

async def main() -> None:
    """Start an approval Workflow and demonstrate Signal and Query interactions."""
    client = await Client.connect("localhost:7233", namespace=NAMESPACE)

    document = DocumentSubmission(
        document_id="doc-2026-001",
        title="Q1 Budget Proposal",
        submitter_email="alice@example.com",
        submitter_name="Alice Chen",
        content_url="https://docs.example.com/q1-budget-v1",
        approver=ApproverConfig(
            approver_email="bob@example.com",
            approver_name="Bob Martinez",
            sla_seconds=172800,  # 48 hours
            escalation_email="carol@example.com",
            reminder_interval_seconds=86400,  # Remind every 24 hours
            max_reminders=3,  # Send max 3 reminders
        ),
    )

    workflow_id = f"approval-{document.document_id}"

    search_attributes = TypedSearchAttributes([
        SearchAttributePair(DOCUMENT_ID_KEY, document.document_id),
        SearchAttributePair(SUBMITTER_EMAIL_KEY, document.submitter_email),
        SearchAttributePair(APPROVAL_STATUS_KEY, "SUBMITTED"),
    ])

    handle = await client.start_workflow(
        DocumentApprovalWorkflow.run,
        document,
        id=workflow_id,
        task_queue=TASK_QUEUE,
        search_attributes=search_attributes,
    )

    print(f"Workflow started: {workflow_id}")
    print(f"Run Id: {handle.result_run_id}")

    status = await handle.query(DocumentApprovalWorkflow.get_status)
    print(f"Current status: {status}")

    await handle.signal(
        DocumentApprovalWorkflow.submit_decision,
        ApprovalDecision(
            approver_email="bob@example.com",
            status=ApprovalStatus.APPROVED,
            comment="Budget looks good. Approved.",
        ),
    )
    print("Approval Signal sent")

    # The Workflow sets the final ApprovalStatus Search Attribute before it completes
    result = await handle.result()
    print(f"Workflow result: {result}")

    # Query the audit trail
    audit_trail = await handle.query(
        DocumentApprovalWorkflow.get_audit_trail
    )
    print(f"Audit trail ({len(audit_trail)} entries):")
    for entry in audit_trail:
        print(f"  [{entry['timestamp']}] {entry['action']}: {entry['details']}")

if __name__ == "__main__":
    asyncio.run(main())
```

### Phase 6: Configure timeouts and retry policies

Choosing appropriate timeouts and retry policies is critical for the reliability of the system. The following table summarizes the timeout and retry configuration used in this pattern:

| Activity | `start_to_close_timeout` | `heartbeat_timeout` | Retry policy | Rationale |
|----------|--------------------------|---------------------|--------------|-----------|
| `send_notification` | 30 seconds | Not set | 5 attempts, 5s initial, 2x backoff, 2m max | Notifications are important but not blocking. Five retries with exponential backoff handle transient network failures. |
| `record_audit_entry` | 30 seconds | Not set | 10 attempts, 2s initial, 2x backoff, 5m max | Audit entries are critical for compliance. More aggressive retry to ensure entries are persisted. |
| `store_document` | 60 seconds | Not set | 5 attempts, 1s initial, 2x backoff, 1m max | Document storage should complete quickly. Longer `start_to_close_timeout` accommodates large documents. |
| `generate_approval_report` | 5 minutes | 30 seconds | 3 attempts, 10s initial, 2x backoff, 5m max | Report generation may take time. Heartbeating detects Worker failures quickly. Fewer retries because the operation is expensive. |

Do not set `schedule_to_close_timeout` unless you need to bound the total time across all retry attempts.

## Outcomes

You've built a document approval system that holds up under real conditions — slow reviewers, missed deadlines, infrastructure restarts, and repeated resubmissions. The techniques here aren't specific to documents. Any process that has to wait for a human — loan applications, employee onboarding, content moderation, procurement sign-off, insurance claims — can be built the same way. The Workflow waits, the timers fire, the audit trail fills in, and the process completes regardless of what happens underneath it.

## Related resources

- [Approval Pattern](/design-patterns/approval) — pattern catalog reference for this Workflow with Go, Java, Python, and TypeScript implementations, plus multi-level and escalating approval variants
- [Temporal Python SDK documentation](/develop/python)
- [Message Passing — Signals, Queries, Updates](/develop/python/workflows/message-passing)
- [Continue-As-New](/develop/python/workflows/continue-as-new)
- [Failure Detection — Timeouts, Activity Heartbeating, and Retry Policies](/develop/python/best-practices/error-handling)
- [Child Workflows](/develop/python/workflows/child-workflows)
- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning)
- [Temporal Python SDK API Reference](https://python.temporal.io)
- [Temporal Python SDK GitHub repository](https://github.com/temporalio/sdk-python)
- [Temporal Python SDK samples](https://github.com/temporalio/samples-python)
- [Entity Workflow Pattern](entity-pattern-loyalty-points) — covers the related Entity Workflow pattern for modeling long-lived domain objects

---

# Route GPU and high-memory Activities to dedicated Workers

Source: https://docs.temporal.io/guides/route-specialized-workloads

> Give GPU, high-memory, and CPU Activities their own Task Queues so each runs only on Workers with the right hardware, shown with a Python ML pipeline.

**by Cecil Phillip**

Modern applications have workloads with diverse resource requirements. ML/AI workloads require expensive GPU-equipped Workers with specific CUDA libraries, video processing needs specialized encoding hardware, and data analytics may require high-memory instances. Running all Activities on the same Worker type is cost-prohibitive and inefficient.

## Problem statement

Without separate Task Queues for different resource types, these scenarios create problems:
- **Resource waste:** GPU Workers sit idle waiting for ML work, or expensive GPUs run standard CPU tasks that don't need them
- **Cost inefficiency:** Running all Workers on GPU instances when only 10% of workloads need GPUs increases costs by 20-30x
- **Environment conflicts:** ML libraries (TensorFlow, PyTorch) have complex dependencies that conflict with other Activity requirements
- **Scaling complexity:** Cannot independently scale GPU Workers based on ML demand vs CPU Workers based on standard workload demand

## Solution

Use separate Task Queues to route Activities based on their resource requirements. Create dedicated Worker pools for:
- **GPU-intensive ML workloads** (NVIDIA GPUs, CUDA libraries, limited concurrency due to GPU memory)
- **Standard CPU workloads** (high concurrency on cost-effective instances)
- **High-memory analytics** (memory-optimized instances for large dataset processing)
- **Specialized hardware** (video encoding hardware, TPUs, ARM processors)

## Outcomes

- **Cost optimization:** GPU instances (\$3/hr) only handle ML inference while standard Activities run on cost-effective CPU instances (\$0.10/hr), reducing infrastructure costs by 60-80%
- **Resource efficiency:** GPU Workers run at 2-4 concurrent activities (GPU memory constraints) while CPU Workers run at 100+ concurrent activities, maximizing hardware utilization
- **Independent scaling:** Scale GPU Workers based on ML workload demand, high-memory Workers based on analytics load, and CPU Workers based on standard workload, without coordination
- **Environment isolation:** ML Workers have TensorFlow/PyTorch dependencies isolated from standard Workers, preventing library conflicts and simplifying deployments

## Background and best practices

### Task Queue fundamentals
Task Queues in Temporal are dynamically created when first referenced. If your Workflow references Task Queue `gpu-processing` but your Worker polls `gpu-procesing` (typo), two separate queues are created and the Worker never receives Tasks.

**Best practice:** Always define Task Queue names as constants in a shared module that both Workflows and Workers import.

### Resource-based routing approaches

Route Activities to appropriate Worker pools based on their resource requirements:
- **GPU-intensive Activities:** Route to GPU-equipped Workers with ML libraries (TensorFlow, PyTorch, CUDA)
- **CPU-intensive Activities:** Route to high-CPU instances for compute-heavy tasks
- **Memory-intensive Activities:** Route to memory-optimized instances for large dataset processing
- **Standard Activities:** Route to cost-effective general-purpose instances

### Worker configuration for specialized hardware

Key considerations for resource-specific Workers:
- **GPU concurrency:** Limit GPU Workers to 2-4 concurrent activities due to GPU memory constraints
- **CPU concurrency:** Standard Workers can handle 100+ concurrent activities
- **Memory allocation:** High-memory Workers need careful concurrency tuning to avoid OOM errors
- **Hardware dependencies:** GPU Workers require NVIDIA drivers, CUDA toolkit, and specific ML library versions
- **Container images:** Use specialized images with pre-installed dependencies (nvidia/cuda, tensorflow/tensorflow:latest-gpu)

### Operational considerations

- **Resource utilization:** Monitor GPU memory, compute utilization, and CPU/memory usage per Worker pool
- **Queue latency:** Track `schedule_to_start_latency` per Task Queue to detect under-provisioning of specialized hardware
- **Cost tracking:** Tag resources by hardware type (GPU, CPU, high-memory) for cost analysis and optimization
- **Hardware health:** Monitor GPU temperature, driver errors, and CUDA out-of-memory errors
- **Dependency management:** Use container images with pinned library versions to ensure reproducible environments

## Target audience

- **Temporal Workflow & Activity developers:** Implementing resource-aware routing logic
- **Platform operators:** Configuring Worker deployments for specialized hardware
- **ML/AI Engineers:** Deploying GPU-based inference and training Workers
- **DevOps/SRE teams:** Managing heterogeneous Worker pools and infrastructure
- **FinOps teams:** Optimizing cloud infrastructure costs through efficient resource allocation

This implementation requires code changes, Worker configuration, and deployment of specialized infrastructure (GPU instances, high-memory instances, specialized hardware).

## Prerequisites

### Required software, infrastructure, and tools

- Temporal Service (Self-hosted or Temporal Cloud)
- Python 3.8 or later
- Temporal Python SDK v1.0.0 or later (`pip install temporalio`)
- GPU infrastructure (AWS EC2 with NVIDIA GPUs, GCP with GPUs) for ML workloads
- Process manager or container orchestration for multi-Worker deployments

### Resources & access privileges

- Temporal Namespace with permissions to start Workflows and register Workers
- Infrastructure provisioning access for GPU instances and multiple Worker pools
- Ability to configure autoscaling policies

### Required concepts

- Temporal Workflows, Activities, and Task Queues
- Python async/await patterns
- GPU computing basics (for ML scenarios)
- Basic deployment and process management

## Architecture diagram(s)

### Resource-based routing architecture

```mermaid
flowchart TB
    subgraph Workflow["Workflow Execution"]
        WF[ML Pipeline<br/>Workflow]
    end

    subgraph Routing["Resource-Based Routing Logic"]
        Decision{Activity<br/>Resource<br/>Requirements}
    end

    subgraph Queues["Task Queues by Resource Type"]
        CPUTQ[cpu-standard<br/>Task Queue]
        GPUTQ[gpu-ml-inference<br/>Task Queue]
        MEMTQ[high-memory-analytics<br/>Task Queue]
    end

    subgraph Workers["Specialized Worker Pools"]
        subgraph CPUPool["Standard CPU Pool"]
            CPU1[Instance: c5.xlarge]
            CPU2[4 vCPU, 8GB RAM]
            CPU3[Replicas: 10]
            CPU4[Concurrency: 100]
            CPU5[Cost: $0.17/hr]
        end
        subgraph GPUPool["GPU ML Pool"]
            GPU1[Instance: g4dn.xlarge]
            GPU2[NVIDIA T4 16GB]
            GPU3[CUDA 12.0, PyTorch]
            GPU4[Replicas: 2]
            GPU5[Concurrency: 2]
            GPU6[Cost: $0.526/hr]
        end
        subgraph MEMPool["High-Memory Pool"]
            MEM1[Instance: r5.2xlarge]
            MEM2[8 vCPU, 64GB RAM]
            MEM3[Replicas: 3]
            MEM4[Concurrency: 10]
            MEM5[Cost: $0.504/hr]
        end
    end

    WF --> Decision

    Decision -->|CPU Activities<br/>validation, API calls| CPUTQ
    Decision -->|GPU Activities<br/>inference, embeddings| GPUTQ
    Decision -->|Memory Activities<br/>large dataset processing| MEMTQ

    CPUTQ --> CPUPool
    GPUTQ --> GPUPool
    MEMTQ --> MEMPool

    classDef blue stroke-width:1px
    classDef amber stroke-width:1px
    classDef purple stroke-width:1px
    class CPUTQ blue
    class GPUTQ amber
    class MEMTQ purple
    class CPUPool blue
    class GPUPool amber
    class MEMPool purple
```

## Implementation plan

### Step 1: Define Task Queue constants

Create a shared Python module with Task Queue name constants based on resource requirements.

**File: `task_queues.py`**

```python
"""Task Queue constants for resource-based routing."""

# Resource-specific task queues
STANDARD_CPU_QUEUE = "cpu-standard"
GPU_ML_QUEUE = "gpu-ml-inference"
HIGH_MEMORY_QUEUE = "high-memory-analytics"
VIDEO_ENCODING_QUEUE = "video-encoding-hardware"
```

### Step 2: Configure Workers for standard CPU processing

Deploy Workers on cost-effective CPU instances for standard Activities that don't require specialized hardware.

**File: `worker_cpu.py`**

```python
"""Worker for standard CPU-based Activities."""
import asyncio
import logging
from temporalio.client import Client
from temporalio.worker import Worker

from task_queues import STANDARD_CPU_QUEUE
from workflows import MLPipelineWorkflow
from activities import validate_data, preprocess_data, store_results

logging.basicConfig(level=logging.INFO)

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue=STANDARD_CPU_QUEUE,
        workflows=[MLPipelineWorkflow],
        activities=[validate_data, preprocess_data, store_results],
        max_concurrent_activities=100,  # High concurrency on CPU
    )

    logging.info(f"Starting CPU worker on {STANDARD_CPU_QUEUE}")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**Deployment guidance:**
- Deploy 10 instances of `worker_cpu.py` on standard compute instances (c5.xlarge: 4 vCPU, 8GB RAM)
- Use autoscaling based on queue depth and CPU utilization
- Consider using spot instances for cost savings if workload tolerates interruptions

### Step 3: Configure Workers for GPU processing

Deploy Workers on GPU-equipped instances for ML/AI workloads. GPU Workers should have limited concurrency due to GPU memory constraints.

**File: `worker_gpu.py`**

```python
"""Worker for GPU-intensive ML Activities."""
import asyncio
import logging
from temporalio.client import Client
from temporalio.worker import Worker

from task_queues import GPU_ML_QUEUE
from activities import generate_embeddings, run_inference, train_model

logging.basicConfig(level=logging.INFO)

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue=GPU_ML_QUEUE,
        activities=[generate_embeddings, run_inference, train_model],
        # Limited concurrency - GPU memory constraint
        # T4 16GB can typically handle 2-4 concurrent inference tasks
        max_concurrent_activities=2,
    )

    logging.info(f"Starting GPU worker on {GPU_ML_QUEUE}")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**Deployment guidance:**
- Deploy 2-3 instances of `worker_gpu.py` on GPU instances (g4dn.xlarge with NVIDIA T4, or p3.2xlarge with V100)
- GPU Workers require:
  - NVIDIA drivers (version 525.60 or later)
  - CUDA toolkit (12.0 or later)
  - ML frameworks (PyTorch, TensorFlow) with GPU support
- Use container images: `nvidia/cuda:12.0-cudnn8-runtime-ubuntu22.04` or `pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime`
- Monitor GPU memory usage - adjust `max_concurrent_activities` if OOM errors occur
- Consider using GPU-optimized instance types based on model requirements (T4 for inference, V100/A100 for training)

### Step 4: Configure Workers for high-memory analytics

Deploy Workers on memory-optimized instances for Activities processing large datasets.

**File: `worker_high_memory.py`**

```python
"""Worker for high-memory data processing Activities."""
import asyncio
import logging
from temporalio.client import Client
from temporalio.worker import Worker

from task_queues import HIGH_MEMORY_QUEUE
from activities import process_large_dataset, aggregate_analytics, build_large_index

logging.basicConfig(level=logging.INFO)

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(
        client,
        task_queue=HIGH_MEMORY_QUEUE,
        activities=[process_large_dataset, aggregate_analytics, build_large_index],
        # Limited concurrency due to high memory usage per activity
        # r5.2xlarge (64GB RAM) can handle ~10 concurrent activities using 5-6GB each
        max_concurrent_activities=10,
    )

    logging.info(f"Starting high-memory worker on {HIGH_MEMORY_QUEUE}")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

**Deployment guidance:**
- Deploy 3-5 instances of `worker_high_memory.py` on memory-optimized instances (r5.2xlarge: 8 vCPU, 64GB RAM)
- Adjust `max_concurrent_activities` based on per-Activity memory usage to prevent OOM errors
- Monitor memory utilization and swap usage
- Consider using instances with local NVMe storage for faster I/O on large datasets

### Step 5: Implement resource-aware routing in Workflows

**File: `ml_pipeline_workflow.py`**

```python
"""Workflow with resource-aware routing."""
from datetime import timedelta
from dataclasses import dataclass
from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from task_queues import STANDARD_CPU_QUEUE, GPU_ML_QUEUE, HIGH_MEMORY_QUEUE

@dataclass
class MLPipelineRequest:
    pipeline_id: str
    dataset_url: str
    model_type: str
    customer_id: str

@workflow.defn
class MLPipelineWorkflow:
    """
    ML pipeline workflow with resource-aware routing.

    Routes Activities to appropriate Worker pools based on resource requirements:
    - CPU queue: Data validation, preprocessing, result storage
    - GPU queue: Model training, inference, embeddings generation
    - High-memory queue: Large dataset processing, analytics aggregation
    """

    @workflow.run
    async def run(self, request: MLPipelineRequest) -> dict:
        workflow.logger.info(f"Starting ML pipeline {request.pipeline_id}")

        # Step 1: Validate data (CPU queue - lightweight validation)
        validation_result = await workflow.execute_activity(
            "validate_data",
            {"dataset_url": request.dataset_url},
            task_queue=STANDARD_CPU_QUEUE,
            start_to_close_timeout=timedelta(minutes=2),
        )

        if not validation_result["valid"]:
            return {"status": "validation_failed", "error": validation_result["error"]}

        # Step 2: Process large dataset (High-memory queue - loads full dataset into memory)
        processed_data = await workflow.execute_activity(
            "process_large_dataset",
            {"dataset_url": request.dataset_url, "pipeline_id": request.pipeline_id},
            task_queue=HIGH_MEMORY_QUEUE,
            start_to_close_timeout=timedelta(hours=1),
        )

        # Step 3: Train model (GPU queue - requires GPU for training)
        model_result = await workflow.execute_activity(
            "train_model",
            {
                "model_type": request.model_type,
                "data_path": processed_data["output_path"],
                "pipeline_id": request.pipeline_id,
            },
            task_queue=GPU_ML_QUEUE,
            start_to_close_timeout=timedelta(hours=4),
        )

        # Step 4: Generate embeddings (GPU queue - GPU-accelerated inference)
        embeddings = await workflow.execute_activity(
            "generate_embeddings",
            {
                "model_path": model_result["model_path"],
                "customer_id": request.customer_id,
            },
            task_queue=GPU_ML_QUEUE,
            start_to_close_timeout=timedelta(minutes=30),
        )

        # Step 5: Store results (CPU queue - simple I/O operation)
        await workflow.execute_activity(
            "store_results",
            {
                "pipeline_id": request.pipeline_id,
                "embeddings": embeddings,
                "model_metrics": model_result["metrics"],
            },
            task_queue=STANDARD_CPU_QUEUE,
            start_to_close_timeout=timedelta(minutes=5),
        )

        return {
            "status": "completed",
            "pipeline_id": request.pipeline_id,
            "model_path": model_result["model_path"],
            "embeddings_count": len(embeddings["vectors"]),
        }
```

**File: `activities.py`**

```python
"""Activity implementations for different resource requirements."""
from temporalio import activity
import asyncio

# ===== CPU-based activities (STANDARD_CPU_QUEUE) =====

@activity.defn
async def validate_data(data: dict) -> dict:
    """
    Validate input data (runs on standard CPU workers).

    Lightweight validation that doesn't require specialized hardware.
    """
    activity.logger.info(f"Validating dataset at {data['dataset_url']}")

    # Validation logic
    is_valid = True  # Simplified validation

    return {
        "valid": is_valid,
        "dataset_url": data["dataset_url"],
        "error": None if is_valid else "Invalid dataset format",
    }

@activity.defn
async def preprocess_data(data: dict) -> dict:
    """Basic data preprocessing (runs on CPU workers)."""
    activity.logger.info("Preprocessing data")

    # Lightweight preprocessing
    return {"preprocessed": True, "records": 1000}

@activity.defn
async def store_results(data: dict) -> dict:
    """Store pipeline results (runs on CPU workers)."""
    activity.logger.info(f"Storing results for pipeline {data['pipeline_id']}")

    # Store to database or object storage
    return {"stored": True, "pipeline_id": data["pipeline_id"]}

# ===== GPU-based activities (GPU_ML_QUEUE) =====

@activity.defn
async def train_model(data: dict) -> dict:
    """
    Train ML model (runs on GPU workers).

    Requires GPU for acceptable training performance.
    """
    activity.logger.info(f"Training {data['model_type']} model on GPU")

    import torch

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    activity.logger.info(f"Using device: {device}")

    # Simulate training
    # In production: load data, initialize model, train
    await asyncio.sleep(5)  # Simulate training time

    model_path = f"/models/{data['pipeline_id']}/model.pth"

    activity.logger.info(f"Training complete, model saved to {model_path}")

    return {
        "model_path": model_path,
        "metrics": {"accuracy": 0.95, "loss": 0.05},
        "device": str(device),
    }

@activity.defn
async def generate_embeddings(data: dict) -> dict:
    """
    Generate embeddings using ML model (runs on GPU workers).

    Requires GPU hardware for acceptable performance.
    """
    activity.logger.info("Generating embeddings on GPU")

    import torch
    from sentence_transformers import SentenceTransformer

    # Load model (cached after first load)
    model = SentenceTransformer("all-MiniLM-L6-v2")
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = model.to(device)

    # Generate embeddings (simplified for example)
    texts = [f"Sample text {i}" for i in range(100)]
    embeddings = model.encode(texts, convert_to_tensor=True)

    activity.logger.info(f"Generated {len(embeddings)} embeddings on {device}")

    return {
        "vectors": embeddings.cpu().tolist(),
        "customer_id": data["customer_id"],
        "dimension": embeddings.shape[1],
    }

@activity.defn
async def run_inference(data: dict) -> dict:
    """Run ML inference (runs on GPU workers)."""
    activity.logger.info("Running inference on GPU")

    import torch
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # Model inference logic
    activity.logger.info(f"Inference completed on {device}")

    return {"predictions": [], "confidence": 0.95, "device": str(device)}

# ===== High-memory activities (HIGH_MEMORY_QUEUE) =====

@activity.defn
async def process_large_dataset(data: dict) -> dict:
    """
    Process large dataset (runs on high-memory workers).

    Loads entire dataset into memory for processing.
    Requires 32-64GB RAM for typical datasets.
    """
    activity.logger.info(f"Processing large dataset from {data['dataset_url']}")

    # In production: load dataset (pandas, dask, spark)
    # Process in-memory transformations
    await asyncio.sleep(10)  # Simulate processing time

    output_path = f"/data/processed/{data['pipeline_id']}/dataset.parquet"

    activity.logger.info(f"Dataset processed, saved to {output_path}")

    return {
        "output_path": output_path,
        "records_processed": 10_000_000,
        "memory_used_gb": 45,
    }

@activity.defn
async def aggregate_analytics(data: dict) -> dict:
    """
    Aggregate analytics from large datasets (runs on high-memory workers).

    Performs in-memory aggregations on large datasets.
    """
    activity.logger.info("Running analytics aggregation")

    # In-memory aggregation logic
    await asyncio.sleep(5)

    return {
        "aggregations": {"total": 1000000, "average": 42.5},
        "memory_used_gb": 38,
    }

@activity.defn
async def build_large_index(data: dict) -> dict:
    """
    Build search index from large dataset (runs on high-memory workers).

    Requires significant memory to build index structures.
    """
    activity.logger.info("Building large search index")

    # Build index in memory
    await asyncio.sleep(8)

    return {
        "index_path": "/indexes/search_index.bin",
        "index_size_gb": 25,
        "documents_indexed": 50_000_000,
    }
```

**Starter example:**

```python
# start_ml_pipeline.py
"""Start ML pipeline workflow with resource-aware routing."""
import asyncio
from temporalio.client import Client
from ml_pipeline_workflow import MLPipelineWorkflow, MLPipelineRequest

async def main():
    client = await Client.connect("localhost:7233")

    pipeline_request = MLPipelineRequest(
        pipeline_id="ml-pipeline-2024-001",
        dataset_url="s3://datasets/customer-behavior-10m.parquet",
        model_type="transformer",
        customer_id="customer-456",
    )

    handle = await client.start_workflow(
        MLPipelineWorkflow.run,
        pipeline_request,
        id=f"ml-pipeline-{pipeline_request.pipeline_id}",
        task_queue="workflows",
    )

    print(f"Started ML pipeline workflow: {handle.id}")

    # Wait for result (or use handle.describe() to check status)
    result = await handle.result()
    print(f"Pipeline completed: {result}")

if __name__ == "__main__":
    asyncio.run(main())
```

## Conclusion

By implementing separate Task Queues for resource-based routing, you have achieved:

1. **Cost optimization:** GPU instances (g4dn.xlarge at $0.526/hr) only handle ML training and inference, while standard CPU Activities run on cost-effective instances (c5.xlarge at $0.17/hr), and high-memory analytics run on memory-optimized instances (r5.2xlarge at $0.504/hr). This targeted resource allocation reduces infrastructure costs by 60-80% compared to running everything on GPU instances.

2. **Resource efficiency:** GPU Workers run at 2-4 concurrent activities (constrained by GPU memory), CPU Workers run at 100+ concurrent activities, and high-memory Workers run at 10 concurrent activities (constrained by RAM). Each Worker pool is optimized for its specific resource constraints.

3. **Independent scaling:** Scale GPU Workers based on ML workload demand, high-memory Workers based on analytics load, and CPU Workers based on standard workload volume, without coordination between pools. Each pool can scale independently based on its queue depth and resource utilization.

4. **Environment isolation:** ML Workers with TensorFlow/PyTorch/CUDA dependencies are isolated from standard Workers, preventing library conflicts and simplifying deployment. Each Worker type can use specialized container images with only the dependencies it needs.

Your application now intelligently routes work based on resource requirements, ensuring efficient hardware utilization and cost-effective infrastructure allocation.

## Related resources

### Official documentation
- [Temporal Documentation - Task Routing](/task-routing)
- [Temporal Best Practices - Separate Task Queues](/best-practices/worker#separate-task-queues-logically)
- [Python SDK Worker Configuration](https://python.temporal.io/temporalio.worker.Worker.html)

### Related patterns
- [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue) - Pattern catalog reference for routing Activities to a specific Worker via a dedicated Task Queue
- [Separate Task Queues - Worker Affinity](worker-execution-affinity) - For Activities that must run on the same Worker instance
- [Separate Task Queues - Rate Limiting](rate-limit-downstream-apis) - For protecting downstream APIs with rate limits

### Code samples
- [GPU-Accelerated Image Processing (Python)](https://github.com/temporalio/samples-python) - Example of GPU worker configuration
- [Worker Versioning with Task Queues](https://github.com/temporalio/samples-python/tree/main/worker_versioning) - Managing different Worker versions

### Community resources
- [Blog: AI, ML, and Data Engineering Workflows with Temporal](https://temporal.io/blog/ai-ml-and-data-engineering-workflows-with-temporal)
- [Forum: Task Queue Best Practices](https://community.temporal.io/t/in-what-situation-should-we-use-multiple-separated-task-queues/1254)
- [Temporal ML/AI Use Cases](https://temporal.io/use-cases/machine-learning)

---

# Temporarily increase Namespace capacity on Temporal Cloud

Source: https://docs.temporal.io/guides/temporary-rate-limit-increases

> Add provisioned capacity to a Namespace for a load spike and remove it after a set time, using a Go Workflow and a Child Workflow with a durable Timer.

**by Taylor Khan**

This pattern provides the steps to dynamically provision and automatically deprovision capacity on Temporal Cloud.

Applications often experience predictable or temporary spikes in throughput that require additional Temporal Cloud capacity. This pattern demonstrates how to grant time-limited increases to your capacity limits and guarantee release of those resources after a specific duration.

Permanently allocating peak capacity requirements for a namespace results in unnecessary costs. However, manually adjusting capacity quotas before and after load spikes is prone to human error, risking either workflow throttling if limits are not raised in time, or runaway costs if operators forget to reduce limits after the workload subsides.

You will use Temporal Workflows to orchestrate the provisioning process. A parent Workflow executes an Activity to raise the capacity limit, then starts an asynchronous Child Workflow configured with an abandon policy. The parent Workflow completes to unblock the client, while the Child Workflow waits for a designated duration before executing an Activity to revert the capacity to its original limit.

By completing this pattern, you will:

- Automate capacity management to reduce Temporal Cloud costs.
- Guarantee the execution of cleanup operations using durable Timers.
- Unblock client requests while long-running deprovisioning tasks continue in the background.

## Background and best practices

Temporal Cloud offers different capacity modes to accommodate varying workloads. When using provisioned capacity, you define the maximum throughput your Namespace can consume. Modifying this limit via an API call allows you to scale resources up for intensive tasks.

The primary architectural challenge is ensuring that temporary capacity increases revert reliably. Temporal solves this by persisting the state of your Workflow, including durably storing and starting Timers.

By separating the provisioning and deprovisioning steps into distinct Workflows, you adhere to the best practice of returning control to the caller as soon as the provisioning command succeeds. The `ParentClosePolicyAbandon` setting instructs the Temporal Service to allow the Child Workflow to continue running even after the parent completes. This decouples the client's synchronous request from the long-running *Time-To-Live (TTL)* wait state. When the Child Workflow enters a sleep state, it consumes no Worker memory, allowing you to scale this pattern across thousands of concurrent requests efficiently.

**Note:** Latency to the caller can be further reduced via [the Early Return pattern paired with an SDK call to update-with-start](/sending-messages#update-with-start).

## Target audience

This guide references the following roles:

- **Temporal Workflow and Activity developers**: Implement the Go code for the Workflows and Activities and deploy the Worker processes.
- **Platform operators**: Maintain the Temporal Namespace configuration and monitor usage against provisioned capacity limits.

## Prerequisites

To execute the steps in this pattern, you must have:

- **Required software, infrastructure, and tools:** Temporal Go SDK v1.40.0 or later, Go v1.23 or later, Temporal CLI v.1.6.1 or later.
- **Resources & Access Privileges:** Temporal Cloud Account with Admin role for the target Namespace to access the Cloud Operations API.
    - You will need to create an API key using your Temporal Cloud account to access the Cloud Operations API
- **Required Concepts:**
    - Familiarity with
        - Temporal Workflows
        - Temporal Activities
        - Temporal Child Workflows
        - Temporal Timers
        - Temporal Workers
        - Temporal Cloud Namespaces

**Note:** This pattern uses the Temporal Go SDK and it is necessary for direct replication. However, any Temporal SDK supported by Temporal Cloud is sufficient to achieve the same outcomes outlined in this document.

## People & process considerations

### Platform operators

The **Platform operators** own the Temporal Cloud Namespace and API keys. This team is responsible for the following:

1. Generate and manage Temporal Cloud API keys required for the provisioning Activities.
2. Monitor overall throughput and ensure the base provisioned capacity meets everyday operational needs.

### Application developers

The **Application developers** are the primary authors of the Temporal Workflows and Activities. This team will have the following responsibilities:

1. Implement and test the Temporal code outlined in this pattern.
2. Configure Worker scaling policies to handle the expected frequency of capacity adjustment requests.

## Architecture diagram

![Architecture diagram for temporary rate limit increases](https://docs.temporal.io/guides/assets/temporary-rate-limit-increases/orchestrate-temporary-rate-limit-increases.svg)

The following diagram illustrates the flow of the requests to provision capacity and handle automatic cleanup after a 60 minute time-to-live.

1. A client application sends a gRPC request to the `my-service` application with the requested limit and namespace.
2. The `my-service` application uses the Temporal SDK client to start the `provisionTRU` Workflow.
3. The `provisionTRU` Workflow schedules the `addTRUs` Activity and waits for it to complete.
4. The `provisionTRU` Workflow starts the `deprovisionTRU` asynchronous Child Workflow using a parent close policy of abandon.
5. The `provisionTRU` Workflow completes, and `my-service` returns a success response to the client.
6. The `deprovisionTRU` Child Workflow sets a Timer for 60 minutes.
7. The Timer fires after the TTL expires.
8. The `deprovisionTRU` Child Workflow schedules the `removeTRUs` Activity to revert the capacity limits.

## Implementation plan

In this implementation plan, you will build and verify the capacity management solution. You will begin by defining the required Activities to interact with the Temporal Cloud. Next, you will create the asynchronous Child Workflow responsible for the delayed cleanup. You will then orchestrate the process with the parent Workflow and expose it through a service handler. Finally, you will manually trigger the Workflow to simulate a client request and write tests using the Temporal Go SDK test suite to validate your orchestration logic locally.

Prior to executing this plan, ensure you have your Temporal Cloud API key and your target Namespace name documented

### Define messages for Activities and Workflows

Create the messages for passing to Temporal Workflows and Activities. It is a best practice to use a single serializable input to Workflows and Activities. You will create one for each Activity and Workflow, and helper functions for generating unique identifiers for each capacity request.

```go
// capacity/messages.go
package capacity

import "fmt"

type ProvisionTRUInput struct {
  Namespace          string
  APSLimit           int32
  MinutesToProvision int32
}

type DeprovisionTRUInput struct {
  Namespace          string
  MinutesToProvision int32
}

type AddTRUInput struct {
  Namespace string
  APSLimit  int32
}

type RemoveTRUInput struct {
  Namespace string
}

func generateProvisioningId(namespace string) string {
  return fmt.Sprintf("provision-%s", namespace)
}

func generateDeprovisioningId(namespace string) string {
  return fmt.Sprintf("deprovision-%s", namespace)
}

```

### Define the Activities

You must create functions that interact with the Temporal Cloud Operations API to adjust capacity. Temporal Activities encapsulates these functions to guarantee retries in case of intermittent failures. 

```go
// capacity/activities.go
package capacity

import (
  "bytes"
  "context"
  "encoding/json"
  "fmt"
  "go.temporal.io/sdk/temporal"
  "io"
  "net/http"
  "strconv"
)

const (
  defaultBaseURL = "https://saas-api.tmprl.cloud"
)

// Activities holds the dependencies required by the provisioning activities.
type Activities struct {
  HTTPClient *http.Client
  BaseURL    string
  APIKey     string
}

type getNamespaceResponse struct {
  Namespace struct {
    Spec            json.RawMessage `json:"spec"`
    ResourceVersion string          `json:"resourceVersion"`
  } `json:"namespace"`
}

type updateNamespaceRequest struct {
  Spec            json.RawMessage `json:"spec"`
  ResourceVersion string          `json:"resourceVersion"`
}

func (a *Activities) baseURL() string {
  if a.BaseURL != "" {
    return a.BaseURL
  }
  return defaultBaseURL
}

func (a *Activities) httpClient() *http.Client {
  if a.HTTPClient != nil {
    return a.HTTPClient
  }
  return http.DefaultClient
}

func (a *Activities) getNamespace(ctx context.Context, namespace string) (json.RawMessage, string, error) {
  url := fmt.Sprintf("%s/cloud/namespaces/%s", a.baseURL(), namespace)
  req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
  if err != nil {
    return nil, "", err
  }
  req.Header.Set("Authorization", "Bearer "+a.APIKey)

  resp, err := a.httpClient().Do(req)
  if err != nil {
    return nil, "", err
  }
  defer resp.Body.Close()

  body, err := io.ReadAll(resp.Body)
  if err != nil {
    return nil, "", err
  }
  if resp.StatusCode != http.StatusOK {
    msg := fmt.Sprintf("%s", body)
    return nil, "", temporal.NewApplicationError(msg, strconv.Itoa(resp.StatusCode))
  }
  var result getNamespaceResponse
  if err := json.Unmarshal(body, &result); err != nil {
    return nil, "", err
  }
  return result.Namespace.Spec, result.Namespace.ResourceVersion, nil
}

func (a *Activities) updateNamespace(ctx context.Context, namespace string, spec json.RawMessage, resourceVersion string) error {
  url := fmt.Sprintf("%s/cloud/namespaces/%s", a.baseURL(), namespace)

  payload, err := json.Marshal(updateNamespaceRequest{
    Spec:            spec,
    ResourceVersion: resourceVersion,
  })
  if err != nil {
    return err
  }

  req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
  if err != nil {
    return err
  }
  req.Header.Set("Authorization", "Bearer "+a.APIKey)
  req.Header.Set("Content-Type", "application/json")

  resp, err := a.httpClient().Do(req)
  if err != nil {
    return err
  }
  defer resp.Body.Close()

  if resp.StatusCode != http.StatusOK {
    body, _ := io.ReadAll(resp.Body)
    msg := fmt.Sprintf("%s", body)
    return temporal.NewApplicationError(msg, strconv.Itoa(resp.StatusCode))
  }
  return nil
}

// AddTRUs increases the provisioned capacity of the target namespace to the requested APS limit.
func (a *Activities) AddTRUs(ctx context.Context, input AddTRUInput) error {
  spec, resourceVersion, err := a.getNamespace(ctx, input.Namespace)
  if err != nil {
    return fmt.Errorf("get namespace %q: %w", input.Namespace, err)
  }

  var specMap map[string]interface{}
  if err := json.Unmarshal(spec, &specMap); err != nil {
    return fmt.Errorf("unmarshal namespace spec: %w", err)
  }
  specMap["capacitySpec"] = map[string]interface{}{
    "provisioned": map[string]interface{}{
      // 1 TRU = 500 APS; convert the requested APS limit to TRUs.
      "value": float64(input.APSLimit) / 500.0,
    },
  }

  updatedSpec, err := json.Marshal(specMap)
  if err != nil {
    return fmt.Errorf("marshal updated spec: %w", err)
  }

  if err := a.updateNamespace(ctx, input.Namespace, updatedSpec, resourceVersion); err != nil {
    return err
  }
  return nil
}

// RemoveTRUs reverts the target namespace to on-demand capacity mode.
func (a *Activities) RemoveTRUs(ctx context.Context, input RemoveTRUInput) error {
  spec, resourceVersion, err := a.getNamespace(ctx, input.Namespace)
  if err != nil {
    return err
  }

  var specMap map[string]interface{}
  if err := json.Unmarshal(spec, &specMap); err != nil {
    return fmt.Errorf("unmarshal namespace spec: %w", err)
  }
  specMap["capacitySpec"] = map[string]interface{}{
    "onDemand": map[string]interface{}{},
  }

  updatedSpec, err := json.Marshal(specMap)
  if err != nil {
    return fmt.Errorf("marshal updated spec: %w", err)
  }

  if err := a.updateNamespace(ctx, input.Namespace, updatedSpec, resourceVersion); err != nil {
    return err
  }
  return nil
}
```

The `AddTRUs` and `RemoveTRUs` methods define the actions taken to alter the capacity. By grouping these as methods on a `Activities` struct, you can inject dependencies like API clients or logger instances. Both methods accept a struct, which contains the target namespace and the desired limit.

## Implement the deprovisioning Child Workflow

To ensure capacity reverts after the given duration without consuming active computing resources, you will define a Workflow that sleeps for the specified duration and then executes the Activity that deprovisions the capacity.

```go
// capacity/deprovision_workflow.go
package capacity

import (
  "go.temporal.io/sdk/temporal"
  "time"

  "go.temporal.io/sdk/workflow"
)

// DeprovisionTRUWorkflow sleeps for the TTL duration then reverts the namespace
// to on-demand capacity. It is started as an asynchronous Child Workflow by
// ProvisionTRUWorkflow and runs independently after the parent completes.
func DeprovisionTRUWorkflow(ctx workflow.Context, input DeprovisionTRUInput) error {
  err := workflow.Sleep(ctx, time.Duration(input.MinutesToProvision)*time.Minute)
  if err != nil {
    return err
  }

  ao := workflow.ActivityOptions{
    StartToCloseTimeout: 2 * time.Minute,
    RetryPolicy: &temporal.RetryPolicy{
      NonRetryableErrorTypes: []string{unauthorized, forbidden, badRequest},
    },
  }
  ctx = workflow.WithActivityOptions(ctx, ao)

  var a *Activities
  return workflow.ExecuteActivity(ctx, a.RemoveTRUs, input).Get(ctx, nil)
}
```

## Implement the provisioning Workflow

You will now create the main orchestration Workflow. This Workflow applies the capacity increase, and sets the deprovisioning process to run with an "abandon policy," so it will run regardless of what happens to the parent Workflow.

```go
// capacity/provision_workflow.go
package capacity

import (
  "go.temporal.io/sdk/temporal"
  "net/http"
  "strconv"
  "time"

  "go.temporal.io/api/enums/v1"
  "go.temporal.io/sdk/workflow"
)

var (
  unauthorized = strconv.Itoa(http.StatusUnauthorized)
  forbidden    = strconv.Itoa(http.StatusForbidden)
  badRequest   = strconv.Itoa(http.StatusBadRequest)
)

// ProvisionTRUWorkflow raises the namespace capacity limit, then starts the
// DeprovisionTRUWorkflow as an asynchronous Child Workflow with an abandon
// policy so the parent can return immediately while cleanup runs independently.
func ProvisionTRUWorkflow(ctx workflow.Context, input ProvisionTRUInput) error {
  ao := workflow.ActivityOptions{
    StartToCloseTimeout: 2 * time.Minute,
    RetryPolicy: &temporal.RetryPolicy{
      NonRetryableErrorTypes: []string{unauthorized, forbidden, badRequest},
    },
  }
  actCtx := workflow.WithActivityOptions(ctx, ao)

  var a *Activities
  err := workflow.ExecuteActivity(actCtx, a.AddTRUs, AddTRUInput{
    Namespace: input.Namespace,
    APSLimit:  input.APSLimit,
  }).Get(ctx, nil)
  if err != nil {
    return err
  }

  cwo := workflow.ChildWorkflowOptions{
    WorkflowID:        generateDeprovisioningId(input.Namespace),
    ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON,
  }
  childCtx := workflow.WithChildOptions(ctx, cwo)

  childFuture := workflow.ExecuteChildWorkflow(childCtx, DeprovisionTRUWorkflow, DeprovisionTRUInput{
    Namespace:          input.Namespace,
    MinutesToProvision: input.MinutesToProvision,
  })

  var childExec workflow.Execution
  err = childFuture.GetChildWorkflowExecution().Get(ctx, &childExec)
  if err != nil {
    return err
  }

  return nil
}
```

The `ProvisionTRUWorkflow` orchestrates the entire request. First, it synchronously waits for `AddTRUs` to complete to ensure the capacity is available before returning. Second, it configures `workflow.ChildWorkflowOptions` with `enums.PARENT_CLOSE_POLICY_ABANDON`. This prevents the Temporal Service from cancelling the Child Workflow when this parent completes. Finally, the code uses `GetChildWorkflowExecution().Get()` to block execution just until the Temporal Service confirms the Child Workflow has started. Once started, the parent completes, leaving the child to run independently. 

## Implement the service handler

Connect the Workflow to your application's API layer. You will write the code that receives the incoming request and triggers the Workflow.

```go
// capacity/handler.go
package capacity

import (
  "context"
  "errors"
  "fmt"
  "go.temporal.io/sdk/client"
)

const TaskQueue = "capacity-management"

// HandleProvisionRequest starts a ProvisionTRUWorkflow execution and returns
// after the Temporal Cluster accepts the request. It does not wait
// for the Workflow to complete.
func HandleProvisionRequest(c client.Client, namespace string, apsLimit int32, minutesToProvision int32) error {
  id := generateProvisioningId(namespace)
  preExistingRun := c.GetWorkflow(context.Background(), id, "").GetRunID()
  if preExistingRun != "" {
    return errors.New("provisioning request already in-progress")
  }
  deprovisionId := fmt.Sprintf("deprovision-%s", namespace)
  preExistingDeprovisionRun := c.GetWorkflow(context.Background(), deprovisionId, "").GetRunID()
  if preExistingDeprovisionRun != "" {
    err := c.CancelWorkflow(context.Background(), deprovisionId, preExistingDeprovisionRun)
    if err != nil {
      return errors.Join(errors.New("unable to cancel deprovisioning workflow"), err)
    }
  }
  options := client.StartWorkflowOptions{
    ID:        id,
    TaskQueue: TaskQueue,
  }

  input := ProvisionTRUInput{
    Namespace:          namespace,
    APSLimit:           apsLimit,
    MinutesToProvision: minutesToProvision,
  }

  we, err := c.ExecuteWorkflow(context.Background(), options, ProvisionTRUWorkflow, input)
  if err != nil {
    return fmt.Errorf("failed to start workflow: %w", err)
  }

  fmt.Printf("Successfully started provision workflow. ID: %s, RunID: %s\n",
    we.GetID(), we.GetRunID())
  return nil
}
```

The `HandleProvisionRequest` function represents your gRPC or HTTP endpoint. It prepares the `client.StartWorkflowOptions` and calls `ExecuteWorkflow`. Because it uses `ExecuteWorkflow` rather than waiting on the result with `.Get()`, the function returns an acknowledgment to the client after the Temporal Service accepts the Workflow initiation. `MinutesToProvision` determines how long the `DeprovisionTRUWorkflow`  waits before provisioned capacity reverts.

## Create the Worker Program

Now that you have built the Activities and Workflows here is the complete set of imports and definitions required to register and run the system on a Worker.

```go
// cmd/worker.go
package main

import (
  "github.com/temporal-sa/temporary-rate-limit-increases/capacity"
  "go.temporal.io/sdk/client"
  "go.temporal.io/sdk/worker"
  "log"
  "os"
)

func main() {
  apiKey := os.Getenv("TEMPORAL_CLOUD_API_KEY")
  if apiKey == "" {
    log.Fatalln("TEMPORAL_CLOUD_API_KEY missing and required")
  }

  c, err := client.Dial(client.Options{})
  if err != nil {
    log.Fatalln("Unable to create Temporal client", err)
  }
  defer c.Close()

  w := worker.New(c, capacity.TaskQueue, worker.Options{})

  w.RegisterWorkflow(capacity.ProvisionTRUWorkflow)
  w.RegisterWorkflow(capacity.DeprovisionTRUWorkflow)

  activities := &capacity.Activities{
    APIKey: apiKey,
  }
  w.RegisterActivity(activities)

  err = w.Run(worker.InterruptCh())
  if err != nil {
    log.Fatalln("Unable to start worker", err)
  }
}
```

Run the following commands sequentially, each in their own terminal:

```bash
temporal server start-dev
```

```bash
export TEMPORAL_CLOUD_API_KEY={YOUR_TEMPORAL_CLOUD_API_KEY}
go run cmd/worker.go
```

## Test the Workflow execution

To verify the capacity provisioning process, you must simulate the client request. You will write a short script that initializes a Temporal client, defines the requested limits, and calls your service handler to trigger the parent Workflow. Replace `{TEST_NAMESPACE}` with the target namespace you identified in the prerequisites section.

```go
// cmd/trigger.go
package main

import (
  "github.com/temporal-sa/temporary-rate-limit-increases/capacity"
  "go.temporal.io/sdk/client"
  "log"
  "os"
  "strconv"
)

func main() {
  c, err := client.Dial(client.Options{})
  if err != nil {
    log.Fatalln("Unable to create Temporal client", err)
  }
  defer c.Close()

  targetNamespace := os.Getenv("TEMPORAL_CLOUD_NAMESPACE")
  if targetNamespace == "" {
    log.Fatalln("TEMPORAL_CLOUD_NAMESPACE missing and required")
  }
  minutesToProvisionRaw := os.Getenv("MINUTES_TO_PROVISION")
  if minutesToProvisionRaw == "" {
    log.Fatalln("MINUTES_TO_PROVISION missing and required")
  }
  minutesToProvision, err := strconv.Atoi(minutesToProvisionRaw)
  if err != nil {
    log.Fatalln("Unable to parse MINUTES_TO_PROVISION: " + err.Error())
  }
  var newLimit int32 = 1000

  err = capacity.HandleProvisionRequest(c, targetNamespace, newLimit, int32(minutesToProvision))
  if err != nil {
    log.Fatalln("Unable to execute provision request", err)
  }
}
```

The `main` function connects to the Temporal Service using `client.Dial`. It defines the `targetNamespace` and `newLimit` variables to represent the data payload that a gRPC request would contain. It then passes the client and these variables to the `HandleProvisionRequest` function. By executing this file while your Worker is running, the Temporal Service will start the `ProvisionTRUWorkflow**.**`

Execute the script from your command line:

```bash
export MINUTES_TO_PROVISION=5
export TEMPORAL_CLOUD_NAMESPACE={YOUR_TEMPORAL_NAMESPACE}
go run trigger.go 
```

The script will complete and print the confirmation from the handler:

```bash
Started provision workflow. ID: provision-production-workload, RunID: {RUN_ID}
```

After this output appears, the Worker continues to process the provisioning Activity and the 5-minute sleep Timer in the background. 

**Note:** a 5-minute sleep time is used for testing the Workflow end-to-end. Set `MINUTES_TO_PROVISION` to your needs in production environments. We recommend at least 60 minutes.

## Unit test the Workflow

Verify that the Workflow orchestrates the Activities and Child Workflow without interacting with the live Temporal Service. The Temporal Go SDK provides a test environment that simulates time and cluster behavior, allowing you to validate your code logic locally.

```go
// capacity/provision_workflow_test.go
package capacity

import (
  "testing"

  "github.com/stretchr/testify/mock"
  "github.com/stretchr/testify/require"
  "go.temporal.io/sdk/testsuite"
)

func Test_ProvisionTRUWorkflow(t *testing.T) {
  testSuite := &testsuite.WorkflowTestSuite{}
  env := testSuite.NewTestWorkflowEnvironment()

  env.RegisterWorkflow(DeprovisionTRUWorkflow)

  var a *Activities
  env.OnActivity(a.AddTRUs, mock.Anything, mock.Anything).Return(nil)

  input := ProvisionTRUInput{
    Namespace:          "test-namespace",
    APSLimit:           2000,
    MinutesToProvision: 5,
  }

  env.ExecuteWorkflow(ProvisionTRUWorkflow, input)

  require.True(t, env.IsWorkflowCompleted())
  require.NoError(t, env.GetWorkflowError())
  env.AssertExpectations(t)
}
```

The `Test_ProvisionTRUWorkflow` function initializes a `testsuite.WorkflowTestSuite` and creates a `TestWorkflowEnvironment`. This environment simulates the Temporal Service in memory. Because the parent Workflow starts a Child Workflow asynchronously, you must register `DeprovisionTRUWorkflow` with the test environment so it does not fail when invoked. 

You then use `env.OnActivity` to mock the `AddTRUs` Activity, instructing it to return `nil` without making real network calls. After calling `env.ExecuteWorkflow`, the test uses `require.True` and `require.NoError` to guarantee the parent Workflow completes. Finally, `env.AssertExpectations` confirms that the mocked Activity executed as defined.

Execute the test from your command line: 

```bash
go test -v provision_workflow_test.go provision_workflow.go deprovision_workflow.go activities.go
```

The terminal will output the test results:

```bash
=== RUN   Test_ProvisionTRUWorkflow
--- PASS: Test_ProvisionTRUWorkflow (0.01s)
PASS
```

Additional log lines from the test results are a non-issue.

## Unit test the deprovisioning Workflow

When testing long-running Workflows, waiting for real time to pass is inefficient. You must verify that the `DeprovisionTRUWorkflow` schedules the `RemoveTRUs` Activity after the 60-minute Timer without actually waiting an hour during your test execution.

```go
// capacity/deprovision_workflow_test.go
package capacity

import (
  "testing"

  "github.com/stretchr/testify/mock"
  "github.com/stretchr/testify/require"
  "go.temporal.io/sdk/testsuite"
)

func Test_DeprovisionTRUWorkflow(t *testing.T) {
  testSuite := &testsuite.WorkflowTestSuite{}
  env := testSuite.NewTestWorkflowEnvironment()

  var a *Activities
  env.OnActivity(a.RemoveTRUs, mock.Anything, mock.Anything).Return(nil)

  input := DeprovisionTRUInput{
    Namespace:          "test-namespace",
    MinutesToProvision: 5,
  }

  env.ExecuteWorkflow(DeprovisionTRUWorkflow, input)

  require.True(t, env.IsWorkflowCompleted())
  require.NoError(t, env.GetWorkflowError())
  env.AssertExpectations(t)
}
```

The `Test_DeprovisionTRUWorkflow` function uses the `testsuite.WorkflowTestSuite` to run the Child Workflow. When a Timer blocks a Workflow execution the Temporal test environment automatically skips time forward, such as the `workflow.Sleep` command used in your code. This mechanism causes the 60-minute sleep to complete instantly within the test framework.

You configure `env.OnActivity` to mock the `RemoveTRUs` Activity, ensuring the test does not make live API calls to Temporal Cloud. After execution, `require.True` confirms the Workflow finished, and `env.AssertExpectations` guarantees that the deprovisioning Activity executed as expected after the simulated time elapsed. 

Execute the test from your command line:

```bash
go test -v deprovision_workflow_test.go deprovision_workflow.go activities.go
```

The terminal will output the test results:

```bash
=== RUN   Test_DeprovisionTRUWorkflow
--- PASS: Test_DeprovisionTRUWorkflow (0.01s)
PASS
ok      command-line-arguments  0.015s
```

Additional log lines from the test results are a non-issue.

## Outcomes

By following this guide, you have implemented a time-bound capacity provisioning system including:

- Requesting increased limits through a retryable Temporal Activity.
- Using an asynchronous Child Workflow to decouple application responses from long-term cleanup tasks.
- Using durable Timers to guarantee that capacity limits are strictly reverted after 60 minutes, ensuring you do not overpay for unused compute.

## Related resources

- [Temporal Capacity Modes](/cloud/capacity-modes)
- [Temporal Best Practices](/best-practices)
- [Python Capacity Mode Sample](https://github.com/lainecsmith/temporal-cloud-ops-capacity-modes)
- [Go Capacity Mode Sample](https://github.com/temporal-sa/temporary-rate-limit-increases)

---

# Run all of a Workflow's Activities on the same Worker

Source: https://docs.temporal.io/guides/worker-execution-affinity

> Keep downloaded files, loaded ML models, and open connections on one machine by routing each Activity to a Worker-specific Task Queue, with Python code.

**by Cecil Phillip**

## Problem statement

Many Workflows require multiple Activities to execute on the same Worker to maintain data locality. Common scenarios include:

- **File processing:** Download a file (Activity 1), process it (Activity 2), and upload it (Activity 3). The file exists on the Worker's local disk, and requiring another Worker to re-download multi-GB files from the original storage (or via an object store) is slow and expensive.
- **ML model caching:** Load a large ML model into memory (Activity 1), then run multiple inference calls (Activities 2-N) using the cached model. Loading the model on each Worker wastes time and memory.
- **Database connection pooling:** Establish expensive database connections that should be reused across multiple Activities in the same Workflow Execution.

Without Worker affinity, Temporal distributes Activities across available Workers. This means:
- Files downloaded in Activity 1 aren't available for Activity 2 (different Worker)
- Network transfer overhead: 10GB video file must be re-fetched from the original storage or an object store (incurs large network transfer and egress costs)
- Duplicate resource initialization: ML models loaded multiple times across Workers
- Increased latency: Each Activity pays setup costs instead of reusing resources

## Solution

Use Worker-specific Task Queues to ensure all Activities in a Workflow execute on the same Worker. To achieve this:

1. Each Worker polls two Activity Task Queues: a shared queue and a unique queue (generated per Worker instance)
2. The Workflow calls an Activity on the shared queue to discover an available Worker's unique queue name
3. The Workflow routes all subsequent Activities to that Worker's unique queue
4. All Activities execute on the same Worker, maintaining data locality

## Outcomes

- **Data locality:** Files downloaded in one Activity are immediately available to subsequent Activities on the same Worker
- **Performance:** Eliminate repeated re-downloads from remote storage (multi-GB transfers)
- **Resource efficiency:** Load expensive resources (ML models, DB connections) once per Workflow instead of per Activity
- **Cost reduction:** Reduce network egress costs from repeated downloads from remote/object storage

## Background and best practices

### Task Queue fundamentals

Task Queues in Temporal are dynamically created when first referenced. With this pattern a unique Task Queue is created per Worker instance (for example, `file-processing-abc123`) that only that Worker polls.

**Recommended practice:** Generate unique queue names using UUIDs (or the hostname if running in a containerized environment) to avoid collisions across Worker instances.

### Worker-specific vs Worker sessions

- **Go SDK** has a built-in [Worker Sessions API](/develop/go/workers/sessions) that handles Worker-specific routing automatically
- **Other SDKs** (such as Python and TypeScript) must implement the pattern manually using unique Task Queue names

This pattern provides the same guarantees as Go's Sessions API for non-Go SDKs.

### Worker failure handling

If a Worker crashes while processing Activities on its unique queue:
- Running activities will timeout after `heartbeat_timeout` (if configured) or `start_to_close_timeout`
- Retries and pending activities will wait in the unique queue until `schedule_to_start_timeout` expires
- To recover, the Workflow catches the timeout error and can route to a different Worker (on a new unique task queue)

**Recommended practice:** Set a short `heartbeat_timeout` (for example, 30s) to detect crashes quickly, and a short `schedule_to_start_timeout` (for example, 1m) to stop waiting on dead queues.

**Recommended practice:** Set appropriate `schedule_to_start_timeout` values to detect Worker failures quickly (for example, 5 minutes for file processing).

> Activity Executions in most Workflows are constrained by the Start-to-Close Timeout, which limits the maximum duration of a single attempt. Its value is set to slightly longer than the Activity should take to complete. The pattern described here also relies on the Schedule-to-Start Timeout, which limits the maximum amount of time that a Task may remain enqueued. Although otherwise seldom used, this Timeout is valuable here because it enables the system to detect a Worker crash. That is, when a Worker crashes, it will no longer dequeue Tasks and the Schedule-to-Start Timeout will be reached.

### Operational considerations

> **⚠️ Caution:**
> Worker Versioning
>
> Do not enable [Worker Versioning](/production-deployment/worker-deployments/worker-versioning#definition) for worker-specific Task Queues — it will unnecessarily exhaust the [per-version Task Queue limit](/evaluate/cloud/limits#max-task-queues-in-deployment-version-limits). A per-worker Task Queue's lifetime is tied to the Worker itself. When the Worker is updated to a new version, it gets a new Task Queue and the old one is gone. So versioning this queue does not add any value.
>

- **Queue proliferation:** Each Worker creates a unique queue. Monitor total queue count.
- **Queue lifecycle:** Consider setting a reasonable `schedule_to_close_timeout` on Activities to bound how long work tied to a unique queue can remain active, or implement explicit cleanup job to remove stale unique queues when Workers terminate.
- **Worker scaling:** When scaling down, ensure Workers complete in-progress Workflows before [termination](/encyclopedia/workers/worker-shutdown)
- **Monitoring:** Track Activities stuck in unique queues (indicates Worker crash/unavailability)

## Target audience

- **Temporal Workflow & Activity developers:** Implementing file processing and data-local Workflows
- **Platform operators:** Deploying and monitoring Worker-specific queue patterns
- **Data engineers:** Building ETL pipelines with large file processing
- **ML Engineers:** Deploying inference Workflows with model caching

This implementation requires code changes to Workers and Workflows, and consideration for Worker lifecycle management.

## Prerequisites

### Required software, infrastructure, and tools

- Temporal Service (Self-hosted or Temporal Cloud)
- Python 3.8 or later
- Temporal Python SDK v1.0.0 or later (`pip install temporalio`)
- File storage accessible to Workers (local disk, shared filesystem, or cloud storage)

### Resources & access privileges

- Temporal Namespace with permissions to start Workflows and register Workers
- File storage with appropriate read/write permissions for Workers
- Sufficient disk space on Worker hosts for file processing

### Required concepts

- Temporal Workflows, Activities, and Task Queues
- Python async/await patterns
- File I/O operations
- Basic understanding of Worker lifecycle

## Architecture diagram(s)

### Worker-specific Task Queue pattern

```mermaid
sequenceDiagram
    participant WF as Workflow
    participant Shared as file-processing-shared<br/>Task Queue
    participant W1 as Worker 1<br/>(UUID: abc123)
    participant Unique as file-processing-abc123<br/>Task Queue
    participant Disk as Worker 1<br/>Local Disk

    Note over WF: Need to process file<br/>on same worker

    WF->>Shared: get_available_task_queue()
    Shared->>W1: Activity dispatched
    W1-->>WF: "file-processing-abc123"

    Note over WF: Route all activities<br/>to Worker 1's unique queue

    WF->>Unique: download_file(url)
    Unique->>W1: Routed to Worker 1
    W1->>Disk: Save file to /tmp/video.mp4
    Note over Disk: 10GB video file
    W1-->>WF: "/tmp/video.mp4"

    WF->>Unique: process_file(path)
    Unique->>W1: Same worker
    W1->>Disk: Read /tmp/video.mp4
    Note over W1: Process locally<br/>No network transfer!
    W1->>Disk: Write /tmp/video-processed.mp4
    W1-->>WF: "/tmp/video-processed.mp4"

    WF->>Unique: upload_file(path)
    Unique->>W1: Same worker
    W1->>Disk: Read /tmp/video-processed.mp4
    Note over W1: Upload to S3
    W1->>Disk: Clean up files
    W1-->>WF: "https://s3.../video.mp4"
```

## Implementation

### Step 1: Define Task Queue constants

**File: `task_queues.py`**

```python
"""Task Queue constants for worker-specific routing."""

# Shared queue for discovering available workers
FILE_PROCESSING_SHARED_QUEUE = "file-processing-shared"

# Note: Unique per-worker queues are generated at runtime
# Pattern: f"file-processing-{uuid.uuid4()}"
```

### Step 2: Configure Worker with shared and unique queues

Each Worker polls two queues:
1. Shared queue: Returns this Worker's unique queue name
2. Unique queue: Handles file processing Activities

**File: `worker_file_processing.py`**

```python
"""Worker with unique task queue for file processing affinity."""
import asyncio
import hashlib
import httpx
import uuid
import logging
import os
from pathlib import Path
from temporalio.client import Client
from temporalio.exceptions import ApplicationError
from temporalio.worker import Worker
from temporalio import activity

from task_queues import FILE_PROCESSING_SHARED_QUEUE

logging.basicConfig(level=logging.INFO)

# Generate a unique Task Queue name for this worker instance
UNIQUE_WORKER_TASK_QUEUE = f"file-processing-{uuid.uuid4()}"

@activity.defn
async def get_available_task_queue() -> str:
    """
    Return this worker's unique task queue name.

    The Workflow calls this on the shared queue to discover a Worker's
    unique queue, then routes all subsequent file operations to that queue.
    """
    activity.logger.info(f"Returning unique queue: {UNIQUE_WORKER_TASK_QUEUE}")
    return UNIQUE_WORKER_TASK_QUEUE

@activity.defn
async def download_file(url: str) -> str:
    """
    Download file from URL and save to local disk.

    Returns the local file path. Subsequent activities on the same Worker
    can access this file without network transfer.
    """

    local_path = f"/tmp/temporal_file_{uuid.uuid4()}"

    activity.logger.info(f"Downloading {url} to {local_path}")

    async with httpx.AsyncClient() as client:
        response = await client.get(url, timeout=300.0)
        activity.heartbeat(f"Downloading {url}")
        
        # Raise exception for 4xx/5xx errors
        # By default, Temporal retries all application failures.
        # Treat 4xx errors as non retryable         
        try:
            response.raise_for_status()
        except httpx.HTTPStatusError as e:
            if 400 <= e.response.status_code < 500:                
                raise ApplicationError(f"Client error downloading file: {e}", non_retryable=True) from e
            raise e

        # Use asyncio.to_thread to avoid blocking the event loop
        await asyncio.to_thread(Path(local_path).write_bytes, response.content)

    activity.logger.info(
        f"Downloaded {len(response.content)} bytes to {local_path}"
    )

    return local_path

@activity.defn
async def process_file(local_path: str) -> str:
    """
    Process the file that was downloaded by previous activity.

    Because this runs on the same worker, the file is already present
    on the local disk - no network transfer needed.
    """
    activity.logger.info(f"Processing file at {local_path}")

    # Read the local file (async to avoid blocking the event loop)
    content = await asyncio.to_thread(Path(local_path).read_bytes)

    # Simulate processing (e.g., video transcoding, image transformation)
    # Send periodic heartbeats to detect worker crashes quickly
    processing_duration = 3  # seconds (simulated; real work could be minutes/hours)
    heartbeat_interval = 1  # seconds

    elapsed = 0
    while elapsed < processing_duration:
        await asyncio.sleep(heartbeat_interval)
        elapsed += heartbeat_interval
        # Send heartbeat periodically to prove the worker is alive
        activity.heartbeat(f"Processing file at {local_path} - {elapsed}s elapsed")

    # Compute checksum as proof of processing
    checksum = hashlib.sha256(content).hexdigest()

    result_path = f"{local_path}.processed"
    activity.heartbeat(f"Writing results to {result_path}")
    await asyncio.to_thread(Path(result_path).write_text, f"Processed. Checksum: {checksum}")

    activity.logger.info(f"Processed file, checksum: {checksum[:16]}...")

    return result_path

@activity.defn
async def upload_file(local_path: str) -> str:
    """
    Upload the processed file to destination.

    Reads from local disk (no network transfer from previous Activities).
    """
    activity.logger.info(f"Uploading file from {local_path}")

    # Use asyncio.to_thread for blocking file read
    content = await asyncio.to_thread(Path(local_path).read_text)
    activity.heartbeat(f"File read complete: {len(content)} bytes")

    # Simulate upload to S3, GCS, etc. with periodic heartbeats
    # In real scenarios: stream file, track upload progress, handle retries, etc.
    upload_duration = 3  # seconds (simulated; real uploads could take minutes)
    heartbeat_interval = 1  # seconds

    elapsed = 0
    while elapsed < upload_duration:
        await asyncio.sleep(heartbeat_interval)
        elapsed += heartbeat_interval
        progress = int((elapsed / upload_duration) * 100)
        activity.heartbeat(f"Uploading file - {progress}% complete")

    upload_url = f"https://storage.example.com/results/{uuid.uuid4()}"
    activity.logger.info(f"Uploaded to {upload_url}")

    # Clean up local files with heartbeats
    activity.heartbeat(f"Starting cleanup of {local_path}")
    original_path = local_path.replace(".processed", "")
    for path in [local_path, original_path]:
        try:
            os.remove(path)
            activity.heartbeat(f"Cleaned up {path}")
            activity.logger.info(f"Cleaned up {path}")
        except FileNotFoundError:
            pass

    activity.heartbeat("Upload and cleanup complete")
    return upload_url

async def main():
    config = ClientConfig.load_client_connect_config()
    client = await Client.connect(**config)

    # Worker for the shared queue
    # Handles "get_available_task_queue" requests from workflows
    shared_worker = Worker(
        client,
        task_queue=FILE_PROCESSING_SHARED_QUEUE,
        activities=[get_available_task_queue],
    )

    # Worker for this process's unique queue
    # Handles the actual file operations
    unique_worker = Worker(
        client,
        task_queue=UNIQUE_WORKER_TASK_QUEUE,
        activities=[download_file, process_file, upload_file],
        max_concurrent_activities=5,  # Limit based on disk I/O
    )

    logging.info(
        f"Starting file processing Worker\n"
        f"  Shared queue: {FILE_PROCESSING_SHARED_QUEUE}\n"
        f"  Unique queue: {UNIQUE_WORKER_TASK_QUEUE}"
    )

    # Run both Workers concurrently
    await asyncio.gather(
        shared_worker.run(),
        unique_worker.run(),
    )

if __name__ == "__main__":
    asyncio.run(main())
```

**Deployment guidance:**
- Deploy multiple instances of this Worker (for example, 5-10 instances)
- Each instance generates its own unique queue UUID
- Workers should have adequate disk space for file processing
- Consider using local SSD storage for better I/O performance

### Step 3: Implement Workflow with Worker affinity and retries

When a Worker crashes, Activities scheduled to its unique queue will wait. Implement failure detection and recovery using a multi-layered resilience strategy:

**Failure Detection Mechanisms:**

1. **Heartbeat Timeout** (fastest): Activities periodically send heartbeats to signal they're alive. If a Worker crashes mid-execution, no Heartbeats are sent, and the Activity fails within seconds (for example, 30s) rather than waiting for the full Task duration.

2. **Schedule-to-Start Timeout** (medium): If a Worker crashes before picking up a Task from its queue, this timeout detects it within minutes (for example, 5 min) instead of waiting for the full Start-To-Close Timeout. This is critical for identifying dead Workers early.

3. **Workflow-Level Retries** (recovery): The `for` loop catches all exceptions and retries the entire Workflow sequence on a different Worker. This provides recovery after detecting a Worker failure—the Workflow doesn't retry on the failed Worker's queue, but instead requests a new unique queue from a healthy Worker.

**Failure Scenarios:**

- **Worker crashes mid-Activity** → Detected by `heartbeat_timeout` (30s) → Exception caught by try/except → Workflow retries on new Worker
- **Worker crashes before picking up Task** → Detected by `schedule_to_start_timeout` (5 min) → Exception caught → Workflow retries on new Worker
- **All Workers fail** → Loop exhausts max attempts → Workflow fails after 3 attempts (configurable)

**Enhanced Workflow with fallback:**

```python
@workflow.defn
class FileProcessingWorkflowWithFallback:
    """File processing with Worker failure handling.

    Uses a combination of:
    - Heartbeats for fast crash detection (30s)
    - Schedule-to-start timeout for dead Worker detection (5 min)
    - Workflow-level retries to attempt on a different Worker
    """

    @workflow.run
    async def run(self, file_url: str) -> str:
        max_worker_attempts = 3

        for attempt in range(max_worker_attempts):
            try:
                workflow.logger.info(
                    f"Attempt {attempt + 1}/{max_worker_attempts}"
                )

                # Get unique queue
                unique_queue = await workflow.execute_activity(
                    get_available_task_queue,
                    task_queue=FILE_PROCESSING_SHARED_QUEUE,
                    start_to_close_timeout=timedelta(minutes=1),
                )

                # Process file on that Worker
                local_path = await workflow.execute_activity(
                    download_file,
                    file_url,
                    task_queue=unique_queue,
                    start_to_close_timeout=timedelta(minutes=10),
                    # Key: schedule_to_start timeout detects dead workers before they pick up
                    schedule_to_start_timeout=timedelta(minutes=5),
                    # Key: heartbeat timeout detects worker crashes during execution (faster!)
                    heartbeat_timeout=timedelta(seconds=30),
                )

                processed_path = await workflow.execute_activity(
                    process_file,
                    local_path,
                    task_queue=unique_queue,
                    start_to_close_timeout=timedelta(minutes=30),
                    schedule_to_start_timeout=timedelta(minutes=5),
                    # Heartbeat detection for mid-execution worker crashes
                    heartbeat_timeout=timedelta(seconds=30),
                )

                upload_url = await workflow.execute_activity(
                    upload_file,
                    processed_path,
                    task_queue=unique_queue,
                    start_to_close_timeout=timedelta(minutes=10),
                    schedule_to_start_timeout=timedelta(minutes=5),
                    # Heartbeat detection for mid-execution worker crashes
                    heartbeat_timeout=timedelta(seconds=30),
                )

                return upload_url

            except Exception as e:
                workflow.logger.warning(
                    f"Worker attempt {attempt + 1} failed: {e}"
                )
                if attempt == max_worker_attempts - 1:
                    raise
                # Try a different Worker
                await workflow.sleep(timedelta(seconds=10))

        raise RuntimeError("Failed to process file after all attempts")
```

## Conclusion

By implementing Worker-specific Task Queues for file processing, you have achieved:

1. **Data locality:** Files downloaded in one Activity are immediately available to subsequent Activities on the same Worker, eliminating multi-GB network transfers

2. **Performance improvement:** Reduced execution time by 80%+ for Workflows processing large files (for example, 10GB video file no longer transferred between Workers)

3. **Resource efficiency:** Load expensive resources (ML models, database connections) once per Workflow instead of per Activity, reducing memory usage and initialization overhead

4. **Cost reduction:** Eliminated network egress costs from transferring large files between Workers, potentially saving thousands of dollars per month

Your file processing Workflows now guarantee that all Activities execute on the same Worker, maintaining data locality and maximizing performance.

## Related resources

### Official documentation
- [Temporal Documentation - Task Routing](/task-routing)
- [Temporal Documentation - Worker Sessions (Go SDK)](/develop/go/workers/sessions)

### Related patterns
- [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue) - Pattern catalog reference for this two-tier Task Queue technique, with Go, Java, Python, and TypeScript implementations
- [Separate Task Queues - Priorities](/guides/route-specialized-workloads) - For priority-based routing
- [Separate Task Queues - Rate Limiting](/guides/rate-limit-downstream-apis) - For protecting downstream APIs

### Code samples
- [Worker-Specific Task Queues Sample (Python)](https://github.com/temporalio/samples-python/tree/main/worker_specific_task_queues)
- [Worker-Specific Task Queues Sample (TypeScript)](https://github.com/temporalio/samples-typescript/tree/main/worker-specific-task-queues)
- [Worker-Specific Task Queues Sample (Go)](https://github.com/temporalio/samples-go/tree/main/worker-specific-task-queues)
- [Worker-Specific Task Queues Sample (.NET)](https://github.com/temporalio/samples-dotnet/tree/main/src/WorkerSpecificTaskQueues)
- [Worker-Specific Task Queues Sample (Ruby)](https://github.com/temporalio/samples-ruby/tree/main/worker_specific_task_queues)

### Community resources
- [Forum: When to Use Multiple Task Queues](https://community.temporal.io/t/in-what-situation-should-we-use-multiple-separated-task-queues/1254)
- [Slack: Using Dynamic Task Queues for Traffic Routing](https://community.temporal.io/t/using-dynamic-task-queues-for-traffic-routing/3045)

---

# Handling Signals, Queries, & Updates

Source: https://docs.temporal.io/handling-messages

> Signals, Queries, and Updates facilitate interactions with Workflow Executions.

When Signals, Updates, and Queries arrive at your Workflow, the handlers for these messages will operate on the current state of your Workflow and can use the fields you have set.
In this section, we’ll give you an overview of how messages work with Temporal and cover how to write correct and robust handlers by covering topics like atomicity, guaranteeing completion before the Workflow exits, exceptions, and idempotency.

## Handling messages 

### Message handler concurrency 

If your Workflow receives messages, you may need to consider how those messages interact with one another or with the main Workflow method.
Behind the scenes, Temporal is running a loop that looks like this:

![Diagram that shows the execution ordering of Workflows](https://docs.temporal.io/img/info/messages-workflow-loop.png)

Every time the Workflow wakes up--generally, it wakes up when it needs to--it will process messages in the order they were received, followed by making progress in the Workflow’s main method.

This execution is on a single thread–while this means you don’t have to worry about parallelism, you do need to worry about concurrency if you have written Signal and Update handlers that can block. These can run interleaved with the main Workflow and with one another, resulting in potential race conditions. These methods should be made reentrant.

#### Initializing the Workflow first 

Initialize your Workflow's state before handling messages.
This prevents your handler from reading uninitialized instance variables.

To see why, refer to the [diagram](#message-handler-concurrency).
It shows that your Workflow processes messages before the first run of your Workflow's main method.

The message handler runs first in several scenarios, such as:

- When using [Signal-with-Start](/sending-messages#signal-with-start).
- When your Worker experiences delays, such as when the Task Queue it polls gets backlogged.
- When messages arrive immediately after a Workflow continues as new but before it resumes.

For all languages except Go and TypeScript, use your constructor to set up state.
Annotate your constructor as a Workflow Initializer and take the same arguments as your Workflow's main method.

Note that you can't make blocking calls from your constructor.
If you need to block, make your Signal or Update handler [wait](#waiting) for an initialization flag.

In Go and TypeScript, register any message handlers only after completing initialization.

### Message handler patterns 

Here are several common patterns for write operations, Signal and Update handlers. They don't apply to pure read operations, that is Queries or [Update Validators](/handling-messages#update-validators):

- Returning immediately from a handler
- Waiting for the Workflow to be ready to process them
- Kicking off activities and other asynchronous tasks
- Injecting work into the main Workflow
- Finishing handlers before the Workflow completes
- Ensuring your messages are processed exactly once

#### Synchronous handlers

Synchronous handlers don’t kick off any long-running operations or otherwise block. They're guaranteed to run atomically.

#### Waiting 

A Signal or Update handler can block waiting for the Workflow to reach a certain state using a Wait Condition. See the links below to find out how to use this with your SDK.

#### Running asynchronous tasks

Sometimes, you need your message handler to wait for long-running operations such as executing an Activity. When this happens, the handler will yield control back to [the loop](#message-handler-concurrency). This means that your handlers can have race conditions if you’re not careful.
You can guard your handlers with concurrency primitives like mutexes or semaphores, but you should use versions of these primitives provided for Workflows in most languages. See the links below for examples of how to use them in your SDK.

#### Inject work into the main Workflow 

Sometimes you want to process work provided by messages in the main Workflow. Perhaps you’d like to accumulate several messages before acting on any of them. For example, message handlers might put work into a queue, which can then be picked up and processed in an event loop that you yourself write.
This option is advanced, but it lets you control exactly when each message is processed. And if you serialize the handling of your messages inside your main Workflow, you can avoid using concurrency primitives like mutexes and semaphores. See the links above for how to do this in your SDK.

#### Finishing handlers before the Workflow completes 

You should generally finish running all handlers before the Workflow run completes or continues as new. For some Workflows, this means you should explicitly check to make sure that all the handlers have completed before finishing. You can await a condition called All Handlers Finished at the end of your Workflow.

If you don’t need to ensure that your handlers complete, you may specify your handler’s Handler Unfinished Policy as Abandon to turn off the warnings. However, note that clients waiting for Updates will get Not Found errors if they're waiting for Updates that never complete before the Workflow run completes.

See the links below for how to ensure handlers are finished in your SDK.

#### Message IDs and handling Continue-As-New 

Usually, you'll want your message handlers to run exactly once--to be idempotent--in cases where the same Signal or Update is delivered twice. For Updates, Temporal handles this for you on the server, by deduplicating according to the Update ID. The Update ID is set automatically to a UUID, but you can set it yourself.

For Signals, you should use a custom idempotency key that you send as part of your own signal inputs, implementing the deduplication in your Workflow code.

However, if you are using Updates with [Continue-As-New](/workflow-execution/continue-as-new) you should implement the deduplication in your Workflow code, since Update ID deduplication by the server is per Workflow run.

> **ℹ️ Info:**
>
> In addition to these application-level identifiers, both Signals and Updates automatically use request IDs to deduplicate retried client calls. You do not need to do anything to enable this.
>

See the links below for examples of handling idempotency and Continue-As-New in your SDK.

#### Authoring message handler patterns

See examples of the above patterns.

**Related:**

- [Author message handler patterns in .NET](/develop/dotnet/workflows/message-passing)
- [Author message handler patterns in Go](/develop/go/workflows/message-passing#message-handler-patterns)
- [Author message handler patterns in Java](/develop/java/workflows/message-passing)
- [Author message handler patterns in PHP](/develop/php/workflows/message-passing)
- [Author message handler patterns in Python](/develop/python/workflows/message-passing#message-handler-patterns)
- [Author message handler patterns in TypeScript](/develop/typescript/workflows/message-passing#message-handler-patterns)
- [Author message handler patterns in Ruby](/develop/ruby/workflows/message-passing#message-handler-patterns)
- [Author message handler patterns in Rust](/develop/rust/workflows/message-passing#message-handler-patterns)

### Update validators 

When you define an Update handler, you may optionally define an Update Validator: a read operation that's responsible for accepting or rejecting the Update. You can use Validators to verify arguments or make sure the Workflow is ready to accept your Updates.

- If it accepts, the Update will become part of your Workflow’s history and the client will be notified that the operation has been Accepted. The Update handler will then run until it returns a value.
- If it rejects, the client will be informed that it was Rejected, and the Workflow will have no indication that it was ever requested, similar to a Query handler.

> **📝 Note:**
>
> Like Queries, Validators are not allowed to block.
>

Once the Update handler is finished and has returned a value, the operation is considered Completed.

**Related:**

- [Validate updates in Go](/develop/go/workflows/message-passing#updates)
- [Validate updates in Java](/develop/java/workflows/message-passing#updates)
- [Validate updates in .NET](/develop/dotnet/workflows/message-passing#updates)
- [Validate updates in Python](/develop/python/workflows/message-passing#updates)
- [Validate updates in TypeScript](/develop/typescript/workflows/message-passing#updates)
- [Validate updates in PHP](/develop/php/workflows/message-passing#handle-updates)
- [Validate updates in Ruby](/develop/ruby/workflows/message-passing#updates)
- [Validate updates in Rust](/develop/rust/workflows/message-passing#updates)

### Exceptions in message handlers 

When throwing an exception in a message handler, you should decide whether to make it an [Application Failure](/references/failures#application-failure). The implications are different between Signals and Updates.

> **⚠️ Caution:**
> The following content applies in every SDK except the Go SDK. See below.

#### Exceptions in Signals

In Signal handlers, throw [Application Failures](/references/failures#application-failure) only for unrecoverable errors, because the entire Workflow will fail.
Similarly, allowing a failing Activity or Child Workflow to exhaust its retries, so that it throws an [Activity Failure](/references/failures#activity-failure) or [Child Workflow Failure](/references/failures#child-workflow-failure) will cause the entire Workflow to fail.
Note that for Activities, this will only happen if you change the default Activity [Retry Policy](/encyclopedia/retry-policies), since by default they retry forever.
If you throw any other exception, by default, it will cause a [Workflow Task Failure](/references/failures#workflow-task-failures). This means the Workflow will get stuck and will retry the handler periodically until the exception is fixed, for example by a code change.

#### Exceptions in Updates

Doing any of the following will fail the Update and cause the client to receive the error:

- Reject the Update by throwing any exception from your [Validator](/handling-messages#update-validators).
- Allow a failing Activity or Child Workflow to exhaust its retries, so that it throws an [Activity Failure](/references/failures#activity-failure) or [Child Workflow Failure](/references/failures#child-workflow-failure). Note that for Activities, this will only happen if you change the default Activity [Retry Policy](/encyclopedia/retry-policies), since by default they retry forever.
- Throw an [Application Failure](/references/failures#application-failure) from your Update handler.

Unlike with Signals, the Workflow will keep going in these cases.

If you throw any other exception, by default, it will cause a [Workflow Task Failure](/references/failures#workflow-task-failures). This means the Workflow will get stuck and will retry the handler periodically until the exception is fixed, for example by a code change or infrastructure coming back online. Note that this will cause a delay for clients waiting for an Update result.

#### Errors and panics in message handlers in the Go SDK

In Go, returning an error behaves like an [Application Failure](/references/failures#application-failure) in the other SDKs. Panics behave like non-Application Failure exceptions in other languages, in that they cause a [Workflow Task Failure](/references/failures#workflow-task-failures).

### Writing Signal handlers 

Use these links to see a simple Signal handler.

**Related:**

- [Handle Signals in Go](/develop/go/workflows/message-passing#signals)
- [Handle Signals in Java](/develop/java/workflows/message-passing#signals)
- [Handle Signals in Python](/develop/python/workflows/message-passing#signals)
- [Handle Signals in TypeScript](/develop/typescript/workflows/message-passing#signals)
- [Handle Signals in .NET](/develop/dotnet/workflows/message-passing#signals)
- [Handle Signals in PHP](/develop/php/workflows/message-passing#handle-signal)
- [Handle Signals in Ruby](/develop/ruby/workflows/message-passing#signals)
- [Handle Signals in Rust](/develop/rust/workflows/message-passing#signals)

### Writing Update handlers 

Use these links to see a simple update handler.

**Related:**

- [Handle Updates in Go](/develop/go/workflows/message-passing#updates)
- [Handle Updates in Java](/develop/java/workflows/message-passing#updates)
- [Handle Updates in Python](/develop/python/workflows/message-passing#updates)
- [Handle Updates in TypeScript](/develop/typescript/workflows/message-passing#updates)
- [Handle Updates in .NET](/develop/dotnet/workflows/message-passing#updates)
- [Handle Updates in PHP](/develop/php/workflows/message-passing#handle-updates)
- [Handle Updates in Ruby](/develop/ruby/workflows/message-passing#updates)
- [Handle Updates in Rust](/develop/rust/workflows/message-passing#updates)

### Writing Query handlers 

Author queries using these per-language guides.

**Related:**

- [Handle Queries in Go](/develop/go/workflows/message-passing#queries)
- [Handle Queries in Java](/develop/java/workflows/message-passing#queries)
- [Handle Queries in Python](/develop/python/workflows/message-passing#queries)
- [Handle Queries in TypeScript](/develop/typescript/workflows/message-passing#queries)
- [Handle Queries in .NET](/develop/dotnet/workflows/message-passing#queries)
- [Handle Queries in PHP](/develop/php/workflows/message-passing#handle-query)
- [Handle Queries in Ruby](/develop/ruby/workflows/message-passing#queries)
- [Handle Queries in Rust](/develop/rust/workflows/message-passing#queries)

---

# Temporal Platform Documentation

Source: https://docs.temporal.io/index

> Official docs for Temporal, the open source Durable Execution platform for crash-proof applications and AI agents. Guides, SDKs, and references for reliable workflows.

# Build applications that never fail

Temporal is an open-source platform for building reliable applications. Temporal delivers crash-proof execution by guaranteeing that applications resume exactly where they left off after crashes, network failures, or infrastructure outages, whether that happens seconds, days, or even years later.

Temporal enables developers to focus on building features that drive the business while ensuring that mission-critical processes such as order fulfillment, customer onboarding, payment processing, and AI agents never fail or disappear, regardless of what goes wrong.

- [Quickstart](/quickstarts): Setup your local and run a Hello World workflow.

- [Developer Guide](/develop): Dive into everything you need to know about building Temporal workflows.

- [Deploy your Workflows](/production-deployment): Deploy your Temporal Application to your environment. Self-Host the Temporal Service or use Temporal Cloud.

- [Get started for free with $150 in credits](https://temporal.io/cloud): Sign up for Temporal Cloud and let us host the Temporal Service for you.

- [Slack Community](https://temporal.io/slack): Join us on [temporal.io/slack](https://temporal.io/slack) and say hi or ask us a question.

- [Developer Forum](https://community.temporal.io): [Find out](https://community.temporal.io) if your question has already been asked.

- [Learn it all](https://learn.temporal.io/courses/): [Master Temporal](https://learn.temporal.io/courses/) with our courses and tutorials.

---

# Integrations

Source: https://docs.temporal.io/integrations

> Browse all integrations available for Temporal.

Browse integrations for Temporal SDKs and Temporal Cloud.
Filter by SDK, tag, or search to find what you need.

- [AI SDK by Vercel](/develop/typescript/integrations/ai-sdk) — Build AI-powered applications with Durable Execution using the Vercel AI SDK. _(TypeScript · Agent framework)_
- [Baseten](/develop/python/integrations/baseten) — Run OpenAI Agents SDK agents on open-weight models served by Baseten, with every model call as a durable Temporal Activity. _(Python · Model provider)_
- [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#go) — Monitor and evaluate AI application performance with Braintrust observability. _(Go · Agent observability)_
- [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#python) — Monitor and evaluate AI application performance with Braintrust observability. _(Python · Agent observability)_
- [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#typescript) — Monitor and evaluate AI application performance with Braintrust observability. _(TypeScript · Agent observability)_
- [ClickStack](https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/temporal-metrics) — Ingest Temporal Cloud metrics into ClickHouse via an OpenTelemetry collector with HyperDX dashboards. _(Observability · Temporal Cloud)_
- [Datadog (Cost Management)](https://docs.datadoghq.com/integrations/temporal-cloud-costs/) — Send Temporal Cloud billing data to Datadog Cloud Cost Management for spend attribution by namespace and service type. _(Observability · Temporal Cloud)_
- [Datadog (Metrics)](https://docs.datadoghq.com/integrations/temporal-cloud-openmetrics/) — Export Temporal Cloud metrics to Datadog with a serverless integration and pre-built dashboard. _(Observability · Temporal Cloud)_
- [Deep Agents](/develop/python/integrations/deepagents) — Make LangChain Deep Agents durable with Temporal Workflows and Activities. _(Python · Agent framework)_
- [Elastic](https://www.elastic.co/docs/reference/integrations/temporal) — Export Temporal Cloud metrics to Elastic with the Elastic Managed Temporal integration and get OOTB dashboards, alerts and slo templates. _(Observability · Temporal Cloud)_
- [Fireworks AI](/develop/python/integrations/fireworks-ai) — Run OpenAI Agents SDK agents on open-weight models served by Fireworks AI, with every model call as a durable Temporal Activity. _(Python · Model provider)_
- [Google ADK](/develop/python/integrations/google-adk) — Run Google ADK agents as durable Temporal Workflows. _(Python · Agent framework)_
- [Google ADK](/develop/go/integrations/google-adk) — Run Google ADK agents as durable Temporal Workflows. _(Go · Agent framework)_
- [Google ADK](/develop/typescript/integrations/google-adk-agents) — Run Google ADK agents as durable Temporal Workflows. _(TypeScript · Agent framework)_
- [Google GenAI](/develop/python/integrations/google-genai) — Call Google Gemini models durably from Temporal Workflows with the Google Gen AI SDK. _(Python · Agent framework)_
- [Grafana Cloud](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-temporal/) — Export Temporal Cloud metrics to Grafana Cloud with a serverless integration and pre-built dashboard. _(Observability · Temporal Cloud)_
- [Laminar](https://laminar.sh/docs/integrations/temporal) — Trace Temporal Workflows, Activities, and the LLM calls inside them with Laminar. _(Python · Agent observability)_
- [Laminar](https://laminar.sh/docs/integrations/temporal) — Trace Temporal Workflows, Activities, and the LLM calls inside them with Laminar. _(TypeScript · Agent observability)_
- [Langfuse](https://langfuse.com/integrations/frameworks/temporal) — Trace and debug LLM calls in Temporal Workflows with Langfuse. _(Python · Agent observability)_
- [LangGraph](/develop/python/integrations/langgraph) — Run LangGraph agent graphs as durable, resumable Temporal Workflows. _(Python · Agent framework)_
- [LangSmith](/develop/python/integrations/langsmith) — Trace and debug LLM calls in Temporal Workflows with LangSmith. _(Python · Agent observability)_
- [LangSmith](/develop/typescript/integrations/langsmith) — Trace and debug LLM calls in Temporal Workflows with LangSmith. _(TypeScript · Agent observability)_
- [Mastra](https://mastra.ai/integrations/deploy/temporal) — Build durable AI agents and workflows with the Mastra TypeScript framework. _(TypeScript · Agent framework)_
- [Model Context Protocol (MCP)](/develop/python/integrations/mcp) — Call MCP tools, prompts, and resources from Python Workflows as durable Temporal Activities. _(Python · Framework)_
- [New Relic](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/temporal-cloud-integration/) — Export Temporal Cloud metrics to New Relic via the nri-flex infrastructure agent integration. _(Observability · Temporal Cloud)_
- [OpenAI Agents SDK](/develop/python/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(Python · Agent framework)_
- [OpenAI Agents SDK](/develop/typescript/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(TypeScript · Agent framework)_
- [OpenBox](https://docs.openbox.ai/getting-started/temporal) — Add governance and observability guardrails to AI-powered Temporal Workflows with OpenBox. _(Python · Governance · Agent observability)_
- [OpenRouter](/develop/python/integrations/openrouter) — Call hundreds of models through OpenRouter from Temporal Activities, with durable retries, cost tracking, and budgets. _(Python · Model gateway)_
- [OpenRouter](https://github.com/temporalio/samples-typescript/tree/main/openrouter) — Call hundreds of models through OpenRouter from Temporal Activities, with durable retries and cost tracking. _(TypeScript · Model gateway)_
- [OpenTelemetry v2](/develop/go/integrations/opentelemetry-v2) — Export tracing and metrics from Temporal Go SDK applications with OpenTelemetry. _(Go · Observability)_
- [Parseable](https://github.com/parseablehq/temporal-plugin-java/blob/main/INTEGRATION.md) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(Java · Agent observability)_
- [Parseable](https://github.com/parseablehq/temporal-plugin-python/blob/main/INTEGRATION.MD) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(Python · Agent observability)_
- [Parseable](https://github.com/parseablehq/temporal-plugin/blob/main/INTEGRATION.md) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(TypeScript · Agent observability)_
- [Prometheus + Grafana](/cloud/metrics/openmetrics/metrics-integrations#prometheus-grafana) — Scrape Temporal Cloud metrics with self-hosted Prometheus and visualize with Grafana dashboards. _(Observability · Temporal Cloud)_
- [Pydantic AI](https://pydantic.dev/docs/ai/capabilities/durable_execution/temporal/) — Build type-safe AI agents with Durable Execution through Pydantic AI. _(Python · Agent framework)_
- [Rails](/develop/ruby/integrations/rails-integration) — Integrate Temporal durable workflows into Ruby on Rails applications. _(Ruby · Framework)_
- [SigNoz](https://signoz.io/docs/integrations/temporal-cloud-metrics/) — Export Temporal Cloud metrics to SigNoz via an OpenTelemetry Collector with a pre-built dashboard. _(Observability · Temporal Cloud)_
- [Spring AI](/develop/java/integrations/spring-ai) — Build AI-powered Java applications with durable Spring AI tool calls. _(Java · Agent framework)_
- [Spring Boot](/develop/java/integrations/spring-boot-integration) — Use Temporal natively in Spring Boot with auto-configuration and dependency injection. _(Java · Framework)_
- [Strands Agents](/develop/python/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(Python · Agent framework)_
- [Strands Agents](/develop/typescript/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(TypeScript · Agent framework)_
- [Tenuo](https://tenuo.ai/temporal) — Add governance and compliance guardrails to AI-powered Temporal Workflows. _(Python · Governance)_
- [Together AI](/develop/python/integrations/together-ai) — Run OpenAI Agents SDK agents on open-weight models served by Together AI, with every model call as a durable Temporal Activity. _(Python · Model provider)_
- [Vantage](https://docs.vantage.sh/connecting_temporal/) — Track and analyze Temporal Cloud costs with detailed reports, filtering, and multi-account support. _(Observability · Temporal Cloud)_
- [xMemory](https://xmemory.ai/temporal/#quickstart--python) — Store and recall agent memory across Workflow Executions with xMemory. _(Python · Agent memory)_
- [xMemory](https://xmemory.ai/temporal/#quickstart--typescript) — Store and recall agent memory across Workflow Executions with xMemory. _(TypeScript · Agent memory)_

---

# Key management

Source: https://docs.temporal.io/key-management

> Learn about key management practices for securing encryption keys in Temporal applications.

This page discusses [Key Management](#key-management).

## What is Key Management? 

Key Management is a fundamental part of working with encryption keys.

There are many computational and logistical aspects to generating and rotating keys, and this usually calls for a dedicated application in your stack. Here are some general recommendations for working with encryption keys for Temporal applications:

- [Symmetric Encryption](https://en.wikipedia.org/wiki/Symmetric-key_algorithm) is generally faster and will produce smaller payloads than asymmetric. Normally, an advantage of _asymmetric_ encryption is that it allows you to distribute your encryption and decryption keys separately, but depending on your infrastructure, this might not offer any security benefits with Temporal.

- AES-based algorithms are [hardware accelerated in Go](https://pkg.go.dev/crypto/aes) and other languages. AES algorithms are widely vetted and trusted, and there are many different variants that may suit your requirements. Load tests using `ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY` have performed well.

- Store your encryption keys in the same manner as you store passwords, config details, and other sensitive data. When possible, load the key into your application, so you don't need to make a network call to retrieve it. Separate keys for each environment or namespace as much as possible.

- Make sure you have a key rotation strategy in place in the event that your keys are compromised or need to be replaced for another reason. Consider using a dedicated secrets engine or a key management system (KMS). Note that when you rotate keys, you may also need to retain old keys to query old Workflows.

### Key rotation

National Institute of Standards and Technology (NIST) guidance recommends periodic rotation of encryption keys. For AES-GCM keys, rotation should occur before approximately 2^32 encryptions have been performed by a key version, following the guidelines of NIST publication 800-38D.

It is recommended that operators estimate the encryption rate of a key and use that to determine a frequency of rotation that prevents the guidance limits from being reached. For example, if one determines that the estimated rate is 40 million operations per day, then rotating a key every three months is sufficient.

Key rotation should generally be transparent to the Temporal Data Converter implementation. Temporal's `Encode()` and `Decode()` steps only need to trigger as expected, and Temporal has no knowledge of how or when you are generating your encryption keys.

You should design your Encode and Decode steps to accept all the necessary parameters for your key management, such as the key version, alongside your payloads. Like the Data Converters, keys should be mapped to a Namespace in Temporal.

---

# List Filter

Source: https://docs.temporal.io/list-filter

> This guide on Temporal List Filters explains how to set up, configure, and use the List Filter API in Temporal Server versions. Filter and retrieve Workflow Executions, apply supported operators, and optimize queries for efficiency.

This page discusses [List Filter](#list-filter).

## What is a List Filter? 

The [Visibility](/temporal-service/visibility) List API requires you to provide a List Filter as an SQL-like string parameter.

A List Filter includes [Search Attribute](/search-attribute) names, Search Attribute values, and [operators](#supported-operators) so that it can retrieve a filtered list of Workflow Executions from the Visibility Store.

List Filter [Search Attribute](/search-attribute) names are case sensitive.
A single [Namespace](/namespaces) scopes each List Filter.

A List Filter using a time range provides a resolution of 1 ns on [Elasticsearch](/self-hosted-guide/visibility/elasticsearch) and 1 µs for [SQL databases](/self-hosted-guide/visibility).

### Supported operators

List Filters support the following operators:

- **`=, !=, >, >=, <, <=`**
- **`AND, OR, ()`**
- **`BETWEEN ... AND`**
- **`IN`**
- **STARTS_WITH**

> **📝 Note:**
>
> The **ORDER BY** operator is currently not supported in Temporal Cloud.
>
> The default ordering is: `ClosedTime DESC NULL FIRST`, `StartTime DESC`. 
>
> Custom Search Attributes of the `Text` type cannot be used in **ORDER BY** clauses.
>

### Partial string match

There are different options for partial string matching when the type of the Search Attribute is [Text](#text) versus [Keyword](#keyword).

#### Text

`Text` Search Attributes support word-level search.
Both stored values and query strings are analyzed by the [Elasticsearch standard analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html), and the `=` operator returns a result if any query token matches any indexed token.

If you need exact matching on identifiers, UUIDs, or other structured strings, use `Keyword` instead.
See [Choose a string type](/search-attribute#choose-a-string-type).

##### Tokenization 

The standard analyzer applies [Unicode word boundary rules](https://unicode.org/reports/tr29/) to split values into tokens.
All tokens are lowercased by the analyzer's lowercase filter.

Most punctuation and whitespace act as delimiters.
For example, `order-processing-v2` produces three tokens: `order`, `processing`, `v2`.

The following characters do **not** split when they appear between two letters or between two digits:

| Character | Example | Tokens |
| --- | --- | --- |
| Underscores (`_`) | `payment_retry_handler` | one token |
| Dots (`.`) | `com.example.workflows.ProcessOrder` | one token |
| Colons (`:`) | `alpha:bravo` | one token |
| Apostrophes (`'`) | `it's` | one token |

Dots and colons between digits also stay connected (`v1.2.3` → one token).
However, a dot or colon between a digit and a letter does split: `v1.ProcessOrder` → `v1`, `processorder`.

##### Search matching 

The `=` operator on `Text` Search Attributes uses **OR matching**: the query string is tokenized using the same rules as above, and a result is returned if **any** query token matches **any** indexed token.

For example, if you have a custom `Text` Search Attribute named `Description` with either of the following values—

```
my-business-id-foobar
my business id foobar
```

—then the following List Filter matches—

```
Description = 'foobar'
```

—but a partial word does not:

```
// Doesn't match
Description = 'foo'
```

Because the query string is also tokenized, a query like `= "processing-v2"` is split into `processing` and `v2`, matching any workflow that contains **either** token.
This can return unexpected results when tokens are shared across different values.

> **📝 Note:**
>
> Broader tokenizer and matching improvements for `Text` Search Attributes are being evaluated as part of future Visibility enhancements.
>

#### Keyword

For Search Attributes of type `Keyword` like `WorkflowId`, perform partial string matching using STARTS_WITH for prefixes and BETWEEN for suffixes.

- `WorkflowId STARTS_WITH "order-"` matches Workflow IDs with the "order-" prefix, regardless of the following text.

  ```
  order-
  order-1234
  order-abracadabra
  order-~~~abracadabra
  ```

- `WorkflowId BETWEEN "order-" AND "order-~"` matches Workflow IDs that have characters after `order-` with ASCII values lower than `~` (126, the highest-value printable character), such as the following:

  ```
  order-
  order-1234
  order-abracadabra
  ```

  It does not match `order-~~`.

> **📝 Note:**
> Filter Composition Quick Reference
>
> **Composition**
>
> - Data types:
>   - String literals with single or double quotes
>   - Numbers (Integer and Floating Point)
>   - Booleans
> - Comparison: `=`, `!=`, `>`, `>=`, `<`, `<=`
> - Expressions/Operators:
>   - `IN array`
>   - `BETWEEN value AND value`
>   - `STARTS_WITH string`
>   - `IS NULL`, `IS NOT NULL`
>   - `expr AND expr`, `expr OR expr`, `( expr )`
> - Array: `( comma-separated-values )`
>
> **Please note**
>
> - Wrap attributes with backticks if it contains characters not in
>   `[a-zA-Z0-9]`.
> - `STARTS_WITH` is only available for Keyword Search Attributes.
>

### Efficient API usage

If the Advanced List Filter API retrieves a substantial number of Workflow Executions (more than 10,000), the response time might be longer.

Beginning with Temporal Server v1.20, you can employ the `CountWorkflow` API to efficiently count the number of [Workflow Executions](/workflow-execution).

To paginate the results using the `ListWorkflow` API, use the page token to retrieve the next page.
Continue until the page token becomes `null`/`nil`.

#### List Filter examples

Here are examples of List Filters set with the [Temporal CLI](/cli/command-reference/workflow#list):

```
WorkflowType = "main.YourWorkflowDefinition" and ExecutionStatus != "Running" and (StartTime > "2021-06-07T16:46:34.236-08:00" or CloseTime > "2021-06-07T16:46:34-08:00")
```

When you use the preceding example, you receive a list of Workflows fulfilling the following criteria:

- Workflow Type is `main.YourWorkflowDefinition`.
- Workflow isn't in a running state.
- Workflow either started after "2021-06-07T16:46:34.236-08:00" or closed after "2021-06-07T16:46:34-08:00".

The following are additional examples of List Filters.

```sql
WorkflowId = '<workflow-id>'
```

```sql
WorkflowId = '<workflow-id>' or WorkflowId = '<another-workflow-id>'
```

```sql
WorkflowId IN ('<workflow-id>', '<another-workflow-id>')
```

```sql
WorkflowId = '<workflow-id>' and ExecutionStatus = 'Running'
```

```sql
WorkflowId = '<workflow-id>' or ExecutionStatus = 'Running'
```

```sql
WorkflowId = '<workflow-id>' and StartTime > '2021-08-22T15:04:05+00:00'
```

```sql
ExecutionTime between '2021-08-22T15:04:05+00:00' and '2021-08-28T15:04:05+00:00'
```

```sql
ExecutionTime < '2021-08-28T15:04:05+00:00' or ExecutionTime > '2021-08-22T15:04:05+00:00'
```

```sql
WorkflowType STARTS_WITH '<workflow-type-prefix>'
```

#### Standalone Activity Execution List Filter examples

[Standalone Activity Executions](/standalone-activity) use the same List Filter syntax over their own Search Attributes.
Pass these to `ListActivities` and `CountActivities`, or to [`temporal activity list`](/cli/command-reference/activity#list) and [`temporal activity count`](/cli/command-reference/activity#count).
These APIs return Standalone Activity Executions only; Activities that run inside a Workflow are not included.

Find every running Standalone Activity of one Activity Type:

```sql
ActivityType = '<activity-type>' and ExecutionStatus = 'Running'
```

Find the Standalone Activities an operator has [paused](/activity-operations):

```sql
ExecutionStatus = 'Paused'
```

```sql
ExecutionStatus = 'Paused' and ActivityType = '<activity-type>'
```

Find delayed jobs, meaning Standalone Activities whose first Activity Task has not been dispatched yet because of a [Start Delay](/standalone-activity#start-delay):

```sql
ExecutionStatus = 'Running' and ExecutionTime > '2021-08-22T15:04:05+00:00'
```

Find one Standalone Activity by its business identifier:

```sql
ActivityId = '<activity-id>'
```

### Search Attribute aliasing

Temporal prefixes most [default Search Attributes](./search-attributes.mdx#default-search-attribute) with `Temporal` to avoid naming conflicts with custom Search Attributes.
To make it easier to reference default Search Attributes in List Filters, Temporal supports aliasing, which lets you use the non-prefixed name of a default Search Attribute. However, if you choose to define a custom Search Attribute with the same name as the non-prefixed alias of a default Search Attribute, your custom Search Attribute overrides the alias.

> **ℹ️ Info:**
> Server Version Requirement
>
> Search Attribute aliasing requires Temporal Server version 1.30 and greater.
>

For example, the default Search Attribute `TemporalWorkflowVersioningBehavior` has the alias `WorkflowVersioningBehavior`. If you haven't defined a custom Search Attribute named `WorkflowVersioningBehavior`, you can use either name in a List Filter, and both refer to the same Search Attribute.

```sql
-- Using the original attribute name
WorkflowVersioningBehavior = 'pinned'

-- Using the Temporal-prefixed alias (equivalent)
TemporalWorkflowVersioningBehavior = 'pinned'
```

#### Alias resolution with custom Search Attributes

When resolving a Search Attribute in a List Filter, Temporal Server checks for matches in the following order:

1. Custom Search Attributes defined in the current Namespace

2. Default Search Attributes

This means that if you define a custom Search Attribute with the same name as the alias of a default Search Attribute, the non `Temporal` prefixed name will refer to your custom attribute. You can still search with the default Search Attribute by using the `Temporal` prefix.

For example, if you have a custom Search Attribute named `SchedulePaused`, List Filters using the following Search Attributes will return different results:

```sql
-- If you have a custom Search Attribute named 'SchedulePaused'
-- This will use your custom attribute, not the default Search Attribute
SchedulePaused = true

-- The original system attribute still works by using the Temporal prefix
TemporalSchedulePaused = true
```

`SchedulePaused` will refer to your custom Search Attribute, while `TemporalSchedulePaused` will refer to the default Search Attribute.

---

# Local Activity

Source: https://docs.temporal.io/local-activity

> Learn about Local Activities that run in the same Worker process as the Workflow that schedules it, trading full durability for lower latency and a smaller Event History.

## What is a Local Activity? 

A Local Activity is an [Activity Execution](/activity-execution) that executes in the same Worker process as the [Workflow Execution](/workflow-execution) that schedules it.

Unlike a regular Activity, a Local Activity never enters an Activity Task Queue. Instead, the Workflow Worker executes it directly in an in-process queue. Because it avoids the round trip through the Temporal Service, a Local Activity has significantly lower latency and produces fewer Event History entries than a regular Activity.

Local Activities help with performance optimization and are not a replacement for regular Activities.

Consider using a Local Activity only when the operation:

- is short-lived (completes in a few seconds), including retries
- can execute in the same binary as the Workflow
- does not require routing to a specific Worker or Task Queue
- does not require global rate limiting
- [is idempotent](/develop/python/best-practices/error-handling#make-activities-idempotent)

For most production workloads, regular Activities remain the recommended default.

> **💡 Tip:**
> Recommendation
>
> Use Local Activities only when your use case requires the performance optimization they provide, such as high-throughput Workflows with many very short-lived operations. For most business logic, regular Activities are the better choice.
>

## How Local Activities execute

Regular Activities are coordinated through the Temporal Service.

The execution flow is:

1. A Workflow schedules an Activity.
2. The Workflow completes its Workflow Task by sending a `ScheduleActivityTask` command to the Temporal Service.
3. The Temporal Service creates an Activity Task.
4. An Activity Worker polls the Activity Task Queue and executes the Activity.
5. The Activity result is recorded in Event history.
6. The Workflow resumes in a new Workflow Task.

Local Activities follow a shorter execution path:

1. A Workflow schedules a Local Activity.
2. The Local Activity is placed into an in-process queue within the Workflow Worker.
3. The Workflow Worker executes the Local Activity immediately.
4. The result is returned directly to the Workflow.
5. When the Workflow Task completes, the Worker records a `MarkerRecorded` event containing the Local Activity result.

Unlike regular Activities, scheduling and execution occur entirely within the Worker process. Only the final marker is persisted to Event history.

## Workflow Task heartbeating

Local Activities do **not** support Activity heartbeats. Instead, the SDK supports Workflow Task heartbeating.

Workflow Task heartbeating means that if a Local Activity approaches approximately 80% of the Workflow Task Timeout (10 seconds by default), the Worker completes the current Workflow Task and requests a new one from the Temporal Service. This renews the Worker's authorization to continue executing the Workflow and allows the Local Activity to keep running without exceeding the Workflow Task Timeout.

This enables Local Activities to run longer than a single Workflow Task timeout, but it comes with tradeoffs:

- Each Workflow Task heartbeat adds additional Events to the Event history.
- Signals and other external Workflow events are not processed until the Local Activities finish.
- Commands generated by the Workflow are not sent to the Temporal Service until one of the following occurs:
  - the Local Activity completes
  - the next Workflow Task heartbeat occurs

If your operation regularly approaches the Workflow Task timeout, it is usually better implemented as a regular Activity.

You can monitor your Local Activities by using the [`local_activity_total` metric](/references/sdk-metrics#local_activity_total) to determine how many Local Activity Executions have been made. You can also check your Event History for `RecordMarker` entries.

## Failure and durability

Regular Activities are durably tracked by the Temporal Service. Scheduling, completion, retries, and failures are all recorded in the Event history. If a Worker crashes after an Activity completes, the completed Activity is not executed again.

Local Activities behave differently. A Local Activity result becomes durable only when the enclosing Workflow Task successfully completes and records a `MarkerRecorded` event. Until then, execution exists only in Worker memory.

If the Worker crashes before the Workflow Task completes, the Workflow Task is retried, causing Local Activities executed during that Workflow Task to run again.

Because of this behavior, Local Activities provide at-least-once execution semantics and should always be idempotent. You can learn more about the behavior at shutdown in the [Workers section on Local Activities](encyclopedia/workers/worker-shutdown#local-activities).

Once a `MarkerRecorded` event has been written to the Event history, replay uses the recorded result rather than executing the Local Activity again.

## Mixing Local Activities and regular Activities

A Workflow can freely combine Local Activities and regular Activities.

For example:

```mermaid
flowchart LR
    LA[Local Activity A] --> LB[Local Activity B] --> RC[Regular Activity C]
```

When the Workflow Task completes after scheduling Activity C, the Worker sends commands similar to:

- `MarkerRecorded` (Local Activity A)
- `MarkerRecorded` (Local Activity B)
- `ScheduleActivityTask` (Activity C)

If Activity C later fails or retries, the Workflow replays using the recorded markers for A and B. Those completed Local Activities are not executed again because their results have already been persisted in the Event history. Only Activity C is retried according to its Retry Policy.

The only time completed Local Activities execute again is if the Worker fails before their completion markers are recorded. Long retry intervals are inefficient because retries eventually require Workflow Timers and additional Event history entries.

## Choosing between regular Activities and Local Activities

Choose a regular Activity unless you have a specific need for the performance optimization that Local Activities provide.

Use a Local Activity when:

- execution completes in a few seconds
- retries are expected to be short
- the operation is idempotent
- low latency is more important than full durability
- routing, rate limiting, and separate Activity Workers are unnecessary

Use a regular Activity when:

- interacting with external systems
- execution may take longer than a few seconds
- retries may span minutes or hours
- Activity heartbeating is required
- strong durability guarantees are important

Regular Activities are the right choice for most applications. Local Activities are an optimization for specialized, high-throughput workloads where minimizing latency and Event history size outweighs their reduced durability guarantees.

### Use cases for Local Activities

Good use cases for Local Activities include:

- Lightweight data transformations
- Small computations
- Reading from an in-memory cache
- Fast local filesystem operations
- High-throughput Workflows with many very short-lived operations

Use regular Activities for:

- Network requests
- Database operations
- External API calls
- Long-running work
- Operations requiring durable retries
- Operations that benefit from Task Queue routing or rate limiting

### Activity versus Local Activity 

| Feature | Activity | Local Activity |
| --- | --- | --- |
| Execution | Activity Worker | Workflow Worker |
| Task Queue | Yes | No |
| Service round trip | Required | Not required |
| Latency | Higher | Lower |
| Event history | Fully recorded | `MarkerRecorded` on completion |
| Heartbeating | Activity heartbeats | Workflow Task heartbeating |
| Retry durability | Durable | At-least-once until marker is recorded |
| Signal responsiveness | Unaffected | Delayed while the Workflow Task executes |
| Best for | General-purpose work | Short, high-throughput operations |

---

# Temporal Namespace

Source: https://docs.temporal.io/namespaces

> A Namespace is a unit of isolation within the Temporal Platform that provides resource separation, Workflow ID uniqueness, and configuration boundaries.

> **ℹ️ Info:**
> Open source and Temporal Cloud
> This page covers core namespace concepts that apply to both open source Temporal and Temporal Cloud.
>
> Temporal Cloud namespaces include additional capabilities, such as [API key](/cloud/api-keys) and [mTLS authentication](/cloud/certificates), [built-in role-based access controls](/cloud/manage-access/roles-and-permissions#namespace-level-permissions), [separate rate limits](/evaluate/cloud/limits#namespace-level), [high availability replication](/cloud/high-availability), [namespace tags](/cloud/namespaces#tag-a-namespace), and [namespace descriptions](/cloud/namespaces#describe-a-namespace).
>
> Moving from self-hosting to Cloud, or the reverse, requires zero code changes and incurs zero downtime.

A Namespace is a unit of isolation within the [Temporal Platform](/temporal#temporal-platform).

[Task Queues](/task-queue) and [Workflow Executions](/workflow-execution) belong to a Namespace.
When a Workflow Execution is spawned, it does so within a specific Namespace.

## Usage

- **Workflow ID uniqueness**: Temporal guarantees a unique Workflow Id within a Namespace.
  Workflow Executions may have the same Workflow Id if they are in different Namespaces.
- **Resource isolation**: Heavy traffic from one Namespace will not impact other Namespaces running on the same Temporal Service.
- **Configuration boundaries**: Options like the [Retention Period](/temporal-service/temporal-server#retention-period) and [Archival](/temporal-service/archival) destination are configured per Namespace.
- **Default Namespace**: If no Namespace is specified, the Temporal Service uses the Namespace "default" for all Temporal SDKs and the Temporal CLI.
  You must create a Namespace before using it in your Client.
- **Access control**: In self-hosted Temporal, use a custom [Authorizer](/self-hosted-guide/security#authorization) on your Frontend Service to restrict who can access each Namespace.
  Temporal Cloud provides [role-based access controls](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) instead.
- **Inter-Namespace communication**: Namespaces are isolated by default.
  [Nexus](/nexus) is the controlled path for Workflows in one Namespace to call operations in another.
- **Multi-tenancy**: A single Namespace is still multi-tenant.
  Multiple applications or teams can share a Namespace, but must coordinate on Workflow ID and Task Queue naming to avoid conflicts.
  For architectural guidance, see [Multi-tenant application patterns](/best-practices/multi-tenant-patterns).

## Namespace operations

For how to create and manage Namespaces:

- **Open source Temporal**: [Managing Namespaces](/self-hosted-guide/namespaces)
- **Temporal Cloud**: [Temporal Cloud Namespaces](/cloud/namespaces)

---

# Temporal Nexus

Source: https://docs.temporal.io/nexus

> Temporal Nexus connects Temporal Applications across isolated Namespaces with clean service contracts, built-in durability, observability, and security.

> **ℹ️ Info:**
> NEW TO NEXUS?
>
> This page explains what Nexus is and how it works. To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/features/nexus).
>

As a platform grows, coordinating work across teams and applications becomes increasingly difficult. 
Nexus lets teams selectively expose functionality that other teams can discover and reuse, without exposing their internal implementation details.
This approach makes it easier to build new applications by reusing what already exists, forming the foundation for a more modular and collaborative platform.

## What is Nexus?

Nexus connects Temporal Applications across (and within) isolated Namespaces.
Each team gets their own Namespace for security and fault isolation, while exposing a clean service contract for others to use through a [Nexus Endpoint](/nexus/endpoints).

Designed for Durable Execution, Nexus combines a familiar SDK programming model with reliable execution, built-in observability, and multi-region connectivity in Temporal Cloud.

Nexus is peer-to-peer, not hierarchical.
Caller and handler Workflows are siblings that communicate across Namespace boundaries.

![Nexus connects caller and handler Namespaces through a Nexus Endpoint](https://docs.temporal.io/img/cloud/nexus/nexus-overview-short.png)

## How Nexus works

### Services and Operations

A [Nexus Service](/nexus/services) is a named collection of [Nexus Operations](/nexus/operations) that a team exposes.
Operations abstract the underlying implementation - callers don't need to know whether an Operation starts a Workflow, sends a Signal, runs a Query, or executes other reliable code.
If you need to orchestrate multiple Nexus Operations, call them from a Workflow. But if you just need to execute a single Nexus Operation across Namespace boundaries, use a [Standalone Nexus Operation](/standalone-nexus-operation).
For a side-by-side comparison of these options, including when to skip Nexus and start a Workflow directly, see [Choosing a Nexus approach](/nexus/choosing-an-approach).

The [Operation lifecycle](/nexus/operations#operation-lifecycle) supports two modes:

- **Asynchronous** - Starts a Workflow (on the same or a different Task Queue, with optional [Eager Start](/develop/worker-performance#eager-workflow-start)) or a Standalone Activity, or sends an Update to an existing Workflow (pre-release). The handler returns an Operation token, and the backing Execution delivers its result when it completes. The maximum Schedule-to-Close timeout is [60 days](/evaluate/cloud/limits#nexus-operation-duration-limits).
- **Synchronous** - Completes within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout). Use for Signals, Queries, Updates, or other reliable low-latency calls using the [Temporal SDK Client](/nexus/operations#executing-arbitrary-code-from-a-sync-handler).

Use a Synchronous Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the 10-second handler deadline; otherwise, use an Asynchronous Operation.

Services and Operations are built with the Temporal SDK and typically [collocated](/nexus/patterns#collocated-pattern) in the same Worker as the Temporal primitives they abstract, or in a dedicated router Worker using the [router-queue pattern](/nexus/patterns#router-queue-pattern).

See the SDK feature guides for your language:

- [Nexus - Go](/develop/go/nexus/feature-guide)
- [Nexus - Java](/develop/java/nexus/feature-guide)
- [Nexus - Python](/develop/python/nexus/feature-guide)
- [Nexus - TypeScript](/develop/typescript/nexus/feature-guide)
- [Nexus - .NET](/develop/dotnet/nexus/feature-guide)

> **💡 Tip:**
> QUICKSTART
>
> To get started with Nexus, follow a Quickstart:
> [Go](/develop/go/nexus/quickstart) |
> [Java](/develop/java/nexus/quickstart) |
> [Python](/develop/python/nexus/quickstart) |
> [TypeScript](/develop/typescript/nexus/quickstart) |
> [.NET](/develop/dotnet/nexus/quickstart)
>

### Endpoints and Registry

A [Nexus Endpoint](/nexus/endpoints) is a reverse proxy that decouples callers from handlers.
Callers reference an Endpoint by name. The Endpoint routes requests to a target Namespace and Task Queue.
Callers never need to know the handler's Namespace, Task Queue, or internal implementation.

Endpoints are managed in the [Nexus Registry](/nexus/registry) using the UI, CLI, or Cloud Ops API.

In Temporal Cloud, each Endpoint also belongs to a [Project](/cloud/projects#use-nexus-across-projects).
The Project controls who can manage the Endpoint in the Cloud Ops API and UI.
It does not restrict cross-Namespace Nexus calls. Runtime access remains the Endpoint allowlist of caller Namespaces.

### Queue-based Worker architecture

Nexus uses the same queue-based Worker architecture as the rest of Temporal.
Handler Workers poll the Endpoint's target Task Queue for [Nexus Tasks](/tasks#nexus-task).
If a Nexus Service is down, caller Workflows continue to schedule Operations - they process when the service is back up.
No bespoke service deployments needed. Load balancing is automatic.

![Nexus Workers poll the Endpoint](https://docs.temporal.io/img/cloud/nexus/nexus-workers-short.png)

See [system interactions](/nexus/operations#system-interactions) for the full request flow.

### Built-in Nexus Machinery

When a caller Workflow executes a Nexus Operation, the [Nexus Machinery](/glossary#nexus-machinery) handles delivery with [at-least-once](/nexus/operations#execution-semantics) execution:

- Automatic retries with exponential backoff
- Rate limiting and concurrency limiting
- [Circuit breaking](/nexus/operations#circuit-breaking) (trips after 5 consecutive retryable errors)
- Automatic load balancing

The Machinery uses [Nexus RPC](/glossary#nexus-rpc) on the wire - a protocol supporting arbitrary-duration Operations.
You interact only with the Temporal SDK, not Nexus RPC directly.

### Multi-level calls

Nexus Operations can be composed across multiple services and teams. A handler Workflow can call another Nexus Operation, forming a chain:

- Workflow A -> Nexus Op 1 -> Workflow B -> Nexus Op 2 -> Workflow C

Each step is a separate, durable Operation with its own retries and failure handling. This enables service composition across Namespaces without requiring direct connectivity or shared configuration between teams.

## Operational reference

- [Nexus security](/nexus/security) - Access controls, secure connectivity, and payload encryption.
- [Nexus execution debugging](/nexus/execution-debugging) - Bi-directional linking, pending Operations, and tracing.
- [Nexus error handling](/nexus/error-handling) - Error types and how they surface in caller Workflows.
- [Nexus metrics](/nexus/metrics) - SDK, Cloud, and OSS cluster metrics.

---

# Choosing a Nexus approach

Source: https://docs.temporal.io/nexus/choosing-an-approach

> Pick between starting a Workflow directly, exposing a Nexus Service, calling one without a caller Workflow, and backing an Operation with an Activity.

[Workflows](/workflows) and [Activities](/activities) provide durable execution.
An Activity that runs without a Workflow is a [Standalone Activity](/standalone-activity).

[Nexus](/nexus) exposes capabilities through a service contract, with durable communication and controlled access across [Namespace](/namespaces) boundaries.

Two options remove a Workflow from either end of a Nexus call.
[Standalone Nexus Operations](/standalone-nexus-operation) (SANO) let applications call a service without a caller Workflow.
[Nexus Standalone Activities](/nexus/standalone-activity) (SAA) let a service execute a single durable action without a handler Workflow.

> **ℹ️ Info:**
>
> Nexus is Generally Available.
> Standalone Nexus Operations are at [Public Preview](/evaluate/product-release-stages#public-preview), where API breaking changes are kept to a minimum.
> Nexus Standalone Activities are at [Pre-release](/evaluate/product-release-stages#pre-release), where APIs are experimental and may be subject to backwards-incompatible changes.
>

## When to use Workflows vs Nexus vs SANO vs SAA

| Capability | When to use it | Example |
| :---- | :---- | :---- |
| **Start a Workflow directly** | The application owns the process and can depend on its Workflow API. The same team invokes and owns the Workflow. | An order application starts its own fulfillment Workflow. |
| **Expose a Nexus Service Interface** | Other teams want to call your application using clean contracts. | A billing team exposes `ChargeCustomer` for other teams to consume. |
| **Call Nexus from a Workflow** | The process orchestrates multiple Nexus Operations across Namespace boundaries. | An order Workflow reserves inventory, calls billing, and arranges shipping. |
| **Use a Standalone Nexus Operation (SANO)** (Public Preview) | An application, script, or gateway needs to invoke a Nexus Service with minimal overhead and no multi-step orchestration requirements. | An admin application, which is not implemented on Temporal, calls `ProvisionAccount` through Nexus. |
| **Use a Nexus Standalone Activity (SAA)** (Pre-release) | A Service Operation performs one durable action and needs Activity retries and timeouts. | The `ChargeCustomer` Nexus Operation uses a Nexus Standalone Activity so that the Stripe API call can be made durably. |

## Decide whether you need a service boundary

Starting a Workflow or Standalone Activity directly already provides durability.
Use Nexus when callers should depend on a business capability, such as `ProvisionAccount`, while the service owner controls its implementation.
Callers use the service contract without needing to know its Workflow IDs, Task Queues, or backing implementation.
This is particularly useful across teams and isolated Namespaces.

## Choose how to call the Nexus Service

- **Call from a Workflow** when the caller needs durable business logic, coordination, or state. An existing Workflow can also call a single [Nexus Operation](/nexus/operations).
- **Use a Standalone Nexus Operation** when code outside a Workflow needs to invoke the service. The called Operation can still run a complex Workflow; a Standalone Nexus Operation removes only the need for a *caller* Workflow. That is, when the Nexus Operation is the only work you need to do, and there is no other processing required.
- An HTTP endpoint or webhook receiver can use a Standalone Nexus Operation through an adapter that makes the SDK Client call. A Standalone Nexus Operation does not itself provide a public HTTP endpoint.

## Choose how to implement the Nexus Operation

- **Use a Workflow** when the Operation needs multiple steps, durable intermediate state, timers, or coordination.
- **Use a Nexus Standalone Activity** when the Operation performs one durable action, such as calling a third-party API, writing to another system, or running a computation.

These choices work together.
For example, an existing checkout application can use a **Standalone Nexus Operation** to call a **Nexus billing service** implemented with a **Nexus Standalone Activity**.
An order **Workflow** can call that same service as part of a larger process.

---

# Nexus Endpoints

Source: https://docs.temporal.io/nexus/endpoints

> Nexus Endpoints are reverse proxies that connect Nexus callers and handlers forwarding Nexus requests to an upstream target Namespace and Task Queue that a Worker is polling.

A [Nexus Endpoint](/glossary#nexus-endpoint) is a fully managed reverse proxy for [Nexus Services](/nexus/services).
It routes requests from a caller Workflow to a target Namespace and Task Queue.
Callers only need to know the Endpoint name - the target Namespace, Task Queue, and internal implementation are encapsulated.

Workers handle Nexus requests by registering one or more Services and polling the Endpoint's target Task Queue.
Multiple Endpoints can target different Task Queues in the same Namespace.

The Endpoint description field supports markdown for documenting available Operations, contact information, or schema links.

## Reverse proxy for Nexus Services, not a general purpose proxy
A Nexus Endpoint acts as a reverse proxy for a single Nexus Service, routing requests to one target Namespace and Task Queue.

Unlike general-purpose proxies, it does not route to multiple backends. Instead, it provides a secure, managed connection to a specific upstream target, which can be in any region or cloud.
The [EndpointSpec](https://github.com/temporalio/api/blob/70cec27214ec2173974fb045667df324b45e705c/temporal/api/nexus/v1/message.proto#L182) support the following [target type](https://github.com/temporalio/api/blob/70cec27214ec2173974fb045667df324b45e705c/temporal/api/nexus/v1/message.proto#L197):

- **Worker**: Route to a target Namespace and Task Queue.

## Deploying a Nexus Endpoint

Adding an Endpoint to the [Nexus Registry](/nexus/registry) deploys it immediately. The Endpoint is available at runtime as soon as it's registered.

---

# Error Handling - Temporal Nexus

Source: https://docs.temporal.io/nexus/error-handling

> Learn about error handling for Temporal Nexus.

Nexus Operations can return errors for a caller Workflow to handle.
Errors from an asynchronous Operation's underlying Workflow propagate back to the caller.

## Errors in Nexus handlers

Nexus handlers may return [different error types](/references/failures#nexus-errors).
By default, handler errors are retryable unless they are:

- [Application Failures](/references/failures#nexus-errors) explicitly marked as non-retryable.
- [Nexus Operation errors](/references/failures#nexus-errors) that resolve an Operation as failed or canceled.
- [Non-retryable Nexus errors](/references/failures#non-retryable-nexus-errors).

When the caller's Nexus Machinery receives an error:

- **Non-retryable** - A [NexusOperationFailed](/references/events#nexusoperationfailed) event is added to the caller's Event History.
- **Retryable** - The Nexus Machinery [automatically retries](/nexus/operations#automatic-retries). These errors surface in [Pending Operations](/nexus/execution-debugging/#pending-operations).

> **💡 Tip:**
>
> Return a [specific Nexus error type](/references/failures#nexus-errors) to avoid infinite retries.
> See [errors in Nexus Operations](/references/failures#errors-in-nexus-operations) for additional details.
>

## Nexus error handling in caller Workflows

When a Nexus Operation fails, the caller receives a Nexus Operation Failure containing the operation name, token, and failure reason.
The cause field indicates the type of error (for example, Application Error or Canceled Error).

> **💡 Tip:**
> RESOURCES
>
> - [Errors in Nexus Operations](/references/failures#errors-in-nexus-operations)
> - [Nexus Errors](/references/failures#nexus-errors)
> - [Nexus Operation Failures](/references/failures#nexus-operation-failure)
>

---

# Execution Debugging - Temporal Nexus

Source: https://docs.temporal.io/nexus/execution-debugging

> Learn about execution debugging in Temporal Nexus.

Nexus supports end-to-end execution debugging across caller Workflows, Nexus Operations, and handler Workflows - even across [multi-level calls](/nexus#multi-level-calls) spanning multiple Namespaces.

## Bi-directional linking

Bidirectional links connect Nexus Operation events in the caller's Event History to corresponding events in the handler's Event History.
They are automatically wired by SDK builder functions like New-Workflow-Run-Operation, enabling click-through navigation across Namespaces, regions, and clouds in the Temporal UI.

![Bi-directional linking](https://docs.temporal.io/img/cloud/nexus/nexus-bi-directional-linking.png)

- **Forward**: From a caller's Nexus Operation event to the handler's Workflow.
- **Backward**: From the handler's Workflow back to the caller's Nexus Operation event.

## Pending Operations

Pending Nexus Operations are displayed in the UI on the Workflow details page and can be listed from the CLI using the `temporal workflow describe` command.

From the UI:

![Pending Operations](https://docs.temporal.io/img/cloud/nexus/pending-nexus-operations.png)

From the CLI:

```
temporal workflow describe

Pending Nexus Operations: 1

  Endpoint                 myendpoint
  Service                  my-hello-service
  Operation                echo
  OperationToken
  State                    BackingOff
  Attempt                  6
  ScheduleToCloseTimeout   0s
  NextAttemptScheduleTime  20 seconds from now
  LastAttemptCompleteTime  11 seconds ago
  LastAttemptFailure       {"message":"handler error (INTERNAL): internal error","applicationFailureInfo":{}}
```

[Retryable errors](/nexus/error-handling#errors-in-nexus-handlers) surface in the Pending Operation.
Non-retryable errors resolve the Operation with a [Failed](/references/events#nexusoperationfailed), [TimedOut](/references/events#nexusoperationtimedout), or [Canceled](/references/events#nexusoperationcanceled) event.

## Pending callbacks

Nexus completion callbacks are sent from the handler's Namespace to the caller's Namespace for asynchronous Operations.
These can be viewed in the UI or from the CLI using the `temporal workflow describe` command.

From the UI:

![Pending Callbacks](https://docs.temporal.io/img/cloud/nexus/nexus-callback.png)

From the CLI:

```
temporal workflow describe

Callbacks: 1

  URL               https://nexus.phil-caller-Namespace.a2dd6.cluster.tmprl.cloud:7243/Namespaces/phil-caller-Namespace.a2dd6/nexus/callback
  Trigger           WorkflowClosed
  State             Succeeded
  Attempt           1
  RegistrationTime  32 minutes ago
```

## Tracing

Temporal integrates with [OpenTelemetry](https://opentelemetry.io/) and [OpenTracing](https://opentracing.io/) to visualize call graphs across Activities, Nexus Operations, and Child Workflows.
Enable tracing by installing an interceptor on the Client or Worker:

- [Go SDK](https://github.com/temporalio/samples-go/tree/main/opentelemetry)
- [Java SDK](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/tracing)
- [Python SDK](https://github.com/temporalio/samples-python/tree/main/open_telemetry)
- [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry)
- [.NET SDK](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry)

---

# Nexus Metrics

Source: https://docs.temporal.io/nexus/metrics

> Learn about available metrics emitted by Temporal Nexus.

Nexus provides SDK metrics, Cloud metrics, and self-hosted Temporal Service metrics in addition to integrated [execution debugging](/nexus/execution-debugging).

## SDK Metrics

[SDK metrics](/references/sdk-metrics) are emitted from a Nexus Worker, including:

- [nexus_poll_no_task](/references/sdk-metrics#nexus_poll_no_task)
- [nexus_task_schedule_to_start_latency](/references/sdk-metrics#nexus_task_schedule_to_start_latency)
- [nexus_task_execution_failed](/references/sdk-metrics#nexus_task_execution_failed)
- [nexus_task_execution_latency](/references/sdk-metrics#nexus_task_execution_latency)
- [nexus_task_endtoend_latency](/references/sdk-metrics#nexus_task_endtoend_latency)

## Cloud Metrics

Temporal Cloud emits [OpenMetrics metrics](/cloud/metrics/openmetrics/metrics-reference#nexus-metrics) for both sides of a Nexus call:

- Caller-side Nexus Operation metrics report completion outcomes and schedule-to-close, schedule-to-start, and start-to-close latency.
- Handler-side Nexus request metrics report request rates, errors, and latency.

These metrics are in
[Public Preview](/evaluate/product-release-stages#public-preview).

## Self-hosted metrics

A self-hosted Temporal Service emits [Nexus metrics](/references/service-metrics#nexus-metrics), including:

- History Service metrics
- Concurrency Limiter metrics
- Frontend Service metrics

---

# NexGen

Source: https://docs.temporal.io/nexus/nexgen

> NexGen turns one schema into typed models, runtime validators, and Nexus Service definitions for Go, Java, Python, and TypeScript.

> **Pre-release** — Go, Java, Python, TypeScript
> APIs are experimental and may be subject to backwards-incompatible changes.

> **📝 Note:**
>
> For setup instructions, see the [NexGen GitHub project](https://github.com/temporalio/nexgen#installation)
>

A [Nexus Service](/nexus/services) is called across a team boundary, often by a caller written in a different language than the handler implementation and deployed on its own schedule.
When each side hand-writes its own request and response types, the two copies drift, and nothing catches it until a call fails.

[`nexgen`](https://github.com/temporalio/nexgen) generates client code for Go, Java, Python, and TypeScript from a schema file that defines the contract. The schema's types are modeled with [JSON Schema 2020-12](https://json-schema.org).
Both sides can then use code generated from the same file, which gives data validation and type safety across the languages and helps prevent drift.

For each type it emits:

- **Typed models** — an idiomatic struct, class, interface, or dataclass, with doc comments carried over from the schema.
- **A runtime validator**, automatically applied when a value is parsed off the wire and again when it is serialized onto it.
- **A [Nexus Service](/nexus/services) definition**, for a file that declares Services. The handler implements it; the caller uses it to invoke Operations.

## How it works

You write the contract once, as a JSON or YAML definition file, and run `nexgen` against it.
The generator emits contract code in Go, Java, Python, or TypeScript as requested.

Both sides use that generated code: the handler implements the Service, and the caller invokes its Operations.
Because both were generated from the same file, they agree on the contract by construction, and the generated validators enforce it at runtime on every payload.

This is what makes a Nexus Service polyglot.
A Python handler and a Go caller never share code — they share a definition file.
Generate from it in each language and they interoperate, with no coordination between the teams beyond the contract itself.

## Data validation

The generated validators check every payload against the contract when a value is parsed off the wire and again when it is serialized onto it.
Bad data is rejected at the boundary instead of reaching your Workflow or Activity.

Failures aggregate into a single error listing every violation, each naming the offending field and the constraint it broke.
A handler maps that to a `BAD_REQUEST` [Nexus handler error](/nexus/error-handling), so a malformed request tells the caller everything that was wrong in one response.

A value is validated identically in every language, which is what lets a caller and a handler written in different ones trust the same contract.
Keeping that promise is why the supported schema subset is deliberately strict: anything ambiguous, or anything that cannot be expressed the same way everywhere, is rejected at generation time rather than becoming code that validates differently in one language than another.

Three numeric and timestamp edge cases do not yet behave the same way in every language, covering negative zero, very large fractional integers, and nanosecond precision in Python.
See [Known cross-language divergences](https://github.com/temporalio/nexgen#known-cross-language-divergences) in the `nexgen` README for what each language does.

---

# Nexus Operations

Source: https://docs.temporal.io/nexus/operations

> Learn about the Nexus Operation lifecycle, execution semantics, and automatic retries for reliable execution.

[Nexus Operations](/glossary#nexus-operation) can be synchronous or asynchronous. Unlike a traditional RPC, an asynchronous Nexus Operation has an operation token that can be used to re-attach to a long-running Operation backed by a Workflow.
An Operation's lifecycle spans scheduling, reliable delivery with retries, handler execution, and result or callback completion.
Use a synchronous Nexus Operation only when the complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database.

## SDK support 

> **💡 Tip:**
> SDK GUIDES
>
> - [Go](/develop/go/nexus/feature-guide) |
>   [Java](/develop/java/nexus) |
>   [Python](/develop/python/nexus) |
>   [TypeScript](/develop/typescript/nexus) |
>   [.NET](/develop/dotnet/nexus)
>

**Caller side:** A caller Workflow executes a Nexus Operation through a [Nexus Endpoint](/nexus/endpoints) using the Temporal SDK.
A [Standalone Nexus Operation](/standalone-nexus-operation) is a top-level Nexus Operation Execution started directly by a Client, without using a caller Workflow.

**Handler side:** [Nexus Services](/nexus/services) and their Operations are registered with a Worker that polls the Endpoint's target Task Queue. Operations are defined using SDK builder functions:

- **New-Workflow-Run-Operation** - Start a Workflow as an asynchronous Operation.
- **New-Sync-Operation** - Run a synchronous Operation: invoke a Query, Signal, or Update, or execute other reliable code using the Temporal SDK Client.

## Nexus Operation lifecycle 

When a caller Workflow executes a Nexus Operation, the command is atomically handed off to the [Nexus Machinery](/glossary#nexus-machinery).
The Machinery ensures [at-least-once](#at-least-once-execution-semantics-and-idempotency) execution with [automatic retries](#automatic-retries) and reliable result delivery.

![Nexus Overview](https://docs.temporal.io/img/cloud/nexus/nexus-overview.png)

### Synchronous Operation lifecycle

Synchronous Operations must complete within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout), as measured from the caller's Nexus Machinery.

![Nexus Sync Operation Lifecycle](https://docs.temporal.io/img/cloud/nexus/nexus-sync-operation.png)

Lifecycle for a synchronous Operation (for example, to Signal, Query, or Update a Workflow, or to run other reliable code):

1. Caller Workflow executes a Nexus Operation.
1. Caller Worker issues a [ScheduleNexusOperation](/references/commands#schedulenexusoperation) command.
1. Caller Namespace records a [NexusOperationScheduled](/references/events#nexusoperationscheduled) event.
1. Caller Nexus Machinery sends the start request.
1. Handler Nexus Machinery sync-matches the request to a handler Worker.
1. Handler Worker receives a [Nexus Task](/tasks#nexus-task) by polling the Endpoint's target Task Queue.
1. Handler processes the task using **New-Sync-Operation**.
1. Handler responds with the Operation result.
1. Caller Namespace records a [Completed](/references/events#nexusoperationcompleted) or [Failed](/references/events#nexusoperationfailed) event.
1. Caller Worker polls for a Workflow Task.
1. Caller Workflow receives the result.

![Nexus](https://docs.temporal.io/img/cloud/nexus/nexus-workers-short-sync-op-sequence.png)

> **💡 Tip:**
>
> Use a synchronous Operation only when its complete execution path is highly reliable, predictably low latency, and finishes well within the [10-second handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
> Timed-out handlers are retried until the Operation's Schedule-to-Close timeout is exceeded.
>

### Asynchronous Operation lifecycle 

Asynchronous Operations can run up to [60 days](/evaluate/cloud/limits#nexus-operation-duration-limits) (the maximum Schedule-to-Close timeout in Temporal Cloud).
Differences from the synchronous lifecycle are in **bold**.

![Nexus Async Operation Lifecycle](https://docs.temporal.io/img/cloud/nexus/nexus-async-operation.png)

1. Caller Workflow executes a Nexus Operation.
1. Caller Worker issues a [ScheduleNexusOperation](/references/commands#schedulenexusoperation) command.
1. Caller Namespace records a [NexusOperationScheduled](/references/events#nexusoperationscheduled) event.
1. Caller Nexus Machinery sends the start request.
1. Handler Nexus Machinery sync-matches the request to a handler Worker.
1. Handler Worker receives a [Nexus Task](/tasks#nexus-task) by polling the Endpoint's target Task Queue.
1. Handler processes the task using **New-Workflow-Run-Operation**.
1. Handler responds with the **start Operation response**.
1. Caller Namespace records a **[NexusOperationStarted](/references/events#nexusoperationstarted)** event.
1. **Handler Workflow completes and a [Nexus Completion Callback](/glossary#nexus-async-completion-callback) is delivered to the caller's Nexus Machinery.**
1. Caller Namespace records a [Completed](/references/events#nexusoperationcompleted) or [Failed](/references/events#nexusoperationfailed) event.
1. Caller Worker polls for a Workflow Task.
1. Caller Workflow receives the result.

![Nexus](https://docs.temporal.io/img/cloud/nexus/nexus-workers-short-async-op-sequence.png)

### Executing code from a synchronous handler 

Synchronous handlers can execute code directly but must complete within the [handler deadline](/evaluate/cloud/limits#nexus-operation-request-timeout).
Use the Temporal SDK Client to invoke Signals, Queries, Updates, or other reliable code.

> **⚠️ Caution:**
>
> Use [async Operations](#asynchronous-operation-lifecycle) for long-running work.
> Repeated sync handler failures can trip the [circuit breaker](#circuit-breaking), blocking all Operations from that caller to the Endpoint.
>

![Nexus Operations with Arbitrary Code](https://docs.temporal.io/img/cloud/nexus/nexus-sync-operation-arbitrary-code.png)

### System interactions

Nexus uses the same queue-based Worker architecture as the rest of Temporal.
Workers interact with their Namespace gRPC endpoint. Nexus Machinery on both sides handles cross-Namespace communication.

![Nexus Queue-based Worker Architecture](https://docs.temporal.io/img/cloud/nexus/nexus-workers.png)

At a high level, when a caller Workflow executes a Nexus Operation:

1. The caller Worker schedules the Operation with a [ScheduleNexusOperation command](/references/commands#schedulenexusoperation), atomically handing off execution to the caller's Nexus Machinery.
2. The handler Worker receives a [Nexus Task](/tasks#nexus-task) by polling the Endpoint's target Task Queue.
3. The handler processes the task and returns the result (synchronous) or an Operation token (asynchronous).
4. The caller's Nexus Machinery records a NexusOperation event ([Started](/references/events#nexusoperationstarted), [Completed](/references/events#nexusoperationcompleted), [Failed](/references/events#nexusoperationfailed), [Canceled](/references/events#nexusoperationcanceled), or [TimedOut](/references/events#nexusoperationtimedout)) in the caller's Event History.

## Automatic retries 

Once the caller Workflow schedules an Operation with the caller's Temporal Service, the caller's Nexus Machinery keeps trying to start the Operation.
If a [retryable Nexus error](/references/failures#nexus-errors) is returned the Nexus Machinery will retry until the Nexus Operation's [Schedule-to-Start timeout](#schedule-to-start-timeout) or [Schedule-to-close timeout](#schedule-to-close-timeout) is exceeded.

For example, if a Nexus handler returns a [retryable error](/references/failures#nexus-errors), or an [upstream timeout](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors) is encountered by the caller, the Nexus request will be retried up to the [default Retry Policy's](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/components/nexusoperations/config.go#L204) max attempts and expiration interval.

> **📝 Note:**
> This differs from Activity and Workflow error handling.
> See [errors in Activities](/references/failures#errors-in-activities) and [non-retryable errors](/references/failures#non-retryable).

To control retry behavior, return a [non-retryable Nexus error](/references/failures#non-retryable-nexus-errors).
See [errors in Nexus handlers](/nexus/error-handling#errors-in-nexus-handlers).

## Timeouts 

Nexus Operations support three types of timeouts that control how long the caller is willing to wait at different stages of the Operation lifecycle.
These timeouts are set by the caller when scheduling the Operation.

### Schedule-to-Close timeout 

The Schedule-to-Close timeout limits the total duration from when the Operation is scheduled to when it completes.
This is the overall timeout for the entire Operation.
The Nexus Machinery [automatically retries](#automatic-retries) failed requests internally until this timeout is exceeded, at which point the Operation fails with a [NexusOperationTimedOut](/references/events#nexusoperationtimedout) event.

This timeout covers the full [Nexus Operation lifecycle](/nexus/operations#operation-lifecycle). Asynchronous Operations are scheduled, started, and completed. Synchronous Operations don't have an intermediate started state because they complete as part of the start request.

In Temporal Cloud, the [maximum Schedule-to-Close timeout is 60 days](/evaluate/cloud/limits#nexus-operation-duration-limits).

### Schedule-to-Start timeout 

The Schedule-to-Start timeout limits how long the caller is willing to wait for the Operation to be started (or completed, if synchronous) by the handler.
If the Operation is not started within this timeout, it fails with `TIMEOUT_TYPE_SCHEDULE_TO_START`.

If not set or set to zero, no Schedule-to-Start timeout is enforced.

> **📝 Note:**
>
> The Schedule-to-Start timeout requires Temporal Server version 1.31.0 or later.
>

### Start-to-Close timeout 

The Start-to-Close timeout limits how long the caller is willing to wait for an asynchronous Operation to complete after it has been started.
If the Operation does not complete within this timeout after starting, it fails with `TIMEOUT_TYPE_START_TO_CLOSE`.

This timeout only applies to asynchronous Operations.
Synchronous Operations ignore this timeout because they complete as part of the start request.

If not set or set to zero, no Start-to-Close timeout is enforced.

> **📝 Note:**
>
> The Start-to-Close timeout requires Temporal Server version 1.31.0 or later.
>

## Circuit breaking 

Nexus implements circuit breaking per caller-Namespace/Endpoint pair ("destination pair").
Each destination pair trips and resets independently.
By default, the circuit breaker activates after 5 consecutive [retryable errors](/references/failures#nexus-errors).

After tripping, the circuit breaker enters the _open_ state and stops sending requests.
After 60 seconds, it transitions to _half-open_, allowing a single probe request.
If the probe succeeds, the circuit breaker returns to _closed_ (normal operation).
If it fails, the circuit breaker returns to _open_ for another 60 seconds.

> **📝 Note:**
> Note that worker availability affects the circuit breaker as well. 
> If no workers are polling the handler task queue — due to a deployment issue, crash, or scale-down — Nexus requests will time out.
> Consecutive timeouts count as retryable errors and will trip the circuit breaker just as application-level errors do. 
> Ensure handler workers maintain sufficient availability to avoid unintended circuit breaker trips.
>

![Flow chart showing the states of the Temporal Nexus Circuit Breaker](https://docs.temporal.io/img/cloud/nexus/circuit-breaker.png)

Circuit breaker state surfaces in [Pending Nexus Operations](/nexus/execution-debugging#pending-operations) and [Pending Callbacks](/nexus/execution-debugging#pending-callbacks).
Check it in the UI, CLI, or `DescribeWorkflowExecution` API.

When open, pending Operations show a `Blocked` state with a `BlockedReason`:

![Circuit Breaking](https://docs.temporal.io/img/cloud/nexus/circuit-breaking.png)

Different Operations within the same destination pair contribute to the trip count.
A given Operation may have fewer than 5 attempts when the circuit breaker opens.

From the CLI:

```sh
temporal workflow describe -w my-workflow-id
```

```sh
Pending Nexus Operations: 1

  Endpoint                 my-nexus-endpoint
  Service                  nexus-playground
  Operation                sync-op-ok
  State                    Blocked
  Attempt                  1
  LastAttemptFailure       {"message":"handler error (UPSTREAM_TIMEOUT): upstream timeout",...}
  BlockedReason            The circuit breaker is open.
```

Cancellation requests surface the same pattern with `CancelationState: Blocked` and `CancelationBlockedReason`.

```sh
Execution Info:
  WorkflowId            my-workflow-id
  ...

Pending Activities: 0
Pending Child Workflows: 0
Pending Nexus Operations: 1

  Endpoint                            my-nexus-endpoint
  Service                             nexus-playground
  Operation                           async-op-workflow-wait-for-cancel
  OperationToken                      eyJ2IjowLCJ0IjoxLCJucyI6Im5zIiwid2lkIjoidyJ
  State                               Started
  Attempt                             1
  ScheduleToCloseTimeout              1d 0h 0m 0s
  LastAttemptCompleteTime             51 seconds ago
  CancelationState                    Blocked
  CancelationAttempt                  5
  CancelationRequestedTime            37 seconds ago
  CancelationLastAttemptCompleteTime  27 seconds ago
  CancelationLastAttemptFailure       {"message":"handler error (UPSTREAM_TIMEOUT): upstream timeout","cause":{"message":"upstream timeout","applicationFailureInfo":{"type":"NexusFailure"}},"applicationFailureInfo":{"type":"NexusHandlerError"}}
  CancelationBlockedReason            The circuit breaker is open.
```

## Execution semantics 

### At-least-once execution semantics and idempotency

The Nexus Machinery provides reliable execution with at-least-once execution semantics for a Nexus Operation, until the caller's [Schedule-to-Close timeout](#schedule-to-close-timeout) is exceeded, at which time the overall Nexus Operation times out.
The Machinery retries on handler timeouts or retryable errors, so a handler may be invoked multiple times for the same Operation.

Nexus Operation handlers should be idempotent, similar to Activities.
Not strictly required in all cases, but highly recommended.

### Exactly-once execution semantics

To upgrade to exactly-once, back your Operation with a Workflow that uses a WorkflowIDReusePolicy of RejectDuplicates.
This allows only one Workflow Execution per Workflow ID within a Namespace for the Retention Period.

## Cancelation

Cancelling a caller Workflow automatically propagates to all pending Nexus Operations and their underlying handler Workflows.
A canceled handler Workflow reports a [Canceled Failure](/references/failures#cancelled-failure) to the caller.

## Termination

Terminating a caller Workflow abandons all pending Nexus Operations. Unlike cancellation, no cancel request is sent to the 
handler Namespace, so handler Workflows continue running indefinitely, consuming resources until they time out or are manually 
stopped. Because the handler runs in a separate Namespace, it has no signal that the caller is gone, making orphaned Operations 
difficult to detect and correlate. If the Nexus Operation was part of a multi-step process, termination also leaves no opportunity 
to run compensation logic, potentially leaving the system in a partially completed state.
Prefer [cancellation](#cancelation) when possible.

## Versioning 

Task Routing is the simplest way to version Nexus service code.
For backward-incompatible changes, use a different Service name and Task Queue (for example, `prod.payments.v2`).
Callers migrate to the new version on their own deployment schedule.

## Attaching multiple Nexus callers to a handler Workflow 

Operations started with [New-Workflow-Run-Operation](/nexus/operations#sdk-support) automatically attach a completion Callback to the handler Workflow.
Additional callers can attach to the same handler Workflow using a [Conflict-Policy of Use-Existing](/workflow-execution/workflowid-runid#workflow-id-conflict-policy).

Each handler Workflow has a [Callback limit](/workflow-execution/limits#workflow-execution-callback-limits) (configurable for self-hosted, see [Cloud limits](/evaluate/cloud/limits#per-workflow-callback-limits) for Temporal Cloud).
Callers that exceed the limit receive an error.

When a handler Workflow uses [Continue-As-New](/workflow-execution/continue-as-new), existing completion Callbacks are copied to the new Execution.
The previous Execution's Callbacks remain in `Standby` state indefinitely.

---

# Nexus Patterns

Source: https://docs.temporal.io/nexus/patterns

> Common patterns for building Nexus Services including the collocated pattern and the router-queue pattern for flexible deployment topologies.

There are two common patterns for building and deploying [Nexus Services](/nexus/services): 
- **[Collocated pattern](#collocated-pattern)**: Runs on the same Workers as your existing Workflows and Activities. Use by default.
- **[Router pattern](#router-queue-pattern)**: Separates Nexus routing from Workflow execution. A dedicated Nexus Worker on a “router” Task Queue routes Operations to Workflows on other Task Queues. 
Use when you need independent scaling, different IAM permissions per Worker fleet, or want to add Nexus to without modifying existing Workers.

## Collocated pattern (default) 

The **collocated pattern** runs Nexus Operation handlers in the same Worker and on the same Task Queue as the underlying Workflows.
This is the default and simplest deployment.

The Nexus Endpoint targets the same Task Queue used by the underlying Workflows.
A single Worker registers both Nexus Services and Workflow types, so everything runs together.

### Why start here

- **Simplest setup:** One Worker, one Task Queue, one deployment. No extra infrastructure.
- **Eager Workflow Start:** When the handler starts a Workflow in the same Worker, you can use [Eager Workflow Start](/develop/worker-performance#eager-workflow-start) to execute the first Workflow Task locally without an extra call to the Temporal Server - while still recording durable state. If the process crashes, the Workflow resumes on another Worker.
- **Clean facade:** Operations act as a stable contract. You can change the underlying implementation (Signal today, Workflow tomorrow) without impacting callers.

### When to use this pattern

- Getting started with Nexus.
- The same team owns both the Nexus Service and underlying Workflows.
- You don't need to scale Nexus routing separately from Workflow execution.
- You are setting up a simple test environment

Use this pattern by default unless you have a good reason to use the Router-queue pattern below

## Router-queue pattern

The **router-queue pattern** separates Nexus routing from Workflow execution. A dedicated Nexus Worker on a "router" Task Queue routes Operations to Workflows on other Task Queues in the same Namespace.

### When to use this pattern

- **Separate scaling:** Scale Nexus routing independently from Workflow execution.
- **Dedicated routing layer:** A single Nexus Worker routes requests to multiple Workflow types on different Task Queues.
- **Different IAM permissions:** Worker fleets behind different Task Queues may have different IAM permissions to different underlying resources.
- **Avoid modifying existing Workers:** Add a router Worker to a Namespace without changing any existing Workers or Workflows.

### How it works

1. Register a Nexus Worker that polls a dedicated "router" Task Queue.
2. Configure the Nexus Endpoint's target Task Queue to point to this router Task Queue.
3. In each Nexus Operation handler, specify a different target Task Queue in the Workflow start options.
4. Existing Workers continue to poll their own Task Queues and execute the Workflows started by the router.

### Production usage

Used in production by organizations running self-service platforms where a central gateway routes requests to domain-specific Namespaces and Task Queues.
The router Worker is lightweight - it only handles routing logic.

---

# Nexus Registry

Source: https://docs.temporal.io/nexus/registry

> Use the Nexus Registry to manage Nexus Endpoints.

The [Nexus Registry](/glossary#nexus-registry) manages [Nexus Endpoints](/nexus/endpoints). 
Developers can advertise available Endpoints and Services, so others can find and use them in their caller Workflows.
Adding an Endpoint to the Registry deploys it for immediate runtime use.
Endpoint names must be unique within the Registry.
In Temporal Cloud, the Registry is global across your entire Account, spanning all Namespaces.
Endpoint names remain unique Account-wide, and caller Workflows in any Namespace can use an Endpoint if they are on its allowlist.
The Endpoint resource is Project-scoped for management. Creating, updating, and deleting an Endpoint requires access to that Project **and** Namespace Admin on the target Namespace. See [Roles and permissions](#roles-and-permissions) and [Use Nexus across Projects](/cloud/projects#use-nexus-across-projects).
In self-hosted deployments, the Registry is scoped to a Cluster.

## View and manage Nexus Endpoints

Manage Endpoints using the Temporal UI, CLI, Terraform provider, or [Cloud Ops API](/ops).

> **💡 Tip:**
> RESOURCES
>
> - [Terraform support](/cloud/terraform-provider#manage-temporal-cloud-nexus-endpoints-with-terraform) for Temporal Cloud.
> - [tcld nexus](/cloud/tcld/nexus) for Temporal Cloud.
> - [temporal operator nexus](/cli/command-reference/operator#nexus) for self-hosted deployments.

### Search for a Nexus Endpoint

Search by Endpoint name or target Namespace.

![Nexus Endpoints](https://docs.temporal.io/img/cloud/nexus/nexus-endpoints-ss.png)

The details page shows the target Namespace, Task Queue, and description rendered as markdown.

![Nexus Billing](https://docs.temporal.io/img/cloud/nexus/nexus-billing-ss.png)

### Create a Nexus Endpoint

Creating an Endpoint includes setting an Access Policy - the allowlist of caller Namespaces permitted to use the Endpoint.
No callers are allowed by default, even if in the same Namespace as the Endpoint target.

![Create Nexus Endpoint](https://docs.temporal.io/img/cloud/nexus/create-nexus-endpoint-ss.png)

### Edit a Nexus Endpoint

Everything except the Endpoint name can be edited.
New Operations route to the updated target immediately.

> **⚠️ Caution:**
> Changing the target Namespace
>
> - **In-flight async Operations** - Completion callbacks point to the original handler Namespace and are unaffected, but Cancel requests route to the new target.
> - **Workflow ID uniqueness** - IDs are scoped per Namespace. Signal-With-Start creates a new Workflow in the new target even if the same ID is active in the old target, resulting in potential duplicates.
> - **Recommendation:** Drain existing Nexus Operations and underlying handler Workflows before changing the target Namespace.
>

![Edit Nexus Endpoint](https://docs.temporal.io/img/cloud/nexus/target-namespace-ss.png)

### Configure runtime access controls

The Access Policy controls which caller Namespaces can use an Endpoint at runtime.
No callers are allowed by default.

See [Runtime Access Controls](/nexus/security#runtime-access-controls) for details.

![Configure runtime access controls](https://docs.temporal.io/img/cloud/nexus/create-nexus-endpoint-ss.png)

## Roles and permissions

> **ℹ️ Info:**
>
> The Nexus Registry uses default roles in Temporal Cloud. For self-hosted deployments, you can implement [custom Authorizers](/self-hosted-guide/security#authorizer-plugin) to restrict access.
>

In Temporal Cloud, Nexus Registry respects RBAC permissions, and restricts functionality based on user role:

| Action                     | Required permissions                                      |
|---------------------------|-----------------------------------------------------------|
| View or search Endpoints  | Read-only role (or higher) at the Account level           |
| Manage Endpoints          | Account Developer (or higher), or Project Contribute (or higher) on the Endpoint's Project, **and** Namespace Admin on the target Namespace. Effective Namespace Admin on the target Namespace is required separately. Changing an Endpoint's target Namespace requires Namespace Admin on both the current target Namespace and the new target Namespace. |

See [Nexus security in Temporal Cloud](/cloud/nexus/security).

## Automate Nexus Endpoint provisioning and lifecycle management

There are two ways to automate endpoint provisioning and lifecycle management: Terraform and the Operator API.

> **💡 Tip:**
> RESOURCES
>
> - [Terraform support](/cloud/terraform-provider#manage-temporal-cloud-nexus-endpoints-with-terraform) for Temporal Cloud.
> - [Cloud Ops API](/ops) for Temporal Cloud.
> - [Operator API](https://github.com/temporalio/api/blob/main/temporal/api/operatorservice/v1/service.proto) for self-hosted deployments.
>

---

# Security in Temporal Nexus

Source: https://docs.temporal.io/nexus/security

> Explore the security aspects of Temporal Nexus, including managing Nexus Endpoints, runtime access control, secure routing, and payload encryption to ensure safe and reliable cross-namespace operations.

Temporal Cloud provides built-in Endpoint access controls and secure connectivity across Namespaces.
Self-hosted deployments can implement [custom Authorizers](/self-hosted-guide/security#authorizer-plugin).

## Runtime access controls 

In Temporal Cloud, each Endpoint has an access control policy: an allowlist of caller Namespaces.

![Nexus Security](https://docs.temporal.io/img/cloud/nexus/nexus-workers-short.png)

Workers authenticate with their Namespace using mTLS or API key.
When a caller Workflow executes a Nexus Operation, Temporal Cloud verifies the caller's Namespace is in the Endpoint's allowlist before routing the request to the handler.
Temporal Cloud acts as a trusted broker across Namespace boundaries.

See [Configure runtime access controls](/nexus/registry#configure-runtime-access-controls).

## Secure connectivity 

> **ℹ️ Info:**
>
> Temporal Cloud has built-in secure connectivity across all Namespaces in an Account.
>
> Self-hosted deployments rely on the Temporal Cluster being secure.
>

Temporal Cloud secures all Nexus communication:

- Workers authenticate to their Namespace using mTLS or API key.
- mTLS encrypts all cross-Namespace Nexus traffic (start, cancel, and completion callbacks) across cells and regions.
- Endpoints are only accessible from within a Temporal Cloud Account through the Temporal SDK - not externally accessible.

## Payload encryption and Data Converter 

Nexus uses the same Data Converter as Workflows and Activities - JSON, Proto, and binary payloads are all supported.
If you use a Codec for encryption, it also encrypts Nexus payloads.

Caller and handler Workers must have compatible Data Converters.
Payloads are encrypted by the sender (caller encrypts input, handler encrypts result).

Three common approaches for cross-Namespace payload encryption:

### Option 1: Same encryption key 

Both Namespaces share the same encryption key.
Simplest approach - no additional configuration needed.

### Option 2: Pass KMS key ID in payload metadata 

Each Namespace uses its own encryption key, with the KMS key ID passed in Temporal payload metadata.
The receiver reads the key ID from metadata and decrypts using KMS IAM permissions.

Works bi-directionally: caller encrypts input with the caller's key, handler decrypts using the key ID from metadata, then encrypts the result with the handler's key.
The Codec Server needs KMS decrypt permissions for all relevant keys.

See the [encryption sample](https://github.com/temporalio/samples-go/blob/main/encryption/data_converter.go) and the [reference-app-orders-go data converter](https://github.com/temporalio/reference-app-orders-go/blob/main/app/temporalutil/data_converter.go).

### Option 3: Wrapper types for endpoint-specific encryption keys 

Use wrapper types (for example, `EndpointValue`) so the Data Converter selects an Endpoint-specific encryption key.
This encrypts only Nexus traffic with a dedicated key, without sharing Namespace keys across teams.

> **⚠️ Warning:**
> Limited to string-based operation names
>
> When calling `ExecuteOperation` with a non-string operation (for example, an operation definition or reference), the wrapper changes the input type and `ExecuteOperation` fails type-checking:
>
> ```
> cannot assign argument of type "<WrapperType>" to type "<ActualInputType>"
> ```
>
> This approach only works when the operation is passed as a string name.
> For typed operations, pass the Nexus endpoint info through the Workflow context instead of the wrapped input.
>

See the [draft endpoint-based encryption sample](https://github.com/temporalio/samples-go/compare/main...bergundy:samples-go:nexus-encryption-by-endpoint).

### Choosing an approach

Options 1 and 2, where both sides share the same key or flow the KMS key ID in payload metadata, work with the standard Data Converter.
Option 3 is more advanced and is intended for teams that don't want to share their Namespace encryption keys with other teams.

## Nexus Registry security 

See [Nexus Registry Roles and Permissions](/nexus/registry#roles-and-permissions).

---

# Nexus services

Source: https://docs.temporal.io/nexus/services

> Nexus Services are named collections of arbitrary-duration Nexus Operations that provide a microservice contract suitable for sharing across team boundaries.

[Nexus Services](/glossary#nexus-service) are named collections of [Nexus Operations](/nexus/operations) that provide a contract for sharing across team boundaries.
A [Nexus Endpoint](/nexus/endpoints) exposes Services for callers to use.

Services are registered in a Worker that polls the Endpoint's target Task Queue.
Multiple Services can run in the same Worker.
Services typically run alongside the Workflows they abstract, or in a dedicated router Worker using the [router-queue pattern](/nexus/patterns#router-queue-pattern).

Callers reference a Service by name when executing a Nexus Operation.

You can hand-write a Service definition, or generate it from a schema file with [NexGen](/nexus/nexgen), which emits the definition along with typed models and runtime validators for each language.

---

# Nexus Standalone Activity

Source: https://docs.temporal.io/nexus/standalone-activity

> Back a Nexus Operation with a Standalone Activity when the work is a single durable step, with no Workflow wrapped around it.

> **Pre-release** — Go, Java, Python, TypeScript, .NET
> APIs are experimental and may be subject to backwards-incompatible changes.

> **📝 Note:**
> Not the same as a Standalone Nexus Operation
>
> The two names are close and describe opposite ends of the call.
> A [Standalone Nexus Operation](/standalone-nexus-operation) is about the **caller**: a Client starts an Operation directly, with no caller Workflow around it.
> A Nexus Standalone Activity is about the **handler**: an Operation is backed by a single Activity, with no Workflow behind it.
> They are independent choices, and either can be used without the other.
> See [Choosing a Nexus approach](/nexus/choosing-an-approach) for how to decide each one.
>

An Activity-backed [Nexus Operation](/nexus/operations) runs a [Standalone Activity](/standalone-activity) and completes when that Activity returns.
Use it when the work behind an Operation is one durable step rather than a process: calling an external API, running a computation, writing to another system.

Two things combine to make this happen.
The [Activity](/activities) supplies durability — retries on the policy you set, timeouts you control, and a record of every attempt.
The Operation supplies a typed contract and a [Namespace](/namespaces) boundary, so another team can call it without sharing your code, your deployment, or write access to your Namespace.

Because the Activity carries the durability, no Workflow is needed behind the Operation.
A Workflow wrapping a single Activity costs two [Billable Actions](/cloud/actions-usage#actions-in-workflows) in Temporal Cloud — one to start the Workflow, one to start the Activity — whereas a Standalone Activity costs one.
Retries and heartbeats are billed the same way in either shape.

## Required options

Starting an Activity this way needs values a Workflow-called Activity does not, because there is no parent Workflow to supply them:

- **An Activity ID**, unique within the Namespace. Deriving it from the Nexus request ID makes the start idempotent, so a retried request targets the same Activity Execution instead of sending a second notification or charge.
- **A timeout.** At least one of start-to-close or schedule-to-close.

The Task Queue is optional and defaults to the one the Operation is running on. Set it explicitly to run the Activity on its own Worker fleet.

## Cancellation

An Activity is not interrupted by a cancellation request the way a Workflow is.
The Worker only learns about it on the next Heartbeat, so an Activity that never Heartbeats runs until it completes or times out.
Nothing here is Nexus-specific — see [Activity Cancellation](/activity-execution#cancellation).

Back an Operation with a **Workflow** instead when the work has more than one step, needs to wait for something, needs to receive [messages](/sending-messages), or needs durable intermediate state.

---

# Temporal Operation Handler

Source: https://docs.temporal.io/nexus/temporal-operation-handler

> The Temporal Operation Handler is a single handler type that backs a Nexus Operation with a Workflow, an Update, or an Activity, and links every Execution back to the caller.

> **Pre-release** — Go, Java, Python, TypeScript, .NET
> APIs are experimental and may be subject to backwards-incompatible changes.

Temporal has unified the Workflow handler and the synchronous Operation handler into a single handler, and added the ability to back an Operation with a [Standalone Activity](/nexus/standalone-activity).

What runs behind an Operation remains private to the handler, so you can change it later without touching the contract or any caller.

## The Nexus-aware Client

The Operation handler receives a context, the Operation input, and a Client.

That Client is not an ordinary Temporal Client.
It propagates [bidirectional links](/nexus/execution-debugging#bi-directional-linking) and request IDs on every call, so caller-side and handler-side Executions are connected in the UI and in [Event History](/encyclopedia/event-history) without wiring anything.
Constructing your own Client inside a handler works, but the Executions it starts are not linked back to the caller.

It exposes two kinds of call:

- **Async backings**, at most one per Operation invocation. These determine what the Operation *is*, and their result reaches the caller through the Nexus completion callback. Starting a Workflow, starting an Activity, and starting a Workflow Update are all async backings.
- **Sync messaging**, as many as you need. Signals and Signal-with-Start take effect during the handler call and do not require an async backing.

Deriving the backing Execution's ID from the Nexus request ID keeps a retried start request targeting the same Execution instead of creating a second one.

---

# Cloud Ops API

Source: https://docs.temporal.io/ops

> The Temporal Cloud Operations API (Cloud Ops) allows programmatic management of Temporal Cloud Control Plane resources.

The Temporal Cloud Operations API, or the Cloud Ops API, is an open source, public [HTTP API](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction) and [gRPC API](https://github.com/temporalio/cloud-api/tree/main) for programmatically managing Temporal Cloud Control Plane resources, including [Namespaces](/cloud/namespaces), [Users](/cloud/manage-access/users), [Service Accounts](/cloud/manage-access/service-accounts), [API keys](/cloud/api-keys), and others. The Temporal Cloud [Terraform Provider](/cloud/terraform-provider), [tcld CLI](/cloud/tcld), and Web UI all use the Cloud Ops API.

## Develop applications with the Cloud Ops API

You can use the HTTP API or the gRPC API depending on how you need to integrate with your platform.

- HTTP: `https://saas-api.tmprl.cloud`
- gRPC: `saas-api.tmprl.cloud:443`

If you connect to Temporal Cloud over [AWS PrivateLink](/cloud/connectivity/aws-connectivity) or [GCP Private Service Connect](/cloud/connectivity/gcp-connectivity), configure private DNS for `saas-api.tmprl.cloud` so control-plane clients (Cloud Ops API, Terraform, tcld) reach the private endpoint.

### Prerequisites

These prerequisites are required for using either HTTP or gRPC.

- A Temporal Cloud [User](/cloud/manage-access/users) or [Service Account](/cloud/manage-access/service-accounts)
- An [API Key](/cloud/api-keys#manage-api-keys) for authentication (owned by that User or Service Account)

Required [roles and permissions](/cloud/manage-access/roles-and-permissions) vary by operation. Some account-wide management operations require Account Owner or Global Admin. Others work with Developer, Finance Admin, Namespace Admin, or custom roles.

### Use cases

Some common reasons you might use the API are to:

- Provision Namespaces per environment or tenant via pipelines.
- Bootstrap new projects by creating users, assigning roles, and creating Namespaces via custom scripts.
- Rotate service account keys on a schedule with a job.
- Audit and report access across orgs with scheduled HTTP requests.

### Using HTTP

[The HTTP API](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction) supports the same operations as the [gRPC API](#using-grpc), but it's usable via standard HTTP methods and authentication. This may be a more convenient option if you are writing automation scripts for CI/CD or you can't use gRPC due to network policies, proxies, tooling gaps, or language/runtime constraints. Since it's standard HTTP, it's language agnostic giving you the ability to run cloud operations consistently.

> **📝 Note:**
> This *does not* allow interaction with individual Workflows or Activities via HTTP.

### Using gRPC

*For Go developers:*
- Use the [Go SDK](https://github.com/temporalio/cloud-sdk-go) for the simplest setup experience

*For other programming languages:*
- Basic familiarity with gRPC and Protocol Buffers (protobuf)
- [Protocol Buffers](https://github.com/protocolbuffers/protobuf/releases)
- [gRPC](https://grpc.io/docs/languages/) in your preferred programming language

You can use the provided proto files to generate client libraries in your desired programming language, and then use that client to access the gRPC API. You can also find the [full gRPC docs on Buf](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CloudService).

#### Using the Go SDK

If you're developing in Go, we recommend using the [Go SDK](https://github.com/temporalio/cloud-sdk-go) which provides pre-compiled Go bindings and a more idiomatic interface. The Go SDK handles all the protobuf compilation and provides ready-to-use Go types and client interfaces. You can also use the [Go samples](https://github.com/temporalio/cloud-samples-go) to help you get started with the Cloud Ops API using the Go SDK.

To start using the Go SDK with the Cloud Ops API, follow these steps:

1. Install the Go SDK:
   ```go
   go get go.temporal.io/cloud-sdk@latest
   ```

2. Import and use the SDK:
   ```go
   import (
       "go.temporal.io/cloud-sdk/cloudclient"
   )
   ```

3. The Go SDK provides pre-built client interfaces that handle authentication and connection setup. Refer to the [Go samples](https://github.com/temporalio/cloud-samples-go) for detailed usage examples, including [Cloud Ops API client setup](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go).

The Go SDK eliminates the need to work directly with generated protobuf files and provides a more idiomatic Go experience.

#### Compile the API and use the generated code (For other languages)

For programming languages other than Go, download the gRPC protobufs from the [Cloud Ops API repository](https://github.com/temporalio/cloud-api/tree/main/temporal/api/cloud) and compile them manually.

Use [gRPC](https://grpc.io/docs/) to compile and generate code in your preferred [programming language](https://grpc.io/docs/#official-support). The steps below use Python as an example and require [Python's gRPC tools](https://grpc.io/docs/languages/python/quickstart/#grpc-tools) to be installed, but the approach can be adapted for other supported programming languages.

You can also generate clients from the published Buf module at [buf.build/temporalio/cloud-api](https://buf.build/temporalio/cloud-api).

1. Clone the Temporal Cloud API repository:

   ```command
   git clone https://github.com/temporalio/cloud-api.git
   cd cloud-api
   ```

2. Compile the Protobuf files from the repository root (protos live under `temporal/` and import each other):

   ```command
   python -m grpc_tools.protoc \
     -I. \
     --python_out=. \
     --grpc_python_out=. \
     $(find temporal -name '*.proto')
   ```

   - `-I.` adds the repository root to the import path.
   - `--python_out=` and `--grpc_python_out=` set the output directories for generated classes.
   - `find temporal -name '*.proto'` includes all Cloud Ops API protos and their dependencies under `temporal/`.

   After compiling the Protobuf files, you will have generated code files in your project directory.
   These files enable interaction with the Temporal Cloud API in your chosen programming language.

3. Import the generated files into your application and call the Cloud Ops API. Handle authentication with an API key and set the API version header as described in [Usage guidelines](#usage-guidelines).

This approach can be adapted for other programming languages by following their respective import and usage conventions for the generated code files.

## Usage guidelines

When interacting with the Temporal Cloud Ops API, follow these guidelines:

### API version header

Use the `temporal-cloud-api-version` header to select an API version. The backend uses this version to safely mutate resources. The current API version is in the [`cloud-api` VERSION file](https://github.com/temporalio/cloud-api/blob/main/VERSION).

**gRPC**

gRPC clients must send a `temporal-cloud-api-version` header on every request.

**HTTP**

For HTTP clients, the version header is optional. If omitted, the HTTP gateway defaults it to the latest API version. This supports simple `curl` usage without looking up a version first.

For production HTTP automation, still pin an explicit version so behavior does not change when the gateway’s latest version advances.

### API versioning and deprecation

We recommend using the latest Cloud Ops API version for new integrations, and keeping your client version up to date to within one year of the latest release instead of pinning to an older version indefinitely.

Temporal supports each Cloud Ops API version for at least one year after its release. Before a version becomes unsupported, Temporal provides at least six months’ notice and migration guidance if you need to update your integration.

Exceptions to these timelines may be necessary for critical security, legal, or service-integrity reasons. Temporal provides notice whenever practicable.

### Connection URL

- gRPC: `saas-api.tmprl.cloud:443`
- HTTP: `https://saas-api.tmprl.cloud`

### Engagement steps

1. Generate an [API Key for authentication](/cloud/api-keys#manage-api-keys). Required permissions vary by operation; see [roles and permissions](/cloud/manage-access/roles-and-permissions).
2. Establish a secure connection. For Go, see [Cloud Ops API client setup](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go).
3. Execute operations. For request and response messages, see `cloudservice/v1/request_response.proto`. For services, see `cloudservice/v1/service.proto`.

## Availability

Temporal targets 99.99% monthly availability for the Cloud Ops API across its HTTP and gRPC interfaces.

Availability measures whether valid Cloud Ops API requests are successfully served. Server-side failures count against this objective; client-caused errors, including invalid requests and authentication or authorization failures, do not. Responses due to documented rate limits (`ResourceExhausted` for gRPC and HTTP 429) are also excluded because they represent intentional throttling rather than service unavailability.

This availability objective applies only to the Cloud Ops API. It is not a contractual service-level agreement and does not change the Temporal Cloud [SLA](https://temporal.io/sla). Temporal Service operations are covered separately; see the Temporal Cloud SLA for the APIs it covers.

## Rate limits

The Temporal Cloud Operations API implements rate limiting to ensure system stability and fair usage across all users. Rate limits are applied based on identity type, with different limits for users and service accounts.

### Account-level rate limit

**Total rate limit: 160 requests per second (RPS)**

This limit applies to all requests made to the Temporal Cloud Control Plane by any client (tcld, UI, Cloud Ops API) or identity type (user, service account) within your account. The total account throughput cannot exceed the limit regardless of the number of users or service accounts making requests.

### Per-identity rate limits

**User rate limit: 40 RPS per user**

This limit applies to all requests made by each user through any client (tcld, UI, Cloud Ops API), regardless of the authentication method used (SSO or API keys).

**Service account rate limit: 80 RPS per service account**

This limit applies to all requests made by each service account through any client (tcld, Cloud Ops API).

**Asynchronous operations: 10 concurrent operations at a time**

By default, each account can have up to 10 long-running (asynchronous) mutating operations in flight at once. This limit applies to a subset of create, update, and delete operations (for example, Namespace, User, API Key, Export Sink, and Service Account mutations), not every RPC that returns an async operation.

### Important considerations

- Rate limits are enforced across all Temporal Cloud Control Plane operations
- Multiple clients used by the same identity (user or service account) share the same rate limit
- Authentication method (SSO, API keys) does not affect rate limiting
- These limits help ensure system stability and prevent any single account or identity from overwhelming the service

### Request limit increases

If your use case requires higher rate limits, you can request an increase by [submitting a support ticket](/evaluate/cloud/support#support-ticket). When requesting a limit increase, please provide:

- Your current usage patterns and requirements
- The specific limits you need increased
- A description of your use case and why higher limits are necessary

### Provide feedback

Your input is valuable!

You can provide feedback through the following channels:

- Submit request or feedback through a [support ticket](/evaluate/cloud/support#support-ticket)
- Open an issue in the [GitHub Repo](https://github.com/temporalio/cloud-api)

---

# Parent Close Policy

Source: https://docs.temporal.io/parent-close-policy

> Understand the Parent-Child Workflow relationship, including when to use Child Workflows and Parent Close Policies.

This page discusses [Parent Close Policy](#parent-close-policy).

## What is a Parent Close Policy? 

A Parent Close Policy determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed out).

- [How to set a Parent Close Policy using the Go SDK](/develop/go/workflows/child-workflows#parent-close-policy)
- [How to set a Parent Close Policy using the Java SDK](/develop/java/workflows/child-workflows#parent-close-policy)
- [How to set a Parent Close Policy using the PHP SDK](/develop/php/workflows/child-workflows#parent-close-policy)
- [How to set a Parent Close Policy using the Python SDK](/develop/python/workflows/child-workflows#parent-close-policy)
- [How to set a Parent Close Policy using the TypeScript SDK](/develop/typescript/workflows/child-workflows#parent-close-policy)
- [How to set a Parent Close Policy using the .NET SDK](/develop/dotnet/workflows/child-workflows#parent-close-policy)
- [How to set a Parent Close Policy using the Rust SDK](/develop/rust/workflows/child-workflows#parent-close-policy)

There are three possible values:

- **Abandon:** the Child Workflow Execution is not affected.
- **Request Cancel:** a Cancellation request is sent to the Child Workflow Execution.
- **Terminate** (default): the Child Workflow Execution is forcefully Terminated.

[`ParentClosePolicy`](https://github.com/temporalio/api/blob/70cec27214ec2173974fb045667df324b45e705c/temporal/api/enums/v1/workflow.proto#L50) proto definition.

Each Child Workflow Execution may have its own Parent Close Policy.
This policy applies only to Child Workflow Executions and has no effect otherwise.

![Parent Close Policy entity relationship](https://docs.temporal.io/diagrams/parent-close-policy.svg)

You can set policies per child, which means you can opt out of propagating terminates / cancels on a per-child basis.
This is useful for starting Child Workflows asynchronously (see [relevant issue here](https://community.temporal.io/t/best-way-to-create-an-async-child-workflow/114) or the corresponding SDK docs).

---

# Patching

Source: https://docs.temporal.io/patching

> This Encyclopedia page provides an in-depth explanation of what happens during Workflow Patching

This page discusses [Patching](#patching).

## What is Patching? 

A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag.
It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions.
When you want to make substantive code changes that may affect existing Workflow executions, create a patch.

### Detailed description of the `patched()` function

This applies to the `patched()` function in the Python, .NET, and Ruby SDKs.

#### Behavior when not replaying

If the execution is not replaying, when it encounters a call to `patched()`, it first checks the event history.

- If the patch ID is not in the event history, the execution adds a marker to the event history, upserts a search attribute, and returns `true`. This happens in the first block of the patch ID.
- If the patch ID is in the event history, the execution doesn't modify the history, and returns `true`.
  This happens in a patch ID's subsequent blocks, because the event history was updated in the first block.

There is a caveat to this behavior, which we will cover below.

#### Behavior when replaying with marker before-or-at current location

If the execution is replaying and has a call to `patched()`, and if the event history has a marker from a call to `patched()` in the same place (which means it will match the original event history), then it writes a marker to the replay event history and returns `true`.

This is similar to the behavior of the non-replay case, and also happens in a given patch ID's first block.

If the code has a call to `patched()`, and the event history has a marker with that Patch ID earlier in the history, it will return `true` and will not modify the replay event history.

This is also similar to the behavior of the non-replay case, and also happens in a given patch ID's subsequent blocks.

#### Behavior when replaying with marker after current location

If the Event History's Marker Event is after the current execution point, that means the new patch is too early.
The execution will encounter the new patch before the original.
The execution will attempt to write the marker to the replay event history, but it will throw a non-deterministic exception because the replay and original event histories don't match.

#### Behavior when replaying with no marker for that patch ID

During a Replay, if there is no marker for a given patch ID, the execution will return `false` and will not add a marker to the event history. In addition, all future calls to `patched()` with that ID will return `false` -- even after it is done replaying and is running new code.

The [preceding section](#behavior-when-not-replaying) states that if the execution is not replaying, the `patched()` function will always return `true`.
If the marker doesn't exist, it will be added, and if the marker already exists, it won't be re-added.

However, this behavior doesn't occur if there was already a call to `patched()` with that ID in the replay code, but not in the event history.
In this situation, the function won't return `true`.

#### A summary of the two potentially unexpected behaviors

Recapping the potentially unexpected behaviors that may occur during a Replay:

If the execution hits a call to `patched()`, but that patch ID isn't _at or before that point_ in the event history, you may not realize that the event history _after_ the current execution location matters.
This behavior occurs because:

- If that patch ID exists later, you get a non-determinism error
- If the patch doesn't exist later, you don't get a non-determinism error, and the call returns `false`

If the execution hits a call to `patched()` with an ID that doesn't exist in the history, then not only will it return `false` in that occurrence, but it will also return `false` if the execution surpasses the Replay threshold and is running new code.

#### Implications of the behaviors

If you deploy new code while Workflows are executing, any Workflows that were in the middle of executing will Replay up to the point they were at when the Worker was shut down.
When they do this Replay, they will not follow the `patched()` branches in the code.
For the rest of the execution after they have replayed to the point before the deployment and worker restart, they will either:

- Use new code if there was no call to `patched()` in the replay code
- If there was a call to `patched()` in the replay code, they will
  run the non-patched code during and after replay

This might sound odd, but it's actually exactly what's needed because that means that if the future patched code depends on earlier patched code, then it won't use the new code -- it will use the old code instead.

But if there's new code in the future, and there was no code earlier in the body that required the new patch, then it can switch over to the new code, which it will do.

Note that this behavior means that the Workflow _does not always run the newest code_.
It only does that if not replaying or if replay is surpassed and there hasn't been a call to `patched()` (with that ID) throughout the replay.

#### Recommendations

Based on this behavior and the implications, when patching in new code, always put the newest code at the top of an if-patched-block.

```python
if patched('v3'):
    # This is the newest version of the code.
    # put this at the top, so when it is running
    # a fresh execution and not replaying,
    # this patched statement will return true
    # and it will run the new code.
    pass
elif patched('v2'):
    pass
else:
    pass
```

The following sample shows how `patched()` will behave in a conditional block that's arranged differently.
In this case, the code's conditional block doesn't have the newest code at the top.
Because `patched()` will return `True` when not Replaying (except with the preceding caveats), this snippet will run the `v2` branch instead of `v3` in new executions.

```python
if patched('v2'):
    # This is bad because when doing a new execution (i.e. not replaying),
    # patched statements evaluate to True (and put a marker
    # in the event history), which means that new executions
    # will use v2, and miss v3 below
    pass
elif patched('v3'):
    pass
else:
  pass
```

---

# Payload Codec

Source: https://docs.temporal.io/payload-codec

> A Payload Codec performs bytes-to-bytes transformations on Temporal Payloads, often for compression and encryption.

This page discusses [Payload Codec](#payload-codec).

## What is a Payload Codec? 

A Payload Codec transforms an array of [Payloads](/dataconversion#payload) (for example, a list of Workflow arguments) into another array of Payloads. Unlike Payload Converters, Payload Codecs do not operate within the Workflow sandbox.

This allows them to execute operations that can include calls to remote services and the use of non-deterministic modules, which are critical for tasks such as encrypting Payloads, compressing data, or offloading large payloads to an object store. Payload Codecs can also be implemented as a Codec Server.

When serializing to Payloads, the Payload Converter is applied first to convert your objects to bytes, followed by codecs that convert bytes to bytes. When deserializing from Payloads, codecs are applied first to last to reverse the effect, followed by the Payload Converter.

Use a custom Payload Codec to transform your Payloads; for example, implementing compression and/or encryption on your Workflow Execution data.

### Operational chain

In practice, the [Payload Converter](/payload-converter) and Payload Codec operate in a chain to handle data securely. Incoming data first passes through a Payload Converter through the `toPayload` method, turning application objects into Payloads. These Payloads are then processed by the Payload Codec through the `encode` method, which adjusts the Payload according to the required security or efficiency needs before it is sent to the Temporal Cluster.

The process is symmetric for outgoing data. Payloads retrieved from the Temporal Cluster first pass through the Payload Codec through the `decode` method, which reverses any transformations applied during encoding. Finally, the resulting Payload is converted back into an application object by the Payload Converter through the `fromPayload` method, making it ready for use within the application.

### Encryption 

Using end-to-end encryption in your custom Data Converter ensures that sensitive application data is secure when handled by the Temporal Server.

Apply your encryption logic in a custom Payload Codec and use it locally to encrypt data.
You maintain all the encryption keys, and the Temporal Server sees only encrypted data. Refer to [What is Key Management?](/key-management) for more guidance.

Your data exists unencrypted only on the Client and the Worker process that is executing the Workflows and Activities, on hosts that you control. For details, see [Securing your data](/production-deployment/data-encryption).

The following samples use encryption (AES GCM with 256-bit key) in a custom Data Converter:

- [Go sample](https://github.com/temporalio/samples-go/tree/main/encryption)
- [Java sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads)
- [Python sample](https://github.com/temporalio/samples-python/tree/main/encryption)
- [TypeScript sample](https://github.com/temporalio/samples-typescript/tree/main/encryption)

---

# Payload Converter

Source: https://docs.temporal.io/payload-converter

> A Payload Converter serializes and deserializes values to and from bytes for use in the Temporal SDK.

This page discusses [Payload Converter](#payload-converter).

## What is a Payload Converter? 

A Payload Converter serializes data, converting values to bytes and back. This is how you can serialize application objects into a Payload and deserialize them back into application objects.

A Payload, in this context, is a binary form suitable for network transmission that may include some metadata. This serialization process transforms an object (like those in JSON or Protobuf formats) into a binary format and vice versa. For example, an object might be serialized to JSON with UTF-8 byte encoding or to a protobuf binary using a specific set of protobuf message definitions.

When you initiate a Workflow Execution through a Client and pass data as input, the input is serialized using a Data Converter that runs it through a set of Payload Converters. When your Workflow Execution starts, this data input is deserialized and passed as input to your Workflow.

Due to their operation within the Workflow context, Payload Converters run inside the Workflow sandbox. Consequently, Payload Converters cannot access external services or employ non-deterministic modules, which excludes most types of encryption due to their non-deterministic nature.

### Composite Data Converters 

A Composite Data Converter is used to apply custom, type-specific Payload Converters in a specified order.
A Composite Data Converter can be comprised of custom rules that you created, and it can also leverage the default Data Converters built into Temporal.
In fact, the default Data Converter logic is implemented internally in the Temporal source as a Composite Data Converter. It defines these rules in this order:

```go
defaultDataConverter = NewCompositeDataConverter(
    NewNilPayloadConverter(),
    NewByteSlicePayloadConverter(),
    NewProtoJSONPayloadConverter(),
    NewProtoPayloadConverter(),
    NewJSONPayloadConverter(),
)
```

The order in which the Payload Converters are applied is important.
During serialization, the Data Converter tries the Payload Converters in that specific order until a Payload Converter returns a non-nil Payload.
A custom PayloadConverter must implement the functions:

- `FromPayload` (for a single value) or
- `FromPayloads` (for a list of values) to convert to values from a Payload, and
- `ToPayload` (for a single value) or
- `ToPayloads` (for a list of values) to convert values to a Payload.

Defining a new Composite Data Converter is not always necessary to implement custom data handling.
Each SDK allows you to override or configure the default Converter with a custom Payload Codec.

---

# Temporal Platform production deployments

Source: https://docs.temporal.io/production-deployment

To take your application to production, you'll need to deploy the following components:

- Your application code, including your Workflows, Activities, and Workers, on your infrastructure using your existing
  build, test and deploy tools.

- A production-ready Temporal Service to coordinate the execution of your Workflows and Activities.
  - You can use Temporal Cloud, a fully managed platform, or you can self-host the service.

> **💡 Tip:**
> Do you need a production Temporal Service?
>
> If you're still developing and testing your application locally, you may not need a production Temporal Service. Use the
> [Temporal CLI development server](/cli/command-reference/server#start-dev) — a single binary with no external dependencies:
>
> `temporal server start-dev`
>
> This starts a complete Temporal Service with Web UI on your local machine. We recommend this for local development
> regardless of whether you plan to use Temporal Cloud or self-host in production. See the
> [Temporal CLI server](/cli/command-reference/server) page for configuration options.
>

## Use Temporal Cloud

You can let us handle the operations of running the Temporal Service, and focus on your application. Follow the
[Temporal Cloud guide](/cloud) to get started.

![Connect your application instances to Temporal Cloud](https://docs.temporal.io/diagrams/basic-platform-topology-cloud.svg)

## Run a Self-hosted Temporal Service

Alternatively, you can run your own production level Temporal Service to orchestrate your durable applications. Follow
the [Self-hosted guide](/self-hosted-guide) to get started.

![Connect your application instances to your self-hosted Temporal Service](https://docs.temporal.io/diagrams/basic-platform-topology-self-hosted.svg)

## Worker deployments

Whether you're hosting with Temporal Cloud or on your own, you have control over where to run and scale your Workers. We
provide guidance on [Worker Deployments](/production-deployment/worker-deployments).

---

# Codecs and Encryption

Source: https://docs.temporal.io/production-deployment/data-encryption

> Encrypt data in Temporal Server to secure Workflow, Activity, and Worker information. Use custom Payload Codecs for encryption/decryption, set up Codec Servers for remote decoding, and ensure secure access.

The Temporal Service persists data from your Workflow Executions, including inputs, outputs, and results. To protect
sensitive data, use a [Payload Codec](/payload-codec) to encrypt payloads before they reach the Temporal Service. With
encryption enabled, data exists unencrypted only on the Client and the Worker process, on hosts that you control.

Alternatively, use [Temporal Proxy](/production-deployment/temporal-proxy) to encrypt payloads before they reach the
Temporal Service without changing Worker or Client code. You run the proxy in your environment, where it has access to
plaintext payloads and your KMS. See the proxy's
[encryption guide](/production-deployment/temporal-proxy/encrypt-payloads) for configuration and coverage limits.

The following data is persisted in the Event History and can be encrypted:

- Inputs and outputs/results in your [Workflow](/workflow-execution), [Activity](/activity-execution), and [Child Workflow](/child-workflows)
- [Signal](/sending-messages#sending-signals) inputs
- [Memo](/workflow-execution#memo)
- Headers (verify if applicable to your SDK)
- [Query](/sending-messages#sending-queries) inputs and results
- Results of [Local Activities](/local-activity) and [Side Effects](/workflow-execution/event#side-effect)
- [Application errors and failures](/references/failures).
  Failure messages and call stacks are not encoded as codec-capable Payloads by default; you must explicitly enable
  encoding these common attributes on failures. For more details, see [Failure Converter](/failure-converter).

To view encrypted data in the Web UI and CLI, set up a [Codec Server](/codec-server). The following sections cover how
to set up a Codec Server and configure the Web UI and CLI to use it.

For encryption implementation examples, see the following samples:

- [Go](https://github.com/temporalio/samples-go/tree/main/encryption)
- [Java](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads)
- [Python](https://github.com/temporalio/samples-python/tree/main/encryption)
- [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/encryption)
- [.NET](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption)

## Codec Server setup 

Use a Codec Server to programmatically decode your encoded [payloads](/dataconversion#payload).

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
The Codec Server is independent of the Temporal Service and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment.
When you configure a Codec Server endpoint in the Web UI or CLI, the Web UI and CLI use the remote endpoint to send and receive payloads from the Codec Server.
See [API contract requirements](#api-contract-specifications).

Decoded payloads can then be displayed in the Workflow Execution Event History on the Web UI. When you use a Codec Server, the decoded payloads are decoded and returned on the client side only. Payloads on the Temporal Service (whether on Temporal Cloud or self-hosted) remain encrypted.

Because you create, operate, and manage access to your Codec Server in your controlled environment, ensure that you consider the following:

- When you register a Codec Server endpoint with your Web UI, expect the Codec Server to receive multiple requests per Workflow Execution.
- Ensure that you secure access to your Codec Server. For details, see [Authorization](#authorization). You might need some form of [Key management infrastructure](/key-management) for sharing your encryption keys between the Workers and your Codec Server.
- You will need to enable [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) on the HTTP/HTTPS endpoints in your Codec Server to receive requests from the Temporal Web UI.
- You may introduce latency in the Web UI when sending and receiving payloads to the Codec Server.

Your Codec Server should share logic with the custom [Payload Codec](/payload-codec) used elsewhere in your application.

### API contract specifications

When you create your Codec Server to handle requests from the Web UI, the following requirements must be met.

#### Endpoints

The Web UI and CLI send POST requests to the following endpoints on your Codec Server:

- `/decode` passes incoming payloads to the decode method in your Payload Codec.
- `/encode` passes incoming payloads to the encode method in your Payload Codec.
- `/download` retrieves and decodes payloads from [External Storage](/external-storage). This endpoint is only needed if
  your Workers use External Storage. See [Codec Server with External Storage](/codec-server#external-storage) for
  details.

For examples on how to create your Codec Server, see the following Codec Server implementation samples:

- [Go](https://github.com/temporalio/samples-go/tree/main/codec-server)
- [Java](https://github.com/temporalio/sdk-java/tree/main/temporal-remote-data-encoder)
- [Python](https://github.com/temporalio/samples-python/blob/main/encryption/codec_server.py)
- [TypeScript](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/codec-server.ts)
- [.NET](https://github.com/temporalio/samples-dotnet/blob/main/src/Encryption/CodecServer/Program.cs)

You can also add a [verification step](#authorization) to check whether the incoming request has the required authorization to access the decode logic in your Payload Codec.

#### Headers

Each request from the Web UI to your Codec Server includes the following headers:

- `Content-Type: application/json`: Ensure that your Codec Server can accommodate this [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types).

- `X-Namespace: {namespace}`: This is a custom HTTP Header. Ensure that the [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) configuration in your Codec Server includes this header.

- [Optional] `Authorization: <credentials>`: Include this in your CORS configuration when enabling authorization with your Codec Server.

For details on setting up authorization, see [Authorization](#authorization).

#### Request body

The general specification for the `POST` request body contains payloads.
By default, all field values in your payload are base64 encoded, regardless of whether they are encrypted by your custom codec implementation.

The following example shows a sample `POST` request body with base64 encoding.

```json
{
  "payloads": [{
    "metadata": {
      "encoding": <base64EncodedEncodingHint>
    },
    "data": <encryptedPayloadData>
  }, ...]
}
```

#### CORS

By default, in cross-origin Fetch/XHR invocations, browsers will not send credentials.
Enable [Cross-Origin Resource Sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) requests on your Codec Server to receive HTTP/HTTPS requests from the Temporal Web UI.

At a minimum, enable the following responses from your Codec Server to allow requests coming from the Temporal Web UI:

- `Access-Control-Allow-Origin`
- `Access-Control-Allow-Methods`
- `Access-Control-Allow-Headers`

For example, for Temporal Cloud Web UI hosted at https://cloud.temporal.io, enable the following in your Codec Server:

- `Access-Control-Allow-Origin: https://cloud.temporal.io`
- `Access-Control-Allow-Methods: POST, GET, OPTIONS`
- `Access-Control-Allow-Headers: X-Namespace, Content-Type`

For details on what a sample request/response looks like from the Temporal Web UI, see [Sample Request/Response](#sample-requestresponse).
If setting authorization, include `Authorization` in your `Access-Control-Allow-Headers`.
For details on setting up authorization, see [Authorization](#authorization).

#### Authorization

It is important to establish how you will provide access to your Codec Server.
Because it is designed to decode potentially sensitive data with a single API call, access to a production Codec Server should be restricted.

Depending on your infrastructure and risk levels, it might be sufficient to restrict HTTP ingress to your Codec Server (such as by using a VPN like [WireGuard](https://www.wireguard.com/)).
The Temporal Web UI can communicate with a Codec Server that is only accessible on `localhost`, so this is a legitimate security pattern.
However, if your Codec Server is exposed to the internet at all, you will likely need an authentication solution.

If you are already using an organization-wide authentication provider, you should integrate it with your Codec Server. Remember, a Codec Server is just a standalone HTTP server, so you can use existing libraries for OAuth, [Auth0](https://auth0.com/), or any other protocol.
[This repository](https://github.com/pvsone/codec-cors-credentials) contains an example of using Auth0 to handle browser-based auth to a Codec Server.

To enable authorization from the Web UI (for both a self-hosted Temporal Service and Temporal Cloud), your Codec Server must be an HTTPS Server.

**Temporal Cloud**

The Temporal Cloud UI provides an option to pass access tokens (JWT) to your Codec Server endpoints.
Use the access tokens to validate access and then return decoded payloads from the Codec Server.

You can enable this by selecting **Pass access token** in your Codec Server endpoint interface where you add your endpoint.
Enabling this option in the Temporal Cloud UI adds an authorization header to each request sent to the Codec Server endpoint that you set.

In your Codec Server implementation, verify the signature on this access token (in your authorization header) against [our JWKS endpoint](https://login.tmprl.cloud/.well-known/jwks.json).

The token provided from Temporal Cloud UI contains the email identifier of the person requesting access to the payloads.
Based on the permissions you have provided to the user in your access control systems, set conditions in your Codec Server whether to return decoded payloads or just return the original encoded payloads.

**Self-hosted Temporal Service**

On a self-hosted Temporal Service, configure [authorization in the Web UI configuration](/references/web-ui-configuration#auth) in your Temporal Service setup.

With this enabled, you can pass access tokens to your Codec Server and validate the requests from the Web UI to the Codec Server endpoints that you set.
Note that with a self-hosted Temporal Service, you must explicitly configure authorization specifications for the Web UI and CLI.

#### Sample request/response

Consider the following sample request/response when creating and hosting a Codec Server with the following specifications:

- Scheme: `https`
- Host: `dev.mydomain.com/codec`
- Path: `/decode`

```json
HTTP/1.1 POST /decode
Host: https://dev.mydomain.com/codec
Content-Type: application/json
X-Namespace: myapp-dev.acctid123
Authorization: Bearer <token>

{"payloads":[{"metadata":{"encoding":"anNvbi9wcm90b2J1Zg==","messageType":"dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q="},"data":"eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0"}]}

200 OK
Content-Type: application/json

{
  "payloads": [{
    "metadata":{
      "encoding": "json/protobuf",
      "messageType": "temporal_shop.orchestrations.v1.StartShoppingCartRequest"
    },
    "data":{
      "cartId":"example-cart",
      "shopperId":"your-shopper-id-example",
      "email":"your-email@domain.com"
    }}]
}
```

You can also perform remote encoding on an `/encode` endpoint, which looks the same in reverse:

- Scheme: `https`
- Host: `dev.mydomain.com/codec`
- Path: `/encode`

```json
HTTP/1.1 POST /encode
Host: https://dev.mydomain.com/codec
Content-Type: application/json
X-Namespace: myapp-dev.acctid123
Authorization: Bearer <token>

{"payloads":[{"metadata":{"encoding":"json/protobuf","messageType":"temporal_shop.orchestrations.v1.StartShoppingCartRequest"},"data":{"cartId":"example-cart","shopperId":"your-shopper-id-example","email":"your-email@domain.com"}}]}

200 OK
Content-Type: application/json

{
  "payloads": [
    {
      "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q="
      },
      "data": "eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0"
    }
  ]
}
```

### Set your Codec Server endpoints with Web UI and CLI

After you create your Codec Server and expose the requisite endpoints, set the endpoints in your Web UI and CLI.

#### Web UI

On Temporal Cloud and self-hosted Temporal Service, you can configure a Codec Server endpoint to be used for a Namespace in the Web UI.

![Codec Server endpoint Namespace setting](https://docs.temporal.io/img/info/set-codec-endpoint-form.png)

> **⚠️ Caution:**
>
> If your Codec Server is on a private network, your browser may block Temporal Web UI from accessing it. On Chrome, your browser may prompt you to allow access to the Codec Server endpoint. Make sure to allow access to the Codec Server endpoint. Refer to the [Chrome for Developers blog: New permission prompt for Local Network Access](https://developer.chrome.com/blog/local-network-access/) for details on this permission prompt.
>
> If your browser has blocked Temporal's access to your Codec Server, refer to [Chrome documentation](https://support.google.com/chrome/answer/114662) for details on how to change the site settings. In **Site settings**, look for the **Local network** setting and change it to **Allow** to only change this setting for Temporal without affecting other sites.
>

To set a Codec Server endpoint on a Namespace, do the following.

1. In the Web UI, go to Namespaces, select the Namespace where you want to configure the Codec Server endpoint, and click **Edit**.
1. In the **Codec Server** section on the Namespace configuration page, enter your Codec Server endpoint and port number.
1. Optional: If your Codec Server is configured to [authenticate requests](#authorization) from Temporal Web UI, enable **Pass access token** to send a JWT access token with the HTTPS requests.
1. Optional: If your Codec Server is configured to [verify origins of requests](#cors), enable **Include cross-origin credentials**.

On Temporal Cloud, you must have [Namespace Admin privileges](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) to add a Codec Server endpoint on the Namespace. Setting a Codec Server endpoint on a Cloud Namespace enables it for all users on the Namespace.

Setting a Codec Server endpoint on a self-hosted Temporal Service enables it for the entire Temporal Service. You can use a single Codec Server to handle different encoding and decoding routes for each namespace.

You can also override the global Codec Server setting at the browser level. This can be useful when developing, testing, or troubleshooting encoding functionality.

![Codec Server endpoint browser setting](https://docs.temporal.io/img/info/data-encoder-button.png)

To set a browser override for the Namespace-level endpoint, do the following.

1. Navigate to **Workflows** in your Namespace.
2. In the top-right corner, select **Configure Codec Server**.
3. Select whether you want to use the Namespace-level (or Temporal Service-level for self-hosted Temporal Service) or the browser-level Codec Endpoint setting as the default for your browser.
   In Temporal Cloud:
   - **Use Namespace-level settings, where available. Otherwise, use my browser setting.**
     Uses the Namespace-level Codec Server endpoint by default.
     If no endpoint is set on the Namespace, your browser setting is applied.
   - **Use my browser setting and ignore Namespace-level setting.**
     Applies your browser-level setting by default, overriding the Namespace-level Codec Server endpoint.
4. Enter your Codec Server endpoint and port number.
5. Optional: If your Codec Server is configured to [authenticate requests](#authorization) from Temporal Web UI, enable **Pass access token** to send a JWT access token with the HTTPS requests.
6. Optional: If your Codec Server is configured to [verify origins of requests](#cors), enable **Include cross-origin credentials**.

In a self-hosted Temporal Service with dedicated UI Server configuration, you can also set the codec endpoint in the UI server [configuration file](/references/web-ui-configuration#codec):

```yaml
codec:
    endpoint: {{ default .Env.TEMPORAL_CODEC_ENDPOINT "{namespace}"}}
```

#### CLI

You can configure a Codec Server endpoint with the Temporal CLI using the `--codec-endpoint` flag.

For example, if you are running your Codec Server on `http://localhost:8888`, you can use `env set` to set the endpoint globally:

```bash
temporal env set --codec-endpoint "http://localhost:8888"
```

If your Codec Server endpoint is not set globally, provide the `--codec-endpoint` option with each command.
For example, to see the decoded output of the Workflow Execution "yourWorkflow" in the Namespace "yourNamespace", run:

```bash
temporal --codec-endpoint "http://localhost:8888" --namespace "yourNamespace" workflow show --workflow-id "yourWorkflow"  --run-id "<yourRunId>" --output "table"
```

For details, see the [CLI reference](/cli/).

If your Codec Server requires authentication, the Temporal CLI will also accept a `--codec-auth` parameter to supply an
authorization header:

```shell
temporal workflow show \
   --workflow-id converters_workflowID \
   --codec-endpoint 'http://localhost:8081/{namespace}' \
   --codec-auth 'auth-header'
```

### Working with large payloads

If your payloads exceed the Temporal Service's size limits, use [External Storage](/external-storage) to offload large
payloads to an external store like Amazon S3. The Web UI displays a claim reference instead of the payload content for
externally stored payloads. When External Storage is configured on your Codec Server, the Codec Server can retrieve
and decode these payloads for viewing in the Web UI and CLI. See
[Codec Server with External Storage](/codec-server#external-storage) for details.

### Temporal Nexus

The Data Converter works the same for a Nexus Operation as it does for other payloads sent between a Worker and Temporal Cloud.
Both the caller and handler Workers must use compatible Data Converters to pass operation inputs and results between them.

See [Nexus Payload Encryption & Data Converter](/nexus/security#payload-encryption-data-converter) for details.

---

# Self-hosted Temporal Nexus

Source: https://docs.temporal.io/production-deployment/self-hosted-guide/nexus

> Use Nexus in your self-hosted Temporal Service.

> **ℹ️ Info:**
> NEW TO NEXUS?
>
> This page explains how to self-host Nexus. To learn about Nexus, see the [how Nexus works page](/nexus). To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/features/nexus).
>

## Enable Nexus

Nexus can be configured by setting static configuration and dynamic configuration entries.

> **📝 Note:**
> Replace `$PUBLIC_URL` with a URL value that's accessible to external callers or internally within the cluster.
> Currently, external Nexus calls are considered experimental so it should be safe to use the address of an internal load balancer for the Frontend Service.

To enable Nexus in your deployment:

1. Enable the HTTP API in the server's static configuration.

   ```yaml
   services:
     frontend:
       rpc:
         # NOTE: keep other fields as they were
         httpPort: 7243

   clusterMetadata:
     # NOTE: keep other fields as they were
     clusterInformation:
       active:
         # NOTE: keep other fields as they were
         httpAddress: $PUBLIC_URL:7243
   ```

2. Set the required dynamic configuration
    1. **Prior to version 1.30.X**, you must set the public callback URL and the allowed callback addresses.

       **NOTE**: the callback endpoint template and allowed addresses should be set when using the experimental
       "external" endpoint targets.

       ```yaml
       component.nexusoperations.callback.endpoint.template:
         # The URL must be publicly accessible if the callback is meant to be called by external services.
         # When using Nexus for cross namespace calls, the URL's host is irrelevant as the address is resolved using
         # membership. The URL is a Go template that interpolates the `NamepaceName` and `NamespaceID` variables.
         - value: https://$PUBLIC_URL:7243/namespaces/{{.NamespaceName}}/nexus/callback
       component.callbacks.allowedAddresses:
         # Limits which callback URLs are accepted by the server.
         # Wildcard patterns (*) and insecure (HTTP) callbacks are intended for development only.
         # For production, restrict allowed hosts and set AllowInsecure to false
         # whenever HTTPS/TLS is supported. Allowing HTTP increases MITM and data exposure risk.
         - value:
             - Pattern: "*" # Update to restrict allowed callers, for example "*.example.com"
               AllowInsecure: true # In production, set to false and ensure traffic is HTTPS/TLS encrypted
       ```

    2. **Version 1.30.X+**: Nexus is enabled by default. Only the system callback URL is needed.
       ```yaml
       component.nexusoperations.useSystemCallbackURL:
         - value: true
       ```

## Build and use Nexus Services

See [how Nexus works](/nexus) for an architectural overview, then follow an SDK guide to build your first Nexus Service.

> **💡 Tip:**
> SDK GUIDES
>
> - [Go](/develop/go/nexus/feature-guide) |
>   [Java](/develop/java/nexus) |
>   [Python](/develop/python/nexus) |
>   [TypeScript](/develop/typescript/nexus) |
>   [.NET](/develop/dotnet/nexus)
>

## Multi-Cluster Replication (multi-region failover)

> **⚠️ Warning:**
> EXPERIMENTAL
>
> Nexus across [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication)
> is experimental.
>

Nexus works across
[Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication). An
asynchronous Nexus Operation started in one Cluster completes even if its Namespace
fails over, or its Cluster is lost, before the Operation finishes.

> **📝 Note:**
> Endpoint target types
>
> This applies to [Worker-target Endpoints](/nexus/endpoints), where the Endpoint
> routes to a target Namespace and Task Queue that a Worker polls. Endpoints can also
> target an external URL (`--target-url`), which is experimental and not covered here.
>

### Configuration

1. **Set up Multi-Cluster Replication.** See
   [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) for
   connecting Clusters and creating replicated Namespaces.

2. **Advertise a frontend HTTP address on every Cluster.** Set
   `frontend.rpc.httpPort` and `clusterInformation.<cluster>.httpAddress` (shown in
   [Enable Nexus](#enable-nexus) for one Cluster) on **each** Cluster, each with its
   own address, so the other Clusters can reach it to deliver callbacks.

3. **Register Endpoints on every Cluster.** The
   [Nexus Endpoint registry](/nexus/registry) isn't replicated across Clusters.
   Create the same Endpoints (same target Namespace and Task Queue) on **each** Cluster.

4. **Cross-Cluster forwarding.** Nexus request and callback forwarding to the
   active Cluster is on by default; no additional configuration is required.

### What to expect on failover

An asynchronous Nexus Operation completes even when the Handler Namespace fails
over before its completion callback is delivered. Two properties make this work:

- The pending completion callback is **durable state on the Handler Workflow**, so
  it fails over together with the Handler Namespace and is re-driven from the new
  active Cluster.
- Each Namespace resolves its active Cluster **independently**. The callback is
  routed to the **Caller Namespace's** current active Cluster (re-resolved on each
  attempt), which is unaffected by the Handler Namespace's failover.

The sequence below shows the Handler Namespace failing over from Cluster A to
Cluster B while the Caller Namespace stays active on Cluster A. The Handler
Workflow finishes on Cluster B, and its completion callback is still delivered back
to the Caller on Cluster A:

```mermaid
sequenceDiagram
    box Cluster A
    participant caA as Caller Namespace
    participant haA as Handler Namespace
    end
    box Cluster B
    participant haB as Handler Namespace
    end
    Note over caA,haA: Caller and Handler Namespaces<br/>active on Cluster A
    caA->>haA: start Nexus Operation
    haA-->>caA: Operation started, token returned
    Note over haA: Handler Workflow running, completion callback pending
    Note over haA,haB: 💥 Handler Namespace fails over to Cluster B<br/>(Caller stays on Cluster A)
    Note over haB: Handler Workflow resumes on B, pending callback re-driven from B
    haB->>caA: deliver completion callback to Caller's active Cluster (A)
    Note over caA: Operation completes on the Caller
```

---

# Temporal Proxy

Source: https://docs.temporal.io/production-deployment/temporal-proxy

> **Public Preview**
> Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between
> releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and
> the definitive configuration schema.

The Temporal Proxy is a gRPC proxy that sits between your Temporal SDK Clients, Workers, and the Temporal Web UI on one
side and one or more upstream Temporal Services on the other. It handles Namespace translation, TLS termination, and
optional payload encryption so your applications can target a single local endpoint while the proxy routes each request
to the right upstream, whether that is a local development Service, a self-hosted Service, or Temporal Cloud.

Temporal Proxy's encryption code and security model have been vetted by security teams at global banks and independent
penetration testing firms. See [Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads) for the
encryption design, configuration, and coverage limits.

## Why use it

Without the proxy, connection details leak into your application code. Every Worker and Client has to know the
upstream's host, TLS material, credentials, and the exact Namespace name the upstream expects. That couples your code to
an environment: moving between a local Service, a self-hosted deployment, and Temporal Cloud becomes a code change.

The proxy owns that concern instead. Workers talk plaintext to a single local endpoint using a short Namespace name, and
the proxy adds TLS, credentials, and Namespace translation on the way out. Point a Worker at a different Namespace and
it reaches a different upstream, with no change to the Worker.

## How it works

The proxy is built from a gateway and one proxy per upstream, running in a single process:

- The **gateway** is the single inbound endpoint that every Worker, SDK Client, and the Web UI connects to.
- Each **upstream** has its own proxy that handles communication with that destination.

```mermaid
flowchart LR
    Worker[Worker]
    Client[SDK Client]
    UI[Web UI]

    subgraph Proxy[Temporal Proxy]
        direction LR
        Gateway["Gateway<br/>routes by Namespace"]
        ProxyA["Per-upstream proxy A<br/>Namespace translation<br/>payload encryption (optional)"]
        ProxyB["Per-upstream proxy B<br/>Namespace translation<br/>payload encryption (optional)"]
        Gateway -->|in process| ProxyA
        Gateway -->|in process| ProxyB
    end

    Cloud[Temporal Cloud]
    SelfHosted[Self-hosted Temporal Service]

    Worker --> Gateway
    Client --> Gateway
    UI --> Gateway
    ProxyA --> Cloud
    ProxyB --> SelfHosted
```

For each request, the gateway:

1. refuses the call if its gRPC service is not one the configuration allows, before any upstream work. By default that
   allows `WorkflowService` and `OperatorService` and nothing else.
2. reads the target Namespace from the request.
3. picks an upstream: the first matching routing rule, otherwise the system upstream for Namespace-less calls, otherwise
   the default.
4. hands the request to that upstream's proxy, in the same process.

The per-upstream proxy then rewrites the local Namespace to the name the upstream expects, attaches that upstream's TLS
and credentials, forwards to the Temporal Service, and translates the Namespace back on responses. When payload
encryption is enabled, it also seals codec-capable Payloads on the way out and opens them on the way back. New Payloads
sent through the proxy reach the upstream as ciphertext. See
[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads) for the scope and limits of this fail-closed
behavior.

The proxy reads only the Namespace and the fields it translates, so it needs no knowledge of your Data Converter or
codec, and payload data passes through untouched unless encryption is configured. The gateway accepts gzip-compressed
requests, so a Client with gRPC compression turned on works through it.

> **📝 Note:**
> Version note
>
> Before v0.9.0, the gateway and each per-upstream proxy were connected by unix sockets. v0.9.0 serves every upstream in
> the gateway's process. Nothing in the configuration changes, but per-upstream sockets no longer exist to dial directly.
>

### Terms

| Term             | Meaning                                                                                                                                                                                         |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| gateway          | The single inbound gRPC endpoint that every SDK Client, Worker, and the Web UI connects to. It routes each request to an upstream by Namespace and request metadata.                           |
| upstream         | A configured destination the proxy forwards to: a Temporal Service (local dev, self-hosted, or Temporal Cloud), or another Temporal Proxy.                                                      |
| system upstream  | The upstream that handles Namespace-less requests, such as the SDK's `GetSystemInfo` call on connect.                                                                                           |
| extension server | A gRPC service you run that the proxy calls out to for a capability it has no built-in backend for: wrapping data encryption keys, or deciding whether an inbound call may proceed.             |
| Temporal Service | A Temporal frontend the proxy connects to.                                                                                                                                                      |

## Prerequisites

- One or more upstream Temporal Services to route to, such as a local development Service, a self-hosted Service, or
  Temporal Cloud.
- The `hostPort` address for each upstream.
- Any credentials the upstreams require, such as a Temporal Cloud API key or mTLS certificates.
- Go installed, if you build the proxy from source. The container image and Helm chart do not require a local Go
  toolchain.

## Install the proxy

**Recommended starting size:** For a Temporal Proxy instance that applies payload encryption for all payload sizes,
use **4 CPUs and 12 GiB of memory**, with `GOMEMLIMIT=11GiB`, and plan for each instance to handle up to
**3,000 requests per second and 100 SDK Workers**. Scale beyond these limits by adding more Proxy replicas (instances)
behind a gRPC-compatible load balancer. For more efficient sizing based on the features and payload sizes you use, see
[Sizing and performance](/production-deployment/temporal-proxy/sizing-and-performance).

Install the `proxy` binary with Go:

```bash
go install go.temporal.io/proxy/cmd/proxy@latest
```

Pin an explicit version instead of `@latest`, using a tag from the
[releases page](https://github.com/temporalio/temporal-proxy/releases):

```bash
go install go.temporal.io/proxy/cmd/proxy@vX.Y.Z
```

The `go.temporal.io/proxy` module path starts with v0.10.0. Install an earlier release from
`github.com/temporalio/temporal-proxy/cmd/proxy`.

Pull the container image:

```bash
docker pull temporalio/temporal-proxy:latest
```

Install with Helm from the Temporal Helm repo, optionally pinning a chart version with `--version`:

```bash
helm install temporal-proxy temporal-proxy \
  --repo https://go.temporal.io/helm-charts
```

Each chart release deploys a proxy version by default. Override it with `--set image.tag=vX.Y.Z`. Supply the proxy
configuration under the `config` key of a Helm values file, as described in
[Deploy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes).

Run the proxy with a configuration file passed through the `-c` (or `--config`) flag:

```bash
proxy serve -c config.yaml
```

`--config` also reads the `PROXY_CONFIG` environment variable, which is how the Helm chart points the proxy at its
mounted configuration. See [Observability](#observability) for the log level flag and the metrics settings.

> **⚠️ Caution:**
> Upgrade to v0.9.0
>
> Two v0.9.0 changes stop an older configuration from starting:
>
> - A configuration with encryption keys must set `encryption.namespaceBinding`. Set `off` before you upgrade a
>   deployment that already holds encrypted payloads, then follow
>   [Bind payloads to their Namespace](/production-deployment/temporal-proxy/encrypt-payloads#namespace-binding) to move
>   to `strict`.
> - The v0.8.0 top-level `codecServer` block moved to `http.codecServer`. See
>   [Run a codec server](/production-deployment/temporal-proxy/codec-server).
>

## Observability 

The proxy serves Prometheus metrics at `/metrics` on `:9090` by default and logs JSON to stderr. The gateway also serves
the standard gRPC health service, with an entry for every allowed service, which is what a Client health check or a
liveness probe reads. See [Report health](/production-deployment/temporal-proxy/configure#health-checks) for what it
reports.

`proxy serve` takes two flags, each with an environment variable equivalent:

| Flag             | Environment variable | Default | Sets                                         |
| ---------------- | -------------------- | ------- | -------------------------------------------- |
| `--config`, `-c` | `PROXY_CONFIG`       | none    | Path to the configuration file. Required.    |
| `--level`        | `LOG_LEVEL`          | `info`  | Log level: `debug`, `info`, `warn`, `error`. |

Everything else about metrics comes from the `metrics` block in the configuration file:

```yaml
metrics:
  hostPort: :9090 # Address serving /metrics.
  namespace: tmprl_proxy # Prometheus namespace prefixed onto every metric.
  labels:
    namespace: false # Report the Temporal Namespace on metrics that can name one.
    fixed: # Constant labels stamped on every series.
      region: us-east-1
    metadata: # Inbound headers reported as labels, written <header>:<name>.
      - x-tenant:tenant
```

> **📝 Note:**
> Version note
>
> `metrics.hostPort` and `metrics.namespace` replace the `--metrics-addr` and `--metrics-namespace` flags, removed in
> v0.6.0 along with their `METRICS_ADDR` and `METRICS_NAMESPACE` environment variables. The defaults are the same, so a
> deployment that never set them needs no change.
>

- `labels.namespace` decides whether the metrics that can name a Temporal Namespace report it. It is `false` by default. The value comes from the request, so every Namespace a Client names adds another series. The label is declared either
  way and only its value is dropped, so a query that already ignores it reads the same before and after you turn enable it.
  The name reported is the local, pre-translation one, the same key you write under `encryption.overrides`.
- `labels.fixed` maps a label name to a constant value stamped on every series the proxy publishes, such as the region
  or cluster a deployment runs in.
- `labels.metadata` reports an inbound request header as a label on the request-scoped series, written
  `<header>:<name>`. Values are truncated at 256 bytes. An encryption operation that came from the
  [codec server](/production-deployment/temporal-proxy/codec-server) reports these labels blank, because an HTTP request
  carries no gRPC metadata.

> **⚠️ Caution:**
>
> Do not list a header that carries a credential, such as an API key or bearer token, under `labels.metadata`. The
> `/metrics` endpoint doesn't require authentication, so anyone who can reach the port can read the header's value.
>
> Also avoid headers whose values the caller can set freely. Each distinct value creates a new copy of every
> request-scoped series, so an unbounded header causes unbounded metric cardinality.
>

Metric names are `<namespace>_<subsystem>_<name>`, so with the default namespace the routing counter is
`tmprl_proxy_router_decisions_total`. There are four subsystems:

| Subsystem      | Metric                             | Labels                               | Reports                                                  |
| -------------- | ---------------------------------- | ------------------------------------ | -------------------------------------------------------- |
| `server`       | `requests_total`                   | `method`, `code`                     | RPCs served, by gRPC status code                         |
| `server`       | `request_duration_seconds`         | `method`                             | End-to-end time serving an RPC                           |
| `server`       | `panics_total`                     | `method`                             | Panics the server recovered from while serving an RPC    |
| `router`       | `decisions_total`                  | `upstream`, `outcome`                | Routing decisions, by chosen upstream                    |
| `router`       | `forwarding_errors_total`          | `upstream`, `reason`                 | Forwarding failures the router originated                |
| `encryption`   | `vault_ops_total`                  | `operation`, `result`, `namespace`   | Envelope operations, sealing and opening payloads        |
| `encryption`   | `vault_ops_duration_seconds`       | `operation`, `namespace`             | Time per envelope operation, end to end                  |
| `encryption`   | `payloads_skipped_total`           | `operation`, `encoding`, `namespace` | Payloads passed through under `skipEncodings`            |
| `encryption`   | `vault_unbound_opens_total`        | `namespace`                          | Payloads with no Namespace binding handed to decrypt     |
| `encryption`   | `vault_namespace_mismatches_total` | `namespace`                          | Payloads bound to another Namespace handed to decrypt    |
| `encryption`   | `dek_ops_total`                    | `operation`, `result`                | Outcome of the AES-256-GCM step alone                    |
| `encryption`   | `dek_ops_duration_seconds`         | `operation`                          | Time in the AES-256-GCM step alone                       |
| `encryption`   | `kek_ops_total`                    | `provider`, `operation`, `result`    | DEK wrap and unwrap calls to your KMS                    |
| `encryption`   | `kek_ops_duration_seconds`         | `provider`, `operation`              | Time spent wrapping and unwrapping DEKs                  |
| `encryption`   | `dek_rotations_total`              | `reason`                             | DEK rotations, by why the DEK was replaced               |
| `encryption`   | `dek_cache_hits_total`             | none                                 | Reads served from the decrypted-DEK cache                |
| `encryption`   | `dek_cache_misses_total`           | none                                 | Reads that required a KMS unwrap                         |
| `encryption`   | `dek_cache_size`                   | none                                 | Current entries in the decrypted-DEK cache               |
| `codec_server` | `requests_total`                   | `route`, `code`                      | Codec server requests, by HTTP status code               |
| `codec_server` | `request_duration_seconds`         | `route`                              | Time serving a codec server request                      |

> **📝 Note:**
> Version note
>
> v0.6.0 renamed the three `*_duration_secs` histograms to `*_duration_seconds` to follow the Prometheus convention.
> Update any dashboard or alert that names the old spelling.
>

The `namespace` label reports a value only when `metrics.labels.namespace` is on. It is declared either way, so the
series exist regardless.

The `encryption` metrics only move when [payload encryption](/production-deployment/temporal-proxy/encrypt-payloads) is
configured. They are layered, so pick the one that matches the question you are asking:

- `vault_ops_*` is the whole envelope operation end to end, including any KEK call and cache lookup, and is the pair to
  alert on.
- `dek_ops_*` is the symmetric AES-256-GCM step by itself, with the KEK work excluded. Its `result` is that step's own
  outcome, so a payload that encrypts cleanly and then fails to wrap its DEK counts as a success here and an error under
  `kek_ops_total`, which keeps the blame with the KMS.
- `kek_ops_*` is the calls to your KMS. Watch `kek_ops_total{result="error"}`, since a failure to wrap or unwrap a DEK
  fails the request that needed it.

Besides `success` and `error`, `vault_ops_total` reports three refusals by `result`:

- `unknown_key`: the payload was sealed under a KMS key this proxy doesn't hold.
- `unbound`: `strict` [Namespace binding](/production-deployment/temporal-proxy/encrypt-payloads#namespace-binding)
  refused a payload that carries no binding.
- `mismatch`: `strict` binding refused a payload bound to another Namespace.

The `codec_server` metrics only move when the [codec server](/production-deployment/temporal-proxy/codec-server) is
enabled. Their `route` label is the matched pattern, such as `/decode`, never the request path, and the subsystem has
no Namespace label, so a Namespace in the path cannot grow the series count.

`dek_rotations_total` splits by `reason`: `initial` for a Namespace's first DEK, `scheduled` for the `renewBefore`
pre-rotation, and `on_demand` for a DEK replaced at request time because no fresh one was ready. A rising `on_demand`
rate means rotation is falling behind, so raise `renewBefore`. Compare the cache counters against `cacheSize` to see
whether the cache is absorbing read traffic.

## Versions and support

The following support policy applies to Temporal Proxy v1.0 and later. Support for Temporal customers includes
troubleshooting through Temporal Support and maintenance releases with critical fixes for security, data loss, and
reliability issues.

Temporal supports the current minor release and at least the three preceding minor releases, when available. Versions
earlier than v1.0 are excluded from this support window. For example, when v1.4 is current, the supported minor releases
include v1.1, v1.2, v1.3, and v1.4; when v1.1 is current, they include v1.0 and v1.1.

Each major version receives support for at least 12 months after its general availability release. Temporal provides at
least six months' notice before ending support for a major version.

Temporal Support may require you to upgrade to the latest bugfix (patch) release within your supported minor version
before proceeding with support if that patch introduces critical functionality related to your reported issue.

Temporal publishes CVE advisories when a fix is available. Report vulnerabilities privately to
[security@temporal.io](mailto:security@temporal.io), following the
[Temporal Proxy security policy](https://github.com/temporalio/temporal-proxy/blob/main/SECURITY.md).

## Related

- [Sizing and performance](/production-deployment/temporal-proxy/sizing-and-performance)
- [Run a codec server](/production-deployment/temporal-proxy/codec-server)
- [Temporal Proxy repository](https://github.com/temporalio/temporal-proxy)
- [Temporal Proxy Helm chart](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy)
- [Temporal Cloud example](https://github.com/temporalio/temporal-proxy/tree/main/examples/cloud)
- [KMS extension server example](https://github.com/temporalio/temporal-proxy/tree/main/examples/kms)
- [Codecs and Encryption](/production-deployment/data-encryption)
- [Self-hosted guide: Security](/self-hosted-guide/security)

---

# Run a codec server with Temporal Proxy

Source: https://docs.temporal.io/production-deployment/temporal-proxy/codec-server

> **Public Preview**
> Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between
> releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and
> the definitive configuration schema.

With [payload encryption](/production-deployment/temporal-proxy/encrypt-payloads) on, anything that reads Event History
without passing through the gateway sees only ciphertext. That includes the Temporal Web UI and `temporal workflow show`
pointed at the Temporal Service directly. The proxy can serve the [codec server](/production-deployment/data-encryption)
HTTP contract with the same keys the gateway uses, so those clients can decode and encode payloads themselves.

The codec server answers `POST /encode`, `POST /decode`, and `POST /download`. Each route is also served under a
Namespace path segment, such as `/{namespace}/decode`, for the CLI's `--codec-endpoint` templating. An unknown path
answers `404`, and a known path called with any method but `POST` answers `405`.

## Configure the HTTP listener

The codec server runs on the proxy's HTTP listener, on its own port separate from the gateway. Configure the listener
under the top-level `http` block and the codec server under `http.codecServer`. Nothing binds unless a route group,
the codec server or the [profiling routes](#profile-the-proxy), is enabled.

This configuration, from the proxy's
[codec server example](https://github.com/temporalio/temporal-proxy/tree/main/examples/codecserver), serves the codec
server on loopback for a local development Service whose Web UI runs on port 8233:

```yaml
http:
  hostPort: 127.0.0.1:8081
  insecure: true # Plaintext, accepted only on a loopback bind.
  codecServer:
    enabled: true
    cors:
      origins: # The Web UI's origin is whichever host was typed, so list both.
        - http://localhost:8233
        - http://127.0.0.1:8233
```

| Key                                 | Default | Meaning                                                                                       |
| ----------------------------------- | ------- | --------------------------------------------------------------------------------------------- |
| `http.hostPort`                     | none    | Address the HTTP listener binds. Required when a route group is enabled.                      |
| `http.insecure`                     | `false` | Serve plaintext. Can't be set together with `http.tls`.                                       |
| `http.tls`                          | none    | `cert` and `key` to terminate TLS with. Add `ca` to require client certificates.              |
| `http.codecServer.enabled`          | `false` | Turns the codec server on.                                                                    |
| `http.codecServer.cors.origins`     | none    | Origins a browser-based caller may reach the codec server from. `*` is rejected.              |
| `http.codecServer.cors.credentials` | `false` | Lets a browser send cookies and `Authorization` from an allowed origin. Requires `origins`.   |
| `http.codecServer.auth`             | none    | Caller authentication, with the same keys as the gateway's [`auth`](/production-deployment/temporal-proxy/configure#authenticate-and-authorize-inbound-requests) block. |

> **📝 Note:**
> Version note
>
> v0.8.0 configured the codec server in a top-level `codecServer` block. v0.9.0 moved it to `http.codecServer`, with
> `hostPort`, `insecure`, and `tls` set on `http` itself. A configuration that still has the top-level block fails at
> startup rather than running with the codec server off.
>

## Secure the codec server

A codec server holds the same KMS unwrap permission the proxy does, so anyone who can reach it can decrypt anything your
keys protect. The proxy enforces that:

- When `http.hostPort` binds anything other than loopback (`127.0.0.1`, `::1`, or `localhost`), every enabled route
  group requires `auth`, and the listener requires `tls`.
- A loopback bind runs without authentication by default, for local development where the caller is on the same host.
  The proxy logs a warning when it does.

```yaml
http:
  hostPort: 0.0.0.0:8081
  tls:
    cert: /etc/temporal-proxy/certs/http/tls.crt
    key: /etc/temporal-proxy/certs/http/tls.key
  codecServer:
    enabled: true
    cors:
      origins:
        - https://cloud.temporal.io
    auth:
      jwks:
        url: https://issuer.example.com/.well-known/jwks.json
        audiences:
          - temporal-codec
```

`auth` verifies who is calling, not which Namespace they may decrypt. A token that passes it authorizes every
Namespace's payloads. If a per-Namespace decision matters, scope a JWKS token narrowly at the issuer, or point
`auth.external` at an
[extension server](/production-deployment/temporal-proxy/configure#delegate-the-decision-to-an-extension-server), which
receives the Namespace the request names.

On Kubernetes, the Helm chart declares only the gateway port. To reach the codec server from outside the Pod, add a
Service for the `http.hostPort` port, for example through the chart's `extraObjects`.

## Name the Namespace 

The codec server compares every payload it opens with the Namespace the request names, using the same
[Namespace binding](/production-deployment/temporal-proxy/encrypt-payloads#namespace-binding) rules as the gateway.
Under `strict`, it requires a Namespace on every request.

Name the Namespace as the Temporal Service knows it, such as `orders.acct` on Temporal Cloud, not the local name a
Worker uses through the gateway. Payloads are labeled with that name, and the codec server maps it to the local name
itself to pick keys. The codec server ignores an upstream's `bindingLabelsFrom`, so during a
[migration between upstreams](/production-deployment/temporal-proxy/encrypt-payloads#move-namespaces-between-upstreams)
a client that names the on-premises Namespace can't open payloads labeled with the Cloud name under `strict`.

The Temporal CLI sends the Namespace for you, so either endpoint form works:

```bash
# The CLI sends an X-Namespace header from --namespace or your profile.
temporal workflow show --namespace orders.acct --workflow-id wf-1 \
  --codec-endpoint https://codec.example.com

# Or name it in the path; the CLI expands {namespace}.
temporal workflow show --namespace orders.acct --workflow-id wf-1 \
  --codec-endpoint 'https://codec.example.com/{namespace}'
```

Add `--codec-auth "Bearer $TOKEN"` when the codec server requires a token. A client that sends neither the header nor
the path segment can add the header itself, for example with `--codec-header X-Namespace=orders.acct`. The proxy's
documentation records this behavior against Temporal CLI 1.9.1.

Under `strict`, a request that names no Namespace answers `400`, and `/decode` refuses a payload labeled for a
different Namespace with the same `400`. That includes the results of Nexus Operations handled in another Namespace,
which the Web UI therefore shows as encrypted. Under `permissive`, those payloads decode and count toward
`vault_namespace_mismatches_total`.

## Use it from the Web UI

Set the codec server endpoint in the Web UI's codec server settings, or start a local development Service with
`temporal server start-dev --ui-codec-endpoint <address>`. The browser fetches encrypted Event History from the Temporal
Service and posts each payload to `/decode`, so the Web UI's origin must be listed under
`http.codecServer.cors.origins`.

When `/decode` answers with a `4xx`, the Web UI shows no error. It renders the payload it already had, so a rejected
token, a missing CORS origin, or a malformed request looks the same as ciphertext the UI hasn't decoded yet: a payload
with the `binary/encrypted` encoding. If payloads stay encrypted, check the browser console. A missing origin is the
usual cause: the codec server answers the preflight without an `Access-Control-Allow-Origin` header, so the browser
never sends the request.

## Profile the proxy

The HTTP listener can also serve the Go runtime's standard pprof endpoints under `/debug/pprof/`. They are off by
default:

```yaml
http:
  hostPort: 127.0.0.1:8080
  insecure: true
  pprof:
    enabled: true
```

| Key                  | Default | Meaning                                                                                       |
| -------------------- | ------- | --------------------------------------------------------------------------------------------- |
| `http.pprof.enabled` | `false` | Turns the pprof routes on.                                                                    |
| `http.pprof.auth`    | none    | Caller authentication, with the same keys as `http.codecServer.auth`. Required off loopback. |

Profiles name hot code paths in the proxy, so the same rule applies as for the codec server: off loopback, the routes
require `auth` and the listener requires `tls`.

Pull a profile with `curl` and read it with `go tool pprof`:

```bash
curl -s -H "Authorization: Bearer $TOKEN" -o heap.pb.gz http://127.0.0.1:8080/debug/pprof/heap
go tool pprof -http=: heap.pb.gz
```

`go tool pprof` can also read from a running proxy directly:
`go tool pprof -http=: http://127.0.0.1:8080/debug/pprof/heap`. To profile a remote proxy whose routes are on loopback
without `auth`, open a tunnel first, such as `ssh -L 8080:localhost:8080 <host>`, then run the same command against
`localhost`.

## Monitor the codec server

The codec server reports `codec_server_requests_total` and `codec_server_request_duration_seconds`, by matched route.
Payloads it decodes and encodes also count toward the `encryption` metrics. See
[Observability](/production-deployment/temporal-proxy#observability).

---

# Configure the Temporal Proxy

Source: https://docs.temporal.io/production-deployment/temporal-proxy/configure

> Configure Temporal Proxy routing, upstreams, TLS, authentication, and payload encryption.

> **Public Preview**
> Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between
> releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and
> the definitive configuration schema.

The proxy reads a single YAML file. Three sections are the core of it: the gateway listener (`hostPort`), `routing`, and
the `upstreams` it forwards to.

Everything else is optional.

- `allowedServices` narrows which gRPC services the proxy forwards
- `tls` and `auth` secure and admit inbound connections
- `encryption` seals payloads
- `extensionServers` declares the servers `auth` and `encryption` delegate to
- `health` tunes the liveness check
- `metrics` configures the Prometheus endpoint
- `http` serves the [codec server and profiling routes](/production-deployment/temporal-proxy/codec-server) on a
  separate port
- `apiTranslations` overrides how the proxy reaches Temporal Cloud's control plane

Values support `${VAR}` and `$VAR` environment variable expansion, and an upstream's `hostPort` can be a template that
resolves per request (for example, `{{ .RemoteNamespace }}`).

The example below is the proxy's
[Temporal Cloud example](https://github.com/temporalio/temporal-proxy/tree/main/examples/cloud), which connects a Worker
to Temporal Cloud with an API key. The Worker carries no Cloud configuration: it talks plaintext to `127.0.0.1:7233`,
and the proxy adds TLS, the API key, and the Namespace rewrite on the way out.

```yaml
# Gateway: the local endpoint your Workers and Clients connect to (plaintext).
hostPort: 127.0.0.1:7233

routing:
  default: cloud # Namespaced requests.
  system: system # Namespace-less requests (for example GetSystemInfo on connect).

upstreams:
  # Namespaced traffic. The host is derived per request from the translated
  # Namespace, so one entry serves any number of Namespaces.
  - name: cloud
    hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233'
    namespaces:
      rules:
        suffix: .$TEMPORAL_ACCOUNT # quickstart becomes quickstart.<account>
    credentials:
      static:
        apiKey: $TEMPORAL_API_KEY

  # Namespace-less calls have no Namespace to derive a host from, so they use a
  # fixed endpoint. Any Namespace endpoint in the account answers them.
  - name: system
    hostPort: ${TEMPORAL_NAMESPACE}.${TEMPORAL_ACCOUNT}.tmprl.cloud:7233
    namespaces:
      rules:
        # Same rule as cloud, so replies that name Namespaces, such as
        # ListNamespaces, come back with local names.
        suffix: .$TEMPORAL_ACCOUNT
    credentials:
      static:
        apiKey: ${TEMPORAL_API_KEY}
```

Neither upstream declares a `tls` block, because the proxy dials over TLS by default. See
[Configure TLS](#configure-tls).

The [chart `values.yaml`](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) and the
[temporal-proxy repository](https://github.com/temporalio/temporal-proxy) hold the complete, current set of options.

## Restrict forwarded services

The top-level `allowedServices` list names the gRPC services the proxy forwards, by proto full name. Omit it and the
proxy forwards `WorkflowService` and `OperatorService`, which is what an SDK Client, a Worker, the CLI, and the Web UI
need.

```yaml
allowedServices:
  - temporal.api.workflowservice.v1.WorkflowService
  - temporal.api.operatorservice.v1.OperatorService
  - grpc.reflection.v1.ServerReflection
```

| Service                                           | Backs                                                                           | In the default set |
| ------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------ |
| `temporal.api.workflowservice.v1.WorkflowService` | Every SDK Client, Worker, and the Web UI                                        | Yes                |
| `temporal.api.operatorservice.v1.OperatorService` | `temporal operator` commands, and the Web UI's Search Attribute and Nexus views | Yes                |
| `grpc.reflection.v1.ServerReflection`             | Service discovery for tools such as `grpcurl`                                   | No                 |

Those three are the whole set to select from. A name the proxy cannot forward fails the configuration at startup rather
than at the first request.

A service you leave out is never forwarded. The gateway answers a call to one with `Unimplemented` and
`unknown service "<name>"`, decided before any upstream work, so the proxy behaves as a server that does not implement
it rather than passing the call through.

Service discovery is opt-in, so name the reflection service only when you want tools such as `grpcurl` to probe the
gateway. Allowing `grpc.reflection.v1.ServerReflection` also allows the superseded
`grpc.reflection.v1alpha.ServerReflection`, because clients probe `v1` and fall back to it. The relation runs one way:
allowing only the superseded spelling does not allow the current one.

## Health checks 

The gateway serves the standard gRPC health service and publishes an entry for every allowed service under its proto
full name, alongside the usual empty service name. Each entry reports the same status, since the proxy's health is
process-wide. A Client health check that names a service, such as the Go SDK's `CheckHealth`, therefore gets a status,
and so does a per-service liveness probe. A service your configuration does not allow returns `NOT_FOUND` instead of
claiming a status.

Behind that status is a liveness check the proxy runs against itself. On each interval it calls
`grpc.health.v1.Health/Watch` over an in-process connection, reads one message, and reports whether the exchange was
answered at all. `Watch` is a streaming method, so the call travels the same interceptor chain every forwarded request
goes through, and a stall in that chain shows up in the reported status instead of going unnoticed. The call never
leaves the process, so the check behaves the same whether the gateway serves plaintext, TLS, or mutual TLS.

What the check does not cover is anything below that chain: accepting on the real listener, and the TLS handshake above
it. Those belong to whatever probes the proxy from outside. See
[Probe the gateway](/production-deployment/temporal-proxy/deploy-kubernetes#probe-the-gateway) for the Kubernetes side.

Tune the cadence under the top-level `health` block. Both fields are optional:

```yaml
health:
  interval: 30s # How often the liveness check runs. Defaults to 30s.
  timeout: 5s # Bounds one run. Defaults to 5s.
```

`timeout` must be positive and strictly shorter than `interval`, so a check always finishes before the next one begins.
A configuration that breaks that rule fails at startup.

## Route requests

The `routing` section selects an upstream for each request:

- `default` is the fallback when no rule matches. It is optional; omit it to reject unmatched requests with an error.
- `system` is the upstream for Namespace-less requests, such as the SDK's `GetSystemInfo` and `GetClusterInfo` calls. It
  is optional; when unset, those requests fall back to `default`.
- `rules` is an ordered list, evaluated top to bottom. The first match wins.

Every upstream named by `default`, `system`, or a rule must exist in `upstreams`. Upstream `name` and `hostPort` values
must each be unique across the list: two upstreams sharing a name make a routing reference ambiguous, and two sharing an
address is a copy-paste error rather than a useful configuration.

```yaml
routing:
  default: local # Fallback when no rule matches.
  system: cloud # Namespace-less requests.
  rules:
    - match:
        namespace: 'prod-*'
        metadata:
          x-tier: gold
      upstream: cloud
    - match:
        namespace: '*-test'
      upstream: local
```

A rule matches when its Namespace matches and every metadata condition matches (AND logic). A `match` must set at least
one of `namespace` or `metadata`; an empty match is a configuration error, since that is what `default` is for. Routing
runs on the local Namespace, before translation.

`namespace` is a string literal or a simple glob with a single leading or trailing `*`:

| Pattern    | Matches                     |
| ---------- | --------------------------- |
| `payments` | exactly `payments`          |
| `prod-*`   | names starting with `prod-` |
| `*-test`   | names ending with `-test`   |
| `*-test-*` | names containing `-test-`   |
| `*`        | any Namespace               |

A `*` in any other position, such as `a*b`, is invalid.

`metadata` matches gRPC request metadata (headers). Keys are case-insensitive and do not support wildcards; values use
the same glob syntax as `namespace`. A key matches when any of the request's values for it match.

## Translate Namespaces

Applications connected to the proxy use short, local Namespace names. Each upstream rewrites those names to the ones its
Temporal Service expects, under `namespaces.rules`. The rewrite applies to requests and is reversed on responses, so
callers only ever see the local name.

```yaml
upstreams:
  - name: cloud
    hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233'
    namespaces:
      rules:
        prefix: '' # Optional string prepended to the local name.
        suffix: .acct # payments becomes payments.acct
        overrides: # Explicit pairs that bypass prefix and suffix.
          - local: billing
            remote: payments.acct
```

- `prefix` and `suffix` wrap every local Namespace: the remote name is `prefix + local + suffix`, and responses are
  unwrapped back to the local name.
- `overrides` lists explicit `local` and `remote` pairs for names that do not follow the prefix and suffix convention.
  An override takes precedence over the prefix and suffix rules. Each local name and each remote name may appear only
  once.
- `bindingLabelsFrom` names another upstream whose rules label encrypted payloads in place of this upstream's own. Use
  it while Namespaces move between upstreams. See
  [Move Namespaces between upstreams](/production-deployment/temporal-proxy/encrypt-payloads#move-namespaces-between-upstreams).

An upstream's `hostPort` and `tls.serverName` can be Go templates resolved per request, so one upstream can serve many
Namespaces. Available variables:

- `{{ .LocalNamespace }}`: the Namespace before translation.
- `{{ .RemoteNamespace }}`: the Namespace after translation.
- `{{ .Metadata.<key> }}` or `{{ index .Metadata "<key>" }}`: a request metadata value.

Upstreams with a static `hostPort` connect eagerly at startup; templated ones connect lazily on first use.

### Translate to a Temporal Cloud upstream

Temporal Cloud takes a fully qualified `<name>.<account-id>`, so the proxy checks the names a Cloud upstream's rules
produce:

- `suffix` must be `.<account-id>`.
- Each override's `remote` must be a complete Cloud Namespace identifier.

A Namespace name runs 2 to 39 characters, starts with a letter, ends with a letter or number, and holds only lowercase
letters, numbers, and `-`. An account identifier runs 5 to 20 characters. Both are checked at startup, so rules that
cannot produce a valid Cloud name fail before the proxy serves anything.

The proxy recognizes Cloud from the address: any `.tmprl.cloud` host, templated ones included. A `tls.serverName` on the
Cloud domain counts too, since a private-link upstream reaches Cloud through a per-VPC hostname but still pins Cloud's
certificate. When neither names the Cloud domain, declare it:

```yaml
upstreams:
  - name: cloud
    hostPort: my-vpc-endpoint.example.internal:7233
    cloud: true # Declare Cloud for an address carrying no Cloud domain.
    namespaces:
      rules:
        suffix: .acct
```

A templated `hostPort` resolves per request, so the name it produces cannot be checked at startup. The proxy logs a
warning when a translated name is not shaped like a Cloud Namespace and forwards the request unchanged.

## Answer the methods Cloud does not serve

A Temporal Cloud Namespace endpoint does not serve every method an SDK Client, the CLI, or the Temporal Web UI calls.
Cloud pins each request to the endpoint's Namespace, so a `ListNamespaces` request, which names none, can never succeed
there. Cloud also refuses every `OperatorService` call made with a customer credential, whatever its role.

The proxy answers these calls from Cloud's control plane, or answers them itself, and converts the reply into the
response the caller expected. That is what lets the open source Temporal Web UI run against Cloud through the proxy:

| Method                                 | Answered with                                                 |
| -------------------------------------- | ------------------------------------------------------------- |
| `WorkflowService/ListNamespaces`       | `CloudService.GetNamespaces`                                  |
| `WorkflowService/GetClusterInfo`       | An empty reply from the proxy, calling no upstream            |
| `OperatorService/ListSearchAttributes` | `CloudService.GetNamespace`, with custom Search Attributes only |
| `OperatorService/ListNexusEndpoints`   | `CloudService.GetNexusEndpoints`                              |
| `OperatorService/GetNexusEndpoint`     | `CloudService.GetNexusEndpoint`                               |

Cloud reports neither a server version nor system Search Attributes, so the proxy does not invent them. Web UI features
gated on a server version stay off, and the UI's filters list only the Namespace's custom Search Attributes.

This needs no configuration. It follows from an upstream already being Temporal Cloud, and the control plane is reached
at `saas-api.tmprl.cloud:443` with that upstream's own credentials, since one Temporal Cloud API key authorizes both.
The connection opens on first use rather than at startup, so a control plane you cannot reach does not stop the proxy
serving everything else.

Translation runs on every Cloud upstream, because a call lands on whichever upstream routing picks.
`ListSearchAttributes` names a Namespace, so it follows that Namespace's route, while `ListNamespaces` and
`ListNexusEndpoints` name none and go to `routing.system`, or `routing.default` when that is unset. Give the system
upstream the same `namespaces.rules` as the upstreams it stands in for, so the Namespace names in those replies come
back translated to local names. To get the untranslated failure back for a Namespace-less call, route those requests at
a Temporal Service that serves them.

> **📝 Note:**
> Version note
>
> Before v0.8.0, the proxy translated only `ListNamespaces`, and only on the upstream serving Namespace-less requests.
>

Override the control plane under `apiTranslations.cloudApi`, which takes the same keys as an upstream:

```yaml
apiTranslations:
  cloudApi:
    hostPort: saas-api.tmprl.cloud:443 # Defaults to Cloud's control plane.
    credentials:
      static:
        apiKey: ${TEMPORAL_CLOUD_API_KEY}
```

Set `cloudApi` in two cases:

- **The upstream authenticates with a client certificate.** Set an API key under `credentials`. The Cloud Ops API
  accepts only API keys, so the proxy has no credential to inherit from the upstream, and it refuses translated calls
  until you provide one.
- **You need to reach a different Cloud environment.** Set `hostPort` to that environment's control plane. If the
  address isn't a Cloud endpoint, the proxy reports it at startup but still accepts it, because test doubles and private
  environments often use addresses outside the Cloud domain.

TLS is not inherited from the upstream, since the control plane is a different host and the upstream's server name or
client certificate would not apply. The dial default stands instead, verifying against the system root pool.

## Authenticate and authorize inbound requests

Inbound authentication runs on the gateway and is off by default: omit the top-level `auth` block to accept all
requests. When present, `auth` must select exactly one authenticator: `staticToken`, `jwks`, or `external`. The gateway
decides before the request is routed, so nothing reaches an upstream until the caller is admitted, and it strips the
credential before forwarding upstream.

Compare an inbound bearer token against a fixed value with `staticToken`:

```yaml
auth:
  staticToken:
    token: ${GATEWAY_TOKEN} # Required. The expected token value.
    header: authorization # Header to read the token from.
    scheme: Bearer # Scheme prefix to strip before comparing.
```

Or verify a JWT's signature and claims against a JWKS endpoint with `jwks`:

```yaml
auth:
  jwks:
    url: https://issuer.example.com/.well-known/jwks.json # Required. Absolute https URL.
    audiences:
      - temporal-proxy
    issuer: https://issuer.example.com/
    header: authorization
    scheme: Bearer
```

`token` (for `staticToken`) and `url` (for `jwks`) are required; the remaining fields are optional.

### Delegate the decision to an extension server

For an identity system neither built-in authenticator covers, or for a decision that turns on more than who the caller
is, point `auth.external` at an extension server you run. Declare the server under the top-level `extensionServers`
block, the same way you would a
[key management backend](/production-deployment/temporal-proxy/encrypt-payloads#plug-in-your-own-key-management-backend),
and the proxy asks it about every stream it accepts, forwarding only the ones it is told to admit.

```yaml
extensionServers:
  - name: authz
    hostPort: 127.0.0.1:9444

auth:
  external:
    name: authz # Required. Names an entry in extensionServers.
    credentialHeaders:
      - authorization
```

`name` must match a configured extension server. `credentialHeaders` names the metadata headers that carry the caller's
credentials: the proxy lifts those into the request it sends the extension server and removes them from the stream it
forwards upstream, so a credential the server consumes never reaches the Temporal Service.

Declaring no headers does not hide the caller's credentials from the extension server, since the proxy forwards the
caller's other metadata on the call either way. What you lose is the field naming them, so the server has to know which
metadata to read and cannot tell a header the proxy vouches for from any other. Nothing is stripped before forwarding
upstream either, so the caller's credential continues to the upstream alongside any credential configured for it.

The server implements `api.auth.v1.AuthService`, one RPC defined in
[`api/auth/v1`](https://github.com/temporalio/temporal-proxy/tree/main/api/auth/v1). Each request carries what the call
is addressing and who is making it:

| Field              | Carries                                                                                                                                                    |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `target.full_name` | The gRPC full method being invoked, leading slash included. Set for every gRPC call, and empty for an HTTP request.                                        |
| `target.namespace` | The Temporal Namespace the request names.                                                                                                                  |
| `target.http`      | For a request to the [codec server or profiling routes](/production-deployment/temporal-proxy/codec-server): the route `group`, HTTP `method`, and `path`. |
| `credentials`      | One entry per declared header the caller sent, each with the canonical header name and every value sent under it.                                         |

For a gRPC call, both `full_name` and `namespace` come from the stream rather than from anything the caller claims, so a
caller cannot forge either by sending a header. `target.namespace` is empty when the method names no Namespace, when the
caller sent no message to read one from, or when the service is not one the proxy forwards. Empty means unknown rather
than a Namespace called nothing, so match a Namespace-scoped rule against `full_name` as well. `credentials` is empty
when you declared no headers or the caller sent none of them, which is an unauthenticated caller rather than a trusted
one.

For an HTTP request, the caller chooses the Namespace through the codec server's path segment or `X-Namespace` header,
and the proxy acts on the Namespace you authorized. Treat any `target.http.group` value you don't recognize as one you
don't authorize, since a release can add groups.

Answer with a `decision`:

- `DECISION_ALLOW` admits the caller and is the only value that does. A response left unfilled is
  `DECISION_UNSPECIFIED`, which denies, so there is no way to admit a caller by omission.
- `DECISION_DENY` refuses the caller, who is told `PERMISSION_DENIED`.
- `reason` is optional and written for whoever operates the server. The proxy records it and keeps it out of what a
  refused caller is told, so it can name internal systems or subjects.

Return a gRPC error only for reaching no verdict at all, such as a backend the server cannot itself reach. The proxy
denies either way, but an error keeps its status code, so `UNAVAILABLE` or `DEADLINE_EXCEEDED` tells a Worker to retry
where a denial tells it not to bother. A server that cannot reach its own backend should report that rather than admit
the caller.

The [authorization example](https://github.com/temporalio/temporal-proxy/tree/main/examples/authz) maps a JWT to claims
and then decides each call against them, the two steps Temporal Server splits across its `ClaimMapper` and `Authorizer`.
It includes a Worker that cannot reach a second Namespace and an auditor that can read Workflow History but not start a
Workflow.

## Present credentials to upstreams

Each upstream can present its own credential to the Temporal Service, set under `credentials`. `static` is the only
variant today; it injects a fixed API key as a bearer header on every outbound request, which is how you connect to
Temporal Cloud:

```yaml
upstreams:
  - name: cloud
    hostPort: my-ns.acct.tmprl.cloud:7233
    credentials:
      static:
        apiKey: ${TEMPORAL_API_KEY} # Required.
        header: authorization # Optional header override.
        scheme: Bearer # Optional scheme override.
```

Credentials require TLS to the upstream. The proxy dials over TLS by default, so this is only an issue when you also set
`insecure: true`. Credentials on a plaintext hop fail the configuration at startup rather than putting the key on the
wire in the clear.

## Tune upstream connections 

Each upstream takes an optional `connection` block. The defaults match the Temporal Go SDK, so a Worker gets the same
limits through the proxy as on a direct connection:

```yaml
upstreams:
  - name: cloud
    hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233'
    connection:
      maxResponseSize: 128MiB # Largest response accepted from the upstream.
      maxConnections: 32 # Connections calls are spread across, per target.
      keepAlive:
        time: 30s # Idle time before the proxy pings the connection.
        timeout: 15s # Wait for the ping's ack before closing the connection.
```

| Field               | Default  | Rules                                                                                                         |
| ------------------- | -------- | ------------------------------------------------------------------------------------------------------------- |
| `maxResponseSize`   | `128MiB` | Positive and under 2GiB. A byte count, or a number with a decimal (`KB`, `MB`, `GB`) or binary (`KiB`, `MiB`, `GiB`) unit. |
| `maxConnections`    | `32`     | 1 to 64.                                                                                                      |
| `keepAlive.time`    | `30s`    | At least `10s`, the floor gRPC and the Temporal Service enforce.                                              |
| `keepAlive.timeout` | `15s`    | Positive.                                                                                                     |

- `maxResponseSize` is the largest response a Worker gets from that upstream, including any growth from decrypting
  payloads on the way back. A response over the limit fails with `ResourceExhausted`, which a Workflow Task with a long
  Event History can hit.
- `maxConnections` spreads calls round-robin across that many connections. Each SDK long poll holds an HTTP/2 stream
  for up to a minute, and a connection carries a limited number of streams, so too few connections stall calls behind
  polls. A static upstream opens every connection at startup. A templated upstream opens up to `maxConnections` for each
  target it resolves, usually one per Namespace, as calls reach them. See
  [Size the connection pool](/production-deployment/temporal-proxy/sizing-and-performance#size-the-connection-pool).
- `keepAlive` detects a connection that a NAT or load balancer dropped without closing, which would otherwise stall
  polls until the operating system gives up on it.

The [Temporal Cloud control plane connection](#answer-the-methods-cloud-does-not-serve) inherits its upstream's
connection settings.

> **📝 Note:**
> Version note
>
> The `connection` block is new in v0.9.0. Earlier releases dialed each upstream over one connection with gRPC's defaults,
> including a 4MiB response limit and no keepalive.
>

### Discover upstreams through DNS SRV records

Set an upstream's `hostPort` to `srv:///<record>` and the proxy balances calls round-robin across every backend the
record lists:

```yaml
upstreams:
  - name: self-hosted
    hostPort: srv:///_grpc._tcp.temporal-frontend.temporal.svc.cluster.local
```

The proxy re-reads the record every 30 seconds, and within 5 seconds of a backend failing. `maxConnections` applies to
each backend, so 4 connections across 10 backends opens 40.

Over TLS, the proxy verifies every backend's certificate against the record name minus its leading `_service._proto`
labels, `temporal-frontend.temporal.svc.cluster.local` in the example above, rather than the host each record points
at. Set `tls.serverName` when the certificates carry a different name.

## Configure TLS 

TLS is configured in two independent places, both using the same keys: `ca`, `cert`, `key`, and `serverName`. `ca`,
`cert`, and `key` are paths to PEM files on disk, not inline PEM content. On Kubernetes, let the chart mount them from a
Secret and fill in the paths for you, as described in
[Supply TLS material](/production-deployment/temporal-proxy/deploy-kubernetes#supply-tls-material).

The two directions default differently, because a dialer can fall back on the system root pool and a listener has no
certificate to fall back on.

**Inbound, on the gateway.** The top-level `tls` block secures connections from your applications. Set `cert` and `key`
for server TLS, and add `ca` to enforce mutual TLS, which requires each client to present a certificate signed by that
CA. A gateway with no `tls` block serves plaintext, which is what local development usually handles.

**Outbound, per upstream.** Each upstream dials over TLS whether or not it declares a `tls` block:

- No `tls` block verifies the upstream against the system root certificate pool and presents no client certificate. This
  is what Temporal Cloud with an API key needs.
- `ca` alone verifies the upstream against a private trust anchor, still presenting no client certificate.
- `cert` and `key` together select mutual TLS and require `ca`. They must be set as a pair.
- `insecure: true` dials plaintext, for a local development Service that terminates no TLS.

Set `serverName` when the host you dial does not match the common name or SAN on the server's certificate.

`insecure` and a `tls` block cannot be set together because they are opposite, so the configuration fails at startup
rather than silently picking one. The same rules govern every outbound hop, so an `extensionServers` entry and the
`apiTranslations.cloudApi` block read exactly the same way.

> **📝 Note:**
> Version note
>
> v0.6.0 made TLS the default for everything the proxy dials. Before it, an upstream with no `tls` block connected in
> plaintext. A `tls: {}` block still means what it always did, so configurations that declared one keep working, but an
> upstream that relied on the old plaintext default now needs `insecure: true`.
>

For payload encryption (envelope encryption, cloud KMS, and custom key backends), see
[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads).

---

# Deploy Temporal Proxy to Kubernetes

Source: https://docs.temporal.io/production-deployment/temporal-proxy/deploy-kubernetes

> Deploy the Temporal Proxy with the Helm chart, including configuration, secrets, TLS, and KMS access.

> **Public Preview**
> Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between
> releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status and
> the definitive configuration schema.

The [Helm chart](https://github.com/temporalio/helm-charts/tree/main/charts/temporal-proxy) provisions everything the
proxy needs: a Deployment, a Service, the ConfigMap that holds your configuration, a ServiceAccount, and an optional
HorizontalPodAutoscaler and PodDisruptionBudget. The `extraObjects` value renders additional manifests as-is, for
example an ExternalSecret that supplies upstream credentials.

## Supply the configuration

`config` mirrors the proxy's own configuration schema. Whatever you put under it is what lands in the ConfigMap the
chart mounts at `/etc/temporal-proxy/config.yaml`. The chart does not run `config` through Helm's `tpl` function, so the
proxy's own per-request templates, such as `{{ .RemoteNamespace }}` in an upstream `hostPort`, pass through untouched
and are evaluated by the proxy at request time.

```yaml
config:
  routing:
    default: default
  upstreams:
    - name: default
      hostPort: localhost:7233
```

If you leave `config.hostPort` unset, the chart defaults the gateway's listen address to `:<service.port>`.

Values the configuration references with `${VAR}` come from `env` and `envFrom` on the Deployment, which is how you
supply things like a Temporal Cloud account identifier:

```yaml
env:
  - name: TEMPORAL_ACCOUNT
    value: a1b2c

envFrom:
  - secretRef:
      name: proxy-env
```

## Supply credentials from a Secret

Two configuration fields accept either a plain string or a `secretKeyRef`, so the ConfigMap never has to hold a
credential in plaintext:

- `upstreams[].credentials.static.apiKey`
- `auth.staticToken.token`

Given a `secretKeyRef`, the chart adds an environment variable to the proxy container sourced from that Secret and
rewrites the configuration value to `${VAR}`, which the proxy expands at startup. The generated names are
`TP_UPSTREAM_<NAME>_API_KEY`, where `<NAME>` is the upstream's name upper-cased with non-alphanumeric characters
replaced by `_`, and `TP_AUTH_STATIC_TOKEN`.

```yaml
config:
  upstreams:
    - name: cloud
      hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233'
      credentials:
        static:
          apiKey:
            secretKeyRef:
              name: temporal-cloud
              key: api-key
```

That renders `apiKey: ${TP_UPSTREAM_CLOUD_API_KEY}` into the ConfigMap and adds a matching environment variable backed
by the `temporal-cloud` Secret's `api-key` entry.

## Supply TLS material 

The gateway's `config.tls` block and each `upstreams[].tls` block accept a `secretName`. The chart mounts that Secret at
`/etc/temporal-proxy/certs/gateway` or `/etc/temporal-proxy/certs/upstream-<name>` and fills in the file paths for you:

| Configuration key | Value                                                         |
| ----------------- | ------------------------------------------------------------- |
| `cert`            | `<mount>/tls.crt`, or `<mount>/<certKey>` if `certKey` is set |
| `key`             | `<mount>/tls.key`, or `<mount>/<keyKey>` if `keyKey` is set   |
| `ca`              | `<mount>/<caKey>`, only when `caKey` is set                   |

`ca` is opt-in because it changes behavior rather than just adding material. On the gateway, a `ca` enforces mutual TLS,
so only set `caKey` on `config.tls` when you intend to require client certificates. See
[Configure TLS](/production-deployment/temporal-proxy/configure#configure-tls) for what each combination means.

```yaml
config:
  tls:
    secretName: temporal-proxy-server-tls
  upstreams:
    - name: cloud
      hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233'
      tls:
        secretName: temporal-cloud-client-tls
        caKey: ca.crt
```

The default keys match the `tls.crt` and `tls.key` pair [cert-manager](https://cert-manager.io/) issues, but
cert-manager is not required: any Secret works, and `certKey`, `keyKey`, and `caKey` point at whatever keys yours uses.
Keep upstream `name` values DNS-safe, meaning lowercase alphanumeric characters and `-`, since the chart builds a volume
name from them. To manage certificate files yourself instead, skip `secretName` and mount them with the chart's generic
`volumes` and `volumeMounts` values.

## Probe the gateway

The gateway serves the standard gRPC health service, so a Kubernetes gRPC probe can read it directly. The chart passes
`livenessProbe` and `readinessProbe` through to the container as written, and its defaults are HTTP GETs against the
gateway port, which a gRPC listener does not answer:

```yaml
livenessProbe:
  grpc:
    port: 8443 # Must be numeric. Match service.port.
readinessProbe:
  grpc:
    port: 8443
    service: temporal.api.workflowservice.v1.WorkflowService
```

Naming a service is optional; omit it to check the empty service name. Either way the answer is the same, because the
proxy's health is process-wide, and a service your configuration does not allow returns `NOT_FOUND`. See
[Health checks](/production-deployment/temporal-proxy/configure#health-checks) for what the gateway publishes.

The status your probe reads is refreshed by a check the proxy runs against itself every 30 seconds, tunable under the
`health` block. The two cover different scenarios, which is why you want both. The internal check exercises the
interceptor chain every forwarded request travels, and your probe proves the gateway still accepts a connection and
completes a handshake, which the internal check deliberately skips.

Kubernetes gRPC probes connect in plaintext, so they cannot reach a gateway that terminates TLS. When `config.tls` is
set, probe the proxy some other way, such as an exec probe running a client that presents the right certificate.

## Shut down cleanly

On `SIGTERM` the proxy stops accepting connections and drains in-flight RPCs. The gateway drains within five seconds,
and the whole stop sequence is bounded at 30 seconds. Long-poll methods
block for much longer than that, so a rollout normally cuts some of them short: the proxy logs a warning naming the
deadline that ended the drain and still exits zero, because the callers re-poll. Treat that warning as ordinary on a
restart rather than a failed shutdown.

An ordinary drain finishes in about five seconds, well inside the Kubernetes default `terminationGracePeriodSeconds`
of 30.

## Grant access to your KMS key

The one thing the chart cannot do for you is grant the proxy access to your cloud KMS key. When you enable
[payload encryption](/production-deployment/temporal-proxy/encrypt-payloads), the proxy needs a cloud identity that
holds the encrypt and decrypt permissions on the key. Bind the chart's Kubernetes ServiceAccount to that identity with
workload identity, so the proxy authenticates to your KMS with no static credentials. If you do not use encryption, you
can skip this: API keys and TLS material come from the configuration and mounted Secrets, so the proxy needs no cloud
identity.

Grant the cloud identity the encrypt and decrypt permissions first, as described under
[Encrypt payloads](/production-deployment/temporal-proxy/encrypt-payloads), then bind it to the ServiceAccount as shown
below. The bindings reference the ServiceAccount by name, so set `serviceAccount.name` in your values to keep it stable
and matching what you bind on the cloud side.

### AWS IRSA and Pod Identity

Create an IAM role whose trust policy lets your cluster's OIDC provider assume it from the proxy's ServiceAccount, and
attach the KMS policy from [AWS KMS](/production-deployment/temporal-proxy/encrypt-payloads#aws-kms). Then annotate the
ServiceAccount with the role ARN:

```yaml
serviceAccount:
  name: temporal-proxy
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/temporal-proxy
```

On EKS you can use [Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) instead: create
an association between the role and the ServiceAccount with `aws eks create-pod-identity-association`, and no annotation
is required.

### Azure Workload Identity

Create a federated identity credential that links a user-assigned managed identity (holding the key permissions) to the
proxy's ServiceAccount subject:

```bash
az identity federated-credential create \
  --name temporal-proxy \
  --identity-name <managed-identity-name> \
  --resource-group <resource-group> \
  --issuer <aks-oidc-issuer-url> \
  --subject system:serviceaccount:<namespace>:temporal-proxy
```

Then annotate the ServiceAccount with the identity's client ID and label the Pods so the webhook injects credentials:

```yaml
serviceAccount:
  name: temporal-proxy
  annotations:
    azure.workload.identity/client-id: <client-id>

podLabels:
  azure.workload.identity/use: 'true'
```

### GCP Workload Identity

Bind the Google service account (GSA) that holds the key permissions to the proxy's Kubernetes service account (KSA):

```bash
gcloud iam service-accounts add-iam-policy-binding \
  proxy@my-project.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:my-project.svc.id.goog[NAMESPACE/KSA_NAME]"
```

Then annotate the ServiceAccount through Helm so GKE maps it to the GSA:

```yaml
serviceAccount:
  name: temporal-proxy
  annotations:
    iam.gke.io/gcp-service-account: proxy@my-project.iam.gserviceaccount.com
```

---

# Encrypt payloads with Temporal Proxy

Source: https://docs.temporal.io/production-deployment/temporal-proxy/encrypt-payloads

> Configure Temporal Proxy payload encryption with envelope encryption and cloud or custom KMS backends.

> **Public Preview**
> Temporal Proxy is under active development and evolving quickly. Behavior and configuration can change between
> releases. See the [temporal-proxy repository](https://github.com/temporalio/temporal-proxy) for the current status
> and the definitive configuration schema.

The proxy can encrypt Workflow and Activity payloads on the hop to an upstream and decrypt them on responses, set under
the top-level `encryption` block. It is off by default. Workers and Clients keep exchanging cleartext with the gateway;
the proxy seals codec-capable Payloads before they leave and opens them on the way back. New Payloads sent through the
proxy reach the upstream Temporal Service as ciphertext. Encryption is transparent, requiring no change to Worker or
Client code.

The proxy uses envelope encryption: a short-lived data encryption key (DEK) encrypts each payload with AES-256-GCM, and a KMS
key you own wraps the DEK. The wrapped DEK and a reference to the key that wrapped it travel with the payload, so the
proxy never holds key material; it calls your KMS to wrap and unwrap DEKs. Supported key schemes are `awskms`,
`azurekeyvault`, and `gcpkms` for cloud KMS, `extension` for a
[key management backend you run yourself](#plug-in-your-own-key-management-backend), and `testing` for local development
only.

```mermaid
flowchart LR
    App[Worker / SDK Client]

    subgraph Proxy[Temporal Proxy]
        direction TB
        Seal["Encrypt on the way out:<br/>DEK seals the payload,<br/>KMS key wraps the DEK"]
        Open["Decrypt on the way back:<br/>KMS key unwraps the DEK,<br/>DEK opens the payload"]
    end

    KMS[("Your KMS<br/>(AWS, Azure, or GCP)")]
    Upstream[Upstream Temporal Service]

    App -->|cleartext request| Seal
    Seal -->|wrap DEK| KMS
    Seal -->|Payloads: ciphertext<br/>other fields: cleartext| Upstream
    Upstream -->|Payloads: ciphertext<br/>other fields: cleartext| Open
    KMS -->|unwrap DEK| Open
    Open -->|cleartext response| App
```

**Configuration**

```yaml
encryption:
  enabled: true # Encrypt new payloads. Optional; defaults to false.
  namespaceBinding: strict # Required with keys: off, permissive, or strict.
  failures: true # Also encrypt failure messages and stack traces. Requires enabled.
  cacheSize: 100 # Bounds the in-memory decrypted-DEK cache. Omit for 100; 0 disables it.
  default: # Fallback key policy for any Namespace without an override.
    uri: awskms:///arn:aws:kms:us-east-1:123456789012:key/abcd-1234?region=us-east-1
    duration: 1h # How long a DEK is used for new encryption before it rotates.
    renewBefore: 15m # Lead time before expiry to pre-rotate. Must be less than duration.
  overrides: # Per-Namespace policies, keyed by local (pre-translation) Namespace.
    payments:
      uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1
      duration: 30m
      renewBefore: 5m
```

- `default` is a key policy required whenever `enabled: true`. It makes encryption fail-closed: any Namespace without
  its own override, including Namespaces created after startup, uses the default rather than sending plaintext. Enabling
  encryption without a `default` is a startup error.
- `overrides` maps a local Namespace to its own key policy, which takes precedence over `default`. The keys are the
  local (pre-translation) Namespace names. Reach for overrides to scope blast radius, spread load across KMS keys, or
  keep a tenant's key in its own region or provider.
- `cacheSize` bounds the in-memory cache of decrypted DEKs, which avoids a KMS call on every message. Omit it for the
  default of 100; `0` disables the cache, so every decryption unwraps its DEK through the KMS. Before v0.6.0 an absent
  `cacheSize` disabled the cache, so upgrading a configuration that never set one turns the cache on.
- `namespaceBinding` ties each payload to the Namespace it was sealed in. It has no default: the proxy refuses to start
  with keys configured and no value set. Use `strict` for a new deployment. See
  [Bind payloads to their Namespace](#namespace-binding) before you set it on an existing one.
- `failures` encrypts the message and stack trace of outbound failures. See [Encrypt failures](#encrypt-failures).
- `skipEncodings` forwards payloads under the listed encodings without sealing them. See
  [Pass through payloads that are already encrypted](#skip-encodings).

> **📝 Note:**
> Scope of fail-closed behavior
>
> Fail-closed applies to outbound codec-capable Payloads on calls that pass through a proxy where `encryption.enabled` is
> `true`. If the proxy cannot encrypt one of these Payloads, such as when the KMS is unavailable, it fails the request
> before sending it upstream. It does not fall back to plaintext.
>
> The proxy cannot enforce encryption for a Client, Worker, Web UI, or CLI that connects directly to the upstream Temporal
> Service. Restrict credentials and network paths when all application traffic must pass through the proxy.
>
> On responses, the proxy decrypts Payloads that carry its encryption metadata. It passes other Payloads through unchanged
> so data written before proxy adoption remains readable during a migration. This behavior continues until existing
> plaintext data ages out under the Namespace's retention period.
>
> Fail-closed does not mean that every request field is encrypted. Search Attributes remain unencrypted so the Temporal
> Service can index them. Failure messages and stack traces are plain strings rather than Payloads, so they stay
> unencrypted unless you set [`failures`](#encrypt-failures). Payloads under an encoding listed in
> [`skipEncodings`](#skip-encodings) pass through as they arrive. See
> [Codecs and Encryption](/production-deployment/data-encryption) for encryption coverage and
> [Failure Converter](/failure-converter) for failure encoding.
>

`default` and each `overrides` entry are key policies with the same shape:

| Field         | Meaning                                                                                                                                                                                                   |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `uri`         | Active key that wraps new DEKs. Required. Scheme must be `awskms`, `azurekeyvault`, `gcpkms`, `extension`, or `testing`. Each policy's `uri` must be unique across `default` and every `overrides` entry. |
| `decryptURIs` | Additional KMS keys accepted only when unwrapping existing DEKs, for key migration. Optional.                                                                                                             |
| `duration`    | How long a DEK is used for new encryption before it rotates. Must be greater than zero.                                                                                                                   |
| `renewBefore` | Lead time before expiry at which the proxy pre-rotates the DEK. Must be at least zero and less than `duration`.                                                                                           |

DEK rotation is automatic on the `duration` and `renewBefore` schedule; you do not manage it. Older payloads stay
decryptable because their wrapped DEK and key reference travel with them.

To change the active key without losing access to existing data, for example when switching providers, promote the new
key to `uri` and move the old one to `decryptURIs` in the same update. New payloads use the new key; older payloads
still resolve against the decrypt-only entry:

```yaml
encryption:
  default:
    uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2
    decryptURIs:
      - gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1
```

> **📝 Note:**
> Multi-region deployments need a multi-region key
>
> To stay available alongside a highly available upstream that fails over across regions, such as a Temporal Cloud High
> Availability Namespace whose endpoint moves between regions, you typically run the proxy in more than one region. After
> a failover, a payload sealed by the proxy in one region may be opened by the proxy in another, so every regional proxy
> must be able to unwrap the others' DEKs. Back the key with a multi-region KMS key so the same key material is reachable
> from every region. How you express that differs by provider:
>
> - **AWS:** use a multi-Region key (its ID starts with `mrk-`) and replicate it into each region. The replicas share key
>   material but each has its own regional ARN, so set each proxy's `uri` to its local ARN and list the other regions'
>   ARNs in `decryptURIs`:
>
>   ```yaml
>   encryption:
>     default:
>       # Config for the us-east-1 proxy.
>       uri: awskms:///arn:aws:kms:us-east-1:123456789012:key/mrk-abcd1234?region=us-east-1
>       decryptURIs:
>         - awskms:///arn:aws:kms:us-west-2:123456789012:key/mrk-abcd1234?region=us-west-2
>   ```
>
> - **Azure:** Key Vault keys are regional. Use a separate vault per region (or geo-replication), and, because each vault
>   gives the key a distinct URI, set each proxy's `uri` to its local vault and list the other vaults' keys in
>   `decryptURIs`.
> - **GCP:** create the key in a multi-region location (for example, `locations/us`) or in `global`. The resource name is
>   the same everywhere, so every proxy uses the identical `uri` and no `decryptURIs` are needed:
>   `gcpkms://projects/my-project/locations/us/keyRings/codec/cryptoKeys/my-key`.
>
> For a single-region proxy deployment, a regional key is the right choice; reach for a multi-region key only when the
> proxy itself spans regions.
>

Decryption always runs on any payload that references a known key, regardless of `enabled`. Setting `enabled: false`
stops new encryption but keeps opening older payloads, which is a valid decrypt-only posture. The proxy reads encryption
configuration at startup, so restart it after changing keys or policies.

The `testing://` scheme holds its key material in the configuration itself and provides no real security. Use it only
for local development, and point production at `awskms`, `azurekeyvault`, `gcpkms`, or an extension server.

The proxy authenticates to each cloud provider through that provider's default credential chain, so no key material or
static cloud credentials live in the proxy configuration. On Kubernetes, bind the proxy's service account to a cloud
identity (IRSA or Pod Identity on AWS, Workload Identity on Azure and GCP); off Kubernetes, the SDKs resolve credentials
from the host. Grant that identity only the permission to encrypt and decrypt with the specific key, as shown for each
provider below.

## AWS KMS 

Wrap DEKs with an AWS KMS symmetric key, referenced by ARN. The `awskms://` scheme uses a triple slash (`awskms:///`) so
the ARN can sit in the URL path; add `?region=` when the key's region differs from the proxy's ambient region. See the
[AWS KMS documentation](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html).

The proxy needs `kms:Encrypt` and `kms:Decrypt` on the key. Attach them as an inline policy to the IAM role the proxy
runs as:

```bash
aws iam put-role-policy \
  --role-name temporal-proxy \
  --policy-name temporal-proxy-kms \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": ["kms:Encrypt", "kms:Decrypt"],
        "Resource": "arn:aws:kms:us-east-1:123456789012:key/abcd-1234"
      }
    ]
  }'
```

```yaml
encryption:
  enabled: true
  namespaceBinding: strict
  default:
    uri: awskms:///arn:aws:kms:us-east-1:123456789012:key/abcd-1234?region=us-east-1
    duration: 1h
    renewBefore: 15m
```

## Azure Key Vault

Wrap DEKs with a Key Vault key. The default algorithm is RSA-OAEP-256, so the key must be an RSA key; append
`?algorithm=` to the URI to override it. The key version is optional and defaults to the latest version. See the
[Azure Key Vault documentation](https://learn.microsoft.com/en-us/azure/key-vault/general/overview).

The proxy needs the `encrypt` and `decrypt` key operations, granted by the built-in Key Vault Crypto User role (Azure
RBAC) or an access policy that allows the Encrypt and Decrypt key permissions:

```bash
az role assignment create \
  --assignee <proxy-identity> \
  --role "Key Vault Crypto User" \
  --scope <key-vault-resource-id>
```

```yaml
encryption:
  enabled: true
  namespaceBinding: strict
  default:
    uri: azurekeyvault://my-vault.vault.azure.net/keys/my-key
    duration: 1h
    renewBefore: 15m
```

## Google Cloud KMS

Wrap DEKs with a Cloud KMS key, referenced by its full resource name. See the
[Cloud KMS documentation](https://cloud.google.com/kms/docs).

The proxy needs `cloudkms.cryptoKeyVersions.useToEncrypt` and `cloudkms.cryptoKeyVersions.useToDecrypt`, granted by the
`roles/cloudkms.cryptoKeyEncrypterDecrypter` role on the key and bound to the proxy's service account:

```bash
gcloud kms keys add-iam-policy-binding my-key \
  --location global --keyring codec \
  --member serviceAccount:proxy@my-project.iam.gserviceaccount.com \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter
```

```yaml
encryption:
  enabled: true
  namespaceBinding: strict
  default:
    uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/my-key
    duration: 1h
    renewBefore: 15m
```

## Plug in your own key management backend 

To wrap DEKs with a backend the proxy has no built-in support for, such as an on-prem HSM or an internal key service,
run that backend as a gRPC server and point the proxy at it. The proxy calls it to wrap and unwrap DEKs only; payload
plaintext never reaches it.

Declare the server under the top-level `extensionServers` block, then address its keys with the `extension://` scheme:

```yaml
extensionServers:
  - name: kms
    hostPort: 127.0.0.1:9443 # Literal host:port. Templates are rejected.
    tls:
      ca: /etc/temporal-proxy/certs/kms/ca.pem
      serverName: kms.internal # Set when the dialed host does not match the certificate.
    credentials:
      static:
        apiKey: ${KMS_API_KEY} # Requires TLS, as with upstreams.

encryption:
  enabled: true
  namespaceBinding: strict
  default:
    uri: extension://kms/payloads # Host names an entry in extensionServers.
    duration: 1h
    renewBefore: 15m
```

- `name` identifies the server so key URIs can reference it. Names and `hostPort` values must be unique across the list,
  and every `extension://` URI must name a configured server.
- `hostPort` must be a literal address. Unlike an upstream, an extension server is dialed at a fixed address rather than
  resolved per request, so a templated value is rejected at startup.
- `tls` takes the same keys as an upstream's, and `credentials` works the same way. The proxy dials an extension server
  over TLS by default, so a plaintext one needs `insecure: true`, and credentials on a plaintext hop fail the
  configuration at startup.
- The path segment in the key URI, `payloads` above, is a proxy-side key name. It never reaches the extension server,
  which selects keys by Namespace, but it must be unique across `default` and every `overrides` entry.

The server implements `api.kms.v1.EncryptionService`, two RPCs defined in
[`api/kms/v1`](https://github.com/temporalio/temporal-proxy/tree/main/api/kms/v1):

| RPC       | Receives                                          | Returns          |
| --------- | ------------------------------------------------- | ---------------- |
| `Encrypt` | the local Namespace and the DEK to wrap           | the wrapped DEK  |
| `Decrypt` | a wrapped DEK, with no Namespace or other context | the original DEK |

Two constraints follow from that shape. `Decrypt` gets ciphertext and nothing else, so whatever your server returns from
`Encrypt` must carry enough information to identify the key that produced it. And the Namespace on `Encrypt` is always
the local, pre-translation name, so a server keying on Namespace must use that name rather than the translated remote
one.

The proxy repository ships [`pkg/ext`](https://github.com/temporalio/temporal-proxy/tree/main/pkg/ext), a Go starting
point for building one. Implement its `KMS` interface and hand it to `ext.Serve`, which supplies the listener, signal
handling, graceful shutdown, and the gRPC health service. Its `NewKeyWrapper` and `NewSealWrapper` helpers handle the
framing constraint above, so an implementation brings key material and writes no cryptography of its own:
`NewKeyWrapper` seals DEKs with an AEAD over keys you supply, and `NewSealWrapper` delegates the sealing to a backend
that will not hand over its keys, such as an HSM. Both record the key version alongside the material, so anything
already sealed stays readable across a rotation.

Whichever route you take, secure the server itself. `ext.Serve` listens in plaintext unless you supply credentials, and
the proxy dials an extension server over TLS unless its entry sets `insecure: true`, so the two ends have to agree.

The proxy builds the connection when it builds its key policies, so a malformed address, credentials without TLS, or an
unreadable `ca` file all fail at startup. Peer verification happens later, because gRPC connects lazily: a `ca` or
`serverName` that does not match the server's certificate is not discovered until the first payload needs a key, and it
surfaces as a hung request rather than a clear error.

The [KMS extension server example](https://github.com/temporalio/temporal-proxy/tree/main/examples/kms) runs the whole
path on localhost, with a reference provider you can read as a starting point.

## Bind payloads to their Namespace 

With binding on, the proxy records the Namespace a payload was sealed in under its `encryption-namespace` metadata and
authenticates it as AES-GCM additional data, so a payload opens only with the label it was sealed with. Under `strict`,
a payload copied from one Namespace into another can't be opened there.

The label is the Namespace's name on the upstream, after
[translation](/production-deployment/temporal-proxy/configure#translate-namespaces). Renaming a local Namespace in the
configuration is safe, and two proxies that name one Namespace differently agree. Changing a translation rule changes
the label, though, so payloads sealed before the change no longer match. To label payloads with another upstream's
translation, for example while Namespaces move to Temporal Cloud, see
[Move Namespaces between upstreams](#move-namespaces-between-upstreams).

When the proxy opens a payload, it also compares the label with the request's Namespace, translated the same way. Nexus
payloads that cross Namespaces by design are exempt, along with everything nested in them, such as an Operation's
failure details. Child Workflows started in another Namespace, and Signals sent from one, carry that Namespace's label
and are not exempt, so deployments that rely on them should stay on `permissive`. The
[codec server](/production-deployment/temporal-proxy/codec-server#name-the-namespace) applies the same comparison to the
Namespace a request names.

| Mode         | Seals     | Opens labeled for the request's Namespace | Opens labeled for another | Opens unlabeled         |
| ------------ | --------- | ----------------------------------------- | ------------------------- | ----------------------- |
| `off`        | Unlabeled | Yes                                       | Yes, counted              | Yes, counted            |
| `permissive` | Labeled   | Yes                                       | Yes, counted              | Yes, counted            |
| `strict`     | Labeled   | Yes                                       | No (`result="mismatch"`)  | No (`result="unbound"`) |

"Counted" means the payload increments `vault_namespace_mismatches_total` or `vault_unbound_opens_total`. See
[Observability](/production-deployment/temporal-proxy#observability).

`namespaceBinding` has no default, because the right mode depends on whether payloads sealed by an earlier release
exist. A new deployment has none, so set `strict`.

### Turn on binding for an existing deployment

An existing deployment has unlabeled payloads in Event History, which `strict` refuses. Set `off` before you upgrade to
v0.9.0, then:

1. **Upgrade.** Roll v0.9.0 out to every replica, and to every chained proxy that holds the keys, leaving `off`. An
   older release can't open a labeled payload, so nothing may label until nothing older is running.
2. **Turn on `permissive`** everywhere, in any order. Every upgraded proxy already opens labeled payloads.
3. **Watch `vault_unbound_opens_total` and `vault_namespace_mismatches_total`.** They count every unlabeled payload,
   and every payload labeled for another Namespace, handed to decrypt, whatever the outcome. When both have stayed at
   zero for longer than the longest Retention Period, plus anything you still read from Archival, set `strict`.

To decide per Namespace, turn on [`metrics.labels.namespace`](/production-deployment/temporal-proxy#observability).
Judge readiness on the proxies that hold the key, because a chained proxy that passes foreign payloads through keeps
counting them.

Going back from `strict` to `permissive` is always safe. Setting `strict` too early makes unlabeled payloads unreadable
but not lost: switching to `permissive` restores them. Going from `permissive` to `off` still opens labeled payloads but
seals unlabeled ones again.

> **⚠️ Caution:**
>
> After you turn on `permissive` or `strict`, downgrading to a release earlier than v0.9.0 leaves every labeled payload
> unreadable to it.
>

### Move Namespaces between upstreams 

Because the label follows the routed upstream's translation, two proxies can label the same Namespace differently. A
migration to Temporal Cloud is the common case. Until handoff, each Namespace routes to an on-premises upstream with no
translation, so the proxy labels payloads `ns1`. Meanwhile, s2s-proxy replicates that history to Cloud and translates
`ns1` to `ns1.acct`. For `strict` to hold on both sides of the handoff, both have to label payloads `ns1.acct`.

Set `namespaces.bindingLabelsFrom` on the upstream the Namespaces are leaving, and name the upstream they are moving to:

```yaml
upstreams:
  - name: local
    hostPort: temporal.internal:7233
    namespaces:
      bindingLabelsFrom: cloud # Label payloads with cloud's name for each Namespace.
  - name: cloud
    hostPort: my-region.tmprl.cloud:7233
    namespaces:
      rules:
        suffix: .acct
```

Payloads sealed or opened through `local` are labeled with the name `cloud` gives the Namespace, `ns1.acct`. Requests to
`local` still use its own translation, and keys are still chosen by the local Namespace name.

- `bindingLabelsFrom` must name another configured upstream, and that upstream can't set `bindingLabelsFrom` itself. The
  proxy refuses to start otherwise.
- Setting or changing it changes the label for every Namespace routed through that upstream, the same as changing a
  translation rule. On a deployment that already labels payloads, switch to `permissive` first and watch
  `vault_namespace_mismatches_total`.
- Once every Namespace routes to `cloud`, removing the setting, or the `local` upstream, leaves every label unchanged.

The [codec server](/production-deployment/temporal-proxy/codec-server#name-the-namespace) has no upstream to consult and
compares payloads with the Namespace the request names. A Web UI or CLI pointed at the on-premises Temporal Service names
`ns1`, so under `strict` it can't open payloads labeled `ns1.acct`. Use `permissive` during the migration if that view
needs to decode payloads.

`bindingLabelsFrom` is new in v0.10.0.

## Encrypt failures 

A failure's message and stack trace are plain strings rather than Payloads, so they reach the upstream as cleartext even
with encryption enabled, and they often carry the same sensitive data as the Payloads. Set `failures: true`, which
requires `enabled: true`, to encrypt them too:

```yaml
encryption:
  enabled: true
  failures: true
```

The proxy does what the SDK's `EncodeCommonAttributes` option does: it moves the message and stack trace into the
failure's encoded attributes, a Payload it then seals like any other, and leaves `Encoded failure` in their place. The
format matches the SDK's, and SDK Clients always decode it on read, so Workers need no changes. Failures a Worker
already encoded are left alone.

On responses, the proxy restores the original text after opening the attributes, so a client that doesn't decode them,
such as the Web UI or the CLI, sees the message where it expects it. Restoring runs whenever keys are configured, so
failures stay readable after you turn `failures` off.

## Pass through payloads that are already encrypted 

With encryption enabled, the proxy seals every outbound Payload, including one a Worker's own codec already encrypted or
one an earlier proxy in a chain already sealed. List those encodings under `skipEncodings` to forward them unchanged:

```yaml
encryption:
  enabled: true
  skipEncodings:
    - binary/encrypted
```

Matching is exact on the Payload's `encoding` metadata. Each skipped Payload counts toward `payloads_skipped_total`.

Listing `binary/encrypted`, the encoding the proxy itself writes, has a second effect: a response sealed under a KMS key
this proxy doesn't hold comes back as it is rather than failing the call. That is what a chained proxy needs, since the
payloads it can't open belong to the next hop. On a standalone proxy, it also hides a key you removed from the
configuration, so alert on `vault_ops_total{result="unknown_key"}`.

> **⚠️ Caution:**
>
> Listing an encoding trusts every Worker that sets it. The proxy can't tell ciphertext from plaintext labeled that way,
> so a Worker that labels plaintext with a listed encoding sends it upstream unencrypted.
>

> **⚠️ Caution:**
> Deleting a key is irreversible
>
> Deleting a KMS key destroys everything encrypted under it, and neither Temporal nor your cloud provider can recover it.
> Keep every key reachable, as `uri` or in `decryptURIs`, until every payload it wrapped is gone from every system the
> proxy decrypts for, including Workflow history, visibility, and Archival storage. In practice, keep a key at least until
> the Workflow Retention Period has elapsed.
>
> To retire a key, revoke the proxy's permissions rather than deleting it. Removing access is reversible: you can re-grant
> it later if a payload still needs opening, whereas a deleted key is gone for good. Revoking only the encrypt permission
> while leaving decrypt in place also lets you stop new encryption without stranding existing payloads.
>

---

# Sizing and performance

Source: https://docs.temporal.io/production-deployment/temporal-proxy/sizing-and-performance

> Size Temporal Proxy CPU, memory, connection pools, and replicas using reference measurements, then validate capacity with your workload.

> **Public Preview**
> Temporal Proxy is under active development. Validate capacity on the version you deploy; these measurements are
> reference results, not throughput guarantees.

**Recommended starting size:** For a Temporal Proxy instance that applies payload encryption for all payload sizes,
use **4 CPUs and 12 GiB of memory**, with `GOMEMLIMIT=11GiB`, and plan for each instance to handle up to
**3,000 requests per second and 100 SDK Workers**. Scale beyond these limits by adding more Proxy replicas (instances)
behind a gRPC-compatible load balancer.

The 3,000-request planning target leaves 40% headroom below the measured ceiling for large encrypted payloads. The
100-Worker target comes from small-payload tests; the report does not establish capacity for 100 Workers with large
payloads. Load-test that combination before relying on it.

For Kubernetes installation and configuration, see
[Deploy to Kubernetes](/production-deployment/temporal-proxy/deploy-kubernetes).

## Compare reference throughput

The tests used one 4-CPU replica with a `12Gi` memory limit, `GOMEMLIMIT=11GiB`, and 32 upstream connections, running on
AWS Graviton arm64 nodes. Traffic used Go SDK Workers, random-byte payloads, mutual TLS on both connections, and a
Temporal Cloud upstream whose Namespace limits did not constrain throughput. Encryption tests enabled the decrypted
key cache. The test report does not identify the pre-release proxy version or commit.

| Workload | Measured throughput ceiling | CPU cores per 1,000 requests per second |
| --- | --- | --- |
| Small payloads, no encryption | About 12,900 requests per second | 0.29 to 0.31 |
| Small payloads, encryption | About 10,500 to 11,400 requests per second | 0.33 to 0.36 |
| Encrypted payloads averaging about 900 KB, up to 1.9 MB | About 5,000 requests per second | About 0.8 |

CPU use depends on both request rate and the payload bytes processed per second, especially with encryption enabled.
At the same request rate, larger payloads require more CPU. Compare requests-per-second figures only for similar
payload sizes and request mixes.

These tests were CPU-bound: beyond saturation, throughput fell. Payload size reduced throughput by more than half.
Request mix matters too: long polls and Activity result polls dominated the reference workload, so a mix with more
payload-carrying calls can have a lower ceiling. Compressible payloads can also increase CPU cost.

Target 65% to 75% of your measured ceiling during normal operation. Do not assume throughput scales linearly with CPU;
re-measure if you change replica size, hardware, or memory limits.

## Choose memory limits

Set `GOMEMLIMIT` to about 85% to 90% of the container memory limit as a starting point. The Go runtime does not derive
this setting from the container limit. Without it, heap growth during a reconnect burst can cause an out-of-memory kill.

| Workload | Provisional memory limit | `GOMEMLIMIT` |
| --- | --- | --- |
| Small payloads, few Workers per replica | `3Gi` | `2700MiB` |
| Small payloads, many Workers per replica | `6Gi` | `5400MiB` |
| Payloads near 1 MB | `12Gi` | `11GiB`, matching the test configuration |

These are starting points, not validated minimums. The throughput tests used `12Gi`; lower limits were not validated
with large Worker fleets. Under those test conditions, memory use ranged from about 1 GiB for small payloads with few
Workers to 5 to 7 GiB for the large-payload workload.

`GOMEMLIMIT` is a soft limit, not a cap on resident set size (RSS). Smaller limits increase garbage collection work and
can reduce throughput. Measure CPU and memory again at the limit you choose, including during Client reconnects.

For encryption, keep the decrypted data encryption key (DEK) cache enabled. A cache miss requires a KMS unwrap, so
cache capacity affects KMS load and request latency. Each replica uses its own DEKs per Namespace. Size the cache for
the keys actively being read across replicas and Namespaces, and monitor its hit and miss counters. See
[encryption settings](/production-deployment/temporal-proxy/encrypt-payloads).

## Size the connection pool

Starting with v0.9.0, `connection.maxConnections` defaults to 32 per upstream target, the pool size the reference tests
used. Each SDK long poll occupies an HTTP/2 stream. In the reference tests, pools of one to four connections exhausted
available streams and caused calls to stall or time out. Increasing replicas does not fix a connection pool that is too
small on each replica.

Change the pool size in your
[proxy configuration](/production-deployment/temporal-proxy/configure#tune-upstream-connections):

```yaml
upstreams:
  - name: cloud
    hostPort: '{{ .RemoteNamespace }}.tmprl.cloud:7233'
    connection:
      maxConnections: 32
```

This fragment sets the pool size; add your routing, Namespace translation, TLS, and credentials. A templated upstream
creates a pool for each resolved target, often one per Namespace. Account for targets multiplied by replicas when
estimating total connections.

## Calculate replicas

Use your peak request rate and measured per-replica ceiling. Reserve one extra replica for a failure or rollout:

```text
planned_requests_per_replica = measured_ceiling * target_utilization
replicas = ceil(peak_requests_per_second / planned_requests_per_replica) + 1
```

For example, at 30,000 requests per second, a 10,500-request ceiling, and a 65% target, this calculation gives six
replicas. This is a capacity estimate, not a measured fleet result.

If you need to tolerate losing a zone, spread replicas evenly across zones and verify that the surviving replicas
can handle peak traffic at no more than 75% of their measured ceiling. In the example above, six replicas across three
zones leave four survivors, each handling 7,500 requests per second, about 71% of the ceiling.

**Check load on each replica.** Long-lived Client connections do not redistribute automatically when you add a replica.
One test left two replicas carrying 99.6% and 0.4% of requests after a rollout. A few high-traffic Clients can overload
one replica while others remain idle. Provision for peak traffic rather than relying on autoscaling to relieve a hot
replica.

Adding proxy replicas does not increase [Temporal Cloud Namespace limits](/evaluate/cloud/limits).
Likewise, a request exceeding the proxy's 4 MiB receive limit needs smaller payloads, not more replicas.

## Measure your workload

Scrape the proxy's [Prometheus metrics](/production-deployment/temporal-proxy#observability). Scope these queries to the
same deployment; replace `job="temporal-proxy"` and the `pod` label with your scraper's labels.

Measure requests per second for each replica to identify uneven traffic:

```promql
sum by (pod) (rate(tmprl_proxy_server_requests_total{job="temporal-proxy"}[5m]))
```

Calculate CPU cores per 1,000 requests per second to compare your workload with the reference results:

```promql
sum(rate(process_cpu_seconds_total{job="temporal-proxy"}[5m]))
  / (sum(rate(tmprl_proxy_server_requests_total{job="temporal-proxy"}[5m])) / 1000)
```

Run the test with your production request mix, payload sizes, encryption, and resource limits. Increase concurrency
until throughput stops rising, recording CPU, RSS, request errors, and latency. Exclude long-poll methods when measuring
short-request latency, since long polls can wait about 60 seconds.

Finally, test the full fleet through pod loss, a rollout, and Client reconnects. Check recovery time and per-replica
load, not only aggregate throughput, before relying on the calculated capacity.

---

# Temporal Worker deployments

Source: https://docs.temporal.io/production-deployment/worker-deployments

> Programmatically scale and deploy your Temporal Workers into common production environments using our provided tooling and best practices.

A core feature of Temporal is that you are able to deploy your Workers to any infrastructure where your Workflow and Activity code will actually run.
This way, you have total control over your runtime environment, and can be responsive to any security or scaling needs that may arise over time, whether you are using Temporal Cloud or self-hosting a Temporal Service.

If you are just getting started, you want more guidance, or a refresher on Temporal concepts, our [Tutorials and Courses](https://learn.temporal.io/) help by using only one or two Temporal Workers to demonstrate core functionality.
Once you have an understanding of the core concepts, the content in this section will provide clarity on real-world deployments that grow far beyond those examples.

Our Worker Deployments guide provides documentation of Temporal product features that make it easier to scale and revise your Workflows.

[Worker Versioning](/production-deployment/worker-deployments/worker-versioning) is the recommended default for safely
deploying new Workflow code. It allows you to pin Workflows to individual versions of your workers, which are called
Worker Deployment Versions.

If your environment cannot yet support versioned worker deployments, you can fall back to patching Workflow code.
However, new production deployments should prefer Worker Versioning whenever possible.

You can optionally use the Temporal [Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to programmatically manage and scale your Worker deployments in Kubernetes pods.

This section also covers specific Worker Deployment examples:

- [**Serverless Workers**](/production-deployment/worker-deployments/serverless-workers)
  Deploy Serverless Workers on AWS Lambda, GCP Cloud Run, or Amazon Bedrock AgentCore Runtime.
  Temporal invokes your Worker when Tasks arrive, with no long-lived processes to manage.

- [**Deploy Workers to Amazon EKS**](/production-deployment/worker-deployments/deploy-workers-to-aws-eks)
  Containerize your Worker, publish it to Amazon Elastic Container Registry (ECR), and deploy it to Amazon Elastic Kubernetes Service (EKS) using the Temporal Python SDK.
  This guide covers the full deployment lifecycle and shows how to configure your Worker to connect to Temporal Cloud using Kubernetes-native tools like ConfigMaps and Secrets.
  Running Workers on EKS gives you fine-grained control over scaling, resource allocation, and availability—ideal for production systems that need reliability and flexibility in the cloud.

---

# Quick launch - Deploying your Workers on Amazon EKS

Source: https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks

> Deploy a Temporal Worker on Amazon Elastic Kubernetes Service (EKS) using the Python SDK.

Temporal Workers running in [Kubernetes](https://kubernetes.io)-based deployments deliver scale, resilience, and flexible resource management.
Amazon Elastic Kubernetes Service (EKS) offers one of the most popular choices for running Temporal Workers.
It integrates smoothly with AWS services and supports auto-scaling and fault tolerance.

Follow this guide to deploy and manage your Temporal Workers in EKS.
This guide walks you through writing Temporal Worker code, containerizing and publishing the Worker to the Amazon Elastic Container Registry (ECR), and deploying the worker to Amazon EKS.
The example on this page uses Temporal’s Python SDK and Temporal Cloud.

For production Kubernetes deployments that use [Worker Versioning](/production-deployment/worker-deployments/worker-versioning),
use the [Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) so deployment
rollouts and autoscaling stay attached to each Worker Deployment Version.

> **💡 Tip:**
>
> This guide applies to running Workers for both Temporal OSS and Temporal Cloud.
> However, there are some differences when working with Temporal OSS.
> For example, you'll need to use mTLS certificates instead of API keys.
> You must modify your Kubernetes deployments to handle and mount the TLS certificates for your use case.
> The specifics will vary depending on your deployment.
>

## Before you begin

To get started deploying your Workers to EKS, you’ll need:

- Your Temporal Cloud account, including:
  - A Namespace using [API key authentication](/cloud/api-keys#namespace-authentication)
  - Your API Key for a [Service Account](/cloud/api-keys#generate-an-api-key-for-a-service-account)
- An AWS account, including:
  - A deployed EKS cluster within your AWS Account
- An installed version of the [`aws` CLI](https://aws.amazon.com/cli/)
- [`docker`](https://www.docker.com/get-started/)
- The [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) command line tool, configured with your deployed EKS cluster

## Write your Worker code

In Temporal applications, business logic lives within your main Workflow code.
Your Worker code runs separately, and is responsible for executing your Workflows and Activities.
Make sure to configure your Worker to use environment variables so you can dynamically route your Worker to different Temporal Instances, Namespaces, and Task Queues on the fly:

```python
TEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")
TEMPORAL_NAMESPACE = os.environ.get("TEMPORAL_NAMESPACE", "default")
TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue")
TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "")
```

After configuration, instantiate your Temporal client:

```python
client = await Client.connect(
    TEMPORAL_ADDRESS,
    namespace=TEMPORAL_NAMESPACE,
    rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE},
    api_key=TEMPORAL_API_KEY,
    tls=True
)
```

Here is a complete Python boilerplate that showcases how to instantiate a Client and pass it to the Worker before starting the Worker execution:

```python
import asyncio
import os

from temporalio.worker import Worker
from temporalio.client import Client

from workflows import your_workflow
from activities import your_first_activity, your_second_activity, your_third_activity

TEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")
TEMPORAL_NAMESPACE = os.environ.get("TEMPORAL_NAMESPACE", "default")
TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue")
TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "your-api-key")

async def main():
  client = await Client.connect(
    TEMPORAL_ADDRESS,
    namespace=TEMPORAL_NAMESPACE,
    rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE},
    api_key=TEMPORAL_API_KEY,
    tls=True
  )

  print("Initializing worker...")

  # Run the worker
  worker = Worker(
    client,
    task_queue=TEMPORAL_TASK_QUEUE,
    workflows=[your_workflow],
    activities=[
      your_first_activity,
      your_second_activity,
      your_third_activity
    ]
  )

  print("Starting worker... Waiting for tasks.")
  await worker.run()

if __name__ == "__main__":
  asyncio.run(main())
```

## Containerize the Worker for Kubernetes

You need to containerize your Worker code to run it with Kubernetes.
Here is a sample Python Dockerfile, complete with the Temporal Python SDK installed:

```docker
# Use Python 3.11 slim image as base
FROM python:3.11-slim

# Set working directory
WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    && rm -rf /var/lib/apt/lists/*

# Install the Temporal Python SDK dependency
RUN pip install --no-cache-dir temporalio

# Copy application code
COPY . .

# Set Python to run in unbuffered mode
ENV PYTHONUNBUFFERED=1

# Run the worker
CMD ["python", "worker.py"]
```

Build the Docker image and target the `linux/amd64` architecture:

```bash
docker buildx build \
    --platform linux/amd64 \
    -t your-app .
```

## Publish the Worker image to Amazon ECR

After building the Docker image, you’re ready to publish it to Amazon ECR.
Make sure that you’re authenticated with AWS, and that you’ve set your `AWS_REGION` and `AWS_ACCOUNT_ID` environment variables:

```bash
export AWS_ACCOUNT_ID=<your_aws_account_id>
export AWS_REGION=<your_aws_region>
```

Create an ECR repository and authenticate ECR with the Docker container client:

```bash
aws ecr create-repository \
    --repository-name your-app
aws ecr get-login-password --region $AWS_REGION | \
    docker login --username AWS --password-stdin \
            $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
```

After authenticating Docker with ECR, tag your container and publish it:

```bash
docker tag your-app $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest
```

## Deploy the Workers to EKS

With your Worker containerized, you’re ready to deploy it to EKS. Create a namespace in your EKS cluster. You’ll use the namespace to run your Temporal Workers:

```bash
kubectl create namespace example-namespace
```

Create a `ConfigMap` to hold non-sensitive values that Kubernetes will inject into your Worker deployment.
These enable dynamic routing for instances, Namespaces, and Task Queues.
To set these values, build a `config-map.yaml` file like the following example:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: temporal-worker-config
  namespace: example-namespace
data:
  TEMPORAL_ADDRESS: "<your-temporal-address>"
  TEMPORAL_NAMESPACE: "<your-temporal-cloud-namespace>"
  TEMPORAL_TASK_QUEUE: "<your-task-queue>"
```

Apply the `ConfigMap` to your namespace:

```bash
kubectl apply -f config-map.yaml \
    --namespace example-namespace
```

For sensitive values, use Kubernetes Secrets.
Create a secret to hold your Temporal API key:

```bash
kubectl create secret generic temporal-secret \
    --from-literal=TEMPORAL_API_KEY=$TEMPORAL_API_KEY \
    --namespace example-namespace
```

With your configuration in place, you can deploy the Worker.
Create a `deployment.yaml` file to configure your Worker image, resources, and secret values.
You can tune the resources and optional configurations you specify to match your deployment needs so they match your production workloads.
Note that the spun-up container reads your Temporal API key from the Kubernetes secret you just created:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
   name: your-app
   namespace: example-namespace
   labels:
      app: your-app
spec:
   selector:
      matchLabels:
         app: your-app
   replicas: 2
   template:
      metadata:
         labels:
            app: your-app
      spec:
         serviceAccountName: your-app
         containers:
            - name: your-app
              image: <your-ecr-image-name>
              env:
                - name: TEMPORAL_ADDRESS
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_ADDRESS
                - name: TEMPORAL_NAMESPACE
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_NAMESPACE
                - name: TEMPORAL_TASK_QUEUE
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_TASK_QUEUE
                - name: TEMPORAL_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: temporal-secret
                      key: TEMPORAL_API_KEY
              resources:
                limits:
                  cpu: "0.5"
                  memory: "512Mi"
                requests:
                  cpu: "0.2"
                  memory: "256Mi"
```

Apply the `deployment.yaml` file to the EKS cluster:

```bash
kubectl apply -f deployment.yaml \
    --namespace example-namespace
```

## Verify that the Workers are Connected

After deploying your Workers to EKS, confirm that they have connected to Temporal Cloud.
Retrieve the pod listing for the Kubernetes/EKS namespace that you created:

```
kubectl get pods -n example-namespace
```

After listing the pods, access the Worker logs to confirm you’re properly connected to Temporal Cloud:

```
kubectl logs <pod-name> -n example-namespace
```

You confirm connection when you see:

```
Initializing worker...
Starting worker... Waiting for tasks.
```

You have now successfully deployed your Temporal Worker to EKS.

---

# Temporal Worker Controller

Source: https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller

> Install the Temporal Worker Controller and configure rollouts, version cleanup, and per-version autoscaling for versioned Workers on Kubernetes.

The [Temporal Worker Controller](https://github.com/temporalio/temporal-worker-controller) is a Kubernetes controller that automates rainbow deployments of your Workers.
It registers new Worker Deployment Versions, creates a Kubernetes Deployment for each version, updates the routing configuration through Temporal APIs, and removes the resources for drained versions.
The Worker Controller is Generally Available, and its APIs are stable.

If you run versioned Workers on Kubernetes, the Worker Controller is the recommended way to manage rollouts and autoscaling together.
You don't need the Worker Controller to use [Worker Versioning](/production-deployment/worker-deployments/worker-versioning), and it integrates with the [Temporal CLI](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin#rolling-out-changes-with-the-cli).

This page uses the following terms:

- [**Worker Deployment**](/worker-versioning#deployments): A logical service that groups similar Workers together for unified management. Each Worker Deployment has a name, such as your service name, and a series of Worker Deployment Versions.
- [**Worker Deployment Version**](/worker-versioning#deployment-versions): An iteration of a Worker Deployment. Each version consists of Workers that share the same code build and environment. When a Worker starts polling for Workflow and Activity Tasks, it reports its Worker Deployment Version to the Temporal Service.
- **Deployment**: A Kubernetes Deployment resource. A Deployment is "versioned" if it runs Workers that report a Worker Deployment Version.

## Features

- Registration of new Worker Deployment Versions
- Creation of versioned Deployment resources that manage the Pods running your Workers
- Deletion of resources associated with drained Worker Deployment Versions
- `Manual`, `AllAtOnce`, and `Progressive` rollouts of new versions
- Automatic rollback when you set the target version back to a recent previous build
- A "gate" Workflow that must succeed on the new version before the Worker Controller routes real traffic to it
- Autoscaling of versioned Deployments using Kubernetes HPA or KEDA

## Autoscaling versioned Workers

The Worker Controller can manage autoscaling for versioned Worker Deployments without forcing you to choose between
safe rollout behavior and elastic capacity.

Use the Worker Controller when you need all of the following:

- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) for safe Workflow code changes
- Kubernetes-native rollout automation
- autoscaling that follows each active Worker Deployment Version separately

The Worker Controller provides Kubernetes-native autoscaling through either HPA or KEDA, so you can scale on any
metric available to your scaling pipeline, including:

- CPU and memory utilization
- Task Queue backlog metrics exposed through your metrics pipeline
- slot utilization and other Worker-specific metrics
- custom metrics surfaced through Prometheus or another Kubernetes metrics adapter

### WorkerResourceTemplate

To attach autoscaling or other Kubernetes resources to each Worker Deployment Version, use a
`WorkerResourceTemplate` (WRT).

A WRT lets you define a resource template once and have the Worker Controller create a version-specific copy for each
active Worker Deployment Version. This is useful for resources such as:

- `HorizontalPodAutoscaler` (HPA)
- `ScaledObject` (KEDA)
- `PodDisruptionBudget`
- other Kubernetes resources that should track the lifecycle of a versioned Deployment

The Worker Controller manages these resources alongside the versioned Deployments it creates, so they are updated and
cleaned up as versions roll forward and drain.

By default, the Worker Controller accepts only `HorizontalPodAutoscaler` resources in a `WorkerResourceTemplate`.
To use other kinds, such as `PodDisruptionBudget` or a KEDA `ScaledObject`, add them to the `workerResourceTemplate.allowedResources` Helm value.
The webhook also checks that you have permission to create the embedded resource yourself, so a user who creates a WRT needs the matching RBAC permissions in that namespace.
For details, see the [WorkerResourceTemplate reference](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/worker-resource-templates.md#allowed-resource-kinds-and-rbac).

### Choosing a scaling strategy

The Worker Controller supports two autoscaling strategies, each attached per Worker Deployment Version through a
`WorkerResourceTemplate`: Kubernetes HPA (with the Prometheus Adapter) and KEDA.

HPA with the Prometheus Adapter is the recommended default for most deployments. It scales independently of the number
of namespaces or Task Queues and handles thousands of Task Queues efficiently. KEDA is a better fit when you need to
scale from zero, have long idle periods, or require sub-minute reactivity, though it is subject to per-namespace
Temporal API rate limits.

For a full comparison and a decision matrix, see
[Scaling recommendations](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/scaling-recommendations.md)
in the Worker Controller repository.

## Configuring Worker lifecycles

Tag your Workers following the guidance for using [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
You then describe each Worker Deployment as a `WorkerDeployment` resource.
The controller creates one Kubernetes Deployment for each build.

The following example runs a Worker with a progressive rollout that is gated on the success of the `HelloWorld` Workflow:

```yaml
apiVersion: temporal.io/v1alpha1
kind: WorkerDeployment
metadata:
  name: my-worker
spec:
  workerOptions:
    connectionRef:
      name: production-temporal
    temporalNamespace: production
  deployment:
    replicas: 3
    template:
      spec:
        containers:
          - name: worker
            image: my-worker:v2.0.0
  rollout:
    strategy: Progressive
    steps:
      - rampPercentage: 1
        pauseDuration: 30s
      - rampPercentage: 10
        pauseDuration: 1m
    gate:
      workflowType: "HelloWorld"
  sunset:
    scaledownDelay: 1h
    deleteDelay: 24h
```

The `connectionRef` points to a `Connection` resource that holds the Temporal Service address and either mTLS or API key credentials.
A `Connection` uses either mTLS or an API key, not both.
For Temporal Cloud, use the Namespace endpoint (`<namespace>.<account>.tmprl.cloud:7233`) as the `hostPort`.
If a Namespace that also allows mTLS rejects your API key connection with a `tls: certificate required` error, switch `hostPort` to the Namespace's [Regional Endpoint](/cloud/connectivity), such as `us-east-1.aws.api.temporal.io:7233`.
For examples, see the [Configuration reference](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/configuration.md#connection-configuration).

When you ship a new image, the Worker Controller detects the new version and gradually makes it the **Current Version** of the Worker Deployment.
Pinned Workflows stay on the version they started on.
When older versions drain, the Worker Controller scales down their Deployments after `scaledownDelay` and deletes them after `deleteDelay`.

When you use autoscaling with the Worker Controller, each active Worker Deployment Version can scale independently while
it is serving traffic. This allows older versions to drain safely while newer versions scale based on live demand.

### Roll back a version

To roll back, set the Worker image back to a previous build.
If that build was the Current Version within the last hour, the Worker Controller routes 100% of traffic to it at once, regardless of the configured rollout strategy.
Rollback doesn't apply to the `Manual` strategy, which leaves routing entirely to you.

## Install the Worker Controller

### Prerequisites

- Kubernetes 1.19 or later
- Helm 3.0 or later
- Temporal Cloud, or a self-hosted Temporal Service v1.29.1 or later
- TLS for the validating webhook. The `WorkerResourceTemplate` webhook is always on, so the controller Pod always needs a certificate. The recommended option is [cert-manager](https://cert-manager.io/docs/installation/), installed before the controller. To manage certificates yourself, see [Webhook TLS](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/worker-resource-templates.md#webhook-tls).

### Install the charts

The Worker Controller ships as two Helm charts so you can upgrade the CRDs separately from the controller.
Install the CRDs chart first.
For the available chart versions, see the [Worker Controller releases](https://github.com/temporalio/temporal-worker-controller/releases).

```bash
VERSION=<chart-version>
NAMESPACE=temporal-system

helm install temporal-worker-controller-crds \
  oci://docker.io/temporalio/temporal-worker-controller-crds \
  --version $VERSION \
  --namespace $NAMESPACE \
  --create-namespace

helm install temporal-worker-controller \
  oci://docker.io/temporalio/temporal-worker-controller \
  --version $VERSION \
  --namespace $NAMESPACE
```

For other deployment templates, see the [Helm chart templates](https://github.com/temporalio/temporal-worker-controller/tree/main/helm/temporal-worker-controller/templates) on GitHub.

### Migrate from the previous CRD names

In Worker Controller v1.7.0, the Worker Controller renamed its CRDs.
The controller no longer reconciles resources of the old kinds, and you can't create new ones.

| Old name                                                  | New name                                      |
| --------------------------------------------------------- | --------------------------------------------- |
| `TemporalWorkerDeployment`                                | `WorkerDeployment`                            |
| `TemporalConnection`                                      | `Connection`                                  |
| `WorkerResourceTemplate.spec.temporalWorkerDeploymentRef` | `WorkerResourceTemplate.spec.workerDeploymentRef` |

Keep the resource names the same, because the Worker Deployment name derives from the namespace and resource name.
Don't roll back the CRDs chart after you migrate, because that can delete your Worker Deployments.
For the upgrade steps, see the [CRD rename migration guide](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/migration-crd-rename.md).

## Learn more

The [Worker Controller repository](https://github.com/temporalio/temporal-worker-controller/tree/main/docs) maintains the detailed guides:

- [Migrate from unversioned Workers](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/migration-to-versioned.md)
- [CD rollouts with Helm, kubectl, Argo CD, and Flux](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/cd-rollouts.md)
- [Configuration reference](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/configuration.md)
- [Upgrade the Worker Controller](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/upgrade.md)
- [Architecture](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/architecture.md)
- [Limits](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/limits.md), including name length constraints

---

# Recover pinned Workflows after a bad rollout

Source: https://docs.temporal.io/production-deployment/worker-deployments/recover-pinned-workflows

> Roll back, identify, and recover pinned Workflows affected by a faulty Worker Deployment Version using Versioning Override and Reset-with-Move.

This runbook covers how to recover pinned Workflows after rolling out a Worker Deployment Version that turned out to be faulty.
Use it when a new code version has caused pinned Workflows to fail, time out, or get stuck retrying Workflow Tasks.

This page assumes you have already configured [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) and that the affected Workflows are pinned to a specific Worker Deployment Version.

> **💡 Tip:**
> Prerequisites
>
> - Worker Versioning is enabled and the affected Workflows are pinned.
> - Your Worker fleet uses [blue-green or rainbow deployments](/production-deployment/worker-deployments/worker-versioning#deployment-systems), not rolling upgrades.
> - You can run the `temporal` CLI against the affected Namespace.
>

## Stop the rollout

Stop sending new Workflows to the faulty version before you do anything else.

If the bad Version is currently ramping, set the ramp percentage to zero:

```bash
temporal worker deployment set-ramping-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBadBuildID" \
    --percentage 0
```

If the bad Version has already become the Current Version, switch the Current Version back to the previous good Version:

```bash
temporal worker deployment set-current-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourPreviousBuildID"
```

After either change, new Workflows stop landing on the bad Version. Existing pinned Workflows still execute on the bad Version until you recover them.

## Identify affected Workflows

Use Search Attributes to find Workflows running on or affected by the bad Version.

Useful filters:

- `ExecutionStatus` — for example, `Running`, `Failed`, or `TimedOut`.
- `TemporalWorkerDeploymentVersion` — formatted as `'YourDeploymentName:YourBuildID'`.
- `TemporalReportedProblems` — accepts values like `category=WorkflowTaskFailed` or `category=WorkflowTaskTimedOut`. See [Detecting Workflow Task Failures](/encyclopedia/detecting-workflow-failures#detecting-workflow-task-failures).
- `WorkflowType` — for example, `'OrderProcessing'`.

Use `temporal workflow count` to quickly check how many Workflows match a query. For Workflows that are still retrying tasks after the upgrade:

```bash
temporal workflow count \
  --query "TemporalWorkerDeploymentVersion='YourDeploymentName:YourBadBuildID' \
    AND ExecutionStatus='Running' \
    AND TemporalReportedProblems IN ('category=WorkflowTaskFailed', 'category=WorkflowTaskTimedOut')"
```

For closed Workflows that failed:

```bash
temporal workflow count \
  --query "TemporalWorkerDeploymentVersion='YourDeploymentName:YourBadBuildID' \
    AND (ExecutionStatus='Failed' OR ExecutionStatus='TimedOut')"
```

To get the Workflow Id and Run Id of matching executions, use `temporal workflow list` with JSON output and extract the relevant fields with [`jq`](https://jqlang.org/):

```bash
temporal workflow list --output json \
  --query "TemporalWorkerDeploymentVersion='YourDeploymentName:YourBadBuildID' \
    AND (ExecutionStatus='Failed' OR ExecutionStatus='TimedOut')" \
  | jq '.[].execution'
```

Example output:

```json
{
  "workflowId": "worker-versioning-pinned-2_032f7b06-f3a0-47a7-a7c2-949fcce7fc42",
  "runId": "019e9a92-1d8e-7a43-a345-721351d2d544"
}
{
  "workflowId": "worker-versioning-pinned-2_99e7c4ac-74cd-48c5-ae2e-94aa3c67c36f",
  "runId": "019e9a91-e8e3-765b-aba8-3a7002ec7d6c"
}
```

## Choose a recovery strategy

The right recovery strategy depends on three questions about each affected Workflow:

1. **Is the Workflow closed, or are its tasks still retrying?**
2. **Can the Workflow safely re-execute from the start of its current run?** Workflows that can are called *restartable* in this runbook. Whether a Workflow is restartable is a property of the Workflow design and must be documented or annotated (for example, via a Custom Search Attribute) by the team that owns it.
3. **Has the Workflow's internal state been corrupted?** Detecting state corruption is difficult to scale. In practice, most teams filter by Workflow Type and make conservative assumptions for an entire batch rather than per-instance.

The answers map to recovery strategies as follows:

| Workflow state | Restartable? | Strategy |
|---|---|---|
| Running, tasks retrying, state intact | Yes | [Reset-with-Move](#recover-workflows) to `FirstWorkflowTask` on the previous good Version. |
| Running, tasks retrying, state intact | No | [Versioning Override](#recover-workflows) to a new replay-safe Version. |
| Running, recently corrupted state | No | [Reset-with-Move](#recover-workflows) to `LastWorkflowTask` on a new replay-safe Version. |
| Closed (Failed, Completed, TimedOut) | Either | [Reset-with-Move](#recover-workflows) to `FirstWorkflowTask`. Critical state may need out-of-band compensation. |
| Stateless or simple replacement is acceptable | Either | Terminate (if still running) and start new Workflows with the original arguments and the new Version. |

For Workflows still retrying without state corruption, you may need to use the [Patching APIs](/patching) to make a new Version replay-safe before pointing Workflows at it.

## Recover Workflows

Temporal exposes two recovery primitives, both available through the CLI or directly through the Worker Versioning APIs (see [Moving a pinned Workflow](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin#moving-a-pinned-workflow)):

- **Versioning Override** — forces the next retried Workflow Task to execute on a different pinned Version. Use [`temporal workflow update-options`](/cli/command-reference/workflow#update-options).
- **Reset-with-Move** — atomically resets a Workflow's Event History and applies a Versioning Override. Use [`temporal workflow reset with-workflow-update-options`](/cli/command-reference/workflow#with-workflow-update-options).

Both commands accept a `--query` argument for batch operations.

### Reset restartable Workflows to the previous Version

Schedule a batch Reset-with-Move targeting the start of execution on the previous good Version. Use `--reapply-exclude All` to skip re-applying signals and Updates, which is typically the right choice for a clean restart:

```bash
temporal workflow reset with-workflow-update-options \
    --query "TemporalWorkerDeploymentVersion='YourDeploymentName:YourBadBuildID' \
        AND ExecutionStatus='Running' \
        AND WorkflowType='YourWorkflowType' \
        AND TemporalReportedProblems IN ('category=WorkflowTaskFailed', 'category=WorkflowTaskTimedOut')" \
    --reason "Reset restartable Workflow to YourPreviousBuildID" \
    --versioning-override-behavior pinned \
    --versioning-override-build-id "YourPreviousBuildID" \
    --versioning-override-deployment-name "YourDeploymentName" \
    --reapply-exclude All \
    --type FirstWorkflowTask \
    --output json --yes
```

### Move running Workflows to a replay-safe Version

For Workflows whose tasks are still retrying and whose state is intact, apply a Versioning Override to a new replay-safe Version. No Reset is needed:

```bash
temporal workflow update-options \
    --query "TemporalWorkerDeploymentVersion='YourDeploymentName:YourBadBuildID' \
        AND ExecutionStatus='Running' \
        AND WorkflowType='YourWorkflowType' \
        AND TemporalReportedProblems IN ('category=WorkflowTaskFailed', 'category=WorkflowTaskTimedOut')" \
    --versioning-override-behavior pinned \
    --versioning-override-build-id "YourGoodBuildID" \
    --versioning-override-deployment-name "YourDeploymentName" \
    --output json --yes
```

### Roll back recently corrupted Workflows

When a Workflow's state was corrupted recently but tasks are still retrying, you can sometimes recover by resetting to `LastWorkflowTask` on a replay-safe Version. This re-applies pending signals and Updates:

```bash
temporal workflow reset with-workflow-update-options \
    --query "TemporalWorkerDeploymentVersion='YourDeploymentName:YourBadBuildID' \
        AND ExecutionStatus='Running' \
        AND WorkflowType='YourWorkflowType' \
        AND TemporalReportedProblems IN ('category=WorkflowTaskFailed', 'category=WorkflowTaskTimedOut')" \
    --reason "Reset corrupted Workflow to YourGoodBuildID" \
    --versioning-override-behavior pinned \
    --versioning-override-build-id "YourGoodBuildID" \
    --versioning-override-deployment-name "YourDeploymentName" \
    --type LastWorkflowTask \
    --output json --yes
```

### Recover closed Workflows

Closed Workflows (Failed, Completed, TimedOut) need Reset-with-Move. Choose `ExecutionStatus` values that match the failure mode:

```bash
temporal workflow reset with-workflow-update-options \
    --query "TemporalWorkerDeploymentVersion='YourDeploymentName:YourBadBuildID' \
        AND (ExecutionStatus='Completed' OR ExecutionStatus='Failed') \
        AND WorkflowType='YourWorkflowType'" \
    --reason "Reset closed Workflow to YourGoodBuildID" \
    --versioning-override-behavior pinned \
    --versioning-override-build-id "YourGoodBuildID" \
    --versioning-override-deployment-name "YourDeploymentName" \
    --reapply-exclude All \
    --type FirstWorkflowTask \
    --output json --yes
```

> **⚠️ Warning:**
> Not idempotent
>
> Resetting a closed Workflow does not change the status of the prior closed execution. Re-running the same command will reset the same closed Workflows again, terminating each previous reset attempt and starting another new run.
> Plan to run this command exactly once per affected batch, after the bad Version has fully [drained](#handle-eventual-consistency).
>

The earlier batch commands targeting Running Workflows are idempotent because they filter on `TemporalWorkerDeploymentVersion` and `ExecutionStatus='Running'`. Once a Workflow is moved off the bad Version, it stops matching the query.

## Handle eventual consistency

The Visibility store is eventually consistent, which means a query that identifies affected Workflows may not return all of them in a single execution.

Use the drainage status of the bad Version as a signal that the Visibility index has caught up.
A Version is **drained** when no new Workflows are expected on it and all existing pinned Workflows on it are closed.

Check drainage status:

```bash
temporal worker deployment describe-version \
  --deployment-name "YourDeploymentName" \
  --build-id "YourBadBuildID" \
  --output json \
  | jq .drainageInfo.drainageStatus
```

Recommended approach:

1. Repeat the idempotent recovery commands on `Running` Workflows until the drainage status reports `drained`. The Temporal Service refreshes drainage status periodically, so it may take a few minutes after the last running Workflow closes.
2. Once the Version is drained, run the non-idempotent Reset-with-Move command against closed Workflows once.

See [Sunsetting an old Deployment Version](/production-deployment/worker-deployments/worker-versioning/sunset-and-gc#sunsetting-an-old-deployment-version) for more on drainage states.

## Clean up the drained Version

After the bad Version has drained and all recovered closed Workflows have been processed, stop the Workers on the bad Version and delete the Version:

```bash
temporal worker deployment delete-version \
  --deployment-name "YourDeploymentName" \
  --build-id "YourBadBuildID"
```

See [`temporal worker deployment delete-version`](/cli/command-reference/worker#delete-version) for prerequisites on deletion (the Version must not be Current, Ramping, or have active pollers, and it must be drained unless you pass `--skip-drainage`).

## Summary

Recovering pinned Workflows from a faulty Worker Deployment Version takes the following steps:

1. **Stop the rollout** by ramping to zero or reverting the Current Version.
2. **Identify** affected Workflows with `TemporalWorkerDeploymentVersion` and `TemporalReportedProblems` queries.
3. **Choose a strategy** based on execution status, restartability, and state integrity.
4. **Recover** using Versioning Override or Reset-with-Move, idempotently while the Version drains.
5. **Clean up** by deleting the drained Version once all affected Workflows are recovered.

---

# Serverless Workers

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview. Amazon Bedrock AgentCore Runtime support is in
> Pre-release. To request AgentCore Runtime access on Temporal Cloud, create a
> [support ticket](/evaluate/cloud/support#support-ticket) or contact your account team. You can also
> [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or try them through self-hosting.

Serverless Workers let you run Temporal Workers on serverless compute. Deploy your Worker code to a serverless provider,
configure a compute provider for the Worker Deployment Version, and Temporal starts and stops Workers in response to
work on the Task Queue. There is no always-on Worker fleet to provision or scale.

Temporal monitors Task Queues that have a compute provider configured. When a Task arrives and no Worker is free to take
it, the [Worker Controller Instance (WCI)](/serverless-workers#how-invocation-works) starts compute. How it starts
compute is where the providers differ. On AWS Lambda and AgentCore Runtime, the WCI invokes compute in response to
unmet Task Queue demand. On GCP Cloud Run, it resizes a Worker Pool of long-lived instances that poll continuously.

> **⚠️ Caution:**
> Multi-cloud Replication is not supported
>
> Serverless Workers do not support [Multi-cloud Replication](/cloud/high-availability#multi-cloud-replication).
> They must be invoked from the same cloud provider where the Namespace is currently active. After a failover to a
> different cloud provider, Serverless Workers can no longer be invoked. This is a temporary restriction during Public Preview.
>

## Supported providers

- [**AWS Lambda**](/production-deployment/worker-deployments/serverless-workers/aws-lambda) - Deploy a Serverless Worker
  as a Lambda function. Temporal assumes an IAM role in your AWS account to invoke the function when Tasks arrive.
- [**GCP Cloud Run**](/production-deployment/worker-deployments/serverless-workers/cloud-run) - Deploy a Serverless
  Worker to a Cloud Run Worker Pool. Temporal impersonates a service account in your GCP project to scale the pool as
  Tasks arrive and drain.
- [**Amazon Bedrock AgentCore Runtime**](/production-deployment/worker-deployments/serverless-workers/agentcore) -
  Deploy a Serverless Worker to AgentCore Runtime. Temporal assumes an IAM role in your AWS account to invoke the
  Runtime endpoint as Tasks arrive.

## Try it locally 

To evaluate Serverless Workers on AWS Lambda without an AWS account, you can run the entire loop on your machine with
[LocalStack](https://www.localstack.cloud/) and the Temporal development server. See
[Test a Serverless Worker locally with LocalStack](/production-deployment/worker-deployments/serverless-workers/aws-lambda/local-development).

---

# Deploy a Serverless Worker on Amazon Bedrock AgentCore Runtime

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/agentcore

> Add a Python Worker to an existing AgentCore application and configure Temporal to start capacity when Task Queue demand increases.

> **Pre-release**
> Amazon Bedrock AgentCore Runtime support is in Pre-release, and its APIs may change in backwards-incompatible ways.
> To request Temporal Cloud access, create a [support ticket](/evaluate/cloud/support#support-ticket) or contact your
> account team. You can also [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or
> [try it through self-hosting](/production-deployment/worker-deployments/serverless-workers/agentcore/self-hosted-setup).

This page shows how to add a Python [Serverless Worker](/serverless-workers) to an existing Amazon Bedrock AgentCore
application, deploy it to AgentCore Runtime, and connect it to a Worker Deployment Version. It assumes that your
Workflow and Activity code and AgentCore project are already in place.

For a tutorial that walks you through setting up an AgentCore project from scratch, see [Build a durable agent on
Amazon Bedrock AgentCore](/guides/durable-agent-on-agentcore). That guide starts with the [Python Strands AgentCore
sample](https://github.com/temporalio/samples-python/tree/main/bedrock_agentcore/strands_agent) and explains
the agent architecture, Workflow and Activity boundaries, AgentCore project configuration, and deployment from start to
finish. Use this page when you only need the Worker deployment procedure.

For details about the Worker implementation and lifecycle, see [Serverless Workers on Amazon Bedrock AgentCore Runtime -
Python SDK](/develop/python/workers/serverless-workers/agentcore).

## Prerequisites 

- A Temporal Cloud account with an AWS-hosted Namespace and access to the AgentCore Serverless Workers Pre-release. For
  a self-hosted Temporal Service v1.32.0 or later, complete the
  [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/agentcore/self-hosted-setup) first.
- [Temporal CLI v1.8.3](https://github.com/temporalio/cli/releases/tag/v1.8.3) or later, configured for your Namespace.
- An existing AgentCore project with `agentcore/agentcore.json`, `agentcore/aws-targets.json`, and a generated AgentCore
  CDK project.
- An AWS account in an [AgentCore-supported Region](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-regions.html),
  with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured for that
  account and permission to create AgentCore resources, CloudFormation stacks, and IAM roles. See
  [IAM permissions for AgentCore Runtime](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-permissions.html).
- Node.js 20 or later, the [AgentCore CLI](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-get-started-cli.html),
  and the [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting-started.html) installed. Bootstrap the CDK in the
  target account and Region.

## 1. Configure the Worker Runtime 

`agentcore/agentcore.json` is the [AgentCore CLI project
configuration](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-get-started-cli.html). Its
`runtimes` array defines the AgentCore Runtime resources that the CLI deploys.

The `entrypoint` field names the Python file that AgentCore starts. That file must implement the [AgentCore Runtime HTTP
protocol contract](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-http-protocol-contract.html) by
serving the Runtime's `/invocations` and `/ping` endpoints. For a Serverless Worker, `/invocations` starts the Temporal
Worker and acknowledges the request. For an implementation example, see the [Python Runtime entry
point](/develop/python/workers/serverless-workers/agentcore#runtime-handler).

The following fragment from the Python Strands AgentCore sample configures that entry point, a public network, and the
named endpoint that Temporal invokes:

```json
{
  "name": "temporal_strands_worker",
  "build": "CodeZip",
  "entrypoint": "agentcore_worker.py",
  "codeLocation": ".",
  "runtimeVersion": "PYTHON_3_12",
  "networkMode": "PUBLIC",
  "protocol": "HTTP",
  "authorizerType": "AWS_IAM",
  "endpoints": {
    "temporal": {
      "version": 1,
      "description": "Invoked by Temporal Cloud Serverless Workers"
    }
  }
}
```

In this initial configuration, `version: 1` selects the first AgentCore Runtime version.

Within the same Runtime object, add the Temporal connection, Task Queue, Worker Deployment name, and Build ID to the
`envVars` array:

```json
{
  "envVars": [
    {
      "name": "TEMPORAL_ADDRESS",
      "value": "<NAMESPACE>.<ACCOUNT>.tmprl.cloud:7233"
    },
    {
      "name": "TEMPORAL_NAMESPACE",
      "value": "<NAMESPACE>.<ACCOUNT>"
    },
    {
      "name": "TEMPORAL_API_KEY",
      "value": "<TEMPORAL_API_KEY>"
    },
    {
      "name": "TEMPORAL_TASK_QUEUE",
      "value": "<TASK_QUEUE>"
    },
    {
      "name": "TEMPORAL_DEPLOYMENT_NAME",
      "value": "<DEPLOYMENT_NAME>"
    },
    {
      "name": "TEMPORAL_BUILD_ID",
      "value": "<BUILD_ID>"
    }
  ]
}
```

Replace `<NAMESPACE>.<ACCOUNT>` with your Temporal Cloud Namespace ID, `<TEMPORAL_API_KEY>` with its API key, and
`<TASK_QUEUE>` with the Task Queue used by your application. Choose `<DEPLOYMENT_NAME>` and `<BUILD_ID>` for this Worker
version.

Do not commit a populated Temporal Cloud API key. For a production deployment, store it in AWS Secrets Manager, grant
the Runtime execution role permission to read it, and load it in the Runtime entry point. The Runtime execution role is
separate from the invocation role that Temporal assumes.

## 2. Add the Worker code 

For a `CodeZip` Runtime, AgentCore packages the directory identified by `codeLocation`. The `entrypoint` path is
relative to that directory. The directory must contain the entry point, every local module that it imports, and a
`pyproject.toml` file that declares the Runtime dependencies.

The sample sets `codeLocation` to `.` and uses the following layout:

```text
project-root/
├── agentcore/
│   ├── agentcore.json
│   ├── aws-targets.json
│   └── cdk/
├── agentcore_worker.py
├── workflows.py
├── activities.py
└── pyproject.toml
```

With this layout, set `entrypoint` to `agentcore_worker.py`. If your AgentCore application keeps code in a directory
such as `app/MyAgent`, set `codeLocation` to that directory and put the entry point, imported modules, and
`pyproject.toml` there.

Declare `temporalio`, `bedrock-agentcore`, and your application dependencies in `pyproject.toml`. The entry point must:

- Connect a Temporal Client and create a standard long-running Worker with your Workflows and Activities.
- Configure the Worker with the deployment name and Build ID from the Runtime environment.
- Use `BedrockAgentCoreApp` to implement the AgentCore Runtime HTTP endpoints.
- Start the Worker as an AgentCore asynchronous task and acknowledge the invocation without waiting for the Worker to
  stop.
- Stop polling and drain the Worker when its idle policy decides to release the Runtime.

The following excerpt from the Python sample implements this structure. It uses the `ActivityTracker`, `DEBOUNCE`, and
`DRAIN` values defined in the same source file to retire the Worker after an idle period. The complete linked source
file also contains the imports, creates the `BedrockAgentCoreApp`, retains the background task in `_worker`, and calls
`app.run()` when the entry point starts. For the idle-policy code and an explanation of each part, see
[Start the Worker from the Runtime handler](/develop/python/workers/serverless-workers/agentcore#runtime-handler).

<!--SNIPSTART python-agentcore-runtime-handler-->
[bedrock_agentcore/strands_agent/agentcore_worker.py](https://github.com/temporalio/samples-python/blob/main/bedrock_agentcore/strands_agent/agentcore_worker.py)
```py
async def run_worker() -> None:
    """Poll until idle, then drain."""
    api_key = os.environ.get("TEMPORAL_API_KEY") or None
    client = await Client.connect(
        os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"),
        namespace=os.environ.get("TEMPORAL_NAMESPACE", "default"),
        api_key=api_key,
        tls=bool(api_key),
        plugins=[StrandsPlugin()],
    )

    tracker = ActivityTracker()
    log.info("polling %s as %s/%s", TASK_QUEUE, DEPLOYMENT_NAME, BUILD_ID)
    # execute_code is a sync Activity, so it needs an executor to block on.
    with ThreadPoolExecutor(max_workers=4) as activity_executor:
        worker = Worker(
            client,
            task_queue=TASK_QUEUE,
            workflows=[workflows.StrandsAgentWorkflow],
            activities=[execute_code],
            activity_executor=activity_executor,
            interceptors=[tracker],
            deployment_config=WorkerDeploymentConfig(
                version=WorkerDeploymentVersion(
                    deployment_name=DEPLOYMENT_NAME, build_id=BUILD_ID
                ),
                use_worker_versioning=True,
                default_versioning_behavior=VersioningBehavior.PINNED,
            ),
            graceful_shutdown_timeout=DRAIN,
        )
        async with worker:
            await tracker.wait_until_idle(DEBOUNCE)
    log.info("worker idle for %ss; drained", DEBOUNCE)

async def _run_until_idle(task_id: int) -> None:
    """Own the Worker's whole life, and always release the async task."""
    try:
        await run_worker()
    except Exception:
        # Nothing awaits this task, so an error would otherwise be swallowed.
        log.exception("worker failed in async task")
    finally:
        # Without this the session stays HealthyBusy until MaxLifetime.
        app.complete_async_task(task_id)

@app.entrypoint
async def invoke(payload: dict) -> dict:
    """Start the Worker and acknowledge. The payload is unused."""
    # Prevent duplicate workers since we exit early
    global _worker
    if _worker is not None and not _worker.done():
        log.info("worker already polling %s", TASK_QUEUE)
        return {"message": "worker already polling", "task_queue": TASK_QUEUE}

    task_id = app.add_async_task("temporal-worker")
    _worker = asyncio.create_task(_run_until_idle(task_id))

    return {"message": "worker starting", "task_queue": TASK_QUEUE}

```
<!--SNIPEND-->

Replace `StrandsPlugin`, `StrandsAgentWorkflow`, and `execute_code` with the plugins, Workflows, and Activities used by
your application.

## 3. Deploy the Worker Runtime 

From the AgentCore project directory, validate and deploy the project. Set `--target` to the `name` of the deployment
target in `agentcore/aws-targets.json`. For example, if the target is named `default`, run:

```bash
agentcore validate
agentcore deploy --target default -y
```

AgentCore packages the Worker and its dependencies, deploys the Runtime, and creates the named endpoint.

Check the deployed resources. The `--runtime` value is the Runtime object's `name` in `agentcore/agentcore.json`. The
sample Runtime is named `temporal_strands_worker`. If your Runtime has a different name, replace this value:

```bash
agentcore status --runtime temporal_strands_worker --json
agentcore status --type runtime-endpoint --json
```

Record the Runtime ARN and the ARN of the named endpoint.

> **❗ Important:**
> Runtime ARN and endpoint ARN are different
>
> The Runtime ARN ends with `/runtime/<runtime-id>`. It identifies the Runtime resource. Use it with a trailing wildcard
> to scope the IAM role that Temporal assumes:
>
> ```text
> arn:aws:bedrock-agentcore:<region>:<account>:runtime/<runtime-id>
> ```
>
> The endpoint ARN adds `/runtime-endpoint/<endpoint-name>` to the Runtime ARN. It identifies the named endpoint that
> routes invocations to its configured Runtime version. Give this ARN to Temporal when you create the Worker Deployment
> Version:
>
> ```text
> arn:aws:bedrock-agentcore:<region>:<account>:runtime/<runtime-id>/runtime-endpoint/<endpoint-name>
> ```
>
> Do not pass the Runtime ARN as the `--aws-agentcore-endpoint-arn` value.
>

If you use a VPC instead of a public network, configure outbound access from the VPC to the Temporal Service. Temporal
invokes the named endpoint by assuming the IAM role that you create in [Step 4](#configure-iam).

## 4. Grant Temporal permission to invoke the Runtime 

> **ℹ️ Info:**
> Self-hosted Temporal Service
>
> If you use a self-hosted Temporal Service, create the invocation role during the
> [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/agentcore/self-hosted-setup#create-invocation-role).
> Use that role when you create the Worker Deployment Version and skip the rest of this step.
>

Temporal Cloud assumes an IAM role in your AWS account to get the named endpoint and invoke the Runtime. Choose an
External ID of at least five characters. Use the same value in the role trust policy and the Worker Deployment Version.
The External ID prevents a [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html)
attack.

[Download the CloudFormation template](/files/temporal-cloud-serverless-worker-agentcore-role.yaml), then deploy it.
Pass the Runtime ARN with a trailing wildcard so the policy covers the Runtime and its endpoints.

> **⚠️ Caution:**
>
> The template names the IAM role `<ROLE_NAME>-<STACK_NAME>`. An [IAM role name can contain at most 64
> characters](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-iam-role.html). Include
> the hyphen when checking the combined length. CloudFormation cannot create the role if the combined name exceeds this
> limit.
>

```bash
aws cloudformation create-stack \
  --stack-name <STACK_NAME> \
  --template-body file://temporal-cloud-serverless-worker-agentcore-role.yaml \
  --parameters \
    ParameterKey=AssumeRoleExternalId,ParameterValue=<EXTERNAL_ID> \
    ParameterKey=AgentRuntimeARNs,ParameterValue='<AGENT_RUNTIME_ARN>*' \
    ParameterKey=RoleName,ParameterValue=<ROLE_NAME> \
  --capabilities CAPABILITY_NAMED_IAM \
  --region <AWS_REGION>
```

Wait for the CloudFormation stack to finish:

```bash
aws cloudformation wait stack-create-complete \
  --stack-name <STACK_NAME> \
  --region <AWS_REGION>
```

Then retrieve the invocation role ARN:

```bash
aws cloudformation describe-stacks \
  --stack-name <STACK_NAME> \
  --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' \
  --output text \
  --region <AWS_REGION>
```

The role grants `bedrock-agentcore:InvokeAgentRuntime` and `bedrock-agentcore:GetAgentRuntimeEndpoint` on the configured
Runtime resources. This role does not run the Worker code.

## 5. Create the Worker Deployment Version 

Create a [Worker Deployment Version](/production-deployment/worker-deployments/worker-versioning) whose compute
configuration points to the named AgentCore Runtime endpoint. The deployment name and Build ID must match
`TEMPORAL_DEPLOYMENT_NAME` and `TEMPORAL_BUILD_ID` in the Runtime environment from
[Step 1](#configure-worker-runtime).

**Temporal Cloud UI**

In the Temporal Cloud UI, open your Namespace and select **Workers** > **Create Worker Deployment**. Provide these
values:

- **Name**: the value of `TEMPORAL_DEPLOYMENT_NAME` in the Runtime environment.
- **Build ID**: the value of `TEMPORAL_BUILD_ID` in the Runtime environment.
- **Compute Provider**: select **Amazon Bedrock AgentCore Runtime**.
- **Runtime endpoint ARN**: the named endpoint ARN from [Step 3](#deploy-runtime).
- **IAM role ARN**: the invocation role ARN from [Step 4](#configure-iam).
- **External ID**: the External ID from [Step 4](#configure-iam).

Save the Worker Deployment. When you create a version through the UI, the version is automatically current. Continue
to [Step 7](#verify-worker-startup).

**Temporal CLI**

Use the Temporal CLI for a self-hosted Temporal Service.

First, create the Worker Deployment if it does not already exist:

```bash
temporal worker deployment create \
  --namespace <TEMPORAL_NAMESPACE> \
  --name <DEPLOYMENT_NAME>
```

Then create the version with the AgentCore compute configuration:

```bash
temporal worker deployment create-version \
  --namespace <TEMPORAL_NAMESPACE> \
  --deployment-name <DEPLOYMENT_NAME> \
  --build-id <BUILD_ID> \
  --aws-agentcore-endpoint-arn <RUNTIME_ENDPOINT_ARN> \
  --aws-agentcore-assume-role-arn <INVOCATION_ROLE_ARN> \
  --aws-agentcore-assume-role-external-id <EXTERNAL_ID>
```

For Temporal Cloud, check whether Temporal can reach the endpoint by opening the Worker Deployment Version in the
Temporal Cloud UI and selecting **Actions** > **Validate Connection**. This checks that Temporal can assume the
invocation role, get the named endpoint, and invoke the Runtime.

## 6. Set the version as current 

If you used the Temporal CLI, set the version as current:

```bash
temporal worker deployment set-current-version \
  --namespace <TEMPORAL_NAMESPACE> \
  --deployment-name <DEPLOYMENT_NAME> \
  --build-id <BUILD_ID>
```

This command asks you to confirm because it changes which version receives new Tasks. Pass `--yes` to skip the prompt.
If you created the version in the Temporal Cloud UI, it is already current.

## 7. Verify Worker startup 

Submit work to the configured Task Queue using your application. When no Worker is polling, Temporal invokes the named
AgentCore Runtime endpoint. The Runtime starts the Worker, and the Worker polls and processes Tasks.

You can confirm the deployment in these places:

- **Temporal UI**: Open the Worker Deployment Version and confirm that a Worker has polled the Task Queue. In Temporal
  Cloud, also confirm that the connection is valid.
- **AgentCore logs**: Run `agentcore logs --runtime <RUNTIME_NAME>` to see the Worker start and process Tasks.
- **Temporal CLI**: Run `temporal worker deployment describe --name <DEPLOYMENT_NAME>` to inspect the deployment and
  current version.

---

# Self-hosted setup for Serverless Workers on Amazon Bedrock AgentCore Runtime

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/agentcore/self-hosted-setup

> Configure AWS access and a self-hosted Temporal Service to run Serverless Workers on Amazon Bedrock AgentCore Runtime.

> **Pre-release**
> Amazon Bedrock AgentCore Runtime support is in Pre-release, and its APIs may change in backwards-incompatible ways.

Serverless Workers on AgentCore Runtime require Temporal Service v1.32.0 or later.

This page covers the prerequisites for running [Serverless Workers](/serverless-workers) on a self-hosted Temporal
Service with AgentCore Runtime:

1. Ensure that AgentCore Runtime and the Temporal Service can reach each other.
2. Enable the Worker Controller Instance (WCI) and AgentCore compute provider through dynamic configuration.
3. Provide the Temporal Service with AWS credentials.
4. Create an IAM role that grants Temporal permission to get and invoke AgentCore Runtime endpoints.

Once setup is complete, follow the
[AgentCore Runtime deployment guide](/production-deployment/worker-deployments/serverless-workers/agentcore) to deploy
your Worker.

## Configure network access 

The [Temporal Service frontend](/temporal-service/temporal-server#frontend-service) must be reachable from the
AgentCore Runtime. If the Temporal Service has a public endpoint, configure the Runtime to use a public network. If the
Temporal Service is available only through a private network, configure the Runtime for
[VPC access](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-vpc.html) and connect that VPC to
the network that hosts the Temporal Service.

The Temporal Service must also reach the AgentCore control-plane and data-plane APIs. If the Service runs in an AWS VPC
without internet access, configure
[AgentCore interface VPC endpoints](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/vpc-interface-endpoints.html)
for both APIs.

## Enable the Worker Controller Instance 

[WCI](/serverless-workers#how-invocation-works) is the server component that monitors Task Queues and invokes compute
providers. It is disabled by default and must be enabled through
[dynamic configuration](/references/dynamic-configuration).

Add the following keys to your dynamic config file:

```yaml
workercontroller.enabled:
  - value: true

workercontroller.compute_providers.enabled:
  - value:
      - aws-agentcore

workercontroller.scaling_algorithms.enabled:
  - value:
      - no-sync
```

`workercontroller.compute_providers.enabled` is an allowlist that defaults to no providers. The `aws-agentcore` value
enables the AgentCore compute provider. AgentCore uses the `no-sync` event-driven scaling algorithm, which invokes a
Runtime session when the Task Queue needs more Worker capacity.

If either allowlist already contains values for other compute providers or scaling algorithms, keep those values and
add `aws-agentcore` or `no-sync` to the existing list.

To enable WCI for specific Namespaces instead of globally, add a `constraints` section with the Namespace name under
`workercontroller.enabled`. For example, to enable WCI only for `your-namespace`:

```yaml
workercontroller.enabled:
  - value: true
    constraints:
      namespace: 'your-namespace'
```

The Temporal Service watches the dynamic config file for changes and applies updates without a restart.

By default, the AWS compute providers require an invocation role and External ID in each Worker Deployment Version.
Keep the default value of `workercontroller.compute_providers.aws.require_role_and_external_id` enabled and create the
role in the following sections.

## Configure AWS credentials 

The Temporal Service needs AWS credentials to assume the AgentCore invocation role. How you provide credentials depends
on where the Temporal Service runs.

**On AWS infrastructure such as EC2, ECS, or EKS:** The server uses the attached instance role, task role, or pod role
automatically. The attached role must have `sts:AssumeRole` permission for the invocation role created in the next
step.

**Outside AWS:** Use [IAM Roles Anywhere](https://aws.amazon.com/iam/roles-anywhere/), or configure static AWS
credentials in the server environment. Static credentials are not recommended:

```text
AWS_ACCESS_KEY_ID=<ACCESS_KEY>
AWS_SECRET_ACCESS_KEY=<SECRET_KEY>
AWS_REGION=<REGION>
```

These credentials must belong to an IAM user or role that has `sts:AssumeRole` permission for the AgentCore invocation
role.

## Create the AgentCore invocation role 

Temporal gets and invokes AgentCore Runtime endpoints by assuming an IAM role in your AWS account. The role trust policy
must allow the AWS identity used by the Temporal Service to assume it.

[Download the CloudFormation template](/files/temporal-self-hosted-serverless-worker-agentcore-role.yaml), then deploy
it. Pass each Runtime ARN with a trailing wildcard so the policy covers the Runtime and its endpoints:

> **⚠️ Caution:**
>
> An [IAM role name can contain at most 64
> characters](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-iam-role.html). If you
> set `RoleName`, keep the complete name within this limit.
>

```bash
aws cloudformation create-stack \
  --stack-name temporal-agentcore-worker \
  --template-body file://temporal-self-hosted-serverless-worker-agentcore-role.yaml \
  --parameters \
    ParameterKey=TemporalIamRoleArn,ParameterValue=<TEMPORAL_SERVER_ROLE_ARN> \
    ParameterKey=AssumeRoleExternalId,ParameterValue=<EXTERNAL_ID> \
    ParameterKey=AgentRuntimeARNs,ParameterValue='<AGENT_RUNTIME_ARN>*' \
  --capabilities CAPABILITY_NAMED_IAM \
  --region <AWS_REGION>
```

| Parameter | Description |
| --- | --- |
| `TemporalIamRoleArn` | ARN of the IAM role or user that the Temporal Service runs as. Run `aws sts get-caller-identity` in the server environment to identify it. |
| `AssumeRoleExternalId` | Unique string that prevents [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html) attacks. Use the same value when creating the Worker Deployment Version. |
| `AgentRuntimeARNs` | Comma-separated Runtime ARNs that Temporal can access. Append a wildcard to each Runtime ARN to include its endpoints. |
| `RoleName` | Name of the IAM role to create. Defaults to `Temporal-AgentCore-Worker`. Use a different name when deploying more than one copy of the stack. |

#### CloudFormation template

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Description:
  Creates an IAM role that a self-hosted Temporal Service can assume to invoke Amazon Bedrock AgentCore runtimes.

Parameters:
  TemporalIamRoleArn:
    Type: String
    Description: The ARN of the IAM role or user that the Temporal Service runs as.

  AssumeRoleExternalId:
    Type: String
    Description: A unique identifier to prevent confused deputy attacks.
    AllowedPattern: '[a-zA-Z0-9_+=,.@-]*'
    MinLength: 5
    MaxLength: 45

  AgentRuntimeARNs:
    Type: CommaDelimitedList
    Description: >-
      Comma-separated list of AgentCore Runtime ARNs that Temporal may access. Append a wildcard to each Runtime ARN
      to include its endpoints.

  RoleName:
    Type: String
    Default: 'Temporal-AgentCore-Worker'

Resources:
  TemporalAgentCoreWorker:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref RoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS: [!Ref TemporalIamRoleArn]
            Action: sts:AssumeRole
            Condition:
              StringEquals:
                'sts:ExternalId': [!Ref AssumeRoleExternalId]
      Description: The role the Temporal Service uses to invoke AgentCore runtimes for Serverless Workers
      MaxSessionDuration: 3600

  TemporalAgentCoreInvokePermissions:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: 'Temporal-AgentCore-Invoke-Permissions'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - bedrock-agentcore:InvokeAgentRuntime
              - bedrock-agentcore:GetAgentRuntimeEndpoint
            Resource: !Ref AgentRuntimeARNs
      Roles:
        - !Ref TemporalAgentCoreWorker

Outputs:
  RoleARN:
    Description: The ARN of the IAM role created for the Temporal Service
    Value: !GetAtt TemporalAgentCoreWorker.Arn

  AgentRuntimeARNs:
    Description: The AgentCore Runtime ARNs that Temporal may access
    Value: !Join [', ', !Ref AgentRuntimeARNs]
```

Wait for the stack to finish, then retrieve the invocation role ARN:

```bash
aws cloudformation wait stack-create-complete \
  --stack-name temporal-agentcore-worker \
  --region <AWS_REGION>

aws cloudformation describe-stacks \
  --stack-name temporal-agentcore-worker \
  --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' \
  --output text \
  --region <AWS_REGION>
```

Use this role ARN and the External ID when creating the Worker Deployment Version.

## Next steps 

Follow the [AgentCore Runtime deployment
guide](/production-deployment/worker-deployments/serverless-workers/agentcore). Configure the Runtime Worker with your
self-hosted Temporal Service address and authentication settings. Skip the Temporal Cloud IAM step and use the
invocation role and External ID from this page when you create the Worker Deployment Version with the Temporal CLI.

---

# Deploy a Serverless Worker on AWS Lambda

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda

> Deploy a Temporal Serverless Worker on AWS Lambda.

> **Public Preview**

This guide walks through deploying a Temporal [Serverless Worker](/serverless-workers) on AWS Lambda.

> **💡 Tip:**
> Try it locally
>
> To evaluate Serverless Workers with no AWS account, or to iterate on Worker code before deploying to AWS, run this whole
> flow on your machine with [LocalStack](https://www.localstack.cloud/) in place of AWS and the Temporal dev server in
> place of a Temporal Service. See
> [Test a Serverless Worker locally with LocalStack](/production-deployment/worker-deployments/serverless-workers/aws-lambda/local-development).
>

## Prerequisites 

- A Temporal Cloud account with an AWS-hosted Namespace, or a self-hosted Temporal Service v1.31.0 or later. The
  Namespace's cloud provider must match the serverless compute provider.
- For self-hosted deployments, complete the
  [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/aws-lambda/self-hosted-setup) before following
  this guide.
- Every Workflow must declare a [versioning behavior](/worker-versioning#versioning-behaviors), or the Worker must set a
  default versioning behavior.
- An AWS account with permissions to create and invoke Lambda functions and create IAM roles.
- The AWS-specific steps in this guide require the [`aws` CLI](https://aws.amazon.com/cli/) installed and configured
  with your AWS credentials. You may use other tools to perform the steps, such as the AWS Console or the AWS SDKs.

- The [Go SDK](/develop/go), [Python SDK](/develop/python), [TypeScript SDK](/develop/typescript),
  [Java SDK](/develop/java), or [.NET SDK](/develop/dotnet), depending on which language you are using.
  Use the tabs to select your language and the rest of the page will update accordingly.

> **💡 Tip:**
>
>   If you are exploring the Serverless Worker feature and don't have a Workflow ready, you can use a sample from
>   the [Go Lambda Worker sample](https://github.com/temporalio/samples-go/tree/main/lambda-worker),
>   [Python Lambda Worker sample](https://github.com/temporalio/samples-python/tree/main/lambda_worker),
>   [TypeScript Lambda Worker sample](https://github.com/temporalio/samples-typescript/tree/main/lambda-worker),
>   [Java Lambda Worker sample](https://github.com/temporalio/samples-java/tree/main/lambda-worker),
>   or [.NET Lambda Worker sample](https://github.com/temporalio/samples-dotnet/tree/main/src/LambdaWorker).
>

## 1. Write Worker code 

Write a Worker that runs inside a Lambda function. The Worker handles the per-invocation lifecycle: connecting to
Temporal, polling for tasks, and gracefully shutting down before the invocation deadline.

**Go**

Use the Go SDK's `lambdaworker` package.

```go
package main

import (
	lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"
	"go.temporal.io/sdk/worker"
	"go.temporal.io/sdk/workflow"
)

func main() {
	lambdaworker.RunWorker(worker.WorkerDeploymentVersion{
		DeploymentName: "my-app",
		BuildID:        "build-1",
	}, func(opts *lambdaworker.Options) error {
		opts.TaskQueue = "my-task-queue"

		opts.RegisterWorkflowWithOptions(MyWorkflow, workflow.RegisterOptions{
			VersioningBehavior: workflow.VersioningBehaviorPinned,
		})
		opts.RegisterActivity(MyActivity)

		return nil
	})
}
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or
`Pinned`. Set it per-Workflow at registration time, or set a Worker-level default with `DefaultVersioningBehavior` in
`DeploymentOptions`.

For details on configuration options, Lambda-tuned defaults, and the invocation lifecycle, see
[Serverless Workers - Go SDK](/develop/go/workers/serverless-workers/aws-lambda).

**Python**

Use the Python SDK's `lambda_worker` contrib package.

```python
from temporalio.common import WorkerDeploymentVersion
from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker

from my_workflows import MyWorkflow
from my_activities import my_activity

def configure(config: LambdaWorkerConfig) -> None:
    config.worker_config["task_queue"] = "my-task-queue"
    config.worker_config["workflows"] = [MyWorkflow]
    config.worker_config["activities"] = [my_activity]

lambda_handler = run_worker(
    WorkerDeploymentVersion(
        deployment_name="my-app",
        build_id="build-1",
    ),
    configure,
)
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or
`AUTO_UPGRADE`. Set it per-Workflow in the `@workflow.defn` decorator, or set a Worker-level default with
`default_versioning_behavior` in the worker config.

```python
from temporalio import workflow
from temporalio.common import VersioningBehavior

@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class MyWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        ...
```

For details on configuration options, Lambda-tuned defaults, and observability, see
[Serverless Workers - Python SDK](/develop/python/workers/serverless-workers/aws-lambda).

**TypeScript**

Use the `@temporalio/lambda-worker` package.

```typescript
import { runWorker } from '@temporalio/lambda-worker';
import * as activities from './activities';

export const handler = runWorker({ deploymentName: 'my-app', buildId: 'build-1' }, (config) => {
  config.workerOptions.taskQueue = 'my-task-queue';
  config.workerOptions.workflowBundle = {
    codePath: require.resolve('./workflow-bundle.js'),
  };
  config.workerOptions.activities = activities;
  config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED';
});
```

Use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold
starts.

Each Workflow must declare a [versioning behavior](/worker-versioning#versioning-behaviors), either `AUTO_UPGRADE` or
`PINNED`. Set it per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with
`defaultVersioningBehavior` in the configure callback.

For details on configuration options, Lambda-tuned defaults, and observability, see
[Serverless Workers - TypeScript SDK](/develop/typescript/workers/serverless-workers/aws-lambda).

**Java**

Use the Java SDK's `temporal-aws-lambda` contrib module.

```java
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import io.temporal.aws.lambda.LambdaWorker;
import io.temporal.common.WorkerDeploymentVersion;

public final class Handler implements RequestHandler<Object, Void> {
  private static final RequestHandler<Object, Void> WORKER =
      LambdaWorker.define(
          new WorkerDeploymentVersion("my-app", "build-1"),
          builder -> {
            builder.setTaskQueue("my-task-queue");
            builder.registerWorkflowImplementationTypes(MyWorkflowImpl.class);
            builder.registerActivitiesImplementations(new MyActivitiesImpl());
          });

  @Override
  public Void handleRequest(Object input, Context context) {
    return WORKER.handleRequest(input, context);
  }
}
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or
`Pinned`. Set it per-Workflow at registration time, or set a Worker-level default with `DefaultVersioningBehavior` in
`DeploymentOptions`.

For details on configuration options, Lambda-tuned defaults, and the invocation lifecycle, see
[Serverless Workers - Java SDK](/develop/java/workers/serverless-workers/aws-lambda).

**.NET**

Use the `Temporalio.Extensions.Aws.Lambda` NuGet package.

```csharp
using Amazon.Lambda.Core;
using Temporalio.Common;
using Temporalio.Extensions.Aws.Lambda;

public class Function
{
    private static readonly Func<object?, ILambdaContext, Task> WorkerHandler =
        TemporalLambdaWorker.CreateHandler(
            new WorkerDeploymentVersion("my-app", "build-1"),
            config =>
            {
                config.WorkerOptions.TaskQueue = "my-task-queue";
                config.WorkerOptions.AddWorkflow<MyWorkflow>();
                config.WorkerOptions.AddActivity(MyActivities.MyActivityAsync);
            });

    public Task HandlerAsync(object? input, ILambdaContext context) =>
        WorkerHandler(input, context);
}
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or
`Pinned`. Set it per-Workflow with the `[Workflow]` attribute, or set a Worker-level default with
`DefaultVersioningBehavior` in `DeploymentOptions`.

```csharp
using Temporalio.Common;
using Temporalio.Workflows;

[Workflow(VersioningBehavior = VersioningBehavior.Pinned)]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string input)
    {
        return await Workflow.ExecuteActivityAsync(
            () => MyActivities.MyActivityAsync(input),
            new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) });
    }
}
```

For details on configuration options, Lambda-tuned defaults, and observability, see
[Serverless Workers - .NET SDK](/develop/dotnet/workers/serverless-workers/aws-lambda).

## 2. Deploy Lambda function 

Build your Worker for the Lambda runtime, package it as a zip, and deploy it to AWS Lambda.

### 2.1 Build and package 

**Go**

Cross-compile for Lambda's Linux runtime:

```bash
GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./worker
```

Package the binary into a zip file:

```bash
zip function.zip bootstrap
```

**Python**

Install dependencies into a local directory for packaging. Use `--platform` to fetch Linux-compatible binaries for the
Lambda runtime:

```bash
pip install --target ./package --platform manylinux2014_x86_64 --only-binary=:all: temporalio
```

To include [OpenTelemetry support](/develop/python/workers/serverless-workers/aws-lambda#add-observability), install
`temporalio[lambda-worker-otel]` instead.

Package the dependencies and your application code into a zip file:

```bash
cd package && zip -r ../function.zip . && cd ..
zip function.zip lambda_function.py my_workflows.py my_activities.py
```

**TypeScript**

Build the Workflow bundle and compile the project:

```bash
npx ts-node src/scripts/build-workflow-bundle.ts
npx tsc
```

Install production dependencies and package everything into a zip:

```bash
npm install --omit=dev
zip -r function.zip lib/ node_modules/ workflow-bundle.js
```

**Java**

Build your Lambda function with all dependencies bundled.
For example, with Gradle's Shadow plugin:

```bash
./gradlew shadowJar
```

The shadow JAR is in `build/libs/` and is a valid zip archive that can be uploaded directly to Lambda.

**.NET**

Publish for the Lambda runtime:

```bash
dotnet publish -c Release -r linux-x64 --self-contained false \
  -o ./publish
```

Package the published output into a zip file:

```bash
cd publish && zip -r ../function.zip . && cd ..
```

### 2.2 Deploy Lambda function 

Replace the placeholder values and run the following command to create the Lambda function in your AWS environment.

**Go**

```bash
aws lambda create-function \
  --function-name my-temporal-worker \
  --runtime provided.al2023 \
  --handler bootstrap \
  --role <EXECUTION_ROLE_ARN> \
  --zip-file fileb://function.zip \
  --timeout 600 \
  --memory-size 256 \
  --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":"<your-temporal-address>:7233","TEMPORAL_NAMESPACE":"<your-namespace>","TEMPORAL_API_KEY":"<your-api-key>"}}'
```

| Parameter         | Description                                                                                   |
| ----------------- | --------------------------------------------------------------------------------------------- |
| `--function-name` | Name of the Lambda function.                                                                  |
| `--runtime`       | Lambda runtime. Use `provided.al2023` for custom Go binaries.                                 |
| `--handler`       | Entry point binary name. Must be `bootstrap` when using the `provided.al2023` custom runtime. |

**Python**

```bash
aws lambda create-function \
  --function-name my-temporal-worker \
  --runtime python3.13 \
  --handler lambda_function.lambda_handler \
  --role <EXECUTION_ROLE_ARN> \
  --zip-file fileb://function.zip \
  --timeout 600 \
  --memory-size 256 \
  --environment '{"Variables":{"TEMPORAL_ADDRESS":"<your-temporal-address>:7233","TEMPORAL_NAMESPACE":"<your-namespace>","TEMPORAL_API_KEY":"<your-api-key>"}}'
```

| Parameter         | Description                                                                                  |
| ----------------- | -------------------------------------------------------------------------------------------- |
| `--function-name` | Name of the Lambda function.                                                                 |
| `--runtime`       | Lambda runtime. Use `python3.13` or another supported Python version.                        |
| `--handler`       | Entry point in `module.function` format. Must point to the handler returned by `run_worker`. |

**TypeScript**

```bash
aws lambda create-function \
  --function-name my-temporal-worker \
  --runtime nodejs22.x \
  --handler lib/index.handler \
  --role <EXECUTION_ROLE_ARN> \
  --zip-file fileb://function.zip \
  --timeout 600 \
  --memory-size 256 \
  --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":"<your-temporal-address>:7233","TEMPORAL_NAMESPACE":"<your-namespace>","TEMPORAL_API_KEY":"<your-api-key>"}}'
```

| Parameter         | Description                                                                               |
| ----------------- | ----------------------------------------------------------------------------------------- |
| `--function-name` | Name of the Lambda function.                                                              |
| `--runtime`       | Lambda runtime. Use `nodejs22.x` or another supported Node.js version (20+).              |
| `--handler`       | Entry point in `module.export` format. Must point to the handler exported by `runWorker`. |

**Java**

```bash
aws lambda create-function \
  --function-name my-temporal-worker \
  --runtime java17 \
  --handler io.temporal.samples.lambdaworker.LambdaFunction::handleRequest \
  --role <EXECUTION_ROLE_ARN> \
  --zip-file fileb://build/libs/<my-worker>-all.jar \
  --timeout 90 \
  --memory-size 1024 \
  --environment '{"Variables":{"TEMPORAL_ADDRESS":"<your-temporal-address>:7233","TEMPORAL_NAMESPACE":"<your-namespace>","TEMPORAL_API_KEY":"<your-api-key>"}}'
```

| Parameter         | Description                                                                                                                    |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `--function-name` | Name of the Lambda function.                                                                                                   |
| `--runtime`       | Lambda runtime. Use `java17` or another supported Java version.                                                                |
| `--handler`       | Fully qualified class name of your `RequestHandler` implementation, with `::handleRequest`.                                    |
| `--zip-file`      | Path to the shadow JAR in `build/libs/`. The filename follows the pattern `<name>-<version>-all.jar` from your `build.gradle`. Shadow JARs are valid zip archives and can be uploaded directly. |
| `--memory-size`   | Java Workers typically need more memory than other runtimes. Start with `1024` and adjust based on CloudWatch memory metrics.   |

**.NET**

```bash
aws lambda create-function \
  --function-name my-temporal-worker \
  --runtime dotnet10 \
  --handler MyAssembly::MyNamespace.Function::HandlerAsync \
  --role <EXECUTION_ROLE_ARN> \
  --zip-file fileb://function.zip \
  --timeout 600 \
  --memory-size 256 \
  --environment '{"Variables":{"TEMPORAL_ADDRESS":"<your-temporal-address>:7233","TEMPORAL_NAMESPACE":"<your-namespace>","TEMPORAL_API_KEY":"<your-api-key>","SSL_CERT_FILE":"/etc/pki/tls/certs/ca-bundle.crt"}}'
```

| Parameter         | Description                                                                                                            |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `--function-name` | Name of the Lambda function.                                                                                           |
| `--runtime`       | Lambda runtime. Use `dotnet10` or another supported .NET version.                                                       |
| `--handler`       | Entry point in `Assembly::Namespace.Class::Method` format. Must point to the method that calls the worker handler.     |
| `SSL_CERT_FILE`   | Required. The .NET Lambda runtime sets a default certificate path that the Temporal SDK cannot read. This overrides it with the system CA bundle so TLS connections work. |

The following parameters apply to all SDKs:

| Parameter                       | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--role`                        | ARN of the Lambda [execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html), which grants the function permission to run. Trusted principal must be `lambda.amazonaws.com`. This is separate from the role Temporal uses to invoke the function in [Step 3](#configure-iam). The role must have at least the [`AWSLambdaBasicExecutionRole`](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaBasicExecutionRole.html) managed policy attached. |
| `--zip-file`                    | Path to your packaged deployment zip.                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `--timeout`                     | Invocation deadline in seconds. This is the maximum time each Lambda invocation can run before AWS terminates it. Set this high enough for the Worker to start, process Tasks, and [shut down gracefully](/serverless-workers#worker-lifecycle).                                                                                                                                                                                                                                                              |
| `--memory-size`                 | Memory in MB allocated to each invocation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `TEMPORAL_ADDRESS`              | Temporal frontend address (for example, `<namespace>.<account>.tmprl.cloud:7233`).                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `TEMPORAL_NAMESPACE`            | Temporal Namespace.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `TEMPORAL_TASK_QUEUE`           | Task Queue name. The Worker package reads it at startup, before the configure callback runs, so a Task Queue set in the callback takes precedence. Set one or the other, or the Worker fails to start.                                                                                                                                                                                                                                                                                                          |
| `TEMPORAL_TLS_CLIENT_CERT_PATH` | Path to the TLS client certificate file for mTLS authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `TEMPORAL_TLS_CLIENT_KEY_PATH`  | Path to the TLS client key file for mTLS authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `TEMPORAL_API_KEY`              | API key for API key authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |

> **⚠️ Caution:**
>
> AWS Lambda functions default to a 3-second timeout, which is too short for the Worker to start, connect to Temporal, and
> register the Task Queue. If the first invocation times out before the Worker polls, the Task Queue binding is never
> created and the Lambda is never invoked again. Always set `--timeout` high enough for the Worker to start, process Tasks,
> and [shut down gracefully](/serverless-workers#worker-lifecycle). See
> [Troubleshoot Serverless Workers on AWS Lambda](/troubleshooting/serverless-workers/aws-lambda#validate-connection) if
> the Task Queue binding is not established.
>

The serverless Worker packages read environment variables and configuration files automatically at startup. For the full
list of supported environment variables, config file format, and profiles, see
[Environment configuration](/develop/environment-configuration).

Sensitive values like TLS keys and API keys should be encrypted at rest. See
[AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options.

To update an existing function with new code:

```bash
aws lambda update-function-code \
  --function-name my-temporal-worker \
  --zip-file fileb://function.zip
```

After updating, increment the Build ID in your Worker code and publish a new Lambda function version.
See [Step 2.3](#publish-lambda-version) for details.

### 2.3 Publish a Lambda function version 

Publish a [Lambda function version](https://docs.aws.amazon.com/lambda/latest/dg/configuration-versions.html) after creating your function and after each subsequent `update-function-code` to create an immutable snapshot of your Lambda code:

```bash
aws lambda publish-version \
  --function-name my-temporal-worker \
  --description "Build ID build-5"
```

The command prints the `FunctionArn` for the new version, for example
`arn:aws:lambda:us-east-1:123456789012:function:my-temporal-worker:5`.
Use this qualified versioned ARN later when you create the Worker Deployment Version in [Step 4](#create-worker-deployment-version).

For production, maintain a one-to-one mapping between each Lambda function version and each Temporal Worker Deployment Version Build ID.
For development or non-critical workloads, you can skip `publish-version` and use an unqualified ARN to iterate faster.
See [Worker Versioning with Serverless Workers](/serverless-workers#worker-versioning-with-serverless-workers) for the trade-offs.

To roll back, revert the Temporal Current Version with
[`temporal worker deployment set-current-version`](/cli/command-reference/worker#set-current-version).
The previous Worker Deployment Version still points at its original Lambda function version and is ready to receive traffic again.

## 3. Configure IAM for Temporal invocation (Cloud only) 

This section applies to Temporal Cloud. For self-hosted Temporal Service deployments, see
[Self-hosted setup](/production-deployment/worker-deployments/serverless-workers/aws-lambda/self-hosted-setup#create-invocation-role)
for IAM configuration with a different CloudFormation template.

Temporal needs permission to invoke your Lambda function and check its status. The Temporal server assumes an IAM role in
your AWS account with a handful of Lambda permissions scoped to your Worker functions. The trust policy on the role
includes an External ID condition to prevent
[confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html) attacks.

Temporal calls the Lambda API with this role and does not connect to your network, so you do not need to open inbound
ports.

The Temporal Cloud UI provides a **Launch Stack** button on the **Create Worker Deployment** page that opens the AWS CloudFormation console with the template and parameters pre-filled. Click it to create the role without downloading or configuring the template manually.

You can also create the role using the templates below.

**CloudFormation**

[Download the CloudFormation template](/files/temporal-cloud-serverless-worker-role.yaml).
Deploy the following CloudFormation template to create the invocation role with these permissions.
[Download the template](/files/temporal-cloud-serverless-worker-role.yaml). See the template details below for the exact
permissions granted.

| Parameter              | Description                                                                                                                                                                                                        |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `AssumeRoleExternalId` | A string you choose to prevent [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html) attacks. Can be any value. Use the same value when creating the Worker Deployment Version. |
| `LambdaFunctionARNs`   | Comma-separated list of Lambda function ARNs that Temporal may invoke. To allow invocation of any published version of a function, use a wildcard suffix (for example, `arn:aws:lambda:...:function:my-temporal-worker:*`). One role can authorize multiple Worker Lambdas. |
| `RoleName`             | Base name for the created IAM role. Defaults to `Temporal-Cloud-Serverless-Worker`. Provide a new role name if creating more than one stack.                                                                                                                                |

#### CloudFormation template

```yaml
# CloudFormation template for creating an IAM role that Temporal Cloud can assume to invoke Lambda functions.
AWSTemplateFormatVersion: '2010-09-09'
Description:
  Creates an IAM role that Temporal Cloud can assume to invoke multiple Lambda functions for Serverless Workers.

Parameters:
  AssumeRoleExternalId:
    Type: String
    Description: A string you choose. Can be any value.
    AllowedPattern: '[a-zA-Z0-9_+=,.@-]*'
    MinLength: 5
    MaxLength: 45

  LambdaFunctionARNs:
    Type: CommaDelimitedList
    Description: >-
      Comma-separated list of Lambda function ARNs to invoke (e.g.,
      arn:aws:lambda:us-west-2:123456789012:function:worker-1,arn:aws:lambda:us-west-2:123456789012:function:worker-2)

  RoleName:
    Type: String
    Default: 'Temporal-Cloud-Serverless-Worker'

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: 'Temporal Cloud Configuration'
        Parameters:
          - AssumeRoleExternalId
      - Label:
          default: 'Lambda Configuration'
        Parameters:
          - LambdaFunctionARNs
          - RoleName
    ParameterLabels:
      AssumeRoleExternalId:
        default: 'External ID'
      LambdaFunctionARNs:
        default: 'Lambda Function ARNs (comma-separated list)'
      RoleName:
        default: 'IAM Role Name'

Resources:
  TemporalCloudServerlessWorker:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref RoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                [
                  arn:aws:iam::902542641901:role/wci-lambda-invoke,
                  arn:aws:iam::160190466495:role/wci-lambda-invoke,
                  arn:aws:iam::819232936619:role/wci-lambda-invoke,
                  arn:aws:iam::829909441867:role/wci-lambda-invoke,
                  arn:aws:iam::354116250941:role/wci-lambda-invoke,
                ]
            Action: sts:AssumeRole
            Condition:
              StringEquals:
                'sts:ExternalId': [!Ref AssumeRoleExternalId]
      Description: 'The role Temporal Cloud uses to invoke Lambda functions for Serverless Workers'
      MaxSessionDuration: 3600 # 1 hour

  TemporalCloudLambdaInvokePermissions:
    Type: AWS::IAM::Policy
    DependsOn: TemporalCloudServerlessWorker
    Properties:
      PolicyName: 'Temporal-Cloud-Lambda-Invoke-Permissions'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - lambda:InvokeFunction
              - lambda:GetFunction
            Resource: !Ref LambdaFunctionARNs
      Roles:
        - !Ref RoleName

Outputs:
  RoleARN:
    Description: The ARN of the IAM role created for Temporal Cloud
    Value: !GetAtt TemporalCloudServerlessWorker.Arn
    Export:
      Name: !Sub '${AWS::StackName}-RoleARN'

  RoleName:
    Description: The name of the IAM role
    Value: !Ref RoleName

  LambdaFunctionARNs:
    Description: The Lambda function ARNs that can be invoked
    Value: !Join [', ', !Ref LambdaFunctionARNs]
```

Deploy the template:

```bash
aws cloudformation create-stack \
  --stack-name <STACK_NAME> \
  --template-body file://temporal-cloud-serverless-worker-role.yaml \
  --parameters \
    ParameterKey=AssumeRoleExternalId,ParameterValue=<EXTERNAL_ID> \
    ParameterKey=LambdaFunctionARNs,ParameterValue='"<LAMBDA_FUNCTION_ARN>"' \
  --capabilities CAPABILITY_NAMED_IAM \
  --region <AWS_REGION>
```

After the stack finishes creating, retrieve the IAM role ARN from the stack outputs:

```bash
aws cloudformation describe-stacks \
  --stack-name <STACK_NAME> \
  --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' \
  --output text \
  --region <AWS_REGION>
```

**Terraform**

Use the [Temporal Terraform module](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/aws/lambda) to create the invocation role.

| Variable                      | Description                                                                                                                                                                                                        | Required |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| `external_id`                 | A string you choose to prevent [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html) attacks. Can be any value. Use the same value when creating the Worker Deployment Version. | Yes      |
| `temporal_cloud_principals`   | Temporal Cloud AWS IAM role ARNs permitted to assume this role. Use the values shown in the example below.                                                                                                         | Yes      |
| `lambda_function_arns`        | List of Lambda function ARNs that Temporal Cloud is permitted to invoke.                                                                                                                                           | Yes      |
| `role_name`                   | Name of the IAM role to create. Defaults to `Temporal-Cloud-Serverless-Worker`.                                                                                                                                    | No       |

Add the module to your Terraform configuration:

```hcl
module "serverless_worker_lambda" {
  source = "github.com/temporalio/terraform-modules//modules/serverless-workers/aws/lambda"

  external_id = "<EXTERNAL_ID>"

  temporal_cloud_principals = [
    "arn:aws:iam::902542641901:role/wci-lambda-invoke",
    "arn:aws:iam::160190466495:role/wci-lambda-invoke",
    "arn:aws:iam::819232936619:role/wci-lambda-invoke",
    "arn:aws:iam::829909441867:role/wci-lambda-invoke",
    "arn:aws:iam::354116250941:role/wci-lambda-invoke",
  ]

  lambda_function_arns = [
    "arn:aws:lambda:<AWS_REGION>:<ACCOUNT_ID>:function:<FUNCTION_NAME>",
  ]
}
```

Apply the configuration:

```bash
terraform init && terraform apply
```

After the apply completes, retrieve the IAM role ARN from the outputs:

```bash
terraform output role_arn
```

Use this role ARN in your Worker Deployment Version's compute configuration.

> **⚠️ Caution:**
> AWS Organizations resource control policies
>
> If your AWS account belongs to an AWS Organization, a
> [resource control policy (RCP)](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_rcps.html)
> can block Temporal from assuming the role even when the trust policy allows it. The failure looks like an ordinary
> AssumeRole access-denied error. Before you create the Worker Deployment Version, ask your AWS Organizations
> administrator to confirm that no RCP denies `sts:AssumeRole` to the Temporal principals in the trust policy. See
> [Troubleshoot Serverless Workers on AWS Lambda](/troubleshooting/serverless-workers/aws-lambda#validate-connection).
>

## 4. Create Worker Deployment Version 

Create a [Worker Deployment Version](/production-deployment/worker-deployments/worker-versioning) with a compute
provider that points to your Lambda function. The compute configuration tells Temporal how to invoke your Worker: the
provider type (`aws-lambda`), the Lambda function ARN, and the IAM role to assume. The deployment name and build ID must
match the values in your Worker code.

You can create the version using the Temporal UI or the Temporal CLI.

**Temporal UI**

1. In the Temporal UI, open your Namespace.
2. In the left pane, select **Workers**.
3. Click **Create Worker Deployment** in the upper right corner.
4. Under **Configuration**, enter a **Name** and **Build ID**. These must match the `DeploymentName` and `BuildID` in
   your Worker code.
5. Under **Compute**, select **AWS Lambda** and provide:
   - **Lambda ARN**: the ARN of your Lambda function.
   - **IAM Role ARN**: the ARN of the role Temporal assumes to invoke your Lambda function. This is the role ARN from
     [Step 3](#configure-iam) (output of the CloudFormation stack). This is not the Lambda execution role from
     [Step 2](#deploy-lambda-function) or your own IAM user/role.
   - **External ID**: the same value you passed to the CloudFormation template.
6. Click **Save**.

When you create a version through the UI, the version is automatically set as current. Skip to
[Verify the deployment](#verify-deployment).

**Temporal CLI**

Use the CLI for manual setup, shell scripts, and CI/CD pipelines. When you create a version through the CLI, you must
[set the version as current](#set-current-version) as a separate step.

First, create the Worker Deployment if it does not already exist:

```bash
temporal worker deployment create \
  --namespace <YOUR_NAMESPACE> \
  --name my-app
```

Then create the version with the compute provider configuration:

```bash
temporal worker deployment create-version \
  --namespace <YOUR_NAMESPACE> \
  --deployment-name my-app \
  --build-id build-1 \
  --aws-lambda-function-arn <LAMBDA_FUNCTION_ARN> \
  --aws-lambda-assume-role-arn <INVOCATION_ROLE_ARN> \
  --aws-lambda-assume-role-external-id <EXTERNAL_ID>
```

| Flag                                   | Description                                                                                                                                                                                                                                       |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--deployment-name`                    | Worker Deployment name. Must match `DeploymentName` in your Worker code.                                                                                                                                                                          |
| `--build-id`                           | Worker Deployment Version build ID. Must match `BuildID` in your Worker code.                                                                                                                                                                     |
| `--aws-lambda-function-arn`            | Qualified versioned ARN of the Lambda function Temporal invokes for this version (for example, `function:my-worker:5`). See [Step 2.3](#publish-lambda-version).                                                                                  |
| `--aws-lambda-assume-role-arn`         | IAM role Temporal assumes to invoke the function. This is the `RoleARN` output from the CloudFormation stack in [Step 3](#configure-iam). This is not the Lambda execution role from [Step 2](#deploy-lambda-function) or your own IAM user/role. |
| `--aws-lambda-assume-role-external-id` | External ID configured in the IAM role trust policy.                                                                                                                                                                                              |

To verify that Temporal can reach your Lambda function, go to **Workers** > **Deployments** > select your deployment >
open the **Actions** menu on the version and click **Validate Connection**. This checks that Temporal can assume the IAM
role and invoke the function.

## 5. Set version as current 

If you created the version through the Temporal UI, the version is already current and you can skip this step.

If you used the CLI, set the version as current. Without this step, tasks on the Task Queue will not route to the
version, and Temporal will not invoke the Lambda function.

```bash
temporal worker deployment set-current-version \
  --deployment-name my-app \
  --build-id build-1
```

## 6. Verify deployment 

Start a Workflow on the same Task Queue to confirm that Temporal invokes your Lambda Worker.

```bash
temporal workflow start \
  --task-queue my-task-queue \
  --type MyWorkflow \
  --input '"Hello, serverless!"'
```

When the task lands on the Task Queue with no active pollers, Temporal detects the compute provider configuration and
invokes your Lambda function. The Worker starts, connects to Temporal, picks up the task, and processes it.

You can verify the invocation by checking:

- **Temporal UI:** The Workflow execution should show task completions in the event history.
- **Connection status:** In **Workers > Deployments**, the **Connection** column shows whether Temporal can reach the Lambda function. You can also check this from the CLI with `temporal worker deployment describe`.
- **AWS CloudWatch Logs:** The Lambda function's log group (`/aws/lambda/my-temporal-worker`) should show invocation
  logs with the Worker startup, task processing, and graceful shutdown. This requires the Lambda execution role from
  [Step 2](#deploy-lambda-function) to have CloudWatch Logs permissions, included in the
  [`AWSLambdaBasicExecutionRole`](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaBasicExecutionRole.html)
  managed policy.

If the Workflow does not progress or the Lambda is not invoked, see
[Troubleshoot Serverless Workers on AWS Lambda](/troubleshooting/serverless-workers/aws-lambda).

---

# Test a Serverless Worker locally with LocalStack

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda/local-development

> **Public Preview**

Run the full [Serverless Worker](/serverless-workers) loop on your machine with no AWS account and no Temporal Cloud.
[LocalStack](https://www.localstack.cloud/) emulates AWS Lambda locally, and the
[Temporal development server](/develop/run-a-development-server) stands in for a Temporal Service. Use this to evaluate
Serverless Workers or to iterate on Worker code before you
[deploy to AWS](/production-deployment/worker-deployments/serverless-workers/aws-lambda).

The flow is the same as a real deployment, with two differences: you deploy the Lambda to the LocalStack endpoint, and
the Worker reaches the development server through `host.docker.internal` instead of a public address.

> **📝 Note:**
> What local testing does and does not cover
>
> LocalStack validates your Worker code, packaging, and the full invoke-poll-complete loop driven by the Worker
> Controller. It does **not** validate real AWS IAM role assumption or the `aws:ExternalId` trust condition, because
> LocalStack's IAM is permissive. Validate those against a real AWS account with the
> [AWS Lambda deployment guide](/production-deployment/worker-deployments/serverless-workers/aws-lambda).
>

## Prerequisites 

- [Docker](https://docs.docker.com/get-docker/).
- The [Temporal CLI](/cli), which includes the development server.
- The [`aws` CLI](https://aws.amazon.com/cli/), or [`awslocal`](https://github.com/localstack/awscli-local), a thin
  wrapper that targets LocalStack automatically.
- A Worker packaged as a Lambda zip. Follow
  [Write Worker code](/production-deployment/worker-deployments/serverless-workers/aws-lambda#write-worker-code) and
  [Build and package](/production-deployment/worker-deployments/serverless-workers/aws-lambda#build-and-package) in the
  AWS Lambda guide, or start from a sample:
  [Go](https://github.com/temporalio/samples-go/tree/main/lambda-worker),
  [Python](https://github.com/temporalio/samples-python/tree/main/lambda_worker),
  [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/lambda-worker).

## 1. Start LocalStack 

Start the LocalStack Community edition with the Lambda, STS, and IAM services. Mount the Docker socket so LocalStack can
run your Lambda in a container.

```bash
docker run --rm -d --name localstack -p 4566:4566 \
  -e SERVICES=lambda,sts,iam \
  -v /var/run/docker.sock:/var/run/docker.sock \
  localstack/localstack:<version>

# Wait until the Lambda service is ready.
until curl -s http://localhost:4566/_localstack/health | grep -q '"lambda"'; do sleep 1; done
echo "LocalStack ready"
```

> **📝 Note:**
> Choose a LocalStack version
>
> Pin a released [Community tag](https://github.com/localstack/localstack/releases) that supports your Lambda runtime.
> Avoid `:latest`, which may try to activate the Pro edition and refuse to start without a token. The Lambda runtime you
> can deploy depends on your LocalStack version: older releases such as `3.8.1` support up to `python3.12`, while recent
> `4.x` releases add `python3.13`. Confirm your version supports the runtime you deploy in the
> [LocalStack Lambda docs](https://docs.localstack.cloud/user-guide/aws/lambda/).
>

## 2. Start the Temporal development server 

The development server is a self-hosted Temporal Service, so enable the [Worker Controller](/serverless-workers#how-invocation-works)
the same way you would for [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/aws-lambda/self-hosted-setup#enable-worker-controller).
The Worker Controller runs inside the server and calls the compute provider, so point it at LocalStack with
`AWS_ENDPOINT_URL` and placeholder credentials. The development server has no dynamic configuration file, so pass the Worker
Controller settings as `--dynamic-config-value` flags:

```bash
AWS_ENDPOINT_URL=http://localhost:4566 \
AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1 \
temporal server start-dev --ip 0.0.0.0 \
  --dynamic-config-value workercontroller.enabled=true \
  --dynamic-config-value workercontroller.compute_providers.aws.require_role_and_external_id=false
```

| Setting                                                        | Why it is needed for local development                                                                                                                                                          |
| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AWS_ENDPOINT_URL` and `AWS_*` credentials                    | Point the server's Worker Controller at LocalStack (`http://localhost:4566`) instead of real AWS, so it invokes your LocalStack function. LocalStack accepts the placeholder `test` credentials. |
| `--ip 0.0.0.0`                                                 | Binds the server to all network interfaces so the Lambda container can reach it. The default `127.0.0.1` is not reachable from the container. The Worker connects back through `host.docker.internal:7233`. |
| `workercontroller.enabled=true`                               | Turns on the Worker Controller, which is disabled by default. This is the same setting used in self-hosted setup.                                                                              |
| `workercontroller.compute_providers.aws.require_role_and_external_id=false` | Relaxes the AWS role and External ID check. LocalStack's IAM cannot enforce the `aws:ExternalId` trust condition, so skip it locally and validate the real path on AWS.                          |

The Web UI is at `http://localhost:8233`.

> **📝 Note:**
>
> This command sets the minimum needed for local use. If Tasks are not picked up, add the `compute_providers.enabled` and
> `scaling_algorithms.enabled` settings from
> [Enable the Worker Controller](/production-deployment/worker-deployments/serverless-workers/aws-lambda/self-hosted-setup#enable-worker-controller)
> as additional `--dynamic-config-value` flags with JSON list values, for example
> `--dynamic-config-value 'workercontroller.scaling_algorithms.enabled=["no-sync"]'`.
>

## 3. Deploy the Worker to LocalStack 

Deploy the packaged Worker as a normal Lambda, but point the `aws` CLI at the LocalStack endpoint and set
`TEMPORAL_ADDRESS` so the Worker connects back to the development server on your host. LocalStack's IAM is permissive, so the
`--role` value is a placeholder.

```bash
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1

aws lambda create-function --endpoint-url http://localhost:4566 \
  --function-name my-temporal-worker \
  --runtime python3.13 \
  --handler lambda_function.lambda_handler \
  --role arn:aws:iam::000000000000:role/dev \
  --zip-file fileb://function.zip \
  --timeout 600 \
  --memory-size 256 \
  --environment '{"Variables":{"TEMPORAL_ADDRESS":"host.docker.internal:7233","TEMPORAL_NAMESPACE":"default"}}'
```

Set `--runtime` and `--handler` to match how you packaged the Worker, as described in
[Deploy Lambda function](/production-deployment/worker-deployments/serverless-workers/aws-lambda#deploy-lambda-function-step)
in the AWS Lambda guide. The example values are for the
[Python sample](https://github.com/temporalio/samples-python/tree/main/lambda_worker). Only two things differ from a real
deployment: the `--endpoint-url` targets LocalStack, and `TEMPORAL_ADDRESS` is `host.docker.internal:7233`.

> **⚠️ Caution:**
>
> AWS Lambda defaults to a 3-second timeout, which is too short for the Worker to start, connect, and register its Task
> Queue. Set `--timeout` high enough for the Worker to start, process Tasks, and
> [shut down gracefully](/serverless-workers#worker-lifecycle).
>

## 4. Create the Worker Deployment Version 

Create the Worker Deployment Version exactly as in the AWS Lambda guide's
[Create Worker Deployment Version](/production-deployment/worker-deployments/serverless-workers/aws-lambda#create-worker-deployment-version)
and [Set version as current](/production-deployment/worker-deployments/serverless-workers/aws-lambda#set-current-version)
steps, but use the LocalStack function ARN and pass `--aws-lambda-skip-role-and-external-id`. The CLI otherwise
requires an assume-role ARN and External ID whenever you set a function ARN; this flag skips that client-side check and
defers to the server, which accepts an ARN-only configuration because you started it with
`require_role_and_external_id=false`. Don't pass the assume-role flags together with it — the CLI rejects that
combination.

```bash
export TEMPORAL_ADDRESS=localhost:7233

# Create the Worker Deployment first; create-version requires it to exist.
temporal worker deployment create --name my-app

temporal worker deployment create-version \
  --deployment-name my-app \
  --build-id build-1 \
  --aws-lambda-function-arn arn:aws:lambda:us-east-1:000000000000:function:my-temporal-worker \
  --aws-lambda-skip-role-and-external-id

temporal worker deployment set-current-version \
  --deployment-name my-app \
  --build-id build-1
```

The `--deployment-name` and `--build-id` must match the values in your Worker code.

## 5. Run a Workflow 

Start a Workflow on the Task Queue. When the Task lands with no Worker polling, the Worker Controller invokes your
LocalStack function; the Worker starts, connects back, polls, and completes the Workflow.

```bash
temporal workflow start \
  --task-queue my-task-queue \
  --type MyWorkflow \
  --input '"Hello, serverless!"'
```

Verify the run:

```bash
docker logs localstack | grep lambda.Invoke   # a Worker Controller-driven invocation
temporal workflow list                         # the Workflow reaches Completed
```

You can also watch the Workflow reach Completed in the Web UI at `http://localhost:8233`.

## Next steps 

- To validate real IAM role assumption and the `aws:ExternalId` trust condition, deploy to a real AWS account with the
  [AWS Lambda deployment guide](/production-deployment/worker-deployments/serverless-workers/aws-lambda).
- To configure a self-hosted Temporal Service to invoke Lambda functions in AWS, see
  [Self-hosted setup](/production-deployment/worker-deployments/serverless-workers/aws-lambda/self-hosted-setup).
- If a Workflow does not progress or the Lambda is not invoked, see
  [Troubleshoot Serverless Workers](/troubleshooting/serverless-workers/aws-lambda).

## Troubleshooting 

| Symptom                                             | Fix                                                                                                                                                                                       |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The Worker cannot reach the server.                 | Start the development server with `--ip 0.0.0.0` and set `TEMPORAL_ADDRESS=host.docker.internal:7233` on the Lambda.                                                                              |
| `create-function` rejects the runtime as unsupported. | Set `--runtime` to a value your LocalStack version supports, or upgrade to a newer LocalStack version.                                                                                    |
| LocalStack does not start or asks for a token.      | Do not use the `:latest` tag on the Community image. Pin a released Community tag instead.                                                                                                 |
| The first invocation times out before the Worker polls. | Increase `--timeout` on the Lambda so the Worker has time to start, register its Task Queue, and shut down gracefully.                                                                 |
| `create-version` fails with `missing required AWS Lambda provider detail: role`. | Add `--aws-lambda-skip-role-and-external-id`. The CLI requires an assume-role ARN and External ID whenever you set a function ARN unless this flag is present, and rejects passing both. |

---

# Self-hosted setup for Serverless Workers

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda/self-hosted-setup

> Configure a self-hosted Temporal Service to use Serverless Workers with AWS Lambda.

> **Public Preview**

Serverless Workers require Temporal Service v1.31.0 or later.

This page covers the prerequisites for running [Serverless Workers](/serverless-workers) on a self-hosted Temporal
Service with AWS Lambda:

1. Ensure Lambda can reach the Temporal Service.
2. Enable the Worker Controller Instance (WCI) on the server through dynamic configuration.
3. Provide the server with AWS credentials to assume IAM roles.
4. Create an IAM role in your AWS account that grants Temporal permission to invoke Lambda functions.

Once setup is complete, follow the
[AWS Lambda deployment guide](/production-deployment/worker-deployments/serverless-workers/aws-lambda) to deploy your
Worker.

## Ensure Lambda can reach the Temporal Service 

The [Temporal Service frontend](/temporal-service/temporal-server#frontend-service) must be reachable from the Lambda execution
environment. How to achieve this depends on your network setup. If the Temporal Service runs on a private network, you
may need [VPC access for Lambda](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html), VPC peering, or a
similar mechanism to allow the Lambda function to connect to the Temporal frontend.

## Enable the Worker Controller Instance 

[WCI](/serverless-workers#how-invocation-works) is the server component that monitors Task Queues and invokes compute
providers. It is disabled by default and must be enabled through
[dynamic configuration](/references/dynamic-configuration).

Add the following keys to your dynamic config file:

```yaml
workercontroller.enabled:
  - value: true

workercontroller.compute_providers.enabled:
  - value:
      - aws-lambda

workercontroller.scaling_algorithms.enabled:
  - value:
      - no-sync
```

To enable WCI for specific Namespaces instead of globally, add a `constraints` section with the
Namespace name under `workercontroller.enabled`. For example, to enable WCI only for `your-namespace`:

```yaml
workercontroller.enabled:
  - value: true
    constraints:
      namespace: 'your-namespace'
```

The Temporal Service watches the dynamic config file for changes and applies updates without a restart.

## Configure AWS credentials 

The Temporal Service needs AWS credentials to assume an IAM role that invokes Lambda functions. How you provide
credentials depends on where the Temporal Service runs.

**On AWS infrastructure (EC2, ECS, EKS):** The server uses the attached instance role, task role, or pod role
automatically. No additional credential configuration is needed. The attached role must have `sts:AssumeRole` permission
for the Lambda invocation role created in the next step.

**Outside AWS:** Use [IAM Roles Anywhere](https://aws.amazon.com/iam/roles-anywhere/), or configure static AWS
credentials in the server's environment (not recommended):

```
AWS_ACCESS_KEY_ID=<access-key>
AWS_SECRET_ACCESS_KEY=<secret-key>
AWS_REGION=<region>
```

These credentials must belong to an IAM user or role that has `sts:AssumeRole` permission for the Lambda invocation
role.

## Create the Lambda invocation role 

Temporal invokes Lambda functions by assuming an IAM role in your AWS account. This role needs `lambda:GetFunction` and
`lambda:InvokeFunction` permission on your Worker Lambda functions, and a trust policy that allows the Temporal server's
identity to assume it.

Deploy the following CloudFormation template to create the role.
[Download the template](/files/temporal-self-hosted-serverless-worker-role.yaml). Replace the parameter values in the
command below and run it in your terminal:

```bash
aws cloudformation create-stack \
  --stack-name temporal-serverless-worker \
  --template-body file://temporal-self-hosted-serverless-worker-role.yaml \
  --parameters \
    ParameterKey=TemporalIamRoleArn,ParameterValue=<TEMPORAL_SERVER_ROLE_ARN> \
    ParameterKey=AssumeRoleExternalId,ParameterValue=<EXTERNAL_ID> \
    ParameterKey=LambdaFunctionARNs,ParameterValue='"<LAMBDA_FUNCTION_ARN>"' \
  --capabilities CAPABILITY_NAMED_IAM \
  --region <AWS_REGION>
```

| Parameter              | Description                                                                                                                                                                                                        |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `TemporalIamRoleArn`   | The ARN of the IAM role or user that the Temporal Service runs as. This is the identity the server uses to call `sts:AssumeRole`. To find the ARN, run `aws sts get-caller-identity` in the server's environment.  |
| `AssumeRoleExternalId` | A unique string to prevent [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html) attacks. Choose any value and pass the same value when creating the Worker Deployment Version. |
| `LambdaFunctionARNs`   | Comma-separated list of Lambda function ARNs that Temporal may invoke. To allow invocation of any published version of a function, use a wildcard suffix (for example, `arn:aws:lambda:...:function:my-temporal-worker:*`). |
| `RoleName`             | Base name for the created IAM role. Defaults to `Temporal-Serverless-Worker`. Provide a new role name if creating more than one stack.                                                                                                                                     |

#### CloudFormation template

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Description:
  Creates an IAM role that a self-hosted Temporal Service can assume to invoke Lambda functions for Serverless Workers.

Parameters:
  TemporalIamRoleArn:
    Type: String
    Description: The ARN of the IAM role or user that the Temporal Service runs as.

  AssumeRoleExternalId:
    Type: String
    Description: A unique identifier to prevent confused deputy attacks.
    AllowedPattern: '[a-zA-Z0-9_+=,.@-]*'
    MinLength: 5
    MaxLength: 45

  LambdaFunctionARNs:
    Type: CommaDelimitedList
    Description: >-
      Comma-separated list of Lambda function ARNs to invoke (e.g.,
      arn:aws:lambda:us-west-2:123456789012:function:worker-1,arn:aws:lambda:us-west-2:123456789012:function:worker-2)

  RoleName:
    Type: String
    Default: 'Temporal-Serverless-Worker'

Resources:
  TemporalServerlessWorker:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref RoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS: [!Ref TemporalIamRoleArn]
            Action: sts:AssumeRole
            Condition:
              StringEquals:
                'sts:ExternalId': [!Ref AssumeRoleExternalId]
      Description: 'The role the Temporal Service uses to invoke Lambda functions for Serverless Workers'
      MaxSessionDuration: 3600

  TemporalLambdaInvokePermissions:
    Type: AWS::IAM::Policy
    DependsOn: TemporalServerlessWorker
    Properties:
      PolicyName: 'Temporal-Lambda-Invoke-Permissions'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - lambda:InvokeFunction
              - lambda:GetFunction
            Resource: !Ref LambdaFunctionARNs
      Roles:
        - !Ref RoleName

Outputs:
  RoleARN:
    Description: The ARN of the IAM role created for the Temporal Service
    Value: !GetAtt TemporalServerlessWorker.Arn
    Export:
      Name: !Sub '${AWS::StackName}-RoleARN'

  RoleName:
    Description: The name of the IAM role
    Value: !Ref RoleName

  LambdaFunctionARNs:
    Description: The Lambda function ARNs that can be invoked
    Value: !Join [', ', !Ref LambdaFunctionARNs]
```

After the stack finishes creating, retrieve the IAM role ARN from the stack outputs:

```bash
aws cloudformation describe-stacks \
  --stack-name temporal-serverless-worker \
  --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' \
  --output text \
  --region <AWS_REGION>
```

Use this role ARN when creating the Worker Deployment Version.

## Next steps 

Follow the [AWS Lambda deployment guide](/production-deployment/worker-deployments/serverless-workers/aws-lambda) to
write your Worker code, deploy it to Lambda, and create a Worker Deployment Version with the IAM role from the previous
step.

---

# Deploy a Serverless Worker on GCP Cloud Run

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/cloud-run

> Deploy a Temporal Serverless Worker to a GCP Cloud Run Worker Pool.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

This guide walks through deploying a Temporal [Serverless Worker](/serverless-workers) to a GCP Cloud Run Worker Pool.

A Cloud Run Worker Pool runs long-lived instances that poll the Task Queue continuously. Temporal's
[Worker Controller Instance (WCI)](/serverless-workers#worker-controller-instance) adjusts the pool's instance count
through the Cloud Run admin API as work arrives and drains. For how the pool scales and how instances are shut down, see
[Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run).

Cloud Run runs a standard long-lived Worker, so no Cloud Run-specific handler is needed and any Temporal SDK can run on
a Worker Pool. Serverless Workers require [Worker Versioning](/worker-versioning). The Go, Python, Java, and .NET examples
below also register a Cloud Run ID plugin to give each instance a distinct Worker Identity.

## Prerequisites 

- A Temporal Cloud account with a GCP-hosted Namespace, or a self-hosted Temporal Service v1.31.0 or later. The
  Namespace's cloud provider must match the serverless compute provider.
- For self-hosted deployments, complete the
  [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/cloud-run/self-hosted-setup) before following
  this guide.
- Every Workflow must declare a [versioning behavior](/worker-versioning#versioning-behaviors), or the Worker must set a
  default versioning behavior.
- A GCP project with the Cloud Run and Artifact Registry APIs enabled, and permissions to create Worker Pools, service
  accounts, and Secret Manager secrets.
- The [`gcloud` CLI](https://cloud.google.com/sdk/docs/install) installed and authenticated. You may use other tools to
  perform the GCP steps, such as the Google Cloud console or Terraform.
- [Terraform](https://developer.hashicorp.com/terraform/install) installed. Temporal provides the IAM setup as a
  Terraform module.
- A Temporal SDK. Use the tabs to select your language and the rest of the page will update accordingly.

> **⚠️ Caution:**
> Open Google Cloud issue: high deployment latency in some regions
>
> Google currently reports that creating or updating Cloud Run resources takes longer than expected in some regions,
> including `us-central1`. Google recommends deploying to another region while the issue is open. For the current status,
> see
> [High deployment latency in some regions](https://cloud.google.com/run/docs/known-issues#deployment-latency) in the
> Cloud Run known issues.
>

## 1. Write Worker code 

A Cloud Run Serverless Worker is a standard long-running Worker. It connects to Temporal, registers its Workflows and
Activities, declares its [Worker Deployment Version](/production-deployment/worker-deployments/worker-versioning), and
polls the Task Queue until the instance is shut down.

**Python**

```python
import asyncio
import os

from temporalio.client import Client
from temporalio.common import VersioningBehavior, WorkerDeploymentVersion
from temporalio.contrib.gcp.cloud_run.id import CloudRunIdPlugin
from temporalio.worker import Worker, WorkerDeploymentConfig

from my_workflows import MyWorkflow
from my_activities import my_activity

async def main() -> None:
    client = await Client.connect(
        os.environ["TEMPORAL_ADDRESS"],
        namespace=os.environ["TEMPORAL_NAMESPACE"],
        api_key=os.environ["TEMPORAL_API_KEY"].strip(),
        plugins=[CloudRunIdPlugin()],
        tls=True,
    )
    worker = Worker(
        client,
        task_queue=os.environ["TEMPORAL_TASK_QUEUE"],
        workflows=[MyWorkflow],
        activities=[my_activity],
        deployment_config=WorkerDeploymentConfig(
            version=WorkerDeploymentVersion(
                deployment_name="my-app",
                build_id="build-1",
            ),
            use_worker_versioning=True,
            default_versioning_behavior=VersioningBehavior.PINNED,
        ),
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or
`AUTO_UPGRADE`. Set it per-Workflow in the `@workflow.defn` decorator, or set a Worker-level default with
`default_versioning_behavior` as shown above.

```python
from temporalio import workflow
from temporalio.common import VersioningBehavior

@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class MyWorkflow:
    @workflow.run
    async def run(self, input: str) -> str:
        ...
```

The example uses the Cloud Run ID plugin to identify each Worker as `<instance-id>@<revision>`. For more on the
Python Worker setup, see [Serverless Workers on GCP Cloud Run - Python SDK](/develop/python/workers/serverless-workers/cloud-run).

**Go**

```go
package main

import (
	"log"
	"os"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/contrib/envconfig"
	"go.temporal.io/sdk/contrib/gcp/cloudrun/id"
	"go.temporal.io/sdk/worker"
	"go.temporal.io/sdk/workflow"

	"example.com/myapp"
)

func main() {
	clientOptions, err := envconfig.LoadDefaultClientOptions()
	if err != nil {
		log.Fatalln("Unable to load Temporal client options", err)
	}
	clientOptions.Plugins = append(clientOptions.Plugins, id.NewCloudRunIDPlugin())
	c, err := client.Dial(clientOptions)
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, os.Getenv("TEMPORAL_TASK_QUEUE"), worker.Options{
		DeploymentOptions: worker.DeploymentOptions{
			UseVersioning: true,
			Version: worker.WorkerDeploymentVersion{
				DeploymentName: "my-app",
				BuildID:        "build-1",
			},
		},
	})

	w.RegisterWorkflowWithOptions(myapp.MyWorkflow, workflow.RegisterOptions{
		VersioningBehavior: workflow.VersioningBehaviorPinned,
	})
	w.RegisterActivity(myapp.MyActivity)

	if err := w.Run(worker.InterruptCh()); err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `VersioningBehaviorPinned` or
`VersioningBehaviorAutoUpgrade`. Set it per Workflow at registration as shown above, or set a Worker-level default in
`DeploymentOptions`:

```go
w := worker.New(c, os.Getenv("TEMPORAL_TASK_QUEUE"), worker.Options{
	DeploymentOptions: worker.DeploymentOptions{
		UseVersioning:             true,
		Version:                   version,
		DefaultVersioningBehavior: workflow.VersioningBehaviorPinned,
	},
})
```

If a Version is set and neither is specified, registration panics with `workflow type does not have a versioning behavior`.

The example uses the Cloud Run ID plugin to identify each Worker as `<instance-id>@<revision>`. For more on the
Go Worker setup, see [Serverless Workers on GCP Cloud Run - Go SDK](/develop/go/workers/serverless-workers/cloud-run).

**TypeScript**

```ts
import { NativeConnection, Worker } from '@temporalio/worker';
import * as activities from './activities';

async function run() {
  const connection = await NativeConnection.connect({
    address: process.env.TEMPORAL_ADDRESS,
    apiKey: process.env.TEMPORAL_API_KEY,
    tls: true,
  });

  const worker = await Worker.create({
    connection,
    namespace: process.env.TEMPORAL_NAMESPACE!,
    taskQueue: process.env.TEMPORAL_TASK_QUEUE!,
    workflowsPath: require.resolve('./workflows'),
    activities,
    workerDeploymentOptions: {
      version: { deploymentName: 'my-app', buildId: 'build-1' },
      useWorkerVersioning: true,
      defaultVersioningBehavior: 'PINNED',
    },
  });

  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`.
Setting `defaultVersioningBehavior` as shown above covers every Workflow on the Worker. To set the behavior per Workflow
instead, pass the Workflow function to `setWorkflowOptions()`:

```ts
import { setWorkflowOptions } from '@temporalio/workflow';

setWorkflowOptions({ versioningBehavior: 'PINNED' }, myWorkflow);
export async function myWorkflow(): Promise<string> {
  // ...
}
```

For more on the TypeScript Worker setup, see
[Serverless Workers on GCP Cloud Run - TypeScript SDK](/develop/typescript/workers/serverless-workers/cloud-run).

**Java**

```java
package example;

import io.temporal.client.WorkflowClient;
import io.temporal.common.VersioningBehavior;
import io.temporal.common.WorkerDeploymentVersion;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.gcp.cloudrun.id.CloudRunIdPlugin;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerDeploymentOptions;
import io.temporal.worker.WorkerFactory;
import io.temporal.worker.WorkerOptions;

public class WorkerMain {
  public static void main(String[] args) throws java.io.IOException {
    ClientConfigProfile profile = ClientConfigProfile.load();

    WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder(profile.toWorkflowServiceStubsOptions())
                .setPlugins(new CloudRunIdPlugin())
                .build());

    WorkflowClient client =
        WorkflowClient.newInstance(service, profile.toWorkflowClientOptions());

    WorkerFactory factory = WorkerFactory.newInstance(client);

    Worker worker =
        factory.newWorker(
            System.getenv("TEMPORAL_TASK_QUEUE"),
            WorkerOptions.newBuilder()
                .setDeploymentOptions(
                    WorkerDeploymentOptions.newBuilder()
                        .setUseVersioning(true)
                        .setVersion(new WorkerDeploymentVersion("my-app", "build-1"))
                        .setDefaultVersioningBehavior(VersioningBehavior.PINNED)
                        .build())
                .build());

    worker.registerWorkflowImplementationTypes(MyWorkflowImpl.class);
    worker.registerActivitiesImplementations(new MyActivitiesImpl());

    factory.start();
  }
}
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or
`AUTO_UPGRADE`. Set it per Workflow with the `@WorkflowVersioningBehavior` annotation, or set a Worker-level default
with `setDefaultVersioningBehavior` as shown above.

The example uses the Cloud Run ID plugin to identify each Worker as `<instance-id>@<revision>`. For more on the
Java Worker setup, see [Serverless Workers on GCP Cloud Run - Java SDK](/develop/java/workers/serverless-workers/cloud-run).

**.NET**

```csharp
using Temporalio.Client;
using Temporalio.Common.EnvConfig;
using Temporalio.Extensions.Gcp.CloudRun.Id;
using Temporalio.Worker;

var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.Plugins = new ITemporalClientPlugin[] { new CloudRunIdPlugin() };
var client = await TemporalClient.ConnectAsync(connectOptions);

var options = new TemporalWorkerOptions(Environment.GetEnvironmentVariable("TEMPORAL_TASK_QUEUE")!)
{
    DeploymentOptions = new(new("my-app", "build-1"), useWorkerVersioning: true)
    {
        DefaultVersioningBehavior = Temporalio.Common.VersioningBehavior.Pinned,
    },
};
options.AddWorkflow<MyWorkflow>();
options.AddActivity(MyActivities.Greet);

using var worker = new TemporalWorker(client, options);
await worker.ExecuteAsync(CancellationToken.None);
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `Pinned` or
`AutoUpgrade`. Set it per Workflow with `[Workflow(VersioningBehavior = ...)]`, or set a Worker-level default with
`DefaultVersioningBehavior` as shown above.

The example uses the Cloud Run ID plugin to identify each Worker as `<instance-id>@<revision>`. For more on the
.NET Worker setup, see [Serverless Workers on GCP Cloud Run - .NET SDK](/develop/dotnet/workers/serverless-workers/cloud-run).

**Ruby**

```ruby
require 'temporalio/client'
require 'temporalio/worker'

client = Temporalio::Client.connect(
  ENV.fetch('TEMPORAL_ADDRESS'),
  ENV.fetch('TEMPORAL_NAMESPACE'),
  api_key: ENV.fetch('TEMPORAL_API_KEY'),
  tls: true
)

worker = Temporalio::Worker.new(
  client:,
  task_queue: ENV.fetch('TEMPORAL_TASK_QUEUE'),
  workflows: [MyWorkflow],
  activities: [Greet],
  deployment_options: Temporalio::Worker::DeploymentOptions.new(
    version: Temporalio::WorkerDeploymentVersion.new(
      deployment_name: 'my-app',
      build_id: 'build-1'
    ),
    use_worker_versioning: true,
    default_versioning_behavior: Temporalio::VersioningBehavior::PINNED
  )
)

worker.run
```

Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or
`AUTO_UPGRADE`. Set it per Workflow by calling `workflow_versioning_behavior` in the Workflow class, or set a
Worker-level default with `default_versioning_behavior` as shown above.

For more on the Ruby Worker setup, see
[Serverless Workers on GCP Cloud Run - Ruby SDK](/develop/ruby/workers/serverless-workers/cloud-run).

**Rust**

```rust
use std::str::FromStr;

use temporalio_client::{Client, ClientOptions, Connection, ConnectionOptions, TlsOptions, Url};
use temporalio_common::worker::{
    VersioningBehavior, WorkerDeploymentOptions, WorkerDeploymentVersion,
};
use temporalio_sdk::{Runtime, Worker, WorkerOptions};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let address = std::env::var("TEMPORAL_ADDRESS")?;

    let runtime = Runtime::from_current_tokio(Default::default())?;

    let connection_options = ConnectionOptions::new(Url::from_str(&format!("https://{address}"))?)
        .api_key(std::env::var("TEMPORAL_API_KEY")?)
        .tls_options(TlsOptions::default())
        .build();
    let connection = Connection::connect(connection_options).await?;
    let client = Client::new(
        connection,
        ClientOptions::new(std::env::var("TEMPORAL_NAMESPACE")?).build(),
    )?;

    let worker_options = WorkerOptions::new(std::env::var("TEMPORAL_TASK_QUEUE")?)
        .deployment_options(
            WorkerDeploymentOptions::new(WorkerDeploymentVersion {
                deployment_name: "my-app".to_owned(),
                build_id: "build-1".to_owned(),
            })
            .use_worker_versioning(true)
            .default_versioning_behavior(VersioningBehavior::Pinned)
            .build(),
        )
        .register_workflow::<MyWorkflow>()?
        .register_activities(MyActivities)
        .build();

    let mut worker = Worker::new(&runtime, client, worker_options)?;
    worker.run().await?;

    Ok(())
}
```

The Rust SDK sets the versioning behavior on the Worker rather than per Workflow, so `default_versioning_behavior`
covers every Workflow the Worker registers. Setting it to `VersioningBehavior::Unspecified` is an error at
startup.

Setting `api_key` does not by itself apply TLS on this connection path, so set `tls_options` as well or the Worker
fails with `Connecting to HTTPS without TLS enabled`.

For more on the Rust Worker setup, see
[Serverless Workers on GCP Cloud Run - Rust SDK](/develop/rust/workers/serverless-workers/cloud-run).

The Cloud Run ID plugins in the Go, Python, Java, and .NET examples require the Cloud Run metadata server. To run these
Workers outside Cloud Run, omit the plugin registration and add it back before deploying.

> **💡 Tip:**
>
> The TypeScript, Ruby, and Rust examples use the SDK's default [Worker Identity](/workers#worker-identity). On Cloud Run,
> it can resolve to `1@localhost`, which does not distinguish Worker instances in the Temporal UI. Set a unique Worker
> Identity using [Cloud Run environment variables](https://cloud.google.com/run/docs/container-contract#worker-pools-env-vars)
> and instance metadata.
>

## 2. Deploy to a Cloud Run Worker Pool 

Containerize the Worker, push the image to Artifact Registry, and create the Worker Pool.

### 2.1 Containerize the Worker 

Package the Worker and its dependencies into a container image that Cloud Run runs for each Worker Pool instance. The
image's entrypoint must start your Worker process, so an instance begins polling the Task Queue as soon as it starts.

**Python**

```dockerfile
FROM python:3.12-slim

RUN pip install --no-cache-dir "temporalio>=1.30.0,<2"

WORKDIR /app
COPY . /app

CMD ["python", "-m", "worker"]
```

**Go**

Build a static binary in one stage and copy it into a minimal runtime image:

```dockerfile
FROM golang:1.25 AS build

WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/worker ./worker

FROM gcr.io/distroless/static-debian12
COPY --from=build /out/worker /worker
CMD ["/worker"]
```

`CGO_ENABLED=0` produces a statically linked binary, which is what the `distroless/static` base image expects.

**TypeScript**

Compile the TypeScript in one stage, then install production dependencies in the runtime image:

```dockerfile
FROM node:22-slim AS build

WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-slim

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY --from=build /app/lib ./lib

CMD ["node", "lib/worker.js"]
```

Three details are required here:

- Keep the `ca-certificates` step. Without it the Worker fails at startup with `TransportError: tonic::transport::Error(Transport, NativeCertsNotFound)`.
- Use a glibc-based image, not Alpine. See [Do not use Alpine](/develop/typescript/workers/run-worker-process#do-not-use-alpine).
- Set `NODE_OPTIONS=--max-old-space-size=<MB>` on the Worker Pool to about 80% of the instance's memory limit. See [Run a Worker on Docker](/develop/typescript/workers/run-worker-process#run-a-worker-on-docker).

**Java**

Build a fat jar in one stage and run it on a JRE image:

```dockerfile
FROM maven:3.9-eclipse-temurin-21 AS build

WORKDIR /src
COPY pom.xml ./
RUN mvn -B -q dependency:go-offline
COPY src ./src
RUN mvn -B -q package -DskipTests

FROM eclipse-temurin:21-jre-noble

WORKDIR /app
COPY --from=build /src/target/my-worker.jar /app/worker.jar
CMD ["java", "-XX:MaxRAMPercentage=75", "-jar", "/app/worker.jar"]
```

The JVM reads the container's memory limit but defaults its maximum heap to 25% of it, so set `-XX:MaxRAMPercentage` to
give the Worker more of the instance.

**.NET**

Publish in one stage and run on a .NET runtime image:

```dockerfile
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build

WORKDIR /src
COPY *.csproj ./
RUN dotnet restore
COPY . .
RUN dotnet publish -c Release -o /out

FROM mcr.microsoft.com/dotnet/runtime:10.0

WORKDIR /app
COPY --from=build /out ./
CMD ["dotnet", "MyWorker.dll"]
```

**Ruby**

Install the precompiled gem rather than building the native extension from source:

```dockerfile
FROM ruby:3.3-slim

WORKDIR /app
RUN gem install temporalio --no-document
COPY worker.rb ./

CMD ["ruby", "worker.rb"]
```

Installing through Bundler in a container can select the source gem, which then fails to build without a Rust
toolchain. If you use Bundler, add the target platform to the lockfile with `bundle lock --add-platform x86_64-linux`.

**Rust**

Compile the Worker in one stage and copy the binary into a runtime image:

```dockerfile
FROM rust:1.92-slim AS build

RUN apt-get update \
    && apt-get install -y --no-install-recommends pkg-config libssl-dev protobuf-compiler libprotobuf-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /src
COPY Cargo.toml ./
COPY src ./src
RUN cargo build --release

FROM debian:bookworm-slim

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY --from=build /src/target/release/my-worker /app/worker
CMD ["/app/worker"]
```

The build stage needs `libprotobuf-dev` alongside `protobuf-compiler`, because the compiler package alone installs
`protoc` without the well-known type definitions. The runtime stage needs `ca-certificates`, which
`debian:bookworm-slim` does not ship.

### 2.2 Build and push the image 

Build the image and push it to Artifact Registry:

```bash
gcloud builds submit \
  --tag <REGION>-docker.pkg.dev/<YOUR_GCP_PROJECT>/<REPOSITORY>/my-temporal-worker:build-1 \
  --project <YOUR_GCP_PROJECT> \
  --region <REGION>
```

### 2.3 Create the Worker Pool 

Create an empty Worker Pool. Start it at zero instances: the WCI raises the instance count once the Worker Deployment
Version is current and Tasks arrive.

Create one Worker Pool per Worker Deployment Version. A pool runs a single container image, and a Worker Deployment
Version pins a single build, so each new build needs its own pool. Include the build ID in the pool name to keep that
mapping visible.

#### Choose a runner service account 

The Worker Pool runs as a *runner service account*, the runtime identity your Worker code uses to reach other Google
Cloud services. It is not the invoker service account from [Step 3](#configure-iam), which Temporal impersonates to
scale the pool but never runs it.

Use a dedicated service account for the pool, and
[create one](https://cloud.google.com/iam/docs/service-accounts-create) if you do not already have a suitable account.
You need the email twice: here, and in [Step 3](#configure-iam) as `runner_service_account_email`.

Store the Temporal Cloud API key (or TLS material) in Secret Manager rather than passing it as a plaintext environment
variable.

The runner service account needs no baseline role to run the Worker. Cloud Run collects `stdout` and `stderr` into
[Cloud Logging](https://cloud.google.com/run/docs/logging) through its own infrastructure, and the Cloud Run service
agent, not the runner service account, pulls the container image. Grant the runner service account only what your code
reaches:

- `roles/secretmanager.secretAccessor` on each secret you mount, including the API key above.
- `roles/logging.logWriter`, only if your Worker writes through the Cloud Logging API instead of `stdout` and `stderr`.
- Access to any other Google Cloud service your Workflows and Activities call.

```bash
gcloud run worker-pools deploy my-temporal-worker-pool-build-1 \
  --image <REGION>-docker.pkg.dev/<YOUR_GCP_PROJECT>/<REPOSITORY>/my-temporal-worker:build-1 \
  --region <REGION> \
  --project <YOUR_GCP_PROJECT> \
  --service-account <RUNNER_SERVICE_ACCOUNT> \
  --instances 0 \
  --set-env-vars TEMPORAL_ADDRESS=<your-temporal-address>:7233,TEMPORAL_NAMESPACE=<your-namespace>,TEMPORAL_TASK_QUEUE=my-task-queue \
  --set-secrets TEMPORAL_API_KEY=<SECRET_NAME>:latest
```

| Parameter           | Description                                                                                                             |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `--image`           | The Worker image you pushed in [Step 2.2](#build-and-push).                                                              |
| `--service-account` | The [runner service account](#runner-service-account) the Worker Pool instances run as. It needs read access to the secrets referenced below. This is distinct from the invoker service account in [Step 3](#configure-iam), which Temporal impersonates to scale the pool. |
| `--instances`       | Initial instance count. Set to `0`; the WCI manages the count after the version is current.                             |
| `--set-env-vars`    | Non-secret Worker configuration. See [Environment configuration](/develop/environment-configuration).                   |
| `--set-secrets`     | Maps a Secret Manager secret to an environment variable. Use for `TEMPORAL_API_KEY` or TLS client cert/key material.    |

For the full list of supported environment variables, config file format, and profiles, see
[Environment configuration](/develop/environment-configuration).

## 3. Grant Temporal permission to manage the Worker Pool (Cloud only) 

This section applies to Temporal Cloud. For self-hosted Temporal Service deployments, see
[Self-hosted setup](/production-deployment/worker-deployments/serverless-workers/cloud-run/self-hosted-setup).

Temporal Cloud scales the Worker Pool by [impersonating a service account](https://docs.cloud.google.com/docs/authentication/use-service-account-impersonation) you create, called the *invoker service account*. The invoker service account
reads and scales the pool through the Cloud Run admin API. It does not run the pool. Temporal calls that API from
outside your network, so you do not need to open inbound ports.

> **⚠️ Caution:**
>
> This guide uses two service accounts, and they are not interchangeable:
>
> - The **runner service account** is the identity the Worker Pool runs as. You set it in
>   [Step 2](#create-worker-pool) with `gcloud run worker-pools deploy --service-account`, and it can be an account you
>   already have.
> - The **invoker service account** is the identity Temporal impersonates to read and scale the pool. The Terraform module
>   below creates it, and its email is the `invoker_email` output you give Temporal in
>   [Step 4](#create-worker-deployment-version).
>

When you create a Worker Deployment in the Temporal Cloud UI (**Workers** → **Create Worker Deployment** →
**Access**), it provides a Terraform template. Copy it. The template uses the
[`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run)
module, with `impersonator_service_account_emails` already filled in for your account:

```hcl
module "serverless-worker-cloud-run" {
  source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run"

  project_id         = "<YOUR_GCP_PROJECT>"
  invoker_account_id = "temporal-worker-pool-invoker"

  impersonator_service_account_emails = [
    "<provided by Temporal Cloud>",
  ]

  runner_service_account_email = "temporal-worker-pool-runner@<YOUR_GCP_PROJECT>.iam.gserviceaccount.com"
}
```

The `impersonator_service_account_emails` values are specific to your Temporal Cloud account, which is why the snippet
above shows a placeholder. Copy them from the template in the UI.

Set these variables:

| Variable | Required | Description |
| --- | --- | --- |
| `project_id` | Yes | The GCP project that hosts the Worker Pool and the invoker service account. |
| `invoker_account_id` | Yes | A name for the invoker service account the module creates. The full email becomes `<invoker_account_id>@<project_id>.iam.gserviceaccount.com`. The template supplies a name, so change it only if you want a different one. |
| `impersonator_service_account_emails` | Yes | Temporal Cloud's service accounts, granted `roles/iam.serviceAccountTokenCreator` on the invoker service account so they can impersonate it. Filled in by the template in the UI. |
| `runner_service_account_email` | Yes | The [runner service account](#runner-service-account) from [Step 2](#create-worker-pool). The module grants the invoker service account `roles/iam.serviceAccountUser` on it, which Cloud Run requires to attach that identity when it scales the pool. |
| `invoker_display_name` | No | Display name for the invoker service account. Defaults to `Temporal Serverless Worker Pool Invoker`. |
| `deploy_roles` | No | Project-level Cloud Run roles granted to the invoker service account. Defaults to `roles/run.developer`. Any role you use instead must include `run.workerPools.get` and `run.workerPools.update`. |

Make sure you are logged in to the GCP project, then apply the configuration:

```bash
terraform init
terraform apply
```

Terraform prints an `invoker_email` output. Use it as the `--gcp-cloud-run-service-account` value when you create the
Worker Deployment Version in [Step 4](#create-worker-deployment-version).

## 4. Create Worker Deployment Version 

Create a [Worker Deployment Version](/production-deployment/worker-deployments/worker-versioning) whose compute
configuration points at your Worker Pool. The compute configuration tells Temporal where the pool lives and which
service account to impersonate to manage it. The deployment name and build ID must match the values in your Worker code.

**Temporal Cloud UI**

In the Temporal Cloud UI, go to **Workers** → **Create Worker Deployment** and fill out the required fields:

- **Name** — the Worker Deployment name. Must match `deployment_name` in your Worker code.
- **Build ID** — the version identifier. Must match `build_id` in your Worker code.
- **Compute Provider** — select **Google Cloud Run**.
- **Resource** — the **Project ID**, **Region**, and **Worker Pool** from [Step 2](#create-worker-pool).
- **Access** — the **Service Account** email Temporal Cloud impersonates: the `invoker_email` output from [Step 3](#configure-iam).

**Scaling and Lifecycle** is optional. Leave the defaults unless you need to change them. If your Worker runs
long-running Activities, use Activity Heartbeats so an interrupted Activity resumes from its last recorded progress. See
[GCP Cloud Run lifecycle](/serverless-workers/cloud-run#lifecycle).

**Temporal CLI**

First, create the Worker Deployment if it does not already exist:

```bash
temporal worker deployment create \
  --namespace <YOUR_NAMESPACE> \
  --name my-app
```

Then create the version with the Cloud Run compute configuration:

```bash
temporal worker deployment create-version \
  --namespace <YOUR_NAMESPACE> \
  --deployment-name my-app \
  --build-id build-1 \
  --gcp-cloud-run-project <YOUR_GCP_PROJECT> \
  --gcp-cloud-run-region <REGION> \
  --gcp-cloud-run-worker-pool my-temporal-worker-pool-build-1 \
  --gcp-cloud-run-service-account <INVOKER_SERVICE_ACCOUNT> \
  --gcp-cloud-run-min-instances 0 \
  --gcp-cloud-run-max-instances 30 \
  --gcp-cloud-run-initial-instances 0 \
  --gcp-cloud-run-utilization-target 0.8 \
  --gcp-cloud-run-scale-down-stabilization-duration 90s
```

| Flag                             | Description                                                                                                     |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `--deployment-name`              | Worker Deployment name. Must match `deployment_name` in your Worker code.                                       |
| `--build-id`                     | Worker Deployment Version build ID. Must match `build_id` in your Worker code.                                  |
| `--gcp-cloud-run-project`        | GCP project ID that contains the Worker Pool.                                                                   |
| `--gcp-cloud-run-region`         | Region of the Worker Pool.                                                                                      |
| `--gcp-cloud-run-worker-pool`    | Name of the Worker Pool created in [Step 2](#create-worker-pool).                                               |
| `--gcp-cloud-run-service-account`| The invoker service account Temporal impersonates to read and scale the pool. This is the `invoker_email` output (or the invoker service account you created manually) from [Step 3](#configure-iam). |
| `--gcp-cloud-run-min-instances`  | Fewest instances the WCI keeps in the pool. Defaults to 0.                                                      |
| `--gcp-cloud-run-max-instances`  | Most instances the WCI scales the pool to. Defaults to 30.                                                      |
| `--gcp-cloud-run-initial-instances` | Instance count the pool starts at. Must fall between the minimum and maximum. Defaults to 0.                 |
| `--gcp-cloud-run-utilization-target` | Average Worker utilization the WCI aims for, as a fraction in the range (0, 1]. Lower values hold more spare capacity. Defaults to 0.8. |
| `--gcp-cloud-run-scale-down-stabilization-duration` | How long the WCI waits after the most recent sync match failure before it scales the pool in. Raise it so the pool does not shrink while long-running Activities are still executing. `0s` disables the wait. Defaults to 90s. See [Scale-in](/serverless-workers/cloud-run#scale-in). |

The five scaling flags are optional, and the values above are the defaults. They form one all-or-none group: pass all
five or none of them. The same rule applies to `temporal worker deployment update-version`, where passing none leaves
the version's current scaling settings unchanged. The scale-down stabilization flag requires Temporal CLI v1.8.3 or
later.

To verify that Temporal can reach your Worker Pool, go to **Workers** > **Deployments** > select your deployment > open
the **Actions** menu on the version and click **Validate Connection**. This checks that Temporal can impersonate the
invoker service account and read the pool. It starts no instance, and it does not exercise the update permission that
scaling needs, so a passing validation is not proof that Temporal can scale the pool.

## 5. Set version as current 

Set the version as current. Without this step, Tasks on the Task Queue will not route to the version, and the WCI will
not start any instances.

```bash
temporal worker deployment set-current-version \
  --namespace <YOUR_NAMESPACE> \
  --deployment-name my-app \
  --build-id build-1
```

This command asks you to confirm, because it changes which version new Tasks route to. Pass `--yes` to skip the prompt.

## 6. Verify deployment 

Start a Workflow on the same Task Queue to confirm that the WCI starts a Worker instance and processes the Task.

```bash
temporal workflow start \
  --namespace <YOUR_NAMESPACE> \
  --task-queue my-task-queue \
  --type MyWorkflow \
  --input '"Hello, serverless!"'
```

When Tasks arrive with no active pollers, the WCI raises the Worker Pool's instance count. Cloud Run starts an instance,
the Worker connects to Temporal, picks up the Task, and processes it.

You can confirm this by checking:

- **Temporal UI:** The Workflow execution should show Task completions in the event history.
- **Cloud Run logs:** Once the WCI starts an instance, the Worker Pool's logs
  (`gcloud run worker-pools logs read my-temporal-worker-pool-build-1 --region <REGION> --project <YOUR_GCP_PROJECT>`) show the
  Worker startup and Task processing. The pool produces no logs until an instance is running.

If the Workflow does not progress or no instance starts, see
[Troubleshoot Serverless Workers on GCP Cloud Run](/troubleshooting/serverless-workers/cloud-run).

---

# Self-hosted setup for Serverless Workers on GCP Cloud Run

Source: https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/cloud-run/self-hosted-setup

> Configure a self-hosted Temporal Service to use Serverless Workers with GCP Cloud Run.

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

Serverless Workers require Temporal Service v1.31.0 or later.

This page covers the prerequisites for running [Serverless Workers](/serverless-workers) on a self-hosted Temporal
Service with GCP Cloud Run:

1. Ensure Cloud Run instances can reach the Temporal Service.
2. Enable the Worker Controller Instance (WCI) on the server through dynamic configuration.
3. Give the Temporal Service a GCP identity.
4. Create the invoker service account that the Temporal Service impersonates to scale the Worker Pool.

Once setup is complete, follow the
[GCP Cloud Run deployment guide](/production-deployment/worker-deployments/serverless-workers/cloud-run) to deploy your
Worker.

## Ensure Cloud Run instances can reach the Temporal Service 

The [Temporal Service frontend](/temporal-service/temporal-server#frontend-service) must be reachable from your Cloud
Run Worker Pool instances. How to achieve this depends on your network setup. If the Temporal Service runs on a private
network, you may need [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a
[Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors) so the pool can connect
to the Temporal frontend.

## Enable the Worker Controller Instance 

[WCI](/serverless-workers#how-invocation-works) is the server component that monitors Task Queues and scales compute
providers. It is disabled by default and must be enabled through
[dynamic configuration](/references/dynamic-configuration).

Add the following keys to your dynamic config file:

```yaml
workercontroller.enabled:
  - value: true

workercontroller.compute_providers.enabled:
  - value:
      - gcp-cloud-run

workercontroller.scaling_algorithms.enabled:
  - value:
      - rate-based
```

Cloud Run uses the `rate-based` algorithm. A Cloud Run Worker Pool is a set of long-lived instances, so the WCI resizes
it from arrival and backlog rates. The `no-sync` algorithm applies only to providers that are invoked once per Task, and
pairing it with `gcp-cloud-run` is rejected.

To enable WCI for specific Namespaces instead of globally, add a `constraints` section with the Namespace name under
`workercontroller.enabled`. For example, to enable WCI only for `your-namespace`:

```yaml
workercontroller.enabled:
  - value: true
    constraints:
      namespace: 'your-namespace'
```

The Temporal Service watches the dynamic config file for changes and applies updates without a restart.

Two optional global keys control how the server reaches the invoker when impersonation needs more than one hop:

| Key | Purpose |
| --- | --- |
| `workercontroller.compute_providers.gcp.intermediary_service_accounts` | Service accounts to impersonate in sequence before the invoker. Unset means the server impersonates the invoker directly. |
| `workercontroller.compute_providers.gcp.first_delegate_as_base` | When `true`, the first entry in the chain is the identity the server's ambient credentials impersonate directly, and the rest are passed as token-creator delegates. Defaults to `false`, which passes the whole chain as delegates from the ambient credentials. |

## Give the Temporal Service a GCP identity 

The Temporal Service reads and scales the Worker Pool by impersonating the invoker service account you create in the
next step. To do that, the server must first run as a GCP identity that is allowed to impersonate the invoker.

**On GCP infrastructure (GCE, GKE):** The server uses the attached service account through
[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials)
automatically. No additional credential configuration is needed. You grant this attached service account permission to
impersonate the invoker in the next step.

**Outside GCP:** Use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) so
the server can obtain GCP credentials without a long-lived key. Point `GOOGLE_APPLICATION_CREDENTIALS` at the credential
configuration file it produces. That variable also accepts a service account key file, but Google
[recommends against long-lived keys](https://cloud.google.com/iam/docs/best-practices-service-accounts).

## Create the invoker service account 

The Temporal Service scales the Worker Pool as an *invoker* service account. This account only reads and scales the
pool; it does not run the pool. The identity your Worker instances run as is the separate runner service account you
set on the pool in the [deployment guide](/production-deployment/worker-deployments/serverless-workers/cloud-run#create-worker-pool).

Two grants make this work:

- The GCP identity the Temporal Service runs as (from the previous step) receives `roles/iam.serviceAccountTokenCreator`
  on the invoker, so the server can impersonate it.
- The invoker receives a project-level Cloud Run role with at least `run.workerPools.get` and `run.workerPools.update`.
  `roles/run.developer` includes both.

Temporal publishes a Terraform module that creates the invoker service account and applies both grants:
[`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run).
For self-hosted deployments, pass the GCP identity the Temporal Service runs as in `impersonator_service_account_emails`:

```hcl
module "serverless-worker-cloud-run" {
  source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run"

  project_id         = "<YOUR_GCP_PROJECT>"
  invoker_account_id = "temporal-serverless-worker"

  runner_service_account_email = "<REPLACE-WITH-WORKER-POOL-RUNNER-SERVICE-ACCOUNT-EMAIL>"

  impersonator_service_account_emails = [
    "<TEMPORAL_SERVICE_GCP_IDENTITY>",
  ]
}
```

After you apply the module, use its `invoker_email` output as the `--gcp-cloud-run-service-account` value when you create
the Worker Deployment Version.

## Next steps 

Follow the [GCP Cloud Run deployment guide](/production-deployment/worker-deployments/serverless-workers/cloud-run) to
write your Worker code, deploy it to a Worker Pool, and create a Worker Deployment Version with the invoker service
account from the previous step.

---

# Migrating from Unversioned to Versioned Temporal Workers

Source: https://docs.temporal.io/production-deployment/worker-deployments/unversioned-to-versioned-migration

This guide will help you implement Worker Versioning when the Temporal Worker Controller isn't used. If you are using the Temporal Worker Controller, follow [this guide](https://github.com/temporalio/temporal-worker-controller/blob/main/docs/migration-to-versioned.md).

## Prerequisites

- Unversioned Temporal Workers currently running in production
- Temporal CLI >= 1.5.0
- Workers that connect to Temporal with Namespace and Task Queue configuration

## Key steps

- Ensure your versioned Worker code is backward-compatible with existing Workflow histories.
- Deploy the versioned Worker. It won't receive Tasks until you activate it.
- Use ramping to gradually shift traffic before full cutover.
- Signal sleeping or idle Workflows to wake them up and migrate them to the versioned Worker.
- Keep unversioned Workers running during the transition period.
- Test thoroughly in a non-production environment before migrating production Workers.

### Step 1: Update your Worker code

Update your Worker initialization to include versioning configuration.

**Before (Unversioned):**

```go
// Worker connects without versioning
worker := worker.New(client, "my-task-queue", worker.Options{})
```

**After (Versioned):**

```go
buildID := os.Getenv("TEMPORAL_WORKER_BUILD_ID")
deploymentName := os.Getenv("TEMPORAL_DEPLOYMENT_NAME")
if buildID == "" || deploymentName == "" {
    // exit with an error
}

workerOptions := worker.Options{
    DeploymentOptions: worker.DeploymentOptions{
        UseVersioning: true,
        Version: worker.WorkerDeploymentVersion{
            DeploymentName: deploymentName,
            BuildID:        buildID,
        },
    },
}
worker := worker.New(client, "my-task-queue", workerOptions)
```

> **ℹ️ Info:**
> Important
>
> Your versioned Worker code must be fully backward-compatible with existing unversioned Workflow histories to avoid non-determinism errors. Don't make breaking Workflow code changes at this stage.
>

### Step 2: Deploy your versioned Worker

Deploy your versioned Worker alongside your existing unversioned Workers. The versioned Worker will begin polling but **won't receive any Tasks** until you explicitly activate it via the CLI.

You can verify it's polling by inspecting the Worker Deployment:

```shell
temporal worker deployment describe --name "YourDeploymentName"
```

### Step 3: Gradually ramp traffic (optional, but recommended)

Instead of cutting over all at once, ramp a small percentage of new Workflow executions to the versioned Worker first:

```shell
temporal worker deployment set-ramping-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID" \
    --percentage=5
```

Then monitor Workflows on the new version:

```shell
temporal workflow describe -w YourWorkflowID
```

This returns versioning info such as:

```
Versioning Info:

  Behavior               AutoUpgrade
  Version                YourDeploymentName.YourBuildID
  OverrideBehavior       Unspecified
```

Increase the ramp percentage incrementally as you test and see that your Workflows are behaving as expected.

### Step 4: Set the versioned Worker as Current

Once validated, promote the versioned Worker to receive 100% of new Workflow executions:

```shell
temporal worker deployment set-current-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID"
```

> **📝 Note:**
>
> Once a Current version is set, **unversioned Workers** will no longer receive any Tasks. Ensure your versioned Workers are healthy before this step.
>

### Step 5: Migrate unversioned in-flight Workflows

After setting the Current version, unversioned in-flight Workflows aren't dropped. On their next Task execution, they will automatically be routed to the versioned Worker. Once they are queued up on a versioned Worker, they will become either *Pinned* or *AutoUpgrade* depending on the Workflow's versioning behavior annotation.

Sleeping or idle Workflows will not automatically begin to receive the new version information. If you have Workflows that are sleeping or waiting for an event, you must send them a Signal to wake them up so they can be dispatched to the versioned Worker on their next Task execution.

Here's an example of how to Signal all running Workflows at once:

```shell
temporal workflow signal \
    --query "ExecutionStatus='Running'" \
    --name "wake-up" \
    --namespace production \
    --rps 100
```

Once signaled, those Workflows will execute a Workflow Task and be routed to the Current versioned Worker. Keep your unversioned Workers running until all in-flight Workflows have migrated over.

### Step 6: Scale down and clean up unversioned Workers

Once you confirm that all Workflows are handled by versioned Workers, shut down
your old unversioned Worker deployments.

---

# Worker Versioning

Source: https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning

Worker Versioning is a Temporal feature that allows you to confidently deploy new changes to the Workflows running on
your Workers without breaking them. Temporal enables this by helping you manage different builds or versions, formally
called [Worker Deployment Versions](/worker-versioning#deployment-versions).

For most teams, Worker Versioning should be the default recommendation for deploying Workflow code changes in
production. If you can run versioned worker deployments, prefer Worker Versioning over patching.

Worker Versioning provides the following benefits for [blue-green or rainbow deployments](#deployment-systems):

- Ramping traffic gradually to a new Worker Deployment Version.
- Verifying a new Deployment Version with tests before sending production traffic to it.
- Instant rollback when you detect that a new Deployment Version is broken.
- Improved error rates when adopting it.

In addition, Worker Versioning introduces **Workflow Pinning**. For pinned Workflow Types, each execution runs entirely
on the Worker Deployment Version where it started. You need not worry about making breaking code changes to running,
pinned Workflows.

To use Workflow Pinning, we recommend using [rainbow deployments](#deployment-systems).

> **💡 Tip:**
>
> Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo.
>
> [Watch: Worker Versioning: a Temporal Replay 2025 talk and demo](https://www.youtube.com/watch?v=rm4BlD9WXqc)
>

## Minimum versions

- Go SDK version [v1.35.0](https://github.com/temporalio/sdk-go/releases/tag/v1.35.0)
- Python [v1.11](https://github.com/temporalio/sdk-python/releases/tag/1.11.0)
- Java [v1.29](https://github.com/temporalio/sdk-java/releases/tag/v1.29.0)
- TypeScript [v1.12](https://github.com/temporalio/sdk-typescript/releases/tag/v1.12.0)
- .NET [v1.7.0](https://github.com/temporalio/sdk-dotnet/releases/tag/1.7.0)
- Ruby [v0.5.0](https://github.com/temporalio/sdk-ruby/releases/tag/v0.5.0)

Self-hosted users:

- Minimum Temporal CLI version [v1.4.1](https://github.com/temporalio/cli/releases/tag/v1.4.1)
- Minimum Temporal Server version: [v1.29.1](https://github.com/temporalio/temporal/releases/tag/v1.29.1)
- Minimum Temporal UI version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0)

## Getting Started with Worker Versioning 

To get started with Worker Versioning, you should understand some concepts around versioning and deployments.

- A **Worker Deployment** is a deployment or service across multiple versions. In a rainbow deployment, more than two
  active Deployment Versions can run at once.
- A **Worker Deployment Version** is a version of a deployment or service. It can have multiple Workers polling on
  multiple Task Queues, but they all run the same build.
- A **Build ID**, in combination with a Worker Deployment name, identifies a single Worker Deployment Version.
- When a versioned worker polls on a task queue, that task queue becomes part of that Worker's version. That version's
  Worker Deployment controls how the task queue matches Workflow Tasks with Workers.
- Using **Workflow Pinning**, you can declare each Workflow type to have a **Versioning Behavior**, either Pinned or
  Auto-Upgrade.
  - A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version.
  - An **Auto-Upgrade** Workflow will automatically move to a new code version as you roll it out, specifically its
    Target Worker Deployment Version (defined below). Therefore, Auto-Upgrade Workflows are not restricted to a single
    Deployment Version and need to be kept replay-safe manually, that is with
    [patching](/workflow-definition#workflow-versioning).
  - Both Pinned and Auto-Upgrade Workflows are guaranteed to start only on the Current or Ramping Version of their
    Worker Deployment.
  - Pinned Workflows are designed for use with rainbow deployments. See [Deployment Systems](#deployment-systems).
  - Pinned Workflows don't need to be patched, as they run on the same worker and build until they complete.
  - If you expect your Workflow to run longer than you want your Worker Deployment Versions to exist, you should mark
    your Workflow Type as Auto-Upgrade.
- Each Worker Deployment has a single [**Current Version**](/worker-versioning#versioning-definitions) which is where
  Workflows are routed to unless they were previously pinned on a different version.
- Each Worker Deployment can have a [**Ramping Version**](/worker-versioning#versioning-definitions) which is where a
  configurable percentage of Workflows are routed to unless they were previously pinned on a different version.
- For a given Workflow, its [**Target Worker Deployment Version**](/worker-versioning#versioning-definitions) is the
  version it will move to next.

> **⚠️ Caution:**
> Before you enable Worker Versioning
>
> Worker Versioning has [limits](/evaluate/cloud/limits#worker-versioning-level) on the number of deployments, versions, and Task Queues. Review those limits before enabling versioning.
>
> In particular, do not enable Worker Versioning for worker-specific Task Queues — versioning is not relevant for such queues, and it will unnecessarily exhaust the [per-version Task Queue limit](/evaluate/cloud/limits#max-task-queues-in-deployment-version-limits).
>

## Setting up your deployment system 

If you haven't already, you'll want to pick a container deployment solution for your Workers.

You also need to pick among three common deployment strategies:

- A **rolling deployment** strategy upgrades Workers in place with little control over how quickly they cut over and
  only a slow ability to roll Workers back. Rolling deploys have minimal footprint but tend to provide lower
  availability than the other strategies and are incompatible with Worker Versioning.
- A **blue-green deployment** strategy maintains two "colors," or Worker Deployment Versions simultaneously and can
  control how traffic is routed between them. This allows you to maximize your uptime with features like instant
  rollback and ramping. Worker Versioning enables the routing control that blue-green deployments need.
- A **rainbow deployment** strategy is like blue-green but with more colors, allowing Workflow Pinning. You can deploy
  new revisions of your Workflows freely while older versions drain. Using Worker Versioning, Temporal lets you know
  when all the Workflows of a given version are drained so that you can sunset it.

> **📝 Note:**
>
> You also have the option to use the
> [Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to automatically enable
> rainbow deployments of your Workers if you're using Kubernetes.
>

If you cannot yet support blue-green or rainbow style deployments, use [patching](/patching) as a fallback while you
work toward a versioned deployment model.

## Choosing a Versioning Behavior 

The right versioning behavior depends on how long your Workflows run relative to your deployment frequency.

### Decision guide 

| Workflow Duration                        | Uses Continue-as-New? | Recommended Behavior                                     | Patching Required? |
| ---------------------------------------- | --------------------- | -------------------------------------------------------- | ------------------ |
| **Short** (completes before next deploy) | N/A                   | `PINNED`                                                 | Never              |
| **Medium** (spans multiple deploys)      | No                    | `AUTO_UPGRADE`                                           | Yes                |
| **Long** (weeks to years)                | Yes                   | `PINNED` + [upgrade on CaN](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new) | Never              |
| **Long** (weeks to years)                | No                    | `AUTO_UPGRADE` + patching                                | Yes                |

### Examples by Workflow type 

| Workflow Type        | Duration     | Recommended Behavior       | Notes                               |
| -------------------- | ------------ | -------------------------- | ----------------------------------- |
| Order processing     | Minutes      | `PINNED`                   | Completes before next deploy        |
| Payment retry        | Hours        | `PINNED` or `AUTO_UPGRADE` | Depends on deploy frequency         |
| Subscription billing | Days         | `AUTO_UPGRADE`             | May span multiple deploys           |
| Customer entity      | Months-Years | `PINNED` + upgrade on CaN  | Uses Continue-as-New pattern        |
| AI agent / Chatbot   | Weeks        | `PINNED` + upgrade on CaN  | Long sleeps, uses CaN               |
| Compliance audit     | Months       | `AUTO_UPGRADE` + patching  | Cannot use CaN (needs full history) |

> **ℹ️ Info:**
> Long-running Workflows with Continue-as-New
>
> If your Workflow uses Continue-as-New to manage history size, you can upgrade to new Worker Deployment Versions at the
> CaN boundary without patching. See [Upgrading on Continue-as-New](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new).
>

### Default versioning behavior considerations

If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning.
Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`.

In general, each Workflow Type should be annotated as Auto-Upgrade or Pinned. If all of your Workflows will be
short-running for the foreseeable future, you can default to Pinned.

Many users who are migrating to Worker Versioning will start by defaulting to Auto-Upgrade until they have had time to
annotate their Workflows. This default is the most similar to the legacy behavior. Once each Workflow Type is annotated,
you can remove the `DefaultVersioningBehavior`.

There is a possibility of a queue blocking limitation for new or Auto-Upgrade Workflows if there is a ramp, but one of
the Current or Ramping versions is down or doesn't have enough capacity. This leads to other versions not getting Tasks
or slowing down.

For example, you have a Current Version and a Ramping Version at 50%. If all of your Current Version Workers go down,
you would expect at least 50% of new Workflows to go to the Ramping Version. This won't happen because the Tasks for the
Current Version are blocking the queue.

> **📝 Note:**
>
> Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow default to Auto-Upgrade behavior and not
> Unspecified.
>

You also want to make sure you understand how your Activities are going to work across different Worker Deployment
Versions. Refer to the [Worker Versioning Activity behavior docs](/worker-versioning#activity-behavior-across-versions)
for more details.

If you run [Serverless Workers](/serverless-workers), each Worker Deployment Version's compute provider points at a
specific Lambda function. In production, configure it with a qualified versioned ARN so Pinned Workflows remain truly
pinned to the code they started against, regardless of whether your default behavior is Pinned or Auto-Upgrade. See
[Worker Versioning with Serverless Workers](/serverless-workers#worker-versioning-with-serverless-workers).

---

# Configure a Worker for Versioning

Source: https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning/configure-worker

> Enable Worker Versioning on your Workers with UseVersioning, Deployment Version, and default Versioning Behavior.

You'll need to add a few additional configuration parameters to your Workers to toggle on Worker Versioning. There are
three new parameters, with different names depending on the language:

- `UseVersioning`: This enables the Versioning functionality for this Worker.
- A `Version` to identify the revision that this Worker will be allowed to execute. This is a combination of a
  deployment name and a build ID number.
- (Optional) The [Default Versioning Behavior](/production-deployment/worker-deployments/worker-versioning#definition). If unset, you'll be required to specify the behavior on
  each Workflow. Or you can default to Pinned or Auto-Upgrade.

Follow the example for your SDK below. Select a concept to highlight the matching lines:

**Go**

  annotations={[
    {
      label: 'UseVersioning',
      description:
        'Enables Worker Versioning for this Worker so Tasks are matched to Deployment Versions.',
      lines: [4],
    },
    {
      label: 'Version',
      description:
        'Identifies the revision this Worker may execute: a deployment name plus a Build ID.',
      lines: [5, 6, 7, 8],
    },
    {
      label: 'Default Versioning Behavior',
      description:
        'Optional. If unset, you must set the Versioning Behavior on each Workflow. Otherwise default to Pinned or Auto-Upgrade.',
      lines: [9],
    },
  ]}
>
```go
buildID := mustGetEnv("MY_BUILD_ID")
w := worker.New(c, myTaskQueue, worker.Options{
  DeploymentOptions: worker.DeploymentOptions{
    UseVersioning: true,
    Version: worker.WorkerDeploymentVersion{
      DeploymentName: "llm_srv",
      BuildID: buildID,
    },
    DefaultVersioningBehavior: workflow.VersioningBehaviorUnspecified,
  },
})
```

**Java**

  annotations={[
    {
      label: 'UseVersioning',
      description:
        'Enables Worker Versioning for this Worker so Tasks are matched to Deployment Versions.',
      lines: [10],
    },
    {
      label: 'Version',
      description:
        'Identifies the revision this Worker may execute: a deployment name plus a Build ID.',
      lines: [9],
    },
    {
      label: 'Default Versioning Behavior',
      description:
        'Optional. If unset, you must set the Versioning Behavior on each Workflow. Otherwise default to Pinned or Auto-Upgrade.',
      lines: [11],
    },
  ]}
>
```java
import io.temporal.worker.WorkerOptions;
import io.temporal.common.VersioningBehavior;
import io.temporal.common.WorkerDeploymentVersion;
import io.temporal.worker.WorkerDeploymentOptions;

WorkerOptions.newBuilder()
  .setDeploymentOptions(
      WorkerDeploymentOptions.newBuilder()
      .setVersion(new WorkerDeploymentVersion("llm_srv", "1.0"))
      .setUseVersioning(true)
      .setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE)
      .build())
  .build();
```

**Python**

  annotations={[
    {
      label: 'UseVersioning',
      description:
        'Enables Worker Versioning for this Worker so Tasks are matched to Deployment Versions.',
      lines: [13],
    },
    {
      label: 'Version',
      description:
        'Identifies the revision this Worker may execute: a deployment name plus a Build ID.',
      lines: [10, 11, 12],
    },
    {
      label: 'Default Versioning Behavior',
      description:
        'Optional. If unset, you must set the Versioning Behavior on each Workflow. Otherwise default to Pinned or Auto-Upgrade.',
      lines: [14],
    },
  ]}
>
```python
from temporalio.common import WorkerDeploymentVersion, VersioningBehavior
from temporalio.worker import Worker, WorkerDeploymentConfig

Worker(
    client,
    task_queue="mytaskqueue",
    workflows=workflows,
    activities=activities,
    deployment_config=WorkerDeploymentConfig(
        version=WorkerDeploymentVersion(
            deployment_name="llm_srv",
            build_id=my_env.build_id),
        use_worker_versioning=True,
        default_versioning_behavior=VersioningBehavior.UNSPECIFIED
    ),
)
```

**TypeScript**

  annotations={[
    {
      label: 'UseVersioning',
      description:
        'Enables Worker Versioning for this Worker so Tasks are matched to Deployment Versions.',
      lines: [5],
    },
    {
      label: 'Version',
      description:
        'Identifies the revision this Worker may execute: a deployment name plus a Build ID.',
      lines: [6],
    },
    {
      label: 'Default Versioning Behavior',
      description:
        'Optional. This TypeScript example does not set a default; set Versioning Behavior on each Workflow, or add a default when your SDK supports it on Worker options.',
      lines: [],
    },
  ]}
>
```ts
const myWorker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  taskQueue,
  workerDeploymentOptions: {
    useWorkerVersioning: true,
    version: { buildId: '1.0', deploymentName: 'llm_srv' },
  },
  connection: nativeConnection,
});
```

**.NET**

  annotations={[
    {
      label: 'UseVersioning',
      description:
        'Enables Worker Versioning for this Worker so Tasks are matched to Deployment Versions.',
      lines: [5],
    },
    {
      label: 'Version',
      description:
        'Identifies the revision this Worker may execute: a deployment name plus a Build ID.',
      lines: [5],
    },
    {
      label: 'Default Versioning Behavior',
      description:
        'Optional. If unset, you must set the Versioning Behavior on each Workflow. Otherwise default to Pinned or Auto-Upgrade.',
      lines: [7],
    },
  ]}
>
```csharp
var myWorker = new TemporalWorker(
    Client,
    new TemporalWorkerOptions(taskQueue)
    {
      DeploymentOptions = new(new("llm_srv", "1.0"), true)
      {
        DefaultVersioningBehavior = VersioningBehavior.Unspecified,
      },
    }.AddWorkflow<MyWorkflow>());
```

**Ruby**

  annotations={[
    {
      label: 'UseVersioning',
      description:
        'Enables Worker Versioning for this Worker so Tasks are matched to Deployment Versions.',
      lines: [10],
    },
    {
      label: 'Version',
      description:
        'Identifies the revision this Worker may execute: a deployment name plus a Build ID.',
      lines: [6, 7, 8, 9],
    },
    {
      label: 'Default Versioning Behavior',
      description:
        'Optional. If unset, you must set the Versioning Behavior on each Workflow. Otherwise default to Pinned or Auto-Upgrade.',
      lines: [11],
    },
  ]}
>
```ruby
worker = Temporalio::Worker.new(
  client: client,
  task_queue: task_queue,
  workflows: [MyWorkflow],
  deployment_options: Temporalio::Worker::DeploymentOptions.new(
      version: Temporalio::WorkerDeploymentVersion.new(
          deployment_name: 'llm_srv',
          build_id: '1.0'
      ),
      use_worker_versioning: true,
      default_versioning_behavior: Temporalio::VersioningBehavior::UNSPECIFIED
  )
)
```

---

# Roll out and pin Workflows

Source: https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin

> Roll out Worker Deployment Versions with the CLI, mark Workflow Types as Pinned, move pinned Workflows, and migrate to Auto-Upgrade.

This page covers Rolling out changes with the CLI, Marking a Workflow Type as Pinned, Moving a pinned Workflow, and Migrating a Workflow from Pinned to Auto-Upgrade.

## Rolling out changes with the CLI 

Next, deploy your Worker with the additional configuration parameters. Before making any Workflow revisions, you can use
the `temporal` CLI to check which of your Worker versions are currently polling:

You can view the Versions that are part of a Deployment with `temporal worker deployment describe`:

```bash
temporal worker deployment describe --name="$MY_DEPLOYMENT"
```

To activate a Deployment Version, use `temporal worker deployment set-current-version`, specifying the deployment name
and a Build ID:

```bash
temporal worker deployment set-current-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID"
```

To ramp a Deployment Version up to some percentage of your overall Worker fleet, use `set-ramping version`, with the
same parameters and a ramping percentage:

```bash
temporal worker deployment set-ramping-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID" \
    --percentage=5
```

You can verify that Workflows are cutting over to that version with `describe -w YourWorkflowID`:

```bash
temporal workflow describe -w YourWorkflowID
```

That returns the new Version that the workflow is running on:

```
Versioning Info:

  Behavior               AutoUpgrade
  Version                llm_srv.2.0
  OverrideBehavior       Unspecified
```

## Marking a Workflow Type as Pinned

You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. This will cause it
to remain on its original deployed version:

**Go**

```go
// w is the Worker configured as in the previous example
w.RegisterWorkflowWithOptions(HelloWorld, workflow.RegisterOptions{
	// or workflow.VersioningBehaviorAutoUpgrade
    VersioningBehavior: workflow.VersioningBehaviorPinned,
})
```

**Java**

```java
@WorkflowInterface
public interface HelloWorld {
    @WorkflowMethod
    String hello();
}

public static class HelloWorldImpl implements HelloWorld {
    @Override
    @WorkflowVersioningBehavior(VersioningBehavior.PINNED)
    public String hello() {
        return "Hello, World!";
    }
}

```

**Python**

```python
@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class HelloWorld:
    @workflow.run
    async def run(self):
        return "hello world!"

```

**TypeScript**

```ts
setWorkflowOptions({ versioningBehavior: 'PINNED' }, helloWorld);
export async function helloWorld(): Promise<string> {
  return 'hello world!';
}
```

**.NET**

```csharp
[Workflow(VersioningBehavior = VersioningBehavior.Pinned)]
public class HelloWorld
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        return "hello world!";
    }
}
```

**Ruby**

```ruby
class HelloWorld < Temporalio::Workflow::Definition
  workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED

  def execute
    'hello world!'
  end
end
```

## Moving a pinned Workflow 

Sometimes you'll need to manually move a set of pinned Workflows off of a version that has a bug to a version with the
fix.

If you need to move a pinned Workflow to a new version, use `temporal workflow update-options`:

```bash
temporal workflow update-options \
    --workflow-id "$WORKFLOW_ID" \
    --versioning-override-behavior pinned \
    --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \
    --versioning-override-build-id "$TARGET_BUILD_ID"
```

You can move several Workflows at once matching a `--query` parameter:

```bash
temporal workflow update-options \
  --query="TemporalWorkerDeploymentVersion=$TARGET_DEPLOYMENT:$BAD_BUILD_ID" \
  --versioning-override-behavior pinned \
  --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \
  --versioning-override-build-id "$FIXED_BUILD_ID"
```

In this scenario, you may also need to use the other [Versioning APIs](/workflow-definition#workflow-versioning) to
patch your Workflow in the "fixed" build, so that your target Worker can handle the moved Workflows correctly. If you
made a [version-incompatible change](/workflow-definition#deterministic-constraints) to your Workflow, and you want to
roll back to an earlier version, it's not possible to patch it. Considering using
[Workflow Reset](/workflow-execution/event#reset) along with your move.

"Reset-with-Move" allows you to atomically Reset your Workflow and set a Versioning Override on the newly reset
Workflow, so when it resumes execution, all new Workflow Tasks will be executed on your new Worker.

```bash
temporal workflow reset with-workflow-update-options \
    --workflow-id "$WORKFLOW_ID" \
    --event-id "$EVENT_ID" \
    --reason "$REASON" \
    --versioning-override-behavior pinned \
    --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \
    --versioning-override-build-id "$TARGET_BUILD_ID"
```

For a complete runbook covering batch recovery, drainage handling, and how to choose between Versioning Override and Reset-with-Move based on each Workflow's state, see [Recover pinned Workflows after a bad rollout](/production-deployment/worker-deployments/recover-pinned-workflows).

## Migrating a Workflow from Pinned to Auto-Upgrade

There may be times when you need to migrate your Workflow from Pinned to Auto-Upgrade because you configured your
Workflow Type with the wrong behavior or you've pinned a really long-running Workflow by mistake.

Pinned Workflows can block version drainage, especially when they run for a long time. You could move the Workflow to a
new build, but that would just push the problem to the next build.

In order to make this change, you need to change the versioning behavior for your Workflow from Pinned to Auto-Upgrade.
You can use `temporal workflow update-options` for this:

```bash
temporal workflow update-options \
    --workflow-id "$WORKFLOW_ID" \
    --versioning-override-behavior auto_upgrade
```

If you want to move all your Workflows of a certain type to this new configuration, you can do it with this command:

```bash
temporal workflow update-options \
    --query="WorkflowType='$WORKFLOW_TYPE'" \
    --versioning-override-behavior auto_upgrade
```

You can also filter on a certain build ID to limit the number of Workflows you apply it to:

```bash
temporal workflow update-options \
    --query="WorkflowType='$WORKFLOW_TYPE' AND TemporalWorkerDeploymentVersion='$TARGET_DEPLOYMENT:$OLD_VERSION'" \
    --versioning-override-behavior auto_upgrade
```

> **📝 Note:**
>
> When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the Workflow's Target Version is different from the earlier Pinned Version, you should make sure you [patch](/patching#patching) the Workflow code.
>

---

# Sunset and garbage collection

Source: https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning/sunset-and-gc

> Drain and sunset old Worker Deployment Versions, then garbage-collect unused versions.

This page covers Sunsetting an old Deployment Version and Garbage collection.

## Sunsetting an old Deployment Version 

A Worker Deployment Version moves through the following states:

1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never
   becomes Active, it will never be Draining or Drained.
2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing Auto-Upgrade
   Workflows.
3. **Draining**: The version stopped being Current or Ramping, and it has open pinned Workflows running on it. It is
   possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated
   periodically.
4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed.

You can see these statuses when you describe a Worker Deployment in the `WorkerDeploymentVersionStatus` of each
`VersionSummary`, or by describing the version directly. When a version is Draining or Drained, that is displayed in a
value called `DrainageStatus`. Periodically, the Temporal Service will refresh this status by counting any open pinned
Workflows using that version.

On each refresh, `DrainageInfo.last_checked_time` is updated. Eventually, `DrainageInfo` will report that the version is
fully drained. At this point, no Workflows are still running on that version and no more will be automatically routed to
it, so you can consider shutting down the running Workers.

You can monitor this by checking `WorkerDeploymentInfo.VersionSummaries` or with
`temporal worker deployment describe-version`:

```bash
temporal worker deployment describe-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID"
```

```
Worker Deployment Version:
  Version                  llm_srv.1.0
  CreateTime               5 hours ago
  RoutingChangedTime       32 seconds ago
  RampPercentage           0
  DrainageStatus           draining
  DrainageLastChangedTime  31 seconds ago
  DrainageLastCheckedTime  31 seconds ago

Task Queues:
     Name        Type
  hello-world  activity
  hello-world  workflow
```

If you have implemented [Queries](/sending-messages#sending-queries) on closed pinned Workflows, you may need to keep
some Workers running to handle them.

### Adding a pre-deployment test

Before deploying a new Workflow revision, you can test it with synthetic traffic.

To do this, use pinning in your tests, following the examples below

**Go**

```go
workflowOptions := client.StartWorkflowOptions{
	ID:        "MyWorkflowId",
	TaskQueue: "MyTaskQueue",
	VersioningOverride: &client.PinnedVersioningOverride{
        Version: worker.WorkerDeploymentVersion{
            DeploymentName: "DeployName",
            BuildID: "1.0",
        },
    },
}
// c is an initialized Client
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, HelloWorld, "Hello")
```

**Java**

```java
MyWorkflow handle = client.newWorkflowStub(
    MyWorkflow.class,
    WorkflowOptions.newBuilder()
        .setWorkflowId("MyWorkflowId")
        .setTaskQueue("MyTaskQueue")
        .setVersioningOverride(new VersioningOverride.PinnedVersioningOverride(
            new WorkerDeploymentVersion("DeployName", "1.0")))
        .build()
);
WorkflowExecution we = WorkflowClient.start(handle::execute, "Hello");
```

**Python**

```python
handle = client.start_workflow(
    MyWorkflow.run,
    "Hello",
    id="MyWorkflowId",
    task_queue="MyTaskQueue",
    versioning_override=PinnedVersioningOverride(
        WorkerDeploymentVersion("DeployName", "1.0")
    ),
)
```

**TypeScript**

```ts
const handle = await client.workflow.start('helloWorld', {
  taskQueue: 'MyTaskQueue',
  workflowId: 'MyWorkflowId',
  versioningOverride: {
    pinnedTo: { buildId: '1.0', deploymentName: 'deploy-name' },
  },
});
```

**.NET**

```csharp
var workerV1 = new WorkerDeploymentVersion("deploy-name", "1.0");
var handle = await Client.StartWorkflowAsync(
    (HelloWorld wf) => wf.RunAsync(),
    new(id: "MyWorkflowId", taskQueue: "MyTaskQueue")
    {
        VersioningOverride = new VersioningOverride.Pinned(workerV1),
    });
```

**Ruby**

```ruby
worker_v1 = Temporalio::WorkerDeploymentVersion.new(
  deployment_name: 'deploy-name',
  build_id: '1.0'
)
handle = env.client.start_workflow(
  HelloWorld,
  id: 'MyWorkflowId',
  task_queue: 'MyTaskQueue',
  versioning_override: Temporalio::VersioningOverride.pinned(worker_v1)
)
```

## Garbage collection

Worker Deployments are never garbage collected, but _Worker Deployment Versions_ (often referred to as Versions, Worker
Versions, Deployment Versions) are.

Versions are deleted to keep the total number of versions in one Worker Deployment less than or equal to
[`matching.maxVersionsInDeployment`](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L1501-L1505),
which is currently set to 100 in Temporal Cloud, but that's a conservative number and it could be increased if needed.

For example, when you deploy your 101st Worker Version in a Worker Deployment, the server looks at the oldest drained
version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it. If that version
still has pollers, the server will try the next oldest version. If none of the 100 versions are eligible for deletion
(that is, none of them are drained with no pollers), then no version will be deleted and the poll from the 101st version
would fail.

At that point, to successfully deploy your 101st version, you would need to increase `matching.maxVersionsInDeployment`
or stop polling from one of the old drained versions to make it eligible for clean up.

If you want to re-deploy a previously deleted version, start polling with a Worker that has the same build ID and
Deployment Name as the deleted version and the server will recreate it.

This covers the complete lifecycle of working with Worker Versioning. We are continuing to improve this feature, and we
welcome any feedback or feature requests using the sidebar link!

---

# Upgrade on Continue-as-New

Source: https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new

> Upgrade pinned Workflows to a new Worker Deployment Version by Continue-as-New when a Target Version becomes available.

This page covers how to upgrade pinned Workflows to a new Worker Deployment Version with Continue-as-New.

## Upgrading on Continue-As-New 

Long-running Workflows that use [Continue-as-New](/workflow-execution/continue-as-new) can upgrade to newer Worker
Deployment Versions at Continue-as-New boundaries without requiring patching.

This pattern is ideal for:

- **Entity Workflows** that run for months or years
- **Batch processing** Workflows that checkpoint with Continue-as-New
- **AI agent Workflows** with long sleeps waiting for user input

> **📝 Note:**
> Public Preview
>
> This feature is in Public Preview as an experimental SDK-level option.
>

### How it works 

By default, Pinned Workflows stay on their original Worker Deployment Version even when they Continue-as-New. With the
upgrade option enabled:

1. Each Workflow run remains pinned to its version (no patching needed during a run)
2. The Temporal Server tells the workflow when a new [Target Version](/worker-versioning#versioning-definitions) becomes available
3. When the Workflow performs Continue-as-New with the upgrade option, the new run starts on the [Target Version](/worker-versioning#versioning-definitions)

### Checking for new versions 

When a new Worker Deployment Version becomes Current or Ramping, active Workflows can detect this through
`target_worker_deployment_version_changed`:

**Go**

```go
func (w *Workflows) ContinueAsNewWithVersionUpgradeV1(
  ctx workflow.Context,
  attempt int,
) (string, error) {
  if attempt > 0 {
    return "v1.0", nil
  }

	// Check GetTargetWorkerDeploymentVersionChanged periodically.
	// GetTargetWorkerDeploymentVersionChanged is refreshed after each WFT completes.
  for {
	// Trigger a WFT when timer expires, thereby refreshing the GetTargetWorkerDeploymentVersionChanged flag.
	// Since this is just a test workflow, we aren't doing any real work. In a real workflow regularly
	// doing non-sleep workflow tasks, you would not need to artificially trigger a WFT to refresh the
	// GetTargetWorkerDeploymentVersionChanged flag. You could choose to check the field periodically, or you
	// might want to check before accepting updates, starting activities, or starting child workflows.
	err := workflow.Sleep(ctx, 10*time.Millisecond)
	if err != nil {
	  return "", err
	}
	info := workflow.GetInfo(ctx)
	if info.GetTargetWorkerDeploymentVersionChanged() {
	  return "", workflow.NewContinueAsNewErrorWithOptions(
		ctx,
		workflow.ContinueAsNewErrorOptions{
		  // Pass InitialVersioningBehavior=workflow.ContinueAsNewVersioningBehaviorAutoUpgrade
		  // to make the new run start with AutoUpgrade behavior and use the Target Version of
		  // its Worker Deployment.
		  InitialVersioningBehavior: workflow.ContinueAsNewVersioningBehaviorAutoUpgrade,
		},
		"ContinueAsNewWithVersionUpgrade",
		attempt+1,
	  )
	}
  }
}

func (w *Workflows) ContinueAsNewWithVersionUpgradeV2(
  ctx workflow.Context,
  attempt int,
) (string, error) {
  return "v2.0", nil
}
```

**Java**

```java
public class ContinueAsNewWithVersionUpgradeImpl implements ContinueAsNewWithVersionUpgrade {
  @Override
  public String run(int attempt) {
    if (attempt > 0) {
      return "v1.0";
    }

    // isTargetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes.
    // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial
    // timer; you could check the flag periodically, or before accepting Updates, starting
    // Activities, or starting child Workflows.
    while (true) {
      Workflow.sleep(Duration.ofMillis(10));
      if (Workflow.getInfo().isTargetWorkerDeploymentVersionChanged()) {
        // Set InitialVersioningBehavior to AUTO_UPGRADE so the new run starts with AutoUpgrade
        // behavior and uses the Target Version of its Worker Deployment.
        Workflow.continueAsNew(
            ContinueAsNewOptions.newBuilder()
                .setInitialVersioningBehavior(InitialVersioningBehavior.AUTO_UPGRADE)
                .build(),
            attempt + 1);
      }
    }
  }
}
```

**Python**

```python
@workflow.defn
class ContinueAsNewWithVersionUpgrade:
    @workflow.run
    async def run(self, attempt: int) -> str:
        if attempt > 0:
            return "v1.0"

        # is_target_worker_deployment_version_changed() is refreshed after each Workflow Task
        # completes. In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need
        # an artificial timer; you could check the flag periodically, or before accepting Updates,
        # starting Activities, or starting child Workflows.
        while True:
            await workflow.sleep(timedelta(milliseconds=10))
            if workflow.info().is_target_worker_deployment_version_changed():
                # Set initial_versioning_behavior to AUTO_UPGRADE so the new run starts with
                # AutoUpgrade behavior and uses the Target Version of its Worker Deployment.
                workflow.continue_as_new(
                    attempt + 1,
                    initial_versioning_behavior=ContinueAsNewVersioningBehavior.AUTO_UPGRADE,
                )
```

**TypeScript**

```ts
import * as wf from '@temporalio/workflow';
import { InitialVersioningBehavior } from '@temporalio/common';

export async function continueAsNewWithVersionUpgrade(attempt: number): Promise<string> {
  if (attempt > 0) {
    return 'v1.0';
  }

  // targetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes.
  // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial
  // timer; you could check the flag periodically, or before accepting Updates, starting
  // Activities, or starting child Workflows.
  for (;;) {
    await wf.sleep('10ms');
    if (wf.workflowInfo().targetWorkerDeploymentVersionChanged) {
      // Set initialVersioningBehavior to AUTO_UPGRADE so the new run starts with AutoUpgrade
      // behavior and uses the Target Version of its Worker Deployment.
      return await wf.makeContinueAsNewFunc<typeof continueAsNewWithVersionUpgrade>({
        initialVersioningBehavior: InitialVersioningBehavior.AUTO_UPGRADE,
      })(attempt + 1);
    }
  }
}
```

**.NET**

```csharp
[Workflow]
public class ContinueAsNewWithVersionUpgrade
{
    [WorkflowRun]
    public async Task<string> RunAsync(int attempt)
    {
        if (attempt > 0)
        {
            return "v1.0";
        }

        // TargetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes.
        // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an
        // artificial timer; you could check the flag periodically, or before accepting Updates,
        // starting Activities, or starting child Workflows.
        while (true)
        {
            await Workflow.DelayAsync(TimeSpan.FromMilliseconds(10));
            if (Workflow.TargetWorkerDeploymentVersionChanged)
            {
                // Set InitialVersioningBehavior to AutoUpgrade so the new run starts with
                // AutoUpgrade behavior and uses the Target Version of its Worker Deployment.
                throw Workflow.CreateContinueAsNewException(
                    (ContinueAsNewWithVersionUpgrade wf) => wf.RunAsync(attempt + 1),
                    new ContinueAsNewOptions
                    {
                        InitialVersioningBehavior = InitialVersioningBehavior.AutoUpgrade,
                    });
            }
        }
    }
}
```

**Ruby**

```ruby
class ContinueAsNewWithVersionUpgrade < Temporalio::Workflow::Definition
  workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED

  def execute(attempt)
    return 'v1.0' if attempt.positive?

    # target_worker_deployment_version_changed? is refreshed after each Workflow Task completes.
    # In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial
    # timer; you could check the flag periodically, or before accepting Updates, starting
    # Activities, or starting child Workflows.
    loop do
      Temporalio::Workflow.sleep(0.01)
      next unless Temporalio::Workflow.target_worker_deployment_version_changed?

      # Set initial_versioning_behavior to AUTO_UPGRADE so the new run starts with AutoUpgrade
      # behavior and uses the Target Version of its Worker Deployment.
      raise Temporalio::Workflow::ContinueAsNewError.new(
        attempt + 1,
        initial_versioning_behavior: Temporalio::ContinueAsNewVersioningBehavior::AUTO_UPGRADE
      )
    end
  end
end
```

### Limitations 

> **⚠️ Caution:**
> Current Limitations
>
> - **Lazy moving only:** Workflows must execute a step to receive the target-version-changed information. Sleeping
>   Workflows won't proactively get it. If you have idle Workflows that you want to wake up so they can check the
>   target-version-changed flag, you can send them a Signal.
> - **Interface compatibility:** When continuing as new to a different version, ensure your Workflow input provided by the
>   previous version's workflow definition is compatible with the new version's workflow definition. If incompatible, the
>   new run may fail on its first Workflow Task.
>

---

# Quickstarts

Source: https://docs.temporal.io/quickstarts

> Start building with Temporal using language-specific quickstarts.

Choose your language to get started locally. If you're using Temporal Cloud, check out [Get started with Temporal Cloud](/cloud/get-started).

## Quickstarts by SDK

- [Go](/develop/go/set-up-your-local-go): Install the Go SDK and run a Hello World Workflow in Go.
- [Java](/develop/java/set-up-your-local-java): Install the Java SDK and run a Hello World Workflow in Java.
- [.NET](/develop/dotnet/set-up-your-local-dotnet): Install the .NET SDK and run a Hello World Workflow in C#.
- [PHP](/develop/php/set-up-your-local-php): Install the PHP SDK and run a Hello World Workflow in PHP.
- [Python](/develop/python/set-up-your-local-python): Install the Python SDK and run a Hello World Workflow in Python.
- [Ruby](/develop/ruby/set-up-local-ruby): Install the Ruby SDK and run a Hello World Workflow in Ruby.
- [Rust](/develop/rust/quickstart): Install the Rust SDK and run a Hello World Workflow in Rust.
- [TypeScript](/develop/typescript/set-up-your-local-typescript): Install the TypeScript SDK and run a Hello World Workflow in TypeScript.

## Quickstarts by feature

Learn Temporal features with guided local walkthroughs.

### Standalone Activities

Run an Activity from a Temporal Client without writing a Workflow.

- [Go](/develop/go/activities/standalone-activities-quickstart): Execute a Standalone Activity with the Go SDK.
- [Java](/develop/java/activities/standalone-activities-quickstart): Execute a Standalone Activity with the Java SDK.
- [.NET](/develop/dotnet/activities/standalone-activities-quickstart): Execute a Standalone Activity with the .NET SDK.
- [Python](/develop/python/activities/standalone-activities-quickstart): Execute a Standalone Activity with the Python SDK.
- [Ruby](/develop/ruby/activities/standalone-activities-quickstart): Execute a Standalone Activity with the Ruby SDK.
- [Rust](/develop/rust/activities/standalone-activities-quickstart): Execute a Standalone Activity with the Rust SDK.
- [TypeScript](/develop/typescript/activities/standalone-activities-quickstart): Execute a Standalone Activity with the TypeScript SDK.

### Nexus
Use [Temporal Nexus](/evaluate/features/nexus) to connect Temporal Applications within and across Namespaces.

- [Go](/develop/go/nexus/quickstart): Build a Nexus Service with the Go SDK.
- [Java](/develop/java/nexus/quickstart): Build a Nexus Service with the Java SDK.
- [.NET](/develop/dotnet/nexus/quickstart): Build a Nexus Service with the .NET SDK.
- [Python](/develop/python/nexus/quickstart): Build a Nexus Service with the Python SDK.
- [TypeScript](/develop/typescript/nexus/quickstart): Build a Nexus Service with the TypeScript SDK.

- [Pair with an AI coding agent](/with-ai): Give Claude Code, Cursor, or Codex Temporal expertise with Skills and real-time documentation access.

---

# Temporal Platform references

Source: https://docs.temporal.io/references

- [API reference](/references/api-reference): API documentation for the Temporal SDKs, Temporal Server, Temporal CLI, and Temporal Cloud.
- [Temporal Service metrics](/references/service-metrics): Metrics a self-hosted Temporal Service emits, covering request rates, latencies, errors, and Nexus Operations.
- [SDK metrics](/references/sdk-metrics): Metrics emitted by Temporal Clients and Workers, with each metric's type and SDK availability.
- [Environment configuration](/references/client-environment-configuration): Environment variables and TOML keys that configure a Temporal Client or the Temporal CLI.
- [Temporal Service configuration](/references/service-configuration): The development.yaml settings for global options, persistence, service roles, and archival.
- [Dynamic configuration](/references/dynamic-configuration): Keys that tune rate limits, size limits, and retry defaults without restarting the Temporal Service.
- [Temporal Web UI configuration](/references/web-ui-configuration): YAML keys for the Web UI Server, covering Frontend connectivity, Workflow actions, auth, TLS, CORS, and Codec Server.
- [Web UI environment variables](/references/web-ui-environment-variables): Environment variables that configure the Temporal Web UI in Docker, such as TEMPORAL_ADDRESS and TLS.
- [Server options](/references/server-options): ServerOption functions for embedding the Temporal Server in a Go application, including auth, TLS, and metrics.
- [Commands](/references/commands): Every Command a Worker can issue after completing a Workflow Task, and its corresponding Event.
- [Errors](/references/workflow-task-errors): Workflow Task failure causes and Resource Exhausted causes, with how to resolve each one.
- [Events](/references/events): Every Event that can appear in a Workflow Execution's Event History.
- [Failures](/references/failures): Failure types for Workflows, Activities, and Nexus Operations, and their SDK classes or exceptions.
- [Temporal Cloud operations](/references/operation-list): Operations Temporal Cloud rate-limits per Namespace, with each operation's priority and throttling effect.
- [Glossary](/glossary): Definitions of terms used across the Temporal Platform.

---

# API reference

Source: https://docs.temporal.io/references/api-reference

> Links to API references for the eight Temporal SDKs, the Temporal Server gRPC API, the Temporal CLI, and Temporal Cloud management tools.

API documentation for the Temporal SDKs, Temporal Server, Temporal CLI, and Temporal Cloud.

## SDK API references

- [.NET SDK API](https://dotnet.temporal.io/api/): .NET SDK namespaces, classes, and methods.
- [Go SDK API](https://pkg.go.dev/go.temporal.io/sdk): Go SDK packages, types, and methods on pkg.go.dev.
- [Java SDK API](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/index.html): Java SDK classes, interfaces, and annotations on javadoc.io.
- [PHP SDK API](https://php.temporal.io/namespaces/temporal.html): PHP SDK namespaces, classes, and interfaces.
- [Python SDK API](https://python.temporal.io/): Python SDK modules, classes, and functions.
- [Ruby SDK API](https://ruby.temporal.io/): Ruby SDK modules, classes, and methods.
- [Rust SDK API](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/): Rust SDK Workflow and Worker modules, structs, and methods on docs.rs.
- [Rust Client API](https://docs.rs/temporalio-client/latest/temporalio_client/): Rust Temporal Client for starting and managing Workflow Executions, on docs.rs.
- [TypeScript SDK API](https://typescript.temporal.io): TypeScript SDK interfaces, types, and namespaces.

## Server API references

- [Server Frontend API](/self-hosted-guide/server-frontend-api-reference): gRPC API reference used by Client and Worker SDKs to communicate with Temporal Server.
- [Generated gRPC API docs](https://api-docs.temporal.io/): Generated reference for every Temporal Server gRPC service, method, and message.
- [Server API protobuf definitions](https://github.com/temporalio/api): Source protobuf files for the Temporal Server API, for generating clients in any gRPC-supported language.

## CLI references

- [Temporal CLI command reference](/cli/command-reference): Commands and options for the Temporal CLI, including the development server and Workflow management.
- [Temporal CLI cloud command reference](/cli/command-reference/cloud): Commands for the temporal cloud extension, which manages Temporal Cloud accounts, Namespaces, and access.
- [tcld command reference](/cloud/tcld): Commands for tcld, the standalone Temporal Cloud CLI.

## Temporal Cloud API references

- [Cloud Ops API](/ops): HTTP and gRPC API for managing Temporal Cloud Namespaces, Users, Service Accounts, and API keys.
- [Terraform provider](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs): Resource and data source reference for managing Temporal Cloud with Terraform.
- [OpenMetrics API](/cloud/metrics/openmetrics/api-reference): Endpoint, authentication, and limits for querying Temporal Cloud metrics in OpenMetrics format.

## Need help?

Use **Ask AI** in the top navigation, connect the [Temporal Docs MCP server](/with-ai) to your AI tools, or ask in the [Community Forum](https://community.temporal.io/) or [Slack](https://temporal.io/slack).

---

# Client environment configuration reference

Source: https://docs.temporal.io/references/client-environment-configuration

> Environment variables that configure Temporal SDK Clients and the Temporal CLI, with each one's TOML key, CLI flag, and the clients that read it.

> **ℹ️ Info:**
>
> For how to use these variables with TOML configuration files and profiles, see [Environment configuration](/develop/environment-configuration).
>

Environment variables configure how a [Temporal Client](/encyclopedia/temporal-client) or the Temporal CLI connects to the Temporal Service.
Most variables have a matching key in a TOML configuration file and a matching CLI flag.

When the same setting comes from more than one source, the client uses the first one it finds in this order:

1. A CLI flag passed on the command line.
2. An environment variable.
3. A value in a TOML configuration file.

Each variable on this page lists its **TOML key**, its **CLI flag**, and the clients that read it.
`Read by: every client` means every SDK and the Temporal CLI.

## Configuration file variables

### `TEMPORAL_CONFIG_FILE`

Path to the TOML configuration file.
The default is `temporal.toml` in a platform-specific directory.

- TOML key: none
- CLI flag: `--config-file`
- Read by: every client

### `TEMPORAL_PROFILE`

Name of the configuration profile to load.
The default is `default`.

- TOML key: none
- CLI flag: `--profile`
- Read by: every client

## Connection variables

### `TEMPORAL_ADDRESS`

Host and port of the Temporal Frontend Service, such as `localhost:7233`.

- TOML key: `profile.<name>.address`
- CLI flag: `--address`
- Read by: every client

### `TEMPORAL_NAMESPACE`

Temporal Namespace to connect to.

- TOML key: `profile.<name>.namespace`
- CLI flag: `--namespace`
- Read by: every client

### `TEMPORAL_API_KEY`

API key for authentication.
When you set an API key, the client enables TLS by default.

- TOML key: `profile.<name>.api_key`
- CLI flag: `--api-key`
- Read by: every client

### `TEMPORAL_CLIENT_AUTHORITY`

Overrides the `:authority` gRPC header.
Only the Go SDK reads this variable.
The Temporal CLI has a global `--client-authority` flag, but the flag doesn't read this variable or the TOML key.

- TOML key: `profile.<name>.authority`
- CLI flag: none
- Read by: Go

## TLS variables

### `TEMPORAL_TLS`

Set to `true` to enable TLS or `false` to disable it.
The TOML key inverts the value, so `disabled = true` turns TLS off.

- TOML key: `profile.<name>.tls.disabled`
- CLI flag: `--tls`
- Read by: every client

### `TEMPORAL_TLS_CLIENT_CERT_PATH`

Path to the client's public TLS certificate.
Don't set it together with `TEMPORAL_TLS_CLIENT_CERT_DATA`.

- TOML key: `profile.<name>.tls.client_cert_path`
- CLI flag: `--tls-cert-path`
- Read by: every client

### `TEMPORAL_TLS_CLIENT_CERT_DATA`

PEM data for the client's public TLS certificate.
Don't set it together with `TEMPORAL_TLS_CLIENT_CERT_PATH`.

- TOML key: `profile.<name>.tls.client_cert_data`
- CLI flag: `--tls-cert-data`
- Read by: every client

### `TEMPORAL_TLS_CLIENT_KEY_PATH`

Path to the client's private TLS key.
Don't set it together with `TEMPORAL_TLS_CLIENT_KEY_DATA`.

- TOML key: `profile.<name>.tls.client_key_path`
- CLI flag: `--tls-key-path`
- Read by: every client

### `TEMPORAL_TLS_CLIENT_KEY_DATA`

PEM data for the client's private TLS key.
Don't set it together with `TEMPORAL_TLS_CLIENT_KEY_PATH`.

- TOML key: `profile.<name>.tls.client_key_data`
- CLI flag: `--tls-key-data`
- Read by: every client

### `TEMPORAL_TLS_SERVER_CA_CERT_PATH`

Path to the Certificate Authority (CA) certificate that verifies the server.
Don't set it together with `TEMPORAL_TLS_SERVER_CA_CERT_DATA`.

- TOML key: `profile.<name>.tls.server_ca_cert_path`
- CLI flag: `--tls-ca-path`
- Read by: every client

### `TEMPORAL_TLS_SERVER_CA_CERT_DATA`

PEM data for the CA certificate that verifies the server.
Don't set it together with `TEMPORAL_TLS_SERVER_CA_CERT_PATH`.

- TOML key: `profile.<name>.tls.server_ca_cert_data`
- CLI flag: `--tls-ca-data`
- Read by: every client

### `TEMPORAL_TLS_SERVER_NAME`

Overrides the server name sent for Server Name Indication (SNI) in the TLS handshake.

- TOML key: `profile.<name>.tls.server_name`
- CLI flag: `--tls-server-name`
- Read by: every client

### `TEMPORAL_TLS_DISABLE_HOST_VERIFICATION`

Set to `true` to skip verifying the server's hostname against its certificate.
This exposes the connection to man-in-the-middle attacks, so use it only for testing.
Not every SDK applies this setting.

- TOML key: `profile.<name>.tls.disable_host_verification`
- CLI flag: `--tls-disable-host-verification`
- Read by: every client

## gRPC metadata variables

### `TEMPORAL_GRPC_META_*`

Sets a gRPC header on every request.
The part of the name after `_META_` becomes the header key, lowercased with underscores replaced by hyphens.
For example, `TEMPORAL_GRPC_META_SOME_KEY` sets the `some-key` header.

- TOML key: `profile.<name>.grpc_meta`
- CLI flag: `--grpc-meta`
- Read by: every client except CLI

## Codec Server variables

### `TEMPORAL_CODEC_ENDPOINT`

Endpoint of a remote [Codec Server](/codec-server).
SDKs that read this variable don't apply the codec by default, so set it mainly for the Temporal CLI.

- TOML key: `profile.<name>.codec.endpoint`
- CLI flag: `--codec-endpoint`
- Read by: every client except Java

### `TEMPORAL_CODEC_AUTH`

Authorization header value sent to the remote Codec Server.

- TOML key: `profile.<name>.codec.auth`
- CLI flag: `--codec-auth`
- Read by: every client except Java

## `temporal env` preset variables

The Temporal CLI has a second configuration mechanism alongside TOML configuration files.
`temporal env` stores named key-value presets in `temporal.yaml`.
The variables in this section select which preset the CLI reads.

Both mechanisms work.
The `temporal config` command that manages TOML configuration files is experimental, and `temporal env` is not.

### `TEMPORAL_ENV`

Name of the active `temporal env` preset.
The default is `default`.

- TOML key: none
- CLI flag: `--env`
- Read by: Temporal CLI

### `TEMPORAL_ENV_FILE`

Path to the preset file.
The default is `temporal.yaml` in the CLI configuration directory.

- TOML key: none
- CLI flag: `--env-file`
- Read by: Temporal CLI

## Legacy TLS variables

The Temporal CLI still reads an older set of TLS variable names.
A legacy variable overrides the matching value from the configuration file and from the current variable.
Legacy variables have no CLI flag or TOML key of their own.
Use the current variable named in each entry instead.

### `TEMPORAL_TLS_CERT`

Superseded by `TEMPORAL_TLS_CLIENT_CERT_PATH`.

- Read by: Temporal CLI

### `TEMPORAL_TLS_CERT_DATA`

Superseded by `TEMPORAL_TLS_CLIENT_CERT_DATA`.

- Read by: Temporal CLI

### `TEMPORAL_TLS_KEY`

Superseded by `TEMPORAL_TLS_CLIENT_KEY_PATH`.

- Read by: Temporal CLI

### `TEMPORAL_TLS_KEY_DATA`

Superseded by `TEMPORAL_TLS_CLIENT_KEY_DATA`.

- Read by: Temporal CLI

### `TEMPORAL_TLS_CA`

Superseded by `TEMPORAL_TLS_SERVER_CA_CERT_PATH`.

- Read by: Temporal CLI

### `TEMPORAL_TLS_CA_DATA`

Superseded by `TEMPORAL_TLS_SERVER_CA_CERT_DATA`.

- Read by: Temporal CLI

---

# Temporal Commands reference

Source: https://docs.temporal.io/references/commands

> Reference for every Command a Worker can issue to the Temporal Service after completing a Workflow Task, including its corresponding Event.

A [Command](/workflow-execution#command) is an action a [Worker](/workers#worker) requests from the [Temporal Service](/temporal-service) when it completes a [Workflow Task Execution](/tasks#workflow-task-execution).
The SDK generates Commands from your Workflow code, such as a call to execute an Activity or start a Timer.
The Temporal Service records an [Event](/workflow-execution/event#event) in the [Event History](/workflow-execution/event#event-history) for each Command.

Commands are defined in the [`CommandType` enum](https://github.com/temporalio/api/blob/master/temporal/api/enums/v1/command_type.proto) and [command messages](https://github.com/temporalio/api/blob/master/temporal/api/command/v1/message.proto) of the Temporal gRPC API.

Each Command on this page lists the following:

- **Awaitable:** Whether the Workflow can wait on the result of the Command.
- **Corresponding Event:** The Event the Temporal Service records for the Command.
- **Limit:** The default maximum number of pending operations of this type per Workflow Execution, if one applies.
  When a Workflow Execution reaches the limit, the Workflow Task fails and is retried.
  See [Workflow Execution limits](/workflow-execution/limits#workflow-execution-limits).

## Workflow Execution close Commands

Each of these Commands closes the Workflow Execution.
Its corresponding Event is the last Event in the Event History.

### CompleteWorkflowExecution

The Worker issues this Command when the Workflow Function returns.

- Awaitable: No
- Corresponding Event: [WorkflowExecutionCompleted](/references/events#workflowexecutioncompleted)

### ContinueAsNewWorkflowExecution

The Worker issues this Command when the Workflow calls [Continue-As-New](/workflow-execution/continue-as-new).

- Awaitable: No
- Corresponding Event: [WorkflowExecutionContinuedAsNew](/references/events#workflowexecutioncontinuedasnew)

### FailWorkflowExecution

The Worker issues this Command when the Workflow returns an error or throws an exception that fails the Workflow Execution.

- Awaitable: No
- Corresponding Event: [WorkflowExecutionFailed](/references/events#workflowexecutionfailed)

### CancelWorkflowExecution

The Worker issues this Command when the Workflow finishes cleaning up after a Cancellation Request.
The request appears in the Event History as a [WorkflowExecutionCancelRequested](/references/events#workflowexecutioncancelrequested) Event.

- Awaitable: No
- Corresponding Event: [WorkflowExecutionCanceled](/references/events#workflowexecutioncanceled)

## Activity Commands

### ScheduleActivityTask

The Worker issues this Command when the Workflow calls an [Activity](/activities).

- Awaitable: Yes
- Corresponding Event: [ActivityTaskScheduled](/references/events#activitytaskscheduled)
- Limit: 2,000 pending Activities

### RequestCancelActivityTask

The Worker issues this Command when the Workflow requests cancellation of an [Activity Task](/tasks#activity-task).

- Awaitable: No
- Corresponding Event: [ActivityTaskCancelRequested](/references/events#activitytaskcancelrequested)

## Timer Commands

### StartTimer

The Worker issues this Command when the Workflow starts a Timer, such as a call to sleep.

- Awaitable: Yes
- Corresponding Event: [TimerStarted](/references/events#timerstarted)

### CancelTimer

The Worker issues this Command when the Workflow cancels a Timer.

- Awaitable: No
- Corresponding Event: [TimerCanceled](/references/events#timercanceled)

## Child and external Workflow Commands

### StartChildWorkflowExecution

The Worker issues this Command when the Workflow starts a [Child Workflow Execution](/child-workflows).

- Awaitable: Yes
- Corresponding Event: [StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated)
- Limit: 2,000 pending Child Workflows

### SignalExternalWorkflowExecution

The Worker issues this Command when the Workflow [sends a Signal](/sending-messages#sending-signals) to another Workflow Execution.

- Awaitable: Yes
- Corresponding Event: [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated)
- Limit: 2,000 pending Signals to other Workflows

### RequestCancelExternalWorkflowExecution

The Worker issues this Command when the Workflow requests cancellation of another Workflow Execution.

- Awaitable: Yes
- Corresponding Event: [RequestCancelExternalWorkflowExecutionInitiated](/references/events#requestcancelexternalworkflowexecutioninitiated)
- Limit: 2,000 pending Cancellation Requests to other Workflows

## Nexus Commands

### ScheduleNexusOperation

The Worker issues this Command when the caller Workflow executes a Nexus Operation.

- Awaitable: Yes
- Corresponding Event: [NexusOperationScheduled](/references/events#nexusoperationscheduled)
- Limit: 30 pending Nexus Operations.
  See [Nexus Operation limits](/workflow-execution/limits#workflow-execution-nexus-operation-limits).

### RequestCancelNexusOperation

The Worker issues this Command when the caller Workflow requests cancellation of a Nexus Operation.

- Awaitable: No
- Corresponding Event: [NexusOperationCancelRequested](/references/events#nexusoperationcancelrequested)

## Workflow metadata and SDK Commands

### RecordMarker

The SDK issues this Command to record data it needs to replay the Workflow deterministically, such as the result of a Side Effect, a Local Activity, or a version check.
Your Workflow code doesn't issue it directly.

- Awaitable: No
- Corresponding Event: [MarkerRecorded](/references/events#markerrecorded)

### UpsertWorkflowSearchAttributes

The Worker issues this Command when the Workflow adds or updates its [Search Attributes](/search-attribute).

- Awaitable: No
- Corresponding Event: [UpsertWorkflowSearchAttributes](/references/events#upsertworkflowsearchattributes)

### ModifyWorkflowProperties

The Worker issues this Command when the Workflow adds or updates its Memo.
The new values merge with the existing Memo.

- Awaitable: No
- Corresponding Event: [WorkflowPropertiesModified](/references/events#workflowpropertiesmodified)

### ProtocolMessage 

The Worker issues this Command to keep protocol messages, such as responses to Updates, in order with the Workflow's other Commands.

The Command points to a protocol message, and the Temporal Service creates the Event from that message.
The Command alone doesn't determine the resulting Event type.
For Updates, the Events are [WorkflowExecutionUpdateAccepted](/references/events#workflowexecutionupdateacceptedevent) and [WorkflowExecutionUpdateCompleted](/references/events#workflowexecutionupdatecompletedevent).

---

# Temporal Service dynamic configuration reference

Source: https://docs.temporal.io/references/dynamic-configuration

> Dynamic configuration keys for a self-hosted Temporal Service, with file format, Namespace and Task Queue constraints, and default rate and size limits.

> **ℹ️ Info:**
>
> The information on this page is relevant to open source [Temporal Service deployments](/temporal-service).
>
> For settings that require a restart, see the [Temporal Service configuration reference](/references/service-configuration).
>

[Dynamic configuration](/temporal-service/configuration#dynamic-configuration) keys change the behavior of a running Temporal Service without a restart.
Every key has a default value.
You override a default by setting the key in a dynamic configuration YAML file.
The Temporal Service checks the file for changes at the `pollInterval` set in the [`dynamicConfigClient`](/references/service-configuration#dynamicconfigclient) section of its static configuration.

Keys and their defaults are defined in [constants.go](https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go) in the Temporal Server repository.
Some keys are defined next to the component they configure, such as the Nexus keys under `components/` and `chasm/lib/`.
Keys can be added, renamed, or removed between releases, so check the [Temporal Server release notes](https://github.com/temporalio/temporal/releases) before you upgrade.

Setting dynamic configuration is optional.
Override a default only when your workload needs it, and test the change before you apply it in production.

## Format

A dynamic configuration file maps each key to a list of values.
Each value can have `constraints` that limit where it applies.
A value without constraints applies across the Temporal Service.

```yaml
testGetBoolPropertyKey:
  - value: false
  - value: true
    constraints:
      namespace: 'your-namespace'
  - value: false
    constraints:
      namespace: 'your-other-namespace'
testGetDurationPropertyKey:
  - value: '1m'
    constraints:
      namespace: 'your-namespace'
      taskQueueName: 'longIdleTimeTaskqueue'
testGetFloat64PropertyKey:
  - value: 12.0
    constraints:
      namespace: 'your-namespace'
testGetMapPropertyKey:
  - value:
      key1: 1
      key2: 'value 2'
      key3:
        - false
        - key4: true
          key5: 2.0
```

For a complete file, see the [dynamic configuration sample](https://github.com/temporalio/samples-server/blob/main/tls/config/dynamicconfig/development.yaml) in the `samples-server` repository.

### Constraints

Each key supports a fixed set of constraints, depending on the level it can be set at.
Most keys in this reference are global, per Namespace, or per Task Queue.

When more than one value matches a request, the Temporal Service uses the value with the most specific matching constraints.
The order of the values in the file doesn't affect which one applies.

#### Global values

To set one value for the whole Temporal Service, leave out `constraints`:

```yaml
frontend.globalNamespaceRPS: # Per-Namespace RPC rate limit, applied across the Temporal Service.
  - value: 5000
```

#### Namespace values

For keys that support Namespace constraints, set a value for each Namespace with `namespace`.
A value with empty constraints (`{}`) applies to every other Namespace.

```yaml
frontend.persistenceNamespaceMaxQPS: # Per-Namespace limit on queries the Frontend Service sends to the Persistence store.
  - constraints: {} # Applies to all other Namespaces. The default is 0.
    value: 2000
  - constraints: { namespace: 'namespace1' }
    value: 4000
  - constraints: { namespace: 'namespace2' }
    value: 1000
```

#### Task Queue values

For keys that support Task Queue constraints, set `taskQueueName` and, optionally, `taskType` in addition to `namespace`.
Supported `taskType` values are `Workflow` and `Activity`.
Without `taskType`, the value applies to every Task type on the Task Queue.

For example, to handle a high rate of Workflow and Activity Tasks on a Task Queue, increase its number of partitions.
The default is 4.
Set the read and write partition keys to the same values.

```yaml
matching.numTaskqueueReadPartitions:
  - constraints: { namespace: 'namespace1', taskQueueName: 'tq' } # Applies to the "tq" Task Queue for all Task types.
    value: 8
  - constraints: { namespace: 'namespace1', taskQueueName: 'other-tq', taskType: 'Activity' } # Applies to Activity Tasks on "other-tq" only.
    value: 20
  - constraints: { namespace: 'namespace2' } # Applies to all Task Queues in "namespace2".
    value: 10
  - constraints: {} # Applies to all other Task Queues in all Namespaces.
    value: 16
matching.numTaskqueueWritePartitions:
  - constraints: { namespace: 'namespace1', taskQueueName: 'tq' }
    value: 8
  - constraints: { namespace: 'namespace1', taskQueueName: 'other-tq', taskType: 'Activity' }
    value: 20
  - constraints: { namespace: 'namespace2' }
    value: 10
  - constraints: {}
    value: 16
```

## Commonly used dynamic configuration keys

The following sections list the keys most often changed to tune rate limits, size limits, and defaults.
Default values are from Temporal Server v1.32.0.

If you raise a limit, provision enough compute and database capacity to handle the extra load.

### Service-level RPS limits

The requests per second (RPS) keys limit the rate at which each service accepts requests.
Requests over a limit fail with a `ResourceExhausted` error.

Tune these limits while you load test your workload, and use the results to set provisioning benchmarks.

| Dynamic configuration key              | Type | Description                                                                                                                                                                                                                                                        | Default value |
| -------------------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| Frontend                               |      |                                                                                                                                                                                                                                                                    |               |
| `frontend.rps`                         | Int  | Requests per second accepted by each Frontend Service host.                                                                                                                                                                                                        | 2400          |
| `frontend.namespaceRPS`                | Int  | Requests per second accepted for each Namespace by each Frontend Service host.                                                                                                                                                                                     | 2400          |
| `frontend.namespaceCount`              | Int  | Concurrent long-running requests, such as long polls and Queries, allowed for each Namespace on each Frontend Service host. The limit applies to each API method separately. Despite its name, this key doesn't limit the number of Namespaces. Setting it to `0` fails all long-running requests. | 1200          |
| `frontend.globalNamespaceRPS`          | Int  | Requests per second accepted for each Namespace across the Temporal Service. The limit is divided evenly among Frontend Service hosts. When set, it overrides `frontend.namespaceRPS`.                                                                             | 0             |
| `internal-frontend.globalNamespaceRPS` | Int  | Requests per second accepted for each Namespace across all Internal Frontend Service hosts.                                                                                                                                                                       | 0             |
| History                                |      |                                                                                                                                                                                                                                                                    |               |
| `history.rps`                          | Int  | Requests per second accepted by each History Service host.                                                                                                                                                                                                         | 3000          |
| Matching                               |      |                                                                                                                                                                                                                                                                    |               |
| `matching.rps`                         | Int  | Requests per second accepted by each Matching Service host.                                                                                                                                                                                                        | 1200          |
| `matching.numTaskqueueReadPartitions`  | Int  | Number of read partitions for a Task Queue. Set it with `matching.numTaskqueueWritePartitions`.                                                                                                                                                                   | 4             |
| `matching.numTaskqueueWritePartitions` | Int  | Number of write partitions for a Task Queue. Set it with `matching.numTaskqueueReadPartitions`.                                                                                                                                                                   | 4             |

### QPS limits for Persistence store

The queries per second (QPS) keys limit how many queries each service sends to the Persistence store.

The Temporal Service applies these limits synchronously.
When a service reaches its limit, Task processing slows down and requests can time out.
Set these limits based on your database capacity and workload.

| Dynamic configuration key                 | Type | Description                                                                                                                                                       | Default value |
| ----------------------------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| Frontend                                  |      |                                                                                                                                                                   |               |
| `frontend.persistenceMaxQPS`              | Int  | Queries per second each Frontend Service host can send to the Persistence store.                                                                                  | 2000          |
| `frontend.persistenceNamespaceMaxQPS`     | Int  | Queries per second each Frontend Service host can send to the Persistence store for each Namespace. When `0` or less, `frontend.persistenceMaxQPS` applies.        | 0             |
| History                                   |      |                                                                                                                                                                   |               |
| `history.persistenceMaxQPS`               | Int  | Queries per second each History Service host can send to the Persistence store.                                                                                   | 9000          |
| `history.persistenceNamespaceMaxQPS`      | Int  | Queries per second each History Service host can send to the Persistence store for each Namespace. When `0` or less, `history.persistenceMaxQPS` applies.          | 0             |
| Matching                                  |      |                                                                                                                                                                   |               |
| `matching.persistenceMaxQPS`              | Int  | Queries per second each Matching Service host can send to the Persistence store.                                                                                  | 3000          |
| `matching.persistenceNamespaceMaxQPS`     | Int  | Queries per second each Matching Service host can send to the Persistence store for each Namespace. When `0` or less, `matching.persistenceMaxQPS` applies.        | 0             |
| Worker                                    |      |                                                                                                                                                                   |               |
| `worker.persistenceMaxQPS`                | Int  | Queries per second each Worker Service host can send to the Persistence store.                                                                                    | 500           |
| `worker.persistenceNamespaceMaxQPS`       | Int  | Queries per second each Worker Service host can send to the Persistence store for each Namespace. When `0` or less, `worker.persistenceMaxQPS` applies.            | 0             |
| Visibility                                |      |                                                                                                                                                                   |               |
| `system.visibilityPersistenceMaxReadQPS`  | Int  | Read queries per second each host can send to the Visibility store.                                                                                               | 9000          |
| `system.visibilityPersistenceMaxWriteQPS` | Int  | Write queries per second each host can send to the Visibility store.                                                                                              | 9000          |

### Activity and Workflow default policy setting

These keys set Service-level defaults for Activity and Workflow [Retry Policies](/encyclopedia/retry-policies).

| Dynamic configuration key            | Type | Description                                                                                                                                                         | Default value                                                                                   |
| ------------------------------------ | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `history.defaultActivityRetryPolicy` | Map  | Retry Policy for Activities. Applies when the Activity has no Retry Policy, and fills in any field left unset in an explicit Retry Policy.                           | [Default values for Retry Policy](/encyclopedia/retry-policies#default-values-for-retry-policy) |
| `history.defaultWorkflowRetryPolicy` | Map  | Values for fields left unset in an explicit Workflow Retry Policy. Workflows without a Retry Policy don't retry.                                                     | [Default values for Retry Policy](/encyclopedia/retry-policies#default-values-for-retry-policy) |

### Size limit settings

These keys set size and count limits that protect the Persistence store.
If you raise them, provision enough database capacity for the larger values.

For all platform limits, see [Temporal Platform limits](/self-hosted-guide/defaults).

| Dynamic configuration key               | Type | Description                                                                                                                                                                                                                           | Default value            |
| --------------------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `limit.maxIDLength`                     | Int  | Maximum length of identifiers and names, including Namespace, Task Queue, Workflow ID, Activity ID, Timer ID, Workflow Type, Activity Type, Signal name, Marker name, failure reason, Identity, and Request ID.                        | 1000                     |
| `limit.blobSize.warn`                   | Int  | Payload size, in bytes, in a single Event at which the Temporal Service logs a warning.                                                                                                                                              | 512 KB (512 × 1024)      |
| `limit.blobSize.error`                  | Int  | Maximum payload size, in bytes, in a single Event. Requests over the limit fail.                                                                                                                                                     | 2 MB (2 × 1024 × 1024)   |
| `limit.historySize.warn`                | Int  | Event History size, in bytes, at which the Temporal Service logs a warning.                                                                                                                                                         | 10 MB (10 × 1024 × 1024) |
| `limit.historySize.error`               | Int  | Maximum Event History size, in bytes. The Temporal Service terminates a Workflow Execution that exceeds it.                                                                                                                         | 50 MB (50 × 1024 × 1024) |
| `limit.historyCount.warn`               | Int  | Number of Events in an Event History at which the Temporal Service logs a warning.                                                                                                                                                  | 10,240 Events            |
| `limit.historyCount.error`              | Int  | Maximum number of Events in an Event History. The Temporal Service terminates a Workflow Execution that exceeds it.                                                                                                                 | 51,200 Events            |
| `limit.numPendingActivities.error`      | Int  | Maximum number of pending Activities a Workflow Execution can have. Further `ScheduleActivityTask` Commands fail.                                                                                                                   | 2000                     |
| `limit.numPendingSignals.error`         | Int  | Maximum number of pending Signals a Workflow Execution can send to other Workflow Executions. Further `SignalExternalWorkflowExecution` Commands fail.                                                                              | 2000                     |
| `history.maximumSignalsPerExecution`    | Int  | Maximum number of Signals a Workflow Execution can receive. Further Signals fail with an `InvalidArgument` error.                                                                                                                   | 10000                    |
| `limit.numPendingCancelRequests.error`  | Int  | Maximum number of pending requests a Workflow Execution can have to cancel other Workflow Executions. Further `RequestCancelExternalWorkflowExecution` Commands fail.                                                               | 2000                     |
| `limit.numPendingChildExecutions.error` | Int  | Maximum number of pending Child Workflows a Workflow Execution can have. Further `StartChildWorkflowExecution` Commands fail.                                                                                                       | 2000                     |
| `frontend.visibilityMaxPageSize`        | Int  | Maximum number of Workflow Executions that `ListWorkflowExecutions` returns in one page.                                                                                                                                            | 1000                     |

For how these limits apply to Commands, see the [Commands reference](/references/commands).

### Secondary visibility settings

These keys control [Dual Visibility](/dual-visibility), which writes Visibility data to a secondary store.
Use Dual Visibility to migrate to a new Visibility store or to keep a backup store.
You can set both keys per Namespace.

| Dynamic configuration key                  | Type    | Description                                                                                                                                                                                                                                                                       | Default value |
| ------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `system.enableReadFromSecondaryVisibility` | Boolean | Reads Visibility data from the secondary Visibility store.                                                                                                                                                                                                                       | `false`       |
| `system.secondaryVisibilityWritingMode`    | String  | Which Visibility stores receive writes. Allowed values:<br />`off`: Primary Visibility store only.<br />`on`: Secondary Visibility store only. The primary store stops receiving writes.<br />`dual`: Both primary and secondary Visibility stores.                          | `off`         |

### Server version check settings

The Temporal Service periodically reports its version and the versions of connected SDKs to check for newer releases.
The Temporal Web UI uses the result to show a banner when a new version is available.

To turn off the check, set this key to `false`, or set the `TEMPORAL_VERSION_CHECK_DISABLED` environment variable to any value, such as `1`.

| Dynamic configuration key           | Type    | Description                                     | Default value                                                     |
| ----------------------------------- | ------- | ----------------------------------------------- | ----------------------------------------------------------------- |
| `frontend.enableServerVersionCheck` | Boolean | Turns the periodic server version check on or off. | `true`, or `false` when `TEMPORAL_VERSION_CHECK_DISABLED` is set |

### Nexus settings

These keys configure [Temporal Nexus](/nexus) on a self-hosted Temporal Service.
Several were added, renamed, or removed in recent releases, so check the version notes in each row.

Temporal Server v1.31.0 and later sends callbacks for Worker targets to the system callback URL, `temporal://system`, and needs no callback configuration.
Set `component.nexusoperations.callback.endpoint.template` and the allowed callback addresses only on earlier versions, or to use the experimental external endpoint target feature.

For how the Nexus components work, see the [Nexus architecture document](https://github.com/temporalio/temporal/blob/main/docs/architecture/nexus.md) in the Temporal Server repository.

| Dynamic configuration key                              | Type    | Description                                                                                                                                                                                                                         | Default value                          |
| ------------------------------------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| `component.nexusoperations.callback.endpoint.template` | String  | Go template for the callback URL the Temporal Service includes in Nexus Operation requests. The template can use `{{.NamespaceName}}` and `{{.NamespaceID}}`.                                                                       | `unset`                                |
| `callback.allowedAddresses`                            | List    | Callback URL patterns the Temporal Service accepts, set per Namespace. Each entry has a `Pattern` (`host:port`, supports `*` wildcards) and an optional `AllowInsecure` Boolean that permits HTTP. `temporal://system` is always allowed. v1.32.0 and later. | No rules                               |
| `component.callbacks.allowedAddresses`                 | List    | Name of `callback.allowedAddresses` before v1.32.0.                                                                                                                                                                                 | No rules                               |
| `component.nexusoperations.useSystemCallbackURL`       | Boolean | Sends callbacks for Worker targets to `temporal://system` instead of a URL built from the callback template. v1.30.x and v1.31.x only. Removed in v1.32.0.                                                                         | `true` on v1.31.x, `false` on v1.30.x |
| `system.enableNexus`                                   | Boolean | Turns Nexus on or off. Changes require a restart. Removed in v1.31.0, where Nexus is always on.                                                                                                                                   | `true` (v1.27.0 to v1.30.x)            |

---

# Temporal Events reference

Source: https://docs.temporal.io/references/events

> Event types the Temporal Service records in a Workflow Execution's Event History, with when each one occurs and the attributes it carries.

An [Event](/workflow-execution/event#event) is a record in the [Event History](/workflow-execution/event#event-history) of a [Workflow Execution](/workflow-execution).
The [Temporal Service](/temporal-service) records Events in response to [Commands](/references/commands) from a [Worker](/workers#worker) and to requests from outside the Workflow, such as a Signal or a cancellation request.
The SDK replays the Event History to recreate Workflow state.

Events are defined in the [`EventType` enum](https://github.com/temporalio/api/blob/master/temporal/api/enums/v1/event_type.proto) and [history messages](https://github.com/temporalio/api/blob/master/temporal/api/history/v1/message.proto) of the Temporal gRPC API.
Each table on this page lists the attributes you're most likely to need when reading an Event History.
For the complete set of attributes, see the history messages.

When an Event comes from a Command, its entry lists the **Command**.
The Event's `workflow_task_completed_event_id` attribute points to the [WorkflowTaskCompleted](#workflowtaskcompleted) Event of the Workflow Task that sent the Command.

## Common Event fields

Every Event has these fields, in addition to the attributes for its type.

| Field         | Description                                                                                                       |
| ------------- | ----------------------------------------------------------------------------------------------------------------- |
| event_id      | The position of the Event in the Event History, starting at 1.                                                   |
| event_time    | The time the Temporal Service recorded the Event.                                                                 |
| event_type    | The type of the Event, such as `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED`.                                           |
| user_metadata | A summary and details for the Event, carried over from the Command or Client call, such as a Timer summary.       |
| links         | Links to related entities, such as the Workflow Execution or Nexus Operation that started this Workflow Execution. |

## Workflow Execution Events

These Events record the lifecycle of the Workflow Execution.
WorkflowExecutionCompleted, WorkflowExecutionFailed, WorkflowExecutionTimedOut, WorkflowExecutionCanceled, WorkflowExecutionTerminated, and WorkflowExecutionContinuedAsNew close the Workflow Execution.
Each of them is the last Event in the Event History.

### WorkflowExecutionStarted

The first Event in every Event History.
The Temporal Service records it when it accepts a request to start the Workflow Execution.

| Field                              | Description                                                                                                                                               |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| workflow_type                      | The [Workflow Type](/workflow-definition#workflow-type).                                                                                                  |
| workflow_id                        | The [Workflow ID](/workflow-execution/workflowid-runid#workflow-id).                                                                                      |
| parent_workflow_namespace          | The [Namespace](/namespaces) of the parent Workflow Execution, if this is a [Child Workflow](/child-workflows).                                           |
| parent_workflow_execution          | The Workflow ID and Run ID of the parent Workflow Execution, if this is a Child Workflow.                                                                 |
| parent_initiated_event_id          | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event in the parent's Event History.                     |
| task_queue                         | The [Task Queue](/task-queue) the Workflow Tasks are sent to.                                                                                             |
| input                              | The Workflow arguments. The SDK deserializes them and passes them to the Workflow Function.                                                               |
| workflow_execution_timeout         | The [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout), which covers retries and Continue-As-New.          |
| workflow_run_timeout               | The [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout) for a single run.                                              |
| workflow_task_timeout              | The [Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout) for a single Workflow Task.                                   |
| continued_execution_run_id         | The [Run ID](/workflow-execution/workflowid-runid#run-id) of the previous run, if this run started from Continue-As-New, a retry, or a Cron Schedule.     |
| initiator                          | What started this run from a previous one: the Workflow (Continue-As-New), a Retry Policy, or a Cron Schedule.                                            |
| continued_failure                  | The failure of the previous run, if this run is a retry.                                                                                                  |
| last_completion_result             | The result of the most recent completed run, for Cron and retried Workflow Executions.                                                                    |
| original_execution_run_id          | The Run ID when this Event was recorded. A [Reset](/workflow-execution/event#reset) changes the Run ID but keeps this value.                              |
| identity                           | The identity of the Client that requested the start.                                                                                                      |
| first_execution_run_id             | The first Run ID in the chain of [Continue-As-New](/workflow-execution/continue-as-new), retry, Cron, and Reset runs.                                     |
| retry_policy                       | The [Retry Policy](/encyclopedia/retry-policies) of the Workflow Execution, if one was set.                                                               |
| attempt                            | The attempt number of this run, starting at 1.                                                                                                            |
| workflow_execution_expiration_time | The time at which the Workflow Execution times out. Passed unchanged to later runs.                                                                       |
| cron_schedule                      | The [Cron Schedule](/cron-job), if one was set.                                                                                                           |
| first_workflow_task_backoff        | For a Cron Workflow, the time between when this run was scheduled and when its first Workflow Task runs.                                                  |
| memo                               | The [Memo](/workflow-execution#memo): non-indexed information about the Workflow Execution.                                                               |
| search_attributes                  | The [Search Attributes](/search-attribute) set at start.                                                                                                  |
| prev_auto_reset_points             | The Reset points carried over from the previous run.                                                                                                      |
| header                             | Headers the Client set, such as context propagation data.                                                                                                 |
| completion_callbacks               | Callbacks the Temporal Service calls when the Workflow Execution closes, such as for a Nexus Operation backed by this Workflow.                            |
| versioning_override                | The [Versioning Override](/worker-versioning) applied at start, if any.                                                                                   |
| priority                           | The [Priority](/develop/task-queue-priority-fairness) metadata of the Workflow Execution.                                                                 |

### WorkflowExecutionCompleted

The Workflow Execution completed successfully.

- Command: [CompleteWorkflowExecution](/references/commands#completeworkflowexecution)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| result                           | The serialized return value of the Workflow Function.                                             |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| new_execution_run_id             | The Run ID of the next run, if a [Cron Schedule](/cron-job) started one.                          |

### WorkflowExecutionFailed

The Workflow Execution failed because the Workflow returned an error or threw an exception.

- Command: [FailWorkflowExecution](/references/commands#failworkflowexecution)

| Field                            | Description                                                                                                          |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| failure                          | The serialized Workflow failure.                                                                                     |
| retry_state                      | Whether the Temporal Service retries the Workflow Execution and, if not, why.                                       |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                    |
| new_execution_run_id             | The Run ID of the next run, if a Cron Schedule or a [Retry Policy](/encyclopedia/retry-policies) started one.        |

### WorkflowExecutionTimedOut

The Workflow Execution didn't complete within its [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) or [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout).

| Field                | Description                                                                                                   |
| -------------------- | ------------------------------------------------------------------------------------------------------------- |
| retry_state          | Whether the Temporal Service retries the Workflow Execution and, if not, why.                                |
| new_execution_run_id | The Run ID of the next run, if a Cron Schedule or a [Retry Policy](/encyclopedia/retry-policies) started one. |

### WorkflowExecutionCancelRequested

A Client or another Workflow requested cancellation of the Workflow Execution.
The Workflow can handle the request and clean up before it completes as canceled.

| Field                       | Description                                                                                                                                                      |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cause                       | The reason given for the cancellation request.                                                                                                                   |
| external_initiated_event_id | The Event ID of the [RequestCancelExternalWorkflowExecutionInitiated](#requestcancelexternalworkflowexecutioninitiated) Event, if another Workflow sent the request. |
| external_workflow_execution | The Workflow ID and Run ID of the Workflow Execution that sent the request, if any.                                                                              |
| identity                    | The identity of the Client or Worker that requested cancellation.                                                                                                |

### WorkflowExecutionCanceled

The Workflow finished cleaning up after a cancellation request, and the Workflow Execution closed as canceled.

- Command: [CancelWorkflowExecution](/references/commands#cancelworkflowexecution)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| details                          | Additional information the Workflow reported on cancellation.                                     |

### WorkflowExecutionSignaled

The Workflow Execution received a [Signal](/sending-messages#sending-signals).

| Field                       | Description                                                                                |
| --------------------------- | ------------------------------------------------------------------------------------------ |
| signal_name                 | The name of the Signal.                                                                    |
| input                       | The Signal arguments. The SDK deserializes them and passes them to the Signal handler.    |
| identity                    | The identity of the Client or Worker that sent the Signal.                                 |
| header                      | Headers the sender set, copied into the [Workflow Task](/tasks#workflow-task).             |
| external_workflow_execution | The Workflow ID and Run ID of the Workflow Execution that sent the Signal, if any.         |

### WorkflowExecutionTerminated

The Workflow Execution was forcefully terminated, usually through a call to the terminate Workflow Execution API.
The Workflow code doesn't run again, and no cleanup happens.

| Field    | Description                                               |
| -------- | --------------------------------------------------------- |
| reason   | The reason the Client gave for termination.               |
| details  | Additional information provided with the termination.     |
| identity | The identity of the Client that requested termination.    |

### WorkflowExecutionContinuedAsNew

The Workflow called [Continue-As-New](/workflow-execution/continue-as-new).
The Temporal Service closes this run and starts a new run with the same Workflow ID in the same transaction.

- Command: [ContinueAsNewWorkflowExecution](/references/commands#continueasnewworkflowexecution)

| Field                            | Description                                                                                                              |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| new_execution_run_id             | The Run ID of the new run.                                                                                               |
| workflow_type                    | The Workflow Type of the new run.                                                                                        |
| task_queue                       | The Task Queue of the new run.                                                                                           |
| input                            | The arguments for the new run.                                                                                           |
| workflow_run_timeout             | The Workflow Run Timeout of the new run.                                                                                 |
| workflow_task_timeout            | The Workflow Task Timeout of the new run.                                                                                |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                        |
| backoff_start_interval           | How long the Temporal Service waits before it schedules the first Workflow Task of the new run.                          |
| initiator                        | What started the new run: the Workflow (Continue-As-New), a Retry Policy, or a Cron Schedule.                            |
| last_completion_result           | The result of the most recent completed run, which the SDK makes available to the new run.                               |
| header                           | Headers passed to the new run.                                                                                           |
| memo                             | The Memo of the new run.                                                                                                 |
| search_attributes                | The [Search Attributes](/search-attribute) of the new run.                                                               |

### WorkflowExecutionOptionsUpdated

A Client updated the options of the running Workflow Execution, such as a Versioning Override or Priority.
The Temporal Service also records this Event when it attaches a request ID or completion callbacks to a running Workflow Execution.

| Field                         | Description                                                                                                                    |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| versioning_override           | The [Versioning Override](/worker-versioning) set by this update. Ignored if empty or if `unset_versioning_override` is true. |
| unset_versioning_override     | Whether this update removed the Versioning Override.                                                                           |
| attached_request_id           | A request ID attached to the Workflow Execution. Later requests with the same request ID are deduplicated.                     |
| attached_completion_callbacks | Completion callbacks attached to the Workflow Execution.                                                                       |
| identity                      | The identity of the Client that requested the update.                                                                          |
| priority                      | The full [Priority](/develop/task-queue-priority-fairness) set by this update. Ignored if empty.                              |

### WorkflowExecutionPaused

A Client [paused](/cli/command-reference/workflow#pause) the Workflow Execution.
Workflow Execution pause is experimental.

| Field      | Description                                         |
| ---------- | --------------------------------------------------- |
| identity   | The identity of the Client that paused the Workflow Execution. |
| reason     | The reason given for the pause.                     |
| request_id | The request ID of the pause request.                |

### WorkflowExecutionUnpaused

A Client [unpaused](/cli/command-reference/workflow#unpause) a paused Workflow Execution.
Workflow Execution pause is experimental.

| Field      | Description                                                       |
| ---------- | ----------------------------------------------------------------- |
| identity   | The identity of the Client that unpaused the Workflow Execution. |
| reason     | The reason given for unpausing.                                   |
| request_id | The request ID of the unpause request.                            |

## Workflow Task Events

These Events record each [Workflow Task](/tasks#workflow-task): scheduling it, a Worker picking it up, and how it ended.

### WorkflowTaskScheduled

The Temporal Service scheduled a Workflow Task because the Workflow has new Events to process.

| Field                  | Description                                                                                    |
| ---------------------- | ---------------------------------------------------------------------------------------------- |
| task_queue             | The [Task Queue](/task-queue) the Workflow Task was sent to. This can be a Sticky Task Queue. |
| start_to_close_timeout | How long the Worker has to process the Workflow Task after it picks it up.                     |
| attempt                | The attempt number of this Workflow Task, starting at 1.                                       |

### WorkflowTaskStarted

A Worker picked up the Workflow Task and is processing the new Events.

| Field                   | Description                                                                                                                    |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| scheduled_event_id      | The Event ID of the [WorkflowTaskScheduled](#workflowtaskscheduled) Event for this Workflow Task.                              |
| identity                | The identity of the Worker that picked up the Workflow Task.                                                                   |
| request_id              | The identifier of the request that started the Workflow Task.                                                                  |
| suggest_continue_as_new | Whether the Temporal Service suggests that the Workflow call [Continue-As-New](/workflow-execution/continue-as-new) soon.       |
| history_size_bytes      | The size of the Event History in bytes. The Workflow can use it to decide when to call Continue-As-New.                         |

### WorkflowTaskCompleted

The Worker finished the Workflow Task and sent its [Commands](/references/commands) to the Temporal Service.
The Events for those Commands follow this Event, such as:

- [ActivityTaskScheduled](#activitytaskscheduled)
- [TimerStarted](#timerstarted)
- [UpsertWorkflowSearchAttributes](#upsertworkflowsearchattributes)
- [MarkerRecorded](#markerrecorded)
- [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated)
- [RequestCancelExternalWorkflowExecutionInitiated](#requestcancelexternalworkflowexecutioninitiated)
- [SignalExternalWorkflowExecutionInitiated](#signalexternalworkflowexecutioninitiated)
- [WorkflowExecutionCompleted](#workflowexecutioncompleted)
- [WorkflowExecutionFailed](#workflowexecutionfailed)
- [WorkflowExecutionCanceled](#workflowexecutioncanceled)
- [WorkflowExecutionContinuedAsNew](#workflowexecutioncontinuedasnew)

| Field                  | Description                                                                                                   |
| ---------------------- | ------------------------------------------------------------------------------------------------------------- |
| scheduled_event_id     | The Event ID of the [WorkflowTaskScheduled](#workflowtaskscheduled) Event for this Workflow Task.             |
| started_event_id       | The Event ID of the [WorkflowTaskStarted](#workflowtaskstarted) Event for this Workflow Task.                 |
| identity               | The identity of the Worker that completed the Workflow Task.                                                  |
| sdk_metadata           | Data the SDK records for its own use, such as the SDK flags it used. The Temporal Service doesn't read it.    |
| versioning_behavior    | The [versioning behavior](/worker-versioning) the Worker sent. Unspecified if the Worker isn't versioned.     |
| worker_deployment_name | The name of the Worker Deployment that completed the Workflow Task.                                           |
| deployment_version     | The Worker Deployment Version that completed the Workflow Task.                                               |

### WorkflowTaskTimedOut

The Workflow Task hit a [Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout).
Either no Worker with the Workflow in its cache was available to pick up the Workflow Task, or the Worker took too long to process it.

| Field              | Description                                                                                       |
| ------------------ | ------------------------------------------------------------------------------------------------- |
| scheduled_event_id | The Event ID of the [WorkflowTaskScheduled](#workflowtaskscheduled) Event for this Workflow Task. |
| started_event_id   | The Event ID of the [WorkflowTaskStarted](#workflowtaskstarted) Event for this Workflow Task.     |
| timeout_type       | The type of timeout, such as Start-To-Close or Schedule-To-Start.                                 |

### WorkflowTaskFailed

The Workflow Task failed, and the Temporal Service retries it.
A common cause is a [non-deterministic](/workflow-definition#non-deterministic-change) change to the Workflow code.
A Workflow [Reset](/workflow-execution/event#reset) also records this Event.
For each cause and how to resolve it, see the [Workflow Task errors reference](/references/workflow-task-errors).

| Field              | Description                                                                                                                         |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| scheduled_event_id | The Event ID of the [WorkflowTaskScheduled](#workflowtaskscheduled) Event for this Workflow Task.                                   |
| started_event_id   | The Event ID of the [WorkflowTaskStarted](#workflowtaskstarted) Event for this Workflow Task.                                       |
| cause              | The [Workflow Task failure cause](/references/workflow-task-errors).                                                                |
| failure            | The details of the failure.                                                                                                         |
| identity           | The identity of the Worker that failed the Workflow Task. Set to `history-service` when the Temporal Service generated the failure. |
| base_run_id        | For a Reset, the Run ID of the original Workflow Execution.                                                                         |
| new_run_id         | For a Reset, the Run ID of the new Workflow Execution.                                                                              |
| fork_event_version | For a Reset, the version of the Event where the history branch was forked. Used by Multi-Cluster Replication.                       |

## Activity Events

These Events record each [Activity Execution](/activity-execution) the Workflow schedules.

### ActivityTaskScheduled

The Workflow scheduled an [Activity Task](/tasks#activity-task).
A Worker polling the Task Queue picks it up and runs the Activity.

- Command: [ScheduleActivityTask](/references/commands#scheduleactivitytask)

| Field                            | Description                                                                                                                                                        |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| activity_id                      | The Activity ID, assigned by the Worker or set in the Workflow code.                                                                                               |
| activity_type                    | The [Activity Type](/activity-definition#activity-type).                                                                                                           |
| task_queue                       | The [Task Queue](/task-queue) the Activity Task was sent to.                                                                                                       |
| header                           | Headers the Workflow set, such as context propagation data.                                                                                                        |
| input                            | The Activity arguments. The SDK deserializes them and passes them to the Activity Function.                                                                        |
| schedule_to_close_timeout        | The [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout): how long the Workflow waits for the Activity, including retries. |
| schedule_to_start_timeout        | The [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout): how long the Activity Task can wait in the Task Queue. Not retried. |
| start_to_close_timeout           | The [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout): the maximum time for one attempt after a Worker picks it up. Retried.  |
| heartbeat_timeout                | The [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout): the maximum time between Activity Heartbeats.                                 |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                                                                  |
| retry_policy                     | The [Retry Policy](/encyclopedia/retry-policies) of the Activity. Retries stop when the Schedule-To-Close Timeout is reached.                                      |
| priority                         | The [Priority](/develop/task-queue-priority-fairness) metadata of the Activity. Unset values are inherited from the Workflow.                                      |

### ActivityTaskStarted

A Worker picked up the Activity Task.
The Temporal Service records this Event when it dispatches the Activity Task to the Worker, not when the Worker starts running it.

The Temporal Service doesn't write this Event to the Event History until the Activity Execution closes, for example with [ActivityTaskCompleted](#activitytaskcompleted) or [ActivityTaskFailed](#activitytaskfailed).
If the Activity is retried, the Event reflects the last attempt.

| Field              | Description                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------- |
| scheduled_event_id | The Event ID of the [ActivityTaskScheduled](#activitytaskscheduled) Event for this Activity. |
| identity           | The identity of the Worker that picked up the Activity Task.                                  |
| request_id         | The identifier of the request that started the Activity Task.                                 |
| attempt            | The attempt number, starting at 1.                                                            |
| last_failure       | The failure of the previous attempt, if the Activity was retried.                             |

### ActivityTaskCompleted

The Activity Execution completed successfully.

| Field              | Description                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------- |
| result             | The serialized return value of the Activity Function.                                         |
| scheduled_event_id | The Event ID of the [ActivityTaskScheduled](#activitytaskscheduled) Event for this Activity. |
| started_event_id   | The Event ID of the [ActivityTaskStarted](#activitytaskstarted) Event for this Activity.     |
| identity           | The identity of the Worker that completed the Activity Task.                                  |

### ActivityTaskFailed

The Activity Execution failed, and the Temporal Service won't retry it.

| Field              | Description                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------- |
| failure            | The serialized Activity failure.                                                              |
| scheduled_event_id | The Event ID of the [ActivityTaskScheduled](#activitytaskscheduled) Event for this Activity. |
| started_event_id   | The Event ID of the [ActivityTaskStarted](#activitytaskstarted) Event for this Activity.     |
| identity           | The identity of the Worker that failed the Activity Task.                                     |
| retry_state        | Why the Temporal Service won't retry the Activity, such as reaching the maximum attempts.     |

### ActivityTaskTimedOut

The Activity Execution hit an [Activity timeout](/encyclopedia/detecting-activity-failures), and the Temporal Service won't retry it.

| Field              | Description                                                                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| failure            | A Timeout Failure that holds the timeout type, such as Start-To-Close or Schedule-To-Close. If an earlier attempt failed, that failure is set as its cause. |
| scheduled_event_id | The Event ID of the [ActivityTaskScheduled](#activitytaskscheduled) Event for this Activity.                                                                |
| started_event_id   | The Event ID of the [ActivityTaskStarted](#activitytaskstarted) Event for this Activity.                                                                    |
| retry_state        | Why the Temporal Service won't retry the Activity, such as the timeout being reached.                                                                        |

### ActivityTaskCancelRequested

The Workflow requested [cancellation](/activity-execution#cancellation) of the Activity.
The Activity learns about the request the next time it Heartbeats.

- Command: [RequestCancelActivityTask](/references/commands#requestcancelactivitytask)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| scheduled_event_id               | The Event ID of the [ActivityTaskScheduled](#activitytaskscheduled) Event for this Activity.     |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |

### ActivityTaskCanceled

The Activity Execution was [canceled](/activity-execution#cancellation).

| Field                            | Description                                                                                                     |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| details                          | Additional information the Activity reported when it confirmed cancellation.                                    |
| latest_cancel_requested_event_id | The Event ID of the most recent [ActivityTaskCancelRequested](#activitytaskcancelrequested) Event for this Activity. |
| scheduled_event_id               | The Event ID of the [ActivityTaskScheduled](#activitytaskscheduled) Event for this Activity.                   |
| started_event_id                 | The Event ID of the [ActivityTaskStarted](#activitytaskstarted) Event for this Activity.                       |
| identity                         | The identity of the Worker that canceled the Activity.                                                          |

### ActivityPropertiesModifiedExternally

Code outside the Workflow changed a property of an Activity that was already scheduled, such as its Retry Policy.

| Field              | Description                                                                                            |
| ------------------ | ------------------------------------------------------------------------------------------------------ |
| scheduled_event_id | The Event ID of the [ActivityTaskScheduled](#activitytaskscheduled) Event for this Activity.          |
| new_retry_policy   | The Retry Policy that replaces the Activity's Retry Policy. The attempt count is kept.                 |

## Timer Events

### TimerStarted

The Workflow started a Timer, such as with a call to sleep.

- Command: [StartTimer](/references/commands#starttimer)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| timer_id                         | The Timer ID, assigned by the Worker or set in the Workflow code.                                 |
| start_to_fire_timeout            | How long until the Timer fires.                                                                   |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |

### TimerFired

The Timer fired.

| Field            | Description                                                         |
| ---------------- | ------------------------------------------------------------------- |
| timer_id         | The Timer ID from the [TimerStarted](#timerstarted) Event.          |
| started_event_id | The Event ID of the [TimerStarted](#timerstarted) Event.            |

### TimerCanceled

The Workflow canceled the Timer before it fired.

- Command: [CancelTimer](/references/commands#canceltimer)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| timer_id                         | The Timer ID from the [TimerStarted](#timerstarted) Event.                                        |
| started_event_id                 | The Event ID of the [TimerStarted](#timerstarted) Event.                                          |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| identity                         | The identity of the Worker that canceled the Timer.                                               |

## Child Workflow Events

These Events appear in the parent Workflow's Event History.
The [Child Workflow Execution](/child-workflows) has its own Event History, which starts with [WorkflowExecutionStarted](#workflowexecutionstarted).

### StartChildWorkflowExecutionInitiated

The parent Workflow requested that the Temporal Service start a Child Workflow Execution.

- Command: [StartChildWorkflowExecution](/references/commands#startchildworkflowexecution)

| Field                            | Description                                                                                                  |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| namespace                        | The [Namespace](/namespaces) of the Child Workflow.                                                          |
| workflow_id                      | The Workflow ID of the Child Workflow.                                                                       |
| workflow_type                    | The Workflow Type of the Child Workflow.                                                                     |
| task_queue                       | The Task Queue of the Child Workflow.                                                                        |
| input                            | The arguments for the Child Workflow.                                                                        |
| parent_close_policy              | The [Parent Close Policy](/parent-close-policy) of the Child Workflow.                                       |
| workflow_id_reuse_policy         | The [Workflow ID Reuse Policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy) of the Child Workflow. |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.            |

### StartChildWorkflowExecutionFailed

The Temporal Service couldn't start the Child Workflow Execution, usually because a Workflow Execution with the same Workflow ID already exists.

| Field                            | Description                                                                                                         |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| namespace                        | The Namespace of the Child Workflow.                                                                                |
| workflow_id                      | The Workflow ID of the Child Workflow.                                                                              |
| workflow_type                    | The Workflow Type of the Child Workflow.                                                                            |
| cause                            | Why the start failed, such as the Workflow ID already being in use or the Namespace not being found.                |
| initiated_event_id               | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event.            |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                   |

### ChildWorkflowExecutionStarted

The Child Workflow Execution started.
The Child Workflow's own Event History records a [WorkflowExecutionStarted](#workflowexecutionstarted) Event.

| Field              | Description                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------------- |
| namespace          | The Namespace of the Child Workflow.                                                                     |
| initiated_event_id | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event. |
| workflow_execution | The Workflow ID and Run ID of the Child Workflow Execution.                                              |
| workflow_type      | The Workflow Type of the Child Workflow.                                                                 |
| header             | Headers passed to the Child Workflow.                                                                    |

### ChildWorkflowExecutionCompleted

The Child Workflow Execution completed successfully.
The Child Workflow's own Event History records a [WorkflowExecutionCompleted](#workflowexecutioncompleted) Event.

| Field              | Description                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------------- |
| result             | The serialized result of the Child Workflow.                                                             |
| namespace          | The Namespace of the Child Workflow.                                                                     |
| workflow_execution | The Workflow ID and Run ID of the Child Workflow Execution.                                              |
| workflow_type      | The Workflow Type of the Child Workflow.                                                                 |
| initiated_event_id | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event. |
| started_event_id   | The Event ID of the [ChildWorkflowExecutionStarted](#childworkflowexecutionstarted) Event.               |

### ChildWorkflowExecutionFailed

The Child Workflow Execution failed.
The Child Workflow's own Event History records a [WorkflowExecutionFailed](#workflowexecutionfailed) Event.

| Field              | Description                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------------- |
| failure            | The serialized Child Workflow failure.                                                                   |
| namespace          | The Namespace of the Child Workflow.                                                                     |
| workflow_execution | The Workflow ID and Run ID of the Child Workflow Execution.                                              |
| workflow_type      | The Workflow Type of the Child Workflow.                                                                 |
| initiated_event_id | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event. |
| started_event_id   | The Event ID of the [ChildWorkflowExecutionStarted](#childworkflowexecutionstarted) Event.               |
| retry_state        | Whether the Temporal Service retries the Child Workflow Execution and, if not, why.                      |

### ChildWorkflowExecutionCanceled

The Child Workflow Execution was canceled.
The Child Workflow's own Event History records a [WorkflowExecutionCanceled](#workflowexecutioncanceled) Event.

| Field              | Description                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------------- |
| details            | Additional information the Child Workflow reported on cancellation.                                      |
| namespace          | The Namespace of the Child Workflow.                                                                     |
| workflow_execution | The Workflow ID and Run ID of the Child Workflow Execution.                                              |
| workflow_type      | The Workflow Type of the Child Workflow.                                                                 |
| initiated_event_id | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event. |
| started_event_id   | The Event ID of the [ChildWorkflowExecutionStarted](#childworkflowexecutionstarted) Event.               |

### ChildWorkflowExecutionTimedOut

The Child Workflow Execution timed out.
The Child Workflow's own Event History records a [WorkflowExecutionTimedOut](#workflowexecutiontimedout) Event.

| Field              | Description                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------------- |
| namespace          | The Namespace of the Child Workflow.                                                                     |
| workflow_execution | The Workflow ID and Run ID of the Child Workflow Execution.                                              |
| workflow_type      | The Workflow Type of the Child Workflow.                                                                 |
| initiated_event_id | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event. |
| started_event_id   | The Event ID of the [ChildWorkflowExecutionStarted](#childworkflowexecutionstarted) Event.               |
| retry_state        | Whether the Temporal Service retries the Child Workflow Execution and, if not, why.                      |

### ChildWorkflowExecutionTerminated

The Child Workflow Execution was terminated.
The Child Workflow's own Event History records a [WorkflowExecutionTerminated](#workflowexecutionterminated) Event.

| Field              | Description                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------------- |
| namespace          | The Namespace of the Child Workflow.                                                                     |
| workflow_execution | The Workflow ID and Run ID of the Child Workflow Execution.                                              |
| workflow_type      | The Workflow Type of the Child Workflow.                                                                 |
| initiated_event_id | The Event ID of the [StartChildWorkflowExecutionInitiated](#startchildworkflowexecutioninitiated) Event. |
| started_event_id   | The Event ID of the [ChildWorkflowExecutionStarted](#childworkflowexecutionstarted) Event.               |

## External Workflow Events

These Events record requests the Workflow sends to other Workflow Executions.

### RequestCancelExternalWorkflowExecutionInitiated

The Workflow requested that the Temporal Service cancel another Workflow Execution.

- Command: [RequestCancelExternalWorkflowExecution](/references/commands#requestcancelexternalworkflowexecution)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| namespace                        | The [Namespace](/namespaces) of the Workflow Execution to cancel.                                 |
| workflow_execution               | The Workflow ID and Run ID of the Workflow Execution to cancel.                                   |
| child_workflow_only              | Whether the target must be a Child Workflow of the Workflow that sent the request.                 |
| reason                           | The reason given for the cancellation request.                                                    |

### RequestCancelExternalWorkflowExecutionFailed

The Temporal Service couldn't deliver the cancellation request, usually because it couldn't find the target Workflow Execution.

| Field                            | Description                                                                                                                         |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| cause                            | Why the request failed, such as the target Workflow Execution or Namespace not being found.                                         |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                                   |
| namespace                        | The Namespace of the target Workflow Execution.                                                                                     |
| workflow_execution               | The Workflow ID and Run ID of the target Workflow Execution.                                                                        |
| initiated_event_id               | The Event ID of the [RequestCancelExternalWorkflowExecutionInitiated](#requestcancelexternalworkflowexecutioninitiated) Event.      |

### ExternalWorkflowExecutionCancelRequested

The Temporal Service delivered the cancellation request to the target Workflow Execution.
The target's Event History records a [WorkflowExecutionCancelRequested](#workflowexecutioncancelrequested) Event.

| Field              | Description                                                                                                                    |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| initiated_event_id | The Event ID of the [RequestCancelExternalWorkflowExecutionInitiated](#requestcancelexternalworkflowexecutioninitiated) Event. |
| namespace          | The Namespace of the target Workflow Execution.                                                                                |
| workflow_execution | The Workflow ID and Run ID of the target Workflow Execution.                                                                   |

### SignalExternalWorkflowExecutionInitiated

The Workflow requested that the Temporal Service [Signal](/sending-messages#sending-signals) another Workflow Execution.

- Command: [SignalExternalWorkflowExecution](/references/commands#signalexternalworkflowexecution)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| namespace                        | The [Namespace](/namespaces) of the Workflow Execution to Signal.                                 |
| workflow_execution               | The Workflow ID and Run ID of the Workflow Execution to Signal.                                   |
| signal_name                      | The name of the Signal.                                                                           |
| input                            | The Signal arguments.                                                                             |
| child_workflow_only              | Whether the target must be a Child Workflow of the Workflow that sent the Signal.                  |
| header                           | Headers passed to the target Workflow Execution with the Signal.                                  |

### SignalExternalWorkflowExecutionFailed

The Temporal Service couldn't deliver the Signal, usually because it couldn't find the target Workflow Execution.

| Field                            | Description                                                                                                          |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| cause                            | Why the Signal failed, such as the target Workflow Execution not being found.                                        |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                    |
| namespace                        | The Namespace of the target Workflow Execution.                                                                      |
| workflow_execution               | The Workflow ID and Run ID of the target Workflow Execution.                                                         |
| initiated_event_id               | The Event ID of the [SignalExternalWorkflowExecutionInitiated](#signalexternalworkflowexecutioninitiated) Event.     |

### ExternalWorkflowExecutionSignaled

The Temporal Service delivered the Signal to the target Workflow Execution.
The target's Event History records a [WorkflowExecutionSignaled](#workflowexecutionsignaled) Event.

| Field              | Description                                                                                                      |
| ------------------ | ---------------------------------------------------------------------------------------------------------------- |
| initiated_event_id | The Event ID of the [SignalExternalWorkflowExecutionInitiated](#signalexternalworkflowexecutioninitiated) Event. |
| namespace          | The Namespace of the target Workflow Execution.                                                                  |
| workflow_execution | The Workflow ID and Run ID of the target Workflow Execution.                                                     |

## Workflow Update Events

These Events record [Workflow Updates](/sending-messages#sending-updates).
A rejected Update doesn't produce an Event.

- Command: [ProtocolMessage](/references/commands#protocolmessagecommand), for WorkflowExecutionUpdateAccepted and WorkflowExecutionUpdateCompleted

### WorkflowExecutionUpdateAdmitted

The Temporal Service reapplied an Update during a [Reset](/workflow-execution/event#reset) or replication.
Most Updates don't produce this Event.

| Field   | Description                                     |
| ------- | ----------------------------------------------- |
| request | The Update request.                             |
| origin  | Why the Temporal Service recorded this Event.   |

### WorkflowExecutionUpdateAccepted 

The Workflow accepted an Update, which means the Update passed its validator, if one is defined.
Sending an Update doesn't produce an Event, so this Event stores the original request.

| Field                                | Description                                                                                   |
| ------------------------------------ | --------------------------------------------------------------------------------------------- |
| protocol_instance_id                 | The identifier of the Update protocol instance that produced this Event.                      |
| accepted_request_message_id          | The identifier of the request message. The Worker uses it to recreate the message on replay. |
| accepted_request_sequencing_event_id | The Event ID after which the Update runs.                                                     |
| accepted_request                     | The Update request: its input and metadata.                                                   |

### WorkflowExecutionUpdateCompleted 

The Workflow ran an Update to completion.

| Field             | Description                                                                                                     |
| ----------------- | --------------------------------------------------------------------------------------------------------------- |
| meta              | The Update metadata from the original request, including the Update ID.                                         |
| accepted_event_id | The Event ID of the [WorkflowExecutionUpdateAccepted](#workflowexecutionupdateacceptedevent) Event.             |
| outcome           | The result of the Update handler: a success value or a failure.                                                 |

## Nexus Operation Events

These Events appear in the caller Workflow's Event History.

### NexusOperationScheduled

The caller Workflow scheduled a Nexus Operation.
The caller's [Nexus Machinery](/glossary#nexus-machinery) then tries to start it.

- Command: [ScheduleNexusOperation](/references/commands#schedulenexusoperation)

| Field                            | Description                                                                                                                                         |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| endpoint                         | The Nexus Endpoint name. It must exist in the Endpoint registry.                                                                                    |
| service                          | The Nexus Service name.                                                                                                                             |
| operation                        | The Nexus Operation name.                                                                                                                           |
| input                            | The Operation input. A handler backed by Temporal receives the original Payload.                                                                    |
| schedule_to_close_timeout        | How long the caller waits for the Operation to complete. The Temporal Service retries calls within this time.                                       |
| nexus_header                     | Headers sent with the Nexus request. These aren't Temporal headers; they can go to Operations outside Temporal.                                     |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                                                   |
| request_id                       | A unique identifier the Temporal Service generated for the Operation. It's sent with every start request as an idempotency key.                     |
| endpoint_id                      | The Endpoint ID when this Event was recorded. The Temporal Service uses it if the Endpoint is renamed later.                                        |

### NexusOperationStarted

The Nexus handler started an asynchronous Nexus Operation, such as one backed by a Workflow.
Synchronous Operations don't produce this Event. They move directly to [NexusOperationCompleted](#nexusoperationcompleted) or another final Event, such as [NexusOperationFailed](#nexusoperationfailed).

| Field              | Description                                                                                                     |
| ------------------ | --------------------------------------------------------------------------------------------------------------- |
| scheduled_event_id | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event for this Operation.              |
| operation_token    | The Operation token the handler returned in its start response. The caller uses it to cancel the Operation.    |
| request_id         | The request ID allocated when the Operation was scheduled.                                                      |

### NexusOperationCompleted

The Nexus Operation completed successfully.
Both synchronous and asynchronous Operations record their result with this Event.

| Field              | Description                                                                                                                  |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| scheduled_event_id | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event. It identifies the Operation.                 |
| result             | The serialized result from the handler, delivered in a synchronous response or through a completion callback.                |
| request_id         | The request ID allocated when the Operation was scheduled.                                                                   |

### NexusOperationFailed

The Nexus Operation failed.
Both synchronous and asynchronous Operations record a failure with this Event.
For example, a handler responds synchronously with a non-retryable error, or the Workflow behind an asynchronous Operation fails with [WorkflowExecutionFailed](#workflowexecutionfailed).

| Field              | Description                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| scheduled_event_id | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event. It identifies the Operation. |
| failure            | The failure details: a `NexusOperationFailureInfo` that wraps an `ApplicationFailureInfo`.                   |
| request_id         | The request ID allocated when the Operation was scheduled.                                                   |

### NexusOperationTimedOut

The Nexus Operation didn't complete within its Schedule-To-Close Timeout.

| Field              | Description                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| scheduled_event_id | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event. It identifies the Operation. |
| failure            | The failure details: a `NexusOperationFailureInfo` that wraps a `CanceledFailureInfo`.                       |
| request_id         | The request ID allocated when the Operation was scheduled.                                                   |

### NexusOperationCancelRequested

The caller Workflow requested cancellation of the Nexus Operation.

- Command: [RequestCancelNexusOperation](/references/commands#requestcancelnexusoperation)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| scheduled_event_id               | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event for this Operation. |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |

### NexusOperationCancelRequestCompleted

The Temporal Service delivered the cancellation request to the Nexus handler.

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| requested_event_id               | The Event ID of the [NexusOperationCancelRequested](#nexusoperationcancelrequested) Event.        |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| scheduled_event_id               | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event for this Operation. |

### NexusOperationCancelRequestFailed

The cancellation request to the Nexus handler resulted in an error.

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| requested_event_id               | The Event ID of the [NexusOperationCancelRequested](#nexusoperationcancelrequested) Event.        |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| failure                          | The failure details: a `NexusOperationFailureInfo` that wraps a `CanceledFailureInfo`.            |
| scheduled_event_id               | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event for this Operation. |

### NexusOperationCanceled

The Nexus Operation completed as canceled.
This can happen with or without a cancellation request from the caller Workflow.

| Field              | Description                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| scheduled_event_id | The Event ID of the [NexusOperationScheduled](#nexusoperationscheduled) Event. It identifies the Operation. |
| failure            | The cancellation details.                                                                                    |
| request_id         | The request ID allocated when the Operation was scheduled.                                                   |

## Workflow metadata and SDK Events

### MarkerRecorded

The SDK recorded data it needs to replay the Workflow deterministically, such as the result of a Side Effect, a Local Activity, or a version check.
The Temporal Service stores the Marker without interpreting it.

- Command: [RecordMarker](/references/commands#recordmarker)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| marker_name                      | The kind of Marker, such as Local Activity or Side Effect. Each SDK defines its own names.        |
| details                          | The serialized data recorded in the Marker.                                                       |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| header                           | Headers recorded with the Marker.                                                                 |
| failure                          | The failure, if the recorded operation failed, such as a failed Local Activity.                   |

### UpsertWorkflowSearchAttributes

The Workflow added or updated its [Search Attributes](/search-attribute).
The Temporal Service updates the Visibility store with the new values.

- Command: [UpsertWorkflowSearchAttributes](/references/commands#upsertworkflowsearchattributes)

| Field                            | Description                                                                                       |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command. |
| search_attributes                | The Search Attributes the Workflow added or updated.                                              |

### WorkflowPropertiesModified

The Workflow added or updated its [Memo](/workflow-execution#memo).

- Command: [ModifyWorkflowProperties](/references/commands#modifyworkflowproperties)

| Field                            | Description                                                                                                       |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| workflow_task_completed_event_id | The Event ID of the [WorkflowTaskCompleted](#workflowtaskcompleted) Event that sent the Command.                 |
| upserted_memo                    | The Memo values to merge with the existing Memo. A key with an empty Payload deletes that key.                    |

---

# Temporal Failures reference

Source: https://docs.temporal.io/references/failures

> Failure types Temporal SDKs raise for Workflow, Activity, and Nexus Operation errors, with each type's SDK class, proto message, and retry behavior.

> **ℹ️ Info:**
>
> For the causes the Temporal Service records when a Workflow Task fails, see the [Workflow Task errors reference](/references/workflow-task-errors).
> For symptom-based guidance, see [Troubleshooting Workflow Execution failures](/troubleshooting/execution-failures).
>

A Failure is Temporal's representation of an error in a Workflow, Activity, or Nexus Operation.
Each type of Failure has its own class or error type in each SDK and its own information in the protobuf messages.
The SDKs use these messages to communicate with the Temporal Service, and they appear in the [Event History](/workflow-execution/event#event-history).

Failures are defined in the [failure messages](https://github.com/temporalio/api/blob/master/temporal/api/failure/v1/message.proto) of the Temporal gRPC API.
Each Failure type on this page lists its class or error type in each SDK and its proto message.

## Temporal Failure

Most SDKs have a base class that the other Failure types extend.

- TypeScript: [TemporalFailure](https://typescript.temporal.io/api/classes/common.TemporalFailure)
- Java: [TemporalFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TemporalFailure.html)
- Python: [FailureError](https://python.temporal.io/temporalio.exceptions.FailureError.html)
- PHP: [TemporalFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-TemporalFailure.html)

The base [Failure proto message](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) has these fields:

| Field                | Description                                                                                                                              |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `message`            | The error message.                                                                                                                       |
| `stack_trace`        | The stack trace of the error.                                                                                                            |
| `source`             | The SDK the Failure came from, such as `"TypeScriptSDK"`. Some SDKs use this field to rebuild the call stack into an exception object.   |
| `cause`              | The `Failure` message for the cause of this Failure, if there is one.                                                                    |
| `encoded_attributes` | The encoded `message` and `stack_trace` fields when you use a [Failure Converter](/failure-converter) that encodes them.                 |

## Application Failure

Workflow, Activity, and Nexus Operation code use Application Failures to report application-specific errors.
Application Failure is the only Failure type that your code creates and throws.

- TypeScript: [ApplicationFailure](https://typescript.temporal.io/api/classes/common.ApplicationFailure)
- Java: [ApplicationFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ApplicationFailure.html)
- Go: [ApplicationError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ApplicationError)
- Python: [ApplicationError](https://python.temporal.io/temporalio.exceptions.ApplicationError.html)
- PHP: [ApplicationFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ApplicationFailure.html)
- Proto: [ApplicationFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ApplicationFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

### Errors in Workflows

An error in a Workflow causes either a Workflow Task Failure or a Workflow Execution Failure.
A Workflow Task Failure retries the Workflow Task.
A Workflow Execution Failure closes the Workflow Execution with a Failed status.

By default, only exceptions that are Temporal Failures fail the Workflow Execution.
All other exceptions fail the Workflow Task, and the Workflow Task is retried.
The TypeScript, Java, Python, .NET, and Ruby SDKs let you list your own exception types in Worker or Workflow options so that they fail the Workflow Execution instead.
In Go, any error the Workflow returns fails the Workflow Execution, and a panic fails the Workflow Task.

Temporal raises most Failure types for you, such as a [Cancelled Failure](#cancelled-failure) when the Workflow is canceled or an [Activity Failure](#activity-failure) when an Activity fails.
To fail the Workflow Execution from your Workflow Definition, throw an Application Failure.
In Go, return any error.

#### Workflow Task Failures

A Workflow Task Failure means the Worker couldn't process a Workflow Task.
It happens when your Workflow code throws an exception that isn't a Temporal Failure or one of your listed exception types, or panics in Go.
The Temporal Service retries the Workflow Task until the Workflow Execution Timeout, which is unlimited by default.

For the causes the Temporal Service records for each Workflow Task Failure, see the [Workflow Task errors reference](/references/workflow-task-errors).

#### Workflow Execution Failures

Throw an Application Failure in a Workflow to fail the Workflow Execution.
The Workflow Execution moves to the Failed state, and the Temporal Service makes no more attempts to progress it.

To fail the Workflow Execution with your own error category, throw an [Application Failure](#application-failure) and set its `type` field.
Don't extend the Application Failure class or any other Failure class. See [How Temporal represents failures](/encyclopedia/application-failures#failure-representation).

### Errors in Activities

To fail an Activity Task, throw an Application Failure or any other error.
The SDK converts any other error to an Application Failure and sets these fields:

| Field              | Value                                                 |
| ------------------ | ----------------------------------------------------- |
| `type`             | The error's type name.                                |
| `message`          | The error message.                                    |
| `non_retryable`    | `false`                                               |
| `details`          | Unset.                                                |
| `cause`            | A Failure converted from the error's `cause` property. |
| `next_retry_delay` | Unset.                                                |

The SDK also copies the call stack.

When an [Activity Execution](/activity-execution) fails, the Application Failure from the last Activity Task becomes the `cause` field of the [Activity Failure](#activity-failure).
The Workflow's call to the Activity throws the Activity Failure, and your Workflow Definition can handle it.

### Errors in Nexus Operations

A Nexus Operation ends in one of four states: completed, failed, canceled, or timed out.

The caller's Nexus service splits an Operation into one or more StartOperation requests and completion callbacks.
It retries these requests as long as they fail with retryable errors.

The Operation times out only when the schedule-to-close timeout set by the caller Workflow expires.
The caller's Nexus service enforces this timeout.

The Operation reaches one of the other three states when one of these happens:

- The Operation handler returns a synchronous response or error.
- An asynchronous Operation, such as one backed by a Workflow, reaches a terminal state.

A Nexus Operation handler returns a retryable or non-retryable error to tell the caller's Nexus service whether to retry the request.
If a request times out before the handler sends a response, the caller retries it.

Errors are retryable by default. These errors aren't retried:

- Non-retryable Application Failures.
- Unsuccessful Operation errors, which resolve the Operation as failed or canceled.
- [Handler errors](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors) of type `BAD_REQUEST`, `UNAUTHENTICATED`, `UNAUTHORIZED`, `NOT_FOUND`, or `NOT_IMPLEMENTED`.

#### Nexus Operation Task Failures

A Nexus Operation Task Failure means the handler couldn't process a Nexus Operation Task.
It happens when your Nexus handler code throws an unknown error.
The Nexus Operation Task is retried.

#### Nexus Operation Execution Failures

To fail the whole Nexus Operation Execution, throw a non-retryable Application Failure from the Nexus Operation handler.
The Nexus Operation Execution moves to the Failed state, and no more attempts are made to complete it.

#### Propagation of Workflow errors

When a Workflow started by a Nexus `NewWorkflowRunOperation` handler throws an Application Failure, the error reaches the caller as a non-retryable error.
The Nexus Operation Execution fails.

#### Failures in a Nexus handler 

To fail a single Nexus Operation Task or the whole Nexus Operation Execution, throw an Application Failure, a Nexus error, or any other error from the handler.

The SDK converts unknown errors to a retryable Application Failure and sets these fields:

| Field           | Value                  |
| --------------- | ---------------------- |
| `non_retryable` | `false`                |
| `type`          | The error's type name. |
| `message`       | The error message.     |

#### Retryable failures

The caller retries retryable Nexus Operation Task failures, such as an unknown error, with a built-in Retry Policy.
When a Nexus Task fails, the caller Workflow records the failed attempt on the pending Nexus Operation and sets these fields:

| Field                        | Value                                                                       |
| ---------------------------- | --------------------------------------------------------------------------- |
| `state`                      | The new state, such as `BackingOff`.                                        |
| `attempt`                    | The attempt count, incremented by one.                                      |
| `next_attempt_schedule_time` | When the Nexus Task is retried.                                             |
| `last_attempt_failure`       | The error message in `message` and the Application Failure in `failure_info`. |

For example, the Temporal CLI shows an unknown error thrown in a Nexus handler like this:

```
temporal workflow describe -w my-workflow-id
...
Pending Nexus Operations: 1

  Endpoint                 myendpoint
  Service                  my-hello-service
  Operation                echo
  OperationToken
  State                    BackingOff
  Attempt                  6
  ScheduleToCloseTimeout   0s
  NextAttemptScheduleTime  20 seconds from now
  LastAttemptCompleteTime  11 seconds ago
  LastAttemptFailure       {"message":"unexpected response status: "500 Internal Server Error": internal error","applicationFailureInfo":{}}
```

### Non-retryable

When an Activity or Workflow throws an Application Failure, Temporal compares the Failure's `type` field to the Retry Policy's list of [non-retryable errors](/encyclopedia/retry-policies#non-retryable-errors).
If the type is in the list, the Activity or Workflow isn't retried.
To stop retries regardless of the Retry Policy, set the Application Failure's `non_retryable` field to `true`.

When a Nexus Operation handler throws an Application Failure, the caller retries it with a built-in Retry Policy that cannot be customized.
To stop retries, set the Application Failure's `non_retryable` field to `true`.
A non-retryable error from a Nexus handler fails the Nexus Operation Execution, and the caller's Workflow Execution receives it as a [Nexus Operation Failure](#nexus-operation-failure).

### Next Retry Delay 

Set the Next Retry Delay on an Application Failure to control how long Temporal waits before it retries the Activity or Workflow.
This delay overrides the interval the Retry Policy would have calculated for that failure.

- Java: [NextRetryDelay](/develop/java/activities/timeouts#activity-next-retry-delay)
- TypeScript: [nextRetryDelay](/develop/typescript/activities/timeouts#activity-next-retry-delay)
- PHP: [NextRetryDelay](/develop/php/activities/timeouts#activity-next-retry-delay)

### Nexus errors 

#### Default mapping

A Nexus Operation handler that throws an Application Failure returns one of these Nexus errors, depending on `non_retryable`:

| `non_retryable`   | Nexus error                  | HTTP status code          |
| :---------------- | :--------------------------- | :------------------------ |
| `false` (default) | `HandlerErrorTypeInternal`   | 500 Internal Server Error |
| `true`            | `UnsuccessfulOperationError` | 424 Failed Dependency     |

#### Use Nexus errors directly

Throw a Nexus error from your Nexus Operation handler instead of an Application Failure.
A Nexus error carries the retry behavior listed in the following tables and maps to a more specific HTTP status code for external Nexus callers that support it.

For example, the Nexus Go SDK provides these errors:

- `nexus.HandlerError(nexus.HandlerErrorType, msg)`
- `nexus.UnsuccessfulOperationError{state, failure}`

#### Retryable Nexus errors

| Nexus error type                    | `non_retryable` |
| :---------------------------------- | :-------------- |
| `HandlerErrorTypeResourceExhausted` | `false`         |
| `HandlerErrorTypeInternal`          | `false`         |
| `HandlerErrorTypeUnavailable`       | `false`         |

#### Non-retryable Nexus errors

| Nexus error type                  | `non_retryable` |
| :-------------------------------- | :-------------- |
| `HandlerErrorTypeBadRequest`      | `true`          |
| `HandlerErrorTypeUnauthenticated` | `true`          |
| `HandlerErrorTypeUnauthorized`    | `true`          |
| `HandlerErrorTypeNotFound`        | `true`          |
| `HandlerErrorTypeNotImplemented`  | `true`          |
| `UnsuccessfulOperationError`      | `true`          |

## Cancelled Failure

When [Cancellation](/activity-execution#cancellation) of a Workflow, Activity, or Nexus Operation is requested, each SDK represents it in its own way.
For example, in TypeScript, some Workflow API functions throw a Cancelled Failure directly, and others wrap it in a different Failure.
Use the TypeScript [isCancellation](https://typescript.temporal.io/api/namespaces/workflow#iscancellation) helper to check for both.

When a Workflow, Activity, or Nexus Operation is canceled, the Cancelled Failure is the `cause` field of the Activity Failure, Nexus Operation Failure, or "Workflow failed" error.

- TypeScript: [CancelledFailure](https://typescript.temporal.io/api/classes/common.CancelledFailure)
- Java: [CanceledFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/CanceledFailure.html)
- Go: [CanceledError](https://pkg.go.dev/go.temporal.io/sdk/temporal#CanceledError)
- Python: [CancelledError](https://python.temporal.io/temporalio.exceptions.CancelledError.html)
- PHP: [CanceledFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-CanceledFailure.html)
- Proto: [CanceledFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.CanceledFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

## Activity Failure

The Workflow Execution receives an Activity Failure when an Activity fails.
It carries details about the Activity Execution, such as the Activity Type and Activity ID.
The `cause` field holds the reason for the failure.
For example, if the Activity Execution times out, the `cause` is a [Timeout Failure](#timeout-failure).

- TypeScript: [ActivityFailure](https://typescript.temporal.io/api/classes/common.ActivityFailure)
- Java: [ActivityFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ActivityFailure.html)
- Go: [ActivityError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ActivityError)
- Python: [ActivityError](https://python.temporal.io/temporalio.exceptions.ActivityError.html)
- PHP: [ActivityFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ActivityFailure.html)
- Proto: [ActivityFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ActivityFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

## Nexus Operation Failure

The Workflow Execution receives a Nexus Operation Failure when a Nexus Operation fails.
It carries details about the Nexus Operation Execution, such as the Operation name and Operation token.
The `message` and `cause` fields hold the reason for the failure.
The `cause` is usually an Application Failure or a Cancelled Failure.

- Go: [NexusOperationError](https://pkg.go.dev/go.temporal.io/sdk/temporal#NexusOperationError)
- Proto: [NexusOperationFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.NexusOperationFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

A Nexus Operation Failure has these fields:

| Field                | Value                                                                                                                      |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `endpoint`           | The name of the Nexus Endpoint.                                                                                            |
| `service`            | The name of the Nexus Service.                                                                                             |
| `operation`          | The name of the Operation.                                                                                                 |
| `operation_token`    | The Operation token, set for an asynchronous Operation. Use it to act on the Operation, such as to cancel it.               |
| `scheduled_event_id` | The ID of the Event in the caller's Event History that scheduled the Operation.                                            |
| `message`            | A generic error message for an unsuccessful Operation.                                                                     |
| `cause`              | The underlying Application Failure, with `non_retryable` set to `true`, `type` set to the error's type name, and `message` set to the error message. |
| `nexus_error_code`   | The underlying Nexus error code.                                                                                           |

## Child Workflow Failure

The Workflow Execution receives a Child Workflow Failure when a Child Workflow Execution fails.
It carries details about the Child Workflow Execution, such as the Workflow Type and Workflow ID.
The `cause` field holds the reason for the failure.

- TypeScript: [ChildWorkflowFailure](https://typescript.temporal.io/api/classes/common.ChildWorkflowFailure)
- Java: [ChildWorkflowFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ChildWorkflowFailure.html)
- Go: [ChildWorkflowExecutionError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ChildWorkflowExecutionError)
- Python: [ChildWorkflowError](https://python.temporal.io/temporalio.exceptions.ChildWorkflowError.html)
- PHP: [ChildWorkflowFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ChildWorkflowFailure.html)
- Proto: [ChildWorkflowExecutionFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ChildWorkflowExecutionFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

## Timeout Failure

A Timeout Failure represents the timeout of an Activity or Workflow.
When an Activity times out, the Timeout Failure includes the last Heartbeat details the Activity sent.

- TypeScript: [TimeoutFailure](https://typescript.temporal.io/api/classes/common.TimeoutFailure)
- Java: [TimeoutFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TimeoutFailure.html)
- Go: [TimeoutError](https://pkg.go.dev/go.temporal.io/sdk/temporal#TimeoutError)
- Python: [TimeoutError](https://python.temporal.io/temporalio.exceptions.TimeoutError.html)
- PHP: [TimeoutFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-TimeoutFailure.html)
- Proto: [TimeoutFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.TimeoutFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

## Terminated Failure

When a Workflow is terminated, a Terminated Failure is the `cause` of the error you receive in these places:

- In a parent Workflow that's waiting for the result of a Child Workflow.
- In a Client that's waiting for the result of a Workflow.

The SDK and proto types are:

- TypeScript: [TerminatedFailure](https://typescript.temporal.io/api/classes/common.TerminatedFailure)
- Java: [TerminatedFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TerminatedFailure.html)
- Go: [TerminatedError](https://pkg.go.dev/go.temporal.io/sdk/temporal#TerminatedError)
- Python: [TerminatedError](https://python.temporal.io/temporalio.exceptions.TerminatedError.html)
- PHP: [TerminatedFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-TerminatedFailure.html)
- Proto: [TerminatedFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.TerminatedFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

## Server Failure

A Server Failure represents an error that comes from the Temporal Service.

- TypeScript: [ServerFailure](https://typescript.temporal.io/api/classes/common.ServerFailure)
- Java: [ServerFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ServerFailure.html)
- Go: [ServerError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ServerError)
- Python: [ServerError](https://python.temporal.io/temporalio.exceptions.ServerError.html)
- PHP: [ServerFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ServerFailure.html)
- Proto: [ServerFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ServerFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure)

---

# Temporal Cloud operations reference

Source: https://docs.temporal.io/references/operation-list

> Operations Temporal Cloud counts toward a Namespace's operations per second (OPS) limit, with each one's priority and what happens when it's throttled.

> **ℹ️ Info:**
>
> For the OPS limit itself and how it scales, see [Operations per second](/evaluate/cloud/limits#operations-per-second).
> For the primary rate limit on a Namespace, see [Actions](/evaluate/cloud/actions) and [Capacity Modes](/cloud/capacity-modes).
>

Temporal Cloud rate limits each Namespace using three measures.
Actions per second (APS) is the primary limit, and [Capacity Modes](/cloud/capacity-modes) are expressed in APS.
Requests per second (RPS) and operations per second (OPS) are lower-level limits that protect the underlying services.

An operation is any work that puts load on Temporal Server, whether a user starts it directly or Temporal runs it in the background on the user's behalf.
Visibility queries are the exception.
They pass through Temporal Server, but most of their load falls on the Visibility store, so they have their own rate limits separate from OPS.

Each operation in the table lists the following:

- **Description:** What the operation does.
- **Priority:** Foreground operations have higher priority than background operations.
- **Effect of throttling:** What you observe when Temporal Cloud throttles the operation.

---

# Temporal SDK metrics reference

Source: https://docs.temporal.io/references/sdk-metrics

> Metrics Temporal SDKs emit from Clients and Workers, with each metric's type, tags, and the SDKs that define it.

> **ℹ️ Info:**
>
> For metrics emitted by [Temporal Cloud](/evaluate/cloud), see [Cloud metrics](/cloud/metrics/openmetrics).
> For metrics emitted by a [self-hosted Temporal Service](/temporal-service), see [Temporal Service metrics](/references/service-metrics).
> To tune Workers based on these metrics, see [Worker performance](/develop/worker-performance).
>

The [Temporal SDKs](/encyclopedia/architecture/temporal-sdks) emit metrics from Temporal Client usage and Worker Processes.
To set up metrics in your application, see the observability guide for your SDK:
[Go](/develop/go/platform/observability#metrics),
[Java](/develop/java/platform/observability#metrics),
[Python](/develop/python/platform/observability#metrics),
[TypeScript](/develop/typescript/platform/observability#metrics),
[.NET](/develop/dotnet/platform/observability#metrics), or
[Ruby](/develop/ruby/platform/observability#metrics).

Only the metrics on this page have guaranteed, defined behavior.
Some SDKs emit other metrics.
Treat those as deprecated, inconsistent, or experimental.

The SDKs add the `temporal_` prefix to every metric before exporting it.
Headings on this page omit the prefix.

Each metric on this page lists the following:

- **Type:** Counter, Gauge, or Histogram.
- **Available in:** The SDK that defines the metric.
  Other SDKs inherit their metrics from one of these:
  - **Core:** The [Core SDK](/glossary#core-sdk) defines the metrics for the TypeScript, Python, .NET, and Ruby SDKs, in [Worker metrics](https://github.com/temporalio/sdk-rust/blob/main/crates/sdk-core/src/telemetry/metrics.rs) and [Client metrics](https://github.com/temporalio/sdk-rust/blob/main/crates/client/src/metrics.rs).
  - **Go:** The Go SDK defines the metrics for the Go and PHP SDKs, in [Worker and Client metrics](https://github.com/temporalio/sdk-go/blob/main/internal/common/metrics/constants.go).
  - **Java:** The Java SDK defines its own metrics, in [Worker metrics](https://github.com/temporalio/sdk-java/blob/main/temporal-sdk/src/main/java/io/temporal/worker/MetricsType.java) and [Client metrics](https://github.com/temporalio/sdk-java/blob/main/temporal-serviceclient/src/main/java/io/temporal/serviceclient/MetricsType.java).
- **Tags:** The [tags](/glossary#tag) the SDK attaches to the metric.
  Prometheus and other OpenMetrics-compatible systems call these labels.

> **📝 Note:**
> Metric units across SDKs
>
> The unit for Histogram metrics depends on the SDK:
>
> - **Core-based SDKs:** Milliseconds by default.
>   You can configure Core-based SDKs to use seconds instead.
> - **Go and Java SDKs:** Seconds.
>

Each metric may have some combination of the following tags:

- `task_queue`: The Task Queue the Worker Entity polls.
- `namespace`: The Namespace the Worker is bound to.
- `poller_type`: One of the following:
  - `workflow_task`
  - `workflow_sticky_task`
  - `activity_task`
  - `nexus_task`
- `worker_type`: One of the following:
  - `ActivityWorker`
  - `WorkflowWorker`
  - `LocalActivityWorker`
  - `NexusWorker`
- `activity_type`: The name of the Activity Function.
- `workflow_type`: The name of the Workflow Function.
- `operation`: The RPC method name. Available on Temporal Client gRPC request metrics.
- `status_code`: The gRPC status code the Temporal Service returned, in `UPPER_SNAKE_CASE`, such as `NOT_FOUND`. Available on Temporal Client gRPC request failure metrics.
- `cause`: The reason the Temporal Service rejected a request as resource exhausted, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`. Available on the resource-exhausted metrics.

Not every SDK attaches every tag, and Histogram buckets differ between SDKs.

| Metric name                                                                                  | Emitted by     | Metric type | Availability   |
| -------------------------------------------------------------------------------------------- | -------------- | ----------- | -------------- |
| [temporal_activity_execution_cancelled](#activity_execution_cancelled)                       | Worker         | Counter     | Java           |
| [temporal_activity_execution_failed](#activity_execution_failed)                             | Worker         | Counter     | Core, Go, Java |
| [temporal_activity_execution_latency](#activity_execution_latency)                           | Worker         | Histogram   | Core, Go, Java |
| [temporal_activity_poll_no_task](#activity_poll_no_task)                                     | Worker         | Counter     | Core, Go, Java |
| [temporal_activity_schedule_to_start_latency](#activity_schedule_to_start_latency)           | Worker         | Histogram   | Core, Go, Java |
| [temporal_activity_succeed_endtoend_latency](#activity_succeed_endtoend_latency)             | Worker         | Histogram   | Core, Go, Java |
| [temporal_activity_task_error](#activity_task_error)                                         | Worker         | Counter     | Go             |
| [temporal_corrupted_signals](#corrupted_signals)                                             | Worker         | Counter     | Go, Java       |
| [temporal_local_activity_execution_cancelled](#local_activity_execution_cancelled)           | Worker         | Counter     | Core, Go, Java |
| [temporal_local_activity_execution_failed](#local_activity_execution_failed)                 | Worker         | Counter     | Core, Go, Java |
| [temporal_local_activity_execution_latency](#local_activity_execution_latency)               | Worker         | Histogram   | Core, Go, Java |
| [temporal_local_activity_succeed_endtoend_latency](#local_activity_succeed_endtoend_latency) | Worker         | Histogram   | Core, Go, Java |
| [temporal_local_activity_total](#local_activity_total)                                       | Worker         | Counter     | Core, Go, Java |
| [temporal_long_request](#long_request)                                                       | Service Client | Counter     | Core, Go, Java |
| [temporal_long_request_failure](#long_request_failure)                                       | Service Client | Counter     | Core, Go, Java |
| [temporal_long_request_latency](#long_request_latency)                                       | Service Client | Histogram   | Core, Go, Java |
| [temporal_long_request_resource_exhausted](#long_request_resource_exhausted)                 | Service Client | Counter     | Go             |
| [temporal_nexus_poll_no_task](#nexus_poll_no_task)                                           | Worker         | Counter     | Core, Go, Java |
| [temporal_nexus_task_schedule_to_start_latency](#nexus_task_schedule_to_start_latency)       | Worker         | Histogram   | Core, Go, Java |
| [temporal_nexus_task_execution_failed](#nexus_task_execution_failed)                         | Worker         | Counter     | Core, Go, Java |
| [temporal_nexus_task_execution_latency](#nexus_task_execution_latency)                       | Worker         | Histogram   | Core, Go, Java |
| [temporal_nexus_task_endtoend_latency](#nexus_task_endtoend_latency)                         | Worker         | Histogram   | Core, Go, Java |
| [temporal_num_pollers](#num_pollers)                                                         | Worker         | Gauge       | Core, Go, Java |
| [temporal_poller_start](#poller_start)                                                       | Worker         | Counter     | Go, Java       |
| [temporal_request](#request)                                                                 | Service Client | Counter     | Core, Go, Java |
| [temporal_request_failure](#request_failure)                                                 | Service Client | Counter     | Core, Go, Java |
| [temporal_request_latency](#request_latency)                                                 | Service Client | Histogram   | Core, Go, Java |
| [temporal_request_resource_exhausted](#request_resource_exhausted)                           | Service Client | Counter     | Go             |
| [temporal_resource_slots_cpu_usage](#resource_slots_cpu_usage)                               | Worker         | Gauge       | Core, Java     |
| [temporal_resource_slots_mem_usage](#resource_slots_mem_usage)                               | Worker         | Gauge       | Core, Java     |
| [temporal_sticky_cache_hit](#sticky_cache_hit)                                               | Worker         | Counter     | Core, Go, Java |
| [temporal_sticky_cache_miss](#sticky_cache_miss)                                             | Worker         | Counter     | Core, Go, Java |
| [temporal_sticky_cache_size](#sticky_cache_size)                                             | Worker         | Gauge       | Core, Go, Java |
| [temporal_sticky_cache_total_forced_eviction](#sticky_cache_total_forced_eviction)           | Worker         | Counter     | Core, Go, Java |
| [temporal_unregistered_activity_invocation](#unregistered_activity_invocation)               | Worker         | Counter     | Go             |
| [temporal_worker_start](#worker_start)                                                       | Worker         | Counter     | Core, Go, Java |
| [temporal_worker_task_slots_available](#worker_task_slots_available)                         | Worker         | Gauge       | Core, Go, Java |
| [temporal_worker_task_slots_used](#worker_task_slots_used)                                   | Worker         | Gauge       | Core, Go, Java |
| [temporal_workflow_active_thread_count](#workflow_active_thread_count)                       | Worker         | Gauge       | Java           |
| [temporal_workflow_canceled](#workflow_canceled)                                             | Worker         | Counter     | Core, Go, Java |
| [temporal_workflow_completed](#workflow_completed)                                           | Worker         | Counter     | Core, Go, Java |
| [temporal_workflow_continue_as_new](#workflow_continue_as_new)                               | Worker         | Counter     | Core, Go, Java |
| [temporal_workflow_endtoend_latency](#workflow_endtoend_latency)                             | Worker         | Histogram   | Core, Go, Java |
| [temporal_workflow_failed](#workflow_failed)                                                 | Worker         | Counter     | Core, Go, Java |
| [temporal_workflow_task_execution_failed](#workflow_task_execution_failed)                   | Worker         | Counter     | Core, Go, Java |
| [temporal_workflow_task_execution_latency](#workflow_task_execution_latency)                 | Worker         | Histogram   | Core, Go, Java |
| [temporal_workflow_task_queue_poll_empty](#workflow_task_queue_poll_empty)                   | Worker         | Counter     | Core, Go, Java |
| [temporal_workflow_task_queue_poll_succeed](#workflow_task_queue_poll_succeed)               | Worker         | Counter     | Core, Go, Java |
| [temporal_workflow_task_replay_latency](#workflow_task_replay_latency)                       | Worker         | Histogram   | Core, Go, Java |
| [temporal_workflow_task_schedule_to_start_latency](#workflow_task_schedule_to_start_latency) | Worker         | Histogram   | Core, Go, Java |

### `activity_execution_cancelled`

An Activity Execution was canceled.

- Type: Counter
- Available in: Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `activity_execution_failed`

An Activity Execution failed.
In the Go and Java SDKs, this metric doesn't count Local Activity failures.
For those, see [`local_activity_execution_failed`](#local_activity_execution_failed).

- Type: Counter
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`, `failure_reason`

Valid values for the `failure_reason` tag:

- `PayloadsTooLarge`: A Payload in the Activity Task response exceeded the Namespace size limit, so the Worker failed the Activity Task as retryable before sending it.
  See [Activity Payloads too large](/troubleshooting/execution-failures#activity-payloads-too-large).
- `ActivityError`: The Activity Task failed for any other reason.

### `activity_execution_latency`

Time to complete an Activity Execution, from when the Activity Task is generated to when the SDK responds with a success or failure.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `activity_poll_no_task`

An Activity Worker's poll for an Activity Task timed out because no Activity Task was available on the Task Queue.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `activity_schedule_to_start_latency`

The Schedule-To-Start time of an Activity Task.
Use this metric to check that Workers pick up Activity Tasks from the Task Queue promptly.
To limit this time, set a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) when you start the Activity Execution.

Some SDKs add the `activity_type` and `workflow_type` tags.
The value doesn't vary by type, because the type doesn't affect how fast Workers pull tasks from the Task Queue.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `activity_succeed_endtoend_latency`

Total time for a successful Activity Execution, from when it's scheduled to when it completes.
The SDK doesn't record this metric for Activities that complete asynchronously.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `activity_task_error`

An internal error or panic occurred while the Worker handled or ran an Activity Task.

- Type: Counter
- Available in: Go
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `corrupted_signals`

The number of Signals whose Payload the Worker couldn't deserialize.

- Type: Counter
- Available in: Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `local_activity_execution_cancelled`

A Local Activity Execution was canceled.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `local_activity_execution_failed`

A Local Activity Execution failed.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`, `failure_reason`

Valid values for the `failure_reason` tag:

- `timeout`: The Local Activity exceeded its Schedule-To-Close Timeout or Start-To-Close Timeout.
- `ActivityError`: The Local Activity failed for any other reason.

### `local_activity_execution_latency`

Time to complete a Local Activity Execution, from when the first Activity Task is generated to when the SDK responds that the execution is complete.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `local_activity_succeed_endtoend_latency`

Total time for a successful Local Activity Execution, from when it's scheduled to when it completes.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `local_activity_total`

The total number of [Local Activity Executions](/local-activity).

- Type: Counter
- Available in: Core, Go, Java
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `long_request`

The Temporal Client made a gRPC long-poll request.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `operation`

### `long_request_failure`

A gRPC long-poll request from the Temporal Client failed.
The `long_request` counter also counts these requests.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `operation`, `status_code`

The `status_code` tag holds the gRPC status code the Temporal Service returned.
Values match the gRPC status code names in `UPPER_SNAKE_CASE`, such as `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, and `INTERNAL`.
Client options can suppress this tag, so confirm your metrics endpoint includes it before you write queries against it.

### `long_request_latency`

Latency of a gRPC long-poll request from the Temporal Client.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `operation`

### `long_request_resource_exhausted`

The Temporal Service rejected a gRPC long-poll request from the Temporal Client as resource exhausted.
The `long_request_failure` metric also counts these failures with `status_code=RESOURCE_EXHAUSTED`.
This metric adds the reason for the rejection in the `cause` tag.

- Type: Counter
- Available in: Go
- Tags: `namespace`, `operation`, `cause`

The `cause` tag holds the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`.
Use this metric instead of `long_request_failure` to investigate throttling, because the fix depends on the cause.

### `nexus_poll_no_task`

A Nexus Worker's poll for a Nexus Task timed out because no Nexus Task was available on the Task Queue.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `nexus_task_schedule_to_start_latency`

The Schedule-To-Start time of a Nexus Task, from when the request reaches the Frontend Service to when the SDK starts processing the task.
The `Request-Timeout` header on the request to the Frontend Service limits this time.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `nexus_task_execution_failed`

Handling a Nexus Task resulted in an error.
This includes errors returned from your handler and unexpected internal errors in the SDK.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `nexus_service`, `nexus_operation`, `failure_reason`

Valid values for the `failure_reason` tag:

- `internal_sdk_error`: The SDK hit an unexpected internal error while handling the Nexus Task.
  This indicates a bug in the SDK.
- `handler_error_{TYPE}`: Your handler returned a [predefined handler error](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors) from the Nexus specification.
  If the handler returns an unexpected error, `{TYPE}` is `INTERNAL`.
- `timeout`: Your handler didn't return within the request timeout.
- `operation_failed`: Your handler reported that the Nexus Operation failed.
  In Go, this is an `UnsuccessfulOperationError` with a `failed` state.
- `operation_canceled`: Your handler reported that the Nexus Operation completed as canceled.
  In Go, this is an `UnsuccessfulOperationError` with a `canceled` state.

### `nexus_task_execution_latency`

Time to complete a Nexus Task, from when the SDK starts processing it to when your handler completes.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `nexus_service`, `nexus_operation`

### `nexus_task_endtoend_latency`

Total time for a Nexus Task, from when the request reaches the Frontend Service to when the Temporal Service acknowledges the task's completion to the SDK.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `nexus_service`, `nexus_operation`

### `num_pollers`

The current number of Worker Entities that are polling.

- Type: Gauge
- Available in: Core, Go, Java
- Tags: `namespace`, `poller_type`, `task_queue`

### `poller_start`

A Worker Entity started a poller.

- Type: Counter
- Available in: Go, Java
- Tags: `namespace`, `task_queue`

### `request`

The Temporal Client made a gRPC request.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `operation`

### `request_failure`

A gRPC request from the Temporal Client failed.
The `request` counter also counts these requests.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `operation`, `status_code`

The `status_code` tag holds the gRPC status code the Temporal Service returned.
Values match the gRPC status code names in `UPPER_SNAKE_CASE`, such as `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, and `INTERNAL`.
Client options can suppress this tag, so confirm your metrics endpoint includes it before you write queries against it.

### `request_latency`

Latency of a Temporal Client gRPC request.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `operation`

### `request_resource_exhausted`

The Temporal Service rejected a gRPC request from the Temporal Client as resource exhausted.
The `request_failure` metric also counts these failures with `status_code=RESOURCE_EXHAUSTED`.
This metric adds the reason for the rejection in the `cause` tag.

- Type: Counter
- Available in: Go
- Tags: `namespace`, `operation`, `cause`

The `cause` tag holds the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`.
Use this metric instead of `request_failure` to investigate throttling, because the fix depends on the cause.

### `resource_slots_cpu_usage`

CPU usage, from 0 to 100, as measured by the resource-based slot tuner.
The SDK emits this metric only when the resource-based slot tuner is enabled.

- Type: Gauge
- Available in: Core, Java

### `resource_slots_mem_usage`

Memory usage, from 0 to 100, as measured by the resource-based slot tuner.
The SDK emits this metric only when the resource-based slot tuner is enabled.

- Type: Gauge
- Available in: Core, Java

### `sticky_cache_hit`

A Workflow Task found a cached Workflow Execution to run against.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `sticky_cache_miss`

A Workflow Task didn't find a cached Workflow Execution to run against.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `sticky_cache_size`

The current number of Workflow Executions in the cache.

- Type: Gauge
- Available in: Core, Go, Java
- Tags: `namespace` (TypeScript, Java), `task_queue` (TypeScript)

### `sticky_cache_total_forced_eviction`

The Worker evicted a Workflow Execution from the cache on purpose.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `unregistered_activity_invocation`

The Worker received an Activity Task for an Activity type that isn't registered with it.

- Type: Counter
- Available in: Go
- Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type`

### `worker_start`

A Worker Entity was registered, created, or started.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `worker_type`

### `worker_task_slots_available`

The number of Workflow, Activity, Local Activity, or Nexus Task execution slots currently available.
Use the `worker_type` tag to tell the slot types apart.
For example, `WorkflowWorker` slots run Workflow Tasks and `ActivityWorker` slots run Activity Tasks.

- Type: Gauge
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `worker_type`

### `worker_task_slots_used`

The number of Workflow, Activity, Local Activity, or Nexus Task execution slots currently in use.
Use the `worker_type` tag to tell the slot types apart.
For example, `WorkflowWorker` slots run Workflow Tasks and `ActivityWorker` slots run Activity Tasks.

- Type: Gauge
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `worker_type`

### `workflow_active_thread_count`

The number of Workflow threads in the Worker Process.

- Type: Gauge
- Available in: Java

### `workflow_canceled`

A Workflow Execution ended because of a cancellation request.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `workflow_completed`

A Workflow Execution completed successfully.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `workflow_continue_as_new`

A Workflow Execution ended with Continue-As-New.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `workflow_endtoend_latency`

Total time for a single Workflow Run, from when it's scheduled to when it completes.
Each retry of a Workflow Execution is a separate Run.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `workflow_failed`

A Workflow Execution failed.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `workflow_task_execution_failed`

A Workflow Task Execution failed.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`, `failure_reason`

Valid values for the `failure_reason` tag:

- `NonDeterminismError`: The Workflow Task failed because of a non-determinism error.
- `GrpcMessageTooLarge`: The Workflow Task response exceeded the gRPC message size limit and couldn't be delivered.
  The Temporal Service terminates the Workflow Execution in response.
  See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error).
- `PayloadsTooLarge`: A Payload or Memo in the Workflow Task response exceeded the Namespace size limit, so the Worker failed the Workflow Task as retryable before sending it.
  See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error#payload-size-limit).
- `RequestTooLarge`: The whole Workflow Task completion exceeded the Namespace limit on the size of a single request.
  The Worker failed the Workflow Task instead of sending a request the Temporal Service would reject, and the Temporal Service retries the Workflow Task.
  See [Request too large](/troubleshooting/execution-failures#request-too-large).
- `WorkflowError`: The Workflow Task failed for any other reason.

### `workflow_task_execution_latency`

Time to run a Workflow Task Execution.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `workflow_task_queue_poll_empty`

A Workflow Worker polled a Task Queue and timed out without picking up a Workflow Task.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `workflow_task_queue_poll_succeed`

A Workflow Worker polled a Task Queue and successfully picked up a Workflow Task.

- Type: Counter
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

### `workflow_task_replay_latency`

Time the Worker spends replaying Event History to catch up before running a Workflow Task.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`, `workflow_type`

### `workflow_task_schedule_to_start_latency`

The Schedule-To-Start time of a Workflow Task.

- Type: Histogram
- Available in: Core, Go, Java
- Tags: `namespace`, `task_queue`

---

# Temporal Server options reference

Source: https://docs.temporal.io/references/server-options

> ServerOption functions for embedding the Temporal Server as a Go library with temporal.NewServer(), covering config, shutdown, auth, TLS, and metrics.

> **ℹ️ Info:**
>
> For when and how to run the Temporal Server in-process, see [Embedding Temporal server as a Go library](/self-hosted-guide/embedded-server).
>
> For the settings in the configuration file, see the [Temporal Service configuration reference](/references/service-configuration).
>

You can run the [Temporal Server](/temporal-service/temporal-server) as a Go application.
Import the `go.temporal.io/server/temporal` package, then create and start a server with `temporal.NewServer()`.

```go
s, err := temporal.NewServer()
if err != nil {
    log.Fatal(err)
}
err = s.Start()
if err != nil {
    log.Fatal(err)
}
```

`NewServer()` accepts functions that each return a `ServerOption`.
The server applies each option to the instance it creates.

Server options are defined in [server_option.go](https://github.com/temporalio/temporal/blob/main/temporal/server_option.go) in the Temporal Server repository.

## WithConfig

Sets the static configuration the server starts with.
Without this option, the server looks for a configuration file at `./config/development.yaml`.

For the configuration structure, see the [`config` package](https://pkg.go.dev/go.temporal.io/server/common/config) and the [Temporal Service configuration reference](/references/service-configuration).

```go
s, err := temporal.NewServer(
    temporal.WithConfig(cfg),
)
```

## WithConfigLoader

Loads the configuration from files in `configDir`, selected by environment (`env`) and availability zone (`zone`).

```go
s, err := temporal.NewServer(
    temporal.WithConfigLoader(configDir, env, zone),
)
```

## WithDynamicConfigClient

Sets the `dynamicconfig.Client` that the server reads [dynamic configuration](/references/dynamic-configuration) from.
`dynamicconfig.Client` is defined in the `go.temporal.io/server/common/dynamicconfig` package.

Without this option, the server reads the file named in the [`dynamicConfigClient`](/references/service-configuration#dynamicconfigclient) section of its static configuration.
If that section is also missing, the server uses the default value for every dynamic configuration key.

```go
s, err := temporal.NewServer(
    temporal.WithDynamicConfigClient(myDynamicConfigClient),
)
```

## ForServices

Sets the Temporal services the server runs.
To run every service, pass `temporal.DefaultServices` from the `go.temporal.io/server/temporal` package.

```go
s, err := temporal.NewServer(
    temporal.ForServices(temporal.DefaultServices),
)
```

## InterruptOn

Sets a channel that stops the server when it receives a signal.
The value you pass controls whether `s.Start()` blocks:

| Value                                          | Behavior of `s.Start()`                                               |
| ---------------------------------------------- | --------------------------------------------------------------------- |
| Option not passed                              | Returns without blocking. Call `s.Stop()` to stop the server.         |
| `temporal.InterruptOn(nil)`                    | Blocks until the process is killed.                                   |
| `temporal.InterruptOn(temporal.InterruptCh())` | Blocks until you press Ctrl+C, then shuts the server down gracefully. |
| `temporal.InterruptOn(yourCustomChan)`         | Blocks until a signal is sent to `yourCustomChan`.                    |

```go
s, err := temporal.NewServer(
    temporal.InterruptOn(temporal.InterruptCh()),
)
```

## WithAuthorizer

Sets the [`Authorizer`](/self-hosted-guide/security#authorizer-plugin) that allows or denies inbound API calls.

```go
s, err := temporal.NewServer(
    temporal.WithAuthorizer(myAuthorizer),
)
```

## WithTLSConfigFactory

Overrides the default TLS configuration provider.
`TLSConfigProvider` is defined in the `go.temporal.io/server/common/rpc/encryption` package.

```go
s, err := temporal.NewServer(
    temporal.WithTLSConfigFactory(yourTLSConfigProvider),
)
```

## WithClaimMapper

Sets the [`ClaimMapper`](/self-hosted-guide/security#claim-mapper) that maps a caller's credentials to `Claims` for authorization.

```go
s, err := temporal.NewServer(
    temporal.WithClaimMapper(func(cfg *config.Config) authorization.ClaimMapper {
        logger := getYourLogger() // Replace with how you retrieve or initialize your logger
        return authorization.NewDefaultJWTClaimMapper(
            authorization.NewDefaultTokenKeyProvider(&cfg.Global.Authorization, logger),
            &cfg.Global.Authorization,
            logger,
        )
    }),
)
```

## WithTokenProvider

Sets the [`TokenProvider`](/self-hosted-guide/security#token-provider) that supplies bearer tokens for outbound RPCs to remote clusters in [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication).
`TokenProvider` is defined in the `go.temporal.io/server/common/rpc/auth` package.

The token requires TLS to the remote cluster.
Configure it with [`global.tls.remoteClusters`](/references/service-configuration#tls) or with [WithTLSConfigFactory](#withtlsconfigfactory).

```go
s, err := temporal.NewServer(
    temporal.WithTokenProvider(myTokenProvider),
)
```

## WithLogger

Sets the `log.Logger` that the server writes its logs to.
`log.Logger` is defined in the `go.temporal.io/server/common/log` package.

Without this option, the server builds a logger from the [`log`](/references/service-configuration#log) section of its static configuration.

```go
s, err := temporal.NewServer(
    temporal.WithLogger(myLogger),
)
```

## WithCustomMetricsHandler

Sets a custom implementation of the `metrics.Handler` interface that publishes the server's metrics.

For the metrics the Temporal Server emits, see the [Temporal Service metrics reference](/references/service-metrics).

```go
s, err := temporal.NewServer(
    temporal.WithCustomMetricsHandler(myHandler),
)
```

---

# Temporal Service configuration reference

Source: https://docs.temporal.io/references/service-configuration

> Settings in the development.yaml file for a self-hosted Temporal Service, covering global options, persistence, service roles, TLS, and archival.

> **ℹ️ Info:**
>
> The information on this page is relevant to open source [Temporal Service deployments](/temporal-service).
>
> For settings you can change without a restart, see the [Dynamic configuration reference](/references/dynamic-configuration).
>

A self-hosted Temporal Service reads its static configuration from the `development.yaml` file.
Changes to this file take effect after you restart the Temporal Service process.

The configuration structs are defined in [config.go](https://github.com/temporalio/temporal/blob/main/common/config/config.go) in the Temporal Server repository.

The file can contain the following top-level sections:

- [`global`](#global): Process-wide settings for membership, metrics, profiling, TLS, and authorization.
- [`persistence`](#persistence): Data stores for Workflow state and Visibility.
- [`log`](#log): Log output and level.
- [`clusterMetadata`](#clustermetadata): Local cluster information for Multi-Cluster Replication.
- [`services`](#services): Network settings for each service role.
- [`publicClient`](#publicclient): How internal services connect to the Frontend Service.
- [`archival`](#archival): Archival of Event History and Visibility data.
- [`namespaceDefaults`](#namespacedefaults): Default Archival settings for new Namespaces.
- [`dcRedirectionPolicy`](#dcredirectionpolicy): API forwarding between clusters.
- [`dynamicConfigClient`](#dynamicconfigclient): Location of the dynamic configuration file.

Settings marked **Required** must be set.
Settings with a default use that value when you leave them out.

## global

The `global` section contains process-wide settings.
The following example shows a minimal configuration:

```yaml
global:
  membership:
    broadcastAddress: '127.0.0.1'
  metrics:
    prometheus:
      framework: 'tally'
      listenAddress: '127.0.0.1:8000'
```

### membership

The `membership` section configures the gossip layer that hosts in the Temporal Service use to discover each other.

#### maxJoinDuration

The amount of time the service tries to join the gossip layer before it fails.

Default: `10s`

#### broadcastAddress

The address the gossip protocol advertises to other hosts in the same Temporal Service.
Use an IP address that the other hosts can reach.
If the Temporal Service runs on one host, you can use `127.0.0.1`.

Only IPv4 addresses are supported.
For supported syntax, see [`net.ParseIP`](https://pkg.go.dev/net#ParseIP).

### metrics

The `metrics` section configures the metrics the Temporal Service emits.
Configure a provider by using its name as the key: [`statsd`](#statsd) or [`prometheus`](#prometheus).

For the metrics the Temporal Service emits, see the [Temporal Service metrics reference](/references/service-metrics).

#### prefix

The prefix added to the name of every metric the Temporal Service emits.

#### tags

Key-value pairs added to every metric the Temporal Service emits.

#### excludeTags

A map of tag names to lists of tag values.
Use it to drop tags with unbounded cardinality, such as `task_queue`.
The Temporal Service still reports the tag values you list.

#### statsd

> **⚠️ Caution:**
>
> Temporal does not support `statsd` natively.
>

- `hostPort`: The `host:port` of the statsd server.
- `prefix`: The prefix for metrics reported to statsd.
- `flushInterval`: The maximum interval between packets.
  - Default: `300ms`.
- `flushBytes`: The maximum UDP packet size, in bytes.
  - Default: `1432`.

#### prometheus

- `framework`: The metrics framework. Supported values are `tally` and `opentelemetry`.
  - Default: `tally`.
- `listenAddress`: The address the Temporal Service listens on for Prometheus scrape requests.
- `handlerPath`: The HTTP path of the metrics handler.
  - Default: `/metrics`.

### pprof

- `port`: The port for [pprof](https://pkg.go.dev/net/http/pprof).
  When set, the Temporal Service starts pprof on this port at process start.

### tls

The `tls` section configures TLS for network traffic.
It has two subsections:

- `internode`: Traffic between service roles in the Temporal Service.
- `frontend`: Traffic from SDK Clients to the Frontend Service.

Each subsection contains a `server` section and a `client` section.

The `server` section supports the following settings:

- `certFile`: Path to the PEM-encoded certificate.
- `keyFile`: Path to the PEM-encoded private key.
- `requireClientAuth`: Boolean. When `true`, clients must present a certificate (mutual TLS).
- `clientCaFiles`: Paths to PEM-encoded Certificate Authority (CA) certificates to trust for client authentication.
  The Temporal Service ignores this setting when `requireClientAuth` is `false`.

The `client` section supports the following settings:

- `serverName`: The DNS name expected in the server certificate.
  Service roles connect to each other by IP address, so set this unless your server certificates include IP Subject Alternative Names.
- `rootCaFiles`: Paths to the CA certificates that signed the server certificate.
  Set this when the client host doesn't trust the server's root CA.

For more TLS configurations, see the [server samples repository](https://github.com/temporalio/samples-server/tree/main/tls).

The following example enables TLS between SDKs and the Frontend Service:

```yaml
global:
  tls:
    frontend:
      server:
        certFile: /path/to/cert/file
        keyFile: /path/to/key/file
      client:
        serverName: dnsSanInFrontendCertificate
```

The following example adds the root CA for the Frontend Service:

```yaml
global:
  tls:
    frontend:
      server:
        certFile: /path/to/cert/file
        keyFile: /path/to/key/file
      client:
        serverName: dnsSanInFrontendCertificate
        rootCaFiles:
          - /path/to/frontend/server/CA/files
```

The following example uses mutual TLS for both Frontend and internode traffic, with CAs set manually:

```yaml
global:
  tls:
    internode:
      server:
        certFile: /path/to/internode/cert/file
        keyFile: /path/to/internode/key/file
        requireClientAuth: true
        clientCaFiles:
          - /path/to/internode/serverCa
      client:
        serverName: dnsSanInInternodeCertificate
        rootCaFiles:
          - /path/to/internode/serverCa
    frontend:
      server:
        certFile: /path/to/frontend/cert/file
        keyFile: /path/to/frontend/key/file
        requireClientAuth: true
        clientCaFiles:
          - /path/to/internode/serverCa
          - /path/to/sdkClientPool1/ca
          - /path/to/sdkClientPool2/ca
      client:
        serverName: dnsSanInFrontendCertificate
        rootCaFiles:
          - /path/to/frontend/serverCa
```

When client authentication is enabled, service roles use the `internode.server` certificate as their client certificate.
This adds the following requirements:

- Specify the `internode.server` certificate on every role, including Frontend-only configurations.
- Mint internode server certificates with either no Extended Key Usages (EKUs) or both the ServerAuth and ClientAuth EKUs.
- If your CAs aren't trusted by the host, as in the previous example, add the internode server CA to each of the following settings:
  - `internode.server.clientCaFiles`
  - `internode.client.rootCaFiles`
  - `frontend.server.clientCaFiles`

### authorization

The `authorization` section configures authentication for the Temporal Service.
It selects the plugins that validate inbound gRPC tokens, the signing keys to trust, and the credentials the Temporal Service attaches to outbound cross-cluster replication RPCs.

For how the `ClaimMapper`, `Authorizer`, and `TokenProvider` plugins work together, see [How to secure a Temporal Service](/self-hosted-guide/security).

The section has two subsections, one for each direction of traffic:

- [`jwtKeyProvider`](#jwtkeyprovider): Signing keys for verifying inbound JWTs.
- [`remoteClusterAuth`](#remoteclusterauth): Bearer tokens for outbound cross-cluster RPCs.

The following top-level settings select and configure the inbound plugins:

- `authorizer`: String. The inbound authorizer.
  - Default: `""`.
  An empty string disables authorization, and every request is allowed.
  `default` enables the built-in role-based authorizer.
  The value is case-insensitive.
- `claimMapper`: String. The `ClaimMapper` that extracts roles from a verified token.
  - Default: `""`.
  An empty string disables claim mapping.
  `default` enables the built-in JWT `ClaimMapper`.
- `audience`: String. The `aud` claim value that inbound JWTs must contain.
  - Default: `""`.
  When empty, the Temporal Service skips audience validation.
- `authHeaderName`: String. The gRPC metadata header the `ClaimMapper` reads the bearer token from.
  - Default: `authorization`.

The following example enables JWT-based inbound authorization with the built-in plugins:

```yaml
global:
  authorization:
    jwtKeyProvider:
      keySourceURIs:
        - https://idp.example.com/.well-known/jwks.json
      refreshInterval: 1m
    authorizer: default
    claimMapper: default
    audience: temporal-frontend
```

#### jwtKeyProvider

The `jwtKeyProvider` section configures where the built-in `ClaimMapper` gets the signing keys for verifying inbound JWTs.

- `keySourceURIs`: List of strings. URLs that return JWKS-formatted public keys.
  The built-in `ClaimMapper` fetches and caches the keys from every URL.
- `refreshInterval`: Go duration string, such as `1m` or `5m`. How often to fetch the keys again.
  - Default: `0`.
  With `0`, the Temporal Service loads keys once at startup and doesn't rotate them.

#### remoteClusterAuth

The `remoteClusterAuth` section controls the bearer tokens a [`TokenProvider`](/self-hosted-guide/security#token-provider) attaches to outbound cross-cluster RPCs.
It has no effect unless you also configure a `TokenProvider` with [`temporal.WithTokenProvider`](/references/server-options#withtokenprovider).

- `require`: Boolean.
  - Default: `false`.
  When `true`, every outbound cross-cluster RPC must carry a non-empty token.
  If the token is empty, the RPC fails with `Unauthenticated` instead of sending an empty `authorization` header.
  The Temporal Service doesn't start when `require` is `true` and no `TokenProvider` is configured.

The following example configures inbound JWT validation and outbound replication authentication on a sending cluster:

```yaml
global:
  authorization:
    jwtKeyProvider:
      keySourceURIs:
        - https://idp.example.com/.well-known/jwks.json
      refreshInterval: 1m
    authorizer: default
    claimMapper: default
    audience: temporal-frontend
    remoteClusterAuth:
      require: true
```

## persistence

The `persistence` section configures the data stores for the Temporal Service.
The following example configures a password-protected Cassandra data store, with Elasticsearch as a secondary Visibility store:

```yaml
persistence:
  defaultStore: default
  visibilityStore: cass-visibility # The primary Visibility store.
  secondaryVisibilityStore: es-visibility # A secondary Visibility store added to enable Dual Visibility.
  numHistoryShards: 512
  datastores:
    default:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal'
        user: 'username'
        password: 'password'
    cass-visibility:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal_visibility'
    es-visibility:
      elasticsearch:
        version: 'v7'
        logLevel: 'error'
        url:
          scheme: 'http'
          host: '127.0.0.1:9200'
        indices:
          visibility: temporal_visibility_v1_dev
        closeIdleConnectionsInterval: 15s
```

### numHistoryShards

**Required.** The number of History shards to create when the Temporal Service first starts.

> **⚠️ Warning:**
>
> You can't change this value after the first run.
> The Temporal Service ignores any later changes.
> Set it high enough for your expected peak load.
>

### defaultStore

**Required.** The name of the data store in [`datastores`](#datastores) that the Temporal Server uses for Workflow state.

### visibilityStore

**Required.** The name of the data store in [`datastores`](#datastores) to use as the primary [Visibility](/temporal-service/visibility) store.

### secondaryVisibilityStore

The name of the data store in [`datastores`](#datastores) to use as the secondary Visibility store for [Dual Visibility](/dual-visibility).

### datastores

**Required.** Named data store definitions.
Each key is a name, such as `default` or `cass-visibility` in the previous example, that other `persistence` settings reference.

Each definition must be one of `cassandra`, `sql`, `elasticsearch`, or `customDatastore`.

#### cassandra

- `hosts`: **Required.** Comma-separated Cassandra endpoints, such as `192.168.1.2,192.168.1.3,192.168.1.4`.
- `keyspace`: **Required.** The Cassandra keyspace.
- `port`: The port the `gocql` client connects to.
  - Default: `9042`.
- `user`: The username the `gocql` client authenticates with.
- `password`: The password the `gocql` client authenticates with.
- `datacenter`: The data center filter for Cassandra.
- `maxConns`: The maximum number of connections to this data store for a single TLS configuration.
- `tls`: TLS settings. See [tls](#tls-1).

#### sql

- `pluginName`: **Required.** The SQL database type. Supported values are `mysql` and `postgres`.
- `databaseName`: **Required.** The name of the SQL database.
- `connectAddr`: **Required.** The remote address of the database, such as `192.168.1.2`.
- `connectProtocol`: **Required.** The protocol for `connectAddr`. Supported values are `tcp` and `unix`.
- `user`: The username for authentication.
- `password`: The password for authentication.
- `connectAttributes`: A map of key-value attributes added to the connection `data_source_name` URL.
  SQLite supports the following attributes:
  - `cache`: The SQLite shared-cache mode.
    With `private`, each database connection uses its own cache instead of a shared one.
    Use `private` with the Temporal Server.
  - `setup`: When set, the Temporal Server creates the SQLite database schema at startup.
- `maxConns`: The maximum number of connections to this data store.
- `maxIdleConns`: The maximum number of idle connections to this data store.
- `maxConnLifetime`: The maximum time a connection stays open.
- `tls`: TLS settings. See [tls](#tls-1).

#### tls

The `tls` and `mtls` sections of a data store support the following settings:

- `enabled`: Boolean. Enables TLS.
- `serverName`: The name of the server that hosts the data store.
- `certFile`: Path to the certificate file.
- `keyFile`: Path to the key file.
- `caFile`: Path to the CA file.
- `enableHostVerification`: Boolean. When `true`, the client verifies the hostname and server certificate.
  This is the inverse of `InsecureSkipVerify` in the Go [`crypto/tls`](https://pkg.go.dev/crypto/tls#Config) package.

`certFile` and `keyFile` are optional, depending on your data store configuration.
To connect without a client certificate, leave out both.

## log

The `log` section configures logging for the Temporal Service.

- `stdout`: Boolean. When `true`, logs go to standard output.
- `level`: The minimum log level. Supported values are `debug`, `info`, `warn`, `error`, and `fatal`.
  - Default: `info`.
- `outputFile`: Path to a log file.

## clusterMetadata

The `clusterMetadata` section describes the local cluster for [Multi-Cluster Replication](/temporal-service/multi-cluster-replication).

```yaml
clusterMetadata:
  enableGlobalNamespace: true
  failoverVersionIncrement: 10
  masterClusterName: 'active'
  currentClusterName: 'active'
  clusterInformation:
    active:
      enabled: true
      initialFailoverVersion: 0
      rpcAddress: '127.0.0.1:7233'
```

- `currentClusterName`: **Required.** The name of the local cluster.
  You can't change this value after the first run. The Temporal Service ignores any later changes.
- `enableGlobalNamespace`: Boolean. Enables Global Namespaces.
  - Default: `false`.
- `failoverVersionIncrement`: The amount each cluster's failover version increases by on failover.
- `masterClusterName`: The name of the primary cluster.
  Only the primary cluster can register or update Namespaces. Any cluster can fail over a Namespace.
- `clusterInformation`: A map of cluster names to cluster definitions.
  The local cluster's name must match `currentClusterName`.
  Each definition supports the following settings:
  - `enabled`: Boolean. Enables replication with this cluster.
  - `initialFailoverVersion`: The cluster's starting failover version.
  - `rpcAddress`: The `host:port` of the cluster. The host can be a DNS name.
    Add the `dns:///` prefix to round-robin between the IP addresses of a DNS name.

## services

The `services` section configures each service role.
The supported roles are `frontend`, `internal-frontend`, `matching`, `worker`, and `history`.

The following example configures the `frontend` role:

```yaml
services:
  frontend:
    rpc:
      grpcPort: 8233
      membershipPort: 8933
      bindOnIP: '0.0.0.0'
```

### rpc

**Required.** Network settings for the service role.

- `grpcPort`: The port the gRPC server listens on.
- `membershipPort`: The port for membership traffic with other hosts in the same Temporal Service.
  Each role needs a different port.
  If multiple Temporal Services share a network, such as in one Kubernetes cluster, give each Temporal Service different membership ports.
- `bindOnLocalHost`: Boolean. When `true`, the role listens on `127.0.0.1`.
- `bindOnIP`: The IP address the role listens on, or `0.0.0.0` for all interfaces.
  Only IPv4 addresses are supported.
  You can't set both `bindOnIP` and `bindOnLocalHost`.

Use the same port values for a role on every host.

## publicClient

The `publicClient` section configures how the History, Matching, and Worker roles connect to the Frontend Service.
Most configurations use one of the following options:

- To connect to `internal-frontend` through membership with internode TLS, leave out `publicClient` and add an `internal-frontend` role under [`services`](#services).
- To connect to `frontend` through membership with Frontend TLS, leave out `publicClient` and don't add an `internal-frontend` role.
- To connect to a specific endpoint without membership, set `hostPort`.

The Temporal Service doesn't start if you configure both an `internal-frontend` role and a non-empty `publicClient` section.

- `hostPort`: The IPv4 `host:port` or DNS name of the Frontend Service.
  Add the `dns:///` prefix to round-robin between the IP addresses of a DNS name.
  For supported formats, see [gRPC name resolution](https://github.com/grpc/grpc/blob/master/doc/naming.md).

```yaml
publicClient:
  hostPort: 'localhost:8933'
```

## archival

The `archival` section configures the [Archival](/temporal-service/archival) store for Event History and Visibility data.
It has a `history` subsection and a `visibility` subsection, which each support the following settings:

- `state`: Supported values are `enabled` and `disabled`.
  Set it to `enabled` to use Archival with any Namespace in the Temporal Service.
  - `enabled`: Also set `URI` and [`namespaceDefaults`](#namespacedefaults).
  - `disabled`: Also set `enableRead` to `false`, and set `state` to `disabled` in `namespaceDefaults` with no `provider` or `URI`.
- `enableRead`: Boolean. When `true`, clients can read archived data.
- `provider`: Where to store archived data.
  Supported providers are `filestore`, `gstorage`, `s3store`, and custom providers.
  The default configuration uses `filestore`.

The following example enables Archival:

```yaml
# Service-level Archival config enabled
archival:
  # Event History configuration
  history:
    # Archival is enabled for the History Service data.
    state: 'enabled'
    enableRead: true
    # Namespaces can use either the local filestore provider or the Google Cloud provider.
    provider:
      filestore:
        fileMode: '0666'
        dirMode: '0766'
      gstorage:
        credentialsPath: '/tmp/gcloud/keyfile.json'
  # Configuration for archiving Visibility data.
  visibility:
    # Archival is enabled for Visibility data.
    state: 'enabled'
    enableRead: true
    provider:
      filestore:
        fileMode: '0666'
        dirMode: '0766'
```

The following example disables Archival:

```yaml
# Service-level Archival config disabled
archival:
  history:
    state: 'disabled'
    enableRead: false
  visibility:
    state: 'disabled'
    enableRead: false

namespaceDefaults:
  archival:
    history:
      state: 'disabled'
    visibility:
      state: 'disabled'
```

For setup steps, see [Set up Archival](/self-hosted-guide/archival#set-up-archival).

## namespaceDefaults

The `namespaceDefaults` section sets the default Archival settings for new Namespaces, for both `history` and `visibility` data.

- `state`: The default Archival state. Supported values are `enabled` and `disabled`.
- `URI`: The default Archival URI.

```yaml
# Default values for a Namespace if none are provided at creation.
namespaceDefaults:
  # Archival defaults.
  archival:
    # Event History defaults.
    history:
      state: 'enabled'
      # New Namespaces will default to the local provider.
      URI: 'file:///tmp/temporal_archival/development'
    visibility:
      state: 'disabled'
      URI: 'file:///tmp/temporal_vis_archival/development'
```

For more information, see [Create an archiving Namespace](/self-hosted-guide/archival#create-an-archiving-namespace).

## dcRedirectionPolicy

The `dcRedirectionPolicy` section configures whether the Frontend Service forwards API calls to the active cluster for a Namespace in cross-cluster replication.

- `policy`: Supported values are `noop`, `selected-apis-forwarding`, and `all-apis-forwarding`.
  - Default: `noop`.
  - `noop`: No forwarding.
  - `selected-apis-forwarding`: Forwards the following APIs to the active cluster for the Namespace:
    - `StartWorkflowExecution`
    - `SignalWithStartWorkflowExecution`
    - `SignalWorkflowExecution`
    - `RequestCancelWorkflowExecution`
    - `TerminateWorkflowExecution`
    - `QueryWorkflow`
  - `all-apis-forwarding`: Forwards all APIs for the Namespace to the active cluster.

```yaml
dcRedirectionPolicy:
  policy: 'selected-apis-forwarding'
```

## dynamicConfigClient

The `dynamicConfigClient` section configures the file-based [dynamic configuration](/temporal-service/configuration#dynamic-configuration) client.
Set it to use dynamic configuration.

- `filepath`: Path to the dynamic configuration YAML file, relative to the root directory.
- `pollInterval`: How often the client checks the file for changes. Minimum: `5s`.

```yaml
dynamicConfigClient:
  filepath: 'config/dynamicconfig/development-cass.yaml'
  pollInterval: '10s'
```

---

# OSS Temporal Service metrics reference

Source: https://docs.temporal.io/references/service-metrics

> Reference for the metrics a self-hosted Temporal Service emits, covering request rates, latencies, errors, and Nexus Operation activity.

> **ℹ️ Info:**
>
> The information on this page is relevant to open source [Temporal Service deployments](/temporal-service).
>
> See the [Cloud metrics reference](/cloud/metrics/openmetrics/metrics-reference) for metrics emitted by [Temporal Cloud](/evaluate/cloud).
>
> See [SDK metrics](/references/sdk-metrics) for metrics emitted by the [SDKs](/encyclopedia/architecture/temporal-sdks).
>
> Temporal SDKs and the Temporal Service attach these metric dimensions as [tags](/glossary#tag). Once a metric is scraped by Prometheus or another OpenMetrics-compatible system, the same dimensions are called labels.
>

A Temporal Service emits metrics that operators use to monitor the Temporal Service's performance and to set up alerts.

All metrics emitted by the Temporal Service are listed in [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go).

For details on setting up metrics in your Temporal Service configuration, see the [Temporal Service configuration reference](/references/service-configuration#global).

The [dashboards repository](https://github.com/temporalio/dashboards) contains community-driven Grafana dashboard templates that you can use as a starting point for monitoring the Temporal Service and SDK metrics.
For any metrics that are missing in the dashboards, use [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) as a reference.

In addition to the metrics the Temporal Service emits, monitor infrastructure metrics such as CPU, memory, and network for every host that runs a Temporal Service service.

Each metric on this page lists its type:

- **Counter:** A value that only increases. Query it with `rate()` or `increase()`.
- **Gauge:** A sampled value that can go up or down.
- **Histogram:** A distribution of values. Latency histograms are defined as timers in `metric_defs.go`. Query them through the `_bucket` series with `histogram_quantile()`.

All example queries on this page use [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).

## Common metrics

Temporal emits metrics for each gRPC service request.
These metrics are emitted with `type`, `operation`, and `namespace` tags, which show request rates across services, Namespaces, and operations.

- Use the `operation` tag in your query to get request rates, error rates, or latencies per operation.
- Use the `service_name` tag with a service role value, such as `frontend`, `history`, or `matching`, to get details for a specific service.
  The role values are defined in [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go).

All common tags that you can add in your query are defined in [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go).

For example, to see service requests by operation on the Frontend Service, use the following query:

```promql
sum by (operation) (rate(service_requests{service_name="frontend"}[2m]))
```

Start with the following metrics.

### `service_requests`

Counts the gRPC requests the service receives, tagged by `operation`.

Type: Counter

Example: Request rate for the `AddWorkflowTask` operation

```promql
sum(rate(service_requests{operation="AddWorkflowTask"}[2m]))
```

### `service_latency`

Shows latencies for all Client request operations.
Start here to find which operation has high latency.

Type: Histogram

Example: P95 service latency by operation for the Frontend Service

```promql
histogram_quantile(0.95, sum(rate(service_latency_bucket{service_name="frontend"}[5m])) by (operation, le))
```

### `service_error_with_type`

Counts errors the service encounters, tagged by `error_type`.

Type: Counter

Example: Service errors by type for the Frontend Service

```promql
sum(rate(service_error_with_type{service_name="frontend"}[5m])) by (error_type)
```

### `client_errors`

Counts failed requests between Temporal Service roles.
An increase indicates connection issues between roles.

Type: Counter

Example: Errors on requests from the Frontend Service to the History Service

```promql
sum(rate(client_errors{service_name="frontend",service_role="history"}[5m]))
```

The following sections list metrics for each service.
For metrics not listed here, see [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go).

## Matching Service metrics

### `poll_success`

Counts Tasks that are successfully matched to a poller.

Type: Counter

Example:

```promql
sum(rate(poll_success{}[5m]))
```

### `poll_timeouts`

Counts polls that return because no Task was available within the poll timeout.

Type: Counter

Example:

```promql
sum(rate(poll_timeouts{}[5m]))
```

### `asyncmatch_latency`

Measures the time from creation to delivery for async matched Tasks.
The larger this latency, the longer Tasks wait in the queue for your Workers to pick them up.

Type: Histogram

Example:

```promql
histogram_quantile(0.95, sum(rate(asyncmatch_latency_bucket{service_name="matching"}[5m])) by (operation, le))
```

### `no_poller_tasks`

Counts Tasks added to a Task Queue that has no recent poller.
An increase usually means that the Worker or the starter program is using the wrong Task Queue name.

Type: Counter

## History Service metrics

A History Task is an internal Task that the Temporal Service creates as part of a transaction that updates Workflow state.
The History Service processes these Tasks.
Use the following metrics to monitor the health of History Task processing.

### `task_requests`

Counts every Task process request.

Type: Counter

Example:

```promql
sum(rate(task_requests{operation=~"TransferActive.*"}[1m]))
```

### `task_errors`

Counts every Task process error.

Type: Counter

Example:

```promql
sum(rate(task_errors{operation=~"TransferActive.*"}[1m]))
```

### `task_attempt`

Records the number of attempts for each Task Execution.
A Task is retried until it succeeds, and each retry increases the attempt count.

Type: Histogram

Example:

```promql
histogram_quantile(0.95, sum(rate(task_attempt_bucket{operation=~"TransferActive.*"}[1m])) by (operation, le))
```

### `task_latency_processing`

Shows the processing latency per attempt.

Type: Histogram

Example:

```promql
histogram_quantile(0.95, sum(rate(task_latency_processing_bucket{operation=~"TransferActive.*",service_name="history"}[1m])) by (operation, le))
```

### `task_latency`

Measures the in-memory latency across multiple attempts.

Type: Histogram

### `task_latency_queue`

Measures the end-to-end duration from when the Task should run (when it fired) to when the Task is done.

Type: Histogram

### `task_latency_load`

Measures the duration from Task generation to Task loading.
This is the schedule-to-start latency for the persistence queue.

Type: Histogram

### `task_latency_schedule`

Measures the duration from Task submission to the Task scheduler to processing.
This is the schedule-to-start latency for the in-memory queue.

Type: Histogram

### `queue_latency_schedule`

Measures the time to schedule 100 Tasks in one Task channel in the host-level Task scheduler.
If fewer than 100 Tasks are in the Task channel for 30 seconds, the latency is scaled to 100 Tasks when emitted.

Type: Histogram

### `service_latency_userlatency`

Shows the latency introduced by Workflow logic.
For example, one Workflow that schedules many Activities or Child Workflows at the same time can cause per-Workflow lock contention.
The time spent waiting for the per-Workflow lock counts as `userlatency`.

Type: Histogram

The `operation` tag on History Service metrics contains the Task type and whether the Task is Active or Standby.
Use it to get request rates, error rates, or latencies per operation, which helps identify issues caused by database problems.

## Persistence metrics

The Temporal Service emits metrics for every persistence database read and write.
Start with the following metrics.

### `persistence_requests`

Counts every persistence request.

Type: Counter

Example: Persistence requests by operation for the History Service

```promql
sum by (operation) (rate(persistence_requests{service_name="history"}[1m]))
```

Example: Persistence requests by operation for the Matching Service

```promql
sum by (operation) (rate(persistence_requests{service_name="matching"}[1m]))
```

### `persistence_errors`

Counts all persistence errors.
An increase indicates connection issues between the Temporal Service and the persistence store.

Type: Counter

Example: Persistence errors for the History Service

```promql
sum(rate(persistence_errors{service_name="history"}[1m]))
```

### `persistence_error_with_type`

Counts persistence store errors, tagged by `error_type`.

Type: Counter

Example: Persistence errors by type for the History Service

```promql
sum(rate(persistence_error_with_type{service_name="history"}[1m])) by (error_type)
```

### `persistence_latency`

Shows the latency of persistence operations.

Type: Histogram

Example: P95 persistence latency by operation for the History Service

```promql
histogram_quantile(0.95, sum(rate(persistence_latency_bucket{service_name="history"}[1m])) by (operation, le))
```

## Schedule metrics

The following metrics track the performance and outcomes of the Workflow Executions that [Schedules](/schedule) start.
Schedule metrics are tagged by `namespace`, not by individual Schedule.
A non-zero value tells you that at least one Schedule in the Namespace is affected, but not which one.
To find the affected Schedule, see [Troubleshoot missed Schedule Actions](/troubleshooting/schedule-missed-actions).

### `schedule_buffer_overruns`

Counts the times the buffer that holds Scheduled Workflow Executions exceeds its maximum capacity.
This usually happens when a Schedule with the `BufferAll` overlap policy has an average run length longer than its average interval.

Type: Counter

Example:

```promql
sum(rate(schedule_buffer_overruns{namespace="$namespace"}[5m]))
```

### `schedule_missed_catchup_window`

Counts the Scheduled Actions that the Temporal Service didn't run within the configured catchup window.
An outage that lasts longer than the catchup window can cause missed Actions.

Type: Counter

Example:

```promql
sum(rate(schedule_missed_catchup_window{namespace="$namespace"}[5m]))
```

### `schedule_rate_limited`

Counts the Workflow Executions a Schedule tried to start that were throttled by Namespace rate limits.
Frequent rate limiting can cause missed catchup windows.

Type: Counter

Example:

```promql
sum(rate(schedule_rate_limited{namespace="$namespace"}[5m]))
```

### `schedule_action_success`

Counts the Workflow Executions that a Schedule started successfully, on its schedule or from a manual trigger.

Type: Counter

Example:

```promql
sum(rate(schedule_action_success{namespace="$namespace"}[5m]))
```

## Workflow metrics

These metrics count Workflow Execution outcomes.

### `workflow_cancel`

Counts Workflow Executions canceled before completing.

Type: Counter

### `workflow_continued_as_new`

Counts Workflow Executions that Continued-As-New.

Type: Counter

### `workflow_failed`

Counts Workflow Executions that failed.

Type: Counter

### `workflow_success`

Counts Workflow Executions that completed successfully.

Type: Counter

### `workflow_terminate`

Counts Workflow Executions that were terminated.

Type: Counter

### `workflow_timeout`

Counts Workflow Executions that timed out before completing.

Type: Counter

## Nexus metrics

These metrics cover Nexus Operations.

### Nexus machinery in the History Service

See the [Nexus architecture document](https://github.com/temporalio/temporal/blob/main/docs/architecture/nexus.md#scheduler) for how these components fit together.

#### In-memory buffer

- `dynamic_worker_pool_scheduler_enqueued_tasks` (Counter): Increments when a Task is added to the buffer.
- `dynamic_worker_pool_scheduler_dequeued_tasks` (Counter): Increments when a Task is removed from the buffer.
- `dynamic_worker_pool_scheduler_rejected_tasks` (Counter): Increments when the buffer is full and a Task is rejected.
- `dynamic_worker_pool_scheduler_buffer_size` (Gauge): The sampled size of the buffer.

#### Concurrency limiter

- `dynamic_worker_pool_scheduler_active_workers` (Gauge): The sampled number of running goroutines.

#### Rate limiter

- `rate_limited_task_runnable_wait_time` (Histogram): The time a Task waits for the rate limiter.

#### Circuit breaker

- `circuit_breaker_executable_blocked` (Counter): Increments each time the circuit breaker blocks a Task execution.

#### Task executors

- `nexus_outbound_requests` (Counter): The number of outbound Nexus requests the History Service makes.
- `nexus_outbound_latency` (Histogram): The latency of outbound Nexus requests the History Service makes.
- `callback_outbound_requests` (Counter): The number of outbound callback requests the History Service makes.
- `callback_outbound_latency` (Histogram): The latency of outbound callback requests the History Service makes.

### Nexus machinery on the Frontend Service

#### `nexus_requests`

The number of Nexus requests received by the service.

Type: Counter

#### `nexus_latency`

Latency of Nexus requests.

Type: Histogram

#### `nexus_request_preprocess_errors`

The number of Nexus requests for which pre-processing failed.

Type: Counter

#### `nexus_completion_requests`

The number of Nexus completion (callback) requests received by the service.

Type: Counter

#### `nexus_completion_latency`

Latency of Nexus completion (callback) requests.

Type: Histogram

#### `nexus_completion_request_preprocess_errors`

The number of Nexus completion requests for which pre-processing failed.

Type: Counter

---

# Temporal Web UI configuration reference

Source: https://docs.temporal.io/references/web-ui-configuration

> **ℹ️ Info:**
>
> To set these keys with environment variables in the `temporalio/ui` Docker image, see the
> [Temporal Web UI environment variables reference](/references/web-ui-environment-variables).
>

The Temporal Web UI Server reads its configuration from YAML files in its configuration directory.
It loads `base.yaml` first, then the file for the current environment, such as `development.yaml`.
Values in the environment file override values in `base.yaml`.

The configuration structs are defined in
[config.go](https://github.com/temporalio/ui-server/blob/main/server/config/config.go) in the ui-server repository.
For a complete example, see
[development.yaml](https://github.com/temporalio/ui-server/blob/main/config/development.yaml).

Each key on this page lists its **environment variable** and its **default**.
The default is the value the Web UI Server uses when no configuration file sets the key.
The Docker image sets its own defaults through environment variables.

## Server settings

### `temporalGrpcAddress`

Address of the [Frontend Service](/temporal-service/temporal-server#frontend-service) that the Web UI Server connects
to.
The Web UI Server doesn't start without this value.

- Environment variable: [`TEMPORAL_ADDRESS`](/references/web-ui-environment-variables#temporal_address)
- Default: `127.0.0.1:7233`, set in `base.yaml`

### `host`

Network interface that the Web UI Server listens on.
When empty, the Web UI Server listens on every interface.

- Environment variable: none
- Default: empty

### `port`

Port that the Web UI Server listens on for the browser UI and the HTTP API.

- Environment variable: [`TEMPORAL_UI_PORT`](/references/web-ui-environment-variables#temporal_ui_port)
- Default: `8233`, set in `base.yaml`

### `publicPath`

Subpath to serve the Web UI from, such as `/custom-path`.
Leave it empty to serve the Web UI from the root path.

- Environment variable: [`TEMPORAL_UI_PUBLIC_PATH`](/references/web-ui-environment-variables#temporal_ui_public_path)
- Default: empty

### `enableUi`

Set to `true` to serve the browser UI.
When `false`, the Web UI Server serves only its APIs.

- Environment variable: [`TEMPORAL_UI_ENABLED`](/references/web-ui-environment-variables#temporal_ui_enabled)
- Default: `false`

### `uiAssetPath`

Directory to serve the Web UI's static files from, instead of the files built into the Web UI Server.

- Environment variable: none
- Default: empty

### `cloudUi`

Set to `true` to use the Temporal Cloud version of the Web UI.

- Environment variable: [`TEMPORAL_CLOUD_UI`](/references/web-ui-environment-variables#temporal_cloud_ui)
- Default: `false`

### `refreshInterval`

How often the Web UI Server reloads its configuration files, such as `1m`.
Set it to `0s` to turn off reloading.
Settings that the Web UI Server reads only at startup, such as `host`, `port`, and `publicPath`, still need a restart.

- Environment variable:
  [`TEMPORAL_CONFIG_REFRESH_INTERVAL`](/references/web-ui-environment-variables#temporal_config_refresh_interval)
- Default: `0s`

### `forwardHeaders`

List of HTTP headers that the Web UI Server forwards from HTTP API requests to the Temporal Service's gRPC API.

```yaml
forwardHeaders:
  - X-Forwarded-For
```

- Environment variable: [`TEMPORAL_FORWARD_HEADERS`](/references/web-ui-environment-variables#temporal_forward_headers)
- Default: empty

### `hideLogs`

Set to `true` to stop the Web UI Server from printing its logs to the console.

- Environment variable: [`TEMPORAL_HIDE_LOGS`](/references/web-ui-environment-variables#temporal_hide_logs)
- Default: `false`

### `distribution`

How the Web UI was installed: `cli`, `docker`, `helm`, or `server`.
When [`notifyOnNewVersion`](#notifyonnewversion) is `true`, the Web UI Server uses this value to choose which release
to check for updates.

- Environment variable: [`TEMPORAL_UI_DISTRIBUTION`](/references/web-ui-environment-variables#temporal_ui_distribution)
- Default: empty, which the Web UI Server treats as `server`

### `distributionVersion`

Version of the distribution that installed the Web UI, such as the Temporal CLI version.
Only the `cli` distribution uses this value.

- Environment variable:
  [`TEMPORAL_UI_DISTRIBUTION_VERSION`](/references/web-ui-environment-variables#temporal_ui_distribution_version)
- Default: empty

## Web UI behavior settings

### `defaultNamespace`

[Namespace](/namespaces) that the Web UI opens first.

- Environment variable:
  [`TEMPORAL_DEFAULT_NAMESPACE`](/references/web-ui-environment-variables#temporal_default_namespace)
- Default: empty

### `feedbackUrl`

URL that the Feedback button in the Web UI opens.
When empty, the button opens the Web UI's GitHub issues page.

- Environment variable: [`TEMPORAL_FEEDBACK_URL`](/references/web-ui-environment-variables#temporal_feedback_url)
- Default: empty

### `showTemporalSystemNamespace`

Set to `true` to show the Temporal System Namespace in the Web UI.
The System Namespace holds the Workflow Executions that the Temporal Service runs internally.

- Environment variable:
  [`TEMPORAL_SHOW_TEMPORAL_SYSTEM_NAMESPACE`](/references/web-ui-environment-variables#temporal_show_temporal_system_namespace)
- Default: `false`

### `disableNewsFetch`

Set to `true` to stop the Web UI from requesting the news feed.
The Web UI also hides the button that opens the news feed panel.

- Environment variable:
  [`TEMPORAL_DISABLE_NEWS_FETCH`](/references/web-ui-environment-variables#temporal_disable_news_fetch)
- Default: `false`

### `notifyOnNewVersion`

Set to `true` to show a notice in the Web UI when a newer release is available.
The Web UI Server checks the release that matches [`distribution`](#distribution).

- Environment variable:
  [`TEMPORAL_NOTIFY_ON_NEW_VERSION`](/references/web-ui-environment-variables#temporal_notify_on_new_version)
- Default: `false`

### `navCollapsedByDefault`

Set to `true` to collapse the left navigation and the saved views navigation when the Web UI loads.

- Environment variable:
  [`TEMPORAL_NAV_COLLAPSED_BY_DEFAULT`](/references/web-ui-environment-variables#temporal_nav_collapsed_by_default)
- Default: `false`

### `hideWorkflowQueryErrors`

Set to `true` to hide server errors from Workflow Queries in the Web UI.

- Environment variable:
  [`TEMPORAL_HIDE_WORKFLOW_QUERY_ERRORS`](/references/web-ui-environment-variables#temporal_hide_workflow_query_errors)
- Default: `false`

### `refreshWorkflowCountsDisabled`

Set to `true` to stop the Web UI from refreshing the Workflow status counts on the Workflows page.

- Environment variable:
  [`TEMPORAL_REFRESH_WORKFLOW_COUNTS_DISABLED`](/references/web-ui-environment-variables#temporal_refresh_workflow_counts_disabled)
- Default: `false`

## Workflow and Activity action settings

These keys disable actions in the Web UI that change Workflow Executions or Activities.
Each key hides or disables the matching control in the Web UI.

### `disableWriteActions`

Set to `true` to disable every action in the Web UI that changes a Workflow Execution or Activity, including batch
actions.
This key overrides the other keys in this section.

The Web UI Server also rejects write requests to its HTTP API.
Workflow Queries still work.

- Environment variable:
  [`TEMPORAL_DISABLE_WRITE_ACTIONS`](/references/web-ui-environment-variables#temporal_disable_write_actions)
- Default: `false`

### `workflowTerminateDisabled`

Set to `true` to prevent users from terminating Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_WORKFLOW_TERMINATE_DISABLED`](/references/web-ui-environment-variables#temporal_workflow_terminate_disabled)
- Default: `false`

### `workflowCancelDisabled`

Set to `true` to prevent users from canceling Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_WORKFLOW_CANCEL_DISABLED`](/references/web-ui-environment-variables#temporal_workflow_cancel_disabled)
- Default: `false`

### `workflowSignalDisabled`

Set to `true` to prevent users from sending Signals to Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_WORKFLOW_SIGNAL_DISABLED`](/references/web-ui-environment-variables#temporal_workflow_signal_disabled)
- Default: `false`

### `workflowUpdateDisabled`

Set to `true` to prevent users from sending Updates to Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_WORKFLOW_UPDATE_DISABLED`](/references/web-ui-environment-variables#temporal_workflow_update_disabled)
- Default: `false`

### `workflowResetDisabled`

Set to `true` to prevent users from resetting Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_WORKFLOW_RESET_DISABLED`](/references/web-ui-environment-variables#temporal_workflow_reset_disabled)
- Default: `false`

### `workflowPauseDisabled`

Set to `true` to prevent users from pausing Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_WORKFLOW_PAUSE_DISABLED`](/references/web-ui-environment-variables#temporal_workflow_pause_disabled)
- Default: `false`

### `batchActionsDisabled`

Set to `true` to prevent users from running batch actions on multiple Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_BATCH_ACTIONS_DISABLED`](/references/web-ui-environment-variables#temporal_batch_actions_disabled)
- Default: `false`

### `startWorkflowDisabled`

Set to `true` to prevent users from starting Workflow Executions from the Web UI.

- Environment variable:
  [`TEMPORAL_START_WORKFLOW_DISABLED`](/references/web-ui-environment-variables#temporal_start_workflow_disabled)
- Default: `false`

### `activityCommandsDisabled`

Set to `true` to hide the commands for pending Activities in the Web UI.
These commands pause, unpause, and reset an Activity, and update its options.

- Environment variable:
  [`TEMPORAL_ACTIVITY_COMMANDS_DISABLED`](/references/web-ui-environment-variables#temporal_activity_commands_disabled)
- Default: `false`

## cors

The `cors` section controls which origins can call the Web UI Server APIs and how the Web UI Server sets its
cross-site request forgery (CSRF) cookie.
CORS stands for Cross-Origin Resource Sharing.

```yaml
cors:
  allowOrigins:
    - http://localhost:3000
  unsafeAllowAllOrigins: false
  cookieInsecure: false
```

- `allowOrigins`: List of origins that can make cross-origin requests to the Web UI Server APIs.
  A value of `*` allows every origin.
  - Environment variable: [`TEMPORAL_CORS_ORIGINS`](/references/web-ui-environment-variables#temporal_cors_origins)
  - Default: empty, which allows no cross-origin requests
- `unsafeAllowAllOrigins`: Set to `true` to accept cross-origin requests from any origin and ignore `allowOrigins`.
  Use it only for local development.
  - Environment variable:
    [`TEMPORAL_CORS_UNSAFE_ALLOW_ALL_ORIGINS`](/references/web-ui-environment-variables#temporal_cors_unsafe_allow_all_origins)
  - Default: `false`
- `cookieInsecure`: Set to `true` to send the CSRF cookie over connections the browser considers insecure, such as
  plain HTTP.
  Use it only when something other than HTTPS secures the connection, such as a VPN.
  - Environment variable:
    [`TEMPORAL_CSRF_COOKIE_INSECURE`](/references/web-ui-environment-variables#temporal_csrf_cookie_insecure)
  - Default: `false`

## auth

The `auth` section configures sign-in to the Web UI through an identity provider (IdP).
It controls who can access the Web UI, not authorization against the Temporal Service.

```yaml
auth:
  enabled: true
  providers:
    - label: sso
      type: oidc
      providerUrl: https://accounts.google.com
      issuerUrl:
      clientId: xxxxx-xxxx.apps.googleusercontent.com
      clientSecret: xxxxxxxxxxxxxxxxxxxx
      callbackUrl: https://xxxx.com:8080/auth/sso/callback
      scopes:
        - openid
        - profile
        - email
```

- `enabled`: Set to `true` to require users to sign in to the Web UI.
  The other `auth` keys take effect only when this key is `true`.
  - Environment variable: [`TEMPORAL_AUTH_ENABLED`](/references/web-ui-environment-variables#temporal_auth_enabled)
  - Default: `false`
- `redirectToProvider`: Set to `true` to skip the Web UI sign-in page and send users who aren't signed in directly to
  the IdP.
  - Environment variable:
    [`TEMPORAL_AUTH_REDIRECT_TO_PROVIDER`](/references/web-ui-environment-variables#temporal_auth_redirect_to_provider)
  - Default: `false`
- `maxSessionDuration`: Longest a user session can last, such as `8h` or `168h`.
  After this duration, users must sign in again even if their tokens are still valid.
  When empty, sessions have no maximum duration.
  - Environment variable:
    [`TEMPORAL_MAX_SESSION_DURATION`](/references/web-ui-environment-variables#temporal_max_session_duration)
  - Default: empty
- `providers`: List of IdPs.
  The Web UI Server uses only the first provider in the list.
  - Default: empty

### providers

Each provider takes the following keys.
When `enabled` is `true`, the Web UI Server doesn't start unless `providerUrl`, `clientId`, and `callbackUrl` are set.

- `label`: Label for the IdP.
  - Environment variable: [`TEMPORAL_AUTH_LABEL`](/references/web-ui-environment-variables#temporal_auth_label)
  - Default: empty
- `type`: Authentication type.
  Only `oidc` is supported.
  - Environment variable: [`TEMPORAL_AUTH_TYPE`](/references/web-ui-environment-variables#temporal_auth_type)
  - Default: empty
- `providerUrl`: IdP URL that the Web UI Server uses for OpenID Connect (OIDC) discovery, such as
  `https://accounts.google.com`.
  - Environment variable:
    [`TEMPORAL_AUTH_PROVIDER_URL`](/references/web-ui-environment-variables#temporal_auth_provider_url)
  - Default: empty
- `issuerUrl`: URL of the token issuer.
  Set it only when the issuer differs from `providerUrl`.
  - Environment variable: [`TEMPORAL_AUTH_ISSUER_URL`](/references/web-ui-environment-variables#temporal_auth_issuer_url)
  - Default: empty
- `clientId`: OAuth client ID that the IdP issued for the Web UI.
  - Environment variable: [`TEMPORAL_AUTH_CLIENT_ID`](/references/web-ui-environment-variables#temporal_auth_client_id)
  - Default: empty
- `clientSecret`: OAuth client secret that the IdP issued for the Web UI.
  - Environment variable:
    [`TEMPORAL_AUTH_CLIENT_SECRET`](/references/web-ui-environment-variables#temporal_auth_client_secret)
  - Default: empty
- `callbackUrl`: URL that the IdP redirects users to after they sign in, such as
  `https://xxxx.com:8080/auth/sso/callback`.
  - Environment variable:
    [`TEMPORAL_AUTH_CALLBACK_URL`](/references/web-ui-environment-variables#temporal_auth_callback_url)
  - Default: empty
- `scopes`: List of OIDC scopes to request, such as `openid`, `profile`, and `email`.
  - Environment variable: [`TEMPORAL_AUTH_SCOPES`](/references/web-ui-environment-variables#temporal_auth_scopes)
  - Default: empty
- `options`: Map of query parameters that the Web UI Server adds to the redirect URL for the IdP.
  Use it for IdP-specific sign-in flows, such as the Auth0 `audience` and `organization` parameters.
  - Environment variable: none
  - Default: empty
- `useIdTokenAsBearer`: Set to `true` to send the ID token instead of the access token as the bearer token in the
  `Authorization` header.
  - Environment variable:
    [`TEMPORAL_AUTH_USE_ID_TOKEN_AS_BEARER`](/references/web-ui-environment-variables#temporal_auth_use_id_token_as_bearer)
  - Default: `false`
- `refreshTokenDuration`: Lifetime of the refresh tokens that the IdP issues, such as `24h`.
  Set it only when the IdP issues opaque refresh tokens, because the Web UI Server can't read their expiration.
  For JSON Web Token (JWT) refresh tokens, the Web UI Server uses the token's `exp` claim and ignores this value.
  When neither is available, the Web UI Server assumes a lifetime of 7 days.
  - Environment variable:
    [`TEMPORAL_AUTH_REFRESH_TOKEN_DURATION`](/references/web-ui-environment-variables#temporal_auth_refresh_token_duration)
  - Default: empty

## tls

The `tls` section configures Transport Layer Security (TLS) for the Web UI Server's connection to the Frontend Service.
It doesn't configure TLS for the Web UI itself.
To serve the Web UI over HTTPS, see [`uiServerTLS`](#uiservertls).

```yaml
tls:
  caFile: ../ca.cert
  certFile: ../cluster.pem
  keyFile: ../cluster.key
  caData:
  certData:
  keyData:
  enableHostVerification: true
  serverName: tls-server
```

- `caFile`: Path to the Certificate Authority (CA) certificate that verifies the Frontend Service's certificate.
  - Environment variable: [`TEMPORAL_TLS_CA`](/references/web-ui-environment-variables#temporal_tls_ca)
  - Default: empty
- `certFile`: Path to the client certificate that the Web UI Server presents to the Frontend Service for mutual TLS
  (mTLS).
  - Environment variable: [`TEMPORAL_TLS_CERT`](/references/web-ui-environment-variables#temporal_tls_cert)
  - Default: empty
- `keyFile`: Path to the private key for the certificate in `certFile`.
  - Environment variable: [`TEMPORAL_TLS_KEY`](/references/web-ui-environment-variables#temporal_tls_key)
  - Default: empty
- `caData`: PEM data for the CA certificate.
  Use it instead of `caFile`.
  - Environment variable: [`TEMPORAL_TLS_CA_DATA`](/references/web-ui-environment-variables#temporal_tls_ca_data)
  - Default: empty
- `certData`: PEM data for the client certificate.
  Use it instead of `certFile`.
  - Environment variable: [`TEMPORAL_TLS_CERT_DATA`](/references/web-ui-environment-variables#temporal_tls_cert_data)
  - Default: empty
- `keyData`: PEM data for the private key.
  Use it instead of `keyFile`.
  - Environment variable: [`TEMPORAL_TLS_KEY_DATA`](/references/web-ui-environment-variables#temporal_tls_key_data)
  - Default: empty
- `enableHostVerification`: Set to `true` to verify that the Frontend Service's certificate matches its hostname.
  - Environment variable:
    [`TEMPORAL_TLS_ENABLE_HOST_VERIFICATION`](/references/web-ui-environment-variables#temporal_tls_enable_host_verification)
  - Default: `false`
- `serverName`: Overrides the server name sent for Server Name Indication (SNI) and checked against the Frontend
  Service's certificate.
  - Environment variable: [`TEMPORAL_TLS_SERVER_NAME`](/references/web-ui-environment-variables#temporal_tls_server_name)
  - Default: empty

## uiServerTLS

The `uiServerTLS` section configures the Web UI Server to serve the Web UI over HTTPS.
The Web UI Server starts in TLS mode only when you set both keys.

```yaml
uiServerTLS:
  certFile: ../ui-server.pem
  keyFile: ../ui-server.key
```

- `certFile`: Path to the certificate that the Web UI Server presents to browsers.
  - Environment variable:
    [`TEMPORAL_UI_SERVER_TLS_CERT`](/references/web-ui-environment-variables#temporal_ui_server_tls_cert)
  - Default: empty
- `keyFile`: Path to the private key for the certificate in `certFile`.
  - Environment variable:
    [`TEMPORAL_UI_SERVER_TLS_KEY`](/references/web-ui-environment-variables#temporal_ui_server_tls_key)
  - Default: empty

## codec

The `codec` section configures how the Web UI sends payloads to a [Codec Server](/codec-server) for decoding.

```yaml
codec:
  endpoint: https://your-codec-server-endpoint
  passAccessToken: false
  includeCredentials: false
  defaultErrorMessage:
  defaultErrorLink:
```

- `endpoint`: URL of the Codec Server.
  - Environment variable: [`TEMPORAL_CODEC_ENDPOINT`](/references/web-ui-environment-variables#temporal_codec_endpoint)
  - Default: empty
- `passAccessToken`: Set to `true` to send the user's access token in the `Authorization` header of requests to the
  Codec Server.
  - Environment variable:
    [`TEMPORAL_CODEC_PASS_ACCESS_TOKEN`](/references/web-ui-environment-variables#temporal_codec_pass_access_token)
  - Default: `false`
- `includeCredentials`: Set to `true` to include browser credentials, such as cookies, in requests to the Codec Server.
  - Environment variable:
    [`TEMPORAL_CODEC_INCLUDE_CREDENTIALS`](/references/web-ui-environment-variables#temporal_codec_include_credentials)
  - Default: `false`
- `defaultErrorMessage`: Message that the Web UI shows in its error banner when it can't reach the Codec Server.
  When empty, the Web UI shows its built-in message.
  - Environment variable:
    [`TEMPORAL_CODEC_DEFAULT_ERROR_MESSAGE`](/references/web-ui-environment-variables#temporal_codec_default_error_message)
  - Default: empty
- `defaultErrorLink`: Link that the Web UI shows in its error banner when it can't reach the Codec Server.
  When empty, the Web UI links to
  [Set your Codec Server endpoints with Web UI and CLI](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli).
  - Environment variable:
    [`TEMPORAL_CODEC_DEFAULT_ERROR_LINK`](/references/web-ui-environment-variables#temporal_codec_default_error_link)
  - Default: empty

---

# Temporal Web UI environment variables reference

Source: https://docs.temporal.io/references/web-ui-environment-variables

> **ℹ️ Info:**
>
> For the settings in the Web UI Server configuration file, see the
> [Temporal Web UI configuration reference](/references/web-ui-configuration).
>

Environment variables configure the Temporal Web UI Server when you run it from the `temporalio/ui` Docker image.
Each variable sets a key in the configuration file.

The variables are defined in the
[docker.yaml configuration template](https://github.com/temporalio/ui-server/blob/main/config/docker.yaml) in the
ui-server repository.
Each variable on this page lists its **configuration key** and its **default**.

```bash
docker run \
  -e TEMPORAL_ADDRESS=127.0.0.1:7233 \
  -e TEMPORAL_UI_PORT=8080 \
  -e TEMPORAL_UI_PUBLIC_PATH=path/to/webui \
  -e TEMPORAL_UI_ENABLED=true \
  -e TEMPORAL_CLOUD_UI=false \
  -e TEMPORAL_DEFAULT_NAMESPACE=default \
  -e TEMPORAL_FEEDBACK_URL=https://feedback.here \
  -e TEMPORAL_CONFIG_REFRESH_INTERVAL=0s \
  -e TEMPORAL_SHOW_TEMPORAL_SYSTEM_NAMESPACE=false \
  -e TEMPORAL_DISABLE_WRITE_ACTIONS=false \
  -e TEMPORAL_AUTH_ENABLED=true \
  -e TEMPORAL_AUTH_TYPE=oidc \
  -e TEMPORAL_AUTH_PROVIDER_URL=https://accounts.google.com \
  -e TEMPORAL_AUTH_ISSUER_URL=https://accounts.google.com \
  -e TEMPORAL_AUTH_CLIENT_ID=xxxxx-xxxx.apps.googleusercontent.com \
  -e TEMPORAL_AUTH_CLIENT_SECRET=xxxxxxxxxxxxxxx \
  -e TEMPORAL_AUTH_CALLBACK_URL=https://xxxx.com:8080/auth/sso/callback \
  -e TEMPORAL_AUTH_SCOPES=openid,email,profile \
  -e TEMPORAL_TLS_CA=../ca.cert \
  -e TEMPORAL_TLS_CERT=../cluster.pem \
  -e TEMPORAL_TLS_KEY=../cluster.key \
  -e TEMPORAL_TLS_ENABLE_HOST_VERIFICATION=true \
  -e TEMPORAL_TLS_SERVER_NAME=tls-server \
  -e TEMPORAL_CODEC_ENDPOINT=https://codec.server \
  -e TEMPORAL_CODEC_PASS_ACCESS_TOKEN=false \
  -e TEMPORAL_CODEC_INCLUDE_CREDENTIALS=false \
  -e TEMPORAL_HIDE_LOGS=false \
  temporalio/ui:<tag>
```

## Server variables

### `TEMPORAL_ADDRESS`

Address of the [Frontend Service](/temporal-service/temporal-server#frontend-service) that the Web UI Server connects
to.

- Configuration key: [`temporalGrpcAddress`](/references/web-ui-configuration#temporalgrpcaddress)
- Default: `127.0.0.1:7233`

### `TEMPORAL_UI_PORT`

Port that the Web UI Server listens on for the browser UI and the HTTP API.

- Configuration key: [`port`](/references/web-ui-configuration#port)
- Default: `8080`

### `TEMPORAL_UI_PUBLIC_PATH`

Subpath to serve the Web UI from, such as `/custom-path`.
Leave it empty to serve the Web UI from the root path.

- Configuration key: [`publicPath`](/references/web-ui-configuration#publicpath)
- Default: empty

### `TEMPORAL_UI_ENABLED`

Set to `false` to stop serving the browser UI.
The Web UI Server still serves its APIs.

- Configuration key: [`enableUi`](/references/web-ui-configuration#enableui)
- Default: `true`

### `TEMPORAL_CLOUD_UI`

Set to `true` to use the Temporal Cloud version of the Web UI.

- Configuration key: [`cloudUi`](/references/web-ui-configuration#cloudui)
- Default: `false`

### `TEMPORAL_CONFIG_REFRESH_INTERVAL`

How often the Web UI Server reloads its configuration, such as `1m`.
Set it to `0s` to turn off reloading.
Settings that the Web UI Server reads only at startup, such as `TEMPORAL_UI_PORT` and `TEMPORAL_UI_PUBLIC_PATH`, still need a restart.

- Configuration key: [`refreshInterval`](/references/web-ui-configuration#refreshinterval)
- Default: `0s`

### `TEMPORAL_FORWARD_HEADERS`

Comma-separated list of HTTP headers that the Web UI Server forwards from HTTP API requests to the Temporal Service's
gRPC API.

- Configuration key: [`forwardHeaders`](/references/web-ui-configuration#forwardheaders)
- Default: empty

### `TEMPORAL_HIDE_LOGS`

Set to `true` to stop the Web UI Server from printing its logs to the console.

- Configuration key: [`hideLogs`](/references/web-ui-configuration#hidelogs)
- Default: `false`

### `TEMPORAL_UI_SERVER_TLS_CERT`

Path to the certificate that the Web UI Server uses to serve the Web UI over HTTPS.
The Web UI Server starts in TLS mode only when you set both this variable and `TEMPORAL_UI_SERVER_TLS_KEY`.

- Configuration key: [`uiServerTLS.certFile`](/references/web-ui-configuration#uiservertls)
- Default: empty

### `TEMPORAL_UI_SERVER_TLS_KEY`

Path to the private key for the certificate in `TEMPORAL_UI_SERVER_TLS_CERT`.

- Configuration key: [`uiServerTLS.keyFile`](/references/web-ui-configuration#uiservertls)
- Default: empty

### `TEMPORAL_UI_DISTRIBUTION`

How the Web UI was installed: `cli`, `docker`, `helm`, or `server`.
When [`TEMPORAL_NOTIFY_ON_NEW_VERSION`](#temporal_notify_on_new_version) is `true`, the Web UI Server uses this value to choose which release to check for updates.

- Configuration key: [`distribution`](/references/web-ui-configuration#distribution)
- Default: `docker`

### `TEMPORAL_UI_DISTRIBUTION_VERSION`

Version of the distribution that installed the Web UI, such as the Temporal CLI version.
Only the `cli` distribution uses this value.

- Configuration key: [`distributionVersion`](/references/web-ui-configuration#distributionversion)
- Default: empty

## CORS and CSRF variables

These variables control which origins can call the Web UI Server APIs and how the Web UI Server sets its cross-site request forgery (CSRF) cookie.
For the matching configuration keys, see [`cors`](/references/web-ui-configuration#cors).

### `TEMPORAL_CORS_ORIGINS`

Comma-separated list of origins that can make cross-origin requests to the Web UI Server APIs.
A value of `*` allows every origin.

- Configuration key: [`cors.allowOrigins`](/references/web-ui-configuration#cors)
- Default: `http://localhost:8080`

### `TEMPORAL_CORS_UNSAFE_ALLOW_ALL_ORIGINS`

Set to `true` to accept cross-origin requests from any origin and ignore `TEMPORAL_CORS_ORIGINS`.
Use it only for local development.

- Configuration key: [`cors.unsafeAllowAllOrigins`](/references/web-ui-configuration#cors)
- Default: `false`

### `TEMPORAL_CSRF_COOKIE_INSECURE`

Set to `true` to send the CSRF cookie over connections the browser considers insecure, such as plain HTTP.
Use it only when something other than HTTPS secures the connection, such as a VPN.

- Configuration key: [`cors.cookieInsecure`](/references/web-ui-configuration#cors)
- Default: `false`

## Web UI behavior variables

### `TEMPORAL_DEFAULT_NAMESPACE`

[Namespace](/namespaces) that the Web UI opens first.

- Configuration key: [`defaultNamespace`](/references/web-ui-configuration#defaultnamespace)
- Default: `default`

### `TEMPORAL_FEEDBACK_URL`

URL that the Feedback button in the Web UI opens.
When empty, the button opens the Web UI's GitHub issues page.

- Configuration key: [`feedbackUrl`](/references/web-ui-configuration#feedbackurl)
- Default: empty

### `TEMPORAL_SHOW_TEMPORAL_SYSTEM_NAMESPACE`

Set to `true` to show the Temporal System Namespace in the Web UI.
The System Namespace holds the Workflow Executions that the Temporal Service runs internally.

- Configuration key: [`showTemporalSystemNamespace`](/references/web-ui-configuration#showtemporalsystemnamespace)
- Default: `false`

### `TEMPORAL_DISABLE_NEWS_FETCH`

Set to `true` to stop the Web UI from requesting the news feed.
The Web UI also hides the button that opens the news feed panel.

- Configuration key: [`disableNewsFetch`](/references/web-ui-configuration#disablenewsfetch)
- Default: `false`

### `TEMPORAL_NOTIFY_ON_NEW_VERSION`

Set to `true` to show a notice in the Web UI when a newer release is available.
The Web UI Server checks the release that matches [`TEMPORAL_UI_DISTRIBUTION`](#temporal_ui_distribution).

- Configuration key: [`notifyOnNewVersion`](/references/web-ui-configuration#notifyonnewversion)
- Default: `false`

### `TEMPORAL_NAV_COLLAPSED_BY_DEFAULT`

Set to `true` to collapse the left navigation and the saved views navigation when the Web UI loads.

- Configuration key: [`navCollapsedByDefault`](/references/web-ui-configuration#navcollapsedbydefault)
- Default: `false`

### `TEMPORAL_HIDE_WORKFLOW_QUERY_ERRORS`

Set to `true` to hide server errors from Workflow Queries in the Web UI.

- Configuration key: [`hideWorkflowQueryErrors`](/references/web-ui-configuration#hideworkflowqueryerrors)
- Default: `false`

### `TEMPORAL_REFRESH_WORKFLOW_COUNTS_DISABLED`

Set to `true` to stop the Web UI from refreshing the Workflow status counts on the Workflows page.

- Configuration key: [`refreshWorkflowCountsDisabled`](/references/web-ui-configuration#refreshworkflowcountsdisabled)
- Default: `false`

## Workflow and Activity action variables

These variables disable actions in the Web UI that change Workflow Executions or Activities.
Each variable hides or disables the matching control in the Web UI.

### `TEMPORAL_DISABLE_WRITE_ACTIONS`

Set to `true` to disable every action in the Web UI that changes a Workflow Execution or Activity, including batch actions.
This variable overrides the other variables in this section.

- Configuration key: [`disableWriteActions`](/references/web-ui-configuration#disablewriteactions)
- Default: `false`

### `TEMPORAL_WORKFLOW_TERMINATE_DISABLED`

Set to `true` to prevent users from terminating Workflow Executions from the Web UI.

- Configuration key: [`workflowTerminateDisabled`](/references/web-ui-configuration#workflowterminatedisabled)
- Default: `false`

### `TEMPORAL_WORKFLOW_CANCEL_DISABLED`

Set to `true` to prevent users from canceling Workflow Executions from the Web UI.

- Configuration key: [`workflowCancelDisabled`](/references/web-ui-configuration#workflowcanceldisabled)
- Default: `false`

### `TEMPORAL_WORKFLOW_SIGNAL_DISABLED`

Set to `true` to prevent users from sending Signals to Workflow Executions from the Web UI.

- Configuration key: [`workflowSignalDisabled`](/references/web-ui-configuration#workflowsignaldisabled)
- Default: `false`

### `TEMPORAL_WORKFLOW_UPDATE_DISABLED`

Set to `true` to prevent users from sending Updates to Workflow Executions from the Web UI.

- Configuration key: [`workflowUpdateDisabled`](/references/web-ui-configuration#workflowupdatedisabled)
- Default: `false`

### `TEMPORAL_WORKFLOW_RESET_DISABLED`

Set to `true` to prevent users from resetting Workflow Executions from the Web UI.

- Configuration key: [`workflowResetDisabled`](/references/web-ui-configuration#workflowresetdisabled)
- Default: `false`

### `TEMPORAL_WORKFLOW_PAUSE_DISABLED`

Set to `true` to prevent users from pausing Workflow Executions from the Web UI.

- Configuration key: [`workflowPauseDisabled`](/references/web-ui-configuration#workflowpausedisabled)
- Default: `false`

### `TEMPORAL_BATCH_ACTIONS_DISABLED`

Set to `true` to prevent users from running batch actions on multiple Workflow Executions from the Web UI.

- Configuration key: [`batchActionsDisabled`](/references/web-ui-configuration#batchactionsdisabled)
- Default: `false`

### `TEMPORAL_START_WORKFLOW_DISABLED`

Set to `true` to prevent users from starting Workflow Executions from the Web UI.

- Configuration key: [`startWorkflowDisabled`](/references/web-ui-configuration#startworkflowdisabled)
- Default: `false`

### `TEMPORAL_ACTIVITY_COMMANDS_DISABLED`

Set to `true` to hide the commands for pending Activities in the Web UI.
These commands pause, unpause, and reset an Activity, and update its options.

- Configuration key: [`activityCommandsDisabled`](/references/web-ui-configuration#activitycommandsdisabled)
- Default: `false`

## Authentication variables

These variables configure sign-in to the Web UI through an identity provider (IdP).
They control who can access the Web UI, not authorization against the Temporal Service.
For the matching configuration keys, see [`auth`](/references/web-ui-configuration#auth).

### `TEMPORAL_AUTH_ENABLED`

Set to `true` to require users to sign in to the Web UI.
The other authentication variables take effect only when this variable is `true`.

- Configuration key: [`auth.enabled`](/references/web-ui-configuration#auth)
- Default: `false`

### `TEMPORAL_AUTH_REDIRECT_TO_PROVIDER`

Set to `true` to skip the Web UI sign-in page and send users who aren't signed in directly to the IdP.

- Configuration key: [`auth.redirectToProvider`](/references/web-ui-configuration#auth)
- Default: `false`

### `TEMPORAL_MAX_SESSION_DURATION`

Longest a user session can last, such as `8h` or `168h`.
After this duration, users must sign in again even if their tokens are still valid.
When empty, sessions have no maximum duration.

- Configuration key: [`auth.maxSessionDuration`](/references/web-ui-configuration#auth)
- Default: empty

### `TEMPORAL_AUTH_LABEL`

Label for the identity provider.

- Configuration key: [`auth.providers.label`](/references/web-ui-configuration#providers)
- Default: `sso`

### `TEMPORAL_AUTH_TYPE`

Authentication type.

- Configuration key: [`auth.providers.type`](/references/web-ui-configuration#providers)
- Default: `oidc`

### `TEMPORAL_AUTH_PROVIDER_URL`

IdP URL that the Web UI Server uses for OpenID Connect (OIDC) discovery, such as `https://accounts.google.com`.

- Configuration key: [`auth.providers.providerUrl`](/references/web-ui-configuration#providers)
- Default: none

### `TEMPORAL_AUTH_ISSUER_URL`

URL of the token issuer.
Set it only when the issuer differs from `TEMPORAL_AUTH_PROVIDER_URL`.

- Configuration key: [`auth.providers.issuerUrl`](/references/web-ui-configuration#providers)
- Default: empty

### `TEMPORAL_AUTH_CLIENT_ID`

OAuth client ID that the IdP issued for the Web UI.
Required when authentication is enabled.

- Configuration key: [`auth.providers.clientId`](/references/web-ui-configuration#providers)
- Default: none

### `TEMPORAL_AUTH_CLIENT_SECRET`

OAuth client secret that the IdP issued for the Web UI.

- Configuration key: [`auth.providers.clientSecret`](/references/web-ui-configuration#providers)
- Default: none

### `TEMPORAL_AUTH_CALLBACK_URL`

URL that the IdP redirects users to after they sign in, such as `https://xxxx.com:8080/auth/sso/callback`.

- Configuration key: [`auth.providers.callbackUrl`](/references/web-ui-configuration#providers)
- Default: none

### `TEMPORAL_AUTH_SCOPES`

Comma-separated list of OIDC scopes to request, such as `openid,email,profile`.

- Configuration key: [`auth.providers.scopes`](/references/web-ui-configuration#providers)
- Default: empty

### `TEMPORAL_AUTH_USE_ID_TOKEN_AS_BEARER`

Set to `true` to send the ID token instead of the access token as the bearer token in the `Authorization` header.

- Configuration key: [`auth.providers.useIdTokenAsBearer`](/references/web-ui-configuration#providers)
- Default: `false`

### `TEMPORAL_AUTH_REFRESH_TOKEN_DURATION`

Lifetime of the refresh tokens that the IdP issues, such as `24h`.
Set it only when the IdP issues opaque refresh tokens, because the Web UI Server can't read their expiration.
For JSON Web Token (JWT) refresh tokens, the Web UI Server uses the token's `exp` claim and ignores this value.
When neither is available, the Web UI Server assumes a lifetime of 7 days.

- Configuration key: [`auth.providers.refreshTokenDuration`](/references/web-ui-configuration#providers)
- Default: empty

## TLS variables

These variables configure Transport Layer Security (TLS) for the Web UI Server's connection to the Frontend Service.
They don't configure TLS for the Web UI itself.
For the matching configuration keys, see [`tls`](/references/web-ui-configuration#tls).

### `TEMPORAL_TLS_CA`

Path to the Certificate Authority (CA) certificate that verifies the Frontend Service's certificate.

- Configuration key: [`tls.caFile`](/references/web-ui-configuration#tls)
- Default: empty

### `TEMPORAL_TLS_CERT`

Path to the client certificate that the Web UI Server presents to the Frontend Service for mutual TLS (mTLS).

- Configuration key: [`tls.certFile`](/references/web-ui-configuration#tls)
- Default: empty

### `TEMPORAL_TLS_KEY`

Path to the private key for the certificate in `TEMPORAL_TLS_CERT`.

- Configuration key: [`tls.keyFile`](/references/web-ui-configuration#tls)
- Default: empty

### `TEMPORAL_TLS_CA_DATA`

PEM data for the CA certificate.
Use it instead of `TEMPORAL_TLS_CA`.

- Configuration key: [`tls.caData`](/references/web-ui-configuration#tls)
- Default: empty

### `TEMPORAL_TLS_CERT_DATA`

PEM data for the client certificate.
Use it instead of `TEMPORAL_TLS_CERT`.

- Configuration key: [`tls.certData`](/references/web-ui-configuration#tls)
- Default: empty

### `TEMPORAL_TLS_KEY_DATA`

PEM data for the private key.
Use it instead of `TEMPORAL_TLS_KEY`.

- Configuration key: [`tls.keyData`](/references/web-ui-configuration#tls)
- Default: empty

### `TEMPORAL_TLS_ENABLE_HOST_VERIFICATION`

Set to `true` to verify that the Frontend Service's certificate matches its hostname.

- Configuration key: [`tls.enableHostVerification`](/references/web-ui-configuration#tls)
- Default: `false`

### `TEMPORAL_TLS_SERVER_NAME`

Overrides the server name sent for Server Name Indication (SNI) and checked against the Frontend Service's certificate.

- Configuration key: [`tls.serverName`](/references/web-ui-configuration#tls)
- Default: empty

## Codec Server variables

These variables configure how the Web UI sends payloads to a [Codec Server](/codec-server) for decoding.
For the matching configuration keys, see [`codec`](/references/web-ui-configuration#codec).

### `TEMPORAL_CODEC_ENDPOINT`

URL of the Codec Server.

- Configuration key: [`codec.endpoint`](/references/web-ui-configuration#codec)
- Default: empty

### `TEMPORAL_CODEC_PASS_ACCESS_TOKEN`

Set to `true` to send the user's access token in the `Authorization` header of requests to the Codec Server.

- Configuration key: [`codec.passAccessToken`](/references/web-ui-configuration#codec)
- Default: `false`

### `TEMPORAL_CODEC_INCLUDE_CREDENTIALS`

Set to `true` to include browser credentials, such as cookies, in requests to the Codec Server.

- Configuration key: [`codec.includeCredentials`](/references/web-ui-configuration#codec)
- Default: `false`

### `TEMPORAL_CODEC_DEFAULT_ERROR_MESSAGE`

Message that the Web UI shows in its error banner when it can't reach the Codec Server.
When empty, the Web UI shows its built-in message.

- Configuration key: [`codec.defaultErrorMessage`](/references/web-ui-configuration#codec)
- Default: empty

### `TEMPORAL_CODEC_DEFAULT_ERROR_LINK`

Link that the Web UI shows in its error banner when it can't reach the Codec Server.
When empty, the Web UI links to [Set your Codec Server endpoints with Web UI and CLI](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli).

- Configuration key: [`codec.defaultErrorLink`](/references/web-ui-configuration#codec)
- Default: empty

---

# Workflow Task errors reference

Source: https://docs.temporal.io/references/workflow-task-errors

> Workflow Task failure causes and Resource Exhausted causes in Temporal, with what triggers each one and how to resolve it in your Workflow or Worker.

> **ℹ️ Info:**
>
> For the Failure types that Workflows, Activities, and Nexus Operations return, see the [Temporal Failures reference](/references/failures).
> For symptom-based guidance, see [Troubleshooting Workflow Execution failures](/troubleshooting/execution-failures).
>

When a [Workflow Task](/tasks#workflow-task) fails, the Temporal Service records a [WorkflowTaskFailed](/references/events#workflowtaskfailed) Event with a cause in `workflow_task_failed_event_attributes`.
The Temporal Service retries the failed Workflow Task, and the Workflow Execution stays open unless the entry says otherwise.
Many of these failures come from a [Command](/references/commands) the Worker sent with unset or invalid attributes.

Workflow Task failure causes are defined in the [`WorkflowTaskFailedCause` enum](https://github.com/temporalio/api/blob/master/temporal/api/enums/v1/failed_cause.proto) of the Temporal gRPC API.
The [Resource Exhausted causes](#resource-exhausted-causes) on this page come from the `ResourceExhaustedCause` enum in the same file.
They're carried on a gRPC `RESOURCE_EXHAUSTED` response, not on a Workflow Task failure Event.

Each error on this page lists its enum value as **Cause**.

## Command attribute errors

The Temporal Service validates the attributes of each Command.
If a required attribute is unset or a value exceeds a size limit, the Workflow Task fails.
The SDK sets most Command attributes, so an unset attribute can indicate an outdated SDK.

### Bad Cancel Timer Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_CANCEL_TIMER_ATTRIBUTES`

The [CancelTimer](/references/commands#canceltimer) Command is missing a Timer ID.

Set a valid Timer ID and deploy the change to your Workers.

### Bad Cancel Workflow Execution Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_CANCEL_WORKFLOW_EXECUTION_ATTRIBUTES`

The [CancelWorkflowExecution](/references/commands#cancelworkflowexecution) Command has unset attributes.

Set the missing attributes and deploy the change to your Workers.

### Bad Complete Workflow Execution Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_COMPLETE_WORKFLOW_EXECUTION_ATTRIBUTES`

The [CompleteWorkflowExecution](/references/commands#completeworkflowexecution) Command has unset attributes, or its result exceeds the Payload size limit.

Set the missing attributes.
If the Workflow result is too large, store the data in external storage and return a reference to it.
See [Payload size limit](/troubleshooting/blob-size-limit-error).

### Bad Continue as New Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_CONTINUE_AS_NEW_ATTRIBUTES`

The [ContinueAsNewWorkflowExecution](/references/commands#continueasnewworkflowexecution) Command has an unset or invalid attribute.

Set the missing attributes.
If the input or Memo exceeds the size limit, reduce its size.
Check that every [Search Attribute](/search-attribute) the new run sets is registered on the Namespace.

### Bad Fail Workflow Execution Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_FAIL_WORKFLOW_EXECUTION_ATTRIBUTES`

The [FailWorkflowExecution](/references/commands#failworkflowexecution) Command has unset attributes, such as a missing Failure, or its Failure exceeds the size limit.

Reduce the size of the error details the Workflow returns.
If the attributes are unset, upgrade to the latest release of your [Temporal SDK](/encyclopedia/architecture/temporal-sdks).

### Bad Modify Workflow Properties Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_MODIFY_WORKFLOW_PROPERTIES_ATTRIBUTES`

The [ModifyWorkflowProperties](/references/commands#modifyworkflowproperties) Command has unset attributes, or the [Memo](/workflow-execution#memo) update exceeds the size limit.

Set the missing attributes and reduce the size of the Memo.

### Bad Record Marker Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_RECORD_MARKER_ATTRIBUTES`

The [RecordMarker](/references/commands#recordmarker) Command has an unset or invalid [Marker](/references/events#markerrecorded) name.

The SDK issues RecordMarker Commands, not your Workflow code.
Upgrade to the latest release of your Temporal SDK.

### Bad Request Cancel Activity Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_REQUEST_CANCEL_ACTIVITY_ATTRIBUTES`

The [RequestCancelActivityTask](/references/commands#requestcancelactivitytask) Command has unset attributes, or the SDK's History Builder is in an invalid state.

Upgrade to the latest release of your Temporal SDK.
If the error continues, check your Workflow code for [non-deterministic changes](/workflow-definition#non-deterministic-change).

### Bad Request Cancel External Workflow Execution Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_REQUEST_CANCEL_EXTERNAL_WORKFLOW_EXECUTION_ATTRIBUTES`

The [RequestCancelExternalWorkflowExecution](/references/commands#requestcancelexternalworkflowexecution) Command has unset or invalid attributes.

Set any missing attributes, such as the Workflow ID or Run ID, and shorten any fields that exceed length limits.

A Workflow Task can't both start a [Child Workflow](/child-workflows) and request its cancellation.
If your Workflow does both in the same Workflow Task, request the cancellation in a later Workflow Task.

### Bad Request Cancel Nexus Operation Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_REQUEST_CANCEL_NEXUS_OPERATION_ATTRIBUTES`

The [RequestCancelNexusOperation](/references/commands#requestcancelnexusoperation) Command has unset or invalid attributes.

Read the reason in the error message for the attribute that failed validation.

### Bad Schedule Activity Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_SCHEDULE_ACTIVITY_ATTRIBUTES`

The [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command has unset or invalid attributes, or the Activity input exceeds the Payload size limit.

Set a Start-To-Close Timeout or a Schedule-To-Close Timeout on the [Activity](/activities).
If the input is too large, store the data in external storage and pass a reference to the Activity.

### Bad Schedule Nexus Operation Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_SCHEDULE_NEXUS_OPERATION_ATTRIBUTES`

The [ScheduleNexusOperation](/references/commands#schedulenexusoperation) Command has unset or invalid attributes.
For example, the Nexus Endpoint name in the caller Workflow doesn't exist.

Read the reason in the error message for the attribute that failed validation.

### Bad Search Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_SEARCH_ATTRIBUTES`

The Workflow set unset or invalid [Search Attributes](/search-attribute).
The Workflow Task keeps failing on retry until you fix the Search Attributes.

Register each Search Attribute on the Namespace with the type your Workflow uses, and keep values within the size limits.

### Bad Signal Input Size 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_SIGNAL_INPUT_SIZE`

The input of a [Signal](/sending-messages#sending-signals) sent to another Workflow Execution exceeds the size limit.

Reduce the size of the Signal input and deploy the change to your Workers.

### Bad Signal Workflow Execution Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_SIGNAL_WORKFLOW_EXECUTION_ATTRIBUTES`

The [SignalExternalWorkflowExecution](/references/commands#signalexternalworkflowexecution) Command has unset or invalid attributes, or its input exceeds the size limit.

Set the missing attributes and reduce the size of the input.

### Bad Start Child Execution Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_START_CHILD_EXECUTION_ATTRIBUTES`

The [StartChildWorkflowExecution](/references/commands#startchildworkflowexecution) Command has unset or invalid attributes, or its input exceeds the size limit.

Reduce the size of the Child Workflow input.
Check that every Search Attribute the Child Workflow sets is registered on the Namespace.

### Bad Start Timer Attributes 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_START_TIMER_ATTRIBUTES`

The [StartTimer](/references/commands#starttimer) Command is missing a Timer ID.

Set a valid Timer ID and deploy the change to your Workers.

### Cause Bad Update 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_UPDATE_WORKFLOW_EXECUTION_MESSAGE`

A [Workflow Update](/sending-messages#sending-updates) message from the [Worker](/workers#worker) (Acceptance, Rejection, or Response) has an invalid format or is missing required fields.

This error can indicate an unsupported SDK.
Use a [supported SDK](/encyclopedia/architecture/temporal-sdks) and upgrade to its latest release.

## Duplicate ID errors

### Schedule Activity Duplicate ID 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_SCHEDULE_ACTIVITY_DUPLICATE_ID`

The Workflow scheduled an [Activity](/activities) with an Activity ID that's already in use in the Workflow Execution.

Give each Activity in the Workflow a unique Activity ID.

### Start Timer Duplicate ID 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_START_TIMER_DUPLICATE_ID`

The Workflow started a Timer with a Timer ID that's already in use in the Workflow Execution.

Give each Timer in the Workflow a unique Timer ID.

## Pending operation limit errors

A Workflow Execution has a maximum number of pending operations of each type.
When a Command would exceed a limit, the Workflow Task fails and is retried.
See [Workflow Execution limits](/workflow-execution/limits#workflow-execution-limits).

On a self-hosted Temporal Service, you can change these limits with [dynamic configuration](/references/dynamic-configuration#size-limit-settings).

### Pending Activities Limit Exceeded 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_PENDING_ACTIVITIES_LIMIT_EXCEEDED`
- Default limit: 2,000 pending Activities

Scheduling another [Activity](/activities) would exceed the Workflow Execution's limit of pending Activities.

Reduce the number of Activities the Workflow runs at the same time, for example by processing work in batches.

### Pending Child Workflows Limit Exceeded 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_PENDING_CHILD_WORKFLOWS_LIMIT_EXCEEDED`
- Default limit: 2,000 pending Child Workflows

Starting another [Child Workflow](/child-workflows) would exceed the Workflow Execution's limit of pending Child Workflows.

Reduce the number of Child Workflows the Workflow runs at the same time.

### Pending Nexus Operations Limit Exceeded 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_PENDING_NEXUS_OPERATIONS_LIMIT_EXCEEDED`
- Default limit: 30 pending Nexus Operations

Scheduling another Nexus Operation would exceed the Workflow Execution's limit of pending Nexus Operations.

Reduce the number of Nexus Operations the Workflow runs at the same time.
See [Workflow Execution Nexus Operation limits](/workflow-execution/limits#workflow-execution-nexus-operation-limits) and, for Temporal Cloud, [Per Workflow Nexus Operation limits](/evaluate/cloud/limits#per-workflow-nexus-operation-limits).

### Pending Request Cancel Limit Exceeded 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_PENDING_REQUEST_CANCEL_LIMIT_EXCEEDED`
- Default limit: 2,000 pending Cancellation Requests to other Workflows

Requesting cancellation of another Workflow Execution would exceed the limit of pending Cancellation Requests.

Reduce the number of Cancellation Requests the Workflow sends at the same time.

### Pending Signals Limit Exceeded 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_PENDING_SIGNALS_LIMIT_EXCEEDED`
- Default limit: 2,000 pending Signals to other Workflows

Sending another [Signal](/sending-messages#sending-signals) to an external Workflow Execution would exceed the limit of pending outgoing Signals.

Reduce the number of Signals the Workflow sends to other Workflow Executions at the same time.

## Size limit errors

### gRPC Message Too Large 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`

The Workflow Task completion exceeds the 4 MB gRPC message size limit.
Replay produces the same oversized request, so the Temporal Service terminates the Workflow Execution.

This error usually means the Workflow schedules too many Activities, Child Workflows, or Commands in a single Workflow Task, or returns a large result.

Fix your Workflow code and start a new Workflow Execution.
Split work into smaller batches, return smaller results, use [Continue-As-New](/workflow-execution/continue-as-new) for long-running Workflows, or compress large Payloads with a custom [Payload Codec](/payload-codec).
See [gRPC message size limit](/troubleshooting/blob-size-limit-error).

### Payloads Too Large 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE`

A Payload in the Workflow Task completion exceeds the Payload size limit.
SDKs that report this cause leave the Workflow Execution open.
For the SDK versions that report it, see [Payload size limit error behavior](/troubleshooting/blob-size-limit-error#payload-error-behavior).

Store large data in external storage, pass a reference through the Workflow, and deploy the fix to your Workers.
See [Payload size limit](/troubleshooting/blob-size-limit-error).

## Workflow code and Worker errors

### Nondeterminism Error 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_NON_DETERMINISTIC_ERROR`

During Replay, the Workflow code produced Commands that don't match the Event History.
This happens when Workflow code changes in a [non-deterministic way](/workflow-definition#non-deterministic-change) while Workflow Executions are open.

See [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) for how to find and fix the change.

### Workflow Worker Unhandled Failure 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_WORKFLOW_WORKER_UNHANDLED_FAILURE`

The [Workflow Definition](/workflow-definition) raised an error or panic that the SDK doesn't treat as a Workflow Execution failure.

Read the stack trace in the WorkflowTaskFailed Event or the Worker logs, fix the code, and deploy the change to your Workers.

### Unhandled Command 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_UNHANDLED_COMMAND`

New [Events](/references/events), such as Signals, arrived while the Workflow Task was running, and the Workflow tried to close without handling them.
The Temporal Service fails the Workflow Task and schedules a new one that includes the new Events.

This error is common when a Workflow receives many [Signals](/sending-messages#sending-signals).
Before the Workflow completes, process every pending Signal.
In the Go SDK, drain the Signal Channel with `ReceiveAsync`.

### Cause Unhandled Update 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_UNHANDLED_UPDATE`

The Temporal Service received a Workflow Update while the Worker processed a Workflow Task that would close the Workflow Execution.

The Temporal Service rejects the Workflow Task completion and rewinds the Workflow so that Workflow code handles the pending Update.
This error is common when a Workflow receives frequent Updates.

### Cause Bad Binary 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_BAD_BINARY`

The [Worker](/workers) that completed the Workflow Task has a binary checksum that's marked as bad on the Namespace.

Deploy a Worker build that isn't marked as bad.

### Cause Unspecified 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_UNSPECIFIED`

The Workflow Task failed without a specific cause.

Read the failure message in the WorkflowTaskFailed Event and check your Workflow Definition.

## Temporal Service errors

The Temporal Service fails these Workflow Tasks because of an operation outside your Workflow code.

### Cause Reset Workflow 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_RESET_WORKFLOW`

The Workflow Task failed because the Workflow Execution was [reset](/workflow-execution/event#reset).

The reset starts a new run of the Workflow Execution.
If the new run didn't start, reset the Workflow Execution again.

### Reset Sticky Task Queue 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_RESET_STICKY_TASK_QUEUE`

The Worker failed the Workflow Task so that the next Workflow Task goes to the normal [Task Queue](/task-queue) instead of the Sticky Task Queue.

No action is needed.
The Temporal Service schedules the next Workflow Task automatically.

### Failover Close Command 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_FAILOVER_CLOSE_COMMAND`

A [Namespace](/namespaces) failover forced the Workflow Task to close.

No action is needed.
The Temporal Service schedules a retry automatically.

### Force Close Command 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_FORCE_CLOSE_COMMAND`

The Temporal Service forced the Workflow Task to close.
If the error is recoverable, the Temporal Service schedules a retry.

### Feature Disabled 

- Cause: `WORKFLOW_TASK_FAILED_CAUSE_FEATURE_DISABLED`

The Workflow Task requested a feature that's disabled on the Temporal Service, either for the whole Temporal Service or, more often, for the Workflow's Namespace.

Read the failure message in the WorkflowTaskFailed Event for the feature name, then turn on the feature for the Namespace or remove its use from the Workflow.

## Resource Exhausted causes 

The Temporal Service returns a gRPC `RESOURCE_EXHAUSTED` error with one of these causes when it throttles a request.
SDKs retry these requests automatically.
For triage steps, see [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations).

### Resource Exhausted Cause APS Limit 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_APS_LIMIT`

The caller exceeded the Namespace's actions per second limit.

Reduce your request rate, or on Temporal Cloud, compare your throughput with the Namespace [limits](/evaluate/cloud/limits) and request an increase.

### Resource Exhausted Cause Busy Workflow 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_BUSY_WORKFLOW`

The target Workflow Execution is busy.

Reduce the rate of requests, such as Signals and Updates, that you send to a single Workflow Execution.

### Resource Exhausted Cause Circuit Breaker Open 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`

A circuit breaker in the Temporal Service is open or half-open, and the Temporal Service is shedding load.

Treat this cause as a capacity problem.
Raising a rate limit doesn't help.
Add capacity to the Temporal Service or slow down your workload.

### Resource Exhausted Cause Concurrent Limit 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`

The caller exceeded the maximum number of concurrent requests, such as long-poll requests from Workers.

Reduce the [poller count](/develop/worker-performance/task-queues#poller-count) per [Worker](/workers) or the number of Workers polling the Namespace.

### Resource Exhausted Cause Ops Limit 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_OPS_LIMIT`

The Namespace exceeded its operations rate limit.

Reduce your request rate, or on Temporal Cloud, compare your throughput with the Namespace [limits](/evaluate/cloud/limits) and request an increase.

### Resource Exhausted Cause Persistence Limit 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_PERSISTENCE_LIMIT`

The Namespace exceeded the persistence rate limit.

On a self-hosted Temporal Service, check persistence latency and the [persistence QPS limits](/references/dynamic-configuration#qps-limits-for-persistence-store).

### Resource Exhausted Cause Persistence Storage Limit 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_PERSISTENCE_STORAGE_LIMIT`

The persistence storage limit was exceeded.

### Resource Exhausted Cause RPS Limit 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`

The caller exceeded the requests per second limit.

Reduce your request rate.
On Temporal Cloud, compare your throughput with the Namespace [limits](/evaluate/cloud/limits) and request an increase.
On a self-hosted Temporal Service, check the [Service-level RPS limits](/references/dynamic-configuration#service-level-rps-limits) after you confirm that persistence is healthy.

### Resource Exhausted Cause System Overload 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`

The Temporal Service is overloaded and is shedding load to protect itself.

Treat this cause as a capacity problem.
Raising a rate limit doesn't help.
Add capacity to the Temporal Service or slow down your workload.

### Resource Exhausted Cause Worker Deployment Limits 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_WORKER_DEPLOYMENT_LIMITS`

The request reached a [Worker Versioning limit](/evaluate/cloud/limits#worker-versioning-level), such as the maximum number of Worker Deployments in a Namespace or Versions in a Worker Deployment.

To free space for a new Version, stop polling from an old, drained Version so the Temporal Service can delete it.
See [Garbage collection](/production-deployment/worker-deployments/worker-versioning/sunset-and-gc#garbage-collection).

### Resource Exhausted Cause Unspecified 

- Cause: `RESOURCE_EXHAUSTED_CAUSE_UNSPECIFIED`

The Temporal Service throttled the request without a specific cause.

---

# Remote data encoding

Source: https://docs.temporal.io/remote-data-encoding

> Use remote encoding to transform data for the Temporal CLI and Web UI.

This page discusses [Remote Data Encoding](#remote-data-encoding).

## What is remote data encoding? 

Remote data encoding is exposing your Payload Codec via HTTP endpoints to support remote encoding and decoding.

Running your encoding remotely allows you to use it with the [Temporal CLI](/cli) to encode/decode data for several commands including `temporal workflow show` and with Temporal Web UI to decode data in your Workflow Execution details view.

To run data encoding/decoding remotely, use a [Codec Server](/codec-server). A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
The Codec Server is independent of the Temporal Service and decodes your encrypted payloads through predefined endpoints.
You create, operate, and manage access to your Codec Server in your own environment.
The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.

### Encoding data on the Web UI and CLI

You can perform some operations on your Workflow Execution using the Temporal CLI and the Web UI.
For example, you can start or signal an active Workflow Execution from the Temporal CLI or cancel a Workflow Execution from the Web UI, which might require inputs that contain sensitive data.

To encode this data, specify your [Codec Server endpoints](/codec-server) with the `codec-endpoint` parameter in [the Temporal CLI](/cli) and configure your Web UI to use the Codec Server endpoints.

### Decoding data on the Web UI and CLI

If you use custom encoding, Payload data handled by the Temporal Service is stored encoded. Since the Web UI uses the [Visibility](/temporal-service/visibility) database to show events and data stored on the Temporal Server, all data in the Workflow Execution History in your Web UI is displayed in the encoded format.

To decode output when using the Web UI and the Temporal CLI, use a [Codec Server](/codec-server).

Note that a remote data encoder is a separate system with access to your encryption keys and exposes APIs to encode and decode any data.
Evaluate and ensure that your remote data encoder endpoints are secured and only authorized users have access to them.

Samples:

- [Go](https://github.com/temporalio/samples-go/tree/main/codec-server)
- [Java](https://github.com/temporalio/sdk-java/tree/main/temporal-remote-data-encoder)
- [Python](https://github.com/temporalio/samples-python/tree/main/encryption)
- [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/encryption)

---

# Schedule

Source: https://docs.temporal.io/schedule

> A Schedule contains instructions for starting a Workflow Execution at specific times. Schedules provide a more flexible and user-friendly approach than Temporal Cron Jobs.

This page discusses [Schedule](#schedule).

## What is a Schedule? 

A Schedule contains instructions for starting a [Workflow Execution](/workflow-execution) at specific times.
Schedules provide a more flexible and user-friendly approach than [Temporal Cron Jobs](/cron-job).

- [How to enable Schedules](#limitations)
- [How to operate Schedules using the Temporal CLI](/cli/command-reference/schedule)

A Schedule has an identity and is independent of a Workflow Execution.
This differs from a Temporal Cron Job, which relies on a cron schedule as a property of the Workflow Execution.

> **ℹ️ Info:**
>
> For triggering a Workflow Execution at a specific one-time future point rather than on a recurring schedule, the [Start Delay](/workflow-execution/timers-delays#delay-workflow-execution) option should be used instead of a Schedule.
>

### Action

The Action of a Schedule is where the Workflow Execution properties are established, such as Workflow Type, Task Queue, parameters, and timeouts.

Workflow Executions started by a Schedule have the following additional properties:

- The Action's timestamp is appended to the Workflow Id.
- The `TemporalScheduledStartTime` [Search Attribute](/search-attribute) is added to the Workflow Execution.
  The value is the Action's timestamp.
- The `TemporalScheduledById` Search Attribute is added to the Workflow Execution.
  The value is the Schedule Id.

### Spec

The Schedule Spec defines when the Action should be taken.
Unless many Schedules have Actions scheduled at the same time, Actions should generally start within 1 second of the specified time.
There are two kinds of Schedule Spec:

- A simple interval, like "every 30 minutes" (aligned to start at the Unix epoch, and optionally including a phase offset).
- A calendar-based expression, similar to the "cron expressions" supported by lots of software, including the older Temporal Cron feature.

These two kinds have multiple representations, depending on the interface or SDK you're using, but they all support the same features.

In the Temporal CLI, for example, an interval is specified as a string like `45m` to mean every 45 minutes, or `6h/5h` to mean every 6 hours but at the start of the fifth hour within each period.

In the Temporal CLI, a calendar expression can be specified as either a traditional cron string with five (or six or seven) positional fields, or as JSON with named fields:

```json
{
  "year": "2022",
  "month": "Jan,Apr,Jul,Oct",
  "dayOfMonth": "1,15",
  "hour": "11-14"
}
```

The following calendar JSON fields are available:

- `year`
- `month`
- `dayOfMonth`
- `dayOfWeek`
- `hour`
- `minute`
- `second`
- `comment`

Each field can contain a comma-separated list of ranges (or the `*` wildcard), and each range can include a slash followed by a skip value.
The `hour`, `minute`, and `second` fields default to `0` while the others default to `*`, so you can describe many useful specs with only a few fields.

For `month`, names of months may be used instead of integers (case-insensitive, abbreviations permitted).
For `dayOfWeek`, day-of-week names may be used.

The `comment` field is optional and can be used to include a free-form description of the intent of the calendar spec, useful for complicated specs.

No matter which form you supply, calendar and interval specs are converted to canonical representations.
What you see when you "describe" or "list" a Schedule might not look exactly like what you entered, but it has the same meaning.

#### Phase offset

By default, intervals align to the Unix epoch (January 1, 1970 at midnight UTC), so without a phase offset, a 7-day interval would fire every Thursday at whatever time midnight UTC equates to in your local time, which may not be convenient for your use case.

Use a phase offset when you need your recurring Schedule to fire at a specific, human-meaningful time like every 10 days at 5pm UTC on a particular date, rather than accepting the arbitrary time the epoch alignment produces.

There is an `offset` parameter you can set so an `interval` spec fires exactly at a pre-determined date and time. Here's an example of how you can calculate the `offset` and apply it to your Schedules.

Timestamps are measured with UNIX time. That means 7 days is 604,800 seconds (7 * 24 * 60 * 60), 5 days is 432,000 (5 * 24 * 60 * 60), and so on. The Schedule fires when the following condition is true:

```
((currentTimestampUTC - phase offset) % interval) == 0
```

So using an interval of 7 days and a phase offset of 0, that means the Schedule fires on `December 28th 2023, at midnight UTC`:

```
((1703721600 - 0) % 604800) = 0
```

For an interval of 5 days and a phase offset of 0, the Schedule fires on `December 29th 2023`

```
((1703808000 - 0) % 432000) = 0
```

You can calculate the phase offset you need for any interval. Take the UNIX timestamp of any date and time you want your Schedule to execute, then calculate the following:

```
phase offset = referenceTimestampUTC % interval
```

For example, if you want a Schedule that will fire every 10 days, including Jun 19th 2024 at 5pm UTC, you need to calculate the UNIX timestamp:

```
UNIX Timestamp of 2024-06-19T17:00:00 UTC = 1718816400
```

Then you need to calculate the number of seconds for your interval:

```
Interval of 10 days = (10 * 24 * 60 * 60) = 864000
```

Finally, you can calculate the phase offset you need:

```
phase offset = 1718816400 % 864000 = 320400
```

So you will set your Schedule interval to `864000s` (10 days) and phase to `320400s` and you get exactly every 10 days at 5pm UTC.

#### Other Spec features

**Multiple intervals/calendar expressions:** A Spec can have combinations of multiple intervals and/or calendar expressions to define a specific Schedule.

**Time bounds:** Provide an absolute start or end time (or both) with a Spec to ensure that no actions are taken before the start time or after the end time.

**Exclusions:** A Spec can contain exclusions in the form of zero or more calendar expressions.
This can be used to express scheduling like "each Monday at noon except for holidays".
You'll have to provide your own set of exclusions and include it in each schedule; there are no pre-defined sets.
(This feature isn't currently exposed in the Temporal CLI or the Temporal Web UI.)

**Jitter:** If given, a random offset between zero and the maximum jitter is added to each Action time (but bounded by the time until the next scheduled Action).

**Time zones:** By default, calendar-based expressions are interpreted in UTC.
Temporal recommends using UTC to avoid various surprising properties of time zones.
If you don't want to use UTC, you can provide the name of a time zone.
The time zone definition is loaded on the Temporal Server Worker Service from either disk or the fallback embedded in the binary.

For more operational control, embed the contents of the time zone database file in the Schedule Spec itself.
(Note: this isn't currently exposed in the Temporal CLI or the web UI.)

### Pause

A Schedule can be Paused.
When a Schedule is Paused, the Spec has no effect.
However, you can still force manual actions by using the [temporal schedule trigger](/cli/command-reference/schedule#trigger) command.

To assist communication among developers and operators, a “notes” field can be updated on pause or resume to store an explanation for the current state.

Pausing a Schedule is different from pausing a Workflow Execution.
Schedule Pause stops the Schedule from taking future Actions, but it doesn't pause or otherwise affect any Workflow Executions that the Schedule already started.

### Policies

A Schedule supports a set of Policies that enable customizing behavior.

#### Overlap Policy

The Overlap Policy controls what happens when it is time to start a Workflow Execution but a previously started Workflow Execution is still running.
The following options are available:

- `Skip`: **Default**.
  Nothing happens; the Workflow Execution is not started.
- `BufferOne`: Starts the Workflow Execution as soon as the current one completes.
  The buffer is limited to one.
  If another Workflow Execution is supposed to start, but one is already in the buffer, only the one in the buffer eventually starts.
- `BufferAll`: Buffers Actions while another Workflow Execution is running.
  They are started sequentially, subject to the Catchup Window and internal buffer limits.
- `CancelOther`: Cancels the running Workflow Execution, and then starts the new one after the old one completes cancellation.
- `TerminateOther`: Terminates the running Workflow Execution and then starts the new one without waiting for Workflow cooperation.
- `AllowAll` Starts any number of concurrent Workflow Executions.
  With this policy (and only this policy), more than one Workflow Execution, started by the Schedule, can run simultaneously.

#### Interaction with Workflow Pause 

> **Pre-release**
> Workflow Pause is in Pre-release. APIs and behavior may change before the stable release.

[Workflow Pause](/cli/command-reference/workflow#pause) pauses a specific Workflow Execution.
If that Workflow Execution was started by a Schedule, the Schedule itself stays active unless you also pause the Schedule.

A Paused Workflow Execution is still an open Workflow Execution, so it counts as the currently running execution when the Schedule evaluates its Overlap Policy.
This affects regularly scheduled Actions, manual triggers, and Backfills:

- With `Skip`, each new scheduled Action is skipped while the previous scheduled Workflow Execution remains Paused.
- With `BufferOne`, one Action can wait behind the Paused Workflow Execution. Additional Actions are skipped while the buffer is full.
- With `BufferAll`, matching Actions are buffered, subject to the Catchup Window and internal buffer limits, and won't start until the Paused Workflow Execution closes.
- With `AllowAll`, new Workflow Executions can start even while the earlier Workflow Execution is Paused.
- With `CancelOther`, the Schedule requests cancellation of the Paused Workflow Execution and starts the next Workflow Execution only after the canceled Workflow Execution closes. If the Paused Workflow Execution can't process cancellation until it is unpaused, the next start still waits.
- With `TerminateOther`, the Schedule terminates the Paused Workflow Execution and then starts the next Workflow Execution without waiting for Workflow cooperation.

If you pause a Workflow Execution for investigation and don't want the Schedule to skip or buffer future Actions, pause the Schedule too.
When you're ready to resume the automation, unpause the Workflow Execution and then unpause the Schedule.
If Actions were skipped while the Workflow Execution was Paused, use [Backfill](#backfill) to run them intentionally with the Overlap Policy you want.

#### Catchup Window

The Temporal Service might be down or unavailable, or an Overlap Policy might delay an Action past its scheduled time.
The Catchup Window controls how late an automated Action can be taken.
The default is one year, meaning Actions will be taken unless over one year late.
If your Actions are more time-sensitive, you can set the Catchup Window to a smaller value (minimum ten seconds), accepting that an outage or other delay longer than the window could lead to missed Actions.
The window is measured from the jitter-adjusted Action time, if jitter is configured.
Manual triggers and Backfills are not restricted by the Catchup Window.
(But you can always [Backfill](#backfill).)

For examples of skipped Actions, buffering delays, and cancellation behavior, see [Common pitfalls](/troubleshooting/schedule-missed-actions#common-pitfalls).

#### Pause-on-failure

If this policy is set, a Workflow Execution started by a Schedule that ends with a failure or timeout (but not Cancellation or Termination) causes the Schedule to automatically pause.

Note that with the `AllowAll` Overlap Policy, this pause might not apply to the next Workflow Execution, because the next Workflow Execution might have started before the failed one finished.
It applies only to Workflow Executions that were scheduled to start after the failed one finished.

### Backfill

A Schedule can be Backfilled.
When a Schedule is Backfilled, all the Actions that would have been taken over a specified time period are taken now (in parallel if the `AllowAll` [Overlap Policy](#overlap-policy) is used; sequentially if `BufferAll` is used).
You might use this to fill in runs from a time period when the Schedule was paused due to an external condition that's now resolved, or a period before the Schedule was created.

### Limit number of Actions

A Schedule can be limited to a certain number of scheduled Actions (that is, not trigger immediately).
After that it will act as if it were paused.

### Last completion result

A Workflow started by a Schedule can obtain the completion result from the most recent successful run.
(How you do this depends on the SDK you're using.)

For overlap policies that don't allow overlap, “the most recent successful run” is straightforward to define.
For the `AllowAll` policy, it refers to the run that completed most recently, at the time that the run in question is started.
Consider the following overlapping runs:

```
time -------------------------------------------->
 A     |----------------------|
 B               |-------|
 C                          |---------------|
 D                                |--------------T
```

If D asks for the last completion result at time T, it gets the result of A.
Not B, even though B started more recently, because A completed later.
And not C, even though C completed after A, because the result for D is captured when D is started, not when it's queried.

Failures and timeouts do not affect the last completion result.

> **📝 Note:**
>
> When a Schedule triggers a Workflow that completes successfully and yields a result, the result from the initial Schedule execution can be accessed by the subsequent scheduled execution through `LastCompletionResult`.
>
> Be aware that if, during the subsequent run, the Workflow employs the [Continue-As-New](/workflow-execution/continue-as-new) feature, `LastCompletionResult` won't be accessible for this new Workflow iteration.
>
> It is important to note that the [status](/workflow-execution#workflow-execution-status) of the subsequent run is marked as `Continued-As-New` and not as `Completed`.
>

> **⚠️ Caution:**
>
> A scheduled Workflow Execution may complete with a result up to the maximum blob size (2 MiB by default).
> However, due to internal limitations, results that are within 1 KiB of this limit cannot be passed to the next execution.
> So, for example, a Workflow Execution that returns a result of size 2,096,640 bytes (which is above 2MiB - 1KiB limit)
> will be allowed to complete successfully, but that value will not be available as a last completion result.
> This limitation may be lifted in the future.
>

### Last failure

A Workflow started by a Schedule can obtain the details of the failure of the most recent run that ended at the time when the Workflow in question was started. Unlike last completion result, a _successful_ run _does_ reset the last failure.

### Limitations

Internally, a Schedule is implemented as a Workflow.
If you're using Elasticsearch, these Workflow Executions are hidden from normal views.

`ListSchedules` and `CountSchedules` are served by the Visibility subsystem. They share the [Visibility API rate limit](/evaluate/cloud/limits#visibility-api-rate-limit) and are eventually consistent, so a newly created or deleted schedule may not appear in the list right away. For more detail, see [Operations that use Visibility](/visibility#operations-that-use-visibility).

---

# Search Attributes

Source: https://docs.temporal.io/search-attribute

> This guide on Temporal Search Attributes explains how to set up, configure, and use default and custom Search Attributes in Temporal Server versions. Learn about supported types, limits, and how to use them to enhance Workflow filtering and querying.

This page discusses the following:

- [Search Attributes](#search-attribute)
- [Default Search Attributes](#default-search-attribute)
- [Custom Search Attributes](#custom-search-attribute)

## What is a Search Attribute? 

A Search Attribute is an indexed field used in a [List Filter](/list-filter) to filter a list of [Workflow Executions](/workflow-execution) that have the Search Attribute in their metadata.

Each Search Attribute is a key-value pair metadata object included in a Workflow Execution's Visibility information.
This information is available in the Visibility store.

> **⚠️ Caution:**
> Do not use sensitive data or PII in Search Attributes
>
> Do not include sensitive data, secrets, or personally identifiable information (PII) in Search Attribute **names or values**.
> Search Attribute values are stored unencrypted in the Visibility store and are not processed by a custom [Payload Codec](/payload-codec#payload-codec).
> The Temporal Server must be able to read these values in plain text to support filtering and ordering, so encryption is not possible without breaking search functionality.
>
> Attribute names are also visible in Namespace configuration, query expressions, and Temporal UI.
> Using sensitive data in either names or values risks exposure to anyone with Namespace access and may violate data protection regulations such as GDPR, HIPAA, or SOC 2.
>

Temporal provides some [default Search Attributes](/search-attribute#default-search-attribute), such as `ExecutionStatus`, the current state of your Workflow Executions.
You can also create [custom Search Attribute](/search-attribute#custom-search-attribute) keys in your Visibility store and assign values when starting a Workflow Execution or in Workflow code.

When using [Continue-As-New](/workflow-execution/continue-as-new) or a [Temporal Cron Job](/cron-job), Search Attribute keys are carried over to the new Workflow Run by default.
Search Attribute values are only available for as long as the Workflow is.

Search Attributes are most effective for search purposes or tasks requiring collection-based result sets.
For business logic in which you need to get information about a Workflow Execution, consider one of the following:

- Storing state in a local variable and exposing it with a Query.
- Storing state in an external datastore through Activities and fetching it directly from the store.

If your business logic requires high throughput or low latency, store and fetch the data through Activities.
You might experience lag due to time passing between the Workflow's state change and the Activity updating the Visibility store.

### Default Search Attributes 

> **⚠️ Caution:**
> The `BinaryChecksums` and `BuildIds` Search Attributes are deprecated. They will stop being propagated for new Workflows starting in Temporal Server 1.32 and will be fully removed from query support in Temporal Server 1.34.
>
> Instead use:
> - `TemporalWorkflowVersioningBehavior` for querying the Workflow Versioning behavior or if it is unversioned.
> - `TemporalWorkerDeploymentVersion` for querying the Worker Deployment Version the Workflow is assigned to at the moment.
> - `TemporalUsedWorkerDeploymentVersions` for querying the list of all Worker Deployment Versions that processed the Workflow.
>

A Temporal Service has a set of default Search Attributes already available.
Default Search Attributes are set globally in any Namespace.
These Search Attributes are created when the initial index is created.

| NAME                                 | TYPE         | DEFINITION                                                                                                                                                                                                                                                                             |
|--------------------------------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| BatcherUser                          | Keyword      | Used by internal batcher Workflow that runs in `TemporalBatcher` Namespace division to indicate the user who started the batch operation.                                                                                                                                              |
| ~BinaryChecksums~                    | ~Keyword List~ | List of binary IDs of Workers that run the Workflow Execution. Deprecated since server version 1.21.                                                                                                                                                                                  |
| ~BuildIds~                           | ~Keyword List~ | List of Worker Build IDs that have processed the Workflow Execution, formatted as `versioned:{BuildId}` or `unversioned:{BuildId}`, or the sentinel `unversioned` value. Deprecated since server version 1.31 in favor of the `TemporalUsedWorkerDeploymentVersions` search attribute. |
| CloseTime                            | Datetime     | The time at which the Workflow Execution completed.                                                                                                                                                                                                                                    |
| ExecutionDuration                    | Int          | The time needed to run the Workflow Execution (in nanoseconds). Available only for closed Workflows.                                                                                                                                                                                   |
| ExecutionStatus                      | Keyword      | The current state of the Workflow Execution. Also set on [Standalone Activity Executions](/standalone-activity), where it holds the Activity Execution status.                                                                                                                          |
| ExecutionTime                        | Datetime     | The time at which the Execution actually begins running; same as `StartTime` for most cases but different for Cron Workflows and retried Workflows. Also set on [Standalone Activity Executions](/standalone-activity), where a [Start Delay](/standalone-activity#start-delay) pushes it past `StartTime`, making it the attribute to query for delayed jobs. |
| HistoryLength                        | Int          | The number of events in the history of Workflow Execution. Available only for closed Workflows.                                                                                                                                                                                        |
| HistorySizeBytes                     | Long         | The size of the Event History.                                                                                                                                                                                                                                                         |
| RunId                                | Keyword      | Identifies the current Workflow Execution Run.                                                                                                                                                                                                                                         |
| StartTime                            | Datetime     | The time at which the Workflow Execution started.                                                                                                                                                                                                                                      |
| StateTransitionCount                 | Int          | The number of times that Workflow Execution has persisted its state. Available only for closed Workflows.                                                                                                                                                                              |
| TaskQueue                            | Keyword      | Task Queue used by Workflow Execution.                                                                                                                                                                                                                                                 |
| TemporalChangeVersion                | Keyword List | Stores change/version pairs if the GetVersion API is enabled.                                                                                                                                                                                                                          |
| TemporalReportedProblems             | Keyword List | Stores information about Workflow task failures. Formatted as `category=<category> cause=<cause>`.                                                                                                                                                                                     |
| TemporalScheduledStartTime           | Datetime     | The time that the Workflow is schedule to start according to the Schedule Spec. Can be manually triggered. Set on Schedules.                                                                                                                                                           |
| TemporalScheduledById                | Keyword      | The ID of the Schedule that started the Workflow.                                                                                                                                                                                                                                      |
| TemporalSchedulePaused               | Boolean      | Indicates whether the Schedule has been paused. Set on Schedules.                                                                                                                                                                                                                      |
| TemporalUsedWorkerDeploymentVersions | Keyword List | The list of all Worker Deployment Versions that completed Workflow Tasks for the Workflow Execution. Each item is in the format `<deployment name>:<build id>`.                                                                                                                        |
| TemporalWorkerDeployment             | Keyword      | Indicates the name of the associated Worker Deployment.                                                                                                                                                                                                                                |
| TemporalWorkerDeploymentVersion      | Keyword      | Indicates the Version string of the associated Worker Deployment, in the format `<deployment name>:<build id>`.                                                                                                                                                                        |
| TemporalWorkflowVersioningBehavior   | Keyword      | Indicates the associated Worker Versioning behavior ("Pinned", "Auto-Upgrade", or null if not using Worker Versioning).                                                                                                                                                                |
| WorkflowId                           | Keyword      | Identifies the Workflow Execution.                                                                                                                                                                                                                                                     |
| WorkflowType                         | Keyword      | The type of Workflow.                                                                                                                                                                                                                                                                  |

- All default Search Attributes are reserved and read-only.
  You cannot create a custom one with the same name or alter the existing one.

- Search Attributes are not encrypted in the system.
  Do not use sensitive data or PII as either the Search Attribute name or value.

- To use default Search Attributes with the `Temporal` prefix in a List Filter, you can use their non-prefixed alias.
  Refer to [Search Attribute aliasing](./list-filter.mdx#search-attribute-aliasing) for details.

- On a Workflow Execution, ExecutionStatus values correspond to Workflow Execution statuses: Running, Completed, Failed, Canceled, Terminated, ContinuedAsNew, TimedOut.

- On a [Standalone Activity Execution](/standalone-activity), ExecutionStatus values correspond to Activity Execution statuses: Running, Completed, Failed, Canceled, Terminated, TimedOut, Paused.
  `Paused` is the status of an Activity that an operator has paused with [Pause](/activity-operations). Query it with `ExecutionStatus = 'Paused'`.

- StartTime, CloseTime, and ExecutionTime are stored as dates but are supported by queries that use either EpochTime in nanoseconds or a string in [RFC3339Nano format](https://pkg.go.dev/time#pkg-constants) (such as "2006-01-02T15:04:05.999999999Z07:00").

- ExecutionDuration is stored in nanoseconds but is supported by queries that use integers in nanoseconds, [Golang duration format](https://pkg.go.dev/time#ParseDuration), or "hh:mm:ss" format.

- CloseTime, HistoryLength, StateTransitionCount, and ExecutionDuration are present only in a closed Workflow Execution.

- ExecutionTime can differ from StartTime in retry and cron use cases.

You can use the default Search Attributes in a List Filter, such as in the Temporal Web UI or with the `temporal workflow list` commands, under the following conditions:

- Without advanced Visibility, you can only use the `=` operator with a single default Search Attribute in your List Filter.
  For example: `temporal workflow list --query "ExecutionStatus = 'Completed'"` or `temporal workflow list --query "WorkflowType = 'YourWorkflow'"`.
- With advanced Visibility, you can combine default Search Attributes in a List Filter to get a list of specific Workflow Executions.
  For example: `temporal workflow list --query "WorkflowType = 'main.YourWorkflowDefinition' and ExecutionStatus != 'Running' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"`

### Custom Search Attributes 

You can [create custom Search Attributes](/self-hosted-guide/visibility/custom-search-attributes#create-custom-search-attributes) with unique key names that are relevant to your business needs.

Use custom Search Attributes in a List Filter, such as in the Temporal Web UI or with the `temporal workflow list` commands, under the following conditions:

- Without advanced Visibility, you cannot use a custom Search Attribute in your List Filter.
- With advanced Visibility, you can create multiple custom Search Attributes and use them in combinations with List Filters to get a list of specific Workflow Executions.
  For example: `temporal workflow list --query "WorkflowType = 'main.YourWorkflowDefinition' and YourCustomSA = 'YourCustomSAValue' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"`
  - With Temporal Server v1.19 and earlier, you must [integrate Elasticsearch](/self-hosted-guide/visibility/elasticsearch) to use custom Search Attributes with List Filters.
  - With Temporal Server v1.20 and later, custom Search Attribute capabilities are available on MySQL (v8.0.17 or later), PostgreSQL (v12 and later), and SQLite (v3.31.0 and later), in addition to Elasticsearch.

If you use Elasticsearch as your Visibility store, your custom Search Attributes apply globally and can be used across Namespaces.
However, if using any of the [supported SQL databases](/self-hosted-guide/visibility) with Temporal Server v1.20 and later, your custom Search Attributes are associated with a specific Namespace and can be used for Workflow Executions in that Namespace.

See [custom Search Attributes limits](/search-attribute#custom-search-attribute-limits) for limits on the number and size of custom Search Attributes you can create.

#### Supported types 

Custom Search Attributes must be one of the following types:

- Bool
- Datetime
- Double
- Int
- Keyword
- KeywordList
- Text

Note:

- **Double** is backed up by `scaled_float` Elasticsearch type with scale factor 10000 (4 decimal digits).
- **Datetime** is backed up by `date` type with milliseconds precision in Elasticsearch 6 and `date_nanos` type with nanoseconds precision in Elasticsearch 7.
- **Int** is 64-bit integer (`long` Elasticsearch type).
- With Temporal Server v1.19 and earlier, the **Keyword** type can store a list of values.
- With Temporal Server v1.20 and later, the **Keyword** type supports only a single value.
  To store a list of values, use **KeywordList**.

> **⚠️ Warning:**
> List-of-values deprecation for non-list types
>
> Storing a list of values in any single-value Search Attribute type — for example, multiple strings in a **Keyword** or **Text** attribute, or multiple integers in an **Int** attribute — is deprecated.
> **KeywordList** is the only type designed to store a list of values.
>
> New namespaces on Temporal Cloud are already blocked from using this pattern. Existing Namespaces that rely on it will need to migrate before Sep. 30, 2026. After that date, Workflow Executions with invalid Search Attribute values will fail. 
> This pattern was unintended behavior inherited from the underlying Elasticsearch encoding, not a supported feature.
>
> To migrate, choose the path that matches the type you store today:
>
> - **Keyword:** Migrate to **KeywordList**.
> - **Int:** Either migrate to **KeywordList** (store integers as strings, with string-based comparison only) or split the list into multiple individual **Int** attributes.
> - **Text:** Concatenate the list values into a single space-delimited string (for example, store `["wf-1", "wf-2"]` as `"wf-1 wf-2"`).
>   Because **Text** tokenizes values into words, a filter such as `SourceWorkflowIds = 'wf-1'` still matches.
>   If you only need exact matching rather than word-level search, migrate to **KeywordList** instead.
>
>

- The **Text** type cannot be used in the "Order By" clause.

#### Choose a string type 

Temporal offers three string-based Search Attribute types.
The right choice depends on what you store and how you need to search for it.

- **Use `Keyword`** when you need exact matching on a single value — workflow type names, UUIDs, hyphenated IDs, dotted class names, file paths, enum values, or any structured identifier.
  `Keyword` stores the value as-is and supports `=`, `!=`, `IN`, `STARTS_WITH`, `BETWEEN`, and `ORDER BY`.
- **Use `KeywordList`** when you need to store and match against multiple values on a single attribute — tags, labels, or categories.
  `KeywordList` supports `=`, `!=`, and `IN`.
- **Use `Text`** only for free-form, human-readable content where you want word-level search — descriptions, notes, or error messages.
  `Text` tokenizes values into individual words and uses OR matching with the `=` operator.
  See [Text matching in List Filters](/list-filter#text) for details on how tokenization and matching work.

> **💡 Tip:**
>
> Most custom Search Attributes store structured identifiers, so `Keyword` is the right default choice.
> Choose `Text` only when you specifically need word-level search on prose-like content.
>

> **📝 Note:**
>
> Broader tokenizer and exact-matching improvements for `Text` Search Attributes are being evaluated as part of future Visibility enhancements.
>

#### Custom Search Attributes limits 

The following table lists the maximum number of custom Search Attributes you can create per Namespace by supported Visibility database.

| Search Attribute type | MySQL (v8.0.17 and later) | PostgreSQL (v12 and later) | SQLite (v3.31.0 and later) | Temporal Cloud |
| --------------------- | :-----------------------: | :------------------------: | :------------------------: | :------------: |
| Bool                  |             3             |             3              |             3              |       20       |
| Datetime              |             3             |             3              |             3              |       20       |
| Double                |             3             |             3              |             3              |       20       |
| Int                   |             3             |             3              |             3              |       20       |
| Keyword               |            10             |             10             |             10             |       40       |
| KeywordList           |             3             |             3              |             3              |       5        |
| Text                  |             3             |             3              |             3              |       5        |

Temporal does not impose a limit on the number of custom Search Attributes you can create with Elasticsearch. However, [Elasticsearch sets a default mapping limit](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/mapping-settings-limit.html) that may apply.
Custom Search Attributes are an advanced Visibility feature and are not supported on Cassandra.

Size limits for a custom Search Attribute:

- The default single Search Attribute **value** size limit is 2 KB.

- The maximum total Search Attribute size is 40 KB.

- The maximum total characters per Search Attribute value is 255.

For Temporal Cloud specific configurations, see the [Defaults, limits, and configurable settings -Temporal Cloud](/evaluate/cloud/limits#number-of-custom-search-attributes) guide.

### Search Attributes on Standalone Activity Executions 

[Standalone Activity Executions](/standalone-activity) have their own Visibility records, separate from Workflow Executions.
`ListActivities` and `CountActivities` (and `temporal activity list` and `temporal activity count`) return Standalone Activity Executions only.
Activities that run inside a Workflow are not returned by them, and they have no Visibility record of their own; find them through their Workflow Execution.

The Temporal Service sets the following Search Attributes on every Standalone Activity Execution:

| Name | Type | Definition |
|------|------|------------|
| ActivityId | Keyword | The business identifier of the Standalone Activity Execution. |
| ActivityType | Keyword | The Activity Type name. |
| ExecutionStatus | Keyword | The current state of the Standalone Activity Execution: Running, Completed, Failed, Canceled, Terminated, TimedOut, or Paused. |
| ExecutionTime | Datetime | The time at which the first Activity Task is dispatched. A [Start Delay](/standalone-activity#start-delay) pushes this past the time the Activity was started. |
| TaskQueue | Keyword | Task Queue the Standalone Activity Execution is dispatched on. |

Custom Search Attributes also apply. Set them when you start the Standalone Activity, then filter on them the same way you would for a Workflow Execution.

Search Attributes that describe Workflow structure or Event History, such as `WorkflowType`, `HistoryLength`, `HistorySizeBytes`, `ParentWorkflowId`, and `RootWorkflowId`, apply to Workflow Executions and are not set on Standalone Activity Executions.

### Usage 

Search Attributes available in your Visibility store can be used with Workflow Executions for the Temporal Service.
To actually have results from the use of a [List Filter](/list-filter), Search Attributes must be added to a Workflow Execution as metadata.

- To create custom Search Attributes in your Visibility store, see [Create custom Search Attributes](/self-hosted-guide/visibility/custom-search-attributes#create-custom-search-attributes).
- To remove a custom Search Attribute from the Visibility store, see [Remove custom Search Attributes](/self-hosted-guide/visibility/custom-search-attributes#remove-custom-search-attributes).
  Removing custom Search Attributes is not supported on Temporal Cloud.
- To rename a custom Search Attribute on Temporal Cloud, see [`temporal cloud namespace search-attribute rename`](/cli/command-reference/cloud/namespace#search-attribute-rename).

With Workflows you can do the following:

- Set the value of Search Attributes in your Workflow
- Update the value set for a Search Attribute from within the Workflow code
- Remove the value set for a Search Attribute from within the Workflow code

> **ℹ️ Info:**
> Manage Search Attributes by SDK
>
> - [How to manage Search Attributes using the Go SDK](/develop/go/platform/observability#visibility)
> - [How to manage Search Attributes using the Java SDK](/develop/java/platform/observability#visibility)
> - [How to manage Search Attributes using the PHP SDK](/develop/php/platform/observability#visibility)
> - [How to manage Search Attributes using the Python SDK](/develop/python/platform/observability#visibility)
> - [How to manage Search Attributes using the TypeScript SDK](/develop/typescript/platform/observability#visibility)
> - [How to manage Search Attributes using the .NET SDK](/develop/dotnet/platform/observability#search-attributes)
>

- To get a list of Search Attributes using the Temporal CLI, issue `temporal operator search-attribute list`. See [Search Attributes](/search-attribute).

After you add and set your Search Attributes, use your default or custom Search Attributes in a List Filter.

---

# Temporal Platform security

Source: https://docs.temporal.io/security

> Discover general security practices of Temporal Technologies, SaaS security with Temporal Cloud, and how to self-host a secure Temporal Platform on our security pages.

Find security information for your Temporal deployment, whether you're using Temporal Cloud or self-hosting.

- [Company Security](https://trust.temporal.io): Learn about Temporal Technologies' general security practices, compliance certifications, and organizational security measures.
- [Temporal Cloud Security](/evaluate/cloud/security): Explore the security features of our SaaS offering, including mTLS, end-to-end encryption, and enterprise compliance.
- [Self-Hosted Security](/self-hosted-guide/security): Discover how to deploy and secure your own Temporal Platform infrastructure with production-ready best practices.
- [Temporal Cloud Security Whitepaper](https://temporal.io/pages/cloud-security-white-paper): Learn how Temporal Cloud provides provable security by design - orchestrating encrypted workflows without ever accessing your sensitive data.

---

# Self-hosted Temporal Service guide

Source: https://docs.temporal.io/self-hosted-guide

Welcome to the self-hosted Temporal Service guide. This guide shows you how to self-host open source infrastructure
software that orchestrates your durable applications.

> **💡 Tip:**
> Do you need a production Temporal Service?
>
> If you're still developing and testing your application locally, you may not need a production Temporal Service. Use the
> [Temporal CLI development server](/cli/command-reference/server#start-dev) — a single binary with no external dependencies:
>
> `temporal server start-dev`
>
> This starts a complete Temporal Service with Web UI on your local machine. We recommend this for local development
> regardless of whether you plan to use Temporal Cloud or self-host in production. See the
> [Temporal CLI server](/cli/command-reference/server) page for configuration options.
>

## Plan and deploy your service

- [Deployment](/self-hosted-guide/deployment): Choose a deployment approach (Docker, Kubernetes, or manual) and set up a
  production-ready Temporal Service.
- [Embedded server](/self-hosted-guide/embedded-server): Run Temporal in-process as a Go library for local development
  and testing scenarios.
- [Defaults](/self-hosted-guide/defaults): Review platform limits and default settings that can affect Workflow and
  Activity behavior.
- [Production checklist](/self-hosted-guide/production-checklist): Validate readiness for scale, reliability,
  operations, and long-term maintainability.

## Operate your self-hosted service

- [Namespaces](/self-hosted-guide/namespaces): Create and manage Namespace isolation, retention, and related
  configuration.
- [Security](/self-hosted-guide/security): Configure TLS/mTLS, authentication, authorization, and related hardening
  controls.
- [Monitoring](/self-hosted-guide/monitoring): Collect and visualize service and SDK metrics to troubleshoot and track
  health.
- [Visibility](/self-hosted-guide/visibility): Configure Visibility storage so you can list, filter, and search Workflow
  Executions.
- [Upgrading server](/self-hosted-guide/upgrade-server#upgrade-server): Perform safe, sequential server and schema
  upgrades.

## Protect data and enable advanced features

- [Data encryption](/production-deployment/data-encryption): Use Payload Codecs and Codec Server patterns to protect
  sensitive Workflow data.
- [Archival](/self-hosted-guide/archival): Move closed Workflow Histories and Visibility records to blob storage for longer
  retention.
- [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication): Replicate Workflow state across clusters
  for failover and disaster recovery.
- [Temporal Nexus](/production-deployment/self-hosted-guide/nexus): Enable Nexus in self-hosted environments to connect
  Temporal Applications across boundaries.

---

# Self-hosted Archival setup

Source: https://docs.temporal.io/self-hosted-guide/archival

> Configure Archival to store closed Workflow Event Histories and Visibility records in blob storage.

Use Archival to back up closed Workflow Execution [Event Histories](/workflow-execution/event#event-history) and
Visibility records from Temporal Service persistence to blob storage.

- [How to create a custom Archiver](#custom-archiver)
- [How to set up Archival](#set-up-archival)

When a Workflow Execution closes, Temporal schedules close-processing tasks for both Visibility records and Event
History Archival. Archival then runs asynchronously after a randomized delay. By default, that delay is up to 5 minutes
set by `history.archivalProcessorArchiveDelay`, and is capped by the Namespace
[Retention Period](/temporal-service/temporal-server#retention-period).

The closed execution still stays in Temporal persistence until retention cleanup runs. For some time, the same closed
execution can exist in both persistence and archival storage. Archival enables Workflow Execution data to persist beyond
retention without overwhelming the Temporal Service persistence store.

Use this to keep closed Workflow data available for compliance, audits, and debugging without keeping all closed data in
your primary persistence store.

> **ℹ️ Info:**
> Experimental feature
>
> Archival is an **experimental** feature and not subject to normal
> [versioning and support policy](/temporal-service/temporal-server#versions-and-support).
>

Archival is not supported when running Temporal through Docker. It's disabled by default when installing the system
manually and when deploying through
[helm charts](https://github.com/temporalio/helm-charts/blob/main/charts/temporal/templates/server-configmap.yaml). It
can be enabled in the server [configuration](https://github.com/temporalio/temporal/blob/main/config/development.yaml).

### Set up Archival 

To set up [Archival](/temporal-service/archival), decide the following:

- **Which provider to use:** S3, Google Cloud, local file system, or custom.
- **Which URI to use:** URI scheme and path identify the provider and destination.
- **Which Namespace should use Archival:** Archival must be enabled at both the Temporal Service level and the Namespace
  level.

Take the following steps to set up Archival:

1. [Choose an Archival provider](#choose-an-archival-provider).
2. [Configure Archival options](#configure-archival-options).
3. [Create an Archiving Namespace](#create-an-archiving-namespace).

#### Choose an Archival provider 

Temporal directly supports several providers:

- **Local file system**: The
  [filestore archiver](https://github.com/temporalio/temporal/tree/main/common/archiver/filestore) is used to archive
  data in the file system of whatever host the Temporal server is running on. In the case of
  [temporal helm-charts](https://github.com/temporalio/helm-charts), the archive data is stored in the `history` pod.
  APIs do not function with the filestore archive. This provider is used mainly for local installations and testing and
  should not be relied on for production environments.
- **Google Cloud**: The [gcloud archiver](https://github.com/temporalio/temporal/tree/main/common/archiver/gcloud) is
  used to connect and archive data with [Google Cloud](https://cloud.google.com/storage).
- **S3**: The [s3store archiver](https://github.com/temporalio/temporal/tree/main/common/archiver/s3store) is used to
  connect and archive data with [S3](https://aws.amazon.com/s3).
- **Custom**: If you want to use a provider that is not currently supported, you can
  [create your own archiver](#custom-archiver) to support it.

Save the provider URI so you can pass it when you create a Namespace with Archival enabled.

#### Configure Archival options 

Configure Archival in
[`config/development.yaml`](https://github.com/temporalio/temporal/blob/main/config/development.yaml#L93):

```yaml
# Temporal Service level Archival config
archival:
  # Event History configuration
  history:
    # Archival is enabled at the Temporal Service level
    state: 'enabled'
    enableRead: true
    # Namespaces can use either the local filestore provider or the Google Cloud provider
    provider:
      filestore:
        fileMode: '0666'
        dirMode: '0766'
      gstorage:
        credentialsPath: '/tmp/gcloud/keyfile.json'

# Default values for a Namespace if none are provided at creation
namespaceDefaults:
  # Archival defaults
  archival:
    # Event History defaults
    history:
      state: 'enabled'
      # New Namespaces will default to the local provider
      URI: 'file:///tmp/temporal_archival/development'
```

You can disable Archival by setting `archival.history.state` and `namespaceDefaults.archival.history.state` to
`"disabled"`.

Example:

```yaml
archival:
  history:
    state: 'disabled'

namespaceDefaults:
  archival:
    history:
      state: 'disabled'
```

The following table shows the available configuration options and their accepted values:

| Config                                         | Acceptable values                                                                  | Description                                                                                                                  |
| ---------------------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `archival.history.state`                       | `enabled`, `disabled`                                                              | Must be `enabled` to use the Archival feature with any Namespace in the Temporal Service.                                    |
| `archival.history.enableRead`                  | `true`, `false`                                                                    | Must be `true` to read from the archived Event History.                                                                      |
| `archival.history.provider`                    | Sub provider configs are `filestore`, `gstorage`, `s3store`, or `your_custom_provider`. | Default config specifies `filestore`.                                                                                        |
| `archival.history.provider.filestore.fileMode` | File permission string                                                             | File permissions of the archived files. We recommend using the default value of `"0666"` to avoid read/write issues.         |
| `archival.history.provider.filestore.dirMode`  | File permission string                                                             | Directory permissions of the archive directory. We recommend using the default value of `"0766"` to avoid read/write issues. |
| `namespaceDefaults.archival.history.state`     | `enabled`, `disabled`                                                              | Default state of the Archival feature whenever a new Namespace is created without specifying the Archival state.             |
| `namespaceDefaults.archival.history.URI`       | Valid URI                                                                          | Must be a URI of the file store location and match a schema that correlates to a provider.                                   |

Additional resources: [Temporal Service configuration reference](/references/service-configuration).

#### Create an Archiving Namespace 

Although Archival is configured at the Temporal Service level, it operates independently within each Namespace. If you
don't specify an Archival URI during Namespace creation, the Namespace uses `namespaceDefaults.archival.history.URI`
from `config/development.yaml`. The Archival URI cannot be changed after the Namespace is created. Each Namespace
supports only a single Archival URI, but each Namespace can use a different URI. A Namespace can safely switch Archival
between `enabled` and `disabled` states as long as Archival is enabled at the Temporal Service level.

Archival is supported in [Global Namespaces](/global-namespace) (Namespaces that span multiple clusters). When Archival
is running in a Global Namespace, it first runs on the active cluster; later it runs on the standby cluster. Before
archiving, a history check is done to see what has been previously archived.

#### Test your Archival setup 

To test Archival locally, start by running a Temporal server:

```bash
./temporal-server start
```

Then register a new Namespace with Archival enabled.

```bash
./temporal operator namespace create --namespace="my-namespace" --global false --history-archival-state="enabled" --retention="4d"
```

> **📝 Note:**
>
> If the retention period isn't set, it defaults to 72h. The minimum retention period is one day. For retention maximums,
> check [Temporal Service Retention Period limits](/temporal-service/temporal-server#retention-period) for your server
> version.
>
> Setting the retention period to 0 results in the error _A valid retention period is not set on request_.
>

Next, run a sample Workflow such as the
[helloworld temporal sample](https://github.com/temporalio/samples-go/tree/main/helloworld).

When the Workflow Execution closes, Temporal schedules archival processing.

#### Retrieve archived history 

You can retrieve archived Event Histories by copying the `workflowId` and `runId` of the completed Workflow from the log
output and running the following command:

```bash
./temporal workflow show --workflow-id="my-workflow-id" --run-id="my-run-id" --namespace="my-namespace"
```

### Create a custom Archiver 

To archive data with a given provider, using the [Archival](/temporal-service/archival) feature, Temporal must have a
corresponding Archiver component installed. The platform does not limit you to the existing providers. To use a provider
that is not currently supported, you can create your own Archiver.

#### Create a new package

The first step is to create a new package for your implementation in
[/common/archiver](https://github.com/temporalio/temporal/tree/main/common/archiver). Create a directory in the archiver
folder and arrange the structure to look like the following:

```
temporal/common/archiver
  - filestore/                      -- Filestore implementation
  - provider/
      - provider.go                 -- Provider of archiver instances
  - yourImplementation/
      - historyArchiver.go          -- HistoryArchiver implementation
      - historyArchiver_test.go     -- Unit tests for HistoryArchiver
      - visibilityArchiver.go       -- VisibilityArchiver implementations
      - visibilityArchiver_test.go  -- Unit tests for VisibilityArchiver
```

#### Archiver interfaces

Next, define objects that implement the
[HistoryArchiver](https://github.com/temporalio/temporal/blob/main/common/archiver/interface.go#L80) and the
[VisibilityArchiver](https://github.com/temporalio/temporal/blob/main/common/archiver/interface.go#L121) interfaces.

The objects should live in `historyArchiver.go` and `visibilityArchiver.go`, respectively.

#### Update provider

Update the `GetHistoryArchiver` and `GetVisibilityArchiver` methods of the `archiverProvider` object in the
[/common/archiver/provider/provider.go](https://github.com/temporalio/temporal/blob/main/common/archiver/provider/provider.go)
file so that it knows how to create an instance of your archiver.

#### Add configs

Add configs for your archiver to the `config/development.yaml` file and then modify the
[HistoryArchiverProvider](https://github.com/temporalio/temporal/blob/main/common/config/config.go#L376) and
[VisibilityArchiverProvider](https://github.com/temporalio/temporal/blob/main/common/config/config.go#L393) structs in
`/common/common/config.go` accordingly.

#### Custom archiver FAQ

**If my custom Archive method can automatically be retried by the caller, how can I record and access progress between
retries?**

Handle this situation by using `ArchiverOptions`. Here is an example:

```go
func(a * Archiver) Archive(ctx context.Context, URI string, request * ArchiveRequest, opts...ArchiveOption) error {
    featureCatalog: = GetFeatureCatalog(opts...) // this function is defined in options.go
    var progress progress
    // Check if the feature for recording progress is enabled.
    if featureCatalog.ProgressManager != nil {
        if err: = featureCatalog.ProgressManager.LoadProgress(ctx, & prevProgress);
        err != nil {
            // log some error message and return error if needed.
        }
    }

    // Your archiver implementation...

    // Record current progress
    if featureCatalog.ProgressManager != nil {
        if err: = featureCatalog.ProgressManager.RecordProgress(ctx, progress);
        err != nil {
            // log some error message and return error if needed.
        }
    }
}
```

**If my `Archive` method encounters an error that is non-retryable, how do I indicate to the caller that it should not
retry?**

```go
func(a * Archiver) Archive(ctx context.Context, URI string, request * ArchiveRequest, opts...ArchiveOption) error {
    featureCatalog: = GetFeatureCatalog(opts...) // this function is defined in options.go

    err: = youArchiverImpl()

    if nonRetryableErr(err) {
        if featureCatalog.NonRetryableError != nil {
            return featureCatalog.NonRetryableError() // when the caller gets this error type back it will not retry anymore.
        }
    }
}
```

**How does my history archiver implementation read history?**

The archiver package provides a utility called
[HistoryIterator](https://github.com/temporalio/temporal/blob/main/common/archiver/history_iterator.go) which is a
wrapper of
[ExecutionManager](https://github.com/temporalio/temporal/blob/main/common/persistence/data_interfaces.go#L1014).
`HistoryIterator` is more simple than the `HistoryManager`, which is available in the BootstrapContainer, so archiver
implementations can choose to use it when reading Workflow histories. See the
[historyIterator.go](https://github.com/temporalio/temporal/blob/main/common/archiver/history_iterator.go) file for more
details. Use the
[filestore historyArchiver implementation](https://github.com/temporalio/temporal/tree/main/common/archiver/filestore)
as an example.

**Should my archiver define its own error types?**

Each archiver is free to define and return its own errors. However, many common errors that exist between archivers are
already defined in
[common/archiver/constants.go](https://github.com/temporalio/temporal/blob/main/common/archiver/constants.go).

**Is there a generic query syntax for the visibility archiver?**

Currently, no. But this is something we plan to do in the future. As for now, try to make your syntax similar to the one
used by our advanced list Workflow API.

- [s3store](https://github.com/temporalio/temporal/tree/main/common/archiver/s3store#visibility-query-syntax)
- [gcloud](https://github.com/temporalio/temporal/tree/main/common/archiver/gcloud#visibility-query-syntax)

---

# Self-hosted Temporal Service defaults

Source: https://docs.temporal.io/self-hosted-guide/defaults

> Explore the Temporal Platform defaults, limits, and configurations, including Workflow Execution, Activity, Worker, and Payload size constraints. Learn about error and warning thresholds.

> **ℹ️ Info:**
> Looking for Temporal Cloud defaults?
>
> See the [Temporal Cloud defaults and limits page](/evaluate/cloud/limits)
>

This page details many of the defaults coded into the Temporal Platform that can produce errors and warnings.
Errors are hard limits that fail when reached.
Warnings are soft limits that produce a warning log on the server side.

> **ℹ️ Info:**
>
> These limits might apply specifically to each Workflow Execution and do not pertain to the entire Temporal Platform or individual Namespaces.
>

- **Identifiers:** By default, the maximum length for identifiers (such as Workflow Id, Workflow Type, and Task Queue name) is 1000 characters.
  - This is configurable with the `limit.maxIDLength` dynamic config variable, set to 255 in [this SQL example](https://github.com/temporalio/samples-server/blob/main/compose/dynamicconfig/development-sql.yaml).
  - The character format is UTF-8.
- **gRPC:** gRPC has a limit of 4 MB for [each message received](https://github.com/grpc/grpc/blob/v1.36.2/include/grpc/impl/codegen/grpc_types.h#L466).
- **Event batch size:** The `DefaultTransactionSizeLimit` limit is [4 MB](https://github.com/temporalio/temporal/pull/1363).
  This is the largest transaction size allowed for the persistence of Event Histories.
- **Blob size limit** for Payloads (including Workflow context and each Workflow and Activity argument and return value; _[source](https://github.com/temporalio/temporal/blob/v1.32.1/common/dynamicconfig/constants.go#L386-L396)_):
  - Temporal warns at 512 KB: `Blob size exceeds limit.`
  - Temporal errors at 2 MB: `ErrBlobSizeExceedsLimit: Blob data size exceeds limit.`
  - Refer to [Troubleshoot blob size limit error](/troubleshooting/blob-size-limit-error).
- **Workflow Execution Update limits**:
  - A single Workflow Execution can have a maximum of 10 in-flight Updates and 2000 total Updates in History.
- **History total size limit** (leading to a terminated Workflow Execution):
  - Temporal warns at 10 MB: [history size exceeds warn limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1238).
  - Temporal errors at 50 MB: [history size exceeds error limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1204).
  - This is configurable with [HistorySizeLimitError and HistorySizeLimitWarn](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/configs/config.go#L380-L381).
- **History total count limit** (leading to a terminated Workflow Execution):
  - Temporal warns after 10,240 Events: [history size exceeds warn limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1238).
  - Temporal errors after 51,200 Events: [history size exceeds error limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1204).
  - This is configurable with [HistoryCountLimitError and HistoryCountLimitWarn](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/configs/config.go#L382-L383).
- **Concurrent limit**
  - The following Commands are limited:
    - `ScheduleActivityTask`
    - `SignalExternalWorkflowExecution`
    - `RequestCancelExternalWorkflowExecution`
    - `StartChildWorkflowExecution`
  - These will fail if the concurrent pending count exceeds 2,000.
    For optimal performance, limit concurrent operations to 500 or fewer.
    This reduces Workflow's Event History size and decreases the loading time in the Web UI.
  - As of v1.21, the open source Temporal Service has a default limit of 2,000 pending Activities, Child Workflows, Signals, or Workflow cancellation requests, but you can override the limits in the dynamic configuration using these variables:
    - `limit.numPendingActivities.error`
    - `limit.numPendingSignals.error`
    - `limit.numPendingCancelRequests.error`
    - `limit.numPendingChildExecutions.error`
  - By default, [Batch jobs](/cli/command-reference/batch) are limited to one job at a time.
- [Custom Search Attributes limits](/search-attribute#custom-search-attribute-limits)

For details on dynamic configuration keys, see [Dynamic configuration reference](/references/dynamic-configuration).

---

# Deploying a Temporal Service

Source: https://docs.temporal.io/self-hosted-guide/deployment

There are many ways to self-host a [Temporal Service](/temporal-service). The right way for you depends entirely on your
use case and where you plan to run it.

This page provides instructions for deploying a Temporal Service for sustained workloads that exceed what the
[development server](/cli#start-a-development-server) is designed to handle. For local development or testing, you can use the
Temporal CLI to [start a local development Temporal Service](/cli#start-a-development-server).

> **⚠️ Warning:**
> Temporal Hosts Should Not Be Exposed to the Open Internet
>
> In self-hosted deployments, the Temporal Service is a critical control and persistence component and should be secured
> similarly to a database. Temporal services should run on hosts that are not accessible from the public internet, with
> network configurations that restrict access to trusted internal networks only.
>

For step-by-step guides on deploying and configuring Temporal, refer to our
[Infrastructure tutorials](https://learn.temporal.io/tutorials/infrastructure/).

## Use Docker Compose

You can run a Temporal Service in [Docker](https://docs.docker.com/engine/install) containers using
[Docker Compose](https://docs.docker.com/compose/install).

### Prerequisites

- You have Docker Compose installed.
- Docker is running and the daemon is available.
- Git is installed and available.

### Procedure

1. Clone the [temporalio/samples-server](https://github.com/temporalio/samples-server) repository. The [docker-compose.yml example](https://github.com/temporalio/samples-server/blob/main/compose/docker-compose.yml) uses the `temporalio/server:${TEMPORAL_VERSION}` image. This is what you need to use in your production environments and you can find a thorough list of configs [here](https://github.com/temporalio/temporal/blob/main/docker/config_template.yaml). 

2. Change into the `compose` directory.

   ```
   cd samples-server/compose
   ```

3. Run the `docker compose up` command. This uses the default configuration from the `docker-compose.yaml` file, which
   includes a PostgreSQL database, an Elasticsearch instance, and exposes the Temporal gRPC Frontend on port 7233.

   ```
   docker compose up
   ```

   The Temporal Web UI will be available at `http://localhost:8080`.

4. (Optional) Review
   [the additional configuration options](https://github.com/temporalio/samples-server/tree/main/compose#other-configuration-files)
   available in the samples-server repository and use `docker compose up` with the corresponding configuration file to
   try them out. The configurations include different databases, visibility stores, and TLS settings.

When you use the `temporalio/server` image, you have to manually manage schema updates using the `temporalio/admin-tools` image or by building the tools from the matching release tag.

## Use Temporal Server binaries

You can run a complete Temporal Server by deploying two Go binaries -- the
[core Temporal Server](https://github.com/temporalio/temporal/releases/), and the
[Temporal UI Server](https://github.com/temporalio/ui-server/releases).

Each service can be deployed separately. Refer to
[How to Configure a Temporal Service without a Proxy](https://learn.temporal.io/tutorials/infrastructure/configuring-sqlite-binary/)
to deploy each service using `systemd`. If you need to run the Temporal Server behind a reverse proxy, refer to our
tutorials to deploy the Temporal Service behind an
[Nginx reverse proxy](https://learn.temporal.io/tutorials/infrastructure/nginx-sqlite-binary/) or an
[Envoy edge proxy](https://learn.temporal.io/tutorials/infrastructure/envoy-sqlite-binary/).

### Configuration templating

Configuration templating is how the Temporal Server turns a template config file into the final `config.yaml` it runs
with. It lets you reuse one template across environments by filling in values from environment variables. For example,
database endpoints, TLS paths, or feature flags.

If you are **not** using a custom config template, you can skip this section. The default configuration is rendered
automatically by the server and embedded in the binary.

#### Template compatibility

If you use a custom configuration template, be aware of the following:

- The server renders templates with embedded `sprig`, so any `dockerize`-specific syntax or helpers will fail
- Some template syntax differs, particularly `.Env` and `default` function usage.
- Refer to the [sprig documentation](http://masterminds.github.io/sprig/) for supported template functions
- Use `temporal-server render-config` to verify your templates render correctly

#### Helm chart configuration

When deploying with Helm charts versions 0.73.1 or later, you may need to adjust the following configuration options
depending on the images you are using.

| Configuration Option         | Description                                                               | Default       |
| ---------------------------- | ------------------------------------------------------------------------- | ------------- |
| `server.useEntrypointScript` | Whether to use entrypoint script that autodetects `dockerize` vs `sprig`. | `false`       |
| `server.configMapsToMount`   | Which config template to mount: `"dockerize"`, `"sprig"`, or `"both"`.    | `"dockerize"` |
| `server.setConfigFilePath`   | Set `TEMPORAL_SERVER_CONFIG_FILE_PATH` environment variable.              | `false`       |

Refer to the following guidelines to determine if you need to adjust the configuration options:

- The default settings work if you are only using pre-1.30 images with 0.73.1 or later Helm chart.
- If you are using 1.30+ images with 0.73.1 or later Helm chart, you need to set `server.configMapsToMount` to `"sprig"`
  and `server.setConfigFilePath` to `true`. Keep the `server.useEntrypointScript` as `false`.
- If you need use the Helm chart with both pre-1.30 and 1.30+ images, you need to set `server.configMapsToMount` to
  `"both"` and `server.useEntrypointScript` to `true`. Keep the `server.setConfigFilePath` as `false`.

## Import the Server package

The Temporal Server is a standalone Go application that can be [imported](/references/server-options) into another
project.

You might want to do this to pass custom plugins or any other customizations through the
[Server Options](/references/server-options). Then you can build and run a binary that contains your customizations.

This requires the minimum Go version listed in the Temporal Server
[`go.mod`](https://github.com/temporalio/temporal/blob/main/go.mod) file. See the Temporal Server
[Build prerequisites](https://github.com/temporalio/temporal/blob/main/CONTRIBUTING.md#build-prerequisites).

## Use Helm charts

[Temporal Helm charts](https://github.com/temporalio/helm-charts) enable you to get a Temporal Service running on
[Kubernetes](https://kubernetes.io/) by deploying the Temporal Server services to individual pods and connecting them to
your existing database and Elasticsearch instances.

The Temporal Helm charts repo contains
[extensive documentation](https://github.com/temporalio/helm-charts/blob/main/README.md) about Kubernetes deployments.

> **⚠️ Caution:**
> Helm Chart version compatibility
>
> If you are using Temporal Server images 1.30+, you must upgrade to Helm chart version 0.73.1 or later.
>
> Helm chart versions below 0.73.1 are **not compatible** with `server` and `admin-tools` images **version 1.30 and
> later**. You **cannot** override old chart versions with newer images.
>

---

# Embedding Temporal server as a Go library

Source: https://docs.temporal.io/self-hosted-guide/embedded-server

> Run Temporal server as an embedded Go library for testing and development. Learn how to use temporal.NewServer() to run Temporal server in-process.

You can run Temporal server as an embedded Go library instead of deploying it as a separate service.
This approach is useful for testing and development scenarios where you want to run Temporal in-process without managing external infrastructure.

> **⚠️ Caution:**
> Not for production use
>
> Embedded deployments with SQLite are suitable for **testing and development only**.
> For production workloads, deploy Temporal as a service using [MySQL, PostgreSQL, or Cassandra](/temporal-service/persistence) as the persistence layer.
>

## Reference implementation

The recommended way to run an embedded Temporal server is to use the Temporal CLI's dev server implementation as a reference.
The CLI's [devserver package](https://github.com/temporalio/cli/tree/main/internal/devserver) provides a complete implementation that handles:

- SQLite configuration and schema setup
- Namespace creation
- Service configuration
- Port allocation

You can study and adapt this implementation for your own embedded use case.

## Basic server API

The core API for embedding Temporal is `temporal.NewServer()`:

```go
import (
    "go.temporal.io/server/temporal"
    "go.temporal.io/server/common/config"
)

server, err := temporal.NewServer(
    temporal.ForServices(temporal.DefaultServices),
    temporal.WithConfig(cfg),
    temporal.InterruptOn(temporal.InterruptCh()),
)
if err != nil {
    log.Fatal(err)
}

if err := server.Start(); err != nil {
    log.Fatal(err)
}
```

The challenge is building the `config.Config` struct correctly, especially for SQLite which requires:

1. **Schema setup** - SQLite databases need schema initialization via `sqliteschema.SetupSchema()`
2. **Namespace creation** - Namespaces can be pre-created via `sqliteschema.CreateNamespaces()`
3. **Service configuration** - All four services (frontend, history, matching, worker) need proper port configuration

## Configuration from file

For non-SQLite databases, you can load configuration from a YAML file:

```go
cfg, err := config.Load(
    config.WithConfigFile("/path/to/config.yaml"),
)
if err != nil {
    log.Fatal(err)
}

server, err := temporal.NewServer(
    temporal.ForServices(temporal.DefaultServices),
    temporal.WithConfig(cfg),
)
```

Or load from a directory with environment-specific files:

```go
cfg, err := config.Load(
    config.WithConfigDir("./config"),
    config.WithEnv("development"),
)
```

## Server options reference

The `temporal.NewServer()` function accepts options to customize the server.
For details on each option below, see the [Temporal Server options reference](/references/server-options).
For every option the server accepts, see [server_option.go](https://github.com/temporalio/temporal/blob/main/temporal/server_option.go) in the Temporal Server repository.

Key options include:

| Option | Description |
|--------|-------------|
| `ForServices([]string)` | Services to run (default: frontend, history, matching, worker) |
| `WithConfig(*config.Config)` | Server configuration |
| `WithLogger(log.Logger)` | Custom logger |
| `WithAuthorizer(authorization.Authorizer)` | Custom authorization |
| `WithClaimMapper(func)` | Role/claim mapping for auth |
| `WithCustomMetricsHandler(metrics.Handler)` | Custom metrics handler |
| `WithDynamicConfigClient(dynamicconfig.Client)` | Runtime configuration |
| `InterruptOn(chan)` | Channel for graceful shutdown |

## SQLite limitations

SQLite is intended for testing and development only:

- **Single writer**: SQLite supports only one writer at a time, limiting write throughput
- **No durability in memory mode**: In-memory mode loses data on restart
- **Not scalable**: Cannot handle production workloads
- **Single shard**: Use `NumHistoryShards: 1` for SQLite

For production, use MySQL, PostgreSQL, or Cassandra with a properly scaled multi-node deployment.

## Examples

For complete working examples, see:

- [Temporal CLI dev server](https://github.com/temporalio/cli/tree/main/internal/devserver) - Reference implementation for SQLite embedding
- [samples-server repository](https://github.com/temporalio/samples-server) - Server extensibility examples:
  - [Authorizer](https://github.com/temporalio/samples-server/tree/main/extensibility/authorizer) - Custom authorization and claim mapping
  - [Metrics handler](https://github.com/temporalio/samples-server/tree/main/extensibility/metrics-handler) - Custom metrics handling
  - [TLS](https://github.com/temporalio/samples-server/tree/main/tls) - TLS configuration for secure communication
  - [Docker Compose](https://github.com/temporalio/samples-server/tree/main/compose) - Database configurations (PostgreSQL, MySQL, Cassandra)

## Related

- [Server Options Reference](/references/server-options)
- [Deployment](/self-hosted-guide/deployment)
- [Visibility Storage](/self-hosted-guide/visibility)

---

# Monitor Temporal Platform metrics

Source: https://docs.temporal.io/self-hosted-guide/monitoring

> Monitor and health check a self-hosted Temporal Platform using Prometheus and StatsD to track Temporal Service, Client, and Worker metrics for performance and issue troubleshooting.

The Temporal Service and SDKs emit metrics that can be used to monitor performance and troubleshoot issues. 
You can relay these metrics to any monitoring and observability platform.

This guide will provide an example of configuring [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) to work with the observability metrics emitted from Temporal.
This solution can work on its own, or serve as a baseline for you to further customize and integrate with other observability tooling.
For example, it is also possible to use the [OpenTelemetry Collector](https://temporal.io/code-exchange/temporal-opentelemetry) in your stack instead of scraping metrics directly with Prometheus, or [Datadog](#datadog) as a frontend instead of Grafana.

This configuration assumes that you have [Docker](https://www.docker.com/) installed and are running a [Temporal dev server](https://temporal.io/setup/start-development-server) via the CLI. 

## Prometheus

This section discusses exporting metrics from Temporal SDKs, and setting up Prometheus to collect metrics on Temporal Service, Temporal Client, and Temporal Worker performance.

The Temporal Service and SDKs emit all metrics by default.
However, you must enable Prometheus in your application code (using the Temporal SDKs) and your Temporal Service configuration to collect the metrics emitted from your SDK and Temporal Service.

First, you'll need to create a `prometheus.yml` configuration file with some target ports to collect metrics from.
Here is a sample with one Temporal Service metrics target and two Temporal Worker (SDK) metrics targets:

```
global:
 scrape_interval: 10s
scrape_configs:
 - job_name: 'temporalmetrics'
   metrics_path: /metrics
   scheme: http
   static_configs:
     # Temporal Service metrics target
     - targets:
         - 'host.docker.internal:8000'
       labels:
         group: 'server-metrics'

     # Local app targets (set in SDK code)
     - targets:
         - 'host.docker.internal:8077'
         - 'host.docker.internal:8078'
       labels:
         group: 'sdk-metrics'
```

In this example, Prometheus is configured to scrape at 10-second intervals and to listen for Temporal Service metrics on `host.docker.internal:8000` and SDK metrics on two targets, `host.docker.internal:8077` and `host.docker.internal:8078`.
The `8077` and `8078` ports must be set on `WorkflowServiceStubs` in your application code with your preferred SDK -- there is an example of this in the next section.
You can set up as many targets as required.

> **ℹ️ Info:**
>
> For further Prometheus configuration options, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/).
>

You can use Docker to run the official Prometheus image with this configuration:

```bash
docker run -p 9090:9090 -v /path/to/prometheus.yml /etc/prometheus/prometheus.yml prom/prometheus
```

Next, launch your Temporal dev server from the CLI with an additional `--metrics-port 8000` parameter:

```bash
temporal server start-dev --metrics-port 8000
```

> **ℹ️ Info:**
>
> Refer to the [Temporal Service configuration reference](/references/service-configuration#global) to expose metrics from a production service.
>

You should now have both Prometheus and a Temporal Service running locally, with Temporal providing Service metrics to Prometheus.
Next, you'll want to configure SDK metrics as well.

### SDK metrics setup

SDK metrics are emitted by Temporal Workers and other Clients, and must be configured in your application code.
The Metrics section in the Observability guide details how to create hooks for all supported SDKs:

- [Go](/develop/go/platform/observability#metrics)
- [Java](/develop/java/platform/observability#metrics)
- [PHP](/develop/php/platform/observability)
- [Python](/develop/python/platform/observability#metrics)
- [TypeScript](/develop/typescript/platform/observability#metrics)
- [.NET](/develop/dotnet/platform/observability#metrics)
- [Ruby](/develop/ruby/platform/observability#metrics)

For end-to-end examples of how to expose metrics from each SDK, see the metrics samples:

- [Go SDK Sample](https://github.com/temporalio/samples-go/tree/main/metrics)
- [Java SDK Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics)
- [Python SDK Sample](https://github.com/temporalio/samples-python/tree/main/prometheus)
- [TypeScript SDK Sample](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry)
- [.NET SDK Sample](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry)

Some of these may require you to set different metrics port numbers based on the Prometheus example here, which is configured to scrape port `8077` and `8078` by default.
Follow the instructions from each of the samples to run Workflows and begin emitting metrics.
This will allow you to populate a dashboard in the next section and understand how to further customize Temporal observability for your needs.

### Verifying Prometheus configuration

Once your Workflows are running and emitting metrics, you can visit [http://localhost:9090/targets](http://localhost:9090/targets) on your local Prometheus instance to verify that it is able to scrape the provided endpoints.

![Prometheus scrape targets](https://docs.temporal.io/img/observability/prometheus-targets.png)

This example shows a response from the server metrics endpoint, provided by the Temporal dev server, and two SDK metrics endpoints, as defined in the Prometheus configuration.
To create this example, we used the Go and Python metrics samples, running on port 8077 and 8088 respectively.
If you are not pushing data to exactly 3 metrics endpoints, your environment may be different.

Next, you can visit the [local Prometheus query endpoint](http://localhost:9090/query) to manually run [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) queries on your exported metrics, or proceed to the next section to configure Grafana to generate dashboards from those metrics.

## Grafana

With [Prometheus](#prometheus) configured, deploy Grafana as a metrics frontend, and configure it to use Prometheus as a data source.

As before, you can use Docker to run the official Grafana image:

```bash
docker run -d -p 3000:3000 grafana/grafana-enterprise
```

This will deploy a Grafana instance with a default username and password of `admin`/`admin`.
In production, you would want to [configure authentication](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/) and control port access to Grafana.

> **ℹ️ Info:**
>
> For more information on how to customize your Grafana setup, see the [Grafana documentation](https://grafana.com/docs/grafana/latest/setup-grafana/).
>

Next, configure Grafana to use Prometheus as the data source.
To do this, click on "Add new data source" from the "Connections" menu in the Grafana sidebar, and add Prometheus from the list.

You will be prompted to add additional configuration parameters.
If you are following this guide using Docker, use `http://host.docker.internal:9090` as the Prometheus address.
This is a [DNS name provided by Docker Desktop](https://docs.docker.com/desktop/features/networking/#use-cases-and-workarounds) which resolves to the internal IP address used by the host machine, and allows you to connect applications across Docker containers without additional configuration rules.
This is the only parameter you will need to set for your Prometheus configuration.
After providing it, scroll down to the "Save and Test" button, and you can validate Prometheus as a data source for this Grafana instance.

![Grafana data sources](https://docs.temporal.io/img/observability/grafana-data-sources.png)

In this example, Grafana is set to pull metrics from Prometheus at the port 9090, as defined in the Prometheus configuration.

Now, you'll just need to add some of our provided dashboards for visualizing Temporal metrics.

### Dashboard setup

We provide community-driven Grafana dashboards that can be used for monitoring Temporal Server and SDK metrics in a [dashboards](https://github.com/temporalio/dashboards/) repo.
Follow the instructions in that repo's README to import the dashboards to Grafana.

This way, you can create at least one dashboard for monitoring server metrics:

![Grafana server metrics](https://docs.temporal.io/img/observability/grafana-server-metrics.png)

And at least one other dashboard for monitoring SDK metrics:

![Grafana SDK metrics](https://docs.temporal.io/img/observability/grafana-sdk-metrics.png)

> **ℹ️ Info:**
>
> You can provide additional queries in your dashboard to report other data as needed.
> For more details on configuring Grafana dashboards, see the [Grafana Dashboards documentation](https://grafana.com/docs/grafana/latest/dashboards/).
>

From here, you can configure Grafana [Alerts](https://grafana.com/docs/grafana/latest/alerting/) for any monitored parameters, add custom metrics to your Temporal SDK code, and use these observability features to help scale your Temporal deployment.
Refer to the [Temporal Service metrics](/references/service-metrics) and [SDK metrics](/references/sdk-metrics) reference for more.

## Configuring Temporal Service health checks 

The [Frontend Service](/temporal-service/temporal-server#frontend-service) supports TCP or [gRPC](https://github.com/grpc/grpc/blob/ee389ed222de898ae419c8c5ef8c213573ce5c55/src/proto/grpc/health/v1/health.proto#L45) health checks on port 7233.

If you use [Nomad](https://www.nomadproject.io/) to manage your containers, the [check stanza](https://developer.hashicorp.com/nomad/docs/job-specification/check) would look like this for TCP:

```
service {
  check {
    type     = "tcp"
    port     = 7233
    interval = "10s"
    timeout  = "2s"
  }
```

or like this for gRPC (requires Consul ≥ `1.0.5`):

```
service {
  check {
    type         = "grpc"
    port         = 7233
    interval     = "10s"
    timeout      = "2s"
  }
```

## Installing via Helm chart

If you are installing and running Temporal via [Helm chart](https://github.com/temporalio/helm-charts), you can also [provide additional parameters](https://github.com/temporalio/helm-charts?tab=readme-ov-file#exploring-metrics-via-grafana) to populate and explore a Grafana dashboard out of the box.

## Datadog 

Datadog has a Temporal integration for collecting Temporal Service metrics.
Once you've [configured Prometheus](#prometheus), you can configure the [Datadog Agent](https://docs.datadoghq.com/integrations/temporal/).

If you are using [Temporal Cloud](/evaluate/cloud), you can also [integrate Datadog directly](https://docs.datadoghq.com/integrations/temporal-cloud/), without needing to use Prometheus.

---

# Self-hosted Multi-Cluster Replication

Source: https://docs.temporal.io/self-hosted-guide/multi-cluster-replication

> Multi-Cluster Replication in Temporal ensures asynchronous replication of Workflow Executions from active to passive Clusters for backup and state reconstruction, so you can fail over to a passive Cluster.

Multi-Cluster Replication is a feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters, for backup and state reconstruction.
When necessary, for higher availability, Cluster operators can failover to any of the backup Clusters.

Temporal's Multi-Cluster Replication feature is considered **experimental** and not subject to normal [versioning and support policy](/temporal-service/temporal-server#versions-and-support).

When a client connects to a standby Cluster, its requests can be forwarded to the
active Cluster. This is off by default; enable it by setting
[`dcRedirectionPolicy`](/references/service-configuration#dcredirectionpolicy).

When the feature is enabled, Tasks are sent to the Parent Task Queue partition that matches that Namespace, if it exists.

All Visibility APIs can be used against active and standby Clusters.
As a result, [Temporal UI](/web-ui) works with Global Namespaces the same way it works with any other Namespace.
Applications making API calls directly to the Temporal Visibility API continue to work even if a Global Namespace is in standby mode.
However, they might see a lag due to replication delay when querying the Workflow Execution state from a standby Cluster.

## How to fail over a Global Namespace

For a planned failover, run the built-in `namespace-handover` Workflow against the currently active Temporal Service.
The handover Workflow:

1. Waits for the destination Temporal Service to approach the configured replication-lag thresholds.
2. Puts the Namespace into the `handover` replication state, which pauses Workflow mutations and task processing.
3. Waits for the destination Temporal Service to acknowledge every replication task created before the pause.
4. Makes the destination Temporal Service active and restores the Namespace to the `normal` replication state.

This process creates a brief availability gap while traffic is paused, but it prevents Workflow progress from rolling
back at the cutover. If the destination cannot catch up during the handover timeout, the Workflow restores the `normal`
replication state without changing the active Temporal Service.

Connect Temporal CLI to the currently active Temporal Service, then run the handover Workflow in the `temporal-system`
Namespace. Use a unique Workflow Id for each handover:

```bash
temporal workflow execute \
  --address ActiveClusterFrontendAddress \
  --namespace temporal-system \
  --task-queue default-worker-tq \
  --workflow-id YourHandoverWorkflowId \
  --type namespace-handover \
  --input '{"Namespace":"YourNamespaceName","RemoteCluster":"NewActiveCluster","AllowedLaggingSeconds":10,"AllowedLaggingTasks":0,"HandoverTimeoutSeconds":10}'
```

`RemoteCluster` is the configured name of the destination Temporal Service. `AllowedLaggingSeconds` and
`AllowedLaggingTasks` control when the Workflow can begin pausing traffic. `HandoverTimeoutSeconds` controls how long
it waits for complete replication after the pause and is capped at 30 seconds.

### Force a failover when the active Temporal Service is unavailable

Use a forced failover only when the active Temporal Service cannot complete a handover. Connect Temporal CLI to the
destination Temporal Service and change the active Temporal Service directly:

```bash
temporal operator namespace update \
  --address NewActiveClusterFrontendAddress \
  --namespace YourNamespaceName \
  --active-cluster NewActiveCluster
```

Namespace metadata is stored separately in each Temporal Service and replicated asynchronously. The Temporal Service
that receives the update applies it locally before the other Temporal Services receive it:

- If the destination Temporal Service receives the update, both Temporal Services can temporarily consider the
  Namespace active.
- If the source Temporal Service receives the update, both Temporal Services can temporarily consider the Namespace
  passive.

For this reason, send an emergency failover to the destination when the source is unavailable.

The Namespace cache refresh interval does not add another data-consistency window after a Temporal Service observes the
failover. Tasks record the failover version of the Workflow state that created them. Task processing compares that
version and the Event Id with the current mutable state and rejects a task from a superseded history branch, even if a
local Namespace cache entry is stale.

This task-version check does not make a forced failover lossless. Before the failover metadata reaches the former active
Temporal Service, both Temporal Services can create different history branches. [Conflict
resolution](#conflict-resolution) selects the branch with the highest failover version, which can roll back Workflow
progress that had not replicated.

### Prevent duplicate Activity effects

A forced failover can cause an Activity to execute again. For example, an Activity can complete in the former active
Temporal Service, but its completion Event might not reach the destination before conflict resolution selects the
destination's history branch. The new active Temporal Service then dispatches the Activity again.

Design Activities that create external side effects to be
[idempotent](/best-practices/error-handling#idempotence). Use an idempotency key, a transactional version check, or a
conditional database update so that repeated executions produce one effect. Setting an Activity Retry Policy to one
attempt does not provide exactly-once execution because a Worker can finish the side effect without recording the
completion Event.

Stopping Workers that poll the source Temporal Service is not a correctness mechanism. It can reduce new Activity
starts when the source is reachable, but it cannot cancel side effects that have already started or prevent an Activity
from being dispatched again after history rollback. It also increases the availability gap. Use the handover Workflow
for planned failovers and idempotent Activities for both planned and forced failovers.

#### Namespace versions

A _version_ is a concept in Multi-Cluster Replication that describes the chronological order of events per Namespace.

With Multi-Cluster Replication, all Namespace change events and Workflow Execution History events are replicated asynchronously for high throughput.
This means that data across clusters is **not** strongly consistent.
To guarantee that Namespace data and Workflow Execution data will achieve eventual consistency (especially when there is a data conflict during a failover), a **version** is introduced and attached to Namespaces.
All Workflow Execution History entries generated in a Namespace will also come with the version attached to that Namespace.

All participating Clusters are pre-configured with a unique initial version and a shared version increment:

- `initial version < shared version increment`

When performing failover for a Namespace from one Cluster to another Cluster, the version attached to the Namespace will be changed by the following rule:

- for all versions which follow `version % (shared version increment) == (active cluster's initial version)`, find the smallest version which has `version >= old version in namespace`

When there is a data conflict, a comparison will be made and Workflow Execution History entries with the highest version will be considered the source of truth.

When a cluster is trying to mutate a Workflow Execution History, the version will be checked.
A cluster can mutate a Workflow Execution History only if the following is true:

- The version in the Namespace belongs to this cluster, that is
  `(version in namespace) % (shared version increment) == (this cluster's initial version)`
- The version of this Workflow Execution History's last entry (event) is equal or less than the version in the Namespace, that is
  `(last event's version) <= (version in namespace)`

    Namespace version change example

Assuming the following scenario:

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Shared version increment: 10

T = 0: Namespace α is registered, with active Cluster set to Cluster A

```
namespace α's version is 1
all workflows events generated within this namespace, will come with version 1
```

T = 1: namespace β is registered, with active Cluster set to Cluster B

```
namespace β's version is 2
all workflows events generated within this namespace, will come with version 2
```

T = 2: Namespace α is updated to with active Cluster set to Cluster B

```
namespace α's version is 2
all workflows events generated within this namespace, will come with version 2
```

T = 3: Namespace β is updated to with active Cluster set to Cluster A

```
namespace β's version is 11
all workflows events generated within this namespace, will come with version 11
```

#### Version history

Version history is a concept which provides a high level summary of version information in regards to Workflow Execution History.

Whenever there is a new Workflow Execution History entry generated, the version from Namespace will be attached.
The Workflow Execution's mutable state will keep track of all history entries (events) and the corresponding version.

        Version history example (without data conflict)

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Shared version increment: 10

T = 0: adding event with event ID == 1 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 1               | 1       |
| -------- | -------------   | --------------- | ------- |
```

T = 1: adding event with event ID == 2 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 2: adding event with event ID == 3 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               |                 |         |
| 3        | 1               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 3: Namespace failover triggered, Namespace version is now 2
adding event with event ID == 4 & version == 2

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               | 4               | 2       |
| 3        | 1               |                 |         |
| 4        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 4: adding event with event ID == 5 & version == 2

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               | 5               | 2       |
| 3        | 1               |                 |         |
| 4        | 2               |                 |         |
| 5        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

Since Temporal is AP, during failover (change of active Temporal Cluster Namespace), there can exist cases where more than one Cluster can modify a Workflow Execution, causing divergence of Workflow Execution History. Below shows how the version history will look like under such conditions.

    Version history example (with data conflict)

Below, shows version history of the same Workflow Execution in 2 different Clusters.

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Cluster C comes with initial version: 3
- Shared version increment: 10

T = 0:

View in both Cluster B & C

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 3               | 2       |
| 3        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 1: adding event with event ID == 4 & version == 2 in Cluster B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 4               | 2       |
| 3        | 2               |                 |         |
| 4        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 1: namespace failover to Cluster C, adding event with event ID == 4 & version == 3 in Cluster C

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 3               | 2       |
| 3        | 2               | 4               | 3       |
| 4        | 3               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 2: replication task from Cluster C arrives in Cluster B

Note: below is a tree structure

```
| -------------- | ------------- |
| Events         |               |
| -------------- | ------------- |
| Event ID       | Event Version |
| -------------  | ------------- |
| 1              | 1             |
| 2              | 1             |
| 3              | 2             |
| -------------  | ------------- |
|                |               |
| -------------  | ------------- |
|                |               |
| -------------- | ------------- |  | -------- | ------------- |
| Event ID       | Event Version |  | Event ID | Event Version |
| -------------  | ------------- |  | -------- | ------------- |
| 4              | 2             |  | 4        | 3             |
| -------------- | ------------- |  | -------- | ------------- |

| --------------- | ----------- |
| Version History |              |
| --------------- | ------------ |
| Event ID        | Version      |
| --------------- | ------------ |
| 2               | 1            |
| 3               | 2            |
| --------------- | ------------ |

| -------- | ------- |  | -------- | ------- |
| Event ID | Version |  | Event ID | Version |
| -------- | ------- |  | -------- | ------- |
| 4        | 2       |  | 4        | 3       |
| -------- | ------- |  | -------- | ------- |
```

T = 2: replication task from Cluster B arrives in Cluster C, same as above

#### Conflict resolution

When a Workflow Execution History diverges, proper conflict resolution is applied.

In Multi-cluster Replication, Workflow Execution History Events are modeled as a tree, as shown in the second example in [Version History](#version-history).

Workflow Execution Histories that diverge will have more than one history branch.
Among all history branches, the history branch with the highest version is considered the `current branch` and the Workflow Execution's mutable state is a summary of the current branch.
Whenever there is a switch between Workflow Execution History branches, a complete rebuild of the Workflow Execution's mutable state will occur.

Temporal Multi-Cluster Replication relies on asynchronous replication of Events across Clusters, so in the case of a failover it is possible to have an Activity Task dispatched again to the newly active Cluster due to a replication task lag.
This also means that whenever a Workflow Execution is updated after a failover by the new Cluster, any previous replication tasks for that Execution cannot be applied.
This results in loss of some progress made by the Workflow Execution in the previous active Cluster.
During such conflict resolution, Temporal re-injects any external Events like Signals in the new Event History before discarding replication tasks.
Even though some progress could roll back during failovers, Temporal provides the guarantee that Workflow Executions won't get stuck and will continue to make forward progress.

Activity Execution completions are not forwarded across Clusters.
Any outstanding Activities will eventually time out based on the configuration.
Your application should have retry logic in place so that the Activity gets retried and dispatched again to a Worker after the failover to the new Cluster.
Handling this is similar to handling an Activity Task timeout caused by a Worker restarting.

#### Zombie Workflows

There is an existing contract that for any Namespace and Workflow Id combination, there can be at most one run (Namespace + Workflow Id + Run Id) open / executing.

Multi-cluster Replication aims to keep the Workflow Execution History as up-to-date as possible among all participating Clusters.

Due to the nature of Multi-cluster Replication (for example, Workflow Execution History events are replicated asynchronously) different Runs (same Namespace and Workflow Id) can arrive at the target Cluster at different times, sometimes out of order, as shown below:

```
+-----------+        +----------------+        +-----------+
| Cluster A |        | Network Layer  |        | Cluster B |
+-----------+        +----------------+        +-----------+
      |                       |                        |
      | Run 1 Replication     |                        |
      |---------------------> |                        |
      |                       |                        |
      | Run 2 Replication     |                        |
      |---------------------> |                        |
      |                       |                        |
      |                       | Run 2 Replication      |
      |                       |--------------------->  |
      |                       |                        |
      |                       | Run 1 Replication      |
      |                       |--------------------->  |
      |                       |                        |
```

Because Run 2 appears in Cluster B first, Run 1 cannot be replicated as "runnable" due to the rule `at most one Run open` (see above), thus the "zombie" Workflow Execution state is introduced.
A "zombie" state is one in which a Workflow Execution which cannot be actively mutated by a Cluster (assuming the corresponding Namespace is active in this Cluster). A zombie Workflow Execution can only be changed by a replication Task.

Run 1 will be replicated similar to Run 2, except when Run 1's execution will become a "zombie" before Run 1 reaches completion.

#### Workflow Task processing

In the context of Multi-cluster Replication, a Workflow Execution's mutable state is an entity which tracks all pending tasks.
Prior to the introduction of Multi-cluster Replication, Workflow Execution History entries (events) are from a single branch, and the Temporal Server will only append new entries (events) to the Workflow Execution History.

After the introduction of Multi-cluster Replication, it is possible that a Workflow Execution can have multiple Workflow Execution History branches.
Tasks generated according to one history branch may become invalidated by switching history branches during conflict resolution.

Example:

T = 0: task A is generated according to Event Id: 4, version: 2

```
| -------- | ------------- |
| Events   |
| -------- | ------------- |
| Event ID | Event Version |
| -------- | ------------- |
| 1        | 1             |
| 2        | 1             |
| 3        | 2             |
| -------- | ------------- |
|          |
|          |
| -------- | ------------- |
| Event ID | Event Version |
| -------- | ------------- |
| 4        | 2             | <-- task A belongs to this event
| -------- | ------------- |
```

T = 1: conflict resolution happens, Workflow Execution's mutable state is rebuilt and history Event Id: 4, version: 3 is written down to persistence

```
| --------- | ------------- |
| Events    |
| --------- | ------------- |
| Event ID  | Event Version |
| --------  | ------------- |
| 1         | 1             |
| 2         | 1             |
| 3         | 2             |
| --------  | ------------- |
|           |
| --------- | ------------- |
|           |                
| --------  | ------------- |                                  | -------- | ------------- |
| Event ID  | Event Version |                                  | Event ID | Event Version |
| --------  | ------------- |                                  | -------- | ------------- |
| 4         | 2             | <-- task A belongs to this event | 4        | 3             | <-- current branch / mutable state
| --------  | ------------- |                                  | -------- | ------------- |
```

T = 2: task A is loaded.

At this time, due to the rebuild of a Workflow Execution's mutable state (conflict resolution), Task A is no longer relevant (Task A's corresponding Event belongs to non-current branch).
Task processing logic will verify both the Event Id and version of the Task against a corresponding Workflow Execution's mutable state, then discard task A.

## How to set up Multi-Cluster Replication 

The [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) feature asynchronously replicates Workflow Execution Event Histories from active Clusters to other passive Clusters, and can be enabled by setting the appropriate values in the `clusterMetadata` section of your configuration file.

1. `enableGlobalNamespace` must be set to `true`.
2. `failoverVersionIncrement` has to be equal across connected Clusters.
3. `initialFailoverVersion` in each Cluster has to assign a different value.
   No equal value is allowed across connected Clusters.

After the above conditions are satisfied, you can start to configure a multi-cluster setup.

#### Set up Multi-Cluster Replication prior to v1.14

You can set this up with [`clusterMetadata` configuration](/references/service-configuration#clustermetadata); however, this is meant to be only a conceptual guide rather than a detailed tutorial.

> **💡 Tip:**
>
> If you need help when setting up, please reach out to our [community Slack](https://temporalio.slack.com).
> Good places to start include the **\#support-community** channel, searching through previous conversations, and asking our unusually excellent Temporal-trained large language model (visit **\#ask-ai**).
>
> Need a Slack invitation?
> Here's an [invitation link](https://temporal.io/slack).
>

For example:

```yaml
# cluster A
clusterMetadata:
  enableGlobalNamespace: true
  failoverVersionIncrement: 100
  masterClusterName: "clusterA"
  currentClusterName: "clusterA"
  clusterInformation:
    clusterA:
      enabled: true
      initialFailoverVersion: 1
      rpcAddress: "127.0.0.1:7233"
    clusterB:
      enabled: true
      initialFailoverVersion: 2
      rpcAddress: "127.0.0.1:8233"

# cluster B
clusterMetadata:
  enableGlobalNamespace: true
  failoverVersionIncrement: 100
  masterClusterName: "clusterA"
  currentClusterName: "clusterB"
  clusterInformation:
    clusterA:
      enabled: true
      initialFailoverVersion: 1
      rpcAddress: "127.0.0.1:7233"
    clusterB:
      enabled: true
      initialFailoverVersion: 2
      rpcAddress: "127.0.0.1:8233"
```

#### Set up Multi-Cluster Replication in v1.14 and later

You still need to set up local cluster [`clusterMetadata` configuration](/references/service-configuration#clustermetadata)

For example:

```yaml
# cluster A
clusterMetadata:
  enableGlobalNamespace: true
  failoverVersionIncrement: 100
  masterClusterName: "clusterA"
  currentClusterName: "clusterA"
  clusterInformation:
    clusterA:
      enabled: true
      initialFailoverVersion: 1
      rpcAddress: "127.0.0.1:7233"

# cluster B
clusterMetadata:
  enableGlobalNamespace: true
  failoverVersionIncrement: 100
  masterClusterName: "clusterB"
  currentClusterName: "clusterB"
  clusterInformation:
    clusterB:
      enabled: true
      initialFailoverVersion: 2
      rpcAddress: "127.0.0.1:8233"
```

Then you can use the Temporal CLI tool to add cluster connections. All operations should be executed in both Clusters.

```shell
# Add a cluster
temporal operator cluster upsert --frontend_address="127.0.2.1:8233"

# Disable connections
temporal operator cluster upsert --frontend_address="localhost:8233" --enable_connection false

# Delete connections
temporal operator cluster remove --name="someClusterName"
```

---

# Managing Namespaces

Source: https://docs.temporal.io/self-hosted-guide/namespaces

> **ℹ️ Info:**
> Open source Temporal
>
> This page covers namespace operations for **open source Temporal**. For core namespace concepts, see
> [Temporal Namespace](/namespaces). For Temporal Cloud, see [Temporal Cloud Namespaces](/cloud/namespaces).
>

A [Namespace](/namespaces) is a unit of isolation within the Temporal Platform. Before you can run Workflows, you must
register at least one Namespace with your Temporal Service.

## Create a Namespace

Registering a Namespace creates it on the Temporal Service. When you register a Namespace, you must set a
[Retention Period](/temporal-service/temporal-server#retention-period) that determines how long closed Workflow
execution history is kept.

You can create Namespaces using:

- **Temporal CLI** (recommended): [`temporal operator namespace create`](/cli/command-reference/operator#create)
- **Go SDK**: [`RegisterNamespace`](/develop/go/client/namespaces#register-namespace)
- **Java SDK**: [`RegisterNamespace`](/develop/java/client/namespaces#register-namespace)
- **TypeScript SDK**: [Namespace management](/develop/typescript/client/namespaces#register-namespace)

### The default Namespace

If no Namespace is specified, SDKs and CLI use the `default` Namespace. You must register this Namespace before using
it.

For local development, the [`temporal server start-dev`](/cli/command-reference/server#start-dev) command automatically creates the
`default` Namespace.

For all other deployment methods, create the `default` Namespace manually using the Temporal CLI:

```bash
temporal operator namespace create --namespace default
```

Namespace registration takes up to 15 seconds to complete. Wait for this process to finish before making calls to the
Namespace.

## Manage Namespaces

Common namespace management operations:

| Operation | CLI Command                                                                        | Description                         |
| --------- | ---------------------------------------------------------------------------------- | ----------------------------------- |
| List      | [`temporal operator namespace list`](/cli/command-reference/operator#list)         | List all registered Namespaces      |
| Describe  | [`temporal operator namespace describe`](/cli/command-reference/operator#describe) | Get details for a Namespace         |
| Update    | [`temporal operator namespace update`](/cli/command-reference/operator#update)     | Update Namespace configuration      |
| Delete    | [`temporal operator namespace delete`](/cli/command-reference/operator#delete)     | Delete a Namespace and all its data |

For SDK-based namespace management:

- [Go SDK namespace management](/develop/go/client/namespaces#manage-namespaces)
- [Java SDK namespace management](/develop/java/client/namespaces#manage-namespaces)
- [TypeScript SDK namespace management](/develop/typescript/client/namespaces#manage-namespaces)

### Deprecate vs Delete

- **Deprecate**: Prevents new Workflow Executions from starting, but existing Workflows continue to run.
- **Delete**: Removes the Namespace and all Workflow Executions. This is irreversible.

## Security

Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service to control who can
create, update, or deprecate Namespaces.

Without an Authorizer configured, Temporal uses the `nopAuthority` authorizer that allows all API calls unconditionally.

For Temporal Cloud, [role-based access controls](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) provide namespace-level authorization without custom configuration.

## Best practices

For namespace naming conventions, organizational patterns, and production safeguards, see
[Namespace Best Practices](/best-practices/managing-namespace).

---

# Temporal Platform's production readiness checklist

Source: https://docs.temporal.io/self-hosted-guide/production-checklist

> Prepare a self-hosted Temporal Service for production with guidance on scaling, metrics, load testing, and Workflow versioning.

This page covers common challenges you'll face when self-hosting Temporal for production, along with recommendations and key tests to address them.

> **📝 Note:**
>
> This guide covers self-hosted deployments. If you're weighing self-hosting against a managed offering, see [Temporal Cloud versus self-hosted Temporal](/evaluate/features/cloud-vs-self-hosted-features).
>

## Self-hosting challenge areas

Resolving these challenges takes significant engineering and ongoing effort:

- Scalability with spiky or growing workloads
- Global hosting
- Uptime, availability and reliability
- Management and control plane
- Latency, which you must keep low and consistent
- [Security](/self-hosted-guide/security)
- Maintenance and upgrades
- Expert support to users of the service
- Cost management

Each of these components is an essential part of building a mission critical Temporal Service.
Without demonstrated architectural durability, Temporal's [Durable Execution](https://temporal.io/how-it-works) model loses its value.

## Scalability with variable or growing workloads 

Workloads can be highly variable, and you may experience sustained workload spikes.
Temporal recommends scaling your Temporal Service well above the average throughput.
See [Scaling Temporal: The Basics](https://temporal.io/blog/scaling-temporal-the-basics) for an introduction to the topic.

Temporal server throughput is often limited by the number of [Shards](/temporal-service/temporal-server#history-shard) configured for the Temporal Service.
A Shard is a unit within a Temporal Service that scales concurrent Workflow Execution throughput.
You set Shard capacity, and often overall Temporal Service throughput, at build time and can't adjust it later.
Adding more Shards if needed requires a rebuild and a migration to the new Temporal Service.

The requirements of your Temporal Service will vary widely based on your intended production workload.
You will want to run your own proof of concept tests and watch for key metrics to understand the system health and scaling needs.

**Load testing.** You can use [the Omes benchmarking tool](https://github.com/temporalio/omes/), see how we [stress test Temporal](https://temporal.io/blog/temporal-deep-dive-stress-testing/), or write your own.

Temporal's source [lists](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) all metrics the server emits.
There are also equivalent metrics that you can configure from the client side.
At a high level, you will want to track these 3 categories of metrics:

- **Service metrics**: For each request made by the service handler we emit `service_requests`, `service_errors`, and `service_latency` metrics with `type`, `operation`, and `namespace` tags.
  This gives you basic visibility into service usage and allows you to look at request rates across services, namespaces and even operations.
- **Persistence metrics**: The Server emits `persistence_requests`, `persistence_errors` and `persistence_latency` metrics for each persistence operation.
  These metrics include the `operation` tag such that you can get the request rates, error rates or latencies per operation.
  These are super useful in identifying issues caused by the database.
- **Workflow Execution stats**: The Server also emits counters for when Workflow Executions are complete.
  These are useful in getting overall stats about Workflow Execution completions.
  Use `workflow_success`, `workflow_failed`, `workflow_timeout`, `workflow_terminate` and `workflow_cancel` counters for each type of Workflow Execution completion.
  These include the `namespace` tag.

## Availability

A high level of availability and reliability (99.99%) is a requirement for mission critical deployments. Temporal recommends testing for this availability level while load testing. We also recommend validating this level of reliability while doing server upgrades, to ensure no loss of service availability.

You can deploy Temporal Services in as many regions as you need to meet various requirements:

- Data Residency
- Latency
- Security / Isolation

## Management and control plane

As your use of Temporal increases, you may go from having one or two production use cases in a single region to many use cases in many regions.
Running multiple Temporal Services is complex work, as each needs its own setup, tuning, and configuration.

Needing to monitor and manage all your Temporal Services in a unified way leads to operational management pain.
Consider adding a layer on top of Temporal to manage multiple Temporal Services: a control plane.
A control plane manages and directs data flow, and decides where to send data packets.
Since Temporal does not ship its own open source control plane, building, and maintaining one is your responsibility.

Self-hosted Temporal doesn't support role-based access control (RBAC) or audit logging out of the box. If your organization requires them, plan to build RBAC, audit logging, and related access controls into your control plane.

## Maintenance and upgrades

Temporal recommends keeping up-to-date and not falling behind on your server versions.

Temporal Server is proactively updated, and releases as often as every two weeks.
Temporal recommends [upgrading sequentially](/self-hosted-guide/upgrade-server), not skipping any minor versions, although you can skip patch versions.
Temporal does not guarantee support for Temporal Server, but old servers will be hard for even the community to support, so we encourage you to keep up to date.
You must create and maintain the infrastructure to host and run your self-hosted Temporal installation, such as Kubernetes, and data stores for persistence.

Server upgrades can negatively affect self-hosted Temporal Service availability.
Temporal recommends load and availability testing during the upgrade process to understand the performance implications.

## Expert support

Temporal recommends that customer platform teams who are building out a Temporal service gain deep experience across the lifecycle and breadth of a Temporal Application.

Specific activities include:

- [Worker tuning](/develop/worker-performance)
- [Worker best practices](/workers)
- Code reviews
- Design guidance
- Training
- Security reviews
- [Metrics](/references/sdk-metrics) and monitoring
- Technical onboarding

## Cost management

Running a mission critical, global Temporal Service can be expensive.
Temporal Server is a complex system to run and scale.
Temporal recommends performance testing and planning scaling as your performance requirements evolve.
Following our guidance can oversize your self-hosted Temporal Server installs, but this is necessary to handle unpredictable spiky workloads.
Performance testing can help you right-size your environments.
Running a mission-critical Temporal Service requires [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) for high availability and global coverage.

It's a good practice to have trained, experienced administrators familiar with Temporal Service architecture to maintain your Temporal servers and provide a mission critical service.
Staffing, training, and skill development can be significant costs to maintaining a Temporal Service.

---

# Temporal Platform security features

Source: https://docs.temporal.io/self-hosted-guide/security

> Discover comprehensive security features of the Temporal Platform, including secure network communication with TLS and mTLS, robust authentication, customizable authorization, and single sign-on integration to protect your data and operations.

> **ℹ️ Info:**
> General company security
>
> For information about the general security habits of Temporal Technologies, see our [trust page](https://trust.temporal.io).
>

> **ℹ️ Info:**
> Cloud security
>
> For information about Temporal Cloud security features, see our [Cloud security page](/evaluate/cloud/security)

The Temporal Platform is designed with security in mind, and there are many features that you can use to keep both the Platform itself and your user's data secure.

A secured Temporal Server has its network communication encrypted and has authentication and authorization protocols set up for API calls made to it.
Without these, your server could be accessed by unwanted entities.

What is documented on this page are the built-in opt-in security measures that come with Temporal.
However users may also choose to design their own security architecture with reverse proxies or run unsecured instances inside of a VPC environment.

### Server samples

The https://github.com/temporalio/samples-server repo offers two examples, which are further explained below:

- **TLS:** how to configure Transport Layer Security (TLS) to secure network communication with and within a Temporal Service.
- **Authorizer:** how to inject a low-level authorizer component that can control access to all API calls.

### Encryption in transit with mTLS

Temporal supports Mutual Transport Layer Security (mTLS) as a way of encrypting network traffic between the services of a Temporal Service and also between application processes and a Temporal Service.
Self-signed or properly minted certificates can be used for mTLS.
mTLS is set in Temporal's [TLS configuration](/references/service-configuration#tls).
The configuration includes two sections such that intra-Temporal Service and external traffic can be encrypted with different sets of certificates and settings:

- `internode`: Configuration for encrypting communication between nodes in the Temporal Service.
- `frontend`: Configuration for encrypting the Frontend's public endpoints.

A customized configuration can be passed using either the [WithConfig](/references/server-options#withconfig) or [WithConfigLoader](/references/server-options#withconfigloader) Server options.

See [TLS configuration reference](/references/service-configuration#tls) for more details.

### Authentication

There are a few authentication protocols available to prevent unwanted access such as authentication of servers, clients, and users.

### Servers

To prevent spoofing and [MITM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) you can specify the `serverName` in the `client` section of your respective mTLS configuration.
This enables established connections to authenticate the endpoint, ensuring that the server certificate presented to any connecting Client has the appropriate server name in its CN property.
It can be used for both `internode` and `frontend` endpoints.

More guidance on mTLS setup can be found in [the `samples-server` repo](https://github.com/temporalio/samples-server/tree/main/tls) and you can reach out to us for further guidance.

### Client connections

To restrict a client's network access to Temporal Service endpoints you can limit it to clients with certificates issued by a specific Certificate Authority (CA).
Use the `clientCAFiles`/ `clientCAData` and `requireClientAuth` properties in both the `internode` and `frontend` sections of the [mTLS configuration](/references/service-configuration#tls).

### Users

To restrict access to specific users, authentication and authorization is performed through extensibility points and plugins as described in the [Authorization](#authorization) section below.

#### Authorization

> **📝 Note:**
> Information regarding [`Authorizer`](#authorizer-plugin) and [`ClaimMapper`](#claim-mapper) has been moved to another location.

Temporal offers two plugin interfaces for implementing API call authorization:

- [`ClaimMapper`](#claim-mapper)
- [`Authorizer`](#authorizer-plugin)

The authorization and claim mapping logic is customizable, making it available to a variety of use cases and identity schemes.
When these are provided the frontend invokes the implementation of these interfaces before executing the requested operation.

See https://github.com/temporalio/samples-server/blob/main/extensibility/authorizer for a sample implementation.

![Front-end authorization order of operations](https://docs.temporal.io/diagrams/frontend-authorization-order-of-operations.png)

### Single sign-on integration

Temporal can be integrated with a single sign-on (SSO) experience by using the `ClaimMapper` and `Authorizer` plugins.
The default JWT `ClaimMapper` implementation can be used as is or as a base for a custom implementation of a similar plugin.

### Temporal UI

To enable SSO authentication in the Temporal UI using environment credentials, you need to configure the UI container with specific environment variables that define your identity provider and OAuth settings.
In your docker-compose.yaml, set `TEMPORAL_AUTH_ENABLED=true` to activate authentication.
Next, specify the required OAuth credentials and endpoints using environment variables such as:

- `TEMPORAL_AUTH_CLIENT_ID`
- `TEMPORAL_AUTH_CLIENT_SECRET`
- `TEMPORAL_AUTH_PROVIDER_URL`
- `TEMPORAL_AUTH_CALLBACK_URL`

These values correspond to the client credentials and endpoints provided by your OAuth identity provider (such as Google, Auth0, Okta).
When properly configured, Temporal UI will redirect users to your SSO login page and enforce authentication on access.
This approach does not require any additional configuration files, making it ideal for containerized environments using secure environment variable injection.

```yaml
temporal-ui:
  container_name: temporal-ui
  depends_on:
    - temporal
  environment:
    - TEMPORAL_GRPC_ENDPOINT=temporal:7233
    - TEMPORAL_ADDRESS=temporal:7233
    - TEMPORAL_AUTH_ENABLED=true
    - TEMPORAL_AUTH_PROVIDER_URL=https://example.com
    - TEMPORAL_AUTH_CLIENT_ID=xxxxxxxxxxxxxx
    - TEMPORAL_AUTH_CLIENT_SECRET=xxxxxxxxxxxxxx
    - TEMPORAL_AUTH_CALLBACK_URL=https://your-domain/auth/sso/callback
    - TEMPORAL_AUTH_SCOPES=openid profile email
  image: temporalio/ui:latest
  networks:
    - temporal-network
  ports:
    - 8080:8080
```

For more general guidance for configuration, refer to the [Temporal UI README](https://github.com/temporalio/ui?tab=readme-ov-file#configuration).
For more details on configuration with Docker, refer to [Temporal UI Config](https://github.com/temporalio/ui/blob/main/server/config/docker.yaml).

## Temporal Service plugins 

The Temporal Service supports some pluggable components.

### What is a ClaimMapper Plugin? 

The Claim Mapper component is a pluggable component that extracts Claims from JSON Web Tokens (JWTs).

This process is achieved with the method `GetClaims`, which translates `AuthInfo` structs from the caller into `Claims` about the caller's roles within Temporal.

A `Role` (within Temporal) is a bit mask that combines one or more of the role constants.
In the following example, the role is assigned constants that allow the caller to read and write information.

```go
role := authorization.RoleReader | authorization.RoleWriter
```

`GetClaims` is customizable and can be modified with the `temporal.WithClaimMapper` server option.
Temporal also offers a default JWT `ClaimMapper` for your use.

A typical approach is for `ClaimMapper` to interpret custom `Claims` from a caller's JWT, such as membership in groups, and map them to Temporal roles for the user.
The subject information from the caller's mTLS certificate can also be a parameter in determining roles.

#### `AuthInfo`

`AuthInfo` is a struct that is passed to `GetClaims`. `AuthInfo` contains an authorization token extracted from the `authorization` header of the gRPC request.

`AuthInfo` includes a pointer to the `pkix.Name` struct.
This struct contains an [x.509](https://www.ibm.com/docs/en/ibm-mq/7.5?topic=certificates-distinguished-names) Distinguished Name from the caller's mTLS certificate.

#### `Claims`

`Claims` is a struct that contains information about permission claims granted to the caller.

`Authorizer` assumes that the caller has been properly authenticated, and trusts the `Claims` when making an authorization decision.

#### Default JWT ClaimMapper

Temporal offers a default JWT `ClaimMapper` that extracts the information needed to form Temporal `Claims`.
This plugin requires a public key to validate digital signatures.

To get an instance of the default JWT `ClaimMapper`, call `NewDefaultJWTClaimMapper` and provide it with the following:

- a `TokenKeyProvider` instance
- a `config.Authorization` pointer
- a logger

The code for the default `ClaimMapper` can also be used to build a custom `ClaimMapper`.

#### Token key provider

A `TokenKeyProvider` obtains public keys from specified issuers' URIs that adhere to a specific format.
The default JWT `ClaimMapper` uses this component to obtain and refresh public keys over time.

Temporal provides a `defaultTokenKeyProvider`.
This component dynamically obtains public keys that follow the [JWKS format](https://tools.ietf.org/html/rfc7517).
It supports formats such as `RSA` and `ECDSA`.

```go
provider := authorization.NewDefaultTokenKeyProvider(cfg, logger)
```

> **📝 Note:**
>
> `KeySourceURIs` are the HTTP endpoints that return public keys of token issuers in the [JWKS format](https://tools.ietf.org/html/rfc7517).
> `RefreshInterval` defines how frequently keys should be refreshed.
> For example, [Auth0](https://auth0.com/) exposes endpoints such as `https://YOUR_DOMAIN/.well-known/jwks.json`.
>

By default, "permissions" is used to name the `permissionsClaimName` value.

Configure the plugin with `config.Config.Global.Authorization.JWTKeyProvider`.

#### JSON Web Token format

The default JWT `ClaimMapper` expects authorization tokens to be formatted as follows:

```
Bearer <token>
```

The Permissions Claim in the JWT Token is expected to be a collection of Individual Permission Claims.
Each Individual Permission Claim must be formatted as follows:

```
<namespace> : <permission>
```

These permissions are then converted into Temporal roles for the caller.
This can be one of Temporal's four values:

- read
- write
- worker
- admin

Multiple permissions for the same Namespace are overridden by the `ClaimMapper`.

##### Example of a payload for the default JWT ClaimMapper

```
{
   "permissions":[
      "temporal-system:read",
      "namespace1:write"
   ],
   "aud":[
      "audience"
   ],
   "exp":1630295722,
   "iss":"Issuer"
}
```

##### System permissions

The Namespace name `temporal-system` is reserved for system permissions.
The default JWT `ClaimMapper` maps `temporal-system:<permission>` to the caller's system role instead of a Namespace role.

The default `Authorizer` checks only the system role for cluster-scoped APIs, such as `ListNamespaces` and `GetClusterInfo`.
A Namespace role does not grant access to these APIs, even `admin`.
To call a cluster-scoped read-only API, a token needs at least `temporal-system:read`.

The Temporal Web UI calls `ListNamespaces` and `GetClusterInfo` when it loads.
A user whose token holds only Namespace permissions, such as `my-namespace:admin`, gets permission-denied errors for these calls.
Add `temporal-system:read` to that user's permissions to let the Web UI load.

A system role also applies to every Namespace, so `temporal-system:read` lets the caller read all Namespaces.
If that is broader than you want, use a custom `Authorizer` that allows the cluster-scoped read-only APIs the Web UI needs for callers with any Namespace role.

### What is an Authorizer Plugin? 

The `Authorizer` plugin contains a single `Authorize` method, which is invoked for each incoming API call.
`Authorize` receives information about the API call, along with the role and permission claims of the caller.

`Authorizer` allows for a wide range of authorization logic, including call target, role/permissions claims, and other data available to the system.

#### Configuration

The following arguments must be passed to `Authorizer`:

- `context.Context`: General context of the call.
- `authorization.Claims`: Claims about the roles assigned to the caller. Its intended use is described in the [`Claims`](#claims) section earlier on this page.
- `authorization.CallTarget`: Target of the API call.

`Authorizer` then returns one of two decisions:

- `DecisionDeny`: the requested API call is not invoked and an error is returned to the caller.
- `DecisionAllow`: the requested API call is invoked.

> **⚠️ Warning:**
> Security Warning
>
> If you do **not** explicitly configure an `Authorizer`, Temporal uses the default `noopAuthorizer`. This default allows **every** API request,
> with no authentication or access control. Anyone who can reach your Temporal Server can invoke any API, including sensitive administrative operations.
> This is **not secure** for production or for any environment that is accessible to untrusted clients (such as over the internet).
>
> **To protect your Temporal Server, you must configure an `Authorizer` plugin with a corresponding `ClaimMapper`.** Without this, your deployment is
> effectively open to anyone with network access.
>

Configure your `Authorizer` with the [`temporal.WithAuthorizer`](/references/server-options#withauthorizer) server option, and your `ClaimMapper` with
the [`temporal.WithClaimMapper`](/references/server-options#withclaimmapper) server option.

```go
temporalServer, err := temporal.NewServer(
	temporal.WithAuthorizer(newCustomAuthorizer()),
	temporal.WithClaimMapper(func(cfg *config.Config) authorization.ClaimMapper {
		return newCustomClaimMapper(cfg)
	}),
)
```

#### How to authorize SDK API calls 

When authentication is enabled, you can authorize API calls made to the Frontend Service.

The Temporal Server [expects](#authentication) an `authorization` gRPC header with an authorization token to be passed with API calls if [requests authorization](#authorization) is configured.

Authorization Tokens may be provided to the Temporal Java SDK by implementing a `io.temporal.authorization.AuthorizationTokenSupplier` interface.
The implementation should be used to create `io.temporal.authorization.AuthorizationGrpcMetadataProvider` that may be configured on ServiceStub gRPC interceptors list.

The implementation is called for each SDK gRPC request and may supply dynamic tokens.

**JWT**

One of the token types that may be passed this way are JWT tokens.
Temporal Server provides a [default implementation of JWT authentication](#default-jwt-claimmapper).

**Example**

```java
AuthorizationTokenSupplier tokenSupplier =
  //your implementation of token supplier
  () -> "Bearer <Base64 url-encoded value of the token for default JWT ClaimMapper>";
WorkflowServiceStubsOptions serviceStubOptions =
  WorkflowServiceStubsOptions.newBuilder()
    //other service stub options
    .addGrpcMetadataProvider(new AuthorizationGrpcMetadataProvider(tokenSupplier))
    .build();
WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(serviceStubOptions);
WorkflowClient client = WorkflowClient.newInstance(service);
```

Related read:

- [How to secure a Temporal Service](/security)

### What is a TokenProvider Plugin? 

The Token Provider component is a pluggable component that attaches an authentication token to outbound cross-cluster RPCs.
Where the `ClaimMapper` and `Authorizer` plugins protect the receiving end of a request, `TokenProvider` protects the sending end.
When a Temporal Service replicates Workflows, Schedules, or Namespaces to a peer cluster, `TokenProvider` supplies the bearer token that the peer's `ClaimMapper` validates.

A typical approach pairs a JWT-emitting `TokenProvider` on the sender with the [default JWT `ClaimMapper`](#default-jwt-claimmapper) on the receiver.

`TokenProvider` is a single-method interface:

```go
type TokenProvider interface {
    GetToken(ctx context.Context, rpcAddress string) (token string, err error)
}
```

`GetToken` is called on every outbound cross-cluster RPC.
`rpcAddress` is the receiver cluster's `host:port`. Providers commonly use this value as the JWT's `aud` claim to scope each token to a specific receiver.

The Temporal Service does not cache the returned token.
Implementations are responsible for caching and refresh.
A provider that fetches tokens from an external service must cache internally, otherwise every outbound RPC triggers a network round-trip to that service.
OAuth 2.0 client libraries typically handle this caching, and a `TokenProvider` can often be a thin wrapper around one.

#### Required claims on the receiver

Cross-cluster RPCs target the receiver's `AdminService`, which the default `Authorizer` only admits with `Claims{System: RoleAdmin}`.
Tokens carrying lower roles are rejected.

With the default JWT `ClaimMapper`, the token must carry a `permissions` claim with the entry `temporal-system:admin`.

```
{
   "permissions":[
      "temporal-system:admin"
   ]
}
```

A custom `ClaimMapper` can recognize any JWT shape, such as an OAuth-style `scp` scope or a custom claim, and translate it into the same `Claims{System: RoleAdmin}` result.

#### Stream lifecycle

Cross-cluster replication runs over long-lived gRPC streams.
The token is attached when a stream is opened, and the receiver's `ClaimMapper` and `Authorizer` evaluate it at that point.
Already-open streams are not re-checked when the token later expires.
Rotation and revocation therefore only take effect when a stream is reestablished, for example after a reconnect or a process restart.

#### Transport security

`auth.TokenCredentials` requires transport security, so the gRPC runtime refuses to attach the credential to a plaintext dial.
Configuring `WithTokenProvider` therefore requires also configuring TLS for the destination, either through [`global.tls.remoteClusters`](/references/service-configuration) in the YAML config or by passing a custom provider through [`temporal.WithTLSConfigFactory`](/references/server-options#withtlsconfigfactory).

> **📝 Note:**
>
> If `WithTokenProvider` is set but no remote-cluster TLS source is configured, the Temporal Service fails to boot with a directed error message.
> This catches the misconfiguration at startup rather than on the first cross-cluster RPC.
>

#### Fail-closed mode

Set `global.authorization.remoteClusterAuth.require: true` to require a non-empty token on every outbound remote-cluster RPC.
When `require` is `true`, the Temporal Service refuses to start if no `TokenProvider` is configured, and the credential returns `Unauthenticated` rather than sending an empty `authorization` header.

#### Configuration

Configure your `TokenProvider` with the [`temporal.WithTokenProvider`](/references/server-options#withtokenprovider) server option.
Pair it with the receiver-side plugins so peers validate what the sender attaches.

```go
temporalServer, err := temporal.NewServer(
    temporal.WithTokenProvider(myTokenProvider),
    temporal.WithAuthorizer(authorization.NewDefaultAuthorizer()),
    temporal.WithClaimMapper(func(cfg *config.Config) authorization.ClaimMapper {
        logger := getYourLogger()
        return authorization.NewDefaultJWTClaimMapper(
            authorization.NewDefaultTokenKeyProvider(&cfg.Global.Authorization, logger),
            &cfg.Global.Authorization,
            logger,
        )
    }),
)
```

When `TokenProvider` is not configured, outbound cross-cluster RPCs carry no `authorization` header.
This is the default for deployments that don't replicate to peers, or that rely on transport-level (mTLS) authentication alone.

## Data Converter 

Each Temporal SDK provides a [Data Converter](/dataconversion) that can be customized with a custom [Payload Codec](/payload-codec) to encode and secure your data.

For details on what data can be encoded, how to secure it, and what to consider when using encryption, see [Data encryption](/production-deployment/data-encryption).

#### Codec Server

You can use a [Codec Server](/codec-server) with your custom Payload Codec to decode the data you see on your Web UI and CLI locally through remote endpoints.
However, ensure that you consider all security implications of [remote data encoding](/remote-data-encoding) before using a Codec Server.

For details on how to set up a Codec Server, see [Codec Server setup](/production-deployment/data-encryption#codec-server-setup).

## Configuration options for restricting and securing access  

Temporal provides extensive configuration options to limit and control which callers can access your services and how those services behave in different environments. These options help you enforce stricter security boundaries in self-hosted deployments and tune behavior for production use cases. For more details on available settings and how to configure them, see the following references:

- [Cluster & Server configuration](/references/service-configuration) — Static configuration settings for servers and services.  

- [Dynamic configuration](/references/dynamic-configuration) — Runtime-updatable settings for tuning and behavior control.  

- [Client environment configuration](/references/client-environment-configuration) — Environment variable settings for configuring Temporal clients.

---

# Server frontend API reference

Source: https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference

While it's usually easiest to interact with [Temporal Server](/temporal-service/temporal-server) via a
[Client SDK](/encyclopedia/temporal-client) or the [Temporal CLI](/cli), you can
also use its gRPC API.

Our Client and Worker SDKs use the gRPC API. The API reference is located here:

[`api-docs.temporal.io`](https://api-docs.temporal.io/)

## Use with code

Usually you interact with the API via high-level methods like `client.workflow.start()`. However, Client SDKs also
expose the underlying gRPC services. For instance, the TypeScript SDK has:

- WorkflowService:
  [`Client.connection.workflowService`](https://typescript.temporal.io/api/classes/client.Connection#workflowservice)
- OperatorService:
  [`Client.connection.operatorService`](https://typescript.temporal.io/api/classes/client.Connection/#operatorservice)
- HealthService:
  [`Client.connection.healthService`](https://typescript.temporal.io/api/classes/client.Connection/#healthservice)

If you're not using an SDK Client (rare), you can generate gRPC client stubs by:

- Cloning [`temporalio/api`](https://github.com/temporalio/api) (repo with the protobuf files)
- Generating code in [your language](https://grpc.io/docs/languages/)

## Use manually

To query the API manually via command line or a GUI, first:

- If you don't already have a Server to connect to, run [`temporal server start-dev`](/cli/command-reference/server#start-dev)
- Clone this repo:

```shell
git clone https://github.com/temporalio/api.git
cd api
```

### With command line

Install [`evans`](https://github.com/ktr0731/evans#installation).

```shell
cd /path/to/api
evans --proto temporal/api/workflowservice/v1/service.proto --port 7233
```

To connect to Temporal Cloud, set the host, cert, cert key, and TLS flag:

```shell
evans --proto temporal/api/workflowservice/v1/service.proto --host devrel.a2dd6.tmprl.cloud --port 7233 --tls --cert /Users/me/certs/temporal.pem --certkey /Users/me/certs/temporal.key
```

Once inside the evans prompt, you can run commands like `help`, `show service` to list available methods, and
`call ListWorkflowExecutions`.

### With a GUI

- Install [BloomRPC](https://github.com/bloomrpc/bloomrpc#installation).
- Open the app
- Select "Import Paths" button on the top-left and enter the path to the cloned repo: `/path/to/api`
- Select the "Import protos" + button and select this file:

```shell
/path/to/api/temporal/api/workflowservice/v1/service.proto
```

- A list of methods should appear in the sidebar. Select one.
- Edit the JSON in the left pane.
- Hit `Cmd/Ctrl-Enter` or click the play button to get a response from the server on the right.

![ListWorkflowExecutions](https://docs.temporal.io/img/proto/ListWorkflowExecutions.png)

One downside compared to [command line](#with-command-line) is it doesn't show enum names, just numbers like
`"task_queue_type": 1`.

![DescribeTaskQueue](https://docs.temporal.io/img/proto/DescribeTaskQueue.png)

---

# Upgrade the Temporal Server

Source: https://docs.temporal.io/self-hosted-guide/upgrade-server

## How to upgrade the Temporal Server version 

If a newer version of the [Temporal Server](/temporal-service/temporal-server) is available, a notification appears in
the Temporal Web UI.

> **ℹ️ Info:**
>
> If you are using a version that is older than 1.0.0, reach out to us at
> [community.temporal.io](http://community.temporal.io) to ask how to upgrade.
>

First check to see if an upgrade to the database schema is required for the version you wish to upgrade to. If a
database schema upgrade is required, it will be called out directly in the
[release notes](https://github.com/temporalio/temporal/releases). Some releases require changes to the schema, and some
do not. We ensure that any consecutive versions are compatible in terms of database schema upgrades, features, and
system behavior; however there is no guarantee that there is compatibility between _any_ two non-consecutive versions.

### Key considerations

When upgrading the Temporal Server, there are two key considerations to keep in mind:

1. **Sequential Upgrades:** Temporal Server should be upgraded sequentially, one minor version at a time. Before
   bumping to the next minor version, first upgrade to the highest available patch version of your current minor
   version. For example, if you're on \(v1.n.0\), upgrade to \(v1.n.latest\) first, then proceed to
   \(v1.(n+1).latest\). Repeat this sequence until you reach your desired version.

2. **Data Compatibility:** During an upgrade, the Temporal Server either updates or restructures the existing version
   data to match the data format of the newer version. Temporal Server ensures backward compatibility only between two
   successive minor versions. Consequently, skipping versions during an upgrade may lead to older data formats becoming
   unreadable. If the previous data format cannot be interpreted and converted to the newer format, the upgrade process
   will be unsuccessful.

### Step-by-step upgrade procedure

Upgrading the Temporal Server requires a methodical approach to preserve data integrity and compatibility between versions. The following documentation outlines the step-by-step process to successfully upgrade your
Temporal Server.

When upgrading your Temporal Server version, ensure that you upgrade sequentially.

1. **Upgrade Database Schema:** Before initiating the Temporal Server upgrade, use one of the recommended upgrade tools
   to update your database schema. This ensures it is aligned with the version of Temporal Server you aim to upgrade to.
2. **Upgrade Temporal Server:** Once the database schema is updated, proceed to upgrade the Temporal Server deployment
   to the next sequential version.
3. **Iterative Upgrades** (optional): Continue this process (steps 1 and 2) iteratively until you reach the desired
   Temporal Server version.

By adhering to the above guidelines and following the step-by-step procedure, you can ensure a smooth and successful
upgrade of your Temporal Server.

The Temporal Server upgrade updates or rewrites the old version data with the format introduced in the newer version.
Because Temporal Server guarantees backward compatibility between two consecutive minor versions, and because older
versions of the code are eventually removed from the code base, skipping versions when upgrading might cause older
formats to become unrecognizable. If the old format of the data can't be read to be rewritten to the new format, the
upgrades fail.

Check the [Temporal Server releases](https://github.com/temporalio/temporal/releases) and follow these releases in
order. Before upgrading to the next minor version, upgrade to the highest available patch version of your current minor
version first.

Also, be aware that each upgrade requires the History Service to load all Shards and update the Shard metadata, so allow
approximately 10 minutes on each version for these processes to complete before upgrading to the next version.

Use one of the upgrade tools to upgrade your database schema to be compatible with the Temporal Server version being
upgraded to.

### Upgrade Cassandra schema

If you are using Cassandra for your Temporal Service's persistence, use the `temporal-cassandra-tool` to upgrade both
the default Persistence and Visibility schemas.

**Example default schema upgrade:**

```bash
temporal_v1.2.1 $ temporal-cassandra-tool \
   --tls \
   --tls-ca-file <...> \
   --user <cassandra-user> \
   --password <cassandra-password> \
   --endpoint <cassandra.example.com> \
   --keyspace temporal \
   --timeout 120 \
   update \
   --schema-dir ./schema/cassandra/temporal/versioned
```

**Example visibility schema upgrade:**

```bash
temporal_v1.2.1 $ temporal-cassandra-tool \
   --tls \
   --tls-ca-file <...> \
   --user <cassandra-user> \
   --password <cassandra-password> \
   --endpoint <cassandra.example.com> \
   --keyspace temporal_visibility \
   --timeout 120 \
   update \
   --schema-dir ./schema/cassandra/visibility/versioned
```

### Upgrade Elasticsearch schema

If you are using Elasticsearch for your Temporal Service's Visibility, use the `temporal-elasticsearch-tool` to upgrade
the schema.

**Example schema upgrade:**

```bash
echo "Updating index mappings: $ES_VISIBILITY_INDEX"
temporal-elasticsearch-tool \
  --endpoint "$ES_SCHEME://$ES_HOST:$ES_PORT" \
  --user "$ES_USER" \
  --password "$ES_PWD" \
  update-schema \
  --index "$ES_VISIBILITY_INDEX"
```

### Upgrade PostgreSQL or MySQL schema

If you are using MySQL or PostgreSQL use the `temporal-sql-tool`, which works similarly to the
`temporal-cassandra-tool`.

Refer to this [Makefile](https://github.com/temporalio/temporal/blob/v1.4.1/Makefile#L367-L383) for context.

#### PostgreSQL

**Example default schema upgrade:**

```bash
./temporal-sql-tool \
  --tls \
  --tls-enable-host-verification \
  --tls-cert-file <path to your client cert> \
  --tls-key-file <path to your client key> \
  --tls-ca-file <path to your CA> \
  --ep localhost -p 5432 -u temporal -pw temporal --pl postgres --db temporal update-schema -d ./schema/postgresql/v96/temporal/versioned
```

**Example visibility schema upgrade:**

```bash
./temporal-sql-tool \
  --tls \
  --tls-enable-host-verification \
  --tls-cert-file <path to your client cert> \
  --tls-key-file <path to your client key> \
  --tls-ca-file <path to your CA> \
  --ep localhost -p 5432 -u temporal -pw temporal --pl postgres --db temporal_visibility update-schema -d ./schema/postgresql/v96/visibility/versioned
```

If you're upgrading PostgreSQL to v12 or later to enable advanced Visibility features with Temporal Server v1.20,
upgrade your PostgreSQL version first, and then run `temporal-sql-tool` with the `postgres12` plugin, as shown in the
following example:

```bash
./temporal-sql-tool \
  --tls \
  --tls-enable-host-verification \
  --tls-cert-file <path to your client cert> \
  --tls-key-file <path to your client key> \
  --tls-ca-file <path to your CA> \
  --ep localhost -p 5432 -u temporal -pw temporal --pl postgres12 --db temporal_visibility update-schema -d ./schema/postgresql/v12/visibility/versioned
```

#### MySQL

**Example default schema upgrade:**

```bash
./temporal-sql-tool \
  --tls \
  --tls-enable-host-verification \
  --tls-cert-file <path to your client cert> \
  --tls-key-file <path to your client key> \
  --tls-ca-file <path to your CA> \
  --ep localhost -p 3036 -u root -pw root --pl mysql --db temporal update-schema -d ./schema/mysql/v57/temporal/versioned/
```

**Example visibility schema upgrade:**

```bash
./temporal-sql-tool \
  --tls \
  --tls-enable-host-verification \
  --tls-cert-file <path to your client cert> \
  --tls-key-file <path to your client key> \
  --tls-ca-file <path to your CA> \
  --ep localhost -p 3036 -u root -pw root --pl mysql --db temporal_visibility update-schema -d ./schema/mysql/v57/visibility/versioned/
```

If you're upgrading MySQL to v8.0.17 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade
your MySQL version first, and then run `temporal-sql-tool` with the `mysql8` plugin, as shown in the following example:

```bash
./temporal-sql-tool \
  --tls \
  --tls-enable-host-verification \
  --tls-cert-file <path to your client cert> \
  --tls-key-file <path to your client key> \
  --tls-ca-file <path to your CA> \
  --ep localhost -p 5432 -u temporal -pw temporal --pl mysql8 --db temporal_visibility update-schema -d ./schema/mysql/v8/visibility/versioned.
```

### Roll-out technique

We recommend preparing a staging Temporal Service and then doing the following to verify the upgrade is successful:

1. Create some simulation load on the staging Temporal Service.
2. Upgrade the database schema in the staging Temporal Service.
3. Wait and observe for a few minutes to verify that there is no unstable behavior from both the server and the
   simulation load logic.
4. Upgrade the server.
5. Now do the same to the live environment Temporal Service.

---

# Self-hosted Visibility feature setup

Source: https://docs.temporal.io/self-hosted-guide/visibility

A [Visibility](/temporal-service/visibility) store is set up as a part of your
[Persistence store](/temporal-service/persistence) to enable listing and filtering details about Workflow Executions
that exist on your Temporal Service.

A Visibility store is required in a Temporal Service setup because it is used by [Temporal Web UI](/web-ui) and
[Temporal CLI](/cli) to pull [Workflow Execution](/workflow-execution) data and enables features like batch operations
on a group of Workflow Executions.

With the Visibility store, you can use [List Filters](/list-filter) with [Search Attributes](/search-attribute) to list
and filter Workflow Executions that you want to review or act upon.

Supported Visibility stores include:

- Elasticsearch v7 with Temporal Server v1.7 and later
- Elasticsearch v8 with Temporal Server v1.18 and later
- OpenSearch 2+ with Temporal Server v1.30.1 and later
- MySQL v8.0.17 and later with Temporal Server v1.20 and later
- PostgreSQL v12 and later with Temporal Server v1.20 and later
- SQLite v3.31.0 and later with Temporal Server v1.20 and later

## Current and legacy Visibility support 

[Advanced Visibility](/visibility#advanced-visibility) is the current generation of Temporal Visibility. It supports the
modern query model, including [custom Search Attributes](/search-attribute#custom-search-attribute).

This page also includes guidance for the legacy (deprecated in Temporal Server v1.21 and removed in v1.24)
[standard Visibility](/self-hosted-guide/visibility/legacy-standard) model for older deployments and migration work. In this context,
"advanced" and "standard (legacy)" refer to the current and legacy generations of Temporal Visibility, respectively.

The following compatibility matrix summarizes which generation of Visibility each store supports and the Temporal Server
versions required:

| Store                                                                                                               | Advanced Visibility (current)                                                                                                                                    | Standard Visibility (legacy)                                          |
| :------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------- |
| [Elasticsearch](/self-hosted-guide/visibility/elasticsearch) <br/><br/>Recommended for any setup that spawns more than a few Workflow Executions | v7 on Temporal Server v1.7+, v8 on Temporal Server v1.18+                                                                                                        | Not supported                                                         |
| [OpenSearch](/self-hosted-guide/visibility/elasticsearch) <br/><br/>Recommended for any setup that spawns more than a few Workflow Executions    | 2+ on Temporal Server v1.30.1+                                                                                                                                   | Not supported                                                         |
| [MySQL](/self-hosted-guide/visibility/mysql)                                                                                                     | v8.0.17+ on Temporal Server v1.20+                                                                                                                               | v5.7+ on older deployments before Temporal Server v1.24               |
| [PostgreSQL](/self-hosted-guide/visibility/postgresql)                                                                                           | v12+ on Temporal Server v1.20+                                                                                                                                   | v9.6+ on older deployments before Temporal Server v1.24               |
| [SQLite](/self-hosted-guide/visibility/sqlite)                                                                                                   | v3.31.0+ on Temporal Server v1.20+                                                                                                                               | Not supported                                                         |
| [Cassandra](/self-hosted-guide/visibility/legacy-standard#cassandra)                                                                                             | Not supported.<br/><br/>To migrate from Cassandra to a supported advanced Visibility store, see [Migrating Visibility database](/self-hosted-guide/visibility/migrate). | Deprecated in Temporal Server v1.21, removed in Temporal Server v1.24 |

You can use any combination of the supported databases for your Persistence and Visibility stores. For updates, check
[Server release notes](https://github.com/temporalio/temporal/releases).

Temporal Server v1.21 introduced support for a secondary Visibility store in your Temporal Service to enable
[Dual Visibility](/dual-visibility). This is useful for migrating your Visibility store database.

---

# Manage custom Search Attributes

Source: https://docs.temporal.io/self-hosted-guide/visibility/custom-search-attributes

> Create or rename custom Search Attributes with the Temporal CLI on Temporal Cloud; contact Support to delete them. Use Temporal CLI to create or remove them when self-hosting.

To manage custom Search Attributes on Temporal Cloud, use the [Temporal CLI](/cli/command-reference/cloud/namespace#search-attribute).
With Temporal Cloud, you can create and rename custom Search Attributes. If you need to delete a custom Search Attribute, contact Support at [support.temporal.io](https://support.temporal.io).
To manage custom Search Attributes on a self-hosted Temporal Service, use the [Temporal CLI](/cli/command-reference/operator#search-attribute).
With a self-hosted Temporal Service, you can create and remove custom Search Attributes.

If you're self-hosting, verify whether your [Visibility database](/self-hosted-guide/visibility#supported-databases) version supports custom Search Attributes before proceeding.

> **⚠️ Caution:**
> Do not use sensitive data or PII in Search Attributes
>
> Do not include sensitive data, secrets, or personally identifiable information (PII) in Search Attribute **names or values**.
> Search Attribute values are stored unencrypted in the Visibility store and are not processed by a custom [Payload Codec](/payload-codec#payload-codec).
> The Temporal Server must be able to read these values in plain text to support filtering and ordering, so encryption is not possible without breaking search functionality.
>
> Attribute names are also visible in Namespace configuration, query expressions, and Temporal UI.
> Using sensitive data in either names or values risks exposure to anyone with Namespace access and may violate data protection regulations such as GDPR, HIPAA, or SOC 2.
>

## How to create custom Search Attributes 

Creating a custom Search Attribute in your Visibility store makes it available to use in your Workflow metadata and
[List Filters](/list-filter).

**On Temporal Cloud**

To create custom Search Attributes on Temporal Cloud, use
[`temporal cloud namespace search-attribute create`](/cli/command-reference/cloud/namespace#search-attribute-create).
For example, to add a custom Search Attribute named `CustomSA` of type `Keyword` to your Namespace, run:

```
temporal cloud namespace search-attribute create \
  --namespace <namespace>.<account> \
  --name CustomSA \
  --type Keyword
```

**On self-hosted Temporal Service**

To create custom Search Attributes in your self-hosted Temporal Service Visibility store, use
`temporal operator search-attribute create` with `--name` and `--type` command options.

For example, to create a Search Attribute called `CustomSA` of type `Keyword`, run the following command:

```
temporal operator search-attribute create --name="CustomSA" --type="Keyword"
```

Note that if you use a SQL database with advanced Visibility capabilities, you are required to specify a Namespace when
creating a custom Search Attribute. For example:

```
temporal operator search-attribute create --name="CustomSA" --type="Keyword" --namespace="yournamespace"
```

You can also create multiple custom Search Attributes when you set up your Visibility store.

The following example shows how custom Search Attributes can be created during Visibility store setup for SQL databases.
For setup examples, refer to the [samples-server repository](https://github.com/temporalio/samples-server)

```bash
add_custom_search_attributes() {
    until temporal operator search-attribute list --namespace "${DEFAULT_NAMESPACE}"; do
      echo "Waiting for namespace cache to refresh..."
      sleep 1
    done
    echo "Namespace cache refreshed."

    echo "Adding Custom*Field search attributes."

    temporal operator search-attribute create --namespace "${DEFAULT_NAMESPACE}" --yes \
        --name="CustomKeywordField" --type="Keyword" \
        --name="CustomStringField" --type="Text" \
        --name="CustomTextField" --type="Text" \
        --name="CustomIntField" --type="Int" \
        --name="CustomDatetimeField" --type="Datetime" \
        --name="CustomDoubleField" --type="Double" \
        --name="CustomBoolField" --type="Bool"
}
```

For Temporal Server v1.19 and earlier, or if using Elasticsearch for advanced Visibility, you can create custom Search
Attributes without a Namespace association, as shown in the following example.

```bash
add_custom_search_attributes() {
       echo "Adding Custom*Field search attributes."
       temporal operator search-attribute create \
        --name="CustomKeywordField" --type="Keyword" \
        --name="CustomStringField" --type="Text" \
        --name="CustomTextField" --type="Text" \
        --name="CustomIntField" --type="Int" \
        --name="CustomDatetimeField" --type="Datetime" \
        --name="CustomDoubleField" --type="Double" \
        --name="CustomBoolField" --type="Bool"
}
```

When your Visibility store is set up and running, these custom Search Attributes are available to use in your Workflow
code.

## How to remove custom Search Attributes 

To remove a Search Attribute key from your self-hosted Temporal Service Visibility store, use the command
`temporal operator search-attribute remove`. Removing Search Attributes is not supported on Temporal Cloud.

For example, if using Elasticsearch for advanced Visibility, to remove a custom Search Attribute called `CustomSA` of
type Keyword use the following command:

```
temporal operator search-attribute remove \
    --name="your_custom_attribute"
```

If you use a SQL database for advanced Visibility on Temporal Server v1.20 and later, you need to specify the Namespace
in your command, as shown in the following command:

```
temporal operator search-attribute remove \
    --name="your_custom_attribute" \
    --namespace="your_namespace"
```

To check whether the Search Attribute was removed, run

```
temporal operator search-attribute list
```

and check the list.

If you're on Temporal Server v1.20 and later, specify the Namespace from which you removed the Search Attribute. For
example,

```
temporal search-attribute list --namespace="yournamespace"
```

Note that if you use [SQL databases](/self-hosted-guide/visibility) with Temporal Server v1.20 and later, a new custom
Search Attribute is mapped to a database field name in the Visibility store `custom_search_attributes` table. Removing
this custom Search Attribute removes the mapping with the database field name but does not remove the data. If you
remove a custom Search Attribute and add a new one, the new custom Search Attribute might be mapped to the database
field of the one that was recently removed. This might cause unexpected results when you use the List API to retrieve
results using the new custom Search Attribute. These constraints do not apply if you use Elasticsearch.

---

# Set up Dual Visibility

Source: https://docs.temporal.io/self-hosted-guide/visibility/dual-visibility

> Configure a secondary Visibility store alongside your primary one to migrate or back up Workflow search data without downtime.

To enable [Dual Visibility](/dual-visibility), set up a secondary Visibility store with your primary Visibility store,
and configure your Temporal Service to enable read and/or write operations on the secondary Visibility store.

With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to
write to primary only, secondary only, or to both primary and secondary stores.

## Set up secondary Visibility store

Set the secondary store with the `secondaryVisibilityStore` configuration key in your Persistence configuration, and
then define the secondary Visibility store configuration under `datastores`.

You can configure any of the [supported databases](/self-hosted-guide/visibility) as your secondary store.

Examples:

To configure MySQL as a secondary store with Cassandra as your primary store, do the following.

```yaml
persistence:
  visibilityStore: cass-visibility # This is your primary Visibility store
  secondaryVisibilityStore: mysql-visibility # This is your secondary Visibility store
  datastores:
    cass-visibility:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal_primary_visibility'
    mysql-visibility:
      sql:
        pluginName: 'mysql8' # Verify supported versions. Use a version of SQL that supports advanced Visibility.
        databaseName: 'temporal_secondary_visibility'
        connectAddr: '127.0.0.1:3306'
        connectProtocol: 'tcp'
        user: 'temporal'
        password: 'temporal'
```

To configure Elasticsearch as both your primary and secondary store, use the configuration key
`elasticsearch.indices.secondary_visibility`, as shown in the following example.

```yaml
persistence:
  visibilityStore: es-visibility
  datastores:
    es-visibility:
      elasticsearch:
        version: 'v7'
        logLevel: 'error'
        url:
          scheme: 'http'
          host: '127.0.0.1:9200'
        indices:
          visibility: temporal_visibility_v1
          secondary_visibility: temporal_visibility_v1_new
        closeIdleConnectionsInterval: 15s
```

## Database schema and setup

The database schema and setup for a secondary store depends on the database you plan to use.

- [MySQL](/self-hosted-guide/visibility/mysql)
- [PostgreSQL](/self-hosted-guide/visibility/postgresql)
- [SQLite](/self-hosted-guide/visibility/sqlite)
- [Elasticsearch](/self-hosted-guide/visibility/elasticsearch)

For the Cassandra and MySQL configuration in the previous example, an example setup script would be as follows.

```bash
#...
# set your Cassandra environment variables
: "${KEYSPACE:=temporal}"
: "${VISIBILITY_KEYSPACE:=temporal_primary_visibility}"

: "${CASSANDRA_SEEDS:=}"
: "${CASSANDRA_PORT:=9042}"
: "${CASSANDRA_USER:=}"
: "${CASSANDRA_PASSWORD:=}"
: "${CASSANDRA_TLS_ENABLED:=}"
: "${CASSANDRA_CERT:=}"
: "${CASSANDRA_CERT_KEY:=}"
: "${CASSANDRA_CA:=}"
: "${CASSANDRA_REPLICATION_FACTOR:=1}"
#...
# set connection details
#...
# set up Cassandra schema
setup_cassandra_schema() {
  #...
  # use valid schema for the version of the database you want to set up for Visibility
    VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/cassandra/visibility/versioned
    if [[ ${SKIP_DB_CREATE} != true ]]; then
        temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" create -k "${VISIBILITY_KEYSPACE}" --rf "${CASSANDRA_REPLICATION_FACTOR}"
    fi
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" setup-schema -v 0.0
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"
  #...
}
#...
# set your MySQL environment variables
: "${DBNAME:=temporal}"
: "${VISIBILITY_DBNAME:=temporal_secondary_visibility}"
: "${DB_PORT:=}"
: "${MYSQL_SEEDS:=}"
: "${MYSQL_USER:=}"
: "${MYSQL_PWD:=}"
: "${MYSQL_TX_ISOLATION_COMPAT:=false}"

#...
# set connection details
#...
# set up MySQL schema
setup_mysql_schema() {
    #...
    # use valid schema for the version of the database you want to set up for Visibility
    VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/mysql/${MYSQL_VERSION_DIR}/visibility/versioned
    if [[ ${SKIP_DB_CREATE} != true ]]; then
        temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" create
    fi
    temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" setup-schema -v 0.0
    temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"
#...
}
```

For Elasticsearch as both primary and secondary Visibility store configuration in the previous example, an example setup
script would be as follows.

```bash
#...
# Elasticsearch
: "${ENABLE_ES:=false}"
: "${ES_SCHEME:=http}"
: "${ES_SEEDS:=}"
: "${ES_PORT:=9200}"
: "${ES_USER:=}"
: "${ES_PWD:=}"
: "${ES_VERSION:=v7}"
: "${ES_VIS_INDEX:=temporal_visibility_v1_dev}"
: "${ES_SEC_VIS_INDEX:=temporal_visibility_v1_new}"
: "${ES_SCHEMA_SETUP_TIMEOUT_IN_SECONDS:=0}"

#...

# Validate your ES environment
#...
# Wait for ES to start
#...
# Set up Elasticsearch index
setup_es_index() {
    ES_SERVER="${ES_SCHEME}://${ES_SEEDS%%,*}:${ES_PORT}"
    # ES_SERVER is the URL of Elasticsearch server i.e. "http://localhost:9200".
    SETTINGS_URL="${ES_SERVER}/_cluster/settings"
    SETTINGS_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/cluster_settings_${ES_VERSION}.json
    TEMPLATE_URL="${ES_SERVER}/_template/temporal_visibility_v1_template"
    SCHEMA_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/index_template_${ES_VERSION}.json
    INDEX_URL="${ES_SERVER}/${ES_VIS_INDEX}"
    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${SETTINGS_URL}" -H "Content-Type: application/json" --data-binary "@${SETTINGS_FILE}" --write-out "\n"
    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${TEMPLATE_URL}" -H 'Content-Type: application/json' --data-binary "@${SCHEMA_FILE}" --write-out "\n"
    curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${INDEX_URL}" --write-out "\n"

    # Checks for and sets up Elasticsearch as a secondary Visibility store
    if [[ ! -z "${ES_SEC_VIS_INDEX}" ]]; then
      SEC_INDEX_URL="${ES_SERVER}/${ES_SEC_VIS_INDEX}"
      curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${SEC_INDEX_URL}" --write-out "\n"
    fi
}
```

## Update Temporal Service configuration

With the primary and secondary stores set, update the `system.secondaryVisibilityWritingMode` and
`system.enableReadFromSecondaryVisibility` configuration keys in your self-hosted Temporal Service's dynamic
configuration YAML file to enable read and/or write operations to the secondary Visibility store.

For example, to enable write operations to both primary and secondary stores, but disable reading from the secondary
store, use the following.

```yaml
system.secondaryVisibilityWritingMode:
  - value: 'dual'
    constraints: {}
system.enableReadFromSecondaryVisibility:
  - value: false
    constraints: {}
```

For details on the configuration options, see:

- [Secondary Visibility dynamic configuration reference](/references/dynamic-configuration#secondary-visibility-settings)
- [Migrating Visibility databases](/self-hosted-guide/visibility/migrate)

---

# Integrate Elasticsearch or OpenSearch

Source: https://docs.temporal.io/self-hosted-guide/visibility/elasticsearch

> Configure Elasticsearch or OpenSearch as your self-hosted Visibility store, with version compatibility and index schema setup.

You can integrate Elasticsearch or OpenSearch with your Temporal Service as your Visibility store. We recommend using
one of these backends for large-scale operations on the Temporal Service.

To integrate Elasticsearch or OpenSearch with your Temporal Service, edit the `persistence` section of your
`development.yaml` configuration file to add the search backend as the `visibilityStore`, and run the index schema setup
commands.

Use the following version guidance:

- Elasticsearch v7 is supported with Temporal Server v1.7 and later.
- Elasticsearch v8 is supported with Temporal Server v1.18 and later.
- OpenSearch 2+ is supported with Temporal Server v1.30.1 and later.

The examples in this section use Elasticsearch. For OpenSearch, use the same datastore configuration shape and
operational flow unless a release note for your target Temporal Server version says otherwise.

## Persistence configuration

Set your Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the
search backend configuration under `datastores`.

The following example shows how to set a Visibility store named `es-visibility` and define the Elasticsearch datastore
configuration in your Temporal Service configuration YAML.

```yaml
persistence:
  ...
  visibilityStore: es-visibility
  datastores:
    ...
    es-visibility: # Define the Elasticsearch datastore connection information under the `es-visibility` key
      elasticsearch:
        version: "v7"
        url:
          scheme: "http"
          host: "127.0.0.1:9200"
        indices:
          visibility: temporal_visibility_v1_dev
```

## Index schema and index

To set up Elasticsearch as your Visibility store, use the `temporal-elasticsearch-tool` available in the
`temporalio/admin-tools` image.

The following example shows how to set up an Elasticsearch Visibility store with a MySQL persistence store using
`temporal-elasticsearch-tool`. For more examples with different databases, refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts).

[compose/scripts/setup-mysql-es.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-mysql-es.sh)
```sh
set -eu

# Validate required environment variables
: "${ES_SCHEME:?ERROR: ES_SCHEME environment variable is required}"
: "${ES_HOST:?ERROR: ES_HOST environment variable is required}"
: "${ES_PORT:?ERROR: ES_PORT environment variable is required}"
: "${ES_VISIBILITY_INDEX:?ERROR: ES_VISIBILITY_INDEX environment variable is required}"
: "${ES_VERSION:?ERROR: ES_VERSION environment variable is required}"

: "${MYSQL_SEEDS:?ERROR: MYSQL_SEEDS environment variable is required}"
: "${MYSQL_USER:?ERROR: MYSQL_USER environment variable is required}"

echo 'Starting MySQL and Elasticsearch schema setup...'
echo 'Waiting for MySQL port to be available...'
nc -z -w 10 ${MYSQL_SEEDS} ${DB_PORT:-3306}
echo 'MySQL port is available'

# Create and setup temporal database
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal create
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal setup-schema -v 0.0
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned

# Setup Elasticsearch index
# temporal-elasticsearch-tool is available in v1.30+ server releases
if [ -x /usr/local/bin/temporal-elasticsearch-tool ]; then
  echo 'Using temporal-elasticsearch-tool for Elasticsearch setup'
  temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" setup-schema
  temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" create-index --index $ES_VISIBILITY_INDEX
else
  echo 'Using curl for Elasticsearch setup'
  echo 'WARNING: curl will be removed from admin-tools in v1.30.'
  echo 'Waiting for Elasticsearch to be ready...'
  max_attempts=30
  attempt=0
  until curl -s -f "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s"; do
    attempt=$((attempt + 1))
    if [ $attempt -ge $max_attempts ]; then
      echo "ERROR: Elasticsearch did not become ready after $max_attempts attempts"
      echo "Last error from curl:"
      curl "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s" 2>&1 || true
      exit 1
    fi
    echo "Elasticsearch not ready yet, waiting... (attempt $attempt/$max_attempts)"
    sleep 2
  done
  echo ''
  echo 'Elasticsearch is ready'
  echo 'Creating index template...'
  curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/_template/temporal_visibility_v1_template" -H 'Content-Type: application/json' --data-binary "@/etc/temporal/schema/elasticsearch/visibility/index_template_$ES_VERSION.json"
  echo ''
  echo 'Creating index...'
  curl --head --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX" 2>/dev/null || curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX"
  echo ''
fi

echo 'MySQL and Elasticsearch setup complete'
```

## Elasticsearch privileges

Ensure that the following privileges are granted for the Elasticsearch Temporal index:

- **Read**
  - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices):
    `create`, `index`, `delete`, `read`
- **Write**
  - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices):
    `write`
- **Custom Search Attributes**
  - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices):
    `manage`
  - [cluster privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-cluster):
    `monitor` or `manage`.

---

# Legacy standard Visibility configuration

Source: https://docs.temporal.io/self-hosted-guide/visibility/legacy-standard

> Configure legacy standard Visibility for older Temporal Server versions.

The following section applies to older self-hosted deployments that still use standard Visibility. For new deployments,
use one of the advanced Visibility backends described in this guide.

## How to set up Cassandra Visibility store 

> **💡 Tip:**
> Support, stability, and dependency info
>
> - Cassandra supported only standard Visibility. Standard Visibility was deprecated in Temporal Server v1.21 and removed
>   in v1.24. For updates, check the [Temporal Server release notes](https://github.com/temporalio/temporal/releases).
> - We recommend migrating from Cassandra to any of the other supported databases for Visibility.
>

Advanced Visibility is not supported with Cassandra.

To enable current Visibility features, use MySQL, PostgreSQL, SQLite, Elasticsearch, or OpenSearch as your Visibility
store. We recommend Elasticsearch or OpenSearch for any Temporal Service setup that handles more than a few Workflow
Executions because these backends support the Visibility request load and help optimize performance.

To migrate from Cassandra to a supported SQL database, see
[Migrating Visibility database](/self-hosted-guide/visibility/migrate).

## Persistence configuration

Set your Cassandra Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `cass-visibility` and define the datastore configuration in
your Temporal Service configuration YAML.

```yaml
#...
persistence:
  #...
  visibilityStore: cass-visibility
  #...
  datastores:
    default:
    #...
    cass-visibility:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal_visibility'
#...
```

## Database schema and setup

Visibility data is stored in a database table called `executions_visibility` that must be set up according to the
schemas defined (by supported versions) in https://github.com/temporalio/temporal/tree/main/schema/cassandra/visibility.

The following example shows how to set up your Cassandra Visibility store using `temporal-cassandra-tool`. For more
examples with different databases, refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts).

```bash
#...
# set your Cassandra environment variables
: "${KEYSPACE:=temporal}"
: "${VISIBILITY_KEYSPACE:=temporal_visibility}"

: "${CASSANDRA_SEEDS:=}"
: "${CASSANDRA_PORT:=9042}"
: "${CASSANDRA_USER:=}"
: "${CASSANDRA_PASSWORD:=}"
: "${CASSANDRA_TLS_ENABLED:=}"
: "${CASSANDRA_CERT:=}"
: "${CASSANDRA_CERT_KEY:=}"
: "${CASSANDRA_CA:=}"
: "${CASSANDRA_REPLICATION_FACTOR:=1}"
#...
# set connection details
#...
# set up Cassandra schema
setup_cassandra_schema() {
  #...
  # use valid schema for the version of the database you want to set up for Visibility
    VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/cassandra/visibility/versioned
    if [[ ${SKIP_DB_CREATE} != true ]]; then
        temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" create -k "${VISIBILITY_KEYSPACE}" --rf "${CASSANDRA_REPLICATION_FACTOR}"
    fi
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" setup-schema -v 0.0
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"
  #...
}
```

---

# Migrate Visibility database

Source: https://docs.temporal.io/self-hosted-guide/visibility/migrate

> How to migrate a Visibility database for a self-hosted Temporal Service.

To migrate your Visibility database, [set up a secondary Visibility store](/self-hosted-guide/visibility/dual-visibility) to enable
[Dual Visibility](/dual-visibility), and update the dynamic configuration in your Temporal Service to update the read
and write operations for the Visibility store.

Dual Visibility setup is optional but useful in gradually migrating your Visibility data to another database.

Before you begin, verify [supported databases and versions](/self-hosted-guide/visibility) for a Visibility store.

The following steps describe how to migrate your Visibility database.

After you make any changes to your [Temporal Service configuration](/temporal-service/configuration), ensure that you
restart your services.

## Set up secondary Visibility store

1. In your Temporal Service configuration,
   [add a secondary Visibility store](/references/service-configuration#secondaryvisibilitystore) to your Visibility setup under
   the Persistence configuration.

   Example: To migrate from Cassandra to Elasticsearch, add Elasticsearch as your secondary database and set it up. For
   details, see [secondary Visibility database schema and setup](/self-hosted-guide/visibility/dual-visibility).

   ```yaml
   persistence:
   visibilityStore: cass-visibility
   secondaryVisibilityStore: es-visibility
   datastores:
     cass-visibility:
     cassandra:
       hosts: '127.0.0.1'
       keyspace: 'temporal_visibility'
     es-visibility:
     elasticsearch:
       version: 'v7'
       logLevel: 'error'
       url:
       scheme: 'http'
       host: '127.0.0.1:9200'
       indices:
       visibility: temporal_visibility_v1_dev
       closeIdleConnectionsInterval: 15s
   ```

1. Update the [dynamic configuration](/temporal-service/configuration#dynamic-configuration) keys on your self-hosted
   Temporal Service to enable write operations to the secondary store and disable read operations. Example:

   ```yaml
   system.secondaryVisibilityWritingMode:
   - value: "dual"
   constraints: {}
   system.enableReadFromSecondaryVisibility:
   - value: false
   constraints: {}
   ```

At this point, Visibility data is read from the primary store, and all Visibility data is written to both the primary
and secondary store. This setting applies only to new Visibility data generated after Dual Visibility is enabled. It
does not migrate any existing data in the primary store to the secondary store.

For details on write options to the secondary store, see
[Secondary Visibility dynamic configuration reference](/references/dynamic-configuration#secondary-visibility-settings).

## Run in dual mode

When you enable a secondary store, only new Visibility data is written to both primary and secondary stores. The primary
store still holds the Workflow Execution data from before the secondary store was set up.

Running in dual mode lets you plan for closed and open Workflow Executions data from before the secondary store was set
up in your self-hosted Temporal Service.

Example:

- To manage closed Workflow Executions data, run in dual mode until the Namespace
  [Retention Period](/temporal-service/temporal-server#retention-period) is reached. After the Retention Period,
  Workflow Execution data is removed from the Persistence and Visibility stores. If you want to keep the closed Workflow
  Executions data after the set Retention Period, you must set up [Archival](/self-hosted-guide/archival).
- To manage data for all open Workflow Executions, run in dual mode until all the Workflow Executions started before
  enabling Dual Visibility mode are closed. After the Workflow Executions are closed, verify the Retention Period and
  set up Archival if you need to keep the data beyond the Retention Period.

You can run your Visibility setup in dual mode for an indefinite period, or until you are ready to deprecate the primary
store and move completely to the secondary store without losing data.

## Deprecate primary Visibility store

When you are ready to deprecate your primary store, follow these steps.

1. Update the dynamic configuration YAML to enable read operations from the secondary store. Example:

   ```yaml
   system.secondaryVisibilityWritingMode:
   - value: "dual"
   constraints: {}
   system.enableReadFromSecondaryVisibility:
   - value: true
   constraints: {}
   ```

   At this point, Visibility data is read from the secondary store only. Verify whether data on the secondary store is
   correct.

1. When the secondary store is vetted and ready to replace your current primary store, change your Temporal Service
   configuration to set the secondary store as your primary, and remove the dynamic configuration set in the previous
   steps. Example:

   ```yaml
   persistence:
   visibilityStore: es-visibility
   datastores:
     es-visibility:
     elasticsearch:
       version: 'v7'
       logLevel: 'error'
       url:
       scheme: 'http'
       host: '127.0.0.1:9200'
       indices:
       visibility: temporal_visibility_v1_dev
       closeIdleConnectionsInterval: 15s
   ```

---

# Set up MySQL Visibility store

Source: https://docs.temporal.io/self-hosted-guide/visibility/mysql

> How to set up a MySQL Visibility store for a self-hosted Temporal Service.

> **💡 Tip:**
> Support, stability, and dependency info
>
> - MySQL v5.7 and later.
> - Advanced Visibility is available on MySQL v8.0.17 and later with Temporal Server v1.20 and later.
> - MySQL v5.7 support applied to older standard Visibility deployments before Temporal Server v1.24.
>

You can set MySQL as your [Visibility store](/temporal-service/visibility). Verify
[supported versions](/self-hosted-guide/visibility) before you proceed.

If using MySQL v8.0.17 or later as your Visibility store with Temporal Server v1.20 and later, any
[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a
Namespace in that Temporal Service.

## Persistence configuration

Set your MySQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `mysql-visibility` and define the datastore configuration in
your Temporal Service configuration YAML.

```yaml
#...
persistence:
  #...
  visibilityStore: mysql-visibility
  #...
  datastores:
    default:
      #...
    mysql-visibility:
      sql:
        pluginName: 'mysql8' # For MySQL v8.0.17 and later. For earlier versions, use "mysql" plugin.
        databaseName: 'temporal_visibility'
        connectAddr: ' ' # Remote address of this database; for example, 127.0.0.0:3306
        connectProtocol: ' ' # Protocol example: tcp
        user: 'username_for_auth'
        password: 'password_for_auth'
        maxConns: 2
        maxIdleConns: 2
        maxConnLifetime: '1h'
#...
```

For details on the configuration parameters and values, see
[Temporal Service configuration](/references/service-configuration#sql).

To enable advanced Visibility features on your MySQL Visibility store, upgrade to MySQL v8.0.17 or later with Temporal
Server v1.20 or later. See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) on how to upgrade your
Temporal Server and database schemas.

For example configuration templates, see
[MySQL Visibility store configuration](https://github.com/temporalio/temporal/blob/main/config/development-mysql8.yaml).

## Database schema and setup

Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for
[MySQL v8.0.17 and later](https://github.com/temporalio/temporal/tree/main/schema/mysql/v8/visibility).

The following example shows how to set up your MySQL as both your persistence and Visibility store using
`temporal-sql-tool`. Refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples
with different databases.

[compose/scripts/setup-mysql.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-mysql.sh)
```sh
set -eu

# Validate required environment variables
: "${MYSQL_SEEDS:?ERROR: MYSQL_SEEDS environment variable is required}"
: "${MYSQL_USER:?ERROR: MYSQL_USER environment variable is required}"

echo 'Starting MySQL schema setup...'
echo 'Waiting for MySQL port to be available...'
nc -z -w 10 ${MYSQL_SEEDS} ${DB_PORT:-3306}
echo 'MySQL port is available'

# Create and setup temporal database
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal create
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal setup-schema -v 0.0
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned

# Create and setup visibility database
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility create
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility setup-schema -v 0.0
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility update-schema -d /etc/temporal/schema/mysql/v8/visibility/versioned

echo 'MySQL schema setup complete'
```

Note that the script uses
[temporal-sql-tool](https://github.com/temporalio/temporal/blob/main/tools/sql/main.go)
to run the setup.

---

# Set up PostgreSQL Visibility store

Source: https://docs.temporal.io/self-hosted-guide/visibility/postgresql

> How to set up a PostgreSQL Visibility store for a self-hosted Temporal Service.

> **💡 Tip:**
> Support, stability, and dependency info
>
> - PostgreSQL v9.6 and later.
> - Advanced Visibility is available on PostgreSQL v12 and later with Temporal Server v1.20 and later.
> - PostgreSQL v9.6 through v11 support applied to older standard Visibility deployments before Temporal Server v1.24. We
>   recommend upgrading to PostgreSQL 12 or later.
>

You can set PostgreSQL as your [Visibility store](/temporal-service/visibility). Verify
[supported versions](/self-hosted-guide/visibility) before you proceed.

If using PostgreSQL v12 or later as your Visibility store with Temporal Server v1.20 and later, any
[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a
Namespace in that Temporal Service.

## Persistence configuration

Set your PostgreSQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `postgres-visibility` and define the datastore configuration
in your Temporal Service configuration YAML.

```yaml
#...
persistence:
  #...
  visibilityStore: postgres-visibility
  #...
  datastores:
    default:
    #...
    postgres-visibility:
      sql:
        pluginName: 'postgres12' # For PostgreSQL v12 and later. For earlier versions, use "postgres" plugin.
        databaseName: 'temporal_visibility'
        connectAddr: ' ' # remote address of this database; for example, 127.0.0.0:5432
        connectProtocol: ' ' # protocol example: tcp
        user: 'username_for_auth'
        password: 'password_for_auth'
        maxConns: 2
        maxIdleConns: 2
        maxConnLifetime: '1h'
#...
```

To enable advanced Visibility features on your PostgreSQL Visibility store, upgrade to PostgreSQL v12 or later with
Temporal Server v1.20 or later. See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) for details on
how to upgrade your Temporal Server and database schemas.

## Database schema and setup

Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for
[PostgreSQL v12 and later](https://github.com/temporalio/temporal/tree/main/schema/postgresql/v12/visibility)

The following example shows how to set up your PostgreSQL as both persistence and Visibility store using
`temporal-sql-tool`. Refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples
with different databases.

[compose/scripts/setup-postgres.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-postgres.sh)
```sh
set -eu

# Validate required environment variables
: "${POSTGRES_SEEDS:?ERROR: POSTGRES_SEEDS environment variable is required}"
: "${POSTGRES_USER:?ERROR: POSTGRES_USER environment variable is required}"

echo 'Starting PostgreSQL schema setup...'
echo 'Waiting for PostgreSQL port to be available...'
nc -z -w 10 ${POSTGRES_SEEDS} ${DB_PORT:-5432}
echo 'PostgreSQL port is available'

# Create and setup temporal database
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal create
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal setup-schema -v 0.0
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal update-schema -d /etc/temporal/schema/postgresql/v12/temporal/versioned

# Create and setup visibility database
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility create
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility setup-schema -v 0.0
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility update-schema -d /etc/temporal/schema/postgresql/v12/visibility/versioned

echo 'PostgreSQL schema setup complete'
```

Note that the script uses
[temporal-sql-tool](https://github.com/temporalio/temporal/blob/main/tools/sql/main.go)
to run the setup.

## Maintaining Visibility index health 

On high-volume deployments, the total size of the indexes on `executions_visibility` can grow much larger than the table itself and keep growing even with a Retention Period set. This is expected PostgreSQL b-tree behavior rather than a data leak:

- Each pre-allocated custom and system Search Attribute has its own nullable column and its own index. PostgreSQL b-tree indexes store entries for `NULL` values, so a Search Attribute that a deployment never sets still has one index entry for every Workflow Execution row.
- Closing or updating a Workflow Execution upserts its Visibility row, and because these indexes order on `close_time`, the updates leave dead index entries over time.
- `VACUUM` reclaims table (heap) space but does not compact b-tree index bloat. `REINDEX` rebuilds an index and reclaims that space; removing rows through the Retention Period does not.

To reclaim index space online, run `REINDEX` with `CONCURRENTLY`, which does not block reads or writes. It needs roughly the size of the index in extra disk space and I/O, so prefer a lower-traffic window:

```sql
REINDEX TABLE CONCURRENTLY executions_visibility;
```

You can also target a single index with `REINDEX INDEX CONCURRENTLY <index_name>;`.

Because `executions_visibility` is update-heavy, more aggressive autovacuum settings for the table help keep dead tuples, and the resulting index bloat, under control:

```sql
ALTER TABLE executions_visibility SET (
  autovacuum_vacuum_scale_factor = 0.05,
  autovacuum_analyze_scale_factor = 0.05
);
```

Track sizes with `pg_relation_size('executions_visibility')` and `pg_indexes_size('executions_visibility')`, and use the [`pgstattuple`](https://www.postgresql.org/docs/current/pgstattuple.html) extension for precise bloat estimates. For sustained high-volume Visibility workloads, consider [Elasticsearch or OpenSearch](/self-hosted-guide/visibility/elasticsearch), which Temporal recommends for production at scale.

---

# Set up SQLite Visibility store

Source: https://docs.temporal.io/self-hosted-guide/visibility/sqlite

> How to set up a SQLite Visibility store for a self-hosted Temporal Service.

> **💡 Tip:**
> Support, stability, and dependency info
>
> - SQLite v3.31.0 and later.
>

You can set SQLite as your [Visibility store](/temporal-service/visibility). Verify
[supported versions](/self-hosted-guide/visibility) before you proceed.

Temporal supports only an in-memory database with SQLite; this means that the database is automatically created when
Temporal Server starts and is destroyed when Temporal Server stops.

You can change the configuration to use a file-based database so that it is preserved when Temporal Server stops.
However, if you use a file-based SQLite database, upgrading your database schema to enable advanced Visibility features
is not supported; in this case, you must delete the database and create it again to upgrade.

If using SQLite v3.31.0 and later as your Visibility store with Temporal Server v1.20 and later, any
[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a
Namespace in that Temporal Service.

## Persistence configuration

Set your SQLite Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `sqlite-visibility` and define the datastore configuration in
your Temporal Service configuration YAML.

```yaml
persistence:
  # ...
  visibilityStore: sqlite-visibility
  # ...
  datastores:
    # ...
    sqlite-visibility:
      sql:
        user: 'username_for_auth'
        password: 'password_for_auth'
        pluginName: 'sqlite'
        databaseName: 'default'
        connectAddr: 'localhost'
        connectProtocol: 'tcp'
        connectAttributes:
          mode: 'memory'
          cache: 'private'
        maxConns: 1
        maxIdleConns: 1
        maxConnLifetime: '1h'
        tls:
          enabled: false
          caFile: ''
          certFile: ''
          keyFile: ''
          enableHostVerification: false
          serverName: ''
```

SQLite (v3.31.0 and later) has advanced Visibility enabled by default.

## Database schema and setup

Visibility data is stored in a database table called `executions_visibility` that must be set up according to the
schemas defined (by supported versions) in
https://github.com/temporalio/temporal/blob/main/schema/sqlite/v3/visibility/schema.sql.

For an example of setting up the SQLite schema, see
[Temporalite](https://github.com/temporalio/temporalite/blob/main/server.go) setup.

---

# Sending Signals, Queries, & Updates

Source: https://docs.temporal.io/sending-messages

> Signals, Queries, and Updates facilitate interactions with Workflow Executions.

This section will help you write clients that send messages to Workflows which includes:

- [Sending Signals](#sending-signals)
- [Sending Updates](#sending-updates)
- [Sending Queries](#sending-queries)

### Sending Signals 

You can send Signals from any Temporal Client, the Temporal CLI, or you can Signal one Workflow to another.

You can also Signal-With-Start to lazily initialize a Workflow while sending a Signal.

#### Send a Signal from a Temporal Client or the CLI

**Related:**

- [Send a Signal using the Temporal CLI](/cli/command-reference/workflow#signal)
- [Send Signals with the Go SDK](/develop/go/workflows/message-passing#send-signal-from-client)
- [Send Signals with the Java SDK](/develop/java/workflows/message-passing#send-signal-from-client)
- [Send Signals with the PHP SDK](/develop/php/workflows/message-passing#send-signal-from-client)
- [Send Signals with the Python SDK](/develop/python/workflows/message-passing#send-signal-from-client)
- [Send Signals with the TypeScript SDK](/develop/typescript/workflows/message-passing#send-signal-from-client)
- [Send Signals with the .NET SDK](/develop/dotnet/workflows/message-passing#send-signal-from-client)
- [Send Signals with the Ruby SDK](/develop/ruby/workflows/message-passing#signals)
- [Send Signals with the Rust SDK](/develop/rust/workflows/message-passing#signals)

#### Send a Signal from one Workflow to another

**Related:**

- [Send Signals from Workflows with the Go SDK](/develop/go/workflows/message-passing#send-signal-from-workflow)
- [Send Signals from Workflows with the Java SDK](/develop/java/workflows/message-passing#send-signal-from-workflow)
- [Send Signals from Workflows with the PHP SDK](/develop/php/workflows/message-passing#send-signal-from-workflow)
- [Send Signals from Workflows with the Python SDK](/develop/python/workflows/message-passing#send-signal-from-workflow)
- [Send Signals from Workflows with the TypeScript SDK](/develop/typescript/workflows/message-passing#send-signal-from-workflow)
- [Send Signals from Workflows with the .NET SDK](/develop/dotnet/workflows/message-passing#send-signal-from-workflow)
- [Send Signals from Workflows with the Ruby SDK](/develop/ruby/workflows/message-passing#send-signal-from-workflow)
- [Send Signals from Workflows with the Rust SDK](/develop/rust/workflows/message-passing#from-a-workflow)

#### Signal-With-Start 

Signal-With-Start is a great tool for lazily initializing Workflows. When you send this operation, if there is a running Workflow Execution with the given Workflow Id, it will be Signaled. Otherwise, a new Workflow Execution starts and is immediately sent the Signal.

**Related:**

- [Signal-With-Start using the Go SDK](/develop/go/workflows/message-passing#signal-with-start)
- [Signal-With-Start using the Java SDK](/develop/java/workflows/message-passing#signal-with-start)
- [Signal-With-Start using the PHP SDK](/develop/php/workflows/message-passing#signal-with-start)
- [Signal-With-Start using the Python SDK](/develop/python/workflows/message-passing#signal-with-start)
- [Signal-With-Start using the TypeScript SDK](/develop/typescript/workflows/message-passing#signal-with-start)
- [Signal-With-Start using the .NET SDK](/develop/dotnet/workflows/message-passing#signal-with-start)
- [Signal-With-Start using the Ruby SDK](/develop/ruby/workflows/message-passing#signal-with-start)
- [Signal-With-Start using the Rust SDK](/develop/rust/workflows/message-passing#signal-with-start)

### Sending Updates 

> **📝 Note:**
>
> To use the Workflow Update feature in versions prior to v1.25.0, it must be manually enabled.
>
> Set the [frontend.enableUpdateWorkflowExecution](https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go) and [frontend.enableUpdateWorkflowExecutionAsyncAccepted](https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go) dynamic config values to `true`.
>
> For example, with the Temporal CLI, run these commands:
>
> ```command
> temporal server start-dev --dynamic-config-value frontend.enableUpdateWorkflowExecution=true
> temporal server start-dev --dynamic-config-value frontend.enableUpdateWorkflowExecutionAsyncAccepted=true
> ```
>

Updates can be sent from a Temporal Client or the Temporal CLI to a Workflow Execution. This call is synchronous and will call into the corresponding Update handler. If you’d rather make an asynchronous request, you should use Signals.

In most languages (except Go), you may call `executeUpdate` to complete an Update and get its result.

Alternatively, to start an Update, you may call `startUpdate` and pass in the Workflow Update Stage as an argument. You have two choices on what to await:

- Accepted - wait until the Worker is contacted, which ensures that the Update is persisted. See [Update Validators](/handling-messages#update-validators) for more information.
- Completed - wait until the handler finishes and returns a result. (This is equivalent to `executeUpdate`.)

The start call will give you a handle you can use to track the Update, determine whether it was Accepted, and ultimately get its result or an error.

If you want to send an Update to another Workflow such as a Child Workflow from within a Workflow, you should do so within an Activity and use the Temporal Client as normal.

There are limits on the total number of Updates that may occur during a Workflow Execution run, and also on the number of concurrent in-progress Updates that a Workflow Execution may have.
Use [Update Validators](/handling-messages#update-validators) and [Update IDs](/handling-messages#exactly-once-message-processing) to stay within the system limits in both [Cloud](/evaluate/cloud/limits#per-workflow-execution-update-limits) and [Self-Hosted](/self-hosted-guide/defaults).

**Related:**

- [Send Updates in Go](/develop/go/workflows/message-passing#send-update-from-client)
- [Send Updates in Java](/develop/java/workflows/message-passing#send-update-from-client)
- [Send Updates in PHP](/develop/php/workflows/message-passing#send-update-from-client)
- [Send Updates in Python](/develop/python/workflows/message-passing#send-update-from-client)
- [Send Updates in TypeScript](/develop/typescript/workflows/message-passing#send-update-from-client)
- [Send Updates in .NET](/develop/dotnet/workflows/message-passing#send-update-from-client)
- [Send Updates in Ruby](/develop/ruby/workflows/message-passing#send-update-from-client)
- [Send Updates in Rust](/develop/rust/workflows/message-passing#send-update-from-client)

#### Update-With-Start 

> **💡 Tip:**
>
> For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended.
>

Update-with-Start sends an Update request, starting a Workflow if necessary.
A [`WorkflowIDConflictPolicy`](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) must be specified.
Workflow ID and Update ID can be used as idempotency keys as follows:

- If the Workflow exists and you provided an Update ID, and the Update exists in the latest Workflow Run, then Update-With-Start attaches to the existing Update (regardless of `WorkflowIDConflictPolicy`)
  - If the Workflow is closed, it attaches only if the Update has completed.
- Otherwise it uses [`WorkflowIDConflictPolicy`](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) and [`WorkflowIDReusePolicy`](/workflow-execution/workflowid-runid#workflow-id-reuse-policy) as usual to determine whether to start a Workflow, and then starts a new Update immediately.

Update-With-Start is great for latency-sensitive use cases:

- **Lazy Initialization** -
  Instead of making separate Start Workflow and Update Workflow calls, Update-With-Start allows you to send them together in a single roundtrip.
  For example, a shopping cart can be modeled using Update-With-Start.
  Updates let you add and remove items from the cart.
  Update-With-Start lets the customer start shopping, whether the cart already exists or they've just started shopping.
  It ensures the cart, modeled by a Workflow Execution, exists before applying any Update that changes the state of items within the cart.
  Set your `WorkflowIDConflictPolicy` to `USE_EXISTING` for this pattern.
- **Early Return** -
  Using Update-With-Start you can begin a new Workflow Execution and synchronously receive a response, while the Workflow Execution continues to run to completion.
  For example, you might model a payment process using Update-With-Start.
  This allows you to send the payment validation results back to the client synchronously, while the transaction Workflow continues in the background.
  Set your `WorkflowIDConflictPolicy` to `FAIL` and use a unique Update ID for this pattern if you want to assert it does not reuse an existing Workflow.

> **⚠️ Caution:**
>
> Unlike Signal-with-Start - Update-With-Start is _not_ atomic.
> If the Update can't be delivered, for example, because there's no running Worker available, a new Workflow Execution will still start.
> The SDKs will retry the Update-With-Start request, but there is no guarantee that the Update will succeed.
>

**Related:**

- [Update-With-Start with the Go SDK](/develop/go/workflows/message-passing#update-with-start)
- [Update-With-Start with the Java SDK](/develop/java/workflows/message-passing#update-with-start)
- [Update-With-Start with the PHP SDK](/develop/php/workflows/message-passing#update-with-start)
- [Update-With-Start with the Python SDK](/develop/python/workflows/message-passing#update-with-start)
- [Update-With-Start with the TypeScript SDK](/develop/typescript/workflows/message-passing#update-with-start)
- [Update-With-Start with the .NET SDK](/develop/dotnet/workflows/message-passing#update-with-start)
- [Update-With-Start with the Ruby SDK](/develop/ruby/workflows/message-passing#update-with-start)
- [Update-With-Start with the Rust SDK](/develop/rust/workflows/message-passing#update-with-start)

### Sending Queries 

Queries can be sent from a Temporal Client or the Temporal CLI to a Workflow Execution--even if this Workflow has Completed. This call is synchronous and will call into the corresponding Query handler.
You can also send a built-in "Stack Trace Query" for debugging.

**Related:**

- [Send a Query using the Temporal CLI](/cli/command-reference/workflow#query)
- [Send a Query with the Go SDK](/develop/go/workflows/message-passing#send-query)
- [Send a Query with the Java SDK](/develop/java/workflows/message-passing#send-query)
- [Send a Query with the PHP SDK](/develop/php/workflows/message-passing#send-query)
- [Send a Query with the Python SDK](/develop/python/workflows/message-passing#send-query)
- [Send a Query with the TypeScript SDK](/develop/typescript/workflows/message-passing#send-query)
- [Send a Query with the .NET SDK](/develop/dotnet/workflows/message-passing#send-query)
- [Send a Query with the Ruby SDK](/develop/ruby/workflows/message-passing#send-query)
- [Send a Query with the Rust SDK](/develop/rust/workflows/message-passing#send-query)

#### Stack trace Query 

In many SDKs, the Temporal Client exposes a predefined `__stack_trace` Query that returns the call stack of all the threads owned by that Workflow Execution.
This is a great way to troubleshoot a Workflow Execution in production.
For example, if a Workflow Execution has been stuck at a state for longer than an expected period of time, you can send a `__stack_trace` Query to return the current call stack.
The `__stack_trace` Query name does not require special handling in your Workflow code.

> **📝 Note:**
>
> Stack Trace Queries are available only for running Workflow Executions.
>

---

# Serverless Workers

Source: https://docs.temporal.io/serverless-workers

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview. Amazon Bedrock AgentCore Runtime support is in
> Pre-release. To request AgentCore Runtime access on Temporal Cloud, create a
> [support ticket](/evaluate/cloud/support#support-ticket) or contact your account team. You can also
> [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or try them through self-hosting.

This page covers the following:

- [What is a Serverless Worker?](#serverless-worker)
- [How Serverless invocation works](#how-invocation-works)
- [Autoscaling](#autoscaling)
- [Scaling with long-lived Workers](#scaling-with-long-lived-workers)
- [Worker lifecycle](#worker-lifecycle)
- [Failure handling](#failure-handling)
- [Constraints](#constraints)
- [Worker Versioning with Serverless Workers](#worker-versioning-with-serverless-workers)

## What is a Serverless Worker? 

A Serverless Worker is a Temporal Worker that runs on serverless compute instead of a long-lived process. There is no
always-on infrastructure to provision or scale. You configure a compute provider, and Temporal starts and stops Workers
in response to work on a Task Queue.

A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker, and registers Workflows and
Activities the same way. What differs is that Temporal manages the Worker's lifecycle rather than you running a Worker
process. How that lifecycle works depends on the compute provider: AWS Lambda runs short-lived invocations, Amazon
Bedrock AgentCore Runtime runs sessions with idle and maximum-lifetime limits, and GCP Cloud Run runs a pool of
long-lived instances. See [Worker lifecycle](#worker-lifecycle).

Serverless Workers require [Worker Versioning](/worker-versioning). Each Serverless Worker must be associated with a
[Worker Deployment Version](/worker-versioning#deployment-versions) that has a compute provider configured.

To deploy a Serverless Worker, see
[Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

> **💡 Tip:**
> Use the Serverless Workers Skill
>
> The [Temporal Serverless Workers Skill](https://github.com/temporalio/skill-temporal-serverless) gives your AI coding
> agent the knowledge to write, package, deploy, and troubleshoot Serverless Workers. See
> [Develop with AI](/with-ai#skills) for installation instructions.
>

### Compute providers 

A compute provider is the configuration that tells Temporal how to invoke a Serverless Worker. The compute provider is
set on a [Worker Deployment Version](/worker-versioning#deployment-versions) and specifies the provider type, the
invocation target, and the credentials Temporal needs to trigger the invocation.

For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role Temporal assumes to invoke
it. A GCP Cloud Run compute provider includes the Worker Pool's project, region, and name, and the service account
Temporal impersonates to scale it.

Compute providers are only needed for Serverless Workers. Traditional long-lived Workers do not require a compute
provider because the Worker process lifecycle is not managed by the Temporal server.

Temporal supports three compute providers:

| Provider                       | Description |
| ------------------------------ | ----------- |
| AWS Lambda                     | Temporal assumes an IAM role in your AWS account to invoke a Lambda function. |
| Amazon Bedrock AgentCore Runtime | Temporal assumes an IAM role in your AWS account to invoke an AgentCore Runtime endpoint. |
| GCP Cloud Run                  | Temporal scales a Cloud Run [Worker Pool](https://cloud.google.com/run/docs/resource-model#worker-pools) through the Cloud Run admin API. A Worker Pool is its own Cloud Run resource type, distinct from a Service or a Job. |

## How Serverless invocation works 

With long-lived Workers, you start the Worker process, which connects to Temporal and polls a Task Queue for work.
Temporal does not need to know anything about the Worker's infrastructure.

With Serverless Workers, Temporal starts the Worker.

### Worker Controller Instance 

The Worker Controller Instance (WCI) is a system Workflow that scales Serverless Workers based on Task Queue conditions.
One WCI Workflow runs per Worker Deployment Version that has a compute provider configured. The WCI runs in the same
Namespace as your Worker Deployment.

The WCI adjusts the Worker count based on Task Queue conditions. Sync match failures and Task Queue backlog signal that
more capacity is needed, and the WCI adds Workers by triggering the configured compute provider. It also removes
capacity as work drains. How often the WCI re-evaluates the count, and how it sizes the pool, depends on the compute
provider. For details, see [Autoscaling](#autoscaling).

You can list WCI Workflows in your Namespace:

```bash
temporal workflow list \
  --namespace <NAMESPACE> \
  --query 'TemporalNamespaceDivision = "TemporalWorkerControllerInstance"'
```

WCI Workflow IDs follow the pattern `temporal-sys-worker-controller-instance:<deployment-name>:<build-id>`. You can
inspect a WCI Workflow's history to see its recent Activity results:

```bash
temporal workflow show \
  --namespace <NAMESPACE> \
  --workflow-id 'temporal-sys-worker-controller-instance:<DEPLOYMENT_NAME>:<BUILD_ID>'
```

The following diagram illustrates the invocation flow of a Serverless Worker.

![Serverless invocation flow](https://docs.temporal.io/diagrams/serverless-worker-flow.svg)

The invocation flow works as follows:

1. A Task is submitted (for example, `StartWorkflow` or `ScheduleActivity`).
2. The [Matching Service](/temporal-service/temporal-server#matching-service) attempts to route the Task directly to an
   available Worker (a sync match).
3. If a Worker is available, the Task is routed to that Worker.
4. If no Worker is available (sync match fails), the Matching Service pushes a signal to the WCI, and the WCI triggers
   the configured compute provider to start a Worker.
5. The Serverless Worker starts, creates a Temporal Client, and begins polling the Task Queue.
6. The Worker processes Tasks from the Task Queue.

> **📝 Note:**
>
> Sync match failure is only one of the signals the WCI acts on. It also reacts to Task Queue backlog, and depending on
> the compute provider it periodically re-sizes the pool based on the rate of arriving work. What the WCI monitors and how
> it scales differ by provider. See [Autoscaling](#autoscaling).
>

Whether a Worker handles a single burst of Tasks and exits, or stays running to process Tasks over a long lifetime,
depends on the compute provider's execution model. See [Worker lifecycle](#worker-lifecycle).

## Autoscaling 

The [WCI](#worker-controller-instance) automatically scales Serverless Workers up and down based on Task Queue conditions, including sync match failures and Task Queue backlog.
The autoscaling algorithm differs by compute provider because of differences in cold start latency, invocation duration limits, and provider APIs.
Refer to the autoscaling section for your compute provider:

- [AWS Lambda autoscaling](/serverless-workers/aws-lambda#autoscaling)
- [GCP Cloud Run autoscaling](/serverless-workers/cloud-run#autoscaling)

## Scaling with long-lived Workers 

Serverless Workers can share a Task Queue with long-lived Workers. What the WCI does with the Tasks the long-lived
Workers are already handling depends on the compute provider, and so does the guidance. Refer to the section for your
compute provider:

- [AWS Lambda](/serverless-workers/aws-lambda#sharing-a-task-queue)
- [GCP Cloud Run](/serverless-workers/cloud-run#sharing-a-task-queue)

## Worker lifecycle 

How a Serverless Worker starts, processes Tasks, and shuts down depends on the compute provider's execution model:
short-lived invocations on AWS Lambda, or long-lived pool instances on GCP Cloud Run.
Refer to the lifecycle section for your compute provider:

- [AWS Lambda lifecycle](/serverless-workers/aws-lambda#lifecycle)
- [Amazon Bedrock AgentCore Runtime lifecycle](/serverless-workers/agentcore#lifecycle)
- [GCP Cloud Run lifecycle](/serverless-workers/cloud-run#lifecycle)

## Failure handling 

Serverless Workers rely on Temporal's standard retry and timeout semantics to recover from failures. The following
sections describe common failure scenarios and how they are handled.

### Worker crash 

If a Worker crashes (out of memory, unhandled exception, etc.), the behavior follows standard Temporal retry semantics:

- The Activity Timeout fires after the configured duration.
- Temporal retries the Activity on another Worker.
- No manual intervention is required.

### Provider concurrency limit 

If the compute provider's capacity limit is reached (for example, AWS Lambda account concurrency or a GCP Cloud Run
maximum instance count):

- The WCI cannot add Workers beyond the limit.
- Tasks remain in the Task Queue backlog. No data loss occurs.
- Processing slows until capacity frees up.

### Resource exhaustion across Activity slots 

By default, a single Worker may run multiple Activity slots. A crash or resource exhaustion in one Activity
(for example, out-of-memory from a memory-intensive operation) can affect other Activities running on the same Worker.

To isolate Activities from each other:

- Split Workflow Workers and Activity Workers into separate Worker Deployments.
- Set Activity slots to 1 per Worker.

With single-slot configuration, each Activity gets a dedicated execution environment.

## Constraints 

| Constraint        | Detail                                                                                                                                                                                                                                                          |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Activity duration | Depends on the compute provider. On AWS Lambda, an Activity must finish within the invocation limit (15 minutes maximum), minus the shutdown deadline buffer. On Amazon Bedrock AgentCore Runtime, a microVM session has a maximum lifetime of 8 hours. On GCP Cloud Run, instances are long-lived, so no per-invocation limit applies. See [Worker lifecycle](#worker-lifecycle). |
| Workflow duration | No limit. Workflows of any duration work. A Workflow runs across as many Workers as needed.                                                                                                                           |
| Worker code       | Depends on the compute provider. AWS Lambda uses a serverless Worker package for your SDK. Amazon Bedrock AgentCore Runtime and GCP Cloud Run run standard long-lived Temporal Workers inside provider-specific runtime infrastructure. |
| Versioning        | [Worker Versioning](/worker-versioning) is required. Each Workflow must have an `AutoUpgrade` or `Pinned` behavior, set per-Workflow or as a Worker-level default. See [Worker Versioning](/worker-versioning) for rollout strategies such as ramping, and [Worker Versioning with Serverless Workers](#worker-versioning-with-serverless-workers) for how Worker Deployment Versions map to compute provider primitives. |
| Multi-cloud Replication | Serverless Workers do not support [Multi-cloud Replication](/cloud/high-availability#multi-cloud-replication). They must be invoked from the same cloud provider where the Namespace is currently active. After a failover to a different cloud provider, Serverless Workers can no longer be invoked. This is a temporary restriction during Public Preview. |
| Multi-region Replication | On failover of a Namespace with [Multi-region Replication](/cloud/high-availability) within the same cloud provider, the WCI keeps invoking Workers in the original region unless you manually repoint the compute provider. Compute provider configuration, such as a Lambda ARN or a Cloud Run Worker Pool, is scoped to a single region. See [Serverless Workers and High Availability](/cloud/high-availability#serverless-workers). |

## Worker Versioning with Serverless Workers 

Serverless Workers require [Worker Versioning](/worker-versioning), and the compute provider must invoke a stable,
immutable build for each Worker Deployment Version.
How Worker Deployment Versions map to compute provider primitives differs by provider.
Refer to the versioning section for your compute provider:

- [AWS Lambda versioning](/serverless-workers/aws-lambda#worker-versioning)
- [Amazon Bedrock AgentCore Runtime versioning](/serverless-workers/agentcore#worker-versioning)
- [GCP Cloud Run versioning](/serverless-workers/cloud-run#worker-versioning)

---

# Serverless Workers on Amazon Bedrock AgentCore Runtime

Source: https://docs.temporal.io/serverless-workers/agentcore

> **Pre-release**
> Amazon Bedrock AgentCore Runtime support is in Pre-release, and its APIs may change in backwards-incompatible ways.
> To request Temporal Cloud access, create a [support ticket](/evaluate/cloud/support#support-ticket) or contact your
> account team. You can also [sign up for updates](https://temporal.io/pages/serverless-workers-updates) or
> [try it through self-hosting](/production-deployment/worker-deployments/serverless-workers/agentcore/self-hosted-setup).

This page covers how Serverless Workers run on Amazon Bedrock AgentCore Runtime, including Worker Versioning and the
Runtime session lifecycle.

For a complete tutorial that explains the agent architecture and walks through deployment, see
[Build a durable agent on Amazon Bedrock AgentCore](/guides/durable-agent-on-agentcore). If your Worker and AgentCore
project are already in place, see
[Deploy a Serverless Worker on Amazon Bedrock AgentCore Runtime](/production-deployment/worker-deployments/serverless-workers/agentcore).

On AgentCore Runtime, a Serverless Worker is a standard long-running Temporal Worker that runs inside an AgentCore
Runtime session. When the [Worker Controller Instance (WCI)](/serverless-workers#worker-controller-instance) needs
capacity, it invokes an AgentCore Runtime endpoint. The Runtime starts a Worker, which connects to the Temporal Service
and polls its Task Queue.

## How Temporal and AgentCore work together 

Temporal is the durable execution layer for the agent. A Workflow records the agent's progress, coordinates model and
tool calls, waits for input, and applies retries and timeouts. Model calls and tool calls run as Activities.

AgentCore supplies the compute that hosts Temporal Workers and optional services that those Activities can use.
Configuring AgentCore Runtime as a Serverless Workers compute provider does not automatically configure the other
AgentCore services.

| Concern | Where it belongs | How to use it |
| --- | --- | --- |
| **Agent execution and progress** | Temporal Workflow | Keep the agent loop, completed steps, approvals, and long-running waits in the Workflow so execution can continue on another Worker. |
| **Model and tool operations** | Temporal Activities | Give each external operation its own timeout, Retry Policy, and recorded result. The Temporal Strands integration runs model and tool calls as Activities. |
| **Worker compute** | AgentCore Runtime | Host a Temporal Worker in each Runtime session. Treat the Worker and its process-local state as replaceable. |
| **Credentials** | [AgentCore Identity](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/identity.html) | Resolve credentials when an Activity accesses AWS or third-party services. Do not store credentials in Workflow state. |
| **Tool access and authorization** | [AgentCore Gateway](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-core-concepts.html) and [Policy](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/policy-create-policies.html) | Call governed tools from Activities. Gateway connects the agent to tools, and Policy controls which tool calls are allowed. |
| **Knowledge across conversations** | [AgentCore Memory](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html) | Read or write reusable knowledge through Activities. Memory does not replace Workflow state or Event History. |
| **Runtime and agent telemetry** | [AgentCore Observability](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html) | Use AgentCore telemetry for Runtime, model, and tool behavior. Use Temporal Event History to inspect durable execution progress. |
| **Caches and temporary files** | AgentCore Runtime session | Reuse them while the Runtime compute remains available, but do not require them to continue the Workflow. |

For an implementation of this architecture using Strands and AgentCore Code Interpreter, see
[Build a durable agent on Amazon Bedrock AgentCore](/guides/durable-agent-on-agentcore).

## Choose AgentCore Runtime or AWS Lambda 

AgentCore Runtime and AWS Lambda use the same event-driven autoscaling algorithm and run replaceable Workers. With
either provider, a Workflow can continue for days, months, or longer across multiple Worker processes. Durable Timers,
human approval waits, and waits for external events do not require compute to remain running. Keep durable progress in
the Workflow and treat Worker-local state as replaceable.

| Consideration | AgentCore Runtime | AWS Lambda |
| --- | --- | --- |
| **Primary use** | Agent and tool workloads that use the [AgentCore platform](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html). | General-purpose, event-driven Worker workloads on AWS. |
| **Worker process** | A standard long-running Worker runs as background work in a Runtime session. The Runtime handler defines its idle and drain behavior, subject to AgentCore lifecycle limits. | A Lambda Worker integration starts the Worker, monitors the invocation deadline, and shuts it down before the function ends. |
| **Compute lifetime** | A Runtime session using the serverless microVM compute type can run for up to 8 hours. | One function invocation can run for up to [15 minutes](https://docs.aws.amazon.com/lambda/latest/dg/configuration-timeout.html). A later invocation can continue processing Tasks for the same Workflow. |
| **Process-local reuse** | The Worker can reuse initialization work, in-memory caches, and temporary files while its Runtime compute remains available. A later Task can run on another Worker, so do not depend on this state. | Each invocation is independent. Do not expect process-local state to be available to a later invocation. |
| **Agent services** | AgentCore provides services for identity, tool access, policy, memory, and observability. | Lambda can call AWS services under its execution role, but it does not provide the AgentCore agent platform. |

Both compute providers can serve long-running Workflows because Temporal maintains Workflow state and execution
progress independently of any Worker process. Their differences affect which provider is a better fit for individual
Activity attempts, Worker lifecycle, process-local reuse, and AgentCore services:

| Choose AgentCore Runtime if | Choose AWS Lambda if |
| --- | --- |
| Your application uses AgentCore services for identity, tool access, policy, memory, or observability. | Your Worker needs general-purpose AWS compute but does not use the AgentCore platform. |
| An Activity attempt might need longer than Lambda's 15-minute invocation limit. An AgentCore Runtime session can provide up to 8 hours for uninterrupted work. | Individual Activity attempts finish within the Lambda invocation window or can resume or retry without significant cost. The 15-minute limit does not limit the overall Workflow duration. |
| Reusing initialization work, in-memory caches, or temporary files reduces startup work or latency. | The Worker does not need process-local reuse. |

## Autoscaling 

AgentCore Runtime uses the same event-driven autoscaling model as AWS Lambda. The WCI invokes individual Runtime
sessions when it needs more capacity. It does not manage a target-sized pool of Runtime sessions. For the shared
autoscaling behavior, see [Autoscaling for Serverless Workers on AWS Lambda](/serverless-workers/aws-lambda#autoscaling).

## Worker Versioning 

Serverless Workers require [Worker Versioning](/worker-versioning). Associate each Worker Deployment Version with a
named AgentCore Runtime endpoint that points to one AgentCore Runtime version.

AgentCore creates an immutable Runtime version when you create or update a Runtime. A named endpoint has a stable ARN
and points to a chosen Runtime version. Configure the endpoint ARN as the compute provider for the corresponding Worker
Deployment Version:

```bash
temporal worker deployment create-version \
    --deployment-name my-worker \
    --build-id v1 \
    --aws-agentcore-endpoint-arn <AGENTCORE_RUNTIME_ENDPOINT_ARN> \
    --aws-agentcore-assume-role-arn <INVOKE_ROLE_ARN> \
    --aws-agentcore-assume-role-external-id <EXTERNAL_ID>
```

Use one named endpoint for each Worker Deployment Version. For example, point an endpoint named `temporal-v1` at
AgentCore Runtime version `1` and use its ARN for Temporal Worker Deployment Version `my-worker/v1`.

When you deploy new Worker code, AgentCore creates a new Runtime version. Create another endpoint that points to that
new Runtime version and configure it on a new Worker Deployment Version. Keep the older endpoint while Pinned
Workflows can still need the older Worker code.

> **⚠️ Caution:**
>
> Do not configure a live Worker Deployment Version with AgentCore's `DEFAULT` endpoint. That endpoint moves to the
> latest Runtime version whenever you update the Runtime. Updating code behind a Worker Deployment Version can cause
> non-determinism errors for in-flight Workflows, including Pinned Workflows.
>

For details about AgentCore Runtime versions and endpoints, see [AgentCore Runtime versioning and
endpoints](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agent-runtime-versioning.html).

## Lifecycle 

An AgentCore Runtime session is the compute that runs a Worker, not a durable place to store Workflow state. AgentCore
can resume a session on new compute after the previous compute ends, and a later Task can run on another Worker. Keep
state that a Workflow needs in the Workflow or another durable store.

Unlike an AWS Lambda Worker, an AgentCore Worker does not have a fixed Lambda invocation deadline. Your Runtime handler
starts the Worker as background work. The Worker polls until it drains or AgentCore ends its compute.

Two sets of controls determine when that Worker stops:

- **Worker idle and graceful-shutdown policy**: Your Worker implementation decides when it has been idle, stops
  polling, and waits for in-flight Activities to complete.
- **AgentCore lifecycle settings**: AgentCore can end the session or its compute before the Worker policy does.

The AgentCore lifecycle settings are:

- **Idle Runtime session timeout**: Ends a Runtime session after it has not received an AgentCore Runtime invocation for
  the configured duration. The default is 15 minutes. This is not a Temporal Worker idle timer: polling the Temporal
  Service does not reset it.
- **Maximum lifetime**: Ends the compute running a Runtime session after the configured duration. The default and
  maximum is 8 hours. AgentCore can resume the session on new compute after that.

AgentCore's session idle timeout does not replace a Worker idle policy. It resets with AgentCore Runtime invocations
and does not measure Task Queue activity. To control how long an unused Worker polls, implement a separate shutdown
policy: when its idle condition is met, stop polling and drain in-flight Activities before the Runtime handler returns.
Choose the idle period and drain timeout for your workload, and account for the AgentCore maximum lifetime.

Configure Activity timeouts and, for long-running Activities,
[Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) so a retry can recover if AgentCore
ends the compute before an Activity completes.

For the lifecycle setting ranges and defaults, see [Configure Amazon Bedrock AgentCore lifecycle
settings](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-lifecycle-settings.html).

---

# Serverless Workers on AWS Lambda

Source: https://docs.temporal.io/serverless-workers/aws-lambda

> **Public Preview**

This page covers how Serverless Workers work on AWS Lambda, including the invocation lifecycle, autoscaling behavior, and how Worker Versioning maps to Lambda function versions.

For a step-by-step deployment guide, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda).

## Autoscaling 

The autoscaling algorithm in this section applies to Serverless Workers on AWS Lambda and Amazon Bedrock AgentCore
Runtime. The compute providers have different Worker lifecycles after a scale-out action. For AgentCore Runtime
lifecycle details, see [Serverless Workers on Amazon Bedrock AgentCore Runtime](/serverless-workers/agentcore#lifecycle).

The autoscaling algorithm is event-driven and reactive.
Sync match failure is the primary control signal, and backlog aids sizing.

When the [WCI](/serverless-workers#worker-controller-instance) needs more capacity, it invokes the compute provider to
start new Workers: the Lambda `InvokeFunction` API for Lambda or an AgentCore Runtime endpoint for AgentCore Runtime.
Each call is a discrete action ("start N more Workers"), not a target state. Temporal calls the provider API from
outside your network, so no inbound connection to the Worker is needed. The WCI does not manage a fleet of instances.

### Scale-out 

On sync match failure, the WCI starts new Workers through the compute provider API.

For Lambda, cold start is sub-second to low single-digit seconds, so reactive-only control does not create meaningful
backlog overshoot. The WCI can scale from zero with low latency.

For AgentCore Runtime startup and session behavior, see
[Lifecycle](/serverless-workers/agentcore#lifecycle).

### Scale-in 

The WCI does not maintain a target number of Workers or actively remove capacity. The provider and Worker lifecycle
determine when a Worker stops.

On Lambda, each invocation runs until the Worker has finished processing available Tasks or approaches the 15-minute
execution time limit, then shuts down. There is no drain logic or stabilization window.

On AgentCore Runtime, Worker shutdown and AgentCore session lifecycle settings determine when a Worker stops. See
[Lifecycle](/serverless-workers/agentcore#lifecycle).

### Instance model 

Each invocation is independent.
The Worker starts, creates a fresh client connection, processes multiple Tasks until near the execution time limit, and then shuts down gracefully.
There is no shared state across invocations.

### Sharing a Task Queue with long-lived Workers 

The WCI invokes a Lambda Worker when a sync match fails, or when the Worker Deployment Version's Task Queues have a
backlog. A long-lived fleet that keeps up with the work leaves neither, so no Lambda Workers are invoked. The Serverless
Workers act as spillover capacity for the long-lived fleet.

> **⚠️ Caution:**
>
> Do not enable dynamic scaling on the long-lived Workers sharing the Task Queue. Both scalers react to the same backlog,
> so the long-lived Workers may add capacity for the Tasks Temporal is already invoking Lambda Workers to handle.
>

## Worker Versioning 

Serverless Workers require [Worker Versioning](/worker-versioning), and the compute provider must invoke a stable,
immutable build for each Worker Deployment Version. With AWS Lambda, this means aligning two versioning systems:

- **Temporal Worker Deployment Versions** — identified by deployment name and Build ID. Each Workflow runs against a
  specific Worker Deployment Version (Pinned) or moves between them on routing changes (Auto-Upgrade).
- **AWS Lambda function versions** — immutable numbered snapshots of your Lambda function code (`1`, `2`, `3`, ...).

A Worker Deployment Version is an immutable build identifier. For production workloads, keep the Lambda function code it
invokes immutable as well: map each Worker Deployment Version to exactly one Lambda function version, and configure the
compute provider with the qualified
[versioned ARN](https://docs.aws.amazon.com/lambda/latest/dg/configuration-versions.html) for that Lambda version (for
example, `arn:aws:lambda:us-east-1:123:function:my-worker:5`).

```mermaid
flowchart LR
    subgraph Temporal["Temporal · Worker Deployment 'my-app'"]
        direction TB
        b1["Build ID: v1"]
        b2["Build ID: v2"]
        b3["Build ID: v3<br/>(Current)"]
    end
    subgraph AWS["AWS Lambda · my-temporal-worker"]
        direction TB
        l1["Lambda version 1<br/><code>function:my-temporal-worker:1</code>"]
        l2["Lambda version 2<br/><code>function:my-temporal-worker:2</code>"]
        l3["Lambda version 3<br/><code>function:my-temporal-worker:3</code>"]
    end
    b1 -- qualified ARN --> l1
    b2 -- qualified ARN --> l2
    b3 -- qualified ARN --> l3
```

For development or non-critical workloads, you can use an unqualified ARN to iterate without publishing a new Lambda
function version each time.

> **⚠️ Caution:**
>
> An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Without a versioned ARN,
> deploying replay-unsafe code causes non-determinism errors for in-flight Workflows, even for Workflows annotated as
> Pinned.
>

### How the versioning behavior changes rollout 

The choice of Pinned or Auto-Upgrade controls how _Workflows_ move between Worker Deployment Versions in Temporal. It
does not change how a Worker Deployment Version targets Lambda. Both behaviors expect a versioned ARN that points at one
immutable Lambda function version.
The following table shows what happens to existing Workflows when you set a new Current Version, with and without versioned Lambda ARNs.

| Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN |
| ------------------- | ------------------------- | ---------------------------- |
| **Pinned**          | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. |
| **Auto-Upgrade**    | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. |

For step-by-step instructions on publishing Lambda versions and configuring the compute provider with a versioned ARN,
see
[Publish a Lambda function version](/production-deployment/worker-deployments/serverless-workers/aws-lambda#publish-lambda-version).

## Lifecycle 

A single Serverless Worker invocation has three phases: init, work, and shutdown.

![Serverless Worker lifecycle](https://docs.temporal.io/diagrams/serverless-worker-lifecycle.svg)

During the **init** phase, the Worker initializes and establishes a client connection to Temporal.

During the **work** phase, the Worker polls the Task Queue and processes Tasks.

During the **shutdown** phase, the Worker stops polling, waits for in-flight Tasks to finish, and runs any shutdown
hooks (for example, OpenTelemetry telemetry flushes). Shutdown begins before the invocation deadline so the Worker can
exit cleanly before the compute provider forcibly terminates the execution environment.

### Tuning for long-running Activities

If your Worker handles long-running Activities, set these three values together:

- **Worker stop timeout > longest Activity runtime.** Gives in-flight Activities enough time to finish after polling
  stops.
- **Shutdown deadline buffer > Worker stop timeout + shutdown hook time.** Ensures the drain and any shutdown hooks
  complete before the compute provider terminates the environment.
- **Invocation deadline > longest Activity runtime + shutdown deadline buffer.** Set on the compute provider to give
  each invocation enough total runtime.

> **💡 Tip:**
>
>   If your longest-running Activity runs longer than half the maximum invocation deadline, this constraint may be
>   difficult or impossible to meet. In this case, use
>   [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) to record the state of the
>   Activity execution so that the next retry can pick up where it left off.
>

For example, if your longest Activity runtime is 5 minutes, and your shutdown hooks take 3 seconds to run, set the
Worker stop timeout to more than 5 minutes, and the shutdown deadline buffer to more than 303 seconds (5 minutes + 3
seconds). Set your invocation deadline to at least 10 minutes and 3 seconds (5 minutes + 303 seconds).

The Worker stop timeout controls how long the Worker waits for in-flight Tasks to finish after it stops polling. The
shutdown deadline buffer controls how much time before the invocation deadline the Worker stops polling for Tasks.

Raising only the shutdown deadline buffer makes the Worker stop polling earlier, but does not give in-flight Tasks any
more time to complete.

Raising only the Worker stop timeout does not make the Worker stop polling earlier, which means the compute provider
might terminate the Worker before the full stop timeout completes. In-flight Activities then do not get the full stop
timeout to finish, and the shutdown hooks may not run.

---

# Serverless Workers on GCP Cloud Run

Source: https://docs.temporal.io/serverless-workers/cloud-run

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

This page covers how Serverless Workers work on GCP Cloud Run, including the instance lifecycle, autoscaling behavior, and how Worker Versioning maps to Cloud Run Worker Pools.

On Cloud Run, a Serverless Worker runs in a Worker Pool: a set of long-lived instances that poll the Task Queue continuously for as long as they remain in the pool.
Each instance runs standard Worker code and processes Tasks for its whole lifetime.
The [Worker Controller Instance (WCI)](/serverless-workers#worker-controller-instance) controls how many instances run; each instance manages its own polling and Task processing.

Serverless Workers on Cloud Run use standard long-lived Worker processes, with scaling managed by the WCI.
Some SDKs add GCP-specific conveniences, such as helpers for configuring OpenTelemetry.
These are optional.
Where an SDK provides them, they're documented in the corresponding developer guide for that SDK.

## Autoscaling 

The WCI keeps the Worker Pool sized to the amount of work arriving on the Task Queue. It combines two mechanisms:

- Immediately bring up new instances when a Task arrives and no Worker is free to take it (a sync match failure).
- Resize the Worker Pool periodically based on incoming work.

The immediate reaction absorbs bursts without waiting on the evaluation cycle. The periodic re-sizing is where the WCI does its rate-based planning.

For that planning, the WCI measures two rates over time:

- How fast Tasks arrive.
- How fast a single Worker processes them.

From these, it calculates how many Workers it takes to handle the incoming work, sets the pool's target instance count accordingly, and adjusts it as the arrival rate changes.

The WCI holds back some spare capacity rather than loading every Worker to 100%.
It sizes the pool to a target utilization (80% by default), so there is room to pick up newly arriving Tasks right away instead of queuing them.
When Tasks are already waiting in the backlog, the WCI adds more instances on top to work through them faster.

It applies the resulting count to the pool through the Cloud Run admin API, and keeps the count within a minimum and maximum bound.
Temporal calls that API from outside your network, so no inbound connection to the pool is needed.

### Scale-out 

When arrivals rise or backlog builds, the WCI raises the pool's target instance count and Cloud Run starts more instances.
Because the WCI keeps a utilization buffer, the pool adds capacity ahead of demand rather than waiting until Tasks pile up.
Since a new instance takes time to start and connect before it can poll for Tasks, provisioning ahead of demand keeps the backlog from growing during that startup window.

### Scale-in 

When arrivals and backlog fall, the WCI lowers the target instance count and Cloud Run stops surplus instances.
Scale-in is more conservative than scale-out.
The WCI holds capacity while sync match failures are still occurring, and it applies a cooldown before reducing the pool, so it does not remove instances that are about to be needed again.
The cooldown starts at the most recent sync match failure and defaults to 90 seconds.
Set it with `--gcp-cloud-run-scale-down-stabilization-duration` on `temporal worker deployment create-version` or `update-version`, alongside the other four scaling flags, which requires Temporal CLI v1.8.3 or later.
Raise it to keep the pool from shrinking while long-running Activities are still executing, or set `0s` to scale in as soon as demand drops.
When there is no work, the WCI can scale the pool to zero; the next sync match failure or backlog scales it back up.

### Sharing a Task Queue with long-lived Workers 

Do not share a Task Queue between a Worker Pool and long-lived Workers. The pool scales up to cover the Task Queue's
full workload even when the long-lived Workers are already handling all of it, so you run and pay for pool instances
that duplicate capacity you already have.

The WCI sizes the pool from the rate of Tasks arriving on the Worker Deployment Version's Task Queues, and nothing in
that measurement accounts for the long-lived Workers. Tasks they process raise the pool's target instance count exactly
as unhandled Tasks would. Sync matching to a long-lived Worker suppresses the immediate scale-up when a Task arrives,
but the periodic re-sizing scales the pool up regardless.

## Worker Versioning 

Serverless Workers require [Worker Versioning](/worker-versioning). Create one Worker Pool for each Worker Deployment
Version, and carry the Build ID in the pool name.

The compute configuration names a project, region, and Worker Pool. It does not name a
[revision](https://cloud.google.com/run/docs/managing/revisions), so Temporal runs whichever revision the pool serves at
the time. That ties a pool to one build, and a new build needs a new pool.

Keep an older version's pool in place while Pinned Workflows are still running on it. The pool can sit at zero
instances, and its WCI scales it back up when a Task arrives for that version.

> **⚠️ Caution:**
>
> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run
> promotes it to every instance by default. The version does not change, but the code behind it does. Deploying
> replay-unsafe code this way causes non-determinism errors for in-flight Workflows, including Pinned ones.
>

For step-by-step instructions, see
[Create the Worker Pool](/production-deployment/worker-deployments/serverless-workers/cloud-run#create-worker-pool).

## Lifecycle 

A Cloud Run Serverless Worker is an ordinary long-lived Temporal Worker. Each instance runs the same Worker code you
would run anywhere else: it connects to Temporal, registers its Workflows and Activities, and polls the Task Queue for
its whole lifetime. You do not start or stop instances yourself. Cloud Run runs the containers, and the WCI sets how many
run through its [scaling operations](#autoscaling).

An instance's lifetime is bounded by scale-in. The WCI decides when to remove an instance from Task Queue activity, not
from what any individual instance is doing. It waits a set amount of time after the most recent sync match failure, and
applies a cooldown between successive reductions, before lowering the target instance count. That wait is the
[scale-down stabilization duration](#scale-in). It does not track how long a
given instance has been running or whether that instance is mid-Activity, so the instance Cloud Run stops may be one that
is still executing work.

Use [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) so that if an Activity is
interrupted, a retry resumes from the last recorded progress instead of restarting from the beginning.

---

# Standalone Activity

Source: https://docs.temporal.io/standalone-activity

> Learn about durable job processing with Standalone Activities that run independently from a Workflow.

## What is a Standalone Activity? 

A Standalone [Activity](/activities) is a top-level [Activity Execution](/activity-execution) started directly by a
[Client](/encyclopedia/temporal-client), without using a Workflow.

Standalone Activities are Temporal's [job queue](/evaluate/features/job-queue) - the
simplest way to run durable, retryable background jobs on Temporal. A job is queued, dispatched to one of your Workers,
retried on failure, and kept addressable the whole time.

Use it to run a single Activity reliably - sending an email, processing a webhook, syncing data, transcoding a file.
If you need to orchestrate multiple steps that depend on each other, use a [Workflow](/workflows) instead. Standalone
Activities don't replace Workflows, and you can use both in the same application.

## Coming from another job queue? 

Temporal uses its own names for some job queue concepts, here's how they map:

| In a job queue | In Temporal |
| --- | --- |
| The function a job runs | An [Activity Definition](/activity-definition) - a normal function, registered by name |
| One enqueued job | A **Standalone Activity Execution** - one durable run, addressable by its Activity ID |
| The queue | A [Task Queue](/task-queue) that your Workers poll |
| A worker process | An [Activity Worker](/workers) - your process, running your code |

For the full comparison and a migration path, see
[Job Queue](/evaluate/features/job-queue) and
[Migrate a Celery task queue to a Standalone Activity](/guides/celery-to-standalone-activity).

> **💡 Tip:**
> GET STARTED
>
> Pick your SDK and follow the quickstart:
> [Go](/develop/go/activities/standalone-activities-quickstart)
> | [Java](/develop/java/activities/standalone-activities-quickstart)
> | [Python](/develop/python/activities/standalone-activities-quickstart)
> | [TypeScript](/develop/typescript/activities/standalone-activities-quickstart)
> | [.NET](/develop/dotnet/activities/standalone-activities-quickstart)
> | [Ruby](/develop/ruby/activities/standalone-activities-quickstart)
> | [Rust](/develop/rust/activities/standalone-activities-quickstart)
>

## Key features

**Durable job lifecycle**

Each job is durably persisted before any Worker sees it, so jobs aren't lost.
Workers pull work from a [Task Queue](/task-queue) and there's no head-of-line blocking, so a slow job doesn't block the dispatch of other Tasks.
See [Activity Execution Lifecycle](/activity-execution#activity-execution-lifecycle).

**Retries and timeouts**

Every job carries a [Retry Policy](/encyclopedia/retry-policies) and
[timeouts](/encyclopedia/detecting-activity-failures) you set when you start it. The default is at-least-once: retry
with exponential backoff until the job succeeds or its Schedule-To-Close Timeout elapses. Set Maximum Attempts to 1 for
at-most-once.

Because a retry runs your function again, Activity code should be
[idempotent](/activity-definition#idempotency).

**Long-running jobs and Heartbeats**

Jobs can run for any duration. [Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) to signal
liveness and to checkpoint progress.

A retry restarts your Activity function from the top - it doesn't resume mid-function. The last recorded Heartbeat
details are made available to the next attempt, so your code can read the checkpoint and skip the work it already
finished.

**Deduplicate with the Activity ID**

Use a business identifier you already have as the Activity ID, and Temporal enforces uniqueness for you: an
[Activity ID Conflict Policy](#activity-id-conflict-policy) covers a job that's already running, and an
[Activity ID Reuse Policy](#activity-id-reuse-policy) covers one that already completed.

This is a different problem from idempotency, and you need both. The Activity ID stops you submitting the same job
twice. Idempotent code stops one job's side effects happening twice if retried.

**Priority and fairness**

[Priority](/develop/task-queue-priority-fairness#task-queue-priority) is strict: higher-priority Tasks dispatch before
lower-priority ones. [Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) prevents starvation: each
fairness key gets its own virtual queue and dispatch cycles round-robin across keys, so one tenant's backlog doesn't
starve everyone else.

On Temporal Cloud, enabling Fairness carries a [per-Action surcharge](/evaluate/cloud/actions#fairness).

**Observability**

All existing [Activity metrics](/cloud/metrics/openmetrics/metrics-reference#activity-metrics) apply
to Standalone Activities. This includes counts for scheduled, started, completed, failed, timed out,
and canceled activities.

**Operator commands**

Pause, Unpause, Reset, and Update Options let an operator intervene in a running job from the CLI, the UI, or the gRPC
API.

These commands are in
[Public Preview](/evaluate/product-release-stages#public-preview). Request Cancel, Terminate,
and Delete are Generally Available.

See [Activity Operations](/activity-operations) for behavior, precedence, and batch support.

**Asynchronous completion**

A Standalone Activity can return from its function without completing the Activity Execution, leaving an external
system to Heartbeat progress and deliver the final result by Activity ID or Task Token. See
[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion).

**Reuse Activities for jobs and Workflows**

You define the Activity and register it on an Activity Worker once. The same function runs as a Standalone Activity or
as a step in a Workflow, with no changes to your Activity code or your Worker. Start with background jobs, and add
Workflow orchestration later without a rewrite.

## Schedule a job for later 

Start Delay dispatches the first Activity Task after a delay instead of immediately. Use it for work that shouldn't run
until later, such as a reminder email or a deferred cleanup step.

```bash
temporal activity start \
  --activity-id my-activity \
  --type MyActivity \
  --task-queue my-task-queue \
  --start-to-close-timeout 5m \
  --start-delay 1h
```

The delay applies to the first Activity Task only. Retry attempts are dispatched according to the
[Retry Policy](/encyclopedia/retry-policies), not the delay.

Start Delay schedules one job at a future time. It doesn't create a recurring schedule.

## Visibility 

Query jobs with [List Filter](/list-filter) by type, status, Task Queue, and other attributes, from the SDK or with
`temporal activity list`. See [Search Attributes](/search-attribute) for the attributes set on Standalone Activity
Executions, and add your own to filter on your business data.

`temporal activity list` shows a list of jobs optionally matching a [List Filter](/list-filter).

```
temporal activity list --query "ExecutionStatus='Running'"
  Status          ActivityId             Type       StartTime
  Running  process_files-1786633958  process_files  1 week ago
```

`temporal activity count` returns the total number of Standalone Activity Executions optionally matching a [List Filters](/list-filter), analogous to counting Workflow Executions.

```
temporal activity count --query "GROUP BY ExecutionStatus"
Total: 45
Group total: 30, values: Completed
Group total: 10, values: Canceled
Group total: 4, values: Terminated
Group total: 1, values: Running
```

This is the count of Activity Executions (Completed, Running, Failed, etc.) - not the number of queued tasks.

`temporal activity describe` shows one job's status, attempt count, and last error.

```
temporal activity describe -a process_files-1786633958
Activity Execution Info:
  ActivityId            process_files-1786633958
  RunId                 01a06322-99ac-7972-8829-65352fc5d158
  Type                  process_files
  Status                Running
  RunState              Scheduled
  TaskQueue             demo-task-queue
  StartToCloseTimeout   24h0m0s
  Attempt               1
  ScheduleTime          1 week ago
  StateTransitionCount  3
```

## Lifecycle control 

Because a Standalone Activity has no Workflow to own it, you act on the execution directly by Activity ID:

- **Request Cancel** asks the execution to close gracefully, letting your code clean up. See
  [Cancellation](/activity-execution#cancellation).
- **Terminate** forcefully closes the execution, with no opportunity for your code to clean up.
- **Delete** terminates the execution if it's running, then deletes it asynchronously.

Activities must [Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) to receive Cancellation, so
interrupting an already-running attempt is cooperative: Temporal can accept a Cancel request without the Worker
honoring it. If the Worker is unresponsive, the request takes effect at the next attempt boundary, for example when the
[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) elapses.

Only Terminate and Delete discard Activity progress unconditionally.

## Activity options 

You set Activity Options on the Client when you start the job. At minimum, specify a timeout - typically the
[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).

| Option | What it controls |
| --- | --- |
| [Timeouts](/encyclopedia/detecting-activity-failures) | Schedule-To-Start, Start-To-Close, Schedule-To-Close, and Heartbeat |
| [Retry Policy](/encyclopedia/retry-policies) | Retry backoff and Maximum Attempts. Set Maximum Attempts to 1 for at-most-once |
| Task Queue | Which Workers get the job |
| Activity ID | The deduplication key, with its [Conflict](#activity-id-conflict-policy) and [Reuse](#activity-id-reuse-policy) policies |
| [Priority and fairness](/develop/task-queue-priority-fairness) | Dispatch order when jobs compete for the same Workers |
| [Start Delay](#start-delay) | Run the job after a start delay |

> **💡 Tip:**
> GET STARTED
>
> Get results, get handles, and list Standalone Activities:
> [Go](/develop/go/activities/standalone-activities)
> | [Java](/develop/java/activities/standalone-activities)
> | [Python](/develop/python/activities/standalone-activities)
> | [TypeScript](/develop/typescript/activities/standalone-activities)
> | [.NET](/develop/dotnet/activities/standalone-activities)
> | [Ruby](/develop/ruby/activities/standalone-activities)
> | [Rust](/develop/rust/activities/standalone-activities)
>

## Activity ID and deduplication 

Standalone Activities have a separate ID space from [the Workflow ID space](/workflow-execution/workflowid-runid) and
other Temporal primitives, so the [Activity ID Conflict Policy](#activity-id-conflict-policy) and the
[Activity ID Reuse Policy](#activity-id-reuse-policy) observe only the Standalone Activity ID space for deduplication
and uniqueness.

### What is an Activity ID Reuse Policy? 

An Activity ID Reuse Policy determines whether a [Standalone Activity](/standalone-activity) Execution can start with an
Activity ID that a previous, and now closed, Standalone Activity Execution used. If the request is denied, the Temporal
Service returns an `ActivityExecutionAlreadyStarted` error.

See [Activity ID Conflict Policy](#activity-id-conflict-policy) for resolving a conflict with a running Standalone
Activity Execution.

The Activity ID Reuse Policy can have one of the following values:

- **Allow Duplicate:** The Standalone Activity Execution can start regardless of the closed status of a previous
  Standalone Activity Execution with the same Activity ID.
  **This is the default policy, if one isn't specified.**
- **Allow Duplicate Failed Only:** The Standalone Activity Execution can start only if the previous Standalone Activity
  Execution with the same Activity ID failed, was canceled, was terminated, or timed out.
- **Reject Duplicate:** The Standalone Activity Execution can't start if a previous Standalone Activity Execution has
  the same Activity ID, regardless of its closed status.

These values apply to closed Standalone Activity Executions that are still retained in the Namespace, so the check
reaches back only as far as the [retention period](#result-retention).

### What is an Activity ID Conflict Policy? 

An Activity ID Conflict Policy determines what happens when you start a Standalone Activity with an Activity ID that a
running Standalone Activity Execution already uses. Two Standalone Activity Executions never run at the same time with
the same Activity ID.

See [Activity ID Reuse Policy](#activity-id-reuse-policy) for reusing the Activity ID of a closed Standalone Activity
Execution.

The Activity ID Conflict Policy can have one of the following values:

- **Fail:** Doesn't start a new Standalone Activity Execution and returns an `ActivityExecutionAlreadyStarted` error.
  **This is the default policy, if one isn't specified.**
- **Use Existing:** Doesn't start a new Standalone Activity Execution and returns a handle to the running one.

## Result retention 

A Standalone Activity Execution and its result are retained for the
[Retention Period](/temporal-service/temporal-server#retention-period) of the Namespace it ran in, the same as other
closed Executions. Within that window the Execution stays visible to `temporal activity describe` and
`temporal activity list`. After the Retention Period elapses, the Execution and its result are deleted and the
Activity ID becomes available for reuse.

Retention is also what enforces deduplication: the [Reuse Policy](#activity-id-reuse-policy) checks against the retained
record of a completed job, so a job older than the Retention Period no longer blocks reuse of its Activity ID.

To remove an Execution before then, use `temporal activity delete`.

## Worker configuration 

An Activity Worker's default poller count is lower than the concurrency many job queue frameworks use, and some of them
prefetch several tasks per poll. If you're moving existing work to Standalone Activities and comparing throughput,
match the poller count to your previous system before you measure. Otherwise the comparison reflects poller
configuration rather than the platform.

See [Worker performance](/develop/worker-performance/configuration#configuring-poller-options) for poller autoscaling
and the manual settings.

For long-running Activities, start the Activity and hold the handle rather than blocking on the result, so a Worker
slot and poller aren't held for the duration.

## Serverless Workers 

Job queue load is bursty, so Activity Workers often sit idle between jobs. With
[Serverless Workers](/serverless-workers), Temporal starts the Worker instead: when a job arrives and no Worker is
available to take it, Temporal invokes your configured compute provider, the Worker polls the Task Queue, processes the
job, and scales back down.

Your Activity code and Worker registration are unchanged. The Worker must belong to a
[Worker Deployment Version](/worker-versioning#deployment-versions) with a compute provider configured, which is how
Temporal knows what to invoke.

AWS Lambda support is in [Public Preview](/evaluate/product-release-stages#public-preview) and
GCP Cloud Run is in [Public Preview](/evaluate/product-release-stages#public-preview). See
[Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers).

## Standalone Activity versus Workflow Activity

A Standalone Activity follows the same execution semantics as an Activity in a Workflow: it's queued, retried until it
succeeds or its Schedule-To-Close Timeout elapses, and it requires idempotent Activity code. What differs is that it's
orchestrated by its own state machine, so there's no Workflow [Event History](/workflow-execution/event#event-history)
and no deterministic replay.

Both are durable [Activity Executions](/activity-execution) that use the same Activity Execution
lifecycle. The [Activity Definition](/activity-definition) and Worker registration are identical, so
the same Activity function can run either way with no code changes. What differs is who starts it
and what owns its lifetime.

Running a single Activity as a Standalone Activity also costs fewer [Billable Actions](/cloud/actions-usage#actions-in-workflows) in
Temporal Cloud than wrapping it in a Workflow, and short jobs see lower latency because there are fewer Worker
round-trips. See [cost optimization](/best-practices/cost-optimization#standalone-activities-vs-a-workflow-that-runs-a-single-activity) for details.

## Feature release stages 

Standalone Activities are Generally Available, including Start Delay, Request Cancel, Terminate, and Delete.

These capabilities are in
[Public Preview](/evaluate/product-release-stages#public-preview):

- [Operator commands](/activity-operations): Pause, Unpause, Reset, and Update Options.
- Batch operations by [List Filter](/list-filter): Request Cancel, Terminate, and Delete.

## Limitations 

The following features are not yet supported:

- `TerminateExisting` conflict policy. Use `Fail` or `UseExisting` instead.
- Starting from a recurring [Schedule](/schedule). For a one-time job at a future time, use
  [Start Delay](#start-delay). For recurring work, schedule a Workflow that invokes the Activity, which runs it as a
  [Workflow Activity](/workflow-activity) rather than a Standalone Activity.
- Starting from [Temporal Nexus](/evaluate/features/nexus).
- Batch Reset, Pause, Unpause, Update Options, Complete, and Fail.
- Export for Standalone Activities similar to [Workflow Export](/cloud/export).

## Temporal CLI support

Standalone Activities require [Temporal CLI](https://github.com/temporalio/cli/releases) v1.9.1 or higher and [Temporal Server](https://github.com/temporalio/temporal/releases) v1.32.0 or higher.

Install with Homebrew:

```bash
brew install temporal
```

Or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms.

Verify the installation:

```bash
temporal --version
```

Which should output v1.9.1 or higher, for example:
```
temporal version 1.9.1 (Server 1.32.0, UI 2.53.3)
```

The `temporal activity` subcommand supports Standalone Activities with `start`, `execute`, `result`, `list`, `count`,
`describe`, `cancel`, `terminate`, and `delete`. It also supports the Public Preview operator commands `pause`,
`unpause`, `reset`, and `update-options`. See [Activity Operations](/activity-operations).

## Temporal Cloud support

Standalone Activities are Generally Available in Temporal Cloud, in all [regions](/evaluate/cloud/regions).

Standalone Activity requests have the same [latency SLO](/cloud/slo#latency) as Workflow requests, and List and Count
requests for Activity Executions are covered by the [Visibility availability SLO](/cloud/slo#visibility-availability).
See the [Temporal Cloud SLA](https://temporal.io/sla) for contractual coverage.

> **💡 Tip:**
> RESOURCES
>
> - Try it end to end with the [Standalone Activities demo](/demos/standalone-activities).
> - Build a job queue with priority and fairness:
>   [Go](https://learn.temporal.io/tutorials/go/standalone-activities/),
>   [Java](https://learn.temporal.io/tutorials/java/standalone-activities/),
>   [Python](https://learn.temporal.io/tutorials/python/standalone-activities/),
>   [TypeScript](https://learn.temporal.io/tutorials/typescript/standalone-activities/).
> - Add orchestration when you need it: see [Workflow Activity](/workflow-activity).
>

---

# Standalone Nexus Operation

Source: https://docs.temporal.io/standalone-nexus-operation

> A Client can start a single Nexus Operation directly, without a caller Workflow, using the same Service and Worker setup.

> **Public Preview** — Go, Java, Python, TypeScript, .NET
> Recommended for production use. API breaking changes are kept to a minimum before General Availability.

## What is a Standalone Nexus Operation? 

A Standalone Nexus Operation is a top-level [Nexus Operation Execution](/nexus/operations) started
directly by a [Client](/encyclopedia/architecture/temporal-sdks#temporal-client), without using a caller
Workflow. Instead of calling a Nexus Operation from within a Workflow Definition, you execute it
directly from a Nexus Client created from the Temporal SDK Client and bound to a
[Nexus Endpoint](/nexus/endpoints) and Service.

Standalone Nexus Operations use the same Service contract, Operation handlers, and Worker setup as
Workflow-driven Operations — only the caller side differs. The same Operation can be executed as a
Standalone Nexus Operation and as a Workflow-driven Nexus Operation with no handler code changes.

If you need to orchestrate multiple Nexus Operations, call them from a [Workflow](/workflows). But if
you just need to execute a single Nexus Operation across Namespace boundaries, use a Standalone
Nexus Operation.

### Get started with your SDK

To make your first standalone Nexus call:

1. Create a caller Namespace where durable Nexus Operations will be started.
2. Import the Temporal SDK in your language of choice.
3. Execute a Nexus Operation through your caller Namespace using the Temporal SDK Client.

See the SDK guides below for details:

- [Standalone Nexus Operations - Go](/develop/go/nexus/standalone-operations)
- [Standalone Nexus Operations - Java](/develop/java/nexus/standalone-operations)
- [Standalone Nexus Operations - Python](/develop/python/nexus/standalone-operations)
- [Standalone Nexus Operations - TypeScript](/develop/typescript/nexus/standalone-operations)
- [Standalone Nexus Operations - .NET](/develop/dotnet/nexus/standalone-operations)

## Use cases

### Make Nexus calls from anywhere

Standalone Nexus Operations let you make Nexus calls from anywhere using the Temporal SDK Client.
Any code that can construct a Temporal Client can durably hand off a Nexus Operation to Temporal, such as a UI backend or BFF, a non-Temporal microservice, an HTTP handler, or a script.
You still get automatic retries, built-in rate limiting and circuit breaking, and full execution visibility: the reliability of Nexus without the overhead of a caller (proxy) Workflow.
To weigh this against calling the same Operation from a Workflow, see [Choosing a Nexus approach](/nexus/choosing-an-approach).

## Key features

- Execute any Nexus Operation as a top-level primitive without the overhead of a caller Workflow
- Same Service contract, Operation handlers, and Worker setup as Workflow-driven Operations
- Supports both synchronous and asynchronous Nexus Operations
- At-least-once execution with automatic retries by the Nexus Machinery
- Get a handle to retrieve results, with the Operation token for asynchronous Operations
- List and count Standalone Nexus Operation Executions using [List Filter](/list-filter) queries
- Execute the same Operation from a Workflow or standalone with no handler code changes

## Migrate a single-Operation caller Workflow to a Standalone Nexus Operation 

A common pattern is a caller Workflow whose only purpose is to invoke a single Nexus Operation.
When that's all the Workflow does, you can drop the caller Workflow entirely and call the
Operation as a Standalone Nexus Operation. The handler Namespace's Service contract, Operation
handlers, and Workers do not change — only the caller side does.

Considerations when migrating:

- If the caller Workflow does more than one Operation call, or has business logic between calls,
  keep the Workflow. Standalone Nexus Operations are only a fit when a single top-level Operation
  is all the Workflow did.
- Pick a stable Operation ID if you previously relied on Workflow ID reuse semantics for
  deduplication.
- Update Visibility queries that filtered by caller Workflow attributes.

For SDK-specific syntax, see the per-SDK guide — for example
[Go: Standalone Nexus Operations](/develop/go/nexus/standalone-operations).

## Observability 

You can use [List Filters](/list-filter) to query Standalone Nexus Operation Executions by Endpoint,
Service, Operation, status, and other attributes using the SDK.

`CountNexusOperations` returns the total number of Standalone Nexus Operation Executions matching a
filter. This is the total count of executions (running, completed, failed, etc.) — not the number of
queued tasks.

## Current limitations

Standalone Nexus Operations are at [Public Preview](/evaluate/product-release-stages#public-preview).
API breaking changes are kept to a minimum before General Availability.

The following features are not supported yet:
  - Reset
  - Batch `--query` support for cancel, terminate, and delete
  - UI list-page operator actions
  - HA / multi-region (global) namespaces using selective API forwarding

Delete is supported by the Temporal Service. Neither the Temporal CLI nor the SDKs expose it yet.

## Temporal CLI support

Standalone Nexus Operations require [Temporal CLI](/cli) v1.9.0 or higher, which includes the
`temporal nexus operation` command family as a standard command group. All commands are Experimental.

Install with Homebrew:

```bash
brew install temporal
```

Or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms.

Verify the installation:

```bash
temporal --version
```

Which should output v1.9.0 or higher.

> **⚠️ Warning:**
>
> - If you see `unknown command "nexus operation"`, upgrade to Temporal CLI v1.9.0 or higher.
> - If you see `Standalone Nexus Operations are disabled` when running commands against a local dev server, start the dev server using a matching CLI version. It enables Standalone Nexus Operations by default.
> - If you don't see the Standalone Nexus Operations button in the Temporal UI, make sure the UI is running the latest release and has the `EnableStandaloneNexusOperations` feature flag enabled.
>

The `temporal nexus operation` subcommand supports Standalone Nexus Operations with commands
including: `start`, `execute`, `result`, `list`, `count`, `describe`, `cancel`, and `terminate`.
The Nexus Endpoint must already exist on the server — create it with
[`temporal operator nexus endpoint create`](/cli/command-reference/operator#create-1).

## Temporal Cloud support

Standalone Nexus Operations in Temporal Cloud are available at [Public Preview](/evaluate/product-release-stages#public-preview), with [formal support](/evaluate/cloud/support#support-ticket).

---

# Sticky Execution

Source: https://docs.temporal.io/sticky-execution

> Learn about Sticky Execution and how it optimizes Task processing in Temporal by caching Workflow state locally.

This page discusses [Sticky Execution](#sticky-execution).

## What is a Sticky Execution? 

Workers cache the state of the Workflow they execute.
To make this caching more effective, Temporal employs a performance optimization known as "Sticky Execution", which directs Workflow Tasks to the same Worker that previously processed tasks for a specific Workflow Execution.

### How Sticky Execution works

Once Workflow Execution begins, the Temporal Service schedules a Workflow Task and puts it into a Task Queue with the name you specify.
Any Worker that polls that Task Queue is eligible to accept the Task and begin executing the Workflow.

The Worker that picks up this Workflow Task will continue polling the original Task Queue, but will also begin polling an additional Task Queue, which the Temporal Service shares exclusively with that specific Worker.
This queue, which has an automatically-generated name, is known as a **Sticky Queue**.

The Worker caches the Workflow state in memory, which improves performance by reducing the need to reconstruct the Workflow from its Event History for every Task.
As the Workflow Execution progresses, the Temporal Service schedules additional Workflow Tasks into this Worker-specific Sticky Queue.

If the Worker fails to start a Workflow Task in the Sticky Queue shortly after it's scheduled (within five seconds by default), the Temporal Service disables stickiness for that Workflow Execution.
When stickiness is disabled, the Temporal Service reschedules the Workflow Task in the original queue, allowing any Worker to pick it up and continue the Workflow Execution.

If a Workflow Task fails, the Worker removes that Workflow Execution from its cache (as it's now in an unknown state), which invalidates the Sticky Execution.
The Workflow Task is then put back into the original Task Queue.

### Why Sticky Execution?

The main benefit of Sticky Execution is improved performance.
By caching the Workflow state in memory and directing tasks to the same Worker, it reduces the need to reconstruct the Workflow from its Event History for every Task, which is particularly useful for latency-sensitive Workflows.

Sticky Execution is the default behavior of the Temporal Platform and only applies to Workflow Tasks.
Since Event History is associated with a Workflow, the concept of Sticky Execution is not relevant to Activity Tasks.

- [How to set `StickyScheduleToStartTimeout` on a Worker in Go](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions)

Sticky Executions are the default behavior of the Temporal Platform.

---

# Task Queues

Source: https://docs.temporal.io/task-queue

> Explore the role of Worker Processes in polling Task Queues and executing Tasks.

This page discusses [Task Queues](#task-queue) including [where to set Task Queues](#set-task-queue) and [Task Ordering](#task-ordering).

## What is a Task Queue? 

A Task Queue is a lightweight, dynamically allocated queue that one or more [Worker Entities](/workers#worker-entity) poll for [Tasks](/tasks).
There are three types of Task Queues: Activity Task Queues, Workflow Task Queues, and Nexus Task Queues.

![Task Queue component](https://docs.temporal.io/diagrams/task-queue.svg)

A Nexus Endpoint creates an entry point that separates callers from the underlying Nexus Task Queue.
The Nexus callers only interact with the Nexus Endpoint.
This endpoint routes Nexus Requests to a target Task Queue that's polled by a Nexus Worker.

![Nexus Endpoint component](https://docs.temporal.io/img/encyclopedia/workers/nexus-task-queue.png)

Task Queues are lightweight components that don’t require explicit registration.
They’re created on demand when a Workflow Execution, Activity, or Nexus Operation is invoked, and/or when a Worker Process subscribes to start polling.
When a named Task Queue is created, individual Task Queues for Workflows, Activities, and Nexus are created using the same name.
A Temporal Application can use, and the Temporal Service can maintain, an unlimited number of Task Queues.

Workers poll for Tasks in Task Queues via synchronous RPC.
This implementation offers several benefits:

- A Worker Process polls for a message only when it has spare capacity, avoiding overloading itself.
- In effect, Task Queues enable load balancing across many Worker Processes.
- Task Queues enable [Task Routing](/task-routing), which is the routing of specific Tasks to specific Worker Processes or even a specific process.
- Activity Task Queues support server-side throttling, which enables you to limit the Task dispatching rate to the pool of Worker Processes while still supporting Task dispatching at higher rates when spikes happen.
- Workflow and Activity Tasks persist in a Task Queue.
  When a Worker Process goes down, the messages remain until the Worker recovers and can process the Tasks.
- Nexus and Query Tasks are not persisted.
  Instead, they are sync matched when, and only when, polled by a Worker.
  Sync matching immediately matches and delivers a Task to an available Worker without persisting a Task to the Service database.
  The caller is responsible to retry failed operations.
  Caller Workflows that invoke Nexus Operations will automatically retry Nexus Tasks until exceeding the Schedule-to-Close timeout.
- Worker Processes do not need to advertise themselves through DNS or any other network discovery mechanism.
- Worker Processes connect directly to the Temporal Service for secure communication without needing to open exposed ports.

Any Worker can pick up any Task on a given Task Queue.
You must ensure that if a Worker accepts a Task that it can process that task using one of its registered Workflows, Activities, or Nexus Operation handlers.
This means that all Workers listening to a Task Queue must register all Workflows, Activities, and Nexus Operations that live on that Queue.

There are two exceptions to this "Task Queue Workers with identical registrations" rule.
First, Worker Versioning may be used.
During Worker upgrade binary rollouts, it's okay to have temporarily misaligned registrations.
Second, dynamic Workflows or Activity components may be used.
If a Task arrives with a recognized method signature, the Worker can use a pre-registered dynamic stand-in.

When Workers don't have a registered Workflow, Activity, Nexus Operation, or dynamic Workflow or Activity component for a given Task, the Task will fail with a "Not Found" error.

- "Not Found" Workflow Tasks and Activity Tasks are treated as _retryable_ errors.
- "Not Found" Nexus Operation handlers are _non-retryable_ and must be manually retried from the caller Workflow.

#### Where to set Task Queues 

There are five places where the name of the Task Queue can be set by the developer.

1. A Task Queue must be set when spawning a Workflow Execution:

   - [How to start a Workflow Execution using the Temporal CLI](/cli/command-reference/workflow#start)
   - [How to start a Workflow Execution using the .NET SDK](/develop/dotnet/client/temporal-client#start-workflow)
   - [How to start a Workflow Execution using the Go SDK](/develop/go/client/temporal-client#start-workflow-execution)
   - [How to start a Workflow Execution using the Java SDK](/develop/java/client/temporal-client#start-workflow-execution)
   - [How to start a Workflow Execution using the PHP SDK](/develop/php/client/temporal-client#start-workflow-execution)
   - [How to start a Workflow Execution using the Python SDK](/develop/python/client/temporal-client#start-workflow-execution)
   - [How to start a Workflow Execution using the Ruby SDK](/develop/ruby/client/temporal-client#start-workflow)
   - [How to start a Workflow Execution using the Rust SDK](/develop/rust/client/temporal-client#start-workflow-execution)
   - [How to start a Workflow Execution using the TypeScript SDK](/develop/typescript/client/temporal-client#start-workflow-execution)

2. A Task Queue name must be set when creating a Worker Entity and when running a Worker Process:

   - [How to run a development Worker using the .NET SDK](/develop/dotnet/workers/run-worker-process)
   - [How to run a development Worker using the Go SDK](/develop/go/workers/run-worker-process)
   - [How to run a development Worker using the Java SDK](/develop/java/workers/run-worker-process)
   - [How to run a development Worker using the PHP SDK](/develop/php/workers/run-worker-process#run-a-dev-worker)
   - [How to run a development Worker using the Python SDK](/develop/python/workers/run-worker-process#run-a-dev-worker)
   - [How to run a development Worker using the Ruby SDK](/develop/ruby/workers/run-worker-process#run-worker-process)
   - [How to run a development Worker using the Rust SDK](/develop/rust/workers/worker-process#run-a-dev-worker)
   - [How to run a development Worker using the TypeScript SDK](/develop/typescript/workers/run-worker-process#run-a-dev-worker)<br /><br />
   - [How to connect a Go SDK Worker to Temporal Cloud](/develop/go/workers/run-worker-process#connect-to-temporal-cloud)
   - [How to run a Temporal Cloud Worker using the TypeScript SDK](/develop/typescript/workers/run-worker-process#run-a-temporal-cloud-worker)

   Note that all Worker Entities listening to the same Task Queue name must be registered to handle the exact same Workflows Types, Activity Types, and Nexus Operations.

   If a Worker Entity polls a Task for a Workflow Type or Activity Type it does not know about, it will fail that Task.
   However, the failure of the Task will not cause the associated Workflow Execution to fail.

3. A Task Queue name can be provided when spawning an Activity Execution:

   This is optional.
   An Activity Execution inherits the Task Queue name from its Workflow Execution if one is not provided.

   - [How to start an Activity Execution using the .NET SDK](/develop/dotnet/activities/execution)
   - [How to start an Activity Execution using the Go SDK](/develop/go/activities/execution)
   - [How to start an Activity Execution using the Java SDK](/develop/java/activities/execution)
   - [How to start an Activity Execution using the PHP SDK](/develop/php/activities/execution)
   - [How to start an Activity Execution using the Python SDK](/develop/python/activities/execution)
   - [How to start an Activity Execution using the TypeScript SDK](/develop/typescript/activities/execution)

4. A Task Queue name can be provided when spawning a Child Workflow Execution:

   This is optional.
   A Child Workflow Execution inherits the Task Queue name from its Parent Workflow Execution if one is not provided.

   - [How to start a Child Workflow Execution using the .NET SDK](/develop/dotnet/workflows/child-workflows)
   - [How to start a Child Workflow Execution using the Go SDK](/develop/go/workflows/child-workflows)
   - [How to start a Child Workflow Execution using the Java SDK](/develop/java/workflows/child-workflows)
   - [How to start a Child Workflow Execution using the PHP SDK](/develop/php/workflows/child-workflows)
   - [How to start a Child Workflow Execution using the Python SDK](/develop/python/workflows/child-workflows)
   - [How to start a Child Workflow Execution using the Python SDK](/develop/ruby/workflows/child-workflows)
   - [How to start a Child Workflow Execution using the Rust SDK](/develop/rust/workflows/child-workflows)
   - [How to start a Child Workflow Execution using the TypeScript SDK](/develop/typescript/workflows/child-workflows)

5. A Task Queue name can be provided when creating a Nexus Endpoint.
   Nexus Endpoints route requests to the target Task Queue.
   Nexus Workers poll the target Task Queue to handle the Nexus Tasks, such as starting or cancelling a Nexus Operation.

   - [How to run a Nexus Worker using the Go SDK](/develop/go/nexus/feature-guide#register-a-nexus-service-in-a-worker)
   - [How to run a Nexus Worker using the Java SDK](/develop/java/nexus/feature-guide#register-a-nexus-service-in-a-worker)

#### Task ordering

Task Queues can be scaled by adding partitions.
By [default](/references/dynamic-configuration#service-level-rps-limits) each Task Queue has 4 partitions.

Task Queues with a single partition are almost always first-in, first-out, with rare edge case exceptions.
However, using a single partition limits you to low- and medium-throughput use cases.

In Task Queues with multiple partitions, each task is assigned to a random partition.
Generally partitions will act as FIFO queues, so once a task queue builds up a backlog, the sync match (tasks that can be dispatched immediately) rate will drop to nearly zero because the task queue will instead dispatch tasks from the backlog (that is, async matches) first.

> **📝 Note:**
>
> This section is on the ordering of individual Tasks, and does not apply to the ordering of Workflow Executions, Activity Executions, or [Events](/workflow-execution/event#event) in a single Workflow Execution.
> The order of Events in a Workflow Execution is guaranteed to remain constant once they have been written to that Workflow Execution's [History](/workflow-execution/event#event-history).
>

---

# Task Queues and naming best practices

Source: https://docs.temporal.io/task-queue/naming

> A mismatch in Task Queue names creates separate queues, preventing the Worker from receiving tasks and stalling Workflow Execution.

The Temporal Service maintains a set of Task Queues, which Workers poll to see
what work needs to be done. Each Task Queue is identified by a name, which is
provided to the Temporal Service when launching a Workflow Execution.

## Set the same name on the Client and the Worker

**Python**

**Excerpt of code used to start the Workflow in Python**

```python
client = await Client.connect("localhost:7233", namespace="default")

# Execute a workflow
result = await client.execute_workflow(
    GreetingWorkflow.run,
    name,
    id="my-workflow",
    task_queue="my-task-queue-name",
)
```

**Excerpt of code used to configure the Worker in Python**

```python
worker = Worker(
    client,
    task_queue="my-task-queue-name",
    workflows=[GreetingWorkflow],
    activities=[activities.say_hello],
)
```

**Go**

**Excerpt of code used to start the Workflow in Go**

```go
options := client.StartWorkflowOptions{
    ID:        "my-workflow",
    TaskQueue: "my-task-queue-name",
}

run, err := c.ExecuteWorkflow(ctx, options, ProcessOrderWorkflow, input)
```

**Excerpt of code used to configure the Worker in Go**

```go
w := worker.New(c, "my-task-queue-name", worker.Options{})
```

**Java**

**Excerpt of code used to start the Workflow in Java**

```java
WorkflowOptions options = WorkflowOptions.newBuilder()
        .setWorkflowId("my-workflow")
        .setTaskQueue("my-task-queue-name")
        .build();

MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options);
```

**Excerpt of code used to configure the Worker in Java**

```java
Worker worker = factory.newWorker("my-task-queue-name");
```

**TypeScript**

**Excerpt of code used to start the Workflow in TypeScript**

```typescript
await client.workflow.start(OrderProcessingWorkflow, {
  args: [order],
  taskQueue: 'my-task-queue',
  workflowId: `workflow-order-${order.id},`,
});
```

**Excerpt of code used to configure the Worker in TypeScript**

```typescript
const worker = await Worker.create({
  taskQueue: 'my-task-queue',
  connection,
  workflowsPath: require.resolve('./workflows'),
  activities,
});
```

**.NET**

**Excerpt of code used to start the Workflow in C# and .NET**

```csharp
var options = new WorkflowOptions(
            id: "translation-workflow",
            taskQueue: "my-task-queue");

// Run workflow
var result = await client.ExecuteWorkflowAsync(
    (TranslationWorkflow wf) => wf.RunAsync(input),
    options);
```

**Excerpt of code used to configure the Worker in C# and .NET**

```csharp
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
    .AddAllActivities(activities)
    .AddWorkflow<TestWorkflow>());
```

## Define the name as a shared constant

Since Task Queues are created dynamically when they are first used, a mismatch
between these two values does not result in an error. Instead, it will result
in the creation of two different Task Queues. Consequently, the Worker will
not receive any tasks from the Temporal Service and the Workflow Execution
will not progress. Therefore, we recommend that you define the Task Queue name
in a constant that is referenced by the Client and Worker if possible, as this
will ensure that they always use the same value.

**Python**

**Excerpt of code used to define a constant with the Task Queue name in Python (in a shared.py file)**

```python
TASK_QUEUE_NAME = "my-task-queue-name"
```

**Excerpt of code used to start the Workflow, referencing the constant
defined with the Task Queue name in Python**

```python
from shared import TASK_QUEUE_NAME

...

client = await Client.connect("localhost:7233", namespace="default")

# Execute a workflow
result = await client.execute_workflow(
    GreetingWorkflow.run,
    name,
    id="my-workflow",
    task_queue=TASK_QUEUE_NAME,
)
```

**Excerpt of code used to configure the Worker, referencing the constant
defined with the Task Queue name in Python**

```python
worker = Worker(
    client,
    task_queue=TASK_QUEUE_NAME,
    workflows=[GreetingWorkflow],
    activities=[activities.say_hello],
)
```

**Go**

**Excerpt of code used to define a constant with the Task Queue name in Go**

```go
package app 

const TaskQueueName = "my-taskqueue-name"
```

**Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in Go**

```go
options := client.StartWorkflowOptions{
    ID:        "my-workflow",
    TaskQueue: app.TaskQueueName,
}

run, err := c.ExecuteWorkflow(ctx, options, ProcessOrderWorkflow, input)
```

**Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in Go**

```go
w := worker.New(c, app.TaskQueueName, worker.Options{})
```

**Java**

**Excerpt of code used to define a constant with the Task Queue name in Java**

```java
package app;

public class Constants {

  public static final String taskQueueName = "my-task-queue-name";

}
```

**Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in Java**

```java
WorkflowOptions options = WorkflowOptions.newBuilder()
        .setWorkflowId("my-workflow")
        .setTaskQueue(Constants.taskQueueName)
        .build();

MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options);
```

**Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in Java**

```java
Worker worker = factory.newWorker(Constants.taskQueueName);
```

**TypeScript**

**Excerpt of code used to define a constant with the Task Queue name in TypeScript**

```typescript
const TASK_QUEUE_NAME = 'my-taskqueue-name';
```

**Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in TypeScript**

```typescript
import { TASK_QUEUE_NAME } from './shared';

// additional code would follow

await client.workflow.start(OrderProcessingWorkflow, {
  args: [order],
  taskQueue: TASK_QUEUE_NAME,
  workflowId: `workflow-order-${order.id},`,
});
```

**Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in TypeScript**

```typescript
import { TASK_QUEUE_NAME } from './shared';

// additional code would follow

const worker = await Worker.create({
  taskQueue: TASK_QUEUE_NAME,
  connection,
  workflowsPath: require.resolve('./workflows'),
  activities,
});
```

**.NET**

**Excerpt of code used to define a constant with the Task Queue name in C# and .NET**

```csharp
public static class WorkflowConstants
{
    public const string TaskQueueName = "translation-tasks";
}
```

**Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in C# and .NET**

```csharp
var options = new WorkflowOptions(
            id: "translation-workflow",
            taskQueue: WorkflowConstants.TaskQueueName);

// Run workflow
var result = await client.ExecuteWorkflowAsync(
    (TranslationWorkflow wf) => wf.RunAsync(input),
    options);
```

**Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in C# and .NET**

```csharp
using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions(WorkflowConstants.TaskQueueName)
    .AddAllActivities(activities)
    .AddWorkflow<TranslationWorkflow>());
```

## When a shared constant is not available

A shared constant is not always possible, such as when the Client that starts the Workflow runs on another system or
is written in a different programming language.

---

# Task Routing and Worker sessions

Source: https://docs.temporal.io/task-routing

> Learn about routing Tasks and Worker Sessions.

## What is Task Routing? 

Task Routing is when a Task Queue is paired with one or more Workers, primarily for Activity Task Executions.

This could also mean employing multiple Task Queues, each one paired with a Worker Process.

Task Routing has many applicable use cases.

Some SDKs provide a [Session API](#worker-session) that provides a straightforward way to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names.
It also includes features like concurrent session limitations and worker failure detection.

### Flow control

A Worker that consumes from a Task Queue asks for an Activity Task only when it has available capacity, so it is never overloaded by request spikes.
If Activity Tasks get created faster than Workers can process them, they are backlogged in the Task Queue.

### Throttling

The rate at which each Activity Worker polls for and processes Activity Tasks is configurable per Worker.
Workers do not exceed this rate even if it has spare capacity.
There is also support for global Task Queue rate limiting.
This limit works across all Workers for the given Task Queue.
It is frequently used to limit load on a downstream service that an Activity calls into.

### Specific environments

In some cases, you might need to execute Activities in a dedicated environment.
To send Activity Tasks to this environment, use a dedicated Task Queue.

#### Route Activity Tasks to a specific host

In some use cases, such as file processing or machine learning model training, an Activity Task must be routed to a specific Worker Process or Worker Entity.

For example, suppose that you have a Workflow with the following three separate Activities:

- Download a file.
- Process the file in some way.
- Upload a file to another location.

The first Activity, to download the file, could occur on any Worker on any host.
However, the second and third Activities must be executed by a Worker on the same host where the first Activity downloaded the file.

In a real-life scenario, you might have many Worker Processes scaled over many hosts.
You would need to develop your Temporal Application to route Tasks to specific Worker Processes when needed.

Code samples:

- [Go file processing example](https://github.com/temporalio/samples-go/tree/main/fileprocessing)
- [Java file processing example](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/fileprocessing)
- [PHP file processing example](https://github.com/temporalio/samples-php/tree/master/app/src/FileProcessing)

#### Route Activity Tasks to a specific process

Some Activities load large datasets and cache them in the process.
The Activities that rely on those datasets should be routed to the same process.

In this case, a unique Task Queue would exist for each Worker Process involved.

#### Workers with different capabilities

Some Workers might exist on GPU boxes versus non-GPU boxes.
In this case, each type of box would have its own Task Queue and a Workflow can pick one to send Activity Tasks.

### Multiple priorities

If your use case involves more than one priority, you can create one Task Queue per priority, with a Worker pool per priority.

Alternatively, you can use [Task Queue Priority](/develop/task-queue-priority-fairness), which lets you assign priority levels to Tasks within a single Task Queue.
This approach avoids the overhead of managing multiple Task Queues and separate Worker pools while still ensuring higher-priority Tasks are processed first.

### Versioning

Task Routing is the simplest way to version your code.

If you have a new backward-incompatible Activity Definition, start by using a different Task Queue.

Alternatively, you can use [Worker Versioning](/worker-versioning), which lets you tag Workers with a version and route Workflow and Activity Tasks to specific versions without requiring separate Task Queues.
Worker Versioning supports both Pinned Workflows (which complete on a single Worker Deployment Version) and Auto-Upgrade Workflows (which automatically move to the latest version).

## What is a Worker Session? 

A Worker Session is a feature provided by some SDKs that provides a straightforward API for [Task Routing](#task-routing) to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names.
It also includes features like concurrent session limitations and Worker failure detection.

- [How to use Worker Sessions](/develop/go/workers/sessions)

---

# Tasks

Source: https://docs.temporal.io/tasks

> Learn about the types of Tasks in Temporal and their role in Workflow and Activity Executions.

This page discusses the following:

- [Task](#task)
- [Workflow Task](#workflow-task)
  - [When are Workflow Tasks scheduled?](#when-workflow-tasks-scheduled)
  - [How does a Worker process a Workflow Task?](#how-worker-processes-workflow-task)
  - [How does the SDK know which code to run?](#how-worker-processes-workflow-task)
  - [Workflow Tasks and Determinism](#workflow-task-failures-vs-execution-failures)
  - [Performance characteristics](#workflow-task-execution)
- [Workflow Task Execution](#workflow-task-execution)
- [Workflow Task Failures vs Workflow Execution Failures](#workflow-task-failures-vs-execution-failures)
- [Activity Task](#activity-task)
- [Activity Task Execution](#activity-task-execution)
- [Nexus Task](#nexus-task)
- [Nexus Task Execution](#nexus-task-execution)

## What is a Task? 

A Task is a unit of work for [Workers](/workers). The Temporal Service places Tasks on [Task Queues](/task-queue), and
Workers poll for and process them to advance [Workflows](/workflows), run [Activity](/activities) attempts, or handle
[Nexus](/nexus) requests.

There are three types of Tasks:

- [Workflow Task](#workflow-task)
- [Activity Task](#activity-task)
- [Nexus Task](#nexus-task)

## What is a Workflow Task? 

A Workflow Task advances a [Workflow Execution](/workflow-execution) by one step.

### When are Workflow Tasks scheduled? 

The Temporal Service creates and schedules a new Workflow Task whenever one of the following occurs:

- The Workflow Execution is started
- A Signal is sent to the Workflow
- An Update is sent to the Workflow
- An Activity completes (successfully or with a failure)
- A Timer fires
- A Child Workflow completes
- A Workflow Task fails and needs to be retried

Any event that might affect the Workflow's state triggers a new Workflow Task. The Workflow Task bundles together all
new events that have occurred since the last Workflow Task completed.

### How does a Worker process a Workflow Task? 

When a Worker picks up a Workflow Task, it replays the entire Workflow Execution from the beginning using the Event
History.

- The Worker receives the Workflow Task, which contains the complete Event History for the Workflow Execution
- The Workflow Worker replays the Workflow code from the start, using the Event History to recreate the Workflow's state
- During replay, previously executed operations (like Activity calls or Timers) return their results immediately from
  the Event History instead of executing again
- The replay continues until the Worker reaches a point where it needs to make new progress (a new Activity to schedule,
  a new Timer to set, etc.)
- The Workflow code executes any new decisions and generates Commands
- The Worker sends these Commands back to the Temporal Service, completing the Workflow Task
- The Temporal Service persists the Commands as new Events in the Event History

This replay mechanism makes Temporal Workflows durable and fault-tolerant. If a Worker crashes mid-execution, another
Worker can pick up the Workflow Task and replay the entire history to reconstruct the exact state before continuing

### What is a Workflow Task Execution? 

A Workflow Task Execution occurs when a [Worker](/workers#worker-entity) picks up a [Workflow Task](#workflow-task) and
uses it to make progress on the execution of a [Workflow Definition](/workflow-definition) (also known as a Workflow
function).

Workflow Task Execution is typically very fast (milliseconds). The Worker replays code and makes decisions based on the
Event History. No actual I/O operations occur during replay (Activity results come from history). The time spent in a
Workflow Task is unrelated to how long Activities or Timers take.

## Workflow Task Failures vs Workflow Execution Failures 

Understanding the difference between Workflow Task failures and Workflow Execution failures is essential to working with
Temporal at a deeper level.

**Workflow Task failure** means a Worker can't successfully process a Workflow Task due to infrastructure, Workflow
code, or execution environment issues (not business logic). Common causes include non-determinism, unhandled exceptions,
task timeouts, invalid Commands, or bad binary checksums. The Service automatically retries the task with exponential
backoff, and the Workflow Execution stays Open until a task completes, an operator terminates it, or the Workflow
Execution Timeout is reached. Fixes typically involve correcting code, scaling Workers, or resolving infrastructure
problems.

**Workflow Execution failure** means the Workflow's business logic determines it can't complete. It occurs when Workflow
code throws or returns an error, an Activity failure propagates uncaught, or an external system terminates/cancels the
Workflow. The Workflow closes with a Failed status and does not automatically retry; if a Retry Policy is configured,
the Service starts a new Run with the same Workflow ID and continues retrying until success or exhaustion. Each retry is
a separate Run with its own Event History.

The table summarizes the differences:

| Aspect              | Workflow Task Failure                                  | Workflow Execution Failure                                  |
| ------------------- | ------------------------------------------------------ | ----------------------------------------------------------- |
| **What failed**     | Infrastructure or Workflow code has a bug              | Business logic determined the Workflow cannot succeed       |
| **Workflow state**  | Workflow Execution remains Open                        | Workflow Execution closes (Failed, Terminated, etc.)        |
| **Automatic retry** | Always retried automatically by the Service            | Only retried if a Workflow Retry Policy is configured       |
| **Event History**   | Same Event History continues to grow                   | Each retry run has a separate Event History                 |
| **How to resolve**  | Fix code/infrastructure and redeploy                   | May require business logic changes or external intervention |
| **Visibility**      | Shows as Workflow Task failures in history and metrics | Shows as a Failed Workflow Execution in the UI              |

**Workflow Task failure example:** A new deployment introduces non-determinism, existing Workflows fail Workflow Tasks,
and the executions stay Open and retry. After deploying a fix, the Workflows automatically continue.

**Workflow Execution failure example:** A payment Activity fails due to a declined card, the failure propagates
uncaught, and the Workflow closes as Failed. The customer updates payment details and restarts the order.

## What is an Activity Task? 

An Activity Task runs one attempt of an [Activity](/activities).

### What is an Activity Task Execution? 

An Activity Task Execution occurs when a [Worker](/workers#worker-entity) uses the context provided from the
[Activity Task](#activity-task) and executes the [Activity Definition](/activity-definition) (also known as the Activity Function).

The [ActivityTaskScheduled Event](/references/events#activitytaskscheduled) corresponds to when the Temporal Service
puts the Activity Task into the Task Queue.

The [ActivityTaskStarted Event](/references/events#activitytaskstarted) corresponds to when the Worker picks up the
Activity Task from the Task Queue.

Either [ActivityTaskCompleted](/references/events#activitytaskcompleted) or one of the other Closed Activity Task Events
corresponds to when the Worker has yielded back to the Temporal Service.

The API to schedule an Activity Execution provides an "effectively once" experience, even though there may be several
Activity Task Executions that take place to successfully complete an Activity.

Across retries, an Activity Task can carry forward
[Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) payload from the previous attempt, allowing
long-running Activities to resume from their last checkpoint.

Once an Activity Task finishes execution, the Worker responds to the Temporal Service with a specific Event:

- ActivityTaskCanceled
- ActivityTaskCompleted
- ActivityTaskFailed
- ActivityTaskTerminated
- ActivityTaskTimedOut

## What is a Nexus Task? 

A Nexus Task delivers one Nexus request to start or cancel a [Nexus Operation](/nexus).

### What is a Nexus Task Execution? 

A Nexus Task Execution occurs when a Worker uses the context provided from the Nexus Task and executes an action
associated with a Nexus Operation which commonly includes starting a Nexus Operation using its Nexus Operation handler
plus many additional actions that may be performed on a Nexus Operation.

The NexusOperationScheduled Event corresponds to when the Temporal Service records the Workflow's intent to schedule an
operation.

The NexusOperationStarted Event corresponds to when the Worker picks up the Nexus Task from the Task Queue, starts an
asynchronous Nexus Operation, and returns an Operation token to the caller indicating the asynchronous Nexus Operation
has started.

Either NexusOperationCompleted or one of the other Closed Nexus Operation Events corresponds to when the Nexus Operation
has reached a final state due to successfully completing the operation or unsuccessfully completing the operation in the
case of a failure, timeout, or cancellation.

A Nexus Operation Execution appears to the caller Workflow as a single RPC, while under the hood the Temporal Service
may issue several Nexus Tasks to attempt to start the Operation. Hence, a Nexus Operation Handler implementation should
be idempotent. The WorkflowRunOperation provided by the SDK leverages Workflow ID based deduplication to ensure
idempotency and provide an "effectively once" experience.

A Nexus Task Execution completes when a Worker responds to the Temporal Service with either a RespondNexusTaskCompleted
or RespondNexusTaskFailed call, or when the Task times out.

The Temporal Service interprets the outcome and determines whether to retry the Task or record the progress in a History
Event:

- NexusTaskCompleted
- NexusTaskFailed

---

# What is Temporal?

Source: https://docs.temporal.io/temporal

> Temporal is a scalable platform that ensures the Durable Execution of application code, allowing reliable and resilient Workflow Executions even in the face of failures like network outages or server crashes.

Temporal is a scalable and reliable runtime for durable function executions called [Temporal Workflow Executions](/workflow-execution).

Said another way, it's a platform that guarantees the [Durable Execution](#durable-execution) of your application code.

It enables you to develop as if failures don't even exist.
Your application will run reliably even if it encounters problems, such as network outages or server crashes, which would be catastrophic for a typical application.
The Temporal Platform handles these types of problems, allowing you to focus on the business logic, instead of writing application code to detect and recover from failures.

![The Temporal System](https://docs.temporal.io/diagrams/temporal-system-simple.svg)

## Durable Execution 

Durable Execution in the context of Temporal refers to the ability of a Workflow Execution to maintain its state and progress even in the face of failures, crashes, or server outages.
This is achieved through Temporal's use of an [Event History](/workflow-execution/event#event-history), which records the state of a Workflow Execution at each step.
If a failure occurs, the Workflow Execution can resume from the last recorded event, ensuring that progress isn't lost.

## What is the Temporal Platform? 

> **💡 Tip:**
>
> Watch a short overview of the Temporal Platform:
>
> [Watch: What is the Temporal Platform?](https://www.youtube.com/watch?v=EwweiH2rd7M)
>

The Temporal Platform consists of supervising software called the [Temporal Service](/temporal-service) and application code bundled as [Worker Processes](/workers#worker-process).
Together these components create a runtime for your [Temporal Application](/temporal#temporal-application).

![The Temporal Platform](https://docs.temporal.io/diagrams/temporal-platform-simple.svg)

A Temporal Service consists of the [Temporal Server](https://github.com/temporalio/temporal), written in Go, and a database.

Our software as a service (SaaS) offering, Temporal Cloud, offers an alternative to hosting the Temporal Service yourself.

Worker Processes are hosted and operated by you and execute your code. Workers run using one of our SDKs.

## What is a Temporal Application? 

A Temporal Application is a set of [Temporal Workflow Executions](/workflow-execution).
Each Temporal Workflow Execution has exclusive access to its local state, executes concurrently to all other Workflow Executions, and communicates with other Workflow Executions and the environment via message passing.

Therefore, a Temporal Workflow Execution executes a [Temporal Workflow Definition](/workflow-definition), also called a Temporal Workflow Function, your application code, exactly once and to completion—whether your code executes for seconds or years, in the presence of arbitrary load and arbitrary failures.

## Next steps 

- [Quickstarts](/quickstarts): Build a Hello World Workflow in your language.
- [Developer guides](/develop): SDK feature guides and API references.
- [Develop with AI](/with-ai): Give your coding agent Temporal expertise.
- [Temporal Cloud](/cloud): Deploy and run on Temporal Cloud.
- [Temporal architecture](/encyclopedia/architecture/temporal-architecture): How the platform works under the hood.
- [Understanding Temporal](/evaluate/understanding-temporal): A short overview of Temporal concepts.

---

# Temporal Service

Source: https://docs.temporal.io/temporal-service

> This page provides a comprehensive technical overview of a Temporal Service, detailing its components and subsystems, including the Temporal Server, Frontend Service, History Service, Matching Service, and Worker Service.

> **ℹ️ Info:**
> Please note an important update in our terminology.
>
> We now refer to the Temporal Cluster as the Temporal Service.

This guide provides a comprehensive technical overview of a Temporal Service.

A Temporal Service is the group of services, known as the [Temporal Server](/temporal-service/temporal-server), combined with [Persistence](/temporal-service/persistence) and [Visibility](/temporal-service/visibility) stores, that together act as a component of the Temporal Platform.

See the Self-hosted Temporal Service [production deployment guide](/self-hosted-guide) for implementation guidance.

![A Temporal Service (Server + persistence)](https://docs.temporal.io/diagrams/temporal-cluster.svg)

---

# Archival

Source: https://docs.temporal.io/temporal-service/archival

> Archival is a feature that automatically backs up Event Histories and Visibility records from Temporal Service persistence to a custom blob store.

This page discusses [Archival](#archival).

## What is Archival? 

Use Archival to copy closed Workflow Execution [Event Histories](/workflow-execution/event#event-history) and Visibility records from Temporal Service persistence to blob storage.

- [How to create a custom Archiver](/self-hosted-guide/archival#custom-archiver)
- [How to set up Archival](/self-hosted-guide/archival#set-up-archival)

When a Workflow Execution closes, Temporal schedules close-processing tasks for both Visibility records and Event History archival.
Archival then runs asynchronously after a randomized delay.
By default, that delay is up to 5 minutes (`history.archivalProcessorArchiveDelay`), capped by the Namespace [Retention Period](/temporal-service/temporal-server#retention-period).

The closed execution still stays in Temporal persistence until retention cleanup runs.
For some time, the same closed execution can exist in both persistence and archival storage.
Archival enables Workflow Execution data to persist beyond retention without overwhelming the Temporal Service persistence store.

This feature is helpful for compliance and debugging.

Temporal's Archival feature is considered **experimental** and not subject to normal [versioning and support policy](/temporal-service/temporal-server#versions-and-support).

Archival is not supported when running Temporal through Docker.
It's disabled by default when installing the system manually and when deploying through [helm charts](https://github.com/temporalio/helm-charts/blob/main/charts/temporal/templates/server-configmap.yaml).
It can be enabled in the [config](https://github.com/temporalio/temporal/blob/main/config/development.yaml).

---

# Temporal Service configuration

Source: https://docs.temporal.io/temporal-service/configuration

> A self-hosted Temporal Service uses YAML-based static configuration set at startup, plus dynamic configuration you can update live.

This page discusses the following:

- [Static Configuration](#static-configuration)
- [Dynamic Configuration](#dynamic-configuration)
- [Security Configuration](#temporal-cluster-security-configuration)
- [Observability](#monitoring-and-observation)

## What is Temporal Service configuration? 

Temporal Service configuration is the setup and configuration details of your self-hosted Temporal Service, defined using YAML.
You must define your Temporal Service configuration when setting up your self-hosted Temporal Service.

For details on using Temporal Cloud, see [Temporal Cloud documentation](/cloud).

Temporal Service configuration is composed of two types of configuration: [Static configuration](#static-configuration) and [Dynamic configuration](#dynamic-configuration).

### Static configuration

Static configuration contains details of how the Temporal Service should be set up.
The static configuration is read just once and used to configure service nodes at startup.
Depending on how you want to deploy your self-hosted Temporal Service, your static configuration must contain details for setting up:

- Temporal Services—Frontend, History, Matching, Worker
- Membership ports for the Temporal Services
- Persistence (including History Shard count), Visibility, Archival store setups.
- TLS, authentication, authorization
- Server log level
- Metrics
- Temporal Service metadata
- Dynamic config Client

Static configuration values cannot be changed at runtime.
Some values, such as the Metrics configuration or Server log level can be changed in the static configuration but require restarting the Temporal Service for the changes to take effect.

For details on static configuration keys, see [Temporal Service configuration reference](/references/service-configuration).

For static configuration examples, see [https://github.com/temporalio/temporal/tree/main/config](https://github.com/temporalio/temporal/tree/main/config).

### Dynamic configuration

Dynamic configuration contains configuration keys that you can update in your Temporal Service setup without having to restart the server processes.

All dynamic configuration keys provided by Temporal have default values that are used by the Temporal Service.
You can override the default values by setting different values for the keys in a YAML file and setting the [dynamic configuration client](/references/service-configuration#dynamicconfigclient) to poll this file for updates.
Setting dynamic configuration for your Temporal Service is optional.

Setting overrides for some configuration keys updates the Temporal Service configuration immediately.
However, for configuration fields that are checked at startup (such as thread pool size), you must restart the server for the changes to take effect.

Use dynamic configuration keys to fine-tune your self-deployed Temporal Service setup.

For details on dynamic configuration keys, see [Dynamic configuration reference](/references/dynamic-configuration).

For dynamic configuration examples, see [https://github.com/temporalio/temporal/tree/main/config/dynamicconfig](https://github.com/temporalio/temporal/tree/main/config/dynamicconfig).

## What is Temporal Service security configuration? 

Secure your Temporal Service (self-hosted and Temporal Cloud) by encrypting your network communication and setting authentication and authorization protocols for API calls.

For details on setting up your Temporal Service security, see [Temporal Platform security features](/security).

### mTLS encryption

Temporal supports Mutual Transport Layer Security (mTLS) to encrypt network traffic between services within a Temporal Service, or between application processes and a Temporal Service.

On the self-hosted Temporal Service, configure mTLS in the `tls` section of the [Temporal Service configuration](/references/service-configuration#tls).
mTLS configuration is a [static configuration](#static-configuration) property.

You can then use either the [`WithConfig`](/references/server-options#withconfig) or [`WithConfigLoader`](/references/server-options#withconfigloader) server option to start your Temporal Service with this configuration.

The mTLS configuration includes two sections that serve to separate communication within a Temporal Service and client calls made from your application to the Temporal Service.

- `internode`: configuration for encrypting communication between nodes within the Temporal Service.
- `frontend`: configuration for encrypting the public endpoints of the Frontend Service.

Setting mTLS for `internode` and `frontend` separately lets you use different certificates and settings to encrypt each section of traffic.

### Using certificates for Client connections

Use CA certificates to authenticate client connections to your Temporal Service.

On Temporal Cloud, you can [set your CA certificates in your Temporal Cloud settings](/cloud/certificates) and use the end-entity certificates in your client calls.

On the self-hosted Temporal Service, you can restrict access to Temporal Service endpoints by using the `clientCAFiles` or `clientCAData` property and the [`requireClientAuth`](/references/service-configuration#tls) property in your Temporal Service configuration.
These properties can be specified in both the `internode` and `frontend` sections of the [mTLS configuration](/references/service-configuration#tls).
For details, see the [tls configuration reference](/references/service-configuration#tls).

### Server name specification

On the self-hosted Temporal Service, you can specify `serverName` in the `client` section of your mTLS configuration to prevent spoofing and [MITM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack).

Entering a value for `serverName` enables established connections to authenticate the endpoint.
This ensures that the server certificate presented to any connected client has the specified server name in its CN property.

This measure can be used for `internode` and `frontend` endpoints.

For more information on mTLS configuration, see [tls configuration reference](/references/service-configuration#tls).

### Authentication and authorization

Temporal provides authentication interfaces that can be set to restrict access to your data.
These protocols address three areas: servers, client connections, and users.

Temporal offers two plugin interfaces for authentication and authorization of API calls.

- [`ClaimMapper`](/self-hosted-guide/security#claim-mapper)
- [`Authorizer`](/self-hosted-guide/security#authorizer-plugin)

The logic of both plugins can be customized to fit a variety of use cases.
When plugins are provided, the Frontend Service invokes their implementation before running the requested operation.

## What is Temporal Service observability? 

You can monitor and observe performance with metrics emitted by your self-hosted Temporal Service or by Temporal Cloud.

Temporal emits metrics by default in a format that is supported by Prometheus.
Any metrics software that supports the same format can be used.
Currently, we test with the following Prometheus and Grafana versions:

- **Prometheus >= v2.0**
- **Grafana >= v2.5**

Temporal Cloud exports metrics through an [OpenMetrics endpoint](/cloud/metrics/openmetrics), which an observability platform can scrape and store.

For details on Cloud metrics and setup, see the following:

- [Temporal Cloud metrics reference](/cloud/metrics/openmetrics/metrics-reference)
- [Connect an observability tool](/cloud/metrics/openmetrics/metrics-integrations)

On the self-hosted Temporal Service, expose Prometheus endpoints in your Temporal Service configuration and configure Prometheus to scrape metrics from the endpoints.
You can then set up your observability platform (such as Grafana) to use Prometheus as a data source.

For details on self-hosted Temporal Service metrics and setup, see the following:

- [Temporal Service OSS metrics reference](/references/service-metrics)
- [Set up Prometheus and Grafana to view SDK and self-hosted Temporal Service metrics](/self-hosted-guide/monitoring)

---

# Multi-Cluster Replication

Source: https://docs.temporal.io/temporal-service/multi-cluster-replication

> Multi-Cluster Replication is a feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters, for backup and state reconstruction.

This page discusses the following:

- [Multi-Cluster Replication](#multi-cluster-replication)
- [Namespace Versions](#namespace-versions)
- [Version History](#version-history)
- [Conflict Resolution](#conflict-resolution)
- [Zombie Workflows](#zombie-workflows)
- [Workflow Task Processing](#workflow-task-processing)

## What is Multi-Cluster Replication? 

Multi-Cluster Replication is a feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters, for backup and state reconstruction.
When necessary, for higher availability, Cluster operators can failover to any of the backup Clusters.

Temporal's Multi-Cluster Replication feature is considered **experimental** and not subject to normal [versioning and support policy](/temporal-service/temporal-server#versions-and-support).

When a client connects to a standby Cluster, its requests can be forwarded to the
active Cluster. This is off by default; enable it by setting
[`dcRedirectionPolicy`](/references/service-configuration#dcredirectionpolicy).

When the feature is enabled, Tasks are sent to the Parent Task Queue partition that matches that Namespace, if it exists.

All Visibility APIs can be used against active and standby Clusters.
As a result, [Temporal UI](/web-ui) works with Global Namespaces the same way it works with any other Namespace.
Applications making API calls directly to the Temporal Visibility API continue to work even if a Global Namespace is in standby mode.
However, they might see a lag due to replication delay when querying the Workflow Execution state from a standby Cluster.

## Namespace versions

A _version_ is a concept in Multi-Cluster Replication that describes the chronological order of events per Namespace.

With Multi-Cluster Replication, all Namespace change events and Workflow Execution History events are replicated asynchronously for high throughput.
This means that data across clusters is **not** strongly consistent.
To guarantee that Namespace data and Workflow Execution data will achieve eventual consistency (especially when there is a data conflict during a failover), a **version** is introduced and attached to Namespaces.
All Workflow Execution History entries generated in a Namespace will also come with the version attached to that Namespace.

All participating Clusters are pre-configured with a unique initial version and a shared version increment:

- `initial version < shared version increment`

When performing failover for a Namespace from one Cluster to another Cluster, the version attached to the Namespace will be changed by the following rule:

- for all versions which follow `version % (shared version increment) == (active cluster's initial version)`, find the smallest version which has `version >= old version in namespace`

When there is a data conflict, a comparison will be made and Workflow Execution History entries with the highest version will be considered the source of truth.

When a cluster is trying to mutate a Workflow Execution History, the version will be checked.
A cluster can mutate a Workflow Execution History only if the following is true:

- The version in the Namespace belongs to this cluster, that is
  `(version in namespace) % (shared version increment) == (this cluster's initial version)`
- The version of this Workflow Execution History's last entry (event) is equal or less than the version in the Namespace, that is
  `(last event's version) <= (version in namespace)`

      Namespace version change example

Assuming the following scenario:

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Shared version increment: 10

T = 0: Namespace α is registered, with active Cluster set to Cluster A

```
namespace α's version is 1
all workflows events generated within this namespace, will come with version 1
```

T = 1: namespace β is registered, with active Cluster set to Cluster B

```
namespace β's version is 2
all workflows events generated within this namespace, will come with version 2
```

T = 2: Namespace α is updated to with active Cluster set to Cluster B

```
namespace α's version is 2
all workflows events generated within this namespace, will come with version 2
```

T = 3: Namespace β is updated to with active Cluster set to Cluster A

```
namespace β's version is 11
all workflows events generated within this namespace, will come with version 11
```

## Version history

Version history is a concept which provides a high level summary of version information in regards to Workflow Execution History.

Whenever there is a new Workflow Execution History entry generated, the version from Namespace will be attached.
The Workflow Execution's mutable state will keep track of all history entries (events) and the corresponding version.

        Version history example (without data conflict)

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Shared version increment: 10

T = 0: adding event with event ID == 1 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 1               | 1       |
| -------- | -------------   | --------------- | ------- |
```

T = 1: adding event with event ID == 2 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 2: adding event with event ID == 3 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               |                 |         |
| 3        | 1               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 3: Namespace failover triggered, Namespace version is now 2
adding event with event ID == 4 & version == 2

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               | 4               | 2       |
| 3        | 1               |                 |         |
| 4        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 4: adding event with event ID == 5 & version == 2

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               | 5               | 2       |
| 3        | 1               |                 |         |
| 4        | 2               |                 |         |
| 5        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

Since Temporal is AP, during failover (change of active Temporal Service Namespace), there can exist cases where more than one Cluster can modify a Workflow Execution, causing divergence of Workflow Execution History. Below shows how the version history will look like under such conditions.

      Version history example (with data conflict)

Below, shows version history of the same Workflow Execution in 2 different Clusters.

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Cluster C comes with initial version: 3
- Shared version increment: 10

T = 0:

View in both Cluster B & C

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 3               | 2       |
| 3        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 1: adding event with event ID == 4 & version == 2 in Cluster B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 4               | 2       |
| 3        | 2               |                 |         |
| 4        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 1: namespace failover to Cluster C, adding event with event ID == 4 & version == 3 in Cluster C

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 3               | 2       |
| 3        | 2               | 4               | 3       |
| 4        | 3               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 2: replication task from Cluster C arrives in Cluster B

Note: below are a tree structures

```
| ------------- | ------------- |
| Events        |
| ------------- | ------------- |
| Event ID      | Event Version |
| ------------- | ------------- |
| 1             | 1             |
| 2             | 1             |
| 3             | 2             |
| --------      | ------------- |
|               |
|               |
| --------      | ------------- |  | -------- | ------------- |
| Event ID      | Event Version |  | Event ID | Event Version |
| --------      | ------------- |  | -------- | ------------- |
| 4             | 2             |  | 4        | 3             |
| --------      | ------------- |  | -------- | ------------- |

| --------------- | ------------------- |
| Version History |
| --------------- | ------------------- |
| Event ID        | Version             |
| --------------- | -------             |
| 2               | 1                   |
| 3               | 2                   |
| --------------- | ------------------- |
|                 |
|                 |
| --------------- | -------             |  | --------------- | ------- |
| Event ID        | Version             |  | Event ID        | Version |
| --------------- | -------             |  | --------------- | ------- |
| 4               | 2                   |  | 4               | 3       |
| --------------- | -------             |  | --------------- | ------- |
```

T = 2: replication task from Cluster B arrives in Cluster C, same as above

## Conflict resolution

When a Workflow Execution History diverges, proper conflict resolution is applied.

In Multi-cluster Replication, Workflow Execution History Events are modeled as a tree, as shown in the second example in [Version History](#version-history).

Workflow Execution Histories that diverge will have more than one history branch.
Among all history branches, the history branch with the highest version is considered the `current branch` and the Workflow Execution's mutable state is a summary of the current branch.
Whenever there is a switch between Workflow Execution History branches, a complete rebuild of the Workflow Execution's mutable state will occur.

Temporal Multi-Cluster Replication relies on asynchronous replication of Events across Clusters, so in the case of a failover it is possible to have an Activity Task dispatched again to the newly active Cluster due to a replication task lag.
This also means that whenever a Workflow Execution is updated after a failover by the new Cluster, any previous replication tasks for that Execution cannot be applied.
This results in loss of some progress made by the Workflow Execution in the previous active Cluster.
During such conflict resolution, Temporal re-injects any external Events like Signals in the new Event History before discarding replication tasks.
Even though some progress could roll back during failovers, Temporal provides the guarantee that Workflow Executions won't get stuck and will continue to make forward progress.

Namespace metadata replication and Namespace cache refresh are separate. If you change the active Cluster directly (for example with `temporal operator namespace update --active-cluster` or the `UpdateNamespace` API), the Temporal Service that receives the request applies it locally first and then replicates it asynchronously. Until that replication completes, both Temporal Services can temporarily consider the Namespace active or both can consider it passive, depending on which Temporal Service received the request. After a Temporal Service observes the failover, task processing compares a task's Event Id and failover version with the current mutable state and rejects tasks from a superseded history branch, even if the local Namespace cache entry is stale.

Activity Execution completions are not forwarded across Clusters.
Any outstanding Activities will eventually time out based on the configuration.
Your application should have retry logic in place so that the Activity gets retried and dispatched again to a Worker after the failover to the new Cluster.
Handling this is similar to handling an Activity Task timeout caused by a Worker restarting.
Because conflict resolution can remove an unreplicated Activity completion Event from the current branch, Activities
must be [idempotent](/activity-definition#idempotency) so that another execution does not repeat an external side effect.
For operational guidance, see [How to fail over a Global
Namespace](/self-hosted-guide/multi-cluster-replication#how-to-fail-over-a-global-namespace).

## Zombie Workflows

There is an existing contract that for any Namespace and Workflow Id combination, there can be at most one run (Namespace + Workflow Id + Run Id) open / executing.

Multi-cluster Replication aims to keep the Workflow Execution History as up-to-date as possible among all participating Clusters.

Due to the nature of Multi-cluster Replication (for example, Workflow Execution History events are replicated asynchronously) different Runs (same Namespace and Workflow Id) can arrive at the target Cluster at different times, sometimes out of order, as shown below:

```
+-----------+        +----------------+        +-----------+
| Cluster A |        | Network Layer  |        | Cluster B |
+-----------+        +----------------+        +-----------+
      |                       |                        |
      | Run 1 Replication     |                        |
      |---------------------> |                        |
      |                       |                        |
      | Run 2 Replication     |                        |
      |---------------------> |                        |
      |                       |                        |
      |                       | Run 2 Replication      |
      |                       |--------------------->  |
      |                       |                        |
      |                       | Run 1 Replication      |
      |                       |--------------------->  |
      |                       |                        |
```

Because Run 2 appears in Cluster B first, Run 1 cannot be replicated as "runnable" due to the rule `at most one Run open` (see above), thus the "zombie" Workflow Execution state is introduced.
A "zombie" state is one in which a Workflow Execution which cannot be actively mutated by a Cluster (assuming the corresponding Namespace is active in this Cluster). A zombie Workflow Execution can only be changed by a replication Task.

Run 1 will be replicated similar to Run 2, except when Run 1's execution will become a "zombie" before Run 1 reaches completion.

## Workflow Task processing

In the context of Multi-cluster Replication, a Workflow Execution's mutable state is an entity which tracks all pending tasks.
Prior to the introduction of Multi-cluster Replication, Workflow Execution History entries (events) are from a single branch, and the Temporal Server will only append new entries (events) to the Workflow Execution History.

After the introduction of Multi-cluster Replication, it is possible that a Workflow Execution can have multiple Workflow Execution History branches.
Tasks generated according to one history branch may become invalidated by switching history branches during conflict resolution.

Example:

T = 0: task A is generated according to Event Id: 4, version: 2

```
| -------- | ------------- |
| Events   |
| -------- | ------------- |
| Event ID | Event Version |
| -------- | ------------- |
| 1        | 1             |
| 2        | 1             |
| 3        | 2             |
| -------- | ------------- |
|          |
|          |
| -------- | ------------- |
| Event ID | Event Version |
| -------- | ------------- |
| 4        | 2             | <-- task A belongs to this event |
| -------- | ------------- |
```

T = 1: conflict resolution happens, Workflow Execution's mutable state is rebuilt and history Event Id: 4, version: 3 is written down to persistence

```
| --------- | ------------- |
| Events    |
| --------- | ------------- |
| Event ID  | Event Version |
| --------  | ------------- |
| 1         | 1             |
| 2         | 1             |
| 3         | 2             |
| --------  | ------------- |
|           |
| --------- | ------------- |
|           |                
| --------  | ------------- |                                  | -------- | ------------- |
| Event ID  | Event Version |                                  | Event ID | Event Version |
| --------  | ------------- |                                  | -------- | ------------- |
| 4         | 2             | <-- task A belongs to this event | 4        | 3             | <-- current branch / mutable state
| --------  | ------------- |                                  | -------- | ------------- |
```

T = 2: task A is loaded.

At this time, due to the rebuild of a Workflow Execution's mutable state (conflict resolution), Task A is no longer relevant (Task A's corresponding Event belongs to non-current branch).
Task processing logic will verify both the Event Id and version of the Task against a corresponding Workflow Execution's mutable state, then discard task A.

---

# Persistence

Source: https://docs.temporal.io/temporal-service/persistence

> The Temporal Persistence store is a database used by the Temporal Server to persist events generated and processed in your Temporal Service and SDK.

This page discusses the following:

- [Persistence](#persistence)
- [Dependency Versions](#dependency-versions)

## What is Persistence? 

The Temporal Persistence store is a database used by the [Temporal Server](/temporal-service/temporal-server) to persist events generated and processed in your Temporal Service and SDK.

A Temporal Service's only required dependency for basic operation is the Persistence database.
Multiple types of databases are supported.

![Persistence](https://docs.temporal.io/diagrams/temporal-database.svg)

The database stores the following types of data:

- Tasks: Tasks to be dispatched.
- State of Workflow Executions:
  - Execution table: A capture of the mutable state of Workflow Executions.
  - History table: An append-only log of Workflow Execution History Events.
- Namespace metadata: Metadata of each Namespace in the Temporal Service.
- [Visibility](/temporal-service/visibility) data: Enables operations like "show all running Workflow Executions".
  For production environments, we recommend using Elasticsearch as your Visibility store.

An Elasticsearch database must be configured in a self-hosted Temporal Service to enable [advanced Visibility](/visibility#advanced-visibility) on Temporal Server versions 1.19.1 and earlier.

With Temporal Server version 1.20 and later, advanced Visibility features are available on SQL databases like MySQL (version 8.0.17 and later), PostgreSQL (version 12 and later), SQLite (v3.31.0 and later), and Elasticsearch.

### Dependency versions

Temporal tests compatibility by spanning the minimum and maximum stable major versions for each supported database.
The following versions are used in our test pipelines and actively tested before we release any version of Temporal:

- **Cassandra v3.11, v4.0, and 5.0.4 and later** (Cassandra 5 versions earlier than 5.0.4 are not supported)
- **PostgreSQL 13.18, 14.15, 15.10 and 16.6**
- **MySQL v5.7 and v8.0** (specifically 8.0.19+ due to a bug)

You can verify supported databases in the [Temporal Server release notes](https://github.com/temporalio/temporal/releases).

- Because Temporal Server primarily relies on core database functionality, we do not expect compatibility to break often.

- We do not run tests with vendors like Vitess and CockroachDB.
- Temporal also supports SQLite v3.x persistence, but this is meant only for development and testing, not production usage.

---

# Temporal Server

Source: https://docs.temporal.io/temporal-service/temporal-server

> The Temporal Server consists of four independently scalable services - Frontend gateway, History subsystem, Matching subsystem, and Worker Service.

This page discusses the following:

- [Frontend Service](#frontend-service)
- [History Service](#history-service)
- [History Shard](#history-shard)
- [Matching Service](#matching-service)
- [Worker Service](#worker-service)
- [Retention Period](#retention-period)

## What is the Temporal Server? 

The Temporal Server consists of four independently scalable services:

- Frontend gateway: for rate limiting, routing, authorizing.
- History subsystem: maintains data (mutable state, queues, and timers).
- Matching subsystem: hosts Task Queues for dispatching.
- Worker Service: for internal background Workflows.

For example, a real-life production deployment can have 5 Frontend, 15 History, 17 Matching, and 3 Worker Services per Temporal Service.

The Temporal Server services can run independently or be grouped together into shared processes on one or more physical or virtual machines.
For live (production) environments, we recommend that each service runs independently, because each one has different scaling requirements and troubleshooting becomes easier.
The History, Matching, and Worker Services can scale horizontally within a Temporal Service.
The Frontend Service scales differently than the others because it has no sharding or partitioning; it is just stateless.

Each service is aware of the others, including scaled instances, through a membership protocol via [Ringpop](https://github.com/temporalio/ringpop-go).

### Versions and support

> **💡 Tip:**
>
> We release new versions of the Temporal SDKs and Temporal Server software independently of one another.
> That said, All SDK versions support all server versions.
>
> To take advantage of bug fixes, performance improvements, and new features, please upgrade both SDK and servers to the latest versions on a regular cadence.
>

All Temporal Server releases abide by the [Semantic Versioning Specification](https://semver.org/).

We support upgrade paths from every version beginning with Temporal v1.7.0.
For details on upgrading your Temporal Service, see [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server).

We provide maintenance support for previously published minor and major versions by continuing to release critical bug fixes related to security, the prevention of data loss, and reliability, whenever they are found.

We aim to publish incremental upgrade guides for each minor and major version, which include specifics about dependency upgrades that we have tested for (such as Cassandra 3.0 -> 3.11).

We offer maintenance support of the last three **minor** versions after a release and do not plan to "backport" patches beyond that.

We offer maintenance support of **major** versions for at least 12 months after a GA release, and we provide at least 6 months' notice before EOL/deprecating support.

**Dependencies**

Temporal offers official support for, and is tested against, dependencies with the exact versions described in the `go.mod` file of the corresponding release tag.
(For example, [v1.5.1](https://github.com/temporalio/temporal/tree/v1.5.1) dependencies are documented in [the go.mod for v1.5.1](https://github.com/temporalio/temporal/blob/v1.5.1/go.mod).)

## What is a Frontend Service? 

The Frontend Service is a stateless gateway service that exposes a strongly typed [Proto API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto).
The Frontend Service is responsible for rate limiting, authorizing, validating, and routing all inbound calls.

![Frontend Service](https://docs.temporal.io/diagrams/temporal-frontend-service.svg)

Types of inbound calls include the following:

- [Namespace](/namespaces) CRUD
- External events
- Worker polls
- [Visibility](/temporal-service/visibility) requests
- [Temporal CLI](/cli) (the Temporal CLI) operations
- Calls from a remote Temporal Service related to [Multi-Cluster Replication](/temporal-service/multi-cluster-replication)

Every inbound request related to a Workflow Execution must have a Workflow Id, which is hashed for routing purposes.
The Frontend Service has access to the hash rings that maintain service membership information, including how many nodes (instances of each service) are in the Temporal Service.

Inbound call rate limiting is applied per host and per namespace.

The Frontend Service talks to the Matching Service, History Service, Worker Service, the database, and Elasticsearch (if in use).

- It uses the grpcPort 7233 to host the service handler.
- It uses port 6933 for membership-related communication.

Ports are configurable in the Temporal Service configuration.

## What is a History Service? 

The History Service is responsible for persisting Workflow Execution state to the Event History.
When the Workflow Execution is able to progress, the History Service adds a Task with the Workflow's updated history to the Task Queue.
From there, a Worker can poll for work, receive this updated history, and resume execution.

![Block diagram of how the History Service relates to the other services of the Temporal Server and to the Temporal Service](https://docs.temporal.io/diagrams/temporal-history-service.svg)

The total number of History Service processes can be between 1 and the total number of [History Shards](#history-shard).
An individual History Service can support many History Shards.
Temporal recommends starting at a ratio of 1 History Service process for every 500 History Shards.

Although the total number of History Shards remains static for the life of the Temporal Service, the number of History Service processes can change.

The History Service talks to the Matching Service and the database.

- It uses grpcPort 7234 to host the service handler.
- It uses port 6934 for membership-related communication.

Ports are configurable in the Temporal Service configuration.

### What is a History Shard? 

A History Shard is an important unit within a Temporal Service by which concurrent Workflow Execution throughput can be scaled.

Each History Shard maps to a single persistence partition.
A History Shard assumes that only one concurrent operation can be within a partition at a time.
In essence, the number of History Shards represents the number of concurrent database operations that can occur for a Temporal Service.
This means that the number of History Shards in a Temporal Service plays a significant role in the performance of your Temporal Application.

Before integrating a database, the total number of History Shards for the Temporal Service must be chosen and set in the Temporal Service's configuration (see [persistence](/references/service-configuration#persistence)).
After the Shard count is configured and the database integrated, the total number of History Shards for the Temporal Service cannot be changed.

In theory, a Temporal Service can operate with an unlimited number of History Shards, but each History Shard adds compute overhead to the Temporal Service.
The Temporal Service has operated successfully using anywhere from 1 to 128K History Shards, with each Shard responsible for tens of thousands of Workflow Executions.
One Shard is useful only in small scale setups designed for testing, while 128k Shards is useful only in very large scale production environments.
The correct number of History Shards for any given Temporal Service depends entirely on the Temporal Application that it is supporting and the type of database.

A History Shard is represented as a hashed integer.
Each Workflow Execution is automatically assigned to a History Shard.
The assignment algorithm hashes Workflow Execution metadata such as Workflow Id and Namespace and uses that value to match a History Shard.

Each History Shard maintains the Workflow Execution Event History, Workflow Execution mutable state, and the following internal Task Queues:

- Internal Transfer Task Queue: Transfers internal tasks to the Matching Service.
  Whenever a new Workflow Task needs to be scheduled, the History Service's Transfer Task Queue Processor transactionally dispatches it to the Matching Service.
- Internal Timer Task Queue: Durably persists Timers.
- Internal Replicator Task Queue: Asynchronously replicates Workflow Executions from active Clusters to other passive Clusters.
  (Relies on the experimental Multi-Cluster feature.)
- Internal Visibility Task Queue: Pushes data to the [Advanced Visibility](/visibility#advanced-visibility) index.

## What is a Matching Service? 

The Matching Service is responsible for hosting user-facing [Task Queues](/task-queue) for Task dispatching.

![Matching Service](https://docs.temporal.io/diagrams/temporal-matching-service.svg)

It is responsible for matching Workers to Tasks and routing new Tasks to the appropriate queue.
This service can scale internally by having multiple instances.

It talks to the Frontend Service, History Service, and the database.

- It uses grpcPort 7235 to host the service handler.
- It uses port 6935 for membership related communication.

Ports are configurable in the Temporal Service configuration.

## What is a Worker Service? 

The Worker Service runs background processing for the replication queue, system Workflows, and (in versions older than 1.5.0) the Kafka visibility processor.

<div className="tdiw">
  <div className="tditw">
    <p className="tdit">Worker Service</p>
  </div>
  <div className="tdiiw" height="740">
    <img
      className="img_ev3q"
      src="/diagrams/temporal-worker-service.svg"
      alt="Worker Service"
    />
  </div>
</div>

It talks to the Frontend Service.

- It uses port 6939 for membership-related communication.

Ports are configurable in the Temporal Service configuration.

## What is a Retention Period? 

Retention Period is the duration for which the Temporal Service stores data associated with closed Workflow Executions on a Namespace in the Persistence store.

- [How to set the Retention Period for a Namespace](/cli/command-reference/operator#create)
- [How to set the Retention Period for a Namespace using the Go SDK](/develop/go/client/namespaces)
- [How to set the Retention Period for a Namespace using the Java SDK](/develop/java/client/namespaces)

A Retention Period applies to all closed Workflow Executions within a [Namespace](/namespaces) and is set when the Namespace is registered.

The Temporal Service triggers a Timer task at the end of the Retention Period that cleans up the data associated with the closed Workflow Execution on that Namespace.

The minimum Retention Period is 1 day.
On Temporal Service version 1.18 and later, the maximum Retention Period value for Namespaces can be set to anything over the minimum requirement of 1 day. Ensure that your Persistence store has enough capacity for the storage.
On Temporal Service versions 1.17 and earlier, the maximum Retention Period you can set is 30 days.
Setting the Retention Period to 0 results in the error _A valid retention period is not set on request_.

If you don't set the Retention Period value when using the [`temporal operator namespace create`](/cli/command-reference/operator#create) command, it defaults to 3 days.
If you don't set the Retention Period value when using the Register Namespace Request API, it returns an error.

When changing the Retention Period (with [`temporal operator namespace update`](/cli/command-reference/operator#update) or the `UpdateNamespace` API), the new duration applies to Workflow Executions that close after the change is saved.

> **ℹ️ Info:**
>
> Changing the Retention Period does NOT affect existing closed Workflow Executions: they retain their original cleanup timers based on the Retention Period that was in effect when they closed.
>

### Manual cleanup of closed Workflow Executions

For cases where you need to remove closed Workflow Executions before their retention timer expires, you can use [`temporal workflow delete`](/cli/command-reference/workflow#delete) or the `DeleteWorkflowExecution` command.
This is particularly useful along with reducing the Retention Period to clean up previously closed Workflow Executions to reduce storage costs.

---

# Visibility

Source: https://docs.temporal.io/temporal-service/visibility

> The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Temporal Service.

This page discusses [Visibility](#visibility).

## What is Visibility? 

The term [Visibility](/visibility), within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Temporal Service.

The [Visibility store](/self-hosted-guide/visibility) in your Temporal Service stores persisted Workflow Execution Event History data and is set up as a part of your [Persistence store](/temporal-service/persistence) to enable listing and filtering details about Workflow Executions that exist on your Temporal Service.

- [How to set up a Visibility store](/self-hosted-guide/visibility)

With Temporal Server v1.21, you can set up [Dual Visibility](/dual-visibility) to migrate your Visibility store from one database to another.

Support for separate standard and advanced Visibility setups will be deprecated from Temporal Server v1.21 onwards.
Check [Supported databases](/self-hosted-guide/visibility) for updates.

---

# Error handling and troubleshooting

Source: https://docs.temporal.io/troubleshooting

> Discover effective troubleshooting solutions for potential Temporal errors and edge cases with our comprehensive guides, ensuring smooth Workflow execution and error management.

Even the most reliable systems can encounter issues.
Our troubleshooting guides are designed to help you quickly identify and resolve potential errors, ensuring your Temporal applications run smoothly and efficiently.

- [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error): The `BlobSizeLimitError` happens when the size of a blob (payloads including Workflow context and each Workflow and Activity argument and return value) is too large.
  The maximum payload for a single request is 2 MB, and the maximum size for any Event History transaction is 4 MB.
- [Troubleshoot the Deadline-Exceeded Error](/troubleshooting/deadline-exceeded-error):
  The "Context: deadline exceeded" error occurs when requests to the Temporal Service by the Client or Worker cannot be completed.
  This can be due to network issues, timeouts, server overload, or Query errors.
- [Troubleshoot the Failed Reaching Server Error](/troubleshooting/last-connection-error): The message "Failed reaching server: last connection error" often happens due to an expired TLS certificate or during the Server startup process when Client requests reach the Server before roles are fully initialized.
- [Troubleshoot missed Schedule Actions](/troubleshooting/schedule-missed-actions): Diagnose missed or delayed Actions with examples of common overlap and buffering pitfalls, then use metrics and Schedule state to find the affected Schedule.
- [Troubleshoot Serverless Workers](/troubleshooting/serverless-workers): Diagnose issues with Serverless Workers by tracing the flow from Task Queue to Worker execution, on [AWS Lambda](/troubleshooting/serverless-workers/aws-lambda) or [GCP Cloud Run](/troubleshooting/serverless-workers/cloud-run).
- [Troubleshoot request failures](/troubleshooting/request-failures): Diagnose gRPC failures and slow requests from Workers and Clients to the Temporal Service, including NOT_FOUND on respond operations, RESOURCE_EXHAUSTED throttling, and UNIMPLEMENTED or INTERNAL responses.
- [Troubleshoot Worker capacity](/troubleshooting/worker-capacity): Diagnose exhausted Task slots, disconnected pollers, Task completions dropping to zero, rising schedule-to-start latency, and an empty Sticky Execution cache.
- [Troubleshoot Workflow and Activity execution failures](/troubleshooting/execution-failures): Diagnose non-determinism errors, oversized Workflow Task responses, unhandled exceptions in Workflow and Activity code, and Local Activities running past the Workflow Task heartbeat timeout.
- [Recover pinned Workflows after a bad rollout](/production-deployment/worker-deployments/recover-pinned-workflows): Recover pinned Workflows that have failed or are stuck retrying tasks after rolling out a faulty Worker Deployment Version.

---

# Troubleshoot payload and gRPC message size limit errors

Source: https://docs.temporal.io/troubleshooting/blob-size-limit-error

Temporal enforces size limits on the data that passes between the Temporal Client, Workers, and the Temporal Service.
There are two distinct limits, each producing different error messages and behaviors, and they require different
solutions:

- [Payload size limit](#payload-size-limit)
- [gRPC message size limit](#grpc-message-size-limit)

## Payload size limit

The Temporal Service enforces a size limit on individual payloads. This limit is 2 MB on Temporal Cloud, but is
configurable on self-hosted deployments with a default of 2 MB. A [payload](/dataconversion#payload) represents the
serialized binary data for the input and output of Workflows and Activities.

### Error messages

The error message depends on which operation carried the oversized payload and which SDK version is in use. Examples
include:

- `WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE`
- `[TMPRL1103] Attempted to upload payloads with size that exceeded the error limit.`
- `BadScheduleActivityAttributes: ScheduleActivityTaskCommandAttributes.Input exceeds size limit`
- `Complete result exceeds size limit`
- `CompleteWorkflowExecutionCommandAttributes.Result exceeds size limit`
- `WORKFLOW_TASK_FAILED_CAUSE_BAD_UPDATE_WORKFLOW_EXECUTION_MESSAGE`

### Error behavior 

The behavior when a payload exceeds the size limit depends on the SDK version.

**Newer SDK versions:** The SDK fails the Workflow Task with cause `WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE`. The
Workflow is not terminated and remains open, so you can deploy a fix and allow the Workflow to continue. For Activities,
the Activity Task fails with an explicit error instead of timing out silently. These SDK versions have this behavior:

- Go SDK 1.43.0+
- Python SDK 1.23.0+
- TypeScript SDK 1.21.0+
- .NET SDK 1.18.0+
- Ruby SDK 1.7.0+
- Rust SDK 0.6.0+

**All other SDK versions:** The behavior depends on whether the oversized payload is an input or a result:

- **Inputs (Workflow input, Activity input):** The Temporal Service rejects the command and terminates the Workflow.
  You'll need to resolve the issue and restart the Workflow.
- **Activity result:** The Temporal Service rejects the Activity completion and the Activity fails with an error.
- **Workflow result:** The Workflow gets stuck in a retry loop. The server rejects the `CompleteWorkflowExecution`
  command, and replay produces the same oversized result.

### How to resolve

1. Offload large payloads to an object store to reduce the risk of exceeding payload size limits:
  1. Pass references to the stored payloads within the Workflow instead of the actual data.
  1. Retrieve the payloads from the object store when needed during execution.

  This is called the
  [claim check pattern](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern). The claim
  check pattern is built into the SDKs as [External Storage](/external-storage), or you can implement your own claim
  check pattern by using a custom [Payload Codec](/payload-codec)

  This is the most reliable way to avoid hitting payload size limits. Consider implementing the claim check pattern for
  Workflows and Activities that have the potential to receive or return large payloads, even if they are currently
  within the limit.

> **Public Preview**
>  APIs and configuration may change before General Availability. Join the
> [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for
> help.

1. Use compression with a [custom Payload Codec](/payload-codec) for large payloads. This may address the immediate
   issue, but if payload sizes continue to grow, the problem can arise again.

## gRPC message size limit

All communication between the Temporal Client, Workers, and the Temporal Service uses gRPC, which enforces a 4 MB limit
on each request. This limit applies to the full request, including all payload data and command metadata. For example,
when a Workflow schedules multiple Activities in a single Workflow Task, the Worker sends one request containing all
those commands to schedule the Activities and their inputs.

A Workflow can hit this limit even when every individual payload is under 2 MB. Scheduling several Activities with
moderate-sized inputs, or hundreds of Activities with tiny inputs in the same Workflow Task can push the combined
request past 4 MB. Activity results are also subject to this limit.

### Error messages

The error message depends on which operation carried the oversized gRPC message and which SDK version is in use.

- `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`
- `ScheduleToCloseTimeout` (Activities only, see [error behavior](#grpc-error-behavior) below)

### Error behavior 

The behavior depends on where the oversized message originates:

- **Workflow Tasks:** The Temporal Service terminates the Workflow Execution. When the Worker completes a Workflow Task,
  it sends all the commands the Workflow produced (such as Activity schedules and their inputs) back to the Temporal
  Service. If the combined size exceeds 4 MB, the SDK catches the gRPC error and sends a failed Workflow Task response
  with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Because replay would produce the same oversized request
  on every attempt, the Temporal Service ends the Execution with `TERMINATED` status rather than retrying it. Terminated
  Executions do not retry automatically, so fix the oversized payload and deploy the corrected Worker before restarting
  them. See [Execution failures](/troubleshooting/execution-failures#grpc-message-too-large) for triage steps.

- **Activity Tasks:** The Activity gets stuck in a retry loop or exits with a `ScheduleToCloseTimeout`. The Activity
  executes successfully, but the Worker can't deliver the oversized result over gRPC. The server never receives the
  completion, so it retries the Activity. Each retry completes successfully but fails to deliver the result. The
  Activity retries until the `ScheduleToCloseTimeout` expires. If no `ScheduleToCloseTimeout` is set, the Activity
  retries indefinitely until the Workflow is manually terminated. The `ResourceExhausted` gRPC error only appears in
  Worker logs.

### How to resolve

1. Break larger batches of commands into smaller batch sizes:
   - Workflow-level batching:
     1. Modify the Workflow to process Activities or Child Workflows in smaller batches.
     2. Iterate through each batch, waiting for completion before moving to the next.
   - [Workflow Task](/tasks#workflow-task)-level batching:
     1. Execute Activities in smaller batches within a single Workflow Task.
     2. Introduce brief pauses or sleeps between batches.
2. If the request is large because of payload sizes rather than the number of commands, refer to the
   [Payload size limit](#payload-size-limit) section for solutions.

---

# Troubleshoot the deadline-exceeded error

Source: https://docs.temporal.io/troubleshooting/deadline-exceeded-error

> Troubleshoot Temporal Service's Context deadline exceeded errors caused by network issues, server overload, and timing skews. Sync system clocks, check Frontend logs, metrics, Workflow logic, and service health after configurations or restarts. For unresolved issues, contact support.

All requests made to the [Temporal Service](/temporal-service) by the Client or Worker are [gRPC requests](https://grpc.io/docs/what-is-grpc/core-concepts/#deadlines).
Sometimes, when these frontend requests can't be completed, you'll see this particular error message: `Context: deadline exceeded`.
Network interruptions, timeouts, server overload, and Query errors are some of the causes of this error.

The following sections discuss the nature of this error and how to troubleshoot it.

### Check system clocks

Timing skew can cause the system clock on a Worker to drift behind the system clock of the Temporal Service.
If the difference between the two clocks exceeds an Activity's Start-To-Close Timeout, an `Activity complete after timeout` error occurs.

If you receive an `Activity complete after timeout` error alongside `Context: deadline exceeded`, check the clocks on the Temporal Service's system and the system of the Worker sending that error.
If the Worker's clock doesn't match the Temporal Service, synchronize all clocks to an NTP server.

### Check Frontend Service logs

> **📝 Note:**
>
> Cloud users cannot access some of the logs needed to diagnose the source of the error.
>
> If you're using Temporal Cloud, create a [support ticket](/evaluate/cloud/support#support-ticket) with as much information as possible, including the Namespace Name and the Workflow Ids of some Workflow Executions in which the issue occurs.
>

[Frontend Service](/temporal-service/temporal-server#frontend-service) logs can show which parts of the Temporal Service aren't working.
For the error to appear, a service pod or container must be up and running.

OSS users can verify that the Frontend Service is connected and running by using the Temporal CLI.

```
temporal operator cluster health --address 127.0.0.1:7233
```

Use [`grpc-health-probe`](https://github.com/grpc-ecosystem/grpc-health-probe) to check the Frontend Service, [Matching Service](/temporal-service/temporal-server#matching-service), and [History Service](/temporal-service/temporal-server#history-service).

```
./grpc-health-probe -addr=frontendAddress:frontendPort -service=temporal.api.workflowservice.v1.WorkflowService

./grpc-health-probe -addr=matchingAddress:matchingPort -service=temporal.api.workflowservice.v1.MatchingService

./grpc-health-probe -addr=historyAddress:historyPort -service=temporal.api.workflowservice.v1.HistoryService
```

Logs can also be used to find failed Client [Query](/sending-messages#sending-queries) requests.

### Check your Temporal Service metrics

Temporal Service metrics can be used to detect issues (such as `resource exhausted`) that impact Temporal Service health.
A `resource exhausted` error can cause your client request to fail, which prompts the `deadline exceeded` error.

Use the following query to check for errors in `RpsLimit`, `ConcurrentLimit` and `SystemOverloaded` on your metrics dashboard.

```
sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause)
```

Look for high latencies, short timeouts, and other abnormal [Temporal Service metrics](/references/service-metrics).
If the metrics come from a specific service (such as History Service), check the service's health and performance.

### Check Workflow logic

Check your [Client and Worker configuration](/references/service-configuration) files for missing or invalid target values, such as the following:

- Server names
- Network or host addresses
- Certificates

Invalid targets also cause `connection refused` errors alongside `deadline exceeded`.
Check that the Client connects after updating your files.

### Advanced troubleshooting

In addition to the steps listed in the previous sections, check the areas mentioned in each of the following scenarios.

### After enabling mTLS

Check the health of the Temporal Service with `temporal operator cluster health`.

```
temporal operator cluster health --address [SERVER_ADDRESS]
```

Add any missing [environment variables](/references/web-ui-environment-variables) to the configuration files, and correct any incorrect values.
Server names and certificates must match between Frontend and internode.

### After restarting the Temporal Service

You might not be giving the Temporal Service enough time to respond and reconnect.
Restart the Server, wait, and then check all services for connectivity and further errors.

If the error persists, review your Workflow Execution History and server logs for more specific causes before continuing to troubleshoot.

### When executing or scheduling Workflows

One or more services might be unable to connect to the [Frontend Service](/temporal-service/temporal-server#frontend-service).
The Workflow might be unable to complete requests within the given connection time.

Increase the value of `frontend.keepAliveMaxConnectionAge` so that requests can be finished before the connection terminates.

> **📝 Note:**
>
> If you increase `frontend.keepAliveMaxConnectionAge` values, consider monitoring your server performance for load.
>

---

Still unable to resolve your issue?

- If you use Temporal Cloud, create a [support ticket](/evaluate/cloud/support#support-ticket).
- If you use our open source software or Temporal Cloud, check for similar questions and possible solutions in our [community forum](https://community.temporal.io) or [community Slack](https://temporal.io/slack).

---

# Troubleshoot Workflow and Activity execution failures

Source: https://docs.temporal.io/troubleshooting/execution-failures

> Diagnose non-determinism errors, oversized payloads, failing Workflow and Activity code, and Local Activity latency

This guide covers failures that happen while your Workflow and Activity code is running on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that outrun the Workflow Task heartbeat timeout.
It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

Every section on this page is one of the recommended alerts in [Worker alerting](/best-practices/worker-alerting).
Each one opens with the alert definition, explains what the condition costs you, then walks through triage.
If you got here from a firing alert, the section you landed on is the runbook for it.

For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics).

[`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`.
The reason decides what the Temporal Service does next, and the difference is worth knowing: three of the four retry forever, one terminates the Execution on the spot.
Alert on each `failure_reason` separately.

## Non-determinism error 

**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError`
Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

Replay produced a different sequence of commands than the one recorded in Event History.
The Worker noticed that the Workflow code it is running no longer matches what the Execution has already done.

**Why it matters.**
Affected Executions stop making progress.
The Temporal Service retries the Workflow Task over and over, loading up your Workflow Workers, and by default those Executions sit in Running status the whole time.
This will not clear up on its own.

**Triage.**

1. **Identify the affected Workflow Executions.** This metric does not carry a Workflow Id. Worker logs record the error with the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures.
1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause.
1. **Determine whether this is a code change or a deploy artifact.** Common causes:
   - A code change added, removed, or reordered commands such as Activity scheduling, Timers, Signals, or Child Workflows, without a versioning guard. Executions that built their History under the old code fail on the new code.
   - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its `for` duration past how long a rollout takes.
   - Changed Activity or Timer parameters in existing Workflow code without versioning.
1. **Roll back if it isn't clearing.** If the errors started after a deploy and aren't going away, roll the Worker back. Affected Executions pick up again on their next Workflow Task retry once compatible code is running. Then add a proper versioning guard before you redeploy. See [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
1. **Watch Worker pressure.** All those retries add up. Cross-check [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high). Enough retry volume will saturate your capacity and start hurting healthy Executions on the same Task Queue.

## gRPC message too large 

**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge`
Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

The Workflow Task response was bigger than the gRPC message size limit.
The Worker tried `RespondWorkflowTaskCompleted` and something rejected it: the gRPC library on the SDK side, a proxy or load balancer in the path, or the gRPC library on the Temporal Service side when it went to receive.

The Temporal Service never saw that original request, so the SDK follows up with `RespondWorkflowTaskFailed` and cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`.
Replay would build the same oversized response every time, so the Service terminates the Workflow Execution instead of retrying it.

For the payload size limits themselves, and for the Activity Task version of this (which retries instead of terminating), see [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error).

**Why it matters.**
Affected Executions end immediately and permanently, with `TERMINATED` status and no retry.
Whatever work was in progress is gone, and someone has to restart them by hand.

This is the one `failure_reason` on this page that kills Executions instead of retrying them, so give it a short `for` duration.

**Triage.**

1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Check Worker logs for Workflow Ids and Run Ids, then confirm the cause from the `WorkflowTaskFailed` and `WorkflowExecutionTerminated` events in Event History.
1. **Find what is oversized.** The fix depends entirely on which part of the response is too large:
   - **Oversized Activity inputs or outputs.** Move the payload out of band: put it in blob storage and pass a reference through Event History instead. See [External Storage](/external-storage) for the pattern.
   - **Accumulated Signals or Updates.** A large number buffered into a single Workflow Task. Rate-limit senders or batch Signals.
   - **Too many commands in one response.** A Workflow scheduling a very large fan-out of Activities or Child Workflows in a single step. Break the fan-out into smaller batches across multiple Workflow Tasks.
1. **Fix and deploy before restarting anything.** Terminated Executions do not retry. Restarting them before the cause is fixed means hitting the same limit and being terminated again. Once the corrected Worker is deployed and verified, restart the affected Executions from the Temporal UI or CLI.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check the Workflow terminate rate on your server dashboard. A spike alongside this metric confirms Executions are being terminated in bulk.
>

## Request too large 

**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=RequestTooLarge`
Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

The whole Workflow Task completion was bigger than the Namespace limit on a single `RespondWorkflowTaskCompleted` request.
This is not one oversized Payload, it is the total: every command, Payload, message, and piece of metadata the Worker produced for that one Workflow Task, added together.

Workers that support command pagination split a large completion across several requests, but the recombined size still has to fit the limit.
When it does not, the Worker fails the Workflow Task itself with cause `WORKFLOW_TASK_FAILED_CAUSE_REQUEST_TOO_LARGE` rather than sending pages the Temporal Service is going to reject.

**Why it matters.**
The Temporal Service retries the Workflow Task, so unlike [gRPC message too large](#grpc-message-too-large) the Execution is not terminated.
Replay produces the same oversized completion every time, so affected Executions stop making progress and stay that way until the cause is fixed.
The retries consume Workflow Worker capacity and can start affecting healthy Executions on the same Task Queue.

**Triage.**

1. **Identify the affected Executions.** Find the Workflow Id and Run Id in Worker logs because the metric does not carry a Workflow Id. In the Temporal UI, you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures.
1. **Find and fix what made the completion large.** The limit applies to the sum. Identify what the Workflow produced in a single step, then change the Workflow to produce less in each Workflow Task:
   - **A large fan-out.** Scheduling many Activities, Child Workflows, or Timers in one Workflow Task can make the completion too large. Break the fan-out into smaller batches across multiple Workflow Tasks.
   - **Accumulated Signals or Updates.** Handling a large number of buffered Signals or Updates in one Workflow Task can make the completion too large. Limit how many senders send at once, or batch their requests.
   - **Large Payloads in aggregate.** Payloads can each be under the Payload limit but make the completion too large when combined. Store the data in [External Storage](/external-storage) and pass a reference through Event History instead.
1. **Check the Namespace limit.** The limit is a Namespace setting on the size of a single Workflow Task completion, separate from the Payload and gRPC message limits. On Temporal Cloud, contact support if the workload legitimately needs a higher limit. Self-hosted operators can adjust it in dynamic config.
1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once the corrected Worker is running. They do not need to be restarted, because they were never terminated.

## Workflow Task execution failures elevated 

**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError`
Fires when the rate goes above 10/s, held for 2m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

Workflow Tasks are failing steadily from unhandled exceptions and panics in Workflow code that the SDK catches and reports.
`WorkflowError` is the catch-all reason. It covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors.

**Why it matters.**
The Temporal Service retries the Workflow Task.
If the error is deterministic and shows up on every replay, the Execution is stuck retrying forever, burning Worker capacity and never getting healthy.

At high rates the retry pressure fills up your Workflow Worker slots and starts affecting healthy Executions on the same Task Queue.
Unlike `GrpcMessageTooLarge`, the Temporal Service won't terminate the Execution for you, so this gets worse the longer you leave it.

**Triage.**

1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The `WorkflowTaskFailed` event in Event History carries the error message and type. The `workflow_type` tag on the metric narrows which Workflow is failing.
1. **Determine which failure mode this is.** `WorkflowError` covers several:
   - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool. `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` is too low for the number of concurrent Executions, so new Workflow Tasks get rejected before they run. Raise the thread count, and think about whether the Worker pool needs to scale out too.
   - **Unhandled exception in Workflow code.** A bug or an unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event names the error.
   - **Data Converter error.** Something failed serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and [Payload Codec](/payload-codec) configuration.
1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion tend to show up together under load, and a CPU-starved Worker is slower to finish Workflow Tasks, which makes both worse.
1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once compatible code is running.

## Workflow Task execution latency high 

**Recommended alert:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency), tagged `namespace`, `task_queue`, and `workflow_type`
Fires when p99 goes above 10s, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

Workflow Tasks are taking too long to execute on the Worker.
The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks.

A batch workload where Workflow Tasks routinely run long can sit above this threshold all day without anything being wrong.
Set the threshold from your own observed p99, and only treat the default as meaningful if your Workflows are latency-sensitive.

**Why it matters.**
The Temporal Service writes `WorkflowTaskTimedOut` events to Event History and reschedules timed-out Tasks on the normal Task Queue.
Each timeout forces a [Sticky Execution](/sticky-execution) cache eviction on the Worker holding the Execution, so the next Workflow Task for it requires a full cold replay.

If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried Task, because their results are not checkpointed between Workflow Task heartbeats.
Non-idempotent Local Activities produce duplicate side effects with real business impact.

At scale this compounds: more timeouts cause more cold replays, cold replays drive latency higher, and higher latency causes more timeouts.

**Triage.**

1. **Check replay latency.** Look at [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-running Event History rather than into new commands. Large histories, a slow Data Converter during replay, or a high cache eviction rate forcing cold replays are the usual culprits.
1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load).
1. **Check Worker CPU.** If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker.
1. **Check for blocking Workflow code.** Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). The SDK holds the slot until the respond call succeeds, which inflates this metric even when your Workflow code finished quickly.

## Activity execution failures elevated 

**Recommended alert:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed) with `failure_reason=ActivityError`, tagged `activity_type`
Fires when the rate goes above 10/s, held for 2m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

Activities are failing outright at a sustained rate: returning failures, not timing out.

Filter on `failure_reason=ActivityError` so this alert tracks Activity code and its dependencies.
The other reasons on this metric are Worker-side size limits with different triage, covered in [Activity Payloads too large](#activity-payloads-too-large).

`ApplicationFailure` instances marked with category `BENIGN` don't increment this counter, so how well this metric tracks only the unexpected failures depends on how consistently your application marks the expected ones.

**Why it matters.**
A high failure rate means a burst of retry Tasks.
If your Workers can't keep up with the retry volume, the Activity Task backlog grows. See [Activity schedule-to-start latency elevated](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated).
At scale, sustained retry bursts put real pressure on Task matching and the database underneath.

**Triage.**

1. **Identify which Activity is failing.** The `activity_type` tag narrows it down. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids.
1. **Work out whether this is transient or a bug.** A downstream outage, a network partition, or a database timeout will recover on its own, so watch whether the rate falls. A code bug won't.
1. **Check downstream service health.** A struggling dependency is a common cause of sustained failure bursts. If it is throttling you, check that your [Retry Policy](/encyclopedia/retry-policies) has sensible backoff. Without it, your retries pile more pressure onto something that is already overloaded.
1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops.
1. **Mark expected failures as benign.** If your design fails Activities on purpose, as polling patterns, Saga compensations, and flow control through exceptions all do, mark those `ApplicationFailure` instances with category `BENIGN`. That keeps them out of this metric and lets the alert track only the failures you didn't expect, without tuning a threshold per `activity_type`. Check that your SDK version supports the category before you rely on it.

One caveat: internal failures increment this counter no matter what category you set. Those are things like a context propagation error or a context timeout, rather than an Activity returning a failure.

## Activity Payloads too large 

**Recommended alert:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed) with `failure_reason=PayloadsTooLarge`, tagged `activity_type`
Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

A Payload in the Activity Task response was bigger than the Namespace Payload size limit.
The Worker failed the Activity Task itself rather than send a completion the Temporal Service was going to reject, so the Activity fails with an explicit error instead of timing out silently.

The Worker marks this failure retryable on purpose: the point of catching it on the Worker is to turn something the Temporal Service would hard-fail into an Activity failure you can fix and redeploy past.

For the limits themselves and the ways to stay under them, see [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error#payload-size-limit).

**Why it matters.**
The Activity retries under its Retry Policy and produces the same oversized result every time, so it cannot succeed until the code changes.
The Workflow Execution stays blocked on that Activity until its schedule-to-close timeout expires, and each attempt is wasted work on a Worker.
This will not clear up on its own, which is why it is worth alerting on any occurrence rather than on a rate.

**Triage.**

1. **Identify which Activity is oversized.** The `activity_type` tag narrows it down. Worker logs for that type record the size and the limit that was exceeded.
1. **Find which Payload is too large.** Usually the Activity result. Large collections, embedded files, and full API responses returned verbatim are the common causes.
1. **Move the Payload out of band.** Store it and return a reference instead, using [External Storage](/external-storage) or a custom [Payload Codec](/payload-codec). Compression buys headroom but does not remove the ceiling.
1. **Deploy the fix.** Because the failure is retryable, affected Activities succeed on their next attempt once the corrected Worker is running, as long as the Workflow has not already hit its schedule-to-close timeout.

## Unregistered Activity invocation 

**Recommended alert:** [`temporal_unregistered_activity_invocation`](/references/sdk-metrics#unregistered_activity_invocation), tagged `activity_type`, `task_queue`, and `workflow_type`
Fires on any occurrence, held for 1m. Routed Critical by default. Go SDK only. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

A Workflow scheduled an Activity that the Worker polling that Task Queue has no registered implementation for.

This metric is emitted by the Go SDK only.

**Why it matters.**
The Activity can't run. It keeps getting retried against a Worker that has no implementation for it, until the Activity's `scheduleToClose` timeout expires, or forever if you haven't set one. Meanwhile the Workflow Execution waiting on it goes nowhere.

This is nearly always a deployment mistake rather than a runtime condition: Workflow code scheduling an Activity that the deployed Worker doesn't register. It won't fix itself.

**Triage.**

1. **Identify the Activity and the Task Queue.** The `activity_type` and `task_queue` tags name both. The `workflow_type` tag identifies which Workflow is scheduling it.
1. **Check whether the Activity is registered on the right Worker.** Confirm the Worker polling that Task Queue registers that Activity type. A common cause is registering the Activity on a Worker polling a different Task Queue.
1. **Check for a partial rollout.** If Workflow code that schedules a new Activity deployed ahead of the Worker that implements it, some Workers will be running without the registration. Complete the rollout.
1. **Check for a renamed Activity.** Changing an Activity's registered name while Executions are in flight leaves those Executions scheduling the old name. Register both names until the in-flight Executions drain, or use a versioning guard.

## Local Activity latency exceeds the heartbeat timeout 

**Recommended alert:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency), tagged `activity_type`
Fires when p99 goes above 30m, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

A [Local Activity](/local-activity) is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes.

### How Workflow Task heartbeating works 

A Local Activity executes inside the Workflow Task rather than as a separately scheduled Activity Task.
That means the Workflow Task stays open for as long as the Local Activity runs, which would normally exceed the Workflow Task timeout.

To keep the Task alive, the SDK sends Workflow Task heartbeats: repeated `RespondWorkflowTaskCompleted` calls that tell the Temporal Service work is still going and ask for more time.
The Service goes along with this up to the Workflow Task heartbeat timeout.
After that it times the Task out and reschedules it on the normal Task Queue.

Local Activities cannot heartbeat individually the way regular Activities can, and their results are not recorded in Event History between Workflow Task heartbeats.
So when the Task is rescheduled, every Local Activity in it runs again from the beginning.

**Why it matters.**
When the Temporal Service times out the heartbeating Workflow Task, the Local Activity re-executes from scratch.
A non-idempotent Local Activity produces duplicate side effects with real business impact.

Any pending Signals, Updates, or other events are delayed until the retried Workflow Task completes, so end-to-end Execution latency rises significantly.

The Local Activity also occupies an executor slot for its entire duration.
Several in this state at once can occupy every available slot, blocking new Local Activities from starting.
See [LocalActivityWorker slots](/troubleshooting/worker-capacity#localactivityworker-slots).

Local Activities are designed for short, fast operations.
A single attempt running for 30 minutes is a design problem, not a tuning problem.

**Triage.**

1. **Identify the affected Local Activity.** The `activity_type` tag narrows it down. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids.
1. **Find what it is blocked on.** A Local Activity running this long is nearly always stuck on a downstream call: a slow service, a slow query, or a network call with a very generous timeout. Fix the dependency, or shorten that timeout so the Local Activity fails fast instead of hanging.
1. **Check whether a retry chain is accumulating.** A high failure rate paired with an aggressive Retry Policy can push total elapsed time past the heartbeat timeout even when every individual attempt is short. Check [`temporal_local_activity_execution_failed`](/references/sdk-metrics#local_activity_execution_failed) for the same `activity_type`, and fix the underlying failure first.
1. **Check whether timeouts have already happened.** By the time this fires, the Temporal Service may have timed out heartbeating Workflow Tasks already. Look for timeout errors in Worker logs and `WorkflowTaskTimedOut` events in Event History. If they are there, your Local Activities have already run twice, so check whether they are idempotent and clean up any duplicate side effects.
1. **Fix the design.** If the work genuinely takes this long, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work. If it must stay a Local Activity, set a `scheduleToCloseTimeout` below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed.

---

# Troubleshoot the failed reaching server error

Source: https://docs.temporal.io/troubleshooting/last-connection-error

> Troubleshoot server connection errors often caused by expired TLS certificates. Verify, renew, and update server configurations to resolve temporal client request issues effectively.

The message `Failed reaching server: last connection error` can often result from an expired TLS certificate or during the Server startup process, in which the Client requests reach the Server before the roles are fully initialized.

This troubleshooting guide shows you how to do the following:

- Verify the certification expiration date
- Renew the certification
- Update the server configuration

### Verify TLS certification expiration date

The first step in troubleshooting this error is to verify the expiration date of the TLS certification.
Then you can renew the certification and update the server configuration.

Choose one of the following methods to verify the expiration date of the TLS certification:

**Verify the expiration date of the TLS certification**

List the expiration date with the following command:

**Temporal CLI**

```command
temporal cloud namespace mtls cert-ca list \
    --namespace <namespace_id>.<account_id>
```

Read the expiration date from the certificate details in the output.

**tcld**

```command
tcld namespace accepted-client-ca list \
    --namespace <namespace_id>.<account_id> | \
    jq -r '.[0].notAfter'
```

If the returned date is in the past, the certificate has expired.

**Existing certificate management infrastructure**

If you are using an existing certificate management infrastructure, use it to verify the TLS connection.
For example, if you are using OpenSSL, run the following command:

```command
openssl s_client -connect <namespace_grpc_endpoint> -showcerts -cert ~/certs/path.pem -key .~/certs/path.key -tls1_2
```

**Self-signed certificate**

If you are using a self-signed certificate, run the following Temporal CLI command:

```command
temporal namespace describe \
    --namespace <namespace_id>.<account_id> \
    --address <namespace_grpc_endpoint> \
    --tls-cert-path <path-to-mTLS-pem-file> \
    --tls-key-path <path-to-mTLS-key-file>
```

Your Namespace gRPC endpoint is available on the details page for your [Temporal Cloud Namespace](https://cloud.temporal.io/namespaces).

### Renew TLS certification

If the certificate has expired or is about to expire, the next step is to renew it.

You can do this by contacting the certificate authority (CA) that issued the certificate and requesting a renewal.

**Existing certificate management infrastructure**

If you are using an existing certificate management infrastructure, contact the administrator of the infrastructure to renew the certificate.

**Self-signed certificate**

If you are using a self-signed certificate or don't have an existing infrastructure, you can generate a new certificate using OpenSSL, [step CLI](https://github.com/smallstep/cli), or similar tools.

For information on generating a self-signed certificate, see [Control authorization](/cloud/certificates#control-authorization).

### Update the CA certification in the server configuration

Update the new CA certificate in the Temporal Cloud server configuration.

You can update certificates using any of the following methods:

- [Update certificates using Temporal Cloud UI](/cloud/certificates#update-certificates-using-temporal-cloud-ui)
- [Update certificates using tcld](/cloud/certificates#update-certificates-using-tcld)

After you update the TLS certification in the server configuration, retry your connection.

### Set reminders

Don't let your certificates expire.
Add reminders to your calendar to issue new CA certificates well before the expiration dates of the existing ones.

### Additional resources

The preceding steps should help you troubleshoot the `failed reaching server: last connection error` error caused by an expired TLS certificate.

If this issue persists, verify that the Client you are using to connect to the server is using the correct TLS certification and that the Client requests reach the server after the roles are fully initialized.
If you still need help, [create a support ticket](/evaluate/cloud/support#support-ticket).

---

# Performance bottlenecks troubleshooting guide

Source: https://docs.temporal.io/troubleshooting/performance-bottlenecks

> Diagnose and resolve performance bottlenecks using Temporal SDK metrics

This guide outlines common performance bottlenecks in Temporal Workers and Clients.
It covers key latency metrics and root causes of high values, and provides diagnostic steps and troubleshooting strategies.
These metrics can help you optimize Temporal deployments and Workflow execution.

To get the most out of this guide, you should be familiar with [Temporal architecture](/temporal), [Workflows](/workflows), [Activities](/activities), and [Task Queues](/task-queue).
You should also know how to use key metrics like latency, counter, rate, CPU utilization, and memory usage.

## Task processing metrics

These metrics provide insights into various stages of the [Task](/tasks) lifecycle, from scheduling to completion.
The following sections detail common metrics, their potential causes for high latency or resource depletion, and strategies for diagnosing and resolving performance issues.

### `temporal_workflow_task_schedule_to_start_latency` spike

High [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency) (P95 higher than one second) can be caused by several factors.
This metric represents the time between when a [Workflow Task](/tasks#workflow-task) is scheduled (enqueued) and when it is picked up by a Worker for processing. Here are some potential causes:

- Insufficient Worker capacity: If there aren't enough Workers or if the Workers are overloaded, they may not be able to pick up Tasks quickly enough. This can lead to Tasks waiting longer in the queue ([Detect Task Backlog](/cloud/worker-health#detect-task-backlog)).
- Worker configuration issues: Improperly configured Workers, such as having too few pollers or Task slots, can lead to increased latency ([Detect Task Backlog](/cloud/worker-health#detect-task-backlog)).
- High Workflow lock latency: If many updates are made to a single execution, this can cause Workflow lock latency, which in turn affects the Schedule-to-start latency. Reduce the rate of Signals.
- Network latency: Workers in a different region from the Temporal cluster, or large payload size, can introduce additional latency.

To diagnose and address high `temporal_workflow_task_schedule_to_start_latency`, you should:

1. Check Worker CPU and memory usage.
1. Review Worker configuration (such as number of pollers and Task slots).
1. Look for any spikes in Workflow or Activity starts that might be overwhelming the system.
1. Ensure Workers are in the same region as the Temporal cluster if possible.

### `temporal_activity_schedule_to_start_latency` spike

High [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency) (P95 higher than one second) can be caused by several factors.
This metric represents the time between when an [Activity Task](/tasks#activity-task) is scheduled (enqueued) and when it is picked up by a Worker for processing.
Here are some potential causes:

- Insufficient Worker capacity: If there aren't enough Workers or if the Workers are overloaded, they may not be able to pick up Tasks quickly enough. This can lead to Tasks waiting longer in the queue ([Detect Task Backlog](/cloud/worker-health#detect-task-backlog)).
- Worker configuration issues: Improperly configured Workers, such as having too few pollers or Task slots, can lead to increased latency ([Detect Task Backlog](/cloud/worker-health#detect-task-backlog)).
- Task Queue configuration: Setting `TaskQueueActivitiesPerSecond` too low can limit the rate at which Activities are started, leading to increased Schedule-to-start latency.
- Network latency: Workers in a different region from the Temporal cluster, or large payload size can introduce additional latency.

To diagnose and address high `temporal_activity_schedule_to_start_latency`:

1. Check Worker CPU and memory usage.
1. Review Worker configuration (such as number of pollers and Task slots).
1. Look for any spikes in Workflow or Activity starts that might be overwhelming the system.
1. Ensure Workers are in the same region as the Temporal cluster if possible.

### `temporal_workflow_endtoend_latency` spike

The [`temporal_workflow_endtoend_latency`](/references/sdk-metrics#workflow_endtoend_latency) metric represents the total Workflow Execution time from Schedule to the closure for a single Workflow Run.
Normal ranges for this metric depend on the use case, but here are some potential causes for the unexpected spikes:

- Complex Workflows: If the Workflows have many Activities or if the Activities take a long time to execute.
- Workflow and Activity retries: If Workflows or Activities are configured to retry upon failure and they fail often, this can increase the end-to-end latency as the system will wait for the retry delay before reattempting the failed operation.
- Worker capacity and configuration: If there aren't enough Workers or if the Workers are overloaded, they may not be able to pick up and process Tasks quickly enough. This can lead to Tasks waiting longer in the queue, thereby increasing the end-to-end latency ([Detect Task Backlog](/cloud/worker-health#detect-task-backlog)).
- External dependencies: If your Workflows or Activities depend on external systems or services (such as databases or APIs) and these systems are slow or unreliable, they can increase the end-to-end latency.
- Network latency: Workers in a different region from the Temporal cluster can introduce additional latency.

To diagnose and address high `temporal_workflow_endtoend_latency`:

1. Review your Workflow and Activity designs to ensure they are as efficient as possible.
2. Monitor your Workers to ensure they have sufficient capacity (CPU and memory) and are not overloaded.
3. Monitor your external dependencies to ensure they are performing well.
4. Ensure Workers are in the same region as the Temporal cluster if possible.

### High `temporal_workflow_task_execution_latency`

The [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) metric represents the time taken by a Worker to execute a Workflow Task.
The Temporal SDK raises a “Deadlock detected during Workflow run” error or [TMPRL1101](https://github.com/temporalio/rules/blob/main/rules/TMPRL1101.md) when a Workflow Task takes more than one or two seconds to complete.
Here are some potential causes:

- CPU-intensive work: Performing CPU-intensive operations in your Workflow Task can lead to slow execution.
- Slow local Activities: Workflow Task execution time includes the Local Activity execution time.
- Slow Workflow replay: Workflow Task execution time includes the Workflow Replay time. Refer to `workflow_task_replay_latency` for more details.
- Worker resource constraints: High CPU usage on Worker pods can lead to slower Workflow Task execution. Workers with insufficient CPU resources can cause delays.
- Infinite loops or blocking calls: Workflow code with infinite loops or blocking external API calls can cause the Workflow Task to execute slowly or time out.
- Slow data conversion: Your custom Data Converter is taking too long to encode/decode payloads, for example, when talking to a remote encryption service.

To diagnose and address slow Workflow Task execution, you can:

1. Monitor Worker CPU and memory utilization.
2. Ensure that your Workers have adequate resources and are properly scaled for your workload.
3. Consider running your Workflow code in a profiler using a replayer to see where CPU cycles are spent.
4. Review your Workflow code for potential optimizations or to remove blocking operations.
5. Disable deadlock detection for Data Converter: It does not reduce Task execution latency but does remove the “Deadlock detected during Workflow run” or TMPRL1101 error. In Go, wrap it with `workflow.DataConverterWithoutDeadlockDetection`. In Java, surround your Data Converter code with `WorkflowUnsafe.deadlockDetectorOff`.

### High `workflow_task_replay_latency`

Workflow Task replay is the process of reconstructing the Workflow's state by re-executing the Workflow code from the beginning, using the recorded Event History.
This process ensures that the Workflow can continue from where it left off, even after interruptions or failures.
[`workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) is high if it exceeds a few milliseconds.
Here are the main causes:

- Large Event Histories: Workflows with long histories take more time to replay, as the Worker needs to process all events to reconstruct the Workflow state.
- Data Converter performance: Slow Data Converters, especially those that perform encryption or interact with external services, can impact replay.
- Large payloads: Activities or Signals with large payloads can slow down the replay process, especially if the Data Converter needs to process these payloads.
- Complex Workflow logic: Workflow code with complex logic or computationally intensive operations, such as scheduling many concurrent child Workflows or Activities, can increase replay latency.
- Frequent cache evictions: If workers often evict Workflow Executions from their cache (due to memory constraints or frequent restarts), it leads to more replays and higher latency.
- Worker resource constraints: High CPU utilization or memory pressure on Worker nodes can slow down the replay.

To diagnose and address slow Workflow Task replay, you can:

1. Monitor SDK Metrics: Keep a close eye on the `temporal_workflow_task_replay_latency` metric. This histogram metric measures the time it takes to replay a Workflow Task.
1. Analyze Event History Size: Check the number of events in your Workflow histories and consider using the "Continue-As-New" feature for long-running Workflows.
1. Optimize Data Converters: If you're using custom Data Converters, especially for encryption or complex serialization, look for opportunities to optimize their performance.
1. Review Payload Sizes: Large Activity or Signal payloads can slow down replay. Consider optimizing the size of data being passed in your Workflows.
1. Profile Workflow Code: Use a profiler to identify CPU-intensive parts of your Workflow code that might be slowing down replay.
1. Manage Worker Cache: Frequent cache evictions can lead to more replays. Tune your Worker's cache size and eviction policies.

### `temporal_activity_execution_latency` spike

The [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) metric measures the time from when a Worker starts processing an Activity Task until it reports to the service that the Task is complete or failed.
There are several potential causes for high `temporal_activity_execution_latency`:

- Activity implementation: The most common cause of high Activity Execution latency is the actual implementation of the Activity itself. If the Activity is performing time-consuming operations or making slow external API calls, it will take longer to execute.
- External dependencies: If your Activity is constrained by an external resource or service that all Activities access, it could cause increased latency.
- Worker resource constraints: Under-resourced Worker nodes or experiencing high CPU utilization can lead to slower Activity Execution.
- Network latency: High latency between your Workers and external services or the Temporal service itself can contribute to increased Activity Execution time.

To diagnose and address high Activity Execution latency:

1. Monitor the `activity_execution_latency` metric, which you can filter by Activity type and Activity Task queue.
2. Optimize your Activity implementation to reduce latency, especially with external services or database interactions.
3. Check your Worker CPU and memory utilization to make sure they have adequate resources.
4. Examine your Worker configuration, particularly `(Max)ConcurrentActivityExecutionSize` and `(Max)WorkerActivitiesPerSecond`, to ensure they are not limiting your activity execution.

### Depletion of `temporal_worker_task_slots_available` for `WorkflowWorker`

The [`temporal_worker_task_slots_available{worker_type=”WorkflowWorker”}`](/references/sdk-metrics#worker_task_slots_available) metric indicates the number of available slots for executing Workflow Tasks on a Worker.
This metric may go to zero for several reasons:

- High Workflow Task Load: If there are more Tasks than the Worker can handle concurrently, the available slots will be depleted. This can happen if the rate of incoming Tasks is higher than the rate at which tasks are being completed.
- Worker Configuration: The number of available slots is determined by the Worker configuration, specifically the `MaxConcurrentWorkflowTaskExecutionSize` setting. If these are set too low, the Worker may not have enough slots to handle the Task load.
- High `temporal_workflow_task_execution_latency` and `workflow_task_replay_latency`.

To prevent depletion of Workflow Task slots:

1. Monitor Worker CPU and Memory usage while increasing `(Max)ConcurrentWorkflowTaskExecutionSize` to add more execution slots.
2. Scale up Workers both vertically (increasing CPU and Memory) and horizontally (increasing Worker instances).

### Depletion of `temporal_worker_task_slots_available` for `ActivityWorker`

The [`temporal_worker_task_slots_available{worker_type=”ActivityWorker”}`](/references/sdk-metrics#worker_task_slots_available) metric indicates the number of available slots for executing Activity Tasks on a Worker.
This metric may go to zero for several reasons:

- Blocked Activities and Zombie Activities: The most common cause is activities that are blocked or not returning on time. Zombie Activities are a subset of this category. They occur when an Activity times out (hits its `StartToClose` or `HeartbeatTimeout` timeout) and has stopped Heartbeating but continues to run, occupying some or all the slots as more retries occur. This can happen if:
  - The Activity code is blocking on a downstream service call or an infinite loop.
  - There's a mismatch between the Activity's `StartToClose` timeout and any client-side timeouts for external calls.
- Resource Utilization: High CPU or memory usage on Workers can cause activities to block and not release slots.

To prevent depletion of Activity Task slots:

1. Monitor Worker CPU and Memory usage while increasing `(Max)ConcurrentActivityExecutionSize` to add more execution slots.
2. Add client-side timeout to your downstream API client.
3. Review your Task code to ensure Tasks complete within a reasonable time measured by `temporal_activity_execution_latency`.

## Network requests

Network issues can impact Temporal clients and workers, leading to delays, failures, and overall system instability.
This section focuses on metrics that reveal common network-related problems with your Temporal deployment, specifically related to network connectivity, latency, and request failures.
These metrics can indicate where bottlenecks exist within the communication channels between Temporal clients (including Temporal Workers) and the Temporal server.

### High `temporal_long_request_failure`

The [`temporal_long_request_failure`](/references/sdk-metrics#long_request_failure) metric counts the number of failed RPC long poll requests for `PollWorkflowTaskQueue`, `PollActivityTaskQueue`, and `GetWorkflowExecutionHistory` (when polling new events). High values of this metric can be caused by several factors:

- Network Issues: Problems with the network connection between the Temporal Client and the Temporal Server, including firewalls and proxies, can cause long poll requests to fail.
- Rate Limiting: If the rate of requests exceeds the configured limits on the Temporal Server or Temporal Cloud, additional requests may be rejected, increasing the `temporal_long_request_failure` count. This is often indicated by a `ResourceExhausted` status code.
- Server Errors: If the Temporal Server is experiencing issues, it may fail to respond to long poll requests correctly, leading to an increase in `temporal_long_request_failure`.

To diagnose the cause of high `temporal_long_request_failure`, you can:

1. Check the operation and the status or code tag of the `temporal_long_request_failure` metric to see the type of errors that are occurring.
2. If you receive a `ResourceExhausted` status code, review the rate limits configured on the Temporal Server or ask for help from Temporal Support for Temporal Cloud.
3. Check the network connection between the Temporal Client and the Temporal Server.

### High `temporal_request_failure_total`

The [`temporal_request_failure_total`](/references/sdk-metrics#request_failure) metric counts the number of RPC requests made by the Temporal Client that have failed.
High values of this metric can be caused by several factors:

- Network Issues: Problems with the network connection between the Temporal Client and the Temporal Server can cause requests to fail.
- Client Errors: If there's an issue with the Temporal Client, such as misconfiguration or resource exhaustion, it may fail to make requests correctly.
- Operation Errors: Specific operations like `SignalWorkflowExecution` or `TerminateWorkflowExecution` can fail if they are trying to act on a closed Workflow Execution that no longer exists (because it completed and was removed from persistence when it hit Namespace retention time).
- Rate Limiting: If the rate of requests exceeds the configured limits on the Temporal Server, additional requests may be rejected, increasing the counter. This is often indicated by a `ResourceExhausted` status code.
- Request Size Limit: If the Worker tries to return an Activity response that is larger than the blob size limit (2MB), the service will reject it, causing a request failure.
- Server Errors: If the Temporal Server is experiencing issues, it may fail to respond to requests correctly, leading to an increase in `temporal_request_failure_total`.

To diagnose the cause of high `temporal_request_failure_total`, you can:

1. Check the status or code tag of the `temporal_request_failure_total` metric to see the type of errors that are occurring.
2. Look at the operation tag of the `temporal_request_failure_total` metric to see which operations are failing.
3. Monitor the Temporal Server logs and the Temporal Client logs for any error messages or warnings.
4. Check the network connection between the Temporal Client and the Temporal Server.

### High `temporal_request_latency`

The [`temporal_request_latency`](/references/sdk-metrics#request_latency) metric measures the latency of gRPC requests made by the Temporal Client.
High values for this metric can be caused by several factors:

- Network Latency: The physical distance and network conditions between the Temporal Client and the Temporal Server can affect the latency of requests.
- Network Transfer Time: Larger payloads take longer to transfer over the network, which affects request latency. For example, large payloads in `RespondWorkflowTaskCompleted` can affect the latency of the request. This is especially true when Workflows are scheduling multiple activities with large inputs.
- Resource Exhaustion: Running out of resources (such as CPU or memory) on the client or server can cause delays in processing the request.
- Client Configuration: Improper client configuration, such as setting thread pool sizes too aggressively or having memory constraints that are too low for the number of allocated threads, can lead to situations where Tasks overwhelm the client, causing increased latency.
- Server Load: If the Temporal Server is under heavy load, it may take longer to respond to requests, leading to increased latency.

To diagnose and address high `temporal_request_latency`:

1. Monitor the `temporal_request_latency` metric to identify when and where latency spikes are occurring.
2. Check the network connection between the Temporal Client and the Temporal Server.
3. Monitor the resource usage on both the Temporal Client and the Temporal Server.
4. Review your Temporal Client configuration to ensure it is optimized for your workload.
5. If you're using Temporal Cloud, check if the Cloud’s [service latency](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_latency_p95) metric spikes up and reach out to Temporal Support for help.

### `rate(temporal_long_request_total{operation="PollActivityTaskQueue"})`

The [`rate(temporal_long_request_total{operation="PollActivityTaskQueue"})`](/references/sdk-metrics#long_request) expression measures the per-second average rate of `PollActivityTaskQueue` long poll requests over a certain period of time.

`PollActivityTaskQueue` is an operation where Workers poll for Activity Tasks from the Task Queue.
The `temporal_long_request_total` metric counts the number of these long poll requests.

By applying the `rate()` function in Prometheus, you can calculate the per-second average rate of these requests over the time range specified in the Query.
This can help you understand the load on your Temporal service and how often your Workers are polling for Activity Tasks.

### `rate(temporal_long_request_total{operation="PollWorkflowTaskQueue"})`

The [`rate(temporal_long_request_total{operation="PollWorkflowTaskQueue"})`](/references/sdk-metrics#long_request) expression measures the per-second average rate of `PollWorkflowTaskQueue` long poll requests over a certain period of time.

`PollWorkflowTaskQueue` is an operation where Workers poll for Workflow Tasks from the Task Queue.
The `temporal_long_request_total` metric counts the number of these long poll requests.

By applying the `rate()` function in Prometheus, you can calculate the per-second average rate of these requests over the time range specified in the query.
This can help you understand the load on your Temporal service and how often your Workers are polling for Workflow Tasks.

## Caching

Temporal Workers rely on caching to optimize performance by reducing the overhead of fetching Workflow state from the history and Replaying.
However, unlimited caching is impossible; there's a trade-off between the benefits of cached data and the memory consumed.
These metrics allow you to balance performance gains with responsible memory usage.

### `temporal_sticky_cache_size`

The [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) metric represents the number of Workflow executions currently cached in a Worker's memory.

The sticky cache is used to improve performance by keeping the Workflow state in memory, reducing the need to reconstruct the Workflow from its Event History for every Task.
It’s particularly useful for latency-sensitive Workflows.

There is a direct relationship between the sticky cache size and Worker memory consumption.
As the cache size increases, so does the memory usage of the Worker.

The maximum size of the sticky cache can be configured. For example, the default in the Go SDK is 10,000 Workflows.

A larger sticky cache can improve performance by reducing the need to replay Workflow histories.
However, it also increases memory usage, which can lead to issues if not properly managed.

Monitor this metric alongside Worker memory usage.
A sudden increase in `sticky_cache_size` can correlate with increased memory consumption and potential performance issues.

If memory consumption is too high, you can reduce the maximum sticky cache size.
Conversely, if you have available memory and want to improve performance, you might increase it.

### `temporal_sticky_cache_hit_total` and `temporal_sticky_cache_miss_total`

The [`temporal_sticky_cache_hit_total`](/references/sdk-metrics#sticky_cache_hit) metric is a counter that measures the total number of times a Workflow Task found a cached Workflow Execution to run against, and
the opposite is [`temporal_sticky_cache_miss_total`](/references/sdk-metrics#sticky_cache_miss), which is a counter that measures the total number of times a Workflow Task did not find a cached Workflow Execution to run against.

Sticky Execution is a feature where a Worker caches a Workflow Execution and creates a dedicated Task Queue to listen on.
This improves performance because the Temporal Service only sends new events to the Worker instead of entire Event Histories, and the Workflow doesn't have to Replay.

A “hit” means the Worker finds the Workflow in its cache when processing a Workflow Task, allowing immediate processing without fetching the full Event History from the server and Replaying.
A "miss" means the Worker didn't find the Workflow in its cache, and it must fetch the Event History and Replay.

Monitoring these two metrics and comparing them can help you understand how your sticky cache is being used.
A high rate of cache hits with a low rate of cache misses indicates that your Workflows are being scheduled efficiently, with minimal need for fetching Event Histories and Replaying.

### `temporal_sticky_cache_total_forced_eviction_total`

The [`temporal_sticky_cache_total_forced_eviction_total`](/references/sdk-metrics#sticky_cache_hit) metric is a counter that measures the total number of Workflow Executions that have been forcibly evicted from the sticky cache.

Sticky Execution is a feature where a Worker caches a Workflow Execution and creates a dedicated Task Queue to listen on.
This improves performance because the Temporal Service only sends new events to the Worker instead of entire Event Histories, and the Workflow doesn't have to Replay.

A "forced eviction" in this context means that a Workflow Execution was removed from the cache before it completed, typically because the cache was full and needed to make room for other Workflow Executions.
This means that if the Worker needs to process more Tasks for the evicted Workflow Execution, it will have to fetch the entire Event History from the Temporal Service and Replay.

Monitoring the `temporal_sticky_cache_total_forced_eviction_total` metric can help you understand how often your Workflows are being evicted from the cache.
A high rate of forced evictions could indicate that your cache size is too small for your workload, and you may need to increase the `WorkflowCacheSize` setting if your Worker resources can accommodate it.

---

# Troubleshoot request failures

Source: https://docs.temporal.io/troubleshooting/request-failures

> Diagnose gRPC failures and slow requests from Temporal Workers and Clients

This guide covers gRPC failures and slow requests between your Temporal SDK Workers or Clients and the Temporal Service.
It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

Every section on this page is one of the recommended alerts in [Worker alerting](/best-practices/worker-alerting).
Each one opens with the alert definition, explains what the condition costs you, then walks through triage.
If you got here from a firing alert, the section you landed on is the runbook for it.

For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics).

[`temporal_request_failure`](/references/sdk-metrics#request_failure) increments when the Temporal Service returns a non-OK gRPC status code on a standard operation.
[`temporal_long_request_failure`](/references/sdk-metrics#long_request_failure) covers poll operations and long-poll `GetWorkflowExecutionHistory`.
Both carry `namespace`, `operation`, and `status_code` tags.
How urgent a failure is depends on which status code turned up on which operation.

> **📝 Note:**
>
> Two naming details will trip up your queries.
> `UpdateWithStartWorkflowExecution` shows up in SDK metrics under the gRPC operation name `ExecuteMultiOperation`.
> And `status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names. Client options can also turn the tag off, so check that it is present in your metrics endpoint before you filter on it.
>

## NOT_FOUND on respond operations 

**Recommended alert:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed`
Fires on any occurrence, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).

A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists.
There are three causes:

- The task timed out. The Worker ran past the Workflow Task timeout, or past the Activity `startToClose` or `scheduleToClose` timeout, and the Service discarded the in-flight task.
- The Workflow Execution is no longer running. It completed, was terminated, or hit its Workflow Run Timeout before the task finished.
- The Worker restarted mid-execution. The in-flight Task Token was lost, the Service rescheduled the task, and the original Worker still attempted to respond after coming back up.

The last two happen during normal operation, so a few of these are nothing to worry about.
A sustained rate is.

**Why it matters.**
The Temporal Service threw away the result your Worker just produced.
For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it.
For Workflow Tasks, the Service writes a `WorkflowTaskTimedOut` event to Event History and reschedules the task on the normal Task Queue, which forces a [Sticky Execution](/sticky-execution) cache eviction and a cold replay on the retry.

A sustained rate means your Workers are finishing too late, over and over.
Every discarded result is Worker capacity you paid for and got nothing back from, and every rescheduled task adds to how long your Workflows take end to end.

If you run [Local Activities](/local-activity), a Workflow Task timeout makes them run again from the start on the retried task.
Their results are not written to Event History between Workflow Task heartbeats, so there is nothing to resume from.
If they aren't idempotent, you get duplicate side effects, and that is a business problem rather than a monitoring one.

**Triage.**

1. **Rule out the expected causes first.** Check the status of a few affected Executions in the Temporal UI or with `temporal workflow describe`. If they completed, were terminated, or hit their Run Timeout, the NOT_FOUND is expected. Check Worker restart counts in your infrastructure observability stack for the same reason. If either explains the volume, stop here.
1. **Check task execution latency.** For Workflow Tasks, check [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency). For Activities, check [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type`. If p99 is at or above the corresponding timeout, that is the direct cause.
1. **Check replay latency.** If Workflow Task execution latency is high, look at [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) next. When replay latency is high, the Worker is burning its time re-running Event History instead of getting to the new commands. Look for large histories and a slow [Data Converter](/dataconversion).
1. **Check Worker resources.** High CPU on the Worker slows task execution directly. Look at the `identity` field in the `WorkflowTaskStarted` or `ActivityTaskStarted` event to identify which Worker ran the task, then check that pod for CPU saturation and cold-start delays.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](#resource_exhausted-on-respond-operations). Sustained throttling can delay a respond call long enough for the Service to time out the task before the response lands.

> **📝 Note:**
> Self-hosted Temporal Service
>
> If SDK-side metrics look normal and the Execution was not terminated or timed out, check server-side latency: Frontend Service latency filtered to the affected respond operation, and persistence latency filtered to `UpdateWorkflowExecution`.
>

## NOT_FOUND on Activity heartbeat 

**Recommended alert:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RecordActivityTaskHeartbeat`
Fires on any occurrence, held for 5m. Routed Warning by default, though see the caveat below. [Tune the threshold](/best-practices/worker-alerting#request-failures).

A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists.
The Service has already cancelled the in-flight Activity Task: either the `heartbeatTimeout` fired before the next heartbeat call arrived, the `startToClose` timeout expired while the Activity was still executing, or the Workflow Execution is no longer running.

Normal Workflow-side cancellation is not a cause.
Cancellation returns `CancelRequested=true` in the heartbeat response body rather than a gRPC error, so NOT_FOUND on this operation is a reliable signal of a timeout or forced closure.

**Why it matters.**
If `heartbeatTimeout` is the cause, the Service has already timed out this Activity attempt and scheduled a retry if the Retry Policy allows it.
The Activity runs again from the start on the next attempt, so a non-idempotent Activity will duplicate its side effects. That is worth treating more seriously than the default Warning severity suggests.

A sustained rate means the Worker keeps missing its heartbeat window.
The Activity will time out on every attempt until you fix the cause, holding Task slots and generating retry tasks the whole time.

**Triage.**

1. **Compare the heartbeat interval against `heartbeatTimeout`.** The Worker must call heartbeat more frequently than the timeout. If the Activity slows down between heartbeat calls because of CPU pressure, blocking I/O, or downstream throttling, the effective interval grows past the timeout even though the code is calling heartbeat.
1. **Check Worker CPU.** A CPU-starved Worker slows down between heartbeat calls even when the Activity is making progress. If utilization is consistently high, reduce per-Worker concurrency or scale out horizontally.
1. **Check the `startToClose` timeout.** If the Activity has run longer than `startToClose`, the Service times it out while the Activity is still executing, and the next heartbeat returns NOT_FOUND. Compare [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` against the configured timeout.
1. **Check for throttling on heartbeat calls.** Query `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` and `operation=RecordActivityTaskHeartbeat`. If the Temporal Service is throttling these calls, the effective heartbeat interval grows past `heartbeatTimeout` even when the Worker calls on time. See [RESOURCE_EXHAUSTED on poll operations](#resource_exhausted-on-poll-operations) for how to work through a throttling cause.
1. **Check heartbeat payload size.** The last heartbeat details payload is held in memory for the life of the Activity attempt. Large payloads on high-throughput Activity Workers contribute to memory pressure on the Temporal Service. Store only the minimum progress state needed to resume on retry.

## RESOURCE_EXHAUSTED on user-facing operations 

**Recommended alert:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, or `ExecuteMultiOperation`
Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).

The Temporal Service is throttling the operations your application code uses to start Workflows and deliver Signals and Updates.
The SDK retries these automatically for up to 60 seconds.
Beyond that, the call fails and the error propagates to your caller.

**Why it matters.**
These operations are on your application's critical path.
Within the retry window, callers experience elevated latency.
Past it, calls fail outright and your application must handle the error.

If it does not, starts and Signals are silently dropped.
A dropped start means the Workflow never runs.
A dropped Signal or Update means a running Workflow never receives input it is waiting on, and may stall indefinitely.
Log these failures in your application code so you can backfill starts and Signals afterward.

The Temporal Service throttles these operations last.
Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread.

**Triage.**

1. **Find out which limit you hit.** A Namespace rate limit, a concurrency limit, system-wide overload, and an open circuit breaker are four different problems with four different fixes. On the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by its `cause` tag and it will tell you directly. On other SDKs, work through the steps below.
1. **Check your traffic against your Namespace limits.** For a rate limit cause (`RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT` or `RESOURCE_EXHAUSTED_CAUSE_APS_LIMIT`), compare current throughput against your Namespace's [service limits](/evaluate/cloud/limits) on Temporal Cloud, and open a support request if you need them raised.
1. **Treat overload and circuit-breaker causes as capacity problems.** `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED` and `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN` mean the Temporal Service is shedding load to protect itself. A higher limit will not help; the Service needs capacity, or your workload needs to slow down.
1. **Add backoff in your application.** If throttling is expected during traffic peaks, ensure calling code retries with backoff rather than tight-looping, which amplifies the pressure.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`.
> Slow persistence is the usual root cause of throttling that cascades like this.
> If the cause is a rate limit, `frontend.namespaceRPS` may be too low for your traffic, but confirm persistence is healthy before you raise it.
> If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself. It needs capacity, not a higher limit.
>

## RESOURCE_EXHAUSTED on respond operations 

**Recommended alert:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed`
Fires on any occurrence, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).

The Temporal Service is throttling Workers reporting task results.
The SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands.

**Why it matters.**
This is a leading indicator of [NOT_FOUND on respond operations](#not_found-on-respond-operations).
If throttling goes on long enough the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task gets rescheduled, with all the cache eviction, cold replay, and Local Activity re-execution that section describes.

Meanwhile every in-flight task is still holding its slot, so you have less concurrency for new work.
Left alone, this turns into [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted).

**Triage.**

1. **Find out which limit you hit,** as in the section above. On the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by `cause`.
1. **Check whether timeouts have already started.** If [NOT_FOUND on respond operations](#not_found-on-respond-operations) is also firing, throttling has already cascaded into task timeouts and Executions are losing work.
1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted). Slots aren't released until the respond call succeeds, so throttling here drains the pool.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check persistence latency filtered to `UpdateWorkflowExecution`. Slow persistence on that operation is the usual root cause of throttling on respond operations.
>

## RESOURCE_EXHAUSTED on poll operations 

**Recommended alert:** `temporal_long_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `PollWorkflowTaskQueue` or `PollActivityTaskQueue`
Fires on any occurrence, held for 5m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).

The Temporal Service is throttling Worker poll calls.
Poll operations are long-poll requests, so they increment `temporal_long_request_failure` rather than `temporal_request_failure`.

**Why it matters.**
Throttled Workers back off and poll less frequently, which lowers the effective poll rate for the Task Queue even when every Worker is healthy.
That shows up as rising schedule-to-start latency and, if it persists, as a growing Task backlog.

This is often a symptom rather than a cause.
The Temporal Service throttles poll operations before it throttles respond or user-facing operations, so throttling here can be the first visible sign of pressure that has nothing to do with your Workers.

**Triage.**

1. **Identify the throttle cause.** As with the other throttling sections, a Namespace rate limit, a concurrency limit, and system-wide overload need different responses. On the Go SDK, [`temporal_long_request_resource_exhausted`](/references/sdk-metrics#long_request_resource_exhausted) carries the `cause` tag for poll operations.
1. **Check whether you are over-polling.** A large number of Workers each configured with many concurrent pollers can exceed the Namespace poller limit without processing any more work. Check your configured poller counts against [Worker performance](/develop/worker-performance) guidance before assuming the limit is too low.
1. **Check downstream effects.** See [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) and [Activity schedule-to-start latency elevated](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) to gauge whether throttling is affecting Task dispatch yet.
1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare against your Namespace's [service limits](/evaluate/cloud/limits).

> **📝 Note:**
> Self-hosted Temporal Service
>
> If poll operations are being throttled at scale, you may need to raise the Namespace concurrent poller limit through `frontend.namespaceCount` or `frontend.globalNamespaceCount`.
> Scale Worker capacity first if schedule-to-start latency is the real problem.
>

## UNIMPLEMENTED or INTERNAL from the Temporal Service 

**Recommended alert:** `temporal_request_failure` with `status_code=UNIMPLEMENTED` or `status_code=INTERNAL`, on any operation
Two alerts, one per status code. Each fires on any occurrence, held for 2m, routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).

These two status codes point at the Temporal Service rather than at your application, and they behave differently in the SDK.
Alert on them separately.

**UNIMPLEMENTED** means the Service does not recognize an operation the Worker called.
By the time a Worker reaches steady-state polling it has already called `GetSystemInfo` and `DescribeNamespace` successfully, so this is rarely a plain version mismatch on a freshly deployed Worker.
Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down.

**INTERNAL** means the Service encountered an error it could not attribute to the request.
Short bursts during Service restarts and rolling deploys are normal, so set the `for` duration long enough that your own deploys don't page you.
The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers.
Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency.

**Triage.**

1. **Check SDK and Temporal Service version compatibility.** For UNIMPLEMENTED, confirm your SDK version is not calling an API that has been removed or changed in your Service version.
1. **Check whether recent deploys correlate.** Both codes commonly appear immediately after a Service upgrade or a Worker deploy. If the timing lines up, consider rolling back while you investigate.
1. **Check whether the errors are Namespace-scoped or cluster-wide.** Errors isolated to one Namespace point at Namespace configuration. Cluster-wide errors point at infrastructure.
1. **Check downstream effects.** Sustained errors on poll operations cause Workers to back off. Cross-check [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected) and [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero).

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check service panics first. Any panic is critical, and it is almost always the root cause of sustained INTERNAL errors.
> Then check persistence errors and availability; the Temporal Service wraps database errors as INTERNAL.
> For UNIMPLEMENTED, check that every Frontend, History, and Matching pod is running the binary you intended. A wrong or corrupted binary on a few pods returns UNIMPLEMENTED on perfectly valid operations, usually alongside panics.
>

## Request latency high on user-facing operations 

**Recommended alert:** [`temporal_request_latency`](/references/sdk-metrics#request_latency) on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, or `ExecuteMultiOperation`
Fires when p99 goes above 2s, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).

p99 latency on the operations your application calls synchronously has risen above your threshold.

**Why it matters.**
These calls block your application code while they wait on the Temporal Service, so the latency is felt directly by your users and by anything downstream of the call completing.

The SDK retries transient errors but does not hide the latency cost: every retry adds to the total time this metric observes.
If throttling is the cause and retries exhaust the 60-second budget, the call fails outright.

**Triage.**

1. **Check for throttling on the same operations.** See [RESOURCE_EXHAUSTED on user-facing operations](#resource_exhausted-on-user-facing-operations). If both are firing, throttling is the cause of the latency and retries are what you are measuring.
1. **Check payload sizes.** This metric includes serialization and network time. Large Workflow inputs or Signal payloads, or an expensive [Payload Codec](/payload-codec), raise it without any Service-side slowdown.
1. **Check network path and region.** Clients in a different region from the Temporal Service pay that round trip on every call.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check Frontend Service latency filtered to the affected operations.
> Server-side latency is the sharper signal here, because the SDK metric also includes serialization and network time.
> Then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`, which is what usually drives up Frontend latency on starts and Signals.
> If persistence is healthy and nothing is being throttled, check Frontend pod CPU.
>

---

# Troubleshoot missed Schedule Actions

Source: https://docs.temporal.io/troubleshooting/schedule-missed-actions

> Diagnose missed or delayed Schedule Actions caused by overlap policies, buffering, cancellation, and Catchup Window expiry, with metrics and examples.

When a [Schedule](/schedule) does not start a Workflow Execution at its expected time, check whether the Schedule is paused or its end time has passed.
An [Overlap Policy](/schedule#overlap-policy) can also skip or delay an Action, and a delayed Action can expire outside the [Catchup Window](/schedule#catchup-window).
Use the [common pitfalls](#common-pitfalls) below to recognize these behaviors, and use metrics and Schedule state to investigate missed Actions.

## Alert on missed catchup window

The Temporal Service emits a counter each time it skips a scheduled Action because it could not run it within the configured Catchup Window. Alert on any non-zero value.

### Temporal Cloud

Alert on [`temporal_cloud_v1_schedule_missed_catchup_window_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_schedule_missed_catchup_window_count) grouped by `temporal_namespace`.

Example PromQL:

```
sum by (temporal_namespace) (
  increase(temporal_cloud_v1_schedule_missed_catchup_window_count[5m])
) > 0
```

### Self-hosted

Alert on [`schedule_missed_catchup_window`](/references/service-metrics#schedule_missed_catchup_window) grouped by `namespace`.

Example PromQL:

```
sum by (namespace) (
  increase(schedule_missed_catchup_window[5m])
) > 0
```

The metric is scoped to the Namespace, not to individual Schedules. A non-zero value tells you that at least one Schedule in the Namespace missed an Action, but not which one.

## Investigate which Schedule missed an Action

Once the alert fires, narrow your search down to the affected Schedule in two steps.

### 1. List Schedules in the Namespace

Enumerate the Schedules in the alerting Namespace:

```
temporal schedule list --namespace <your-namespace>
```

[`ListSchedules`](/cli/command-reference/schedule#list) returns Schedule Ids and summary information. It does not return per-Schedule miss counters, so use it only to produce the set of Schedule Ids to inspect.

### 2. Describe each Schedule

For each Schedule Id returned, run:

```
temporal schedule describe \
  --schedule-id <your-schedule-id> \
  --namespace <your-namespace>
```

[`DescribeSchedule`](/cli/command-reference/schedule#describe) returns full Schedule state, including the `info` block with cumulative counters. The relevant fields:

| Field | Meaning |
|-------|---------|
| `missedCatchupWindow` | Actions skipped because they could not run within the Catchup Window. Non-zero here identifies the Schedule responsible for the alert. |
| `overlapSkipped` | Actions skipped because the previous run was still in progress and the Overlap Policy is `Skip`. |
| `bufferDropped` | Buffered Actions dropped because the buffer was full under `BufferOne` or `BufferAll`. |
| `bufferSize` | Current depth of the Action buffer. |
| `recentActions` | Most recent Action times and results. |
| `runningWorkflows` | Workflow Executions currently running for this Schedule. |

Scripting the fan-out against the JSON output (`temporal schedule describe -o json`) is usually faster than inspecting each Schedule interactively.

## Interpret the result

Once you have identified the Schedule with a non-zero `missedCatchupWindow`, use the rest of the `DescribeSchedule` output to determine impact and root cause.

### Assess impact

- Compare `recentActions` to the Schedule's Spec to determine how many Actions were skipped and over what time period.
- If the Schedule uses the `Skip` Overlap Policy and the preceding run was long-running, the miss may reflect that run exceeding the Catchup Window, not a Service outage.
- For business-critical Schedules, [Backfill](/schedule#backfill) the skipped interval once the underlying cause is resolved.

### Common root causes

- **Service or Namespace outage longer than the Catchup Window.** The default Catchup Window is one year, so a miss typically means the Schedule is configured with a tighter window (minimum ten seconds) and the outage exceeded it.
- **Namespace rate limiting.** If scheduled starts are throttled, Actions can queue past the Catchup Window. Cross-check [`temporal_cloud_v1_schedule_rate_limited_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_schedule_rate_limited_count) (Cloud) or [`schedule_rate_limited`](/references/service-metrics#schedule_rate_limited) (self-hosted) in the same time range.
- **Buffer overruns under `BufferAll`.** Long-running Workflow Executions under `BufferAll` can push buffered Actions past the Catchup Window. Cross-check [`temporal_cloud_v1_schedule_buffer_overruns_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_schedule_buffer_overruns_count) (Cloud) or [`schedule_buffer_overruns`](/references/service-metrics#schedule_buffer_overruns) (self-hosted) and examine `bufferSize`.

### Remediate

- Widen the Catchup Window if the current value is tighter than your Service's worst-case unavailability. The trade-off is that more late Actions will fire during recovery.
- Revisit the Overlap Policy if runs routinely exceed the Spec interval. `BufferAll` and `Skip` have different failure modes under sustained delay.
- Increase Namespace throughput limits if rate limiting is the contributing factor.
- [Backfill](/schedule#backfill) the missed interval if the skipped Actions need to run.

## Common pitfalls

A scheduled time does not guarantee that a Workflow Execution starts.
An Action can be skipped by an Overlap Policy, expire outside the Catchup Window, exceed an internal buffer limit, or fail when the Service attempts to start the Workflow Execution.

### `Skip` is the default Overlap Policy

With `Skip`, every Action that overlaps an open Workflow Execution is intentionally discarded.
Skipped Actions are not retained for later execution.

For example, suppose a Schedule runs every five minutes and its first Workflow Execution takes 22 minutes:

```text
09:00  starts
09:05  overlaps and is skipped
09:10  overlaps and is skipped
09:15  overlaps and is skipped
09:20  overlaps and is skipped
09:22  the 09:00 Workflow Execution closes
09:25  starts
```

Only the `09:00` and `09:25` Actions start.
Use `BufferOne` or `BufferAll` if overlapping Actions must be retained, or `AllowAll` if they can run concurrently.

### `BufferOne` preserves the oldest waiting Action

`BufferOne` retains the first Action that overlaps an open Workflow Execution.
It does not replace that Action with the most recent one.
Additional Actions are skipped while the buffer is occupied.

```text
09:00  starts
09:05  is buffered
09:10  is skipped because the buffer is occupied
09:15  is skipped because the buffer is occupied
09:20  is skipped because the buffer is occupied
09:22  the 09:00 Workflow Execution closes
09:22  the 09:05 Action becomes eligible to start
```

The buffered Action must still be inside its Catchup Window when it becomes eligible on the current Scheduler implementation.
Older Scheduler implementations differ in this edge case and might still attempt to start an Action after it has waited beyond the window.
Configure the Catchup Window to include the maximum expected overlap delay.

### `BufferAll` can create a backlog

`BufferAll` starts only one non-overlapping Action after each preceding Workflow Execution closes.
If Workflow Executions take longer than the Schedule interval, the backlog grows faster than it drains.
Buffered Actions are subject to an internal safety limit of approximately 1,000 per Schedule by default.
The effective limit can vary by deployment and Scheduler implementation.
Actions can also expire outside the Catchup Window, so `BufferAll` should not be treated as an unlimited durable queue.

For example, with a five-minute Catchup Window:

```text
09:00  starts
09:05  is buffered
09:10  is buffered
09:15  is buffered
09:20  is buffered
09:22  the 09:00 Workflow Execution closes
```

At `09:22`, only the `09:20` Action is still inside the five-minute window.
On the current Scheduler implementation, the `09:05`, `09:10`, and `09:15` Actions expire, and the `09:20` Action starts.

### Cancellation is asynchronous

`CancelOther` requests cancellation and waits for the running Workflow Execution to close.
The Workflow must process the cancellation before the replacement can start.
If more Actions become due while cancellation is pending, the most recent waiting Action is selected after the running Workflow Execution closes.

```text
09:00  starts
09:05  requests cancellation of 09:00 and waits
09:10  becomes the newest waiting Action
09:15  becomes the newest waiting Action
09:20  becomes the newest waiting Action
09:22  the 09:00 Workflow Execution closes
09:22  the 09:20 Action becomes eligible to start
```

A slow or cancellation-resistant Workflow can therefore delay the replacement beyond its Catchup Window.
`TerminateOther` does not require Workflow cooperation, but task processing and closure-notification latency can still delay the replacement.

### `AllowAll` can create unbounded concurrency

`AllowAll` produces the fewest policy-related misses because every Action can start concurrently.
However, when Workflow Executions take longer than the Schedule interval, concurrency continually increases until executions begin to close.
Ensure that Workers, downstream systems, and Namespace limits can handle the maximum expected concurrency.

## Related reading

- [Schedule concept](/schedule)
- [Catchup Window](/schedule#catchup-window)
- [Temporal CLI schedule reference](/cli/command-reference/schedule)
- [Temporal Cloud OpenMetrics metrics reference](/cloud/metrics/openmetrics/metrics-reference)
- [Self-hosted Temporal Service metrics reference](/references/service-metrics)

---

# Troubleshoot Serverless Workers

Source: https://docs.temporal.io/troubleshooting/serverless-workers

> **Public Preview**
> AWS Lambda and GCP Cloud Run support are in Public Preview.

When a Serverless Worker is not running your Tasks, the failure is almost always in one of three places: the
[Worker Controller Instance (WCI)](/serverless-workers#how-invocation-works) is not starting compute, the compute
provider is rejecting the request, or the Worker starts but cannot do its job.

Every provider follows the same overall shape:

1. You deploy the Worker to the compute provider.
2. You create a [Worker Deployment Version](/worker-versioning#deployment-versions) with a compute configuration. This
   starts a WCI Workflow.
3. A Worker polls the Temporal Service, which binds its [Task Queue](/task-queue) to the Worker Deployment Version.
4. The WCI monitors that Task Queue. The [Matching Service](/temporal-service/temporal-server#matching-service) also
   signals the WCI when a Task arrives with no Worker free to take it.
5. Work arrives, the WCI starts compute, and the Worker processes Tasks.

How the WCI starts compute is where providers differ, and so is most of the diagnosis. On AWS Lambda the WCI invokes a
function per unit of work. On GCP Cloud Run it resizes a long-lived Worker Pool. Pick your provider:

- [**AWS Lambda**](/troubleshooting/serverless-workers/aws-lambda) - Diagnose functions that are never invoked,
  invocation loops, and timeouts that cut Workers short.
- [**GCP Cloud Run**](/troubleshooting/serverless-workers/cloud-run) - Diagnose Worker Pools that never scale up,
  impersonation and IAM failures, and instances stopped mid-Activity.

---

# Troubleshoot Serverless Workers on AWS Lambda

Source: https://docs.temporal.io/troubleshooting/serverless-workers/aws-lambda

> **Public Preview**

This page walks through the Serverless Worker invocation flow on AWS Lambda and helps you identify where a failure is
occurring. For Worker Pools on GCP Cloud Run, see
[Troubleshoot Serverless Workers on GCP Cloud Run](/troubleshooting/serverless-workers/cloud-run).

When a Serverless Worker invocation works correctly, the following sequence happens:

1. You deploy the Worker function on Lambda.
2. You configure a [Worker Deployment Version](/worker-versioning#deployment-versions) with a compute provider. This starts a [Worker Controller Instance (WCI)](/serverless-workers#how-invocation-works) Workflow and a validation invocation of the Lambda function.
3. The Lambda polls the Temporal Service successfully, binding the [Task Queue](/task-queue) configured on the Worker to the Worker Deployment Version.
4. The WCI continuously monitors the associated Task Queue on a schedule. The [Matching Service](/temporal-service/temporal-server#matching-service) also notifies the WCI Workflow of sync match failures immediately as they happen.
5. A Task arrives on the Task Queue and the WCI detects the backlog.
6. The WCI invokes the Lambda function.
7. The Lambda function starts, the Worker connects to Temporal and polls the Task Queue.
8. The Worker processes Tasks and shuts down gracefully.

Start by determining whether the Lambda function is being invoked at all, then narrow down from there.

If no check below matches what you are seeing, or they all pass, [check for AWS-side issues](#check-aws-issues).

## Is the Lambda function being invoked? 

Check the Lambda function's CloudWatch metrics or invocation logs.

In the AWS Console, go to **Lambda > Functions > your function > Monitor**. Look for recent invocations in the
**Invocations** graph. You can also check **CloudWatch > Log groups > /aws/lambda/your-function-name** for execution
logs.

If there are no invocations, continue to [Lambda is not being invoked](#lambda-not-invoked).

If the Lambda is being invoked but Workflows are not progressing, skip to
[Lambda is invoked but Tasks are not completing](#lambda-invoked-not-completing).

## Lambda is not being invoked 

Work through the following checks in order.

### Validate the connection to Lambda 

Start by verifying that Temporal can reach the Lambda function. In the Temporal UI, go to **Workers > Deployments > select your
deployment**, open the **Actions** menu on the version, and click **Validate Connection**.

A successful validation confirms that the Worker Deployment Version has a compute provider configured, that Temporal can assume the invocation role, and that the Lambda function can be invoked.

Validate Connection reports one of the following failure modes:

- **No compute provider configured.** No [Worker Controller Instance (WCI)](/serverless-workers#how-invocation-works) Workflow exists and the Lambda is never
  automatically invoked. A common cause is manually invoking the Lambda function before creating the Worker Deployment
  Version in the UI or CLI. The Worker connects and polls, which registers the version on the server, but without a compute provider.
  To fix the issue, create or update the Worker Deployment Version with the compute provider flags as described in the
  [deploy guide](/production-deployment/worker-deployments/serverless-workers/aws-lambda#create-worker-deployment-version).

- **Invalid connection.** Temporal cannot assume the invocation role or reach the Lambda function.
  Verify that the Lambda function ARN and invocation role ARN are correct. Verify the invocation role was created using the
  [CloudFormation template](/production-deployment/worker-deployments/serverless-workers/aws-lambda#configure-iam)
  and that the External ID matches the value in the Worker Deployment Version configuration.

  If the ARNs, trust policy, and External ID are all correct, check whether your AWS Organization applies a
  [resource control policy (RCP)](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_rcps.html)
  to the account that owns the invocation role. An RCP can deny `sts:AssumeRole` to principals outside your
  organization, including Temporal's, even when the role's trust policy allows them. The failure is an ordinary
  AssumeRole access-denied error, so it looks like a trust policy problem. Ask your AWS Organizations administrator to
  review the RCPs attached to the account, its organizational units, and the organization root, and to allow the
  Temporal principals listed in the role's trust policy.

- **Task Queue not registered.** The Lambda invokes successfully but the Worker errors before registering its Task Queue. Check the Lambda function's CloudWatch logs for configuration or runtime errors (missing environment variables, incorrect TLS configuration, missing dependencies). The server binds a Task Queue to a Worker Deployment Version when a Worker successfully connects and polls. Without a successful poll, the binding is never created and the WCI cannot route Tasks to the version.

  If the version was recently created, the Worker may still be starting up. Wait a moment and retry.

You can also check Task Queue bindings from the CLI:

```bash
temporal worker deployment describe-version \
  --namespace <NAMESPACE> \
  --deployment-name <DEPLOYMENT_NAME> \
  --build-id <BUILD_ID> \
  --report-task-queue-stats
```

### Check that the version is set as current 

The Worker Deployment Version must be set as the current version (or the ramping version) for Tasks to route to it. If you created the
version through the CLI, you need to
[set it as current](/production-deployment/worker-deployments/serverless-workers/aws-lambda#set-current-version).

You can verify the current version with `temporal worker deployment describe`.

## Lambda is invoked but Tasks are not completing 

If CloudWatch shows Lambda invocations but Workflows are not progressing, the problem is in the Worker's execution
within the Lambda function.

### Check Lambda execution logs 

Check CloudWatch logs for errors during Worker startup. In the AWS Console, go to **CloudWatch > Log groups >
/aws/lambda/your-function-name** and look for recent error messages.

Common errors include:

- **Connection failures**: The Worker cannot reach the Temporal Service. Check that the `TEMPORAL_ADDRESS` and
  `TEMPORAL_API_KEY` environment variables (or `temporal.toml` config file) are correctly set on the Lambda function.
  For self-hosted deployments, verify
  [network reachability](/production-deployment/worker-deployments/serverless-workers/aws-lambda/self-hosted-setup#ensure-network-reachability).
- **TLS errors**: The TLS certificate or key is missing, expired, or does not match the Namespace.
- **Authentication errors**: The API key is invalid or does not have access to the Namespace.

### Check for Lambda timeout 

If the Lambda function reaches its configured timeout before the Worker finishes processing, AWS terminates the
invocation.

The Worker begins graceful shutdown before the Lambda deadline. If Activities take longer than the available execution
window, the Activities are abandoned mid-execution and retried on the next invocation.

For long-running Activities, increase the Lambda timeout and the Worker's shutdown buffer together. See
[Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities) for guidance on how these
values relate.

### Check that the deployment name and build ID match 

If CloudWatch shows rapid, repeated invocations with no Workflow progress, the deployment name or build ID in the Worker
code may not match the Worker Deployment Version configuration.

The deployment name and build ID in your Lambda function code must exactly match the values you used when creating the
Worker Deployment Version. Compare the values in your code against the WCI Workflow ID
(`temporal-sys-worker-controller-instance:<deployment-name>:<build-id>`) and the output of
`temporal worker deployment describe`.

A mismatch causes an invocation loop: the WCI invokes the Lambda, the Worker starts and polls with a different
deployment version than the WCI expects, the Task is not processed, and the WCI invokes the Lambda again.

To fix the loop, update the deployment name and build ID in the Worker code to match the Worker Deployment Version, then
redeploy the Lambda function.

## Check for AWS-side issues 

If the checks above all pass, the cause may be in Lambda rather than in your configuration.
Rule out a provider-side cause:

- [AWS Health Dashboard service history](https://health.aws.amazon.com/health/status) reports active incidents by
  service and region.
- [Your account health events](https://health.aws.amazon.com/health/home) reports events scoped to your own resources,
  such as approaching service quotas.

If an incident explains what you are seeing, the fix is usually to wait it out or move to another region. Temporal
addresses a function by ARN, which includes the region, so changing region means deploying a new function and updating
the [compute configuration](/production-deployment/worker-deployments/serverless-workers/aws-lambda#create-worker-deployment-version)
on the Worker Deployment Version.

---

# Troubleshoot Serverless Workers on GCP Cloud Run

Source: https://docs.temporal.io/troubleshooting/serverless-workers/cloud-run

> **Public Preview**
> Cloud Run support is in Public Preview. APIs and configuration may change before the stable release.

This page walks through the Serverless Worker scaling flow on GCP Cloud Run and helps you identify where a failure is
occurring. For Lambda functions on AWS, see
[Troubleshoot Serverless Workers on AWS Lambda](/troubleshooting/serverless-workers/aws-lambda).

Cloud Run Workers run in a Worker Pool of long-lived instances. Temporal does not invoke your Worker per Task. Instead,
Temporal changes how many instances the pool runs. When the flow works correctly, the following sequence happens:

1. You deploy the Worker image to a Cloud Run Worker Pool with an instance count of zero.
2. You create a [Worker Deployment Version](/worker-versioning#deployment-versions) pointing at that pool. This starts a
   [Worker Controller Instance (WCI)](/serverless-workers#how-invocation-works) Workflow.
3. An instance starts, the Worker polls the Temporal Service, and the server binds the Worker's
   [Task Queue](/task-queue) to the Worker Deployment Version.
4. The WCI monitors that Task Queue. The [Matching Service](/temporal-service/temporal-server#matching-service) also
   signals the WCI when a Task arrives with no Worker free to take it.
5. As work arrives, the WCI raises the pool's instance count through the Cloud Run admin API. As work drains, it lowers
   the count, potentially to zero.

Start by determining whether the pool is running instances at all.

If no check below matches what you are seeing, or they all pass, [check for GCP-side issues](#check-gcp-issues).

## Is the Worker Pool running instances? 

Inspect the pool directly. The instance count the WCI has asked Cloud Run for is stored as an annotation on the pool:

```bash
gcloud run worker-pools describe <POOL_NAME> \
  --region <REGION> \
  --project <YOUR_GCP_PROJECT> \
  --format=yaml
```

Three fields under `metadata.annotations` tell you most of what you need:

| Field | What it tells you |
| --- | --- |
| `run.googleapis.com/manualInstanceCount` | The instance count currently requested. `0` means no Worker is running. |
| `run.googleapis.com/scalingMode` | Should be `manual`. The WCI scales the pool by writing its manual instance count. |
| `serving.knative.dev/lastModifier` | Who last changed the pool. If Temporal has ever scaled it, this is your invoker service account. |

Use `lastModifier` to choose where to look next. If `lastModifier` is still the account you deployed
with, Temporal has never successfully written to the pool, so work through
[the Worker Pool is not scaling up](#pool-not-scaling-up). If it is the invoker service account, Temporal is reaching
the pool, and the problem is more likely in the Worker itself. Skip to
[instances are running but Tasks are not completing](#instances-not-completing).

## The Worker Pool is not scaling up 

Work through the following checks in order.

### Validate the connection to the Worker Pool 

Start by verifying that Temporal can reach the pool. Go to **Workers > Deployments > select your deployment**, open the
**Actions** menu on the version, and click **Validate Connection**.

For Cloud Run, this impersonates the invoker service account and reads the Worker Pool. A successful validation confirms
three things at once: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the
invoker can read the pool. No Worker is started as part of validation.

If validation fails, check each part of the compute configuration against the pool:

- **Project, region, and pool name.** Temporal addresses the pool as
  `projects/<PROJECT>/locations/<REGION>/workerPools/<POOL_NAME>`. A wrong region reports the pool as not found, the
  same as a wrong name.
- **Impersonation.** The identity Temporal runs as needs `roles/iam.serviceAccountTokenCreator` on the invoker service
  account. On Temporal Cloud, the
  [Terraform module](/production-deployment/worker-deployments/serverless-workers/cloud-run#configure-iam) grants this.
  On a self-hosted Service, you grant it to the GCP identity the server runs as. See
  [Self-hosted setup](/production-deployment/worker-deployments/serverless-workers/cloud-run/self-hosted-setup#configure-gcp-identity).
- **Invoker permissions.** The invoker needs `run.workerPools.get` to read the pool and `run.workerPools.update` to
  scale it. `roles/run.developer` includes both. Validation only exercises the read, so an invoker that can read but not
  update validates successfully and then fails to scale.

A passing validation does not prove Temporal can scale the pool. If validation succeeds but `lastModifier` never
changes, check the update permission.

### Check that the version is set as current 

The Worker Deployment Version must be the current version before Tasks route to it. If you created the version through
the CLI, you need to
[set it as current](/production-deployment/worker-deployments/serverless-workers/cloud-run#set-current-version) as a
separate step.

Verify with `temporal worker deployment describe`.

### Check that the Task Queue is bound 

The WCI scales the pool in response to activity on the Task Queues bound to the Worker Deployment Version. If no Task
Queue is bound, there is nothing for it to watch.

Check which Task Queues are bound and whether there is a backlog:

```bash
temporal worker deployment describe-version \
  --namespace <NAMESPACE> \
  --deployment-name <DEPLOYMENT_NAME> \
  --build-id <BUILD_ID> \
  --report-task-queue-stats
```

The server creates the binding when a Worker running that Worker Deployment Version connects and polls. If no Task
Queues are listed, no Worker has ever polled successfully under this version. Confirm that an instance can start and
connect at all by running the image locally against the same Namespace, or by checking the pool's logs from a run where
an instance was up.

### Check whether the pool is at its maximum 

If the pool is running instances but stops growing while a backlog builds, it may be at the scaling algorithm's upper
bound. The `max_count` setting defaults to 30 instances. Raise it in the **Scaling and Lifecycle** settings on the
Worker Deployment Version.

If the count stalls below `max_count`, check your project's
[Cloud Run quotas](https://cloud.google.com/run/quotas) for the region. Cloud Run caps instances and CPU per region, and
those limits apply no matter what count the WCI requests.

## Instances are running but Tasks are not completing 

If the pool is running instances but Workflows are not progressing, the problem is in the Worker's execution.

### Check Worker Pool logs 

Read the pool's logs and look for errors during Worker startup:

```bash
gcloud run worker-pools logs read <POOL_NAME> \
  --region <REGION> \
  --project <YOUR_GCP_PROJECT>
```

The pool produces no logs while it is scaled to zero, so run this while an instance is up.

Common errors include:

- **Connection failures**: The Worker cannot reach the Temporal Service. Check the `TEMPORAL_ADDRESS` and
  `TEMPORAL_NAMESPACE` environment variables on the pool. For self-hosted deployments, verify
  [network reachability](/production-deployment/worker-deployments/serverless-workers/cloud-run/self-hosted-setup#ensure-network-reachability)
  from Cloud Run to your Temporal frontend.
- **Missing secrets**: The instance cannot read the Temporal Cloud API key or TLS material from Secret Manager. The
  [runner service account](/production-deployment/worker-deployments/serverless-workers/cloud-run#runner-service-account)
  the pool runs as needs `roles/secretmanager.secretAccessor` on the secret. This is the service account in
  `spec.template.spec.serviceAccountName`, not the invoker service account.
- **Authentication errors**: The API key is invalid, expired, or has no access to the Namespace.

### Check that the deployment name and build ID match 

If instances start and poll but no Task is ever processed, the deployment name or build ID in the Worker code may not
match the Worker Deployment Version.

Both values must match exactly. Compare the values in your Worker code against the output of
`temporal worker deployment describe`. On a mismatch, the Worker polls under a version the WCI does not manage, so its
polls never satisfy the Tasks the WCI is scaling for.

## Activities are interrupted mid-execution 

If Activities fail partway through and retry from the beginning around the time the pool shrinks, scale-in is stopping
instances that are still working.

The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing. It
does not track whether the instance Cloud Run stops is mid-Activity.

Use [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) so an interrupted Activity
resumes from its last recorded progress instead of restarting.

For how scale-in decisions are made, see
[Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#scale-in).

## Check for GCP-side issues 

If the checks above all pass, the cause may be in Cloud Run rather than in your configuration.
Rule out a provider-side cause:

- [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) lists current defects and limitations,
  including ones that affect how long Worker Pool operations take.
- [Google Cloud Service Health](https://status.cloud.google.com/) reports active incidents by product and region.

If a known issue or incident explains what you are seeing, the fix is usually to wait it out or move to another region.
Temporal addresses a Worker Pool by project, region, and name, so changing region means creating a new pool and updating
the [compute configuration](/production-deployment/worker-deployments/serverless-workers/cloud-run#create-worker-deployment-version)
on the Worker Deployment Version.

---

# Troubleshoot Worker capacity

Source: https://docs.temporal.io/troubleshooting/worker-capacity

> Diagnose exhausted Task slots, disconnected pollers, Task backlog, and Sticky Execution cache problems

This guide covers what to do when your Workers stop keeping up with their Task Queues: slots that never free up, pollers that quit polling, Tasks sitting in the queue, and a [Sticky Execution](/sticky-execution) cache that isn't earning its keep.
It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

Every section on this page is one of the recommended alerts in [Worker alerting](/best-practices/worker-alerting).
Each one opens with the alert definition, explains what the condition costs you, then walks through triage.
If you got here from a firing alert, the section you landed on is the runbook for it.

For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker).
To see how Slot, Poller, and cache limits produce these symptoms, try the [Worker tuning simulator](/demos/worker-tuning-simulator).

These failure modes are chained together.
Slots fill up, polling stops, Tasks pile up, completions go to zero.
When several fire at once, don't treat them as separate problems. Work backward to the earliest link in the chain.

## Worker Task slots exhausted 

**Recommended alert:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available), tagged `worker_type`
Fires when the gauge reaches 0, held for 2m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).

Every Task execution slot for this `worker_type` and Task Queue is occupied, so no new Tasks can be picked up.
The SDK blocks before issuing its next poll until a slot is released.

> **📝 Note:**
>
> This metric reports meaningful values only with fixed-size slot suppliers.
> It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance/metrics#slot-availability-metrics).
> If your Workers use resource-based tuning, skip this section and let [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated) tell you when capacity is short.
>

**Why it matters.**
Slots stay occupied because the Tasks already holding them aren't finishing.
Until they do, schedule-to-start latency climbs. In the Go and Java SDKs, [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) also drops to zero, because both block on getting a slot before they bump the poller gauge.

What this costs you and how you fix it both depend on `worker_type`, so check that tag on the firing alert.
The values are `WorkflowWorker`, `ActivityWorker`, and `LocalActivityWorker`.

### WorkflowWorker slots 

1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high).
1. **Check Worker CPU.** High CPU slows Workflow Task execution directly and keeps slots occupied longer.
1. **Check for blocking calls in Workflow code.** A slot is held until the Workflow Task completes. Blocking I/O, heavy computation, or synchronous non-Temporal API calls inside a Workflow hold the slot far longer than expected. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). Slots aren't released until the respond call succeeds.

To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options.

### ActivityWorker slots 

1. **Check Activity execution latency.** Sustained high [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` confirms Activities are holding slots longer than expected.
1. **Check Worker CPU.** High CPU slows Activity execution directly.
1. **Check what the Activities are waiting on.** If slots are exhausted because Activities are blocked on a downstream service, raising the slot count increases pressure on that service and makes the problem worse. Investigate the dependency before scaling concurrency.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations).

To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size. Confirm a downstream dependency isn't the real constraint first, or you will just push harder on something that is already struggling.

### LocalActivityWorker slots 

[Local Activities](/local-activity) run inside the Workflow Task execution loop, so a blocked slot holds up the whole Workflow Task.
The SDK keeps the Task alive by sending repeated Workflow Task heartbeats.
If that goes on past the Workflow Task heartbeat timeout, 30 minutes by default, the Temporal Service times the Task out and reschedules it, and every Local Activity in it runs again from the start.

1. **Check what the Local Activities are waiting on.** The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected `activity_type`. If it calls a downstream service, check whether that service is slow or throttling.
1. **Check Worker CPU.**
1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout). If that is firing too, you are close to the heartbeat timeout or already past it, and re-execution has started.

## All pollers disconnected 

**Recommended alert:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers), tagged `poller_type` and `task_queue`
Fires when the gauge reaches 0, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).

No active pollers remain for this poller type and Task Queue.
Workers have stopped polling entirely and Tasks are accumulating with nothing to process them.

> **📝 Note:**
>
> This metric is tagged `poller_type`, not `worker_type`.
> The values are `workflow_task`, `workflow_sticky_task`, `activity_task`, and `nexus_task`.
> Alert on each poller type your Workers use, and note that a Worker with an active Sticky Execution cache reports `workflow_sticky_task` alongside `workflow_task`.
>

**Why it matters.**
Workflow and Activity Tasks accumulate on the Temporal Service.
At scale this grows into a large backlog that puts pressure on Task matching and persistence.
Depending on your Workflow and Activity timeouts, Executions begin timing out while waiting for Tasks to be processed.

**Triage.**

1. **Check whether Worker processes are running.** Look at pod status, restart counts, and logs. A crash, an eviction, or an OOM kill is the most common cause by a wide margin.
1. **Rule out scale-down.** If you autoscale Workers or scale to zero on an idle Task Queue, a zero poller count is expected. Confirm against your scaling events before investigating further.
1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first.
1. **Check for authentication failures.** Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry.
1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service). Sustained INTERNAL errors make Workers back off until they stop polling altogether.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Cross-check the server-side concurrent poller count for this Task Queue.
> If it has also dropped, Workers have fully disconnected from the Service's perspective.
> For authentication failures, check unauthorized request counts and authorization system failures.
> A non-zero authorization system failure count means the auth plugin itself is broken, which is a bigger problem than an expired credential.
>

## Task completions dropped to zero 

**Recommended alert:** [`temporal_request`](/references/sdk-metrics#request) on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted`
Fires when the rate reaches 0 while the Task Queue still has demand, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).

The SDK only bumps this counter after a response succeeds, so a zero rate means nothing is landing.

**Check for demand before you act on this one.**
A zero rate looks the same whether your Workers are broken or the Task Queue is quiet: outside business hours, or a batch workload between runs, zero is the correct answer.
The metric is tagged `namespace` and `operation`, not `task_queue`, so you can't narrow it to one Task Queue on its own either.

Look for a non-zero Workflow or Activity start rate, a growing Task backlog, or schedule-to-start latency climbing.
If none of those are there, nothing is waiting to be completed and the metric is telling you the truth.

**Why it matters.**
When there is demand and Workflow Task completions are still zero, your Workflow Executions aren't moving.
Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them.

No Activity Task completions means Workflows waiting on Activity results are stalled.
Activities start timing out against their `scheduleToClose` timeouts and the Temporal Service retries them within the Retry Policy, but with no Workers completing anything the retries just pile up alongside the originals.

**Triage.**

1. **Confirm there is demand.** Check the Workflow and Activity start rate, or the Task backlog on the Task Queue. If neither shows demand, the Task Queue is idle and no further action is needed.
1. **Check whether polling also stopped.** Look at [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the relevant `poller_type`. If pollers are at zero too, your Workers are down entirely. Go to [All pollers disconnected](#all-pollers-disconnected).
1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed). Workers may be failing every Task. See [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large).
1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). If every slot is occupied, no new Tasks are picked up or completed.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). Sustained throttling suppresses this counter directly, since the SDK only counts responses that succeeded.
1. **Check schedule-to-start latency.** See [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated). If Tasks are not being dispatched, completions drop to zero even with healthy Workers.
1. **For Activity completions, check Workflow Task health first.** Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check overall cluster health: service errors by Namespace, persistence availability, and resource-exhausted counts by cause.
>

## Workflow Task schedule-to-start latency elevated 

**Recommended alert:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency), tagged `namespace` and `task_queue`
Two alerts at different thresholds. One fires when p99 goes above 5s, the other above 30m. Both held for 5m and routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).

Workflow Tasks are sitting in the queue before a Worker picks them up.
Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue.

Two other pages give thresholds for this metric, for different purposes.
[Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike) treats P95 above one second as worth a look. Use that number when you are tuning Worker capacity, not when you are deciding whether to page someone.
[Worker health](/cloud/worker-health#schedule-to-start-latency) supplies ready-made Prometheus queries for both this metric and the Activity equivalent.
The thresholds on this page are set higher because they are meant to page someone.

**Why it matters.**
Schedule-to-start latency adds directly to Workflow end-to-end latency.
Every second a Task waits is a second added to how long your Workflows take.

Past roughly the Workflow Task heartbeat timeout, Executions are effectively stalled rather than slow: anything waiting on a Workflow Task is making no progress at all.
At scale a large Task backlog accumulates, putting significant pressure on Task matching and the underlying database.
A backlog large enough can affect an entire self-hosted cluster, not just the affected Namespace and Task Queue.

**Triage.**

1. **Check Worker health.** Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else.
1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `poller_type=workflow_task`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected).
1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). With every slot occupied the SDK blocks before its next poll, so the effective poll rate drops.
1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations). Throttled Workers back off and poll less often.
1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, your Worker pool is too small for current throughput. Scale horizontally, and expect latency to stay high for a while afterward. The backlog you already built up has to drain first.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check the approximate Task backlog in the Matching Service section of your server dashboard.
> A growing backlog confirms Tasks are arriving faster than Workers pick them up.
> Then check the server-side concurrent poller count for this Task Queue to see whether your Worker pool has shrunk.
>

## Activity schedule-to-start latency elevated 

**Recommended alert:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency), tagged `namespace` and `task_queue`
Fires when p99 goes above 30m, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).

Activity Tasks are sitting in the queue before a Worker picks them up.

As with the Workflow Task equivalent above, [Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_activity_schedule_to_start_latency-spike) gives a lower investigation threshold for this metric, and [Worker health](/cloud/worker-health#schedule-to-start-latency) supplies the Prometheus query.

**Why it matters.**
Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency.
At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database.

**Triage.**

1. **Check Worker health.** Confirm Activity Worker processes are running and not restarting.
1. **Check poller counts** for `poller_type=activity_task`. See [All pollers disconnected](#all-pollers-disconnected).
1. **Check Task slots** for `worker_type=ActivityWorker`. See [Worker Task slots exhausted](#worker-task-slots-exhausted).
1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations).
1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated). Lots of failures means lots of retry Tasks, which grow the backlog and hold this latency up even after you have restored Worker capacity.
1. **Check for Task Queue rate limits.** A low `TaskQueueActivitiesPerSecond` caps how fast Activities are dispatched regardless of Worker capacity. See [Performance bottlenecks](/troubleshooting/performance-bottlenecks).

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check the approximate Task backlog and the server-side concurrent poller count for this Task Queue, as in the Workflow Task section above.
>

## Sticky cache holding zero entries under load 

**Recommended alert:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size)
Fires when the gauge reaches 0 while the Worker is processing Workflow Tasks, held for 15m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).

This gauge tells you how many Workflow Executions the Worker is holding in its [Sticky Execution](/sticky-execution) cache right now.
It is not the configured cache size.

**Zero is normal on an idle Worker.**
A Worker that isn't processing Workflow Tasks has nothing to cache, and a low-volume Task Queue can sit at zero for hours.
Alert on this gauge by itself and you will get paged for Workers that are doing nothing wrong.

What you actually want to catch is zero occupancy while the Worker is working.
Pair the gauge with a non-zero Workflow Task rate on the same Worker and give it a long `for` duration.
High [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) or a high [`temporal_sticky_cache_miss`](/references/sdk-metrics#sticky_cache_miss) rate will confirm it: both mean Workflow Tasks are replaying from history instead of picking up where they left off.

Tag coverage is patchy, so there is a limit to how tightly you can scope this alert. Only the TypeScript and Java SDKs emit `namespace`, and only TypeScript emits `task_queue`.

**Why it matters.**
With nothing in the cache, every Workflow Task pulls all the History pages from the Temporal Service and re-runs every command from the beginning.
Each one costs you what a cache miss costs.

At any real scale that means sustained load on the Temporal Service and higher Workflow Task execution latency for every Execution on the Worker.
See [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high).

**Triage.**

1. **Confirm the Worker is actually processing Workflow Tasks.** If the Workflow Task rate is zero, the Worker is idle and there is nothing to fix.
1. **Check whether the cache is disabled by configuration.** A cache size of zero in Worker options means no Execution is ever cached:
   - **Go SDK:** `worker.SetStickyWorkflowCacheSize(int)` sets the size, defaulting to 10,000. Passing `0` turns the cache off completely, so restore a non-zero value. Call it before any Worker starts; the cache is shared across every Worker in the process.
   - **Java SDK:** `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets it to the default instead of disabling it, and a negative value is rejected outright, so you can't reach a zero-size cache this way. Check `setMaxWorkflowThreadCount(int)` instead. A thread pool set too low starves Workflow execution and the cache never gets used properly.
1. **Check the eviction rate.** If the cache is configured correctly but occupancy stays at zero, entries are being evicted as fast as they go in. Check [`temporal_sticky_cache_total_forced_eviction`](/references/sdk-metrics#sticky_cache_total_forced_eviction). A high rate points at either a cache too small for the number of concurrent Executions on that Worker, or repeated Workflow Task timeouts forcing evictions.
1. **Verify the fix.** After redeploying, `temporal_sticky_cache_size` should climb from zero under load, and [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) should fall as cold replays are replaced by cache hits.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check persistence latency filtered to `GetWorkflowExecution`. Sustained high latency there confirms the Temporal Service is straining under all the repeated History reads.
>

---

# Temporal Visibility

Source: https://docs.temporal.io/visibility

> Temporal Visibility enables operators to view, filter, and search Workflow Executions using List Filters and Search Attributes. Learn about supported databases, Dual Visibility, and custom Search Attributes.

This page provides an overview of Temporal Visibility.

The term [Visibility](/visibility), within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Temporal Service.

The [Visibility store](/self-hosted-guide/visibility) in your Temporal Service stores persisted Workflow Execution Event History data and is set up as a part of your [Persistence store](/temporal-service/persistence) to enable listing and filtering details about Workflow Executions that exist on your Temporal Service.

- [How to set up a Visibility store](/self-hosted-guide/visibility)

With Temporal Server v1.21, you can set up [Dual Visibility](/dual-visibility) to migrate your Visibility store from one database to another.

## Visibility features 

Visibility enables the listing, filtering, and sorting of [Workflow Executions](/workflow-execution) through a custom SQL-like [List Filter](/list-filter).

Visibility supports [custom Search Attributes](/search-attribute#custom-search-attribute) for user-defined filtering beyond the default system attributes.

- On SQL databases (MySQL v8.0.17+, PostgreSQL v12+), Visibility is available with Temporal Server v1.20 and later.
- On Elasticsearch (v7+ with Temporal Server v1.7+, v8 with Temporal Server v1.18+) and OpenSearch (2+ with Temporal Server v1.30.1+).
- On Temporal Cloud, Visibility is enabled by default for [all users](/cloud/manage-access/users#invite-users).

For self-hosted setup and version compatibility details, see [Visibility store setup](/self-hosted-guide/visibility).

### Count Workflow by ExecutionStatus

The Count API feature lets you count the number of Workflows that match a given query. For example, the command `temporal workflow count -q "WorkflowType='foo'"` returns the number of Workflows that match `WorkflowType='foo'`.

You can send queries to the Count API to group by a given search attribute. For example, `-q "WorkflowType='foo' GROUP BY ExecutionStatus` returns the number of Workflows that match `WorkflowType='foo'` grouped by `ExecutionStatus`.

The `GROUP BY` clause is only supported in the Count API and currently only grouping by `ExecutionStatus` is supported.

> **📝 Note:**
>
> The Count API returns approximate counts.
>

## Operations that use Visibility

Visibility has uses beyond the workflow search APIs. Any operation that lists, searches, or counts across executions reads the Visibility index, some of which relate to other features:

- Workflow [search](/develop/typescript/platform/observability#search-attributes) and count: `ListWorkflowExecutions`, `ScanWorkflowExecutions`, `CountWorkflowExecutions`
- [Schedule list](/troubleshooting/schedule-missed-actions#investigate-which-schedule-missed-an-action) and count: `ListSchedules` and `CountSchedules`
- Batch operations: `ListBatchOperations`
- [Worker task-queue reachability](/develop/worker-performance/task-queues#cli-task-queue-info): `GetWorkerTaskReachability`, and `DescribeTaskQueue` when reachability is requested

These operations share the Visibility rate limit and the eventual-consistency behavior described below. Other read APIs, such as worker deployment listing, are also subject to the [Visibility API rate limit](/evaluate/cloud/limits#visibility-api-rate-limit), where you'll find the full list.

Operations that look up a single entity by ID, such as `DescribeWorkflowExecution` and `DescribeSchedule`, do not use Visibility.

## How current is Visibility data

Visibility is a search index that Temporal updates asynchronously. After a change is recorded, it takes some time to propagate to the index, so a List or Count query can briefly return stale results. This propagation delay is variable. It is usually fast, but it can be several seconds or longer, and Temporal Cloud does not publish an [SLA](https://temporal.io/sla) for how quickly changes become visible. Treat Visibility as eventually consistent and avoid depending on a fixed or guaranteed delay.

Visibility is built for finding and filtering across many executions, not for reading the current state of one execution. When you need the authoritative, up-to-date state of a specific Workflow Execution, use `DescribeWorkflowExecution` instead of a Visibility query.

## Visibility availability in Temporal Cloud 

Temporal Cloud targets 99.5% availability for Visibility read requests, measured over a rolling 30-day window. This covers List and Count operations against Workflow Executions, Activity Executions, Nexus Operations, Schedules, and Batch Operations.

This is a service level objective rather than a contractual commitment. Visibility API requests are not covered by the Temporal Cloud availability [SLA](https://temporal.io/sla), and no service credits are associated with this objective. The target is deliberately lower than the availability guaranteed for core Workflow Service APIs, because Visibility is a search index intended for operational discovery rather than a critical path for application logic.

The objective measures whether a Visibility read request returns a valid response. Requests that fail for reasons within your control are not counted against it, including requests that exceed your Namespace's rate limits and requests rejected as malformed or unsupported.

It says nothing about how current your results are. As described above, Visibility is eventually consistent, and an empty result is a valid response. It also does not cover query latency, which varies with the complexity of your query and the size of your result set.

For self-hosted deployments, Visibility availability depends on the database you operate.

## When to use Visibility

Visibility is intended for operational discovery and search: finding, filtering, and counting executions for monitoring, tooling, dashboards, and investigation. Use it when you want to search or filter executions by Search Attribute in the UI, CLI, or an operational tool, or when you want to count or sample executions for reporting.

For a few common tasks, another approach works better:

- To check whether a specific workflow exists, start it and handle the already-started result, or look it up by ID with [`DescribeWorkflowExecution`](/references/operation-list).
- To react to a workflow's progress or completion, follow its [Event History](/workflow-execution/event) or model the dependent work as [child workflows](/child-workflows) instead of polling Visibility.
- To coordinate between workflows, use signals, updates, or child workflows rather than reading Visibility from inside your workflow or activity code.

Keeping per-workflow work on these paths gives you strongly consistent reads and keeps the shared Visibility index healthy under load.

## Legacy: Standard Visibility 

Prior to Temporal Server v1.20, Temporal had two Visibility modes: "standard" and "advanced." Standard Visibility supported only predefined filters such as Workflow Type, Workflow Id, Run Id, and Execution Status, without custom Search Attributes. Advanced Visibility required Elasticsearch.

Starting with Temporal Server v1.20, advanced Visibility became available on SQL databases. Standard Visibility was deprecated in v1.21 and removed in v1.24. All current deployments use what was formerly called advanced Visibility, now simply called Visibility.

If you are running a Temporal Server version older than v1.24, see the [legacy standard Visibility section](/self-hosted-guide/visibility/legacy-standard) in the self-hosted guide.

---

# Temporal Web UI

Source: https://docs.temporal.io/web-ui

> Browse, filter, and inspect Workflow Executions with Search Attributes and Saved Views, included with the CLI and Cloud.

The Temporal Web UI provides users with Workflow Execution state and metadata for debugging purposes. It ships with
every [Temporal CLI](/cli) release and is available with [Temporal Cloud](/cloud).

You can configure the Temporal Web UI to work in your own environment. See the
[UI configuration reference](/references/web-ui-configuration).

Web UI open source repos:

- [temporalio/ui](https://github.com/temporalio/ui)
- [temporalio/ui-server](https://github.com/temporalio/ui-server)

## Namespaces

All Namespaces in your self-hosted Temporal Service or Temporal Cloud account are listed under **Namespaces** in the
left section of the window. You can also switch Namespaces from the Workflows view by selecting from the Namespace
switcher at the top right corner of the window. After you select a Namespace, the Web UI shows the Recent Workflows page
for that Namespace. In Temporal Cloud, users can access only the Namespaces that they have been granted access to. For
details, see [Namespace-level permissions](/cloud/manage-access/roles-and-permissions#namespace-level-permissions).

## Workflows

The main Workflows page displays a table of all Workflow Executions within the retention period.

Users can list Workflow Executions by any of the following:

- [Status](/workflow-execution#workflow-execution-status)
- [Workflow ID](/workflow-execution/workflowid-runid#workflow-id)
- [Workflow Type](/workflow-definition#workflow-type)
- Start time
- End time
- Any other Default or Custom [Search Attribute](/search-attribute) that uses [List Filter](/list-filter)

For start time and end time, users can set their preferred date and time format as one of the following:

- UTC
- Local
- Relative

Select a Workflow Execution to view the Workflow Execution's History, Workers, Relationships, pending Activities and
Nexus Operations, Queries, and Metadata.

### Saved Views 

Saved Views let you save and reuse your frequently used visibility queries in the Temporal Web UI. Instead of recreating
complex filters every time, you can save them once and apply them with a single click.

Saved Views are stored locally in your browser and are available to you whenever you use the Temporal Web UI in this
browser. Each user will have their own private collection.

#### Apply a Saved View

By default, The Workflows page has several default Saved Views. You can also create your own Saved Views.

Click the name of a Saved View in the list to display the corresponding Workflows that match the query.

The Workflow List page will refresh with the results of the Saved View.

#### Create a Saved View

You can create a new Saved View from the Workflows page.

1. Create a Saved View by using the filter UI to build your criteria, or you can use the raw query editor to write
   custom query strings.
1. Your new view will appear in the Custom Views list as New View. Click the Save as New button to bring up the Save as
   View window. Name your Saved View. Names must be unique to each user and can contain a max of 255 characters.
1. Click Save. Your new view will appear in the Custom Views list

You can create up to 20 Saved Views. When you reach this limit, you'll need to delete some Saved Views before you can
save new ones.

#### Make temporary changes to a Saved View query

You can modify a Saved View temporarily without changing the saved criteria.

1. Select the Saved View you want to change.
1. Adjust the UI filters as needed.
1. The Workflows page will refresh with the results of the new query, without changing the Saved View.
1. If you want to keep your temporary changes, you can:
   - Click Save, which will replace the original Saved View with your modifications.
   - Click Edit, modify the name, and click Save, which will replace the original Saved View with your modifications and
     change the name.
   - Click Edit, modify the name, and click Create New, which will create a new Saved View with your new settings and a
     new name.

#### Rename a Saved View Query

You can rename an existing Saved View from the Workflows page.

1. Select the Saved View you want to change.
1. Click Edit.
1. In the Edit View dialog box, enter a new name for the Saved View.
1. Click Save to apply your changes and rename the existing Saved View, or click Create Copy to create a new Saved View
   with the new name.

#### Deleting Saved Views

You can delete a Saved View from the Workflows page, because it is no longer useful, or to create room for new Saved
Views.

1. Select the Saved View you want to delete. You can only delete queries you’ve created; you cannot delete the system
   defaults.
1. Click “Edit” and then "Delete this Saved View".

> **📝 Note:**
> Deleting Saved Views is permanent
>
> Deleted queries cannot be recovered, so make sure you won't need them again. If you accidentally delete a Saved List,
> you will need to recreate it.
>

#### Share a Saved View

You can share a Saved View as a URL.

1. Select the Saved View you want to share.
1. Click the “Share” button to copy the URL for this Saved View to the clipboard. You can also copy the URL directly
   from the browser.

> **📝 Note:**
> Saved Views and time
>
> Saved Views that use relative times will be shared with absolute time.
>

## Task Failures view 

The Task Failures view is a pre-defined Saved View that displays Workflows that have a Workflow Task failure.
These Workflows are still running, but one of their Tasks has failed or timed out. 

The details of the Task Failures view displays the Workflow's ID, the Run ID, and the Workflow type. 
Clicking on any of the links in the details opens the Workflow page for that Workflow. 
On this page, you will find more information about the Task that failed and remaining pending tasks.
You can also cancel the Workflow by clicking the Request Cancellation button on this page.

Our system monitors Workflow task execution patterns in real-time. When a Workflow experiences five consecutive task failures or timeouts, it gets automatically flagged. The moment the Workflow recovers with a successful task, the flag clears. This smart threshold filters out minor glitches while surfacing Workflows with genuine problems.

### Activating Task Failures view 

This is enabled by default for Temporal Cloud users. If you're self-hosting Temporal, you'll need to update the `system.numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute` [dynamic config](/references/dynamic-configuration).

Here's an example of how to make the config update for the dev server:

```command
temporal server start-dev \
 --dynamic-config-value system.numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute=5
```

`numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute` is the number of consecutive Workflow Task Failures required to trigger the `TemporalReportedProblems` search attribute. The default value is 5. If adding this search attribute causes strain on the visibility system, consider increasing this number.

To turn off the feature for a Namespace, set `numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute` to 0.

## History

A Workflow Execution History is a view of the [Events](/workflow-execution/event#event) and Event fields within the
Workflow Execution. Approximately [40 different Events](/references/events) can appear in a Workflow Execution's Event
History.

The top of the page lists the following execution metadata:

- Start Time, Close Time and Duration
- [Run Id](/workflow-execution/workflowid-runid#run-id)
- [Workflow Type](/workflow-definition#workflow-type)
- [Task Queue](/task-queue)
- Parent and Parent ID
- SDK
- [State Transitions](/workflow-execution#state-transition)
- [Billable Actions Count](/cloud/actions-usage#actions-in-workflows) (Temporal Cloud only)

The Input and Results section displays the function arguments and return values for debugging purposes. Results are not
available until the Workflow finishes.

The History tab has the following views:

- Timeline: A chronological or reverse-chronological order of events with a summary. Clicking into an Event displays all
  details for that Event.
- All: View all History Events.
- Compact: A logical grouping of Activities, Signals and Timers.
- JSON: The full JSON code for the workflow.

### Download Event History

The entire Workflow Execution Event History, in JSON format, can be downloaded from this section.

### Workflow Actions

Workflow Executions can request a Cancellation, send a Signal or Update, or Reset and Terminate directly from the UI.
Start a new Workflow Execution with pre-filled values with the Start Workflow Like This One button.
Signal and Update input, and the pre-filled Workflow input, are each a single argument. See
[Pass input from the Web UI](#input).

### Relationships

Displays the full hierarchy of a Workflow Execution with all parent and child nodes displayed in a tree.

### Workers

Displays the Workers currently polling on the Workflow Task Queue with a count. If no Workers are polling, an error
displays.

### Pending Activities

Displays a summary of recently active and/or pending Activity Executions. Clicking a pending Activity directs the user
to the Pending Activities tab to view details.

### Call stack

The screen shows the captured result from the [\_\_stack_trace](/sending-messages#stack-trace-query) Query. The Query is
performed when the tab is selected. It works only if a Worker is running and available to return the call stack. The
call stack shows each location where Workflow code is waiting.

### Queries

Lists all Queries sent to the Workflow Execution. A Query you send from this tab takes a single argument. See
[Pass input from the Web UI](#input).

### Metadata

Displays User Metadata including static Workflow Summary and Details and dynamic Current Details. Lists all Events with
User Metadata data to give you a human-readable log of what's happening in your Workflow.

## Pass input from the Web UI 

The **Data** box on the Start Workflow form, the Signal and Update dialogs, the Queries tab, and the Schedule form takes
a single JSON value. The Web UI sends that value to your code as one argument.

- For a Workflow, Signal, Update, or Query that takes one argument, enter the argument as a JSON value, such as
  `"order-123"` or `{"orderId": "order-123", "quantity": 2}`.
- You can't pass more than one argument from the Web UI. A JSON array such as `["order-123", 2]` arrives as one argument
  whose value is the array.

To pass more than one argument, use the [Temporal CLI](/cli/command-reference/workflow) and repeat `--input` once for each
argument, in order:

```command
temporal workflow start \
    --type YourWorkflowType \
    --task-queue YourTaskQueue \
    --workflow-id YourWorkflowId \
    --input '"order-123"' \
    --input 2
```

`temporal workflow signal`, `temporal workflow update execute`, and `temporal workflow query` accept `--input` the same
way. To start a Workflow from the Web UI, give its Workflow Definition a single object parameter instead of several
parameters. This is also the general [recommendation for Workflow parameters](/workflow-definition#workflow-parameters).

If the Web UI sends fewer arguments than a Workflow Definition or Signal handler declares, the result depends on the SDK.
For example:

- In the Python SDK, the Workflow Task fails with a `TypeError` and retries. The Workflow Execution stays Running without
  making progress and can appear in the [Task Failures view](#task-failures-view).
- In the Go SDK, a Workflow Execution runs with the zero value for each missing argument. If the input can't be decoded
  into the first parameter, the Workflow Execution fails.

**Start Workflow Like This One** copies only the first argument of the original Workflow Execution. If the original
Workflow Execution took more than one argument, start the new one with the CLI.

## Schedules

On Temporal Cloud and self-hosted Temporal Service Web UI, the Schedules page lists all the [Schedules](/schedule)
created on the selected Namespace.

Click a Schedule to see details, such as configured frequency, start and end times, and recent and upcoming runs.
The Workflow input you set when you create a Schedule in the Web UI is a single argument. See
[Pass input from the Web UI](#input).

> **💡 Tip:**
> Setting Schedules with Strings
>
> Temporal Workflow Schedule Cron strings follow this format:
>
> ```
> ┌───────────── minute (0 - 59)
> │ ┌───────────── hour (0 - 23)
> │ │ ┌───────────── day of the month (1 - 31)
> │ │ │ ┌───────────── month (1 - 12)
> │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
> │ │ │ │ │
> * * * * *
> ```
>

To read more about Schedules, explore these links:

**Related:**

- [Schedules using the Go SDK](/develop/go/workflows/schedules)
- [Schedules using the Java SDK](/develop/java/workflows/schedules)
- [Schedules using the PHP SDK](/develop/php/workflows/schedules)
- [Schedules using the Python SDK](/develop/python/workflows/schedules)
- [Schedules using the TypeScript SDK](/develop/typescript/workflows/schedules)
- [Schedules using the .NET SDK](/develop/dotnet/workflows/schedules)

### Settings

On Temporal Cloud, **Settings** is visible only to Account Owner and Global Admin
[roles](/cloud/manage-access/roles-and-permissions#account-level-roles).

Click **Settings** to see and manage the list of users in your account and to set up integrations such as
[Observability](/cloud/monitor) and [Audit logging](/cloud/audit-logs).

On a self-hosted Temporal Service, manage your users, metrics, and logging in your
[server configuration](/references/service-configuration).

### Archive

On a self-hosted Temporal Service, Archive shows [Archived](/temporal-service/archival) data of your Workflow Executions
on the Namespace.

To see data in your self-hosted Temporal Service, you must have
[Archival set up and configured](/self-hosted-guide/archival).

For information and details on the Archive feature in Temporal Cloud, contact your Temporal representative.

### Codec Server

The Web UI can use a [Codec Server](/codec-server) with a custom Data Converter to decode inputs and return values. For
details, see [Securing your data](/production-deployment/data-encryption).

The UI supports a [Codec Server endpoint](/production-deployment/data-encryption#web-ui). For details on setting the
Codec Server endpoint, see [Codec Server setup](/production-deployment/data-encryption#codec-server-setup).

---

# Develop with AI

Source: https://docs.temporal.io/with-ai

> Give your AI coding agent Temporal expertise and real-time access to Temporal documentation.

Give your AI coding agent Temporal expertise with Skills, real-time documentation access with the Temporal Knowledge Base
MCP Server, or direct access to the docs as Markdown.

## Skills

Skills give AI agents domain-specific Temporal expertise. They work with Claude Code, Codex, Cursor, and any agent that
supports [Skills](https://agentskills.io).

### Install the coding agent plugin

The Temporal plugin installs a bundle of skills. Install it once, then read the sections below for what each skill
covers.

**Claude Code Plugin**

1. Add the Temporal skills marketplace to Claude Code:

   ```bash
   /plugin marketplace add temporalio/claude-temporal-plugin
   ```

2. Install the Temporal plugin:

   ```bash
   /plugin install temporal@temporal-marketplace
   ```

**Cursor**

Install the Temporal plugin from the [Cursor Marketplace](https://cursor.com/marketplace/temporal), or run the following
command in Cursor's agent chat:

```
/add-plugin temporal
```

**Codex**

Install from the Codex app or CLI:

- **Codex app:** Open the plugins menu, search for **temporal**, then click **+** (or **Add to Codex**).
- **Codex CLI:** Run `/plugins`, select the **OpenAI Curated** marketplace, search for **temporal**, and install the
  plugin.

Restart your coding agent after installing.

Every skill in the bundle also has its own repository, linked in the sections below. If you want one skill instead of
the bundle, install it with the `skills` CLI (`npx skills add <repository-url>`) or clone the repository into your
agent's skills directory.

### Temporal Developer skill

The [Temporal Developer skill](https://github.com/temporalio/skill-temporal-developer) gives your AI coding agent
expert-level knowledge of Temporal's programming model, including Workflow Definition determinism rules, Activity patterns, Retry
Policies, error handling, testing strategies, Worker configuration, versioning, and common gotchas.

### Temporal Ops skill

> **Public Preview**
> The skill's coverage and structure may change before the stable release.

The [Temporal Ops skill](https://github.com/temporalio/skill-temporal-ops) administers and diagnoses running Temporal
Cloud and self-hosted environments through the `temporal` and `tcld` CLIs: Namespace management, API key and mTLS
certificate rotation, Cloud capacity, batch cancel, terminate, and reset operations, Export, and Search Attributes. It
also works backwards from a symptom until it reaches a root cause. Symptoms include a stuck Workflow, a non-determinism
error, unhealthy Workers, a Task Queue backlog, or a missed Schedule.

Trigger the skill with a slash command: `/temporal:temporal-ops`.

### Temporal Serverless skill

> **Public Preview**
> The skill's coverage and structure may change before the stable release.

The [Temporal Serverless skill](https://github.com/temporalio/skill-temporal-serverless) guides your AI coding agent
through configuring and deploying [Serverless Workers](/serverless-workers), and troubleshooting them once they're
running.

Trigger the skill with a slash command: `/temporal:temporal-serverless`.

### Temporal Cloud skill

The [Temporal Cloud skill](https://github.com/temporalio/skill-temporal-cloud) is a standalone skill that isn't part of
the plugin bundle. It helps your AI coding agent troubleshoot Temporal Cloud connectivity, authentication, and
configuration issues.

**npx**

This works with Claude Code, Codex, Cline, and other agents.

Install the skill using the `skills` CLI:

```bash
npx skills add https://github.com/temporalio/skill-temporal-cloud
```

**Manual**

Clone the skill repository into your Claude skills directory. Change the target directory if you are using agents other
than Claude:

```bash
git clone https://github.com/temporalio/skill-temporal-cloud.git ~/.claude/skills/temporal-cloud
```

Restart your coding agent after installing.

## Temporal knowledge base MCP server

Connect Temporal expertise directly to your AI assistant for accurate, up-to-date answers about Temporal. The
Temporal knowledge base MCP server gives AI tools real-time access to best practices compiled from our documentation, educational materials, community forum responses, and slack channels, so responses draw from current expertise
rather than training data.

> **ℹ️ Info:**
> Authentication required
>
> The Temporal Knowledge Base MCP Server is publicly available, but requires a one-time login with a Google or GitHub account to enforce rate limits and prevent abuse.
> Only an opaque user ID is used for rate limiting. Your name, email, repositories, and other personal data are not accessed or collected.
>

### Claude Code

Add the Temporal knowledge base MCP server globally so it's available in all your projects:

1. Register the MCP server with Claude Code:

   ```bash
   claude mcp add --scope user --transport http temporal-docs https://temporal.mcp.kapa.ai
   ```

2. Restart Claude Code and run `/mcp` to authenticate with your Google account.

To add the server to a specific project only, omit the `--scope user` flag. This stores the configuration in the
project's `.mcp.json` file:

```bash
claude mcp add --transport http temporal-docs https://temporal.mcp.kapa.ai
```

### Claude Desktop

1. Open Claude Desktop settings
2. Navigate to **Settings > Connectors**
3. Add a new MCP server with the URL: `https://temporal.mcp.kapa.ai`

### Other MCP-compatible tools

The Temporal Knowledge Base MCP Server URL is:

```
https://temporal.mcp.kapa.ai
```

The server requires authentication through MCP OAuth.
Not all MCP clients support this protocol.
If your client supports MCP OAuth, it will open a browser window to verify with Google or GitHub on first connection.
If your client does not support MCP OAuth, you may need to use a stdio proxy that handles the OAuth flow and passes credentials to the remote server.
Check your client's documentation for details on connecting to OAuth-protected MCP servers.

## Markdown pages and llms.txt

Every documentation page is also available as Markdown, so AI tools that fetch a URL can read the docs without an MCP
connection or sign-in.

- Append `.md` to any page's URL to fetch its Markdown version. For example, `https://docs.temporal.io/workflows.md`.
- Each page has **Copy for LLM**, **View Markdown**, **Open in ChatGPT**, and **Open in Claude** buttons above its
  content. **Copy for LLM** copies the page's Markdown, with its source URL, to your clipboard. **Open in ChatGPT** and
  **Open in Claude** start a new chat with a prompt telling the model to read the page.
- [`llms.txt`](https://llmstxt.org) is a standard for indexing a site for LLMs. Temporal's index is at
  `https://docs.temporal.io/llms.txt`, and the full documentation concatenated into one file is at
  `https://docs.temporal.io/llms-full.txt`. The full file is large enough to exceed most model context windows, so
  prefer the index and individual pages' `.md` versions unless you need to bulk-ingest everything at once.

---

# Worker Versioning

Source: https://docs.temporal.io/worker-versioning

> Learn the concepts behind Worker Versioning and the details behind how it works.

This page defines some of the underlying concepts used in [Worker Versioning](/production-deployment/worker-deployments/worker-versioning):

- [Worker Deployments](#deployments)
- [Worker Deployment Versions](#deployment-versions)
- [Versioning behaviors](#versioning-behaviors)
- [Versioning Definitions](#versioning-definitions)
- [Versioning statuses](#versioning-statuses)
- [Continue-as-new, Child Workflow, and Retry Semantics](#inheritance-semantics)

## Worker Deployments 

A Worker Deployment is a logical service that groups similar Workers together for unified management.
Each Deployment has a name (such as your service name) and supports versioning through a series of Worker Deployment Versions.

## Worker Deployment Versions 

A Worker Deployment Version represents an iteration of a Worker Deployment.
Each Deployment Version is identified by a deployment name and a Build ID.
The deployment name groups related Workers across versions, and the Build ID identifies a specific release of your Worker code.
Each Deployment Version consists of Workers that share the same code build and environment.
When a Worker starts polling for Workflow and Activity Tasks, it reports its Deployment Version to the Temporal Server.

## Versioning behaviors 

You can declare each Workflow type to have a **Versioning Behavior**, either Pinned or Auto-Upgrade, in your Workflow configuration using an SDK or the CLI.

To learn more about implementing Worker Versioning, see our [Worker Versioning in production](production-deployment/worker-deployments/worker-versioning) page.

### Pinned Workflows 

A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version. You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. If you need to move a pinned Workflow to a new version, use [`temporal workflow update-options`](/cli/command-reference/workflow#update-options).

### Auto-Upgrade Workflows 

An **Auto-Upgrade** Workflow will move to the latest Worker Deployment Version automatically whenever you change the current version. Auto-upgrade Workflows are not restricted to a single Deployment Version and need to be kept replay-safe manually, that is with [patching](/workflow-definition#workflow-versioning).

### Activity behavior across versions

There are a few scenarios to consider for your Activities when you're handling your Worker Deployment versions.

- Activities generally start on the Worker Deployment Version of their Workflow which means:
  - For Pinned Workflows, an Activity starts on the pinned version.
  - For Auto-Upgrade Workflows, an Activity starts on the Target Worker Deployment Version of the Workflow. In this case, Workflow Execution moves to its Target Version immediately before starting the Activity if the Target Version is different from the last used Version. The Target Worker Deployment Version of a Workflow is the Current or Ramping Version of the Workflow's Task Queue, depending on the Ramp Percentage and Workflow ID.

There is an exception where you will have **Independent Activities**. Independent Activities are specific to Worker Versioning. They start on the Current or Ramping Version of their own Task Queue independently from their Workflow.

- For a Pinned Workflow, Independent Activities are Activities that start on a Task Queue that's not a member of the calling Workflow's Pinned Worker Deployment Version.
- For an Auto-Upgrade Workflow, Independent Activities are Activities that start on a Task Queue that's not a member of the calling Workflow's Target Worker Deployment Version.

Since Independent Activities aren't part of a Workflow's version, they can run in a few different ways:

- The Activity Task Queue is running in a separate Worker Deployment that only has the Independent Activity.
- The Independent Activity is in an unversioned Task Queue.
- The Independent Activity is in a separate Worker Deployment that has its own Workflows, but other Workflows reuse the Activity from other Worker Deployments.

## Versioning Definitions

- **Current Worker Deployment Version**: The version where Workflows are routed to unless they were previously pinned on a different version. Other versions can continue polling to allow pinned Workflows to finish executing or in case you need to roll back. If no current version is specified, the default is unversioned.
- **Ramping Worker Deployment Version**: The version where a configurable percentage of Workflows are routed to unless they were previously pinned on a different version. The ramp percentage can be in the range [0, 100]. Workflows that don't go to the Ramping Version will go to the Current Version. If no Ramping Version is specified, 100% of new Workflows and Auto-Upgrade Workflows will go to the Current Version.
- **Target Worker Deployment Version**: The version your Workflow will upgrade to next. This could be the Deployment's Current Version or the Ramping Version. For example, if an Auto-Upgrade Workflow was running on Version A, the Current Version is B, and there is a 5% ramp to C, there is a 95% chance that its Target Version is B and 5% that it's C. Workflow ID determines whether the workflow falls into the 95% group or the 5% group.

## Versioning statuses 

A Worker Deployment Version moves through the following states:

1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never becomes Active, it will never be Draining or Drained.
2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing auto-upgrade Workflows.
3. **Draining**: The version has open pinned Workflows running on it, but stopped being Current or Ramping, usually because a newer version has been deployed. It is possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated only periodically.
4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed. Closed Workflows may still re-run some code paths if they are [Queried](/sending-messages#sending-queries) within their [Retention Period](/temporal-service/temporal-server#retention-period) and Workers with that version are still polling.

## Continue-as-new, Child Workflow, and Retry Semantics 

When Workflows start new runs (for example, by continuing-as-new or retrying) the new run may inherit their versioning behavior. This section explains how inheritance works across different Workflow execution patterns.

### Ways Workflows Start New Runs

A Workflow can start a new run through:

- Starting a [Child Workflow](/child-workflows)
- Invoking [Continue-As-New](/workflow-execution/continue-as-new)
- Retrying per its [Retry Policy](/encyclopedia/retry-policies)
- Starting another iteration of a [Cron Job](/cron-job) (superseded by [Schedules](/schedule))

### Inheritance rules overview

Auto-upgrade Workflows never inherit versions.
By default, Pinned workflows will pass their version to any Pinned children.

This section provides more detail on specific inheritance scenarios.

### Inheritance by scenario

#### Child Workflows

**When Parent is Pinned:**

- Child inherits the parent's version if the child's Task Queue belongs to that version
- Child's first Workflow task executes in the same version as its parent
- If child is also Pinned: child remains Pinned to the inherited version for its lifetime
- If child is Auto-Upgrade: child's behavior changes to Auto-Upgrade after the first task completes
- If child's Task Queue is not in the same Worker Deployment as parent: no inheritance occurs, child starts on Current Version of its task queue

**When Parent is Auto-upgrade:**

- Child inherits no initial Versioning Behavior
- Child starts on the Current Version of its Worker Deployment like all new Workflow executions

#### Continue-As-New

**When Original Workflow is Pinned:**

- The Pinned version is inherited across the Continue-As-New chain
- If the new run's Task Queue is not in the same Worker Deployment as the original Workflow: no inheritance occurs, new run starts on Current Version of its task queue

**When Original Workflow is Auto-upgrade:**

- No version inheritance occurs

#### Retries

**Inheritance Conditions (all must be met):**

- The retried run is effectively pinned at the time of retry
- The retried run inherited a pinned version when it started (that is, it is a child of a pinned parent, or a Continue-As-New of a pinned run)
- The retried run is running on a Task Queue in the inherited version

**When Conditions Not Met:**

- No version inheritance occurs

#### Cron Jobs

- **Never inherit** versioning behavior or version

### Versioning Override inheritance

- Children, crons, retries, and continue-as-new inherit the source run's override **if**:
  - The override is pinned, **AND**
  - The new Workflow's Task Queue belongs to the override version
- Override inheritance is evaluated separately and takes precedence over inherited base version

---

# What is a Temporal Worker?

Source: https://docs.temporal.io/workers

> Temporal Workers are tightly coupled with Task Queues and Worker Processes.

This page discusses the following:

- [Worker](#worker)
- [Worker Program](#worker-program)
- [Worker Entity](#worker-entity)
- [Worker Identity](#worker-identity)
- [Worker Process](#worker-process)

## What is a Worker? 

In day-to-day conversations, the term Worker is used to denote either a [Worker Program](#worker-program), a [Worker Process](#worker-process), or a [Worker Entity](/workers#worker-entity).
Temporal documentation aims to be explicit and differentiate between them.

## What is a Worker Program? 

A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK.

> **ℹ️ Info:**
>
> - [How to run a development Worker using the .NET SDK](/develop/dotnet/workers/run-worker-process)
> - [How to run a development Worker using the Go SDK](/develop/go/workers/run-worker-process#develop-worker)
> - [How to run a development Worker using the Java SDK](/develop/java/workers/run-worker-process)
> - [How to run a development Worker using the PHP SDK](/develop/php/workers/run-worker-process#run-a-dev-worker)
> - [How to run a development Worker using the Python SDK](/develop/python/workers/run-worker-process#run-a-dev-worker)
> - [How to run a development Worker using the TypeScript SDK](/develop/typescript/workers/run-worker-process#run-a-dev-worker)
> - [How to run a development Worker using the Ruby SDK](/develop/ruby/workers/run-worker-process)
> - [How to run a development Worker using the Rust SDK](/develop/rust/workers/worker-process)
>
> - [How to connect a Go SDK Worker to Temporal Cloud](/develop/go/workers/run-worker-process#connect-to-temporal-cloud)
> - [How to run a Temporal Cloud Worker using the TypeScript SDK](/develop/typescript/workers/run-worker-process#run-a-temporal-cloud-worker)
>

## What is a Worker Entity? 

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

A Worker Entity listens and polls on a single Task Queue.
A Worker Entity contains a Workflow Worker and/or an Activity Worker, which makes progress on Workflow Executions and Activity Executions, respectively.

**Can a Worker handle more Workflow Executions than its cache size or number of supported threads?**

Yes it can.
However, the trade off is added latency.

Workers are stateless, so any Workflow Execution in a blocked state can be safely removed from a Worker.
Later on, it can be resurrected on the same or different Worker when the need arises (in the form of an external event).
Therefore, a single Worker can handle millions of open Workflow Executions, assuming it can handle the update rate and that a slightly higher latency is not a concern.

**Operation guides:**

- [How to tune Workers](/develop/worker-performance)
- [Worker tuning quick reference](/develop/worker-tuning-reference) - SDK defaults and metrics

## What is a Worker Identity? 

Workers have an associated identifier that helps identify the specific Worker instance.
By default, Temporal SDKs set a Worker Identity to `${process.pid}@${os.hostname()}`, which combines the Worker's process ID (`process.pid`) and the hostname of the machine running the Worker (`os.hostname()`).

The Worker Identity is visible in various contexts, such as Event History and the list of pollers on a Task Queue.

You can use the Worker Identity to aid in debugging operational issues.
By providing a user assigned identifier, you can trace issues back to specific Worker instances.

**What are some limitations of the default identity?**

While the default identity format may seem sensible, it often proves to be of limited usefulness in cloud environments.
Some common issues include:

- **Docker containers**: When running Workers inside Docker containers, the process ID is always `1`, as each container typically runs a single process. This makes the process identifier meaningless for identification purposes.
- **Random hostnames**: In some cloud environments, such as Amazon ECS (Elastic Container Service), the hostname is a randomly generated string that does not provide any meaningful information about the Worker's execution context.
- **Ephemeral IP addresses**: In certain cases, the hostname might be set to an ephemeral IP address, which can change over time and does not uniquely identify a Worker instance.

**What are some recommended approaches?**

It is recommended that you ensure that the Worker Identity can be linked back to the corresponding machine, process, execution context, or log stream.
In some execution environments, this might require that you explicitly specify the Worker Identity.

Here are some approaches:

- **Use environment-specific identifiers**: Choose an identifier that is specific to your execution environment. For example, when running Workers on Amazon ECS, you can set the Worker Identity to the ECS Task ID, which uniquely identifies the task running the Worker.
- **Include relevant context**: Incorporate information that helps establish the context of the Worker, such as the deployment environment (`staging` or `production`), region, or any other relevant details.
- **Ensure uniqueness**: Make sure that the Worker Identity is unique within your system to avoid ambiguity when debugging issues.
- **Keep it concise**: While including relevant information is important, try to keep the Worker Identity concise and readable so you can identify and troubleshoot a Worker quickly.

## What is a Worker Process? 

![Component diagram of a Worker Process and the Temporal Server](https://docs.temporal.io/diagrams/worker-and-server-component.svg)

A Worker Process is responsible for polling a [Task Queue](/task-queue), dequeueing a [Task](/tasks#task), executing your code in response to a Task, and responding to the [Temporal Service](/temporal-service) with the results.

More formally, a Worker Process is any process that implements the Task Queue Protocol and the Task Execution Protocol.

- A Worker Process is a Workflow Worker Process if the process implements the Workflow Task Queue Protocol and executes the Workflow Task Execution Protocol to make progress on a Workflow Execution.
  A Workflow Worker Process can listen on an arbitrary number of Workflow Task Queues and can execute an arbitrary number of Workflow Tasks.
- A Worker Process is an Activity Worker Process if the process implements the Activity Task Queue Protocol and executes the Activity Task Processing Protocol to make progress on an Activity Execution.
  An Activity Worker Process can listen on an arbitrary number of Activity Task Queues and can execute an arbitrary number of Activity Tasks.

**Worker Processes are external to a Temporal Service.**
Temporal Application developers are responsible for developing [Worker Programs](#worker-program) and operating Worker Processes.
Said another way, the [Temporal Service](/temporal-service) (including the Temporal Cloud) doesn't execute any of your code (Workflow and Activity Definitions) on Temporal Service machines. The Temporal Service is solely responsible for orchestrating [State Transitions](/workflow-execution#state-transition) and providing Tasks to the next available [Worker Entity](/workers#worker-entity).

While data transferred in Event Histories is [secured by mTLS](/self-hosted-guide/security#encryption-in-transit-with-mtls), by default, it is still readable at rest in the Temporal Service.

To solve this, Temporal SDKs offer a [Data Converter API](/dataconversion) that you can use to customize the serialization of data going out of and coming back in to a Worker Entity, with the net effect of guaranteeing that the Temporal Service cannot read sensitive business data.

In many of our tutorials, we show you how to run both a Temporal Service and one Worker on the same machine for local development.
However, a production-grade Temporal Application typically has a _fleet_ of Worker Processes, all running on hosts external to the Temporal Service.
A Temporal Application can have as many Worker Processes as needed.

A Worker Process can be both a Workflow Worker Process and an Activity Worker Process.
Many SDKs support the ability to have multiple Worker Entities in a single Worker Process.
(Worker Entity creation and management differ between SDKs.)
A single Worker Entity can listen to only a single Task Queue.
But if a Worker Process has multiple Worker Entities, the Worker Process could be listening to multiple Task Queues.

![Entity relationship diagram (meta model) of Worker Processes, Task Queues, and Tasks](https://docs.temporal.io/diagrams/worker-and-server-entity-relationship.svg)

Worker Processes executing Activity Tasks must have access to any resources needed to execute the actions that are defined in Activity Definitions, such as the following:

- Network access for external API calls.
- Credentials for infrastructure provisioning.
- Specialized GPUs for machine learning utilities.

The Temporal Service itself has [internal workers](https://temporal.io/blog/workflow-engine-principles/#system-workflows-1910) for system Workflow Executions.
However, these internal workers are not visible to the developer.

---

# Workflow Activity

Source: https://docs.temporal.io/workflow-activity

> Learn about Activities started as a step in a Workflow, scoped to that Workflow's lifetime and recorded in Workflow Event History.

## What is a Workflow Activity? 

A Workflow Activity is an [Activity Execution](/activity-execution) started as a step in a
[Workflow](/workflows), using the Temporal SDK from your Workflow code.

The Workflow schedules the Activity, the Temporal Service dispatches an Activity Task to an Activity
Worker polling the Activity's Task Queue, and the result is delivered back to the Workflow when the
Activity Execution closes. Each step is recorded in the Workflow Execution's
[Event History](/workflow-execution/event#event-history).

Use a Workflow Activity when you need to orchestrate multiple steps: sequencing, branching on a
result, compensating a failure, waiting on a Signal or Timer between steps.
To execute a single Activity, use a [Standalone Activity](/standalone-activity) instead.

> **💡 Tip:**
> GET STARTED
>
> Start a Workflow Activity:
> [Go](/develop/go/activities/execution)
> | [Java](/develop/java/activities/execution)
> | [PHP](/develop/php/activities/execution)
> | [Python](/develop/python/activities/execution)
> | [TypeScript](/develop/typescript/activities/execution)
> | [.NET](/develop/dotnet/activities/execution)
> | [Ruby](/develop/ruby/activities/execution)
> | [Rust](/develop/rust/activities/execution)
>

## Activity options 

You set Activity Options in your Workflow code when you schedule the Activity. At minimum, specify a
timeout - typically the [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout).

- [Timeouts](/encyclopedia/detecting-activity-failures): Schedule-To-Start, Start-To-Close,
  Schedule-To-Close, and Heartbeat.
- [Retry Policy](/encyclopedia/retry-policies): Activities retry automatically by default, with
  exponential backoff and unlimited Maximum Attempts.
- Task Queue: routes the Activity Task to the Workers that should run it. It doesn't have to be the
  Workflow's Task Queue.

> **💡 Tip:**
> GET STARTED
>
> Set Activity timeouts and Heartbeats:
> [Go](/develop/go/activities/timeouts)
> | [Java](/develop/java/activities/timeouts)
> | [PHP](/develop/php/activities/timeouts)
> | [Python](/develop/python/activities/timeouts)
> | [TypeScript](/develop/typescript/activities/timeouts)
> | [.NET](/develop/dotnet/activities/timeouts)
> | [Ruby](/develop/ruby/activities/timeouts)
> | [Rust](/develop/rust/activities/timeouts)
>

## Event History 

A Workflow Activity writes [Activity Events](/workflow-execution/event#activity-events) to the
Workflow Execution's Event History, such as
[ActivityTaskScheduled](/references/events#activitytaskscheduled),
[ActivityTaskStarted](/references/events#activitytaskstarted), and
[ActivityTaskCompleted](/references/events#activitytaskcompleted).

Two consequences follow:

- **Replay uses the recorded result.** A completed Activity isn't executed again when the Workflow
  [replays](/workflow-execution#replay).
- **Arguments and return values are persisted.** They count toward
  [Event History limits](/workflow-execution/event#event-history-limits), so be mindful of payload
  size and of how many Activities a single Workflow Execution schedules.

## Activity ID 

A Workflow Activity's [Activity ID](/activity-execution#activity-id) is unique among the open
Activity Executions of a [Workflow Run](/workflow-execution/workflowid-runid#run-id). A Workflow Run
may reuse an Activity ID once an earlier Activity Execution with that ID has closed.

## Cancellation 

A Workflow Activity can't be canceled directly. It receives a
[Cancellation](/activity-execution#cancellation) request as a result of its Workflow being canceled,
or because the Workflow code requested it. Activities must
[Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) to receive Cancellation.

The Workflow decides whether to wait for the Cancellation to be accepted or to proceed without
waiting.

## Asynchronous completion 

A Workflow Activity can return from its function without completing the Activity Execution, leaving
an external system to Heartbeat progress and deliver the final result. See
[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion).

## Operator commands 

Pause, Unpause, Reset and Update Options let an operator intervene in a running Workflow Activity.
These commands are in [Public Preview](/evaluate/product-release-stages#public-preview).

See [Activity Operations](/activity-operations) for details.

---

# Temporal Workflow Definition

Source: https://docs.temporal.io/workflow-definition

> A Workflow Definition in Temporal is the code that defines the constraints of a Workflow Execution.

This page covers the following:

- [What is a Workflow Definition?](/workflow-definition)
- [Determinism and constraints](#deterministic-constraints)
- [Handling code changes and non-deterministic behavior](#non-deterministic-change)
- [Intrinsic non-determinism logic](#intrinsic-nondeterministic-logic)
- [Versioning Workflow code and Patching](#workflow-versioning)
- [Handling unreliable Worker Processes](#unreliable-worker-processes)
- [What is a Workflow Type?](#workflow-type)

A Temporal Workflow defines the overall flow of the application.
Conceptually, a Workflow is a sequence of steps written in a general-purpose programming language.
With Temporal, those steps are defined by writing code, known as a Workflow Definition, and are carried out by running that code, which results in a Workflow Execution.

In day-to-day conversations, the term _Workflow_ might refer to [Workflow Type](#workflow-type), a [Workflow Definition](/workflow-definition), or a [Workflow Execution](/workflow-execution).
Temporal documentation aims to be explicit and differentiate between them.

## What is a Workflow Definition? 

A Workflow Definition is the code that defines the Workflow.
It is written with a programming language and corresponding Temporal SDK.
Depending on the programming language, it's typically implemented as a function or an object method and encompasses the end-to-end series of steps of a Temporal application.

Below are different ways to develop a basic Workflow Definition.

**Go**

**[Workflow Definition in Go](/develop/go/workflows/basics)**

```go
func YourBasicWorkflow(ctx workflow.Context) error {
    // ...
    return nil
}
```

**Java**

**[Workflow Definition in Java (Interface)](/develop/java/workflows/basics)**

```java
// Workflow interface
@WorkflowInterface
public interface YourBasicWorkflow {

    @WorkflowMethod
    String workflowMethod(Arguments args);
}
```

**[Workflow Definition in Java (Implementation)](/develop/java/workflows/basics)**

```java
// Workflow implementation
public class YourBasicWorkflowImpl implements YourBasicWorkflow {
    // ...
}
```

**PHP**

**[Workflow Definition in PHP (Interface)](/develop/php/workflows/basics)**

```php
#[WorkflowInterface]
interface YourBasicWorkflow {
    #[WorkflowMethod]
    public function workflowMethod(Arguments args);
}
```

**[Workflow Definition in PHP (Implementation)](/develop/php/workflows/basics)**

```php
class YourBasicWorkflowImpl implements YourBasicWorkflow {
    // ...
}
```

**Python**

**[Workflow Definition in Python](/develop/python/workflows/basics)**

```Python
@workflow.defn
class YourWorkflow:
    @workflow.run
    async def YourBasicWorkflow(self, input: str) -> str:
        # ...
```

**TypeScript**

**[Workflow Definition in TypeScript](/develop/typescript/workflows/basics)**

```typescript
type BasicWorkflowArgs = {
  param: string;
};

export async function WorkflowExample(
  args: BasicWorkflowArgs,
): Promise<{ result: string }> {
  // ...
}
```

**.NET**

**[Workflow Definition in C# and .NET](/develop/dotnet/workflows/basics)**

```csharp
[Workflow]
public class YourBasicWorkflow {

    [WorkflowRun]
    public async Task<string> workflowExample(string param) {
        // ...
    }
}
```

**Rust**

**[Workflow Definition in Rust](/develop/rust/workflows/basics)**

```rust
use temporalio_macros::{workflow, workflow_methods};
use temporalio_sdk::{WorkflowContext, WorkflowContextView, WorkflowResult};

#[workflow]
pub struct YourBasicWorkflow {
    param: String,
}

#[workflow_methods]
impl YourBasicWorkflow {
    #[init]
    fn new(_ctx: &WorkflowContextView, param: String) -> Self {
        Self { param }
    }

    #[run]
    async fn run(ctx: &mut WorkflowContext<Self>) -> WorkflowResult<String> {
        // ...
        Ok(String::new())
    }
}
```

A Workflow Definition may be also referred to as a Workflow Function.
In Temporal's documentation, a Workflow Definition refers to the source for the instance of a Workflow Execution, while a Workflow Function refers to the source for the instance of a Workflow Function Execution.

A Workflow Execution effectively executes once to completion, while a Workflow Function Execution occurs many times during the life of a Workflow Execution.

We strongly recommend that you write a Workflow Definition in a language that has a corresponding Temporal SDK.

### Workflow parameters 

A Workflow Definition can accept more than one parameter, but give it a single parameter instead: an object, such as a `dataclass` in Python, a `struct` in Go, or a class in Java and .NET.
Put all input fields on that object.

- **You can add fields without breaking existing Workflows.**
  Adding an optional field to the object doesn't change the Workflow Definition's signature, so Workflow Executions that are already running keep working.
  Adding, removing, or retyping a separate parameter changes the signature.
- **Type-safe Client calls need a single parameter.**
  In several SDKs, the type-safe forms of starting a Workflow only accept a single argument.
  With multiple parameters, you fall back to starting the Workflow by Workflow Type name without type checking.
- **The Web UI sends one argument.**
  You can't start a multi-parameter Workflow from the Web UI. See [Pass input from the Web UI](/web-ui#input).

### Deterministic constraints 

A critical aspect of developing Workflow Definitions is ensuring that they are deterministic.
Generally speaking, this means you must take care to ensure that any time your Workflow code is executed it makes the same Workflow API calls in the same sequence, given the same input.
Some changes to those API calls are safe to make.

> **💡 Tip:**
> Note on determinism
>
> A Workflow Definition must be deterministic to support replay. To handle non-deterministic operations like API calls, LLM/AI invocations, database queries, and other external interactions, put them in Activities. Activities execute outside the replay path and are automatically retried so they don't cause non-determinism errors.
>

For example, you can change:

- The input parameters, return values, and execution timeouts of Child Workflows and Activities
  - However, it is not safe to change the types or IDs of Child Workflows or Activities
- The input parameters used to Signal an external Workflow
- The duration of Timers (although changing them to 0 is not safe in all SDKs)
- Add or remove calls to Workflow APIs that don't produce [Commands](/workflow-execution#command) (For example - `workflow.GetInfo` in the Go SDK or its equivalent in other SDKs)

The following Workflow API calls all can produce Commands, and thus must not be reordered, added, or removed without proper [Versioning techniques](#workflow-versioning):

- Starting or cancelling a Timer
- Scheduling or cancelling Activity Executions (including local Activities)
- Starting or cancelling Child Workflow executions
- Signalling or cancelling signals to external Workflow Executions
- Scheduling or cancelling Nexus operations
- Ending the Workflow Execution in any way (completing, failing, cancelling, or continuing-as-new)
- `Patched` or `GetVersion` calls for Versioning (although they may be added or removed according to the [patching](#workflow-patching) rules)
- Upserting Workflow Search Attributes
- Upserting Workflow Memos
- Running a `SideEffect` or `MutableSideEffect`

For a complete reference, see the [Command reference](/references/commands).

More formally, the use of certain Workflow APIs in the function is what generates Commands.
Commands tell the Temporal Service which Events to create and add to the Workflow Execution's [Event History](/workflow-execution/event#event-history).
When the Workflow's code [replays](/workflow-execution#replay), the Commands that are emitted are compared with the existing Event History.
If a corresponding Event already exists within the Event History that matches that command, then the Execution progresses.
See [Event History](/encyclopedia/event-history) for a detailed walkthrough of the process.

For example, using an SDK's "Execute Activity" API generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command.
When this API is called upon re-execution, that Command is compared with the Event that is in the same location within the sequence.
The Event in the sequence must be an [ActivityTaskScheduled](/references/events#activitytaskscheduled) Event, where the Activity name is the same as what is in the Command.

If a generated Command doesn't match what it needs to in the existing Event History, then the Workflow Execution returns a _non-deterministic_ error.

The following are the two reasons why a Command might be generated out of sequence or the wrong Command might be generated altogether:

1. Code changes are made to a Workflow Definition that is in use by a running Workflow Execution.
2. There is intrinsic non-deterministic logic (such as inline random branching).

### Code changes can cause non-deterministic behavior 

The Workflow Definition can change in very limited ways once there is a Workflow Execution depending on it.
To alleviate non-deterministic issues that arise from code changes, we recommend using [Workflow Versioning](#workflow-versioning).

For example, let's say we have a Workflow Definition that defines the following sequence:

1. Start and wait on a Timer/sleep.
2. Spawn and wait on an Activity Execution.
3. Complete.

We start a Worker and spawn a Workflow Execution that uses that Workflow Definition.
The Worker would emit the [StartTimer](/references/commands#starttimer) Command and the Workflow Execution would become suspended.

Before the Timer is up, we change the Workflow Definition to the following sequence:

1. Spawn and wait on an Activity Execution.
2. Start and wait on a Timer/sleep.
3. Complete.

When the Timer fires, the next Workflow Task will cause the Workflow Function to re-execute.
The first Command the Worker sees would be ScheduleActivityTask Command, which wouldn't match up to the expected [TimerStarted](/references/events#timerstarted) Event.

The Workflow Execution would fail and return a nondeterminism error.

The following are examples of minor changes that would not result in non-determinism errors when re-executing a History which already contain the Events:

- Changing the duration of a Timer, with the following exceptions:
  - In Java, Python, and Go, changing a Timer's duration from or to 0 is a non-deterministic behavior.
  - In .NET, changing a Timer's duration from or to -1 (which means "infinite") is a non-deterministic behavior.
- Changing the arguments to:
  - The Activity Options in a call to spawn an Activity Execution (local or nonlocal).
  - The Child Workflow Options in a call to spawn a Child Workflow Execution.
  - Call to Signal an External Workflow Execution.
- Adding a Signal Handler for a Signal Type that has not been sent to this Workflow Execution.

### Intrinsic non-deterministic logic 

Intrinsic non-determinism is when a Workflow Function Execution might emit a different sequence of Commands on re-execution, regardless of whether all the input parameters are the same.

For example, a Workflow Definition can not have inline logic that branches (emits a different Command sequence) based off a local time setting or a random number.
In the representative pseudocode below, the `local_clock()` function returns the local time, rather than Temporal-defined time:

```text
fn your_workflow() {
  if local_clock().is_before("12pm") {
    await workflow.sleep(duration_until("12pm"))
  } else {
    await your_afternoon_activity()
  }
}
```

Each Temporal SDK offers APIs that enable Workflow Definitions to have logic that gets and uses time, random numbers, and data from unreliable resources.
When those APIs are used, the results are stored as part of the Event History, which means that a re-executed Workflow Function will issue the same sequence of Commands, even if there is branching involved.

In other words, all operations that do not purely mutate the Workflow Execution's state should occur through a Temporal SDK API.

For SDK-specific replay-safe APIs and examples (logging, random numbers, time, replay detection), see:

- [Go: Develop Workflow logic](/develop/go/workflows/basics#workflow-logic-requirements)
- [Java: Workflow logic requirements](/develop/java/workflows/basics#workflow-logic-requirements)
- [Python: Develop Workflow logic](/develop/python/workflows/basics#workflow-logic-requirements)
- [TypeScript: Develop Workflow logic](/develop/typescript/workflows/basics#workflow-logic-requirements)
- [.NET: Workflow logic requirements](/develop/dotnet/workflows/basics#workflow-logic-requirements)
- [Ruby: Workflow Logic Requirements](/develop/ruby/workflows/basics#workflow-logic-requirements)
- [Rust: Workflow Logic Requirements](/develop/rust/workflows/basics#workflow-logic-requirements)

### Versioning Workflows 

The Temporal Platform requires that Workflow Definitions be deterministic.
This requirement means that developers should consider how they plan to handle changes to Workflow code over time.

A versioning strategy is even more important if your Workflow Executions live long enough to run on multiple versions of your Worker. Temporal Platform provides Workflow Versioning APIs.

Temporal offers two Versioning strategies:

- [Worker Versioning](#worker-versioning): keep Workers tied to specific code revisions, so that old Workers can run old code paths and new Workers can run new code paths.

- [Versioning with patching](#workflow-patching): make sure your code changes are compatible across versions of your Workflow.

You can use either strategy, or a combination.

> **📝 Note:**
> Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance.

#### Worker Versioning 

This is the **recommended** way to handle versioning and users see improved error rates when adopting it. To learn more about Worker Versioning, see our [Worker Versioning in production](production-deployment/worker-deployments/worker-versioning) page.

#### Versioning with Patching 

When keeping Workflows compatible, you should patch and ideally how to test your running Workflows will be safe to run on a new code version.

To patch:

- [How to patch Workflow code in .NET](/develop/dotnet/workflows/versioning#patching)
- [How to patch Workflow code in Go](/develop/go/workflows/versioning#patching)
- [How to patch Workflow code in Java](/develop/java/workflows/versioning#patching)
- [How to patch Workflow code in PHP](/develop/php/workflows/versioning#php-sdk-patching-api)
- [How to patch Workflow code in Python](/develop/python/workflows/versioning#patching)
- [How to patch Workflow code in Ruby](/develop/ruby/workflows/versioning#patching)
- [How to patch Workflow code in TypeScript](/develop/typescript/workflows/versioning#patching)

To test, see [Safe Deployments](/develop/safe-deployments.mdx).

### Handling unreliable Worker Processes 

You do not handle Worker Process failure or restarts in a Workflow Definition.

Workflow Function Executions are completely oblivious to the Worker Process in terms of failures or downtime.
The Temporal Platform ensures that the state of a Workflow Execution is recovered and progress resumes if there is an outage of either Worker Processes or the Temporal Service itself.
The only reason a Workflow Execution might fail is due to the code throwing an error or exception, not because of underlying infrastructure outages.

### What is a Workflow Type? 

A Workflow Type is a name that maps to a Workflow Definition.

- A single Workflow Type can be instantiated as multiple Workflow Executions.
- A Workflow Type is scoped by a Task Queue.
  It is acceptable to have the same Workflow Type name map to different Workflow Definitions if they are using completely different Workers.

![Workflow Type cardinality with Workflow Definitions and Workflow Executions](https://docs.temporal.io/diagrams/workflow-type-cardinality.svg)

---

# Temporal Workflow Execution overview

Source: https://docs.temporal.io/workflow-execution

> A Temporal Workflow Execution is a durable, reliable, and scalable function execution and the main unit of execution of a Temporal Application.

This page provides an overview of Workflow Execution:

- [What is a Workflow Execution?](#workflow-execution)
- [Replay](#replay)
- [Commands and awaitables](#commands-awaitables)
- [What is a Command?](#command)
- [Checking Workflow Execution Status](#workflow-execution-status)
- [Workflow Execution Chain](#workflow-execution-chain)
- [Memo](#memo)
- [State Transition](#state-transition)
- [Workflow cache](#workflow-cache)

## What is a Workflow Execution? 

While the Workflow Definition is the code that defines the Workflow, the Workflow Execution is created by executing that code.
A Temporal Workflow Execution is a durable, reliable, and scalable function execution.
It is the main unit of execution of a [Temporal Application](/temporal#temporal-application).

- [How to start a Workflow Execution using temporal](/cli/command-reference/workflow#start)
- [How to start a Workflow Execution using the .NET SDK](/develop/dotnet/client/temporal-client#start-workflow)
- [How to start a Workflow Execution using the Go SDK](/develop/go/client/temporal-client#start-workflow-execution)
- [How to start a Workflow Execution using the Java SDK](/develop/java/client/temporal-client#start-workflow-execution)
- [How to start a Workflow Execution using the PHP SDK](/develop/php/client/temporal-client#start-workflow-execution)
- [How to start a Workflow Execution using the Python SDK](/develop/python/client/temporal-client#start-workflow-execution)
- [How to start a Workflow Execution using the Ruby SDK](/develop/ruby/client/temporal-client#start-workflow)
- [How to start a Workflow Execution using the Rust SDK](/develop/rust/client/temporal-client#start-workflow-execution)
- [How to start a Workflow Execution using the TypeScript SDK](/develop/typescript/client/temporal-client#start-workflow-execution)

Each Temporal Workflow Execution has exclusive access to its local state.
It executes concurrently to all other Workflow Executions, and communicates with other Workflow Executions through [Signals](/sending-messages#sending-signals) and the environment through [Activities](/activities).
While a single Workflow Execution has limits on size and throughput, a Temporal Application can consist of millions to billions of Workflow Executions.

**Durability**

Durability is the absence of an imposed time limit.

A Workflow Execution is durable because it executes a Temporal Workflow Definition (also called a Temporal Workflow Function), your application code, effectively once and to completion—whether your code executes for seconds or years.

**Reliability**

Reliability is responsiveness in the presence of failure.

A Workflow Execution is reliable, because it is fully recoverable after a failure.
The Temporal Platform ensures the state of the Workflow Execution persists in the face of failures and outages and resumes execution from the latest state.

**Scalability**

Scalability is responsiveness in the presence of load.

A single Workflow Execution is limited in size and throughput but is scalable because it can [Continue-As-New](/workflow-execution/continue-as-new) in response to load.
A Temporal Application is scalable because the Temporal Platform is capable of supporting millions to billions of Workflow Executions executing concurrently, which is realized by the design and nature of the [Temporal Service](/temporal-service) and [Worker Processes](/workers#worker-process).

### Replays 

A Replay is the method by which a Workflow Execution resumes making progress. During a Replay the Commands that are generated are checked against an existing Event History. Replays are necessary and often happen to give the effect that Workflow Executions are resumable, reliable, and durable.

For more information, see [Deterministic constraints](/workflow-definition#deterministic-constraints).

If a failure occurs, the Workflow Execution picks up where the last recorded event occurred in the Event History.

- [How to use Replay APIs using the Go SDK](/develop/go/best-practices/testing-suite#replay)
- [How to use Replay APIs using the Java SDK](/develop/java/best-practices/testing-suite#replay)
- [How to use Replay APIs using the Python SDK](/develop/python/best-practices/testing-suite#replay)
- [How to use Replay APIs using the TypeScript SDK](/develop/typescript/best-practices/testing-suite#replay)
- [How to use Replay APIs using the .NET SDK](/develop/dotnet/best-practices/testing-suite#replay)

### Commands and awaitables 

A Workflow Execution does two things:

1. Issue [Commands](#command).
2. Wait on an Awaitables (often called Futures).

![Command generation and waiting](https://docs.temporal.io/diagrams/workflow-execution-progession-simple.svg)

Commands are issued and Awaitables are provided by the use of Workflow APIs in the [Workflow Definition](/workflow-definition).

Commands are generated whenever the Workflow Function is executed.
The Worker Process supervises the Command generation and makes sure that it maps to the current Event History.
(For more information, see [Deterministic constraints](/workflow-definition#deterministic-constraints).)
The Worker Process batches the Commands and then suspends progress to send the Commands to the Temporal Service whenever the Workflow Function reaches a place where it can no longer progress without a result from an Awaitable.

A Workflow Execution may only ever block progress on an Awaitable that is provided through a Temporal SDK API.
Awaitables are provided when using APIs for the following:

- Awaiting: Progress can block using explicit "Await" APIs.
- Requesting cancellation of another Workflow Execution: Progress can block on confirmation that the other Workflow Execution is cancelled.
- Sending a [Signal](/sending-messages#sending-signals): Progress can block on confirmation that the Signal sent.
- Spawning a [Child Workflow Execution](/child-workflows): Progress can block on confirmation that the Child Workflow Execution started, and on the result of the Child Workflow Execution.
- Spawning an [Activity Execution](/activity-execution): Progress can block on the result of the Activity Execution.
- Starting a Timer: Progress can block until the Timer fires.

### What is a Command? 

A Command is a requested action issued by a [Worker](/workers#worker) to the [Temporal Service](/temporal-service) after a [Workflow Task Execution](/tasks#workflow-task-execution) completes.

The action that the Temporal Service takes is recorded in the [Workflow Execution's](#workflow-execution) [Event History](/workflow-execution/event#event-history) as an [Event](/workflow-execution/event).
The Workflow Execution can await on some of the Events that come as a result from some of the Commands.

Commands are generated by the use of Workflow APIs in your code. During a Workflow Task Execution there may be several Commands that are generated.
The Commands are batched and sent to the Temporal Service as part of the Workflow Task Execution completion request, after the Workflow Task has progressed as far as it can with the Workflow function.
There will always be [WorkflowTaskStarted](/references/events#workflowtaskstarted) and [WorkflowTaskCompleted](/references/events#workflowtaskcompleted) Events in the Event History when there is a Workflow Task Execution completion request.

![Commands are generated by the use of Workflow APIs in your code](https://docs.temporal.io/diagrams/commands.svg)

Commands are described in the [Command reference](/references/commands) and are defined in the [Temporal gRPC API](https://github.com/temporalio/api/blob/main/temporal/api/command/v1/message.proto).

### Status 

A Workflow Execution can be either _Open_ or _Closed_.

![Workflow Execution statuses](https://docs.temporal.io/diagrams/workflow-execution-statuses.svg)

#### Open

An _Open_ status means that the Workflow Execution is able to make progress.

- Running: The Workflow Execution is actively progressing or is waiting on something.
- Paused: The Workflow Execution has been [Paused](/encyclopedia/workflow/workflow-pause). It remains Open, but the Temporal Service doesn't
  dispatch new Workflow Tasks until the Workflow Execution is Unpaused.

#### Closed

A _Closed_ status means that the Workflow Execution cannot make further progress because of one of the following reasons:

- Cancelled: The Workflow Execution successfully handled a cancellation request.
- Completed: The Workflow Execution has completed successfully.
- Continued-As-New: The Workflow Execution [Continued-As-New](/workflow-execution/continue-as-new).
- Failed: The Workflow Execution returned an error and failed.
- Terminated: The Workflow Execution was terminated.
- Timed Out: The Workflow Execution reached a timeout limit.

### Workflow Execution Chain 

A Workflow Execution Chain is a sequence of Workflow Executions that share the same Workflow Id.
Each link in the Chain is often called a Workflow Run.
Each Workflow Run in the sequence is connected by one of the following:

- [Continue-As-New](/workflow-execution/continue-as-new)
- [Retries](/encyclopedia/retry-policies)
- [Temporal Cron Job](/cron-job)
- [Reset](/cli/command-reference/workflow#reset)

A Workflow Execution is uniquely identified by its [Namespace](/namespaces), [Workflow Id](/workflow-execution/workflowid-runid#workflow-id), and [Run Id](/workflow-execution/workflowid-runid#run-id).

The [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) applies to a Workflow Execution Chain.
The [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout) applies to a single Workflow Execution (Workflow Run).

## What is a Memo? 

A Memo is a non-indexed set of Workflow Execution metadata that developers supply at start time or in Workflow code and that is returned when you describe or list Workflow Executions.

The primary purpose of using a Memo is to enhance the organization and management of Workflow Executions.
Add your own metadata, such as notes or descriptions, to a Workflow Execution, which lets you annotate and categorize Workflow Executions based on developer-defined criteria.
This feature is particularly useful when dealing with numerous Workflow Executions because it facilitates the addition of context, reminders, or any other relevant information that aids in understanding or tracking the Workflow Execution.

> **📝 Note:**
> Use Memos judiciously
>
> Memos shouldn't store data that's critical to the execution of a Workflow, for some of the following reasons:
>
> - Unlike Workflow inputs, Memos lack type safety
> - Memos are subject to eventual consistency and may not be immediately available
> - Excessive reliance on Memos hides mutable state from the Workflow Execution History
>

On Temporal Cloud, a Memo is limited to 40 KB.

## What is a State Transition? 

A State Transition is a unit of progress made by a [Workflow Execution](#workflow-execution).
Each State Transition is recorded in a persistence store.

Some operations, such as [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat), require only one or two State Transitions each. With an Activity Heartbeat, there are two: the Activity Heartbeat and a Timer.

Most operations require multiple State Transitions.

For example, a simple Workflow with two sequential [Activity Tasks](/tasks#activity-task) (and no retries) produces 11 State Transitions: two for Workflow start, four for each Activity, and one for Workflow completion.

> **💡 Tip:**
> NEXT STEPS
> For more information on Workflow Execution, please refer to the following subpages:
>
> - [Event](/workflow-execution/event)
> - [Workflow Id and Run Id](/workflow-execution/workflowid-runid)
> - [Limits](/workflow-execution/limits)
> - [Continue-as-New](/workflow-execution/continue-as-new)
> - [Timers and Start Delay](/workflow-execution/timers-delays)

## Workflow cache

A Workflow Cache is an in-memory LRU (least recently used) cache maintained by Workers that stores the state of Workflow Executions they have processed.

When a Worker picks up a Workflow Task, it caches the Workflow's state in memory. This allows the Worker to continue processing subsequent Tasks for that Workflow without having to fetch the full Event History from the server and replay it from scratch. 

This caching mechanism is closely tied to [Sticky Execution](/sticky-execution). The Temporal Service directs future Workflow Tasks to the same Worker that cached the Workflow, via a dedicated "Sticky Queue". If the cached Workflow is evicted, to make room for another for example, the Worker must replay the Event History to restore its state before continuing.

---

# Continue-As-New

Source: https://docs.temporal.io/workflow-execution/continue-as-new

> Continue-As-New is a mechanism by which the latest relevant state is passed to a new Workflow Execution, with a fresh Event History.

This page discusses [Continue-As-New](#continue-as-new) and how to decide [when to use it](#when).

## What is Continue-As-New? 

Continue-As-New allows you to checkpoint your Workflow's state and start a fresh Workflow.

There are two main reasons you might want to start a new Workflow:

- A Workflow Execution with a long, or large [Event History](/workflow-execution/event#event-history), such as one calling many Activities, may bog down and have performance issues.
  It could even generate more Events than allowed by the [Event History limits](/workflow-execution/event#event-history-limits).
- A Workflow Execution can hit [Workflow Versioning](/workflow-definition#workflow-versioning) problems if it started running on an older version of your code and then begins executing on a newer version.

Your goal is to create a new Workflow with a fresh history that picks up where your last one left off.
First, pass your latest relevant state into Continue-As-New.
This hands it to a new Execution in the [Execution Chain](/workflow-execution#workflow-execution-chain).
This state is passed in as arguments to your Workflow.
The parameters are typically optional and left unset by the original caller of the Workflow.

The new Workflow Execution has the same Workflow Id, but a different Run Id, and starts its own Event History.

You can repeat Continue-As-New as often as needed, which means that your Workflow can run forever.
Workflows that do this are often called Entity Workflows because they represent durable objects, not just processes.

- [How to Continue-As-New using the Go SDK](/develop/go/workflows/continue-as-new#how)
- [How to Continue-As-New using the Java SDK](/develop/java/workflows/continue-as-new)
- [How to Continue-As-New using the PHP SDK](/develop/php/workflows/continue-as-new)
- [How to Continue-As-New using the Python SDK](/develop/python/workflows/continue-as-new#how)
- [How to Continue-As-New using the TypeScript SDK](/develop/typescript/workflows/continue-as-new)
- [How to Continue-As-New using the .NET SDK](/develop/dotnet/workflows/continue-as-new)
- [How to Continue-As-New using the Ruby SDK](/develop/ruby/workflows/continue-as-new)
- [How to Continue-As-New using the Rust SDK](/develop/rust/workflows/continue-as-new)

## When in your Workflow is it right to Continue-As-New? 

Temporal will tell your Workflow when it's approaching performance or scalability problems.
Find out if it's time by checking Continue-As-New Suggested in your Workflow at spots in your implementation where you are ready to checkpoint your state.

To prevent long-running Workflows from running on stale versions of code, you may also want to Continue-as-New periodically, depending on how often you deploy. This makes sure you're running only a couple of versions, which avoids some backwards compatibility problems.

- [Determine when to Continue-As-New using the Go SDK](/develop/go/workflows/continue-as-new#when)
- [Determine when to Continue-As-New using the Java SDK](/develop/java/workflows/continue-as-new)
- [Determine when to Continue-As-New using the PHP SDK](/develop/php/workflows/continue-as-new)
- [Determine when to Continue-As-New using the Python SDK](/develop/python/workflows/continue-as-new)
- [Determine when to Continue-As-New using the TypeScript SDK](/develop/typescript/workflows/continue-as-new)
- [Determine when to Continue-As-New using the .NET SDK](/develop/dotnet/workflows/continue-as-new)
- [Determine when to Continue-As-New using the Ruby SDK](/develop/ruby/workflows/continue-as-new)
- [Determine when to Continue-As-New using the Rust SDK](/develop/rust/workflows/continue-as-new)

---

# Events and Event History

Source: https://docs.temporal.io/workflow-execution/event

> Events are created by the Temporal Service in response to external occurrences and Commands generated by a Workflow Execution.

This page discusses the following:

- [Events](#event)
- [Activity Events](#activity-events)
- [Event History](#event-history)
- [Event Loop](#event-loop)
- [Time Constraints](#time-constraints)
- [Reset](#reset)
- [Side Effect](#side-effect)
- [Principal Attribution](#principal-attribution)

The Temporal Service tracks the progress of each Workflow Execution by appending information about Events, such as when the Workflow Execution began or ended, to the Event History associated with that execution.
This information not only enables developers to know what took place, but is also essential for providing Durable Execution, since it enables the Workflow Execution to recover from a crash and continue making progress.
In order to maintain high performance, the Temporal Service places limits on both the number and size of items in the Event History for each Workflow Execution.

## What is an Event? 

Events are created by the Temporal Service in response to external occurrences and Commands generated by a Workflow Execution.
Each Event corresponds to an `enum` that is defined in the [Server API](https://github.com/temporalio/api/blob/main/temporal/api/enums/v1/event_type.proto).

All Events are recorded in the [Event History](#event-history).

A list of all possible Events that could appear in a Workflow Execution Event History is provided in the [Event reference](/references/events).

### Activity Events 

Seven Activity-related Events are added to Event History at various points in an Activity Execution:

- After a [Workflow Task Execution](/tasks#activity-task-execution) reaches a line of code that starts/executes an Activity, the Worker sends the Activity Type and arguments to the Temporal Service, and the Temporal Service adds an [ActivityTaskScheduled](/references/events#activitytaskscheduled) Event to Event History.
- When `ActivityTaskScheduled` is added to History, the Temporal Service adds a corresponding Activity Task to the Task Queue.
- A Worker polling that Task Queue picks up the Activity Task and runs the Activity function or method.
- If the Activity function returns, the Worker reports completion to the Temporal Service, and the Temporal Service adds [ActivityTaskStarted](/references/events#activitytaskstarted) and [ActivityTaskCompleted](/references/events#activitytaskcompleted) to Event History.
- If the Activity function throws a [non-retryable Failure](/references/failures#non-retryable), the Temporal Service adds [ActivityTaskStarted](/references/events#activitytaskstarted) and [ActivityTaskFailed](/references/events#activitytaskfailed) to Event History.
- If the Activity function throws an error or retryable Failure, the Temporal Service schedules an Activity Task retry to be added to the Task Queue (unless you’ve reached the Maximum Attempts value of the [Retry Policy](/encyclopedia/retry-policies), in which case the Temporal Service adds [ActivityTaskStarted](/references/events#activitytaskstarted) and [ActivityTaskFailed](/references/events#activitytaskfailed) to Event History).
- If the Activity’s [Start-to-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) passes before the Activity function returns or throws, the Temporal Service schedules a retry.
- If the Activity’s [Schedule-to-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) passes before Activity Execution is complete, or if [Schedule-to-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) passes before a Worker gets the Activity Task, the Temporal Service writes [ActivityTaskTimedOut](/references/events#activitytasktimedout) to Event History.
- If the Activity is [canceled](/activity-execution#cancellation), the Temporal Service writes [ActivityTaskCancelRequested](/references/events#activitytaskcancelrequested) to Event History, and if the Activity accepts cancellation, the Temporal Service writes [ActivityTaskCanceled](/references/events#activitytaskcanceled).

> **📝 Note:**
>
> While the Activity is running and retrying, [ActivityTaskScheduled](/references/events#activitytaskscheduled) is the only Activity-related Event in History: [ActivityTaskStarted](/references/events#activitytaskstarted) is written along with a terminal Event like [ActivityTaskCompleted](/references/events#activitytaskcompleted) or [ActivityTaskFailed](/references/events#activitytaskfailed).
>

### What is an Event History? 

An append-only log of [Events](#event) for your application.

- Event History is durably persisted by the Temporal Service, so your application state survives crashes or failures.
- It also serves as an audit log for debugging.

### Event History limits 

The Temporal Service stores the complete Event History for the entire lifecycle of a Workflow Execution.

The Temporal Service logs a [warning after 10,240 Events](/workflow-execution/limits) and periodically logs additional warnings as new Events are added.

The Workflow Execution is terminated when the Event History:

- exceeds 51,200 Events.
- contains more than 2000 Updates.
- contains more than 10000 Signals.

To avoid hitting these limits, you can use the [Continue-As-New](/workflow-execution/continue-as-new) feature to close the current Workflow Execution and create a new one.

### Event loop 

A Workflow Execution is made up of a sequence of [Events](#event) called an [Event History](#event-history).
Events are created by the Temporal Service in response to either Commands or actions requested by a Temporal Client (such as a request to spawn a Workflow Execution).

![Workflow Execution](https://docs.temporal.io/diagrams/workflow-execution-swim-lane-01.svg)

## Time constraints 

**Is there a limit to how long Workflows can run?**

No, there is no time constraint on how long a Workflow Execution can run.

However, if your Workflow will perform many actions, or will receive many messages, it can run into [Event History limits](#event-history-limits).

It can also hit [Workflow Versioning](/workflow-definition#workflow-versioning) and other backwards incompatibility problems.

For these reasons, it can be a good idea to [Continue-As-New](/workflow-execution/continue-as-new) periodically.

## What is a Reset? 

A Reset terminates a [Workflow Execution](/workflow-execution) and creates a new Workflow Execution with the same [Workflow Type](/workflow-definition#workflow-type) and [Workflow ID](/workflow-execution/workflowid-runid).
The [Event History](/workflow-execution/event#event-history) is copied from the original execution up to and including the reset point.
The new execution continues from the reset point. Valid reset points are: `WorkflowTaskStarted`, `WorkflowTaskCompleted`, `WorkflowTaskTimedOut`, and `WorkflowTaskFailed`.
Signals in the original history can be optionally copied to the new history, whether they appear after the reset point or not.

## What is a Side Effect? 

> **📝 Note:**
>
> Side Effects are included in the Go, Java, and PHP SDKs.
> They are not included in other SDKs.
> [Local Activities](/local-activity) fit the same use case and are slightly less resource intensive.
>

A Side Effect is a way to execute a short, non-deterministic code snippet, such as generating a UUID, that executes the provided function once and records its result into the Workflow Execution Event History.

A Side Effect does not re-execute upon replay, but instead returns the recorded result.

Do not ever have a Side Effect that could fail, because failure could result in the Side Effect function executing more than once.
If there is any chance that the code provided to the Side Effect could fail, use an Activity.

## What is Principal Attribution? 

> **Pre-release**
> Email addresses can be displayed, which may be considered Personally Identifiable information (PII data), and should be handled according to your organization’s privacy, access control, logging, and retention policies.

Principal Attribution lets you see who triggered each event in a Workflow's Event history. For example, you can determine which user or service started a Workflow, sent a Signal, or requested cancellation.

These identity fields are set by the Temporal Service itself based on the authenticated caller, so they can't be forged by client code. To populate them, you need a custom Authorizer that sets the `Principal` field on the authorization result.

This is especially valuable for:

- compliance and audit use cases
- incident investigation and root cause analysis
- access governance and internal accountability

### Temporal Cloud

When enabled, Temporal Cloud populates the `Principal` value (with `Principal Type` and `Principal Name` fields).

Possible values are as follows:

| Type | Name |
| ---- | ---- |
| `users` | user email address |
| `service-accounts` | service account name |
| `mtls` | Common Name (CN) or Subject Domain Name (DN) if CN is not present |
| `temporal` | Temporal internal services |

Anyone who has permission to read Event history in the Namespace (ReadOnly access and above) can see the Principal (and the metadata such as email address).

To enable Principal Attribution for a Namespace, contact [Temporal Cloud support](/evaluate/cloud/support#support-ticket).

### Self-hosted Temporal

In self-hosted Temporal, you can control Principal Attribution with a dynamic config flag scoped to the Namespace. 
When enabled, the Principal returned by the `Authorizer` is stamped on Event history events.
To enable, set `frontend.enablePrincipalPropagation` to `true` for the appropriate Namespace.

When using the default `Authorizer` with the default JWT `ClaimMapper`, the following values are populated:

| Type | Name |
| ---- | ---- |
| `jwt ` | value of the JWT `sub` claim |
| `temporal` | `internal` (for internal frontend requests) |

A custom `Authorizer` must set the Principal field on `authorization.Result` for the request to be attributed.
Custom `ClaimMapper` implementations control the `AuthType` and `Subject` values that the default `Authorizer` then copies into the `Principal`.

---

# Workflow Execution limits

Source: https://docs.temporal.io/workflow-execution/limits

> The Workflow Execution's Event History is limited to 51,200 Events or 50 MB and will warn you after 10,240 Events or 10 MB.

This page discusses [Workflow Execution limits](#workflow-execution-limits), [Workflow Execution Callback limits](#workflow-execution-callback-limits), and [Nexus Operation limits](#workflow-execution-nexus-operation-limits).

## Limits 

There is no limit to the number of concurrent Workflow Executions, albeit you must abide by the Workflow Execution's Event History limit.

> **⚠️ Caution:**
>
> As a precautionary measure, the Workflow Execution's Event History is limited to [51,200 Events](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L441) or [50 MB](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L425) and will warn you after 10,240 Events or 10 MB.
>

There is also a limit to the number of certain types of incomplete operations.

Each in-progress Activity generates a metadata entry in the Workflow Execution's mutable state.
Too many entries in a single Workflow Execution's mutable state causes unstable persistence.
To protect the system, Temporal enforces a maximum number of incomplete Activities, Child Workflows, Signals, or Cancellation requests per Workflow Execution (by default, 2,000 for each type of operation).
Once the limit is reached for a type of operation, if the Workflow Execution attempts to start another operation of that type (by producing a `ScheduleActivityTask`, `StartChildWorkflowExecution`, `SignalExternalWorkflowExecution`, or `RequestCancelExternalWorkflowExecution` Command), it will be unable to (the Workflow Task Execution will fail and get retried).

These limits are set with the following [dynamic configuration keys](https://github.com/temporalio/temporal/blob/main/service/history/configs/config.go):

- `NumPendingActivitiesLimit`
- `NumPendingChildExecutionsLimit`
- `NumPendingSignalsLimit`
- `NumPendingCancelRequestsLimit`

## Workflow Execution Callback limits 

There is a limit to the total number of Workflow Callbacks that may be attached to a single Workflow Execution.
Attaching [multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) may exceed this limit.

Set this limit with the [MaxCallbacksPerWorkflow](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L1082) dynamic configuration key.

## Workflow Execution Nexus Operation Limits 

There is a limit to the maximum number of Nexus Operations in a Workflow before Continue-As-New is required.
Each in-progress Nexus Operation generates a metadata entry in the Workflow Execution's mutable state.
Too many entries in a single Workflow Execution's mutable state causes unstable persistence.
To protect the system, Temporal enforces a maximum number of incomplete Nexus Operation requests per Workflow Execution (by default, 30 Nexus Operations).
Once the limit is reached for a type of operation, if the Workflow Execution attempts to start another Nexus operation (by producing a ScheduleNexusOperation), it will be unable to do so (the Workflow Task Execution will fail and get retried).

These limits are set with the following [dynamic configuration keys](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/components/nexusoperations/config.go#L36):

- MaxConcurrentOperations

---

# Timers and Start Delays

Source: https://docs.temporal.io/workflow-execution/timers-delays

> Temporal SDKs offer Timer APIs so that the Workflow Definition is deterministic in its handling of time values.

This page discusses [Timer](#timer) and [Start Delay](#delay-workflow-execution).

## What is a Timer? 

Temporal SDKs offer Timer APIs so that the Workflow Definition is deterministic in its handling of time values.

Timers in Temporal are persisted, meaning that even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service become available, the call that is awaiting the Timer in your Workflow code will resolve, causing execution to proceed.
Timers are reliable and efficient.
Workers consume no additional resources while waiting for a Timer to fire, so a single Worker can await millions of Timers concurrently.

- [How to set Timers in .NET](/develop/dotnet/workflows/timers)
- [How to set Timers in Go](/develop/go/workflows/timers)
- [How to set Timers in Java](/develop/java/workflows/timers)
- [How to set Timers in PHP](/develop/php/workflows/timers)
- [How to set Timers in Python](/develop/python/workflows/timers)
- [How to set Timers in Ruby](/develop/ruby/workflows/timers)
- [How to set Timers in Rust](/develop/rust/workflows/timers)
- [How to set Timers in TypeScript](/develop/typescript/workflows/timers)

The duration of a Timer is fixed, and your Workflow might specify a value as short as one second or as long as several years. Although it's possible to specify an extremely precise duration, such as 36 milliseconds or 15.072 minutes, your Workflows should not rely on sub-second accuracy for Timers.

We recommend that you consider the duration as a minimum time, one which will be rounded up slightly due to the latency involved with scheduling and firing the Timer.
For example, setting a Timer for 11.97 seconds is guaranteed to delay execution for at least that long, but will likely be closer to 12 seconds in practice.

## What is a Start Delay? 

> **💡 Tip:**
> COMPATIBILITY
>
> Start Delay Workflow Execution is incompatible with both [Schedules](/schedule) and [Cron Jobs](/cron-job).
>

Start Delay determines the amount of time to wait before initiating a Workflow Execution.
This is useful if you have a Workflow you want to schedule out in the future, but only want it to execute once: in comparison to reoccurring Workflows using Schedules.

If the Workflow receives a Signal-With-Start or Update-With-Start during the delay, it dispatches a Workflow Task and the remaining delay is bypassed. If the Workflow receives a [Signal](/sending-messages#sending-signals) during the delay that is not a Signal-With-Start, the Signal does not interrupt the delay, and the Workflow continues to be delayed until the delay expires or a Signal-With-Start is received.

A delayed-start Workflow can be triggered with an [Update](/sending-messages#sending-updates). This is because a Workflow Task needs to be scheduled to deliver an Update to Worker. Once a Workflow Task is scheduled, the Workflow is unblocked. Delay start works by not scheduling the first Workflow Task on Workflow creation.

You can delay the dispatch of the initial Workflow Execution by setting this option in the Workflow Options field of your chosen SDK. This delay only applies to the initial Workflow Execution and does not affect subsequent executions, such as when the Workflow Continues-as-New.

---

# Workflow Id and Run Id

Source: https://docs.temporal.io/workflow-execution/workflowid-runid

> A Workflow Id is a customizable, application-level identifier for a Workflow Execution that is unique to an Open Workflow Execution within a Namespace.

This page discusses the following:

- [Run Id](#run-id)
- [Operations leading to non-determinism](#run-id-non-determinism)
- [Workflow Id](#workflow-id)
- [Workflow Id Reuse Policy](#workflow-id-reuse-policy)
- [Workflow Id Conflict Policy](#workflow-id-conflict-policy)

Each Workflow Execution is associated with a user-defined [Workflow ID](#workflow-id), a value which typically carries some business meaning (such as an order number or customer number).
Temporal guarantees that there can be at most one Workflow Execution with a given ID running at any point in time, a constraint that helps to protect against unexpected duplication.
In some cases, such as when running the same Workflow at recurring intervals using the Schedules features, there can be multiple "runs" of a single Workflow Execution over a period of time.
In this case, all runs will have the same Workflow ID.
However, each run will have a unique system-generated [Run ID](#run-id).

> **📝 Note:**
> Workflow Id uniqueness and High Availability
>
> This uniqueness constraint is enforced within the active [Namespace](/namespaces).
> On a [High Availability](/cloud/high-availability) Namespace, replication is asynchronous, and a failover preserves the constraint through Temporal Cloud's [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) process.
>

## What is a Run Id? 

A Run Id is a globally unique, platform-level identifier for a [Workflow Execution](/workflow-execution).

The current Run Id is mutable and can change during a [Workflow Retry](/encyclopedia/retry-policies). You shouldn't rely on storing the current Run Id, or using it for any logical choices, because a Workflow Retry changes the Run Id and can lead to non-determinism issues.

Temporal guarantees that only one Workflow Execution with a given [Workflow Id](#workflow-id) can be in an Open state at any given time.
But when a Workflow Execution reaches a Closed state, it is possible to have another Workflow Execution in an Open state with the same Workflow Id.
For example, a Temporal Cron Job is a chain of Workflow Executions that all have the same Workflow Id.
Each Workflow Execution within the chain is considered a _Run_.

A Run Id uniquely identifies a Workflow Execution even if it shares a Workflow Id with other Workflow Executions.

### Which operations lead to non-determinism issues? 

An operation like `ContinueAsNew`, `Retry`, `Cron`, and `Reset` creates a [Workflow Execution Chain](/workflow-execution#workflow-execution-chain) as identified by the [`first_execution_run_id`](https://github.com/temporalio/api/blob/main/temporal/api/history/v1/message.proto).

Each operation creates a new Workflow Execution inside a chain run; the chain remains identified by `first_execution_run_id`.
Thus, the current Run Id is updated during each operation on a Workflow Execution.

- The `first_execution_run_id` is the Run Id of the first Workflow Execution in a Chain run.
- The `original_execution_run_id` is the Run Id when the `WorkflowExecutionStarted` Event occurs.

A Workflow `Reset` changes the current execution Run Id, but preserves both the first execution Run Id and the original execution Run Id.
For example, when a new Workflow Execution in the chain starts, it stores its Run Id in `original_execution_run_id`.
A reset doesn't change that field, but the current Run Id is updated.

> **⚠️ Caution:**
>
> Because of this behavior, you shouldn't rely on the current Run Id in your code to make logical choices.
>

**Learn more**

For more information, see the following link.

- [`message.proto`](https://github.com/temporalio/api/blob/main/temporal/api/history/v1/message.proto#L75-L82)

## What is a Workflow Id? 

A Workflow Id is a customizable, application-level identifier for a [Workflow Execution](/workflow-execution) that is unique to an Open Workflow Execution within a [Namespace](/namespaces).

- [How to set a Workflow Id](/develop/go/client/temporal-client#workflow-id)

A Workflow Id is meant to be a business-process identifier, such as customer identifier or order identifier.

> **⚠️ Caution:**
> Do not use sensitive data or PII in Workflow Ids
>
> Do not include sensitive data, secrets, or personally identifiable information (PII) as a Workflow Id.
> Workflow Ids are stored in plain text, are **not** processed by a custom [Payload Codec](/payload-codec#payload-codec), and are visible in the Temporal Web UI, CLI output, Event History, and system logs.
> The same applies to other user-defined identifiers such as Workflow Type names, Task Queue names, Activity names, and Signal/Query/Update names.
>
> Using sensitive data in these identifiers risks exposure to anyone with Namespace access and may violate data protection regulations such as GDPR, HIPAA, or SOC 2.
>

The Temporal Platform guarantees uniqueness of the Workflow Id within a [Namespace](/namespaces) based on the Workflow Id Reuse Policy.

A [Workflow Id Reuse Policy](#workflow-id-reuse-policy) can be used to manage whether a Workflow Id from a Closed Workflow can be re-used.

A [Workflow Id Conflict Policy](#workflow-id-conflict-policy) can be used to decide how to resolve a Workflow Id conflict with a Running Workflow.

A Workflow Execution can be uniquely identified across all Namespaces by its [Namespace](/namespaces), Workflow Id, and [Run Id](#run-id).

### What is a Workflow Id Reuse Policy? 

A Workflow Id Reuse Policy determines whether a Workflow Execution is allowed to spawn with a particular Workflow Id, if that Workflow Id has been used with a previous, and now Closed, Workflow Execution.

It is not possible for a new Workflow Execution to spawn with the same Workflow Id as another Open Workflow Execution, regardless of the Workflow Id Reuse Policy.

See [Workflow Id Conflict Policy](#workflow-id-conflict-policy) for resolving a Workflow Id conflict.

The Workflow Id Reuse Policy can have one of the following values:

- **Allow Duplicate:** The Workflow Execution is allowed to exist regardless of the Closed status of a previous Workflow Execution with the same Workflow Id.
  **This is the default policy, if one is not specified.**
  Use this when it is OK to have a Workflow Execution with the same Workflow Id as a previous, but now Closed, Workflow Execution.
- **Allow Duplicate Failed Only:** The Workflow Execution is allowed to exist only if a previous Workflow Execution with the same Workflow Id does not have a Completed status.
  Use this policy when there is a need to re-execute a Failed, Timed Out, Terminated, or Cancelled Workflow Execution and guarantee that the Completed Workflow Execution will not be re-executed.
- **Reject Duplicate:** The Workflow Execution cannot exist if a previous Workflow Execution has the same Workflow Id, regardless of the Closed status.
  Use this when there can only be one Workflow Execution per Workflow Id within a Namespace for the given retention period.

The first three values (Allow Duplicate, Allow Duplicate Failed Only, and Reject Duplicate) of the Workflow Id Reuse Policy apply to Closed Workflow Executions that are retained within the Namespace.
For example, given a default Retention Period, the Temporal Service can only check the Workflow Id of the spawning Workflow Execution based on the Workflow Id Reuse Policy against the Closed Workflow Executions for the last _30 days_.

If you need to start a Workflow for a particular implementation only if it hasn't started yet, ensure that your Retention Period is long enough to check against.
If this becomes unwieldy, consider using [Workflow message passing](/encyclopedia/workflow-message-passing) instead of trying to start Workflows atomically.

The fourth value of the Workflow Id Reuse Policy, Terminate if Running, only applies to a Workflow Execution that is currently open within the Namespace.
For Terminate if Running, the Retention Period is not a consideration for this policy.

If there is an attempt to spawn a Workflow Execution with a Workflow Id Reuse Policy that won't allow it, the Server will prevent the Workflow Execution from spawning.

### What is a Workflow Id Conflict Policy? 

A Workflow Id Conflict Policy determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution.
See [Workflow Id Reuse Policy](#workflow-id-reuse-policy) for managing the reuse of a Workflow Id of a Closed Workflow.

By default, this results in a `Workflow execution already started` error.

> **📝 Note:**
>
> The default [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions) behavior in the Go SDK is to not return an error when a new Workflow Execution is attempted with the same Workflow Id as an Open Workflow Execution.
> Instead, it returns a WorkflowRun instance representing the current or last run of the Open Workflow Execution.
>
> To return the `Workflow execution already started` error, set `WorkflowExecutionErrorWhenAlreadyStarted` to `true`.
>

The Workflow Id Conflict Policy can have one of the following values:

- **Fail:** Prevents the Workflow Execution from spawning and returns a `Workflow execution already started` error.
  **This is the default policy, if one isn't specified.**
- **Use Existing:** Prevents the Workflow Execution from spawning and returns a successful response with the Open Workflow Execution's Run Id.
- **Terminate Existing:** Terminates the Open Workflow Execution then spawns the new Workflow Execution with the same Workflow Id.

---

# Workflow Streams

Source: https://docs.temporal.io/workflow-streams

> **Public Preview**

This page covers the following:

- [What are Workflow Streams?](#what-are-workflow-streams)
- [How Workflow Streams works](#how-workflow-streams-works)
- [Choose where to host the stream](#choose-where-to-host-the-stream)
- [Closing the stream](#closing-the-stream)
- [How events are delivered](#how-events-are-delivered)
- [Tuning](#tuning)
- [Stream from long-running Workflows](#stream-from-long-running-workflows)

## What are Workflow Streams? 

A Workflow Stream is a durable event channel hosted inside a Workflow. Publishers append events to topics on the stream.
Subscribers attach to the Workflow by its Workflow Id, optionally filter by topic, and consume events by long-polling.
Subscribers can disconnect and resume from where they left off.

Use Workflow Streams when outside observers need to follow the progress of a Workflow and its Activities as work
happens. For example, updating a UI as an AI agent works, surfacing status from a payment or order pipeline, or reporting
intermediate results from a data job.

Workflow Streams isn't suited to ultra-low-latency cases like real-time voice. It targets modest fan-out: tens of
publishers and subscribers per Workflow, not thousands.

**Related:**

- [Workflow Streams with the Python SDK](/develop/python/workflows/workflow-streams)
- [Workflow Streams with the TypeScript SDK](/develop/typescript/workflows/workflow-streams)
- [Workflow Streams with the Go SDK](/develop/go/workflows/workflow-streams)
- [Workflow Streams with the Java SDK](/develop/java/workflows/workflow-streams)
- [Workflow Streams with the .NET SDK](/develop/dotnet/workflows/workflow-streams)

## How Workflow Streams works

The stream lives inside a Workflow, not on a separate broker.
A Workflow creates the stream by constructing a `WorkflowStream` object, which sets up an in-memory, append-only event
log and registers [Signal](/sending-messages#sending-signals), [Update](/sending-messages#sending-updates), and
[Query](/sending-messages#sending-queries) handlers that publishers and subscribers interact with.
A Workflow can have at most one stream.
The Workflow Id is the address that publishers and subscribers use to connect.

### Publishing

A **publisher** appends events to the stream.
The Workflow itself, its Activities, and external processes can all publish to the same stream:

- **The Workflow itself** appends events synchronously to the in-memory log.
  Events are immediately available to subscribers on the next poll.
- **Activities** scheduled by the Workflow use a client that infers the Temporal Client and parent Workflow Id from
  the Activity context.
- **External processes** (HTTP backends, starters, scripts, Activities of other Workflows) use a client constructed with
  an explicit Temporal Client and Workflow Id.

Activities and external processes publish through a client-side buffer. The client accumulates events in memory and
flushes the buffer as a single Signal on a configurable batch interval (default: 2 seconds). A single flush batches all
events across all topics that have accumulated since the last flush. This amortizes the cost of Signals: instead of one
Signal per event, one Signal carries an entire batch.

To send a specific event without waiting for the next interval, mark the publish with a `force-flush` flag. The
`force-flush` flag wakes the background flusher immediately. The call returns after appending to the buffer and signaling
the flusher; it doesn't wait for delivery.

Each client has a unique publisher Id and a monotonic sequence number. Every batch is tagged with this pair so that a
Signal retried by the SDK or the network deduplicates to a single landing in the log. See
[How events are delivered](#how-events-are-delivered) for the full guarantee.

**Related:**

- [Publish from a Workflow with the Python SDK](/develop/python/workflows/workflow-streams#publish-from-a-workflow)
- [Publish from a client with the Python SDK](/develop/python/workflows/workflow-streams#publish-from-a-client)
- [Publish from a Workflow with the TypeScript SDK](/develop/typescript/workflows/workflow-streams#publish-from-a-workflow)
- [Publish from a client with the TypeScript SDK](/develop/typescript/workflows/workflow-streams#publish-from-a-client)
- [Publish from a Workflow with the Go SDK](/develop/go/workflows/workflow-streams#publish-from-a-workflow)
- [Publish from a client with the Go SDK](/develop/go/workflows/workflow-streams#publish-from-a-client)
- [Publish from a Workflow with the Java SDK](/develop/java/workflows/workflow-streams#publish-from-a-workflow)
- [Publish from a client with the Java SDK](/develop/java/workflows/workflow-streams#publish-from-a-client)
- [Publish from a Workflow with the .NET SDK](/develop/dotnet/workflows/workflow-streams#publish-from-a-workflow)
- [Publish from a client with the .NET SDK](/develop/dotnet/workflows/workflow-streams#publish-from-a-client)

### Topics

A **topic** is a string label attached to each event when published. Topics are implicit: they are created on first
publish, not declared ahead of time.

A topic handle binds a name to a type so that publish and subscribe call sites carry the type with them. Subscribers can
filter by one or more topic names, or subscribe to all topics on the stream.

Multiple publishers can publish to the same topic. Ordering is guaranteed within a single publisher. Across publishers,
events interleave in the order the Workflow receives the Signals.

### Subscribing

A **subscriber** is any process with a Temporal Client that long-polls the Workflow for new events.
Each poll is an Update that blocks until events are available past the subscriber's current offset, then returns a batch.

The subscriber maintains its own offset.
On reconnect, the subscriber resumes from its last offset without coordinating with anyone but the Workflow.
Multiple subscribers can attach to the same Workflow concurrently.

Poll responses are capped at roughly 1 MB.
When a response hits the cap, the subscriber polls again immediately to drain the rest before applying its cooldown.

Subscribing from inside the Workflow that hosts the stream isn't supported.
The Workflow processes only the successful return value of each Activity, while the stream may carry partial output from
attempts that failed and were retried.
Letting the Workflow read its own stream would mix those two views.

The Workflow is the single source of truth for stream state, so any process bridging events to the outside world (an SSE proxy serving a browser, a forwarding Activity) can stay stateless. You store the last delivered item offset and any reconnects resume from that offset without coordinating with anything other than the Workflow.

**Related:**

- [Subscribe with the Python SDK](/develop/python/workflows/workflow-streams#subscribe)
- [Subscribe with the TypeScript SDK](/develop/typescript/workflows/workflow-streams#subscribe)
- [Subscribe with the Go SDK](/develop/go/workflows/workflow-streams#subscribe)
- [Subscribe with the Java SDK](/develop/java/workflows/workflow-streams#subscribe)
- [Subscribe with the .NET SDK](/develop/dotnet/workflows/workflow-streams#subscribe)

### Log management

Two mechanisms bound the growth of the in-memory log:

- **Truncation** drops entries below a given offset from the in-memory log (and from the Continue-As-New payload). It
  doesn't remove publish Signals already recorded in Workflow history. A subscriber whose offset falls below the new
  base after truncation is silently advanced to the current base.
- **Continue-As-New** starts a fresh Workflow history. This is the only way to shrink history. See
  [Stream from long-running Workflows](#stream-from-long-running-workflows).

### Architecture

The user-facing API abstracts three pieces of machinery worth understanding when you tune throughput, debug delivery, or reason about history size.

#### Append-only log inside the Workflow

A Workflow Stream client keeps an in-memory list of `(topic, data)` entries inside the Workflow's state, each with a monotonically increasing global offset. Subscribers maintain their own cursor and on each long-poll receive the next range past it. Because the log lives in Workflow state, it is replay-safe and is carried across Continue-As-New via Workflow Stream state.

Two mechanisms bound log growth and they do different jobs:

- `truncate(up_to_offset)` drops entries from the in-memory log (and therefore from the carried Continue-As-New payload). It doesn't remove publish Signals already recorded in history.
- Continue-As-New starts a fresh history. This is the only way to shrink history because truncate alone can't.

A subscriber whose offset falls below the new base after a `truncate()` is silently advanced. Internally, the poll raises a `TruncatedOffset` error. The SDK client catches it and resets to offset 0, which the Workflow reads as "from the current base." The iterator doesn't raise an exception, but the subscriber may re-receive items already in the log past the new base. Applications that depend on seeing every event exactly once must keep subscribers ahead of truncation or implement their own gap and re-delivery handling using the item offset.

#### Wire-level handlers

The three handlers registered when you construct a `WorkflowStream` are:

- `__temporal_workflow_stream_publish` (the Signal that receives batched publishes)
- `__temporal_workflow_stream_poll` (the long-poll Update that subscribers use)
- `__temporal_workflow_stream_offset` (the Query that reports the current head offset)

Poll responses are capped at roughly 1 MB by accumulating items until the next would exceed the budget, so high-throughput producers see a steady stream of small batches. A single item that exceeds 1 MB on its own is admitted unconditionally. Offload large items via [External Storage](/external-storage) so each item is a small reference.

#### Batching and deduplicating

Every batch carries a unique identifier (the client's Id paired with a monotonic batch sequence number), so a Signal retried by the SDK or the network deduplicates to a single landing in the Workflow's event log. Deduplicate state is part of the Workflow's carried state, so the guarantee survives Continue-As-New (subject to `publisher_ttl`).

This dedup applies at the Signal layer, not the Activity layer. An *Activity retry* is a different concept from a *publish retry*. When Temporal retries the Activity, the retried execution constructs a new Workflow Stream client with its own client Id, so from the stream's perspective every attempt is a fresh publisher whose batches will not deduplicate against the prior attempt's. That is why retried-attempt events appear in the stream alongside the successful attempt's output.

## Choose where to host the stream

A `WorkflowStream` is hosted inside a Workflow, so the first design choice is whether to use a Workflow that handles both the work and the stream or use a dedicated Workflow for the stream.

**Host the stream on the Workflow that does the work** when the events come from what that Workflow is already
orchestrating: an agent run, an order pipeline, a chat session. The stream's lifecycle aligns with the work. The
Workflow Id that starts the work is the same one subscribers attach to.

**Use a dedicated Workflow for the stream** when the stream should outlive any single producer, accept fan-in from
multiple unrelated sources, or be subscribable before any work has started. Producers publish from outside the stream
Workflow (Activities of other Workflows, or external clients). The trade-off is explicit lifecycle management: a
dedicated stream Workflow doesn't terminate on its own, so you need a Signal-driven shutdown or a Continue-As-New
strategy.

Use distinct Workflow Ids for unrelated streams rather than packing them into one Workflow.

## Closing the stream

A subscriber's iterator doesn't know when the publisher is done. End-of-stream is an application-level concern. Without
coordination, a subscriber keeps polling until the Workflow reaches a terminal state.

A Workflow that returns immediately after its last publish can return before the subscriber's next poll fetches that
event. Two patterns handle this.

### Fixed sleep

Sleep between the last publish and the Workflow return so any in-flight poll has time to fetch the final event:

1. The Workflow or its Activity publishes a sentinel event the subscriber recognizes.
    - For example: `{ state: "completed" }`
2. The Workflow sleeps for a duration long enough to cover the subscriber's poll round-trip. 30 seconds is a good default.
3. The Workflow returns.

The cost is small: the Workflow stays open for the sleep duration but does no other work.

### Acknowledgment handshake

The subscriber sends a Signal to the Workflow once it has received the sentinel event. The Workflow waits for that
Signal up to a timeout, returning as soon as the ack arrives:

1. The Workflow or its Activity publishes a sentinel event.
2. The subscriber receives the sentinel and signals the Workflow.
3. The Workflow's wait condition resolves and the Workflow returns.

The timeout is still required because the subscriber may not be attached. With the ack, the typical case (subscriber
online) exits as soon as the subscriber confirms receipt.

### Inspecting terminal status

The subscribe iterator exits cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELLED`, `TERMINATED`, or
`TIMED_OUT`, but doesn't distinguish among them. If your application needs to know which, describe the Workflow handle
after the loop returns to inspect the status.

## How events are delivered

Workflow Streams provides exactly-once publishing, total ordering within a publisher, and transparent handling of Activity retries. This section covers each guarantee and the failure modes to be aware of.

### Exactly-once publishing

Each `(publisher_id, sequence)` batch lands in the log at most once, even if the publisher's underlying Signal is
retried by the SDK or the network. Once an event is in the log, every subscriber that polls past its offset sees it.
Deduplicate state is carried across Continue-As-New, so a retried publish that arrives after a rollover still lands at
most once.

### Ordering

The log imposes a single total order on all events, fixed once written: an event at offset _N_ stays at offset _N_ on every
read. Within one publisher (one Workflow Stream client instance or the Workflow itself), events appear in publish order. Across
concurrent publishers, the interleaving is what the Workflow saw when serializing inbound Signals. The order is
stable once recorded, but not under application control. If event A must precede event B, publish them from the same
publisher.

### Activity retries surface to subscribers

When an Activity that publishes events fails partway through and Temporal retries it, events for both attempts appear in
the stream. An Activity that publishes three events and then errors, then retries and publishes its full output,
delivers three partial events followed by the complete sequence. The Workflow itself sees only the successful attempt's
return value, but a subscriber sees output from all attempts.

Consumers must handle this. The conventional pattern is for an Activity that detects it is on a retry attempt to publish
a retry-sentinel event with `force-flush`. The consumer clears or annotates prior-attempt output when it sees the
sentinel. Because the Workflow processes only Activity return values rather than reading the stream itself, its own state stays independent of these retried events.

This is the trade-off of streaming events as they happen rather than waiting for the Workflow's durable view to settle. If
the library waited for a successful Activity return before surfacing anything, there would be nothing to stream.

### Other failure modes

- Events still in a publisher's in-memory buffer are lost if the process crashes before they ship.
- Subscribers that handle an item and crash before persisting their next offset reprocess that item on resume.

### Deduplication window

Two settings control the deduplication window:

| Setting                | Default    | Description                                                                                                                |
| ---------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------- |
| **publisher TTL**      | 15 minutes | How long the Workflow retains per-publisher deduplicate state. Entries older than this are pruned at each Continue-As-New. |
| **max retry duration** | 10 minutes | How long a Workflow Stream client retries a failed publish batch before dropping it and raising an error.                                  |

These two settings must satisfy **max retry duration < publisher TTL**. If a publisher's retry window exceeds the dedup
retention, the dedup state can age out before the retry lands. A retry that arrives after its dedup record has been
pruned is treated as a fresh publish, producing a duplicate. If you tune one, tune the other to keep the relationship between them.

## Tuning

The most important question when tuning is: how often do you want to update your UI? That answer drives the trade-off
between user-perceived latency and the number of history events your Workflow accumulates.

Each batched publish is one Signal and each subscriber poll is one Update. Each Signal and each Update counts against
the Workflow's history. A more responsive UI means more messages and more history per second. To plan for long-running streams, refer to [Stream from long-running Workflows](#stream-from-long-running-workflows) for details on how to use [Continue-As-New](/workflow-execution/continue-as-new) to avoid performance degradation from large event histories.

### Batch interval

The batch interval (default: 2 seconds) is the maximum time between automatic flushes from the client. Lower it to make
the stream feel live. Raise it to amortize Signal cost. For an LLM token stream feeding a chat UI, 200 milliseconds is a
good starting point: the user perceives it as live and a 30-second response generates roughly 150 publish Signals.
Below 100 milliseconds, the per-Signal RPC overhead starts to dominate.

For per-publish overrides where one event needs lower latency than the batch interval (the first delta of a response, or
punctuated events like retry sentinels), set `force-flush` to `true` on that publish. Per-token `force-flush` on a 500-token
completion produces 500 publish Signals, which is meaningful but tractable. Per-character `force-flush` isn't.

### Other settings

| Setting            | Default   | Description                                                                                                                                                                                                       |
| ------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **max batch size** | unbounded | Caps the number of items per batch. Without this, only batch interval bounds batch size. A hot publisher can accumulate enough items that the resulting Signal exceeds Temporal's per-message gRPC payload limit. For large items, offload via [External Storage](/external-storage) so each item is a small reference. |
| **poll cooldown**  | 100 ms    | Minimum interval between subscriber polls. Skipped only when a poll response was capped at the 1 MB limit and more items remain.                                                                                  |

## Stream from long-running Workflows

If your Workflow runs for minutes and finishes (a single chat completion, an order pipeline, a one-shot agent), you can
skip this section. Long-running streams that run for hours or accumulate thousands of events need to periodically roll
over via Continue-As-New to keep history bounded.

Subscribers automatically follow Continue-As-New chains. Workflow Ids are stable across Continue-As-New, so the
subscriber fetches a fresh handle for the same Workflow Id and continues polling from its carried offset.

To roll a long-running streaming Workflow over without subscribers seeing a gap:

1. Add an optional stream-state field to your Workflow input. Pass it to the `WorkflowStream` constructor. On a fresh
   start, this field is empty. After a rollover, it carries the accumulated state.
2. When the Workflow decides to roll over (for example, when `continueAsNewSuggested` is true), call the stream's
   Continue-As-New helper. The helper drains waiting subscribers, waits for in-flight handlers to finish, then calls
   `continueAsNew` with the snapshot.

The carried state includes the entire in-memory log of the previous run. Streams that carry large items can hit
Temporal's per-payload size limit at the rollover. To keep the carried state small, offload large payloads via external
storage so each item is a small reference, and use truncation to drop entries that subscribers have already consumed.

**Related:**

- [Long-running streams with the Python SDK](/develop/python/workflows/workflow-streams#continue-as-new)
- [Long-running streams with the TypeScript SDK](/develop/typescript/workflows/workflow-streams#continue-as-new)
- [Long-running streams with the Go SDK](/develop/go/workflows/workflow-streams#continue-as-new)
- [Long-running streams with the Java SDK](/develop/java/workflows/workflow-streams#continue-as-new)
- [Long-running streams with the .NET SDK](/develop/dotnet/workflows/workflow-streams#continue-as-new)

---

# Temporal Workflow

Source: https://docs.temporal.io/workflows

This guide provides a comprehensive overview of Temporal Workflows and covers the following:

- [Workflow Definition](/workflow-definition)
- [Workflow Execution](/workflow-execution)
- [Schedules](/schedule)
- [Dynamic Handler](/dynamic-handler)
- [Cron Job](/cron-job)

## Intro to Workflows

> **💡 Tip:**
>
> Watch a short overview of what a Workflow is in Temporal:
>
> [Watch: What Is a Workflow in Temporal?](https://www.youtube.com/watch?v=zLjhNrOKphE)
>

Conceptually, a workflow defines a sequence of steps. With Temporal, those steps are defined by writing code, known as a Workflow Definition, and are carried out by running that code, which results in a Workflow Execution.

In day-to-day conversations, the term Workflow might refer to Workflow Type, a Workflow Definition, or a Workflow Execution.

1. A **Workflow Definition** is the code that defines your Workflow.
2. The **Workflow Type** is the name that maps to a Workflow Definition. It's an identifier that makes it possible to distinguish one type of Workflow (such as order processing) from another (such as customer onboarding).
3. A **Workflow Execution** is a running Workflow, which is created by combining a Workflow Definition with a request to execute it. You can execute a Workflow Definition any number of times, potentially providing different input each time (that is, a Workflow Definition for order processing might process order #123 in one execution and order #567 in another execution). It is the actual instance of the Workflow Definition running in the Temporal Platform.

You'll develop those Workflows by writing code in a general-purpose programming language such as Go, Java, TypeScript, or Python. The code you write is the same code that will be executed at runtime, so you can use your favorite tools and libraries to develop Temporal Workflows.

Temporal Workflows are resilient. They can run—and keep running—for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off.

Each Workflow Execution emits a series of **Commands** and processes a sequence of **Events**, which are recorded in an **Event History**.

### How Workflow replay works

When a Workflow [yields](https://en.wikipedia.org/wiki/Yield_(multithreading)) or encounters an error, the goal of Temporal is to bring the Workflow back to the exact same state it was in before the pause occurred. To make that possible, Temporal keeps the Event History. This is a complete, ordered log of everything that has already happened in a Workflow.

The Event History could look like this for example:

- Started Timer for 5 minutes
- Scheduled Activity X
- Activity X completed with result Y
- Received Signal Z

This history is the source of truth for everything that happens in the Workflow.

#### Resuming a Workflow

When it's time to continue the Workflow, Temporal doesn't restore memory from a snapshot. It starts the Workflow code from the beginning, replays the Event History step by step, and uses that history to guide the code back to the exact state as before. So the Workflow code is re-run, but uses the recorded events instead of redoing work. Although Temporal doesn't always have to start from the beginning if the state is cached.

Because the Workflow is re-executed to rebuild its state:

- It has to make the same decisions when given the same history.
- It shouldn't depend on any values _not_ recorded in the history which would be different between runs.

For example:

- A direct call to `Date.now()` could return a different value on replay.
- A random number could change.
- A network call, which wasn't performed inside an Activity, could return something new.

If those values changed, the Workflow could take a different path and fail to match the recorded history. To solve this, Temporal provides replay-safe versions of common operations:

- Time is read from the Workflow context so it matches the recorded history.
- Timers are recorded as events and don’t “wait” again during replay.
- Randomness and similar values can be captured once and reused.

These APIs make sure the Workflow receives the same values during replay as it did originally. Activities handle everything that interacts with the outside world, like:

- API calls
- Database queries
- LLM invocations
- File I/O

When a Workflow calls an Activity, the Activity runs once, its result is recorded in the Event History. During replay, that result is reused, not recomputed. So Activities aren't executed again during replay.